Yehjin Shin

2025

1. ICML Workshop

   Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation

   Yehjin Shin, Seojin Kim, and Noseong Park

   In Workshop on Efficient Systems for Foundation Models III at International Conference on Machine Learning, 2025

   Abs PDF

   State-space models (SSMs) provide efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, employs selective input gating and a multi-head structure for parallel computation and strong performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we introduce \textbfHierarchical \textbfADaptive filter bank for \textbfEfficient \textbfSSMs (\textitHADES), a GSP-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. \textitHADES features two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved via structured bias on parameter . \textitHADES matches baseline models, including Mamba2, on key benchmark tasks while using only \textbf58.9% of the original parameters.

2. ICML Workshop

   Mitigating Over-Smoothing in Mamba2 via Spectral Domain Analysis

   Seojin Kim, Yehjin Shin, and Noseong Park

   In Workshop on Efficient Systems for Foundation Models III at International Conference on Machine Learning, 2025

   Abs PDF

   Mamba2, a rising contender against transformer-based architectures, has garnered significant attention for its impressive performance across diverse tasks, sparking a wave of research into its analysis and improvement. In this paper, we investigate Mamba2 through the lens of spectral analysis, uncovering a critical structural bias: Mamba2 inherently functions as a low-pass filter, leading to over-smoothing. Over-smoothing, where token representations become overly uniform, hampers the model’s ability to capture rich and diverse features, ultimately contributing to performance degradation. To address this, we propose a straightforward yet effective high-frequency enhancement method. By selectively amplifying high-frequency components at the layer level, our approach mitigates the over-smoothing effect, restoring token diversity and improving representational richness. Experiments confirm the efficacy of our method, demonstrating its ability to enhance Mamba2’s performance across key tasks.

2024

1. ICML

   Polynomial-based Self-Attention for Table Representation learning

   Jayoung Kim, Yehjin Shin, Jeongwhan Choi, and 2 more authors

   In International Conference on Machine Learning, 2024

   Abs PDF

   Structured data, which constitutes a significant portion of existing data types, has been a long-standing research topic in the field of machine learning. Various representation learning methods for tabular data have been proposed, ranging from encoder-decoder structures to Transformers. Among these, Transformer-based methods have achieved state-of-the-art performance not only in tabular data but also in various other fields, including computer vision and natural language processing. However, recent studies have revealed that self-attention, a key component of Transformers, can lead to an oversmoothing issue. We show that Transformers for tabular data also face this problem, and to address the problem, we propose a novel matrix polynomial-based self-attention layer as a substitute for the original self-attention layer, which enhances model scalability. In our experiments with three representative table learning models equipped with our proposed layer, we illustrate that the layer effectively mitigates the oversmoothing problem and enhances the representation performance of the existing methods, outperforming the state-of-the-art table representation methods.

2. AAAI

   An Attentive Inductive Bias for Sequential Recommendation Beyond the Self-Attention

   Yehjin Shin, Jeongwhan Choi, Hyowon Wi, and 1 more author

   In Proceedings of the AAAI Conference on Artificial Intelligence, 2024

   Abs arXiv PDF Code

   Sequential recommendation (SR) models based on Transformers have achieved remarkable successes. The self-attention mechanism of Transformers for computer vision and natural language processing suffers from the oversmoothing problem, i.e., hidden representations becoming similar to tokens. In the SR domain, we, for the first time, show that the same problem occurs. We present pioneering investigations that reveal the low-pass filtering nature of self-attention in the SR, which causes oversmoothing. To this end, we propose a novel method called Beyond Self-Attention for Sequential Recommendation (BSARec), which leverages the Fourier transform to i) inject an inductive bias by considering fine-grained sequential patterns and ii) integrate low and high-frequency information to mitigate oversmoothing. Our discovery shows significant advancements in the SR domain and is expected to bridge the gap for existing Transformer-based SR models. We test our proposed approach through extensive experiments on 6 benchmark datasets. The experimental results demonstrate that our model outperforms 7 baseline methods in terms of recommendation performance.