WorksComputer Science
Muting Tokens: Exploring Token Sparsification for Efficient Audio Transformers
Abstract
Transformer-based models have shown strong performance across diverse tasks, but their reliance on self-attention mechanisms often leads to high computational costs. Token sparsification, which reduces the number of tokens processed in transformer layers, offers a promising solution to maintain performance while improving efficiency. However, although widely studied in the vision and language domains, token sparsification remains underexplored in audio, with existing work often limited to a single method, model size, or dataset. This work presents a comprehensive evaluation of token sparsification techniques for audio transformers. We assess a range of token pruning and merging strategies across multiple datasets and model sizes, under both post-training and fine-tuning settings. Furthermore, we introduce a novel sparsification approach that combines attention scores with spectral cues derived from the frequency content of token embeddings. Our experiments show that frequency-aware approaches, particularly those based on spectral flatness and low-frequency energy, consistently outperform attention-only methods. We also find that sparsification based on spectrogram patch intensity is especially effective for longer audio inputs. These results highlight the value of incorporating frequency-domain and input-specific information into sparsification strategies to enhance performance in audio tasks.
Introduction
Transformer-based models have demonstrated the ability to achieve state-of-the-art performance across various domains such as Natural Language Processing (NLP) (Devlin et al., [14]; Vaswani et al., 2017) and Computer Vision (Khan et al., [31]; Radford et al., [48]). However, the self-attention mechanism can be computationally intensive, as its complexity scales quadratically with the number of input tokens (Bolya et al., [5]; Fayyaz et al., [18]). To address this challenge, recent work has explored compression techniques such as pruning (Chavan et al., [8]; Song et al., [57]), quantization (Li et al., 2022b; Lin et al., [39]), and knowledge distillation (KD) (Rao et al., [50]; Ren et al., [51]), in an effort to reduce model size and inference latency while maintaining task performance.
Recently, token sparsification, a transformer-specific optimization method that discards, merges, or refines input tokens, has emerged as a promising alternative to reduce the
computational cost of self-attention. For example, the Expediting Vision Transformers (EViT) method achieves up to 50% faster inference on ImageNet [13], with only a 1.3 percentage point drop in accuracy when retaining 50% of the tokens, referred to as the keep rate [37].
A growing body of work has explored token sparsification techniques across various tasks, particularly in image classification [6, 11, 27, 32, 37] (Wu et al. 2023), video classification [55] (Wu et al. 2024), and even large language models [7, 10, 30, 40]. Although Vision Transformers (ViT) have demonstrated notable success in audio tasks by using spectrograms as input representations, the application of token sparsification in this domain remains largely underexplored (Chen et al., 2022a; Gong et al., 2021; 2022a;b; Huang et al., 2022). This gap is particularly important because spectrogram inputs yield more tokens as the input duration increases, which can make self-attention increasingly expensive for longer audio clips [20]. Addressing this challenge could not only help advance efficient transformers’ research, but also enable practical, large-scale deployment of audio-based systems, from everyday voice assistants to real-time transcription services at the edge.
Recent work [33] has begun to explore the effectiveness of token sparsification in audio tasks. However, while showing promising results, this study presents considerable limitations. First, it restricts its scope to only two sparsification methods, Top-K pruning [23] and Intensity pruning [33], overlooking other promising approaches such as token merging [5]. Second, their analysis is restricted to only the ViT-Base model [15] limiting the generalizability of their conclusions to a single model size. Third, the study focuses solely on applying sparsification during fine-tuning, whereas many real-world deployment scenarios favor post-training sparsification methods, mainly due to the widespread availability of pretrained models, and the training costs associated with larger datasets [12, 28]. Finally, it reports suboptimal performance of the intensity pruning method at low keep rates, without investigating the potential influence of input-dependent factors such as the audio duration.
In this work, we systematically evaluate a broad range of token sparsification methods for audio transformers, examining their performance across multiple model sizes and datasets, under two key settings: fine-tuning and post-training sparsification. To ensure a fair comparison, we conduct all experiments within a unified training and inference pipeline, establishing a consistent benchmark for evaluating sparsification techniques in this domain. We also revisit the intensity pruning method and show that its effectiveness improves when combined with attention scores. Furthermore, we propose a novel sparsification approach that integrates attention and frequency-based cues, drawing on insights from audio signal processing. As illustrated in Figure 1, methods that incorporate frequency-domain information consistently outperform existing sparsification approaches across our experimental setup. Our contributions can be summarized as:
Figure 1. Post-training sparsification results on the AST-Base (Gong et al., [20]) model, evaluated on AudioSet-20K (Gemmeke et al., [19]) at a 50% keep rate. Methods that include frequency-domain cues, such as the proposed Spectral Flatness (SF) and the adapted Low-Frequency Energy (LFE) (Wang et al., [63]), outperform others in terms of mean average precision (mAP), a standard metric for multi-label classification. The baseline (without sparsification) achieves an average mAP of 0.3572. The Pareto front is highlighted to illustrate the performance-efficiency trade-off.
We conduct a comprehensive evaluation of token sparsification methods for audio transformers, including token pruning and merging approaches under a unified evaluation setup.
We demonstrate the effectiveness of intensity-based pruning, showing that when combined with attention scores, it can outperform pure attention methods for audio inputs longer than 2 seconds across our experimental setup.
We introduce a novel sparsification method based on spectral flatness, and show that it can consistently improve performance across a wide range of model sizes and datasets.
Related Work
Token Sparsification: Overview
Token sparsification reduces the number of tokens processed within transformer layers, with existing methods broadly falling into two main categories [23]. Token pruning methods [16, 32, 38, 41, 44, 59, 70] (Fayaz et al. 2022; Li et al. 2022a; Xu et al. 2022b) reduce the number of tokens by completely removing tokens, at the risk of discarding useful information. In contrast, token merging methods [5, 24, 43, 52, 54, 71, 72] (Wu et al. 2021; Xu et al. 2022a) combine multiple tokens into a single token or cluster, but may introduce noise through averaging.
Our study includes representative methods from both sparsification categories to ensure diversity in evaluation, addressing the lack of systematic assessment in audio tasks.
Token Sparsification Methods for Audio
Recent efforts to apply token sparsification to audio tasks have primarily adapted techniques originally developed for vision models. Applied on a post-training manner, Token Merging (ToMe) [5] merges similar tokens to reduce the input size, achieving substantial throughput gains with merely a 0.4 percentage point drop in mAP on AudioSet-2M [19]. However, its evaluation is limited to a single dataset and a masked autoencoding (MAE) framework [26] (He et al. 2021), which is inherently designed to operate on sparse inputs. This may reduce the model’s sensitivity to sparsification, making it unclear how ToMe performs in settings that rely on full token visibility. Without evaluation on non-MAE models, its general applicability remains uncertain, an aspect we address in this work.
A different line of work explores sparsification in audio transformers during training. PaSST [21] employs “patchout” which randomly drops a subset of spectrogram patches to reduce computation and act as a regularizer. While effective, these approaches require training with patchout, limiting their practicality. In contrast, post-training sparsification offers a broadly applicable and cost-efficient alternative, especially when retraining is costly or infeasible.
Studies on Token Sparsification for Audio
Beyond these early efforts, recent work has begun to explore sparsification methods specifically for audio. As mentioned in § 1, the authors in [33] investigate Top-K pruning and Intensity pruning. While this represents a promising direction, the study remains narrow in scope, focusing on just two methods, a single ViT-Base configuration, and exclusively on fine-tuning scenarios. This highlights the need for broader studies that evaluate different sparsification approaches, model sizes, and across both fine-tuning and post-training contexts, a gap our work seeks to address.
Methodology
In this study, we pursue two complementary directions. First, we evaluate established token sparsification methods developed for vision, in audio tasks, providing the first systematic comparison of their behavior across audio benchmarks. Second, we propose lightweight, task-specific token importance estimation strategies that incorporate audio-relevant cues such as spectrogram intensity and frequency-domain features.
Token Sparsification Methods
We evaluate three sparsification methods: Top-K [23], Expediting Vision Transformers (EViT) [37] and ToMe [5]. These methods were selected based on their strong performance in vision tasks [23] and their representation of the two main paradigms in token sparsification: pruning (Top-K and EViT) and merging (ToMe). Note that
while some of these methods have been applied to audio in a limited setting, as mentioned in § 2.2, they have not yet been systematically compared or evaluated across benchmarks. Top-K is a widely used baseline, where the attention between spatial tokens and the class (CLS) token are used to rank token importance. At each reduction stage , the most attended tokens are kept. EViT extends this approach by introducing a fused token that preserves information from discarded tokens. At each stage , tokens not in the Top-K set are averaged and weighted by their CLS attention scores. ToMe reduces tokens by merging similar token pairs. The token set is split into two partitions, and each token is matched to its most similar counterpart based on cosine similarity. The top matching pairs are then merged by averaging their features.
Task-specific Token Importance Estimation Strategies
Most sparsification methods rely on internal model signals, such as attention weights or intermediate representations like attention keys. However, in post-training settings where model parameters remain fixed, we hypothesize that lightweight, task-specific heuristics can more effectively identify informative tokens, leading to improved performance under sparsification. In audio tasks, spectrograms encode energy across time and frequency, but these patterns may not be fully captured by attention alone [53]. Prior work highlights the utility of frequency-domain processing in transformers [34, 46, 50, 58], and that structure may not always be visible in raw spectrograms [2, 17]. Motivated by this, we explore three lightweight heuristics for token importance, one input patch intensity-based, and two frequency-based, that can be combined with many existing methods: 1) intensity, 2) low-frequency energy [63] and 3) spectral flatness.
Intensity. Following [33], we compute the mean log-Mel energy of each spectrogram patch as a lightweight salience proxy, based on the hypothesis that high intensity regions reflect stronger signal content. However, using intensity alone can be unreliable, as it can sometimes highlight background noise or nondiscriminative events with high energy [3, 33]. We therefore use it as a coarse filter. At each layer, tokens are split into high and low intensity sets based using a dataset-specific threshold. The Top-K tokens are selected by ranking the high-intensity ones via CLS attention scores, with any remainder filled from the low-intensity pool (Figure 2). This method is primarily intended to test whether incorporating spectrogram intensities improves attention-only sparsification and how it compares to frequency-aware approaches.
Figure 2. Intensity-guided token pruning. Tokens are divided into high- () and low- () intensity groups based on an applied threshold. The high-intensity set is then ranked using CLS attention scores, and the Top- tokens are selected. If fewer than tokens are available in , the remaining slots are filled from the low-intensity set.
Low-Frequency Energy. Inspired by the Low-Frequency Components (LFC) model for vision transformers, we adopt their idea of using low-frequency energy (LFE) for token sparsification [63]. In their work, low-pass filtering of token embeddings was shown to preserve task-relevant content more effectively, aligning with the observation that transformers tend to favor low-frequency features [15, 62]. We extend this idea to audio by computing low-frequency energy within each token embedding, after patch extraction and projection. This provides a local measure of how much of the token’s representation is concentrated in low-frequency components. This score is combined with the token’s attention weight, allowing the model to prioritize tokens that are both attended and rich by low-frequency energy, as illustrated in Figure 3.
Figure 3. Low-frequency Energy token pruning. Token embeddings are transformed to the frequency domain, where a Gaussian low-pass filter highlights low-frequency structure. The resulting energy scores are fused with attention scores to guide top-K token selection.
Spectral Flatness. Spectral Flatness (SF) is a widely used measure in audio signal processing for quantifying the degree to which a signal resembles noise or contains structured, tonal components [29, 56]. It is computed as the ratio of the geometric mean to the arithmetic mean of a signal’s magnitude spectrum, with lower values indicating pronounced spectral peaks typical of harmonic or speech-like content, and higher values corresponding to flatter, noise-like distributions [4, 42]. Motivated by these properties, we extend the use of SF to transformer token embeddings by computing it for each token, as illustrated in Figure 4. We treat each embedding as a learned signal, using SF as a proxy for its internal organization, and hypothesize that lower flatness values indicate more structured and content-rich tokens. An illustrative example of this phenomenon is shown in Figure 5, where two spectrogram patches yield token embeddings with contrasting spectral profiles. The token’s flatness score is combined with its attention to the CLS token, prioritizing tokens that are both spectrally structured and highly attended.
Figure 4. Spectral flatness-guided token pruning. Token embeddings undergo Fast Fourier Transform (FFT) along the embedding dimension to compute spectral flatness, capturing tonal richness. These scores are combined with CLS attention to rank token importance.

Figure 5. Spectral flatness at the token level. (a) A spectrogram with two highlighted tokens. (b) FFT of corresponding token embeddings shows Token A has tonal peaks, while Token B has a flatter, noise-like spectrum - making Token A potentially more informative.
Evaluation Strategy
We aim to systematically evaluate token sparsification methods (§ 3.1) across multiple audio benchmarks, while also investigating the role of input- and frequency-aware token importance estimation strategies (§ 3.2). To understand both the practical and performance implications of sparsification, we evaluate all methods under two settings:
Post-training Sparsification. Sparsification is applied only during inference, using already finetuned models.
Sparsified Fine-tuning. Sparsification is applied during fine-tuning, starting from ImageNet-pretrained weights and adapting to the target audio dataset.
Using these two settings, we aim to understand the benefits and tradeoffs of sparsified fine-tuning over post-training sparsification, especially at lower keep rates.
Experimental Setup
Datasets. For this analysis, three well-established datasets are selected, each covering different audio sample lengths and task complexities. Speech Commands V2 (SPC) [64] comprises 105,829 1-second recordings of 35 spoken commands. The dataset is divided into 84,843 training samples, 9,981 validation samples, and 11,005 test samples. Environment Sound Classification (ESC50) [47] consists of 2,000 5-second environmental sound recordings categorized into 50 classes. Its 5-fold cross-validation setup includes 1,600 training samples per fold, ensuring a reliable evaluation framework. AudioSet (AS-20K) [19] is a large-scale multi-class dataset containing 10-second audio clips sourced from YouTube, labeled across 527 categories. The balanced training set consists of around 22,000 samples, while the evaluation set includes around 20,000 samples.
Backbone. We adopt the AST (Audio Spectrogram Transformer) [20] architecture as our backbone for all experiments. The sparsification methods are integrated into the AST model pretrained on ImageNet [13], using the standard DeiT-based [60] configurations: Base (B), Small (S), and Tiny (T), with patch sizes of and an overlap of 6 in both time and frequency, consistent with the default AST recipe.
Sparsification Allocation. As in previous work, the token sparsification module is placed between the attention and multi-layer perceptron (MLP) branches of each transformer block of the AST [5]. The most common and effective practice is to apply sparsification at three equally spaced points in the transformer, effectively dividing the network into four sections [32, 38, 41]. This is the configuration adopted in our experiments. At each sparsification stage, a fixed keep rate is applied uniformly across all sparsified layers.
Implementation Details. All training and fine-tuning procedures follow the original AST setup [20]. Specifically, AS-20K and ESC50 are finetuned for 25 epochs, while SPC is trained for 30 epochs. When token sparsification is enabled, all default masking strategies (e.g., SpecAugment [45]) in AST are disabled to avoid redundancy, as sparsification inherently performs a form of data masking. All model variants (AST-B, AST-S, AST-T) are evaluated under the post-training sparsification setting to assess performance and efficiency across different model scales. For the sparsified fine-tuning experiments, we use only AST-B, as it is the default model size in the original AST implementation. To quantify the compute cost associated with different sparsification methods, we report the number of Floating Point Operations (FLOPs) per instance using a batch size of 1, measured via the DeepSpeed profiler [1].
Results & Analysis
Post-Training Sparsification
Performance Across Methods and Model Sizes. Tables 2a, 2b, and 2c report performance across the AST-B, AST-S, and AST-T models, respectively. Results are shown for all datasets (SPC, ESC50, AS-20K) and across token keep rates from 90% down to 25%. We report the average of three runs.
From these results, we find that methods using token frequency information, particularly LFE and SF, consistently perform well across model sizes and datasets, frequently ranking among the top two methods in our setup. ToMe also shows competitive performance, especially on the Small and Tiny models with the ESC50 dataset, but its inconsistency on the AST-B model may suggest sensitivity to model scale or dataset characteristics. We hypothesize that this may be due to ToMe’s merging strategy, which can interfere with token semantics more noticeably in larger models where attention patterns are more specialized [14].
Compute Cost Analysis. We report compute cost in terms of FLOPs for AS-20K using the AST-B model in Table 1. As expected, the number of FLOPs decreases consistently with lower keep rates across all datasets and methods, reflecting the expected reduction in attention computation as fewer tokens are retained. These trends validate that sparsification mechanisms function as intended, with lower keep rates leading to correspondingly lower compute cost. While this downward trend is consistent across methods, the actual savings differ slightly due to method-specific overheads. To illustrate the trade-off between performance and compute cost, we present Pareto front plots. In Figure 1, we show results for AS-20K, while Figure 6 shows SPC and ESC50 datasets, on AST-B. We find that frequency-domain methods such as Spectral Flatness (SF) and Low-Frequency Energy (LFE) consistently yield the highest performance across datasets, despite slightly higher computational overhead compared to simpler methods like Top-K. Results follow similar trends, showing the performance scalability of most methods across model sizes and tasks.
| AS-20K (GFLOPs) | |||||
| Baseline | 261.43 | ||||
| (%) | 90 | 70 | 50 | 25 | |
| Top-K ([23]) | 222.28 15.0% | 164.16 37.2% | 124.85 52.3% | 93.74 64.3% | |
| EViT ([37]) | 222.61 14.8% | 164.45 37.1% | 125.05 52.2% | 93.99 64.0% | |
| ToMe ([5]) | 226.11 13.5% | 165.77 36.6% | 125.60 52.0% | – | |
| Intensity | 222.28 15.0% | 164.16 37.2% | 124.89 52.2% | 93.79 64.1% | |
| LFE ([63]) | 222.66 14.8% | 164.46 37.1% | 125.10 52.1% | 93.95 64.1% | |
| SF | 222.41 14.9% | 164.26 37.1% | 124.95 52.2% | 93.84 64.1% |
Table 1. GFLOPs of AST-B model on AS-20K after sparsification across keep rates (). denotes percentage reductions over baseline. Maximum reductions per keep rate are highlighted in green.
Figure 6. Post-training sparsification results on the AST-B ([20]) model evaluated on SPC and ESC50 at a 50% keep rate. Methods leveraging frequency-domain cues, such as Spectral Flatness (SF) and Low-Frequency Energy (LFE) ([63]), outperform other methods in terms of accuracy.
Sparsified Fine-tuning
Figure 7 shows how task performance varies with decreasing keep rates across datasets under sparsified fine-tuning, with results averaged over three runs.
Figure 7. Sparsified Fine-tuning: Performance across varying keep rates. Frequency-based methods (SF, LFE), as well as pure attention-based pruning methods (Top-K, EViT), consistently maintain high performance across tasks. Most methods retain competitive performance down to a 50% keep rate. Overall, the performance differences between methods are not very significant, indicating that multiple sparsification methods can be equally effective when fine-tuned.
Across all datasets, we observe that performance remains robust even under aggressive sparsification, with most methods maintaining accuracy close to the baseline at a 25% keep rate. This indicates that when sparsification is integrated during training, the model is able to adapt to the imposed sparsity, learning to preserve task-relevant information despite reduced number of tokens. Furthermore, the performance differences between methods are less pronounced than in the post-training setting. This suggests that the model learns to accommodate whichever tokens are retained during training, thereby reducing sensitivity to the specific scoring heuristic. As a result, even simple methods like Top-K remain competitive, and the relative advantage of more informed strategies, such as SF and LFE, is reduced.
In summary, sparsified fine-tuning makes the model less sensitive to the specific sparsification heuristic, as it learns to adapt to the retained tokens. This contrasts with post-training sparsification, where token importance is fixed and the choice of heuristic plays a more decisive role.
Comparative & Controlled Studies
Contribution of Frequency-domain Information. To understand the source of performance differences between SF and LFE, we conduct a comparative study that isolates the effects of their frequency metrics and attention formulations. SF uses standard CLS-token attention (TopK-CLS), and LFE on the other hand, uses a head-weighted attention variant (TopK-Wt) ([63]). To isolate the impact of these factors, we evaluate all four combinations of frequency metric (SF or LFE) and attention type (TopK-CLS or TopK-Wt). Each configuration is run five times to account for randomness.
As shown in Figure 8, SF consistently outperforms its TopK-only counterparts across keep rates and attention variants. In contrast, while LFE generally improves over its baseline, it occasionally underperforms on AS-20K and ESC50, suggesting that emphasizing low-frequency components may not universally benefit performance and may depend on dataset characteristics. Furthermore, TopK-Wt tends to outperform TopK-CLS, especially at lower keep rates (< 70%), which may be because TopK-Wt also includes peer-token attention, offering a more distributed importance signal than CLS-token attention alone.
Figure 8. Performance comparison of frequency-based methods (LFE and SF) against attention-only baselines (TopK-Wt and TopK-CLS). SF slightly but consistently outperforms attention-based baselines across keep rates, while LFE shows competitive results but occasionally underperforms on ESC50 and AS-20K.
Overall, incorporating frequency-domain information yields modest but consistent gains over attention-only pruning methods in post-training settings.
Effect of Input Duration on Intensity-Based Pruning. In § 5.1 (Table 2a), we observed that intensity-based pruning outperforms purely attention-based methods (Top-K, EViT) on ESC50 and AS-20K across most settings, whereas its advantage is less pronounced on SPC. This relatively strong performance on datasets with longer audio clips (5 and 10 seconds, respectively) - compared to weaker results on SPC (with 1-second clips) raises the question of a potential dependency on input duration.
Table 2. Post-training sparsification results across model sizes.
(a) Post-training sparsification results for the AST-B model across keep rates (). Best performing methods are in bold, second-best are <u>underlined</u>, and results outperforming the AST-B baseline are highlighted. SF consistently achieves the best performance across most keep rates and datasets, while LFE also performs competitively, particularly on SPC.
| SPC (Acc(%)) | ESC50 (Acc(%)) | AS-20K (mAP) | ||||||||||
| AST-B | 97.67 | 89.42 | 0.3572 | |||||||||
| (%) | 90 | 70 | 50 | 25 | 90 | 70 | 50 | 25 | 90 | 70 | 50 | 25 |
| Top-K (Haurum et al., [23]) | 97.46 | 97.43 | 97.21 | 88.73 | 89.02 | 88.42 | 87.32 | 78.47 | 0.3557 | 0.3546 | 0.3470 | 0.2956 |
| EViT (Liang et al., [37]) | 97.44 | 97.27 | 97.29 | 90.03 | 88.77 | 88.17 | 87.00 | 75.42 | 0.3552 | 0.3545 | 0.3476 | 0.2981 |
| ToMe (Bolya et al., [5]) | 97.51 | 97.47 | 97.14 | – | 88.76 | 88.45 | 87.25 | – | 0.3567 | 0.3539 | 0.3470 | – |
| Intensity | 97.55 | 97.46 | 97.19 | 88.63 | 89.10 | 88.52 | 87.43 | 78.62 | 0.3567 | 0.3549 | 0.3479 | 0.3033 |
| LFE (Wang et al., [63]) | 97.66 | 97.62 | 97.50 | 93.49 | <u>89.12</u> | 88.45 | 87.02 | <u>78.92</u> | <u>0.3568</u> | 0.3546 | <u>0.3484</u> | <u>0.3012</u> |
| SF | 97.68 | 97.66 | <u>97.40</u> | 89.00 | 89.20 | 88.67 | 87.47 | 78.97 | 0.3576 | 0.3564 | 0.3499 | 0.3007 |
(b) Post-training sparsification results for the AST-S model across keep rates (). Best performing methods are in bold, second-best are <u>underlined</u>, and results outperforming the AST-S baseline are highlighted. SF and LFE dominate across datasets, mirroring the trends observed with AST-B. ToMe performs competitively on ESC50, indicating that token merging may be effective for smaller models.
| SPC (Acc(%)) | ESC50 (Acc(%)) | AS-20K (mAP) | ||||||||||
| AST-S | 97.21 | 83.75 | 0.3105 | |||||||||
| (%) | 90 | 70 | 50 | 25 | 90 | 70 | 50 | 25 | 90 | 70 | 50 | 25 |
| Top-K (Haurum et al., [23]) | 96.97 | 97.04 | 96.83 | 83.60 | 83.10 | 81.65 | 79.23 | 69.03 | 0.3058 | 0.3037 | 0.2995 | 0.2688 |
| EViT (Liang et al., [37]) | 96.96 | 97.02 | 96.84 | <u>89.39</u> | 83.03 | 81.65 | 79.30 | 69.40 | 0.3052 | 0.3018 | 0.2950 | 0.2672 |
| ToMe (Bolya et al., [5]) | <u>97.16</u> | 96.93 | 96.32 | – | 83.33 | 83.15 | 80.80 | – | 0.3057 | 0.3029 | 0.3001 | – |
| Intensity | 96.89 | 96.95 | 96.76 | 83.33 | 83.08 | <u>82.58</u> | 78.03 | 69.60 | 0.3068 | 0.3043 | 0.2996 | 0.2734 |
| LFE (Wang et al., [63]) | 97.21 | <u>97.24</u> | 97.21 | 93.74 | 83.05 | 81.50 | 78.88 | 68.33 | 0.3089 | 0.3068 | 0.3037 | 0.2792 |
| SF | 97.21 | 97.25 | <u>97.20</u> | 82.37 | <u>83.20</u> | 81.75 | 79.23 | 69.35 | 0.3093 | 0.3074 | 0.3049 | 0.2799 |
(c) Post-training sparsification results for the AST-T model across keep rates (). Best performing methods are in bold, second-best are <u>underlined</u>, and results outperforming the AST-T baseline are highlighted. SF and LFE rank quite consistently among the top, showing strong scalability across model sizes, while ToMe excels on ESC50 but is less consistent across other datasets.
| SPC (Acc(%)) | ESC50 (Acc(%)) | AS-20K (mAP) | ||||||||||
| AST-T | 96.99 | 81.15 | 0.2759 | |||||||||
| (%) | 90 | 70 | 50 | 25 | 90 | 70 | 50 | 25 | 90 | 70 | 50 | 25 |
| Top-K (Haurum et al., [23]) | 96.37 | 96.21 | 95.93 | 88.06 | 80.58 | 78.65 | 73.95 | 56.23 | 0.2721 | 0.2699 | 0.2601 | 0.2189 |
| EViT (Liang et al., [37]) | 96.35 | 96.19 | 95.87 | 92.37 | 80.35 | 78.53 | 73.78 | 57.03 | 0.2668 | 0.2632 | 0.2591 | 0.2161 |
| ToMe (Bolya et al., [5]) | <u>97.01</u> | 96.26 | 95.00 | – | 81.25 | 79.53 | 74.13 | – | 0.2731 | <u>0.2701</u> | 0.2603 | – |
| Intensity | 97.02 | 96.62 | 96.12 | 89.12 | 80.47 | <u>79.01</u> | 73.61 | 54.91 | 0.2695 | 0.2654 | 0.2599 | 0.2233 |
| LFE (Wang et al., [63]) | 97.00 | 97.03 | 96.81 | <u>91.36</u> | 80.40 | 78.70 | 73.45 | 53.68 | 0.2699 | 0.2675 | <u>0.2624</u> | <u>0.2391</u> |
| SF | 97.02 | <u>96.98</u> | <u>96.71</u> | 87.87 | <u>80.63</u> | 78.70 | 74.20 | 56.73 | <u>0.2728</u> | 0.2709 | 0.2638 | 0.2399 |
Table 2.
To investigate this, we conduct a controlled experiment comparing intensity-based pruning with a purely attention-based Top-K method. We apply random temporal windowing at inference time on AS-20K and ESC50, simulating input lengths ranging from 1 second to the full duration of each dataset (i.e., 1s to 10s for AS-20K, and 1s to 5s for ESC50). Each result is averaged over six runs to account for randomness. The sparsification configuration is fixed to the most aggressive setting, with a keep rate of only 25%.
As shown in Figure 9, intensity-based pruning consistently outperforms the Top-K method for audio segments longer than 2 seconds across both datasets. These results provide further empirical support for our hypothesis and suggest that intensity-based pruning works better for longer inputs. We attribute this effect to the interaction between input duration and token redundancy. First, longer audio clips yield more tokens and often contain repeated or sustained events, introducing redundancy. Therefore, even after sparsification, enough informative content remains. Second, they tend to include clear low-intensity regions, such as silence or background noise, which can be safely discarded. In contrast, short clips (1-2 seconds) typically contain a single spoken word or sound event, where nearly all tokens contribute to the core signal. These inputs offer little redundancy and minimal variation in intensity, making it harder to distinguish unimportant tokens. As a result, sparsification based on intensity can mistakenly remove critical information, leading to performance degradation.

Figure 9. Token pruning performance as a function of input duration. Intensity Pruning consistently outperforms Top-K pruning for longer audio windows ( seconds), suggesting that spectrogram intensity is a more informative proxy for longer audio samples.
Practical Guidelines
Based on our analysis, we outline several recommendations for effectively applying token sparsification in audio transformers:
Frequency-aware heuristics, particularly Spectral Flatness (SF) and Low-Frequency Energy (LFE), quite consistently outperform other methods. SF stands out as one of the most practical choices for post-training sparsification, as it requires no additional tuning and generalizes well across configurations. LFE, while also effective, involves tuning a low-pass filter cutoff to maximize performance, which may require dataset-specific calibration.
For scenarios involving longer input durations (e.g., > 2 seconds), intensity-based pruning can also outperform other standard methods mainly for larger model sizes, but it requires threshold tuning to distinguish between high- and low-energy patches.
While token merging methods like ToMe can perform well on small models or specific tasks, their effectiveness is more variable and less consistent across certain tasks.
Finally, when applying sparsification during finetuning, the specific heuristic becomes less critical, as the model can adapt to sparsity - enabling even simple methods to perform well.
Conclusion
In this work, we presented a comprehensive evaluation of token sparsification techniques for audio transformers, covering pruning and merging approaches across multiple model sizes, datasets, and deployment scenarios. Our results show that frequency-aware methods, particularly those drawing on spectral flatness and low-frequency energy, consistently outperform attention-only methods, particularly in post-training sparsification settings. We also introduced a novel strategy that combines attention with spectral priors, which proved effective across a wide range of configurations. These findings underscore the value of integrating audio-specific cues into sparsification strategies, moving beyond generic attention signals. Looking forward, we see opportunities to explore more advanced task-specific token estimation strategies and extend these techniques to real-time applications.
References
- [1]Aminabadi, R. Y., Rajbhandari, S., Zhang, M., Awan, A. A., Li, C. L., Di, C., Zheng, E., Rasley, J., Smith, S., Ruwase, O., and He, Y. DeepSpeed inference: Enabling efficient inference of transformer models at unprecedented scale. In SC ’20, pp. 1–16. IEEE Press, 2020. doi: 10.5555/3433701.3433725. URL https://arxiv.org/abs/1910.02054.
- [2]Atlas, L. Modulation spectral filtering. In Proc. Eurospeech, Geneva, Switzerland, 2003. University of Washington Technical Report.
- [3]Bhattacharjee, M., Prasanna, S., and Guha, P. Foreground-background audio separation using spectral peaks based time-frequency masks. In IEEE International Conference on Signal Processing and Communications (SPCOM), pp. 1–5, 07 2022. doi: 10.1109/SPCOM55316.2022.9840850.DOI
- [4]Boashash, B., Khan, N. A., and Ben-Jabeur, T. Time–frequency features for pattern recognition using high-resolution tfdis: A tutorial review. Digital Signal Processing, 40:1–30, 2015. ISSN 1051-2004. doi: https://doi.org/10.1016/j.dsp.2014.12.015. URL https://www.sciencedirect.com/science/article/pii/S1051200414003571.
- [5]Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. Token merging: Your vit but faster, 2023. URL https://arxiv.org/abs/2210.09461.
- [6]Bonnaerens, M. and Dambre, J. Learned thresholds token merging and pruning for vision transformers. ArXiv, 2023.arxiv.org/abs/2307.10780
- [7]Cao, Q., Paranjape, B., and Hajishirzi, H. Pumer: Pruning and merging tokens for efficient vision language models, 2023. URL https://arxiv.org/abs/2305.17530.
- [8]Chavan, A., Shen, Z., Liu, Z., Liu, Z., Cheng, K.-T., and Xing, E. Vision transformer slimming: Multi-dimension searching in continuous optimization space, 2022. URL https://arxiv.org/abs/2201.00814.
- [10]Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024. URL https://arxiv.org/abs/2403.06764.
- [11]Chen, M., Lin, M., Li, K., Shen, Y., Wu, Y., Chao, F., and Ji, R. Cf-vit: A general coarse-to-fine method for vision transformer, 2022b. URL https://arxiv.org/abs/2203.03821.
- [12]Chen, X., Liu, J., Wang, Y., Brand, M., Pu, Wang, and Koike-Akino, T. Tunecomp: Joint fine-tuning and compression for large foundation models, 2025. URL https://arxiv.org/abs/2505.21835.
- [13]Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009. doi: 10.1109/CVPR.2009.5206848.DOI
- [14]Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805.
- [15]Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929.
- [16]Dou, Z., Wu, Q., Lin, C., Cao, Z., Wu, Q., Wan, W., Komura, T., and Wang, W. Tore: Token reduction for efficient human mesh recovery with transformer, 2023. URL https://arxiv.org/abs/2211.10705.
- [17]Ezzat, T., Bouvrie, J., and Poggio, T. Spectro-temporal analysis of speech using 2-d gabor filters. In Proc. Interspeech, pp. 506–509, Antwerp, Belgium, 2007.DOI
- [18]Fayyaz, M., Koohpayegani, S. A., Jafari, F. R., Sengupta, S., Joze, H. R. V., Sommerlade, E., Pirsiavash, H., and Gall, J. Adaptive token sampling for efficient vision transformers, 2022. URL https://arxiv.org/abs/2111.15667.
- [19]Gemmeke, J., Ellis, D., Freedman, D., Jansen, A., Lawrence, W., Moore, R., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776–780, 03 2017. doi: 10.1109/ICASSP.2017.7952261.DOI
- [20]Gong, Y., Chung, Y.-A., and Glass, J. Ast: Audio spectrogram transformer. NeurIPS, 2021.
- [21]Gong, Y., Chung, Y. A., and Glass, J. Passt: Efficient training of audio transformers with patchout regularization. arXiv preprint arXiv:2203.16352, 2022a.
- [23]Haurum, J. B., Escalera, S., Taylor, G. W., and Moeslund, T. B. Which tokens to use? investigating token reduction in vision transformers. In IEEE/CVF Conference on CVPR, pp. XX–XX. IEEE, 2022a.arxiv.org/abs/2308.04657
- [24]Haurum, J. B., Madadi, M., Escalera, S., and Moeslund, T. B. Multi-scale hybrid vision transformer and sinkhorn tokenizer for sewer defect classification. Automation in Construction, 2022b.DOI
- [26]Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., and Feichtenhofer, C. Audiomae: Masked auto-encoders for audio. ArXiv, 2022.
- [27]Huang, W., Shen, Y., Xie, J., Zhang, B., He, G., Li, K., Sun, X., and Lin, S. A general and efficient training for transformer via token expansion. ArXiv, 2024.arxiv.org/abs/2404.00672
- [28]Ilhan, F., Su, G., Tekin, S. F., Huang, T., Hu, S., and Liu, L. Resource-efficient transformer pruning for finetuning of large models. In IEEE/CVF Conference on CVPR, pp. 16206–16215, June 2024.DOI
- [29]Jayant, N. S. and Noll, P. Digital Coding of Waveforms: Principles and Applications to Speech and Video. Prentice Hall, Englewood Cliffs, NJ, 1993. ISBN 9780132119132.
- [30]Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L. Lmllm-gua: Compressing prompts for accelerated inference of large language models. ArXiv, 2023.
- [31]Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., and Shah, M. Transformers in vision: A survey. ACM Computing Surveys, 54(10s):1–41, January 2022. ISSN 1557-7341. doi: 10.1145/3505244. URL http://dx.doi.org/10.1145/3505244.
- [32]Kong, Z., Dong, P., Ma, X., Meng, X., Sun, M., Niu, W., Shen, X., Yuan, G., Ren, B., Qin, M., Tang, H., and Wang, Y. Spvit: Enabling faster vision transformers via soft token pruning. ArXiv, 2022.arxiv.org/abs/2112.13890
- [33]Lee, T. and Lee, H. Token pruning in audio transformers: Optimizing performance and decoding patch importance, 2025. URL https://arxiv.org/abs/2504.01690.
- [34]Lee-Thorp, J., Ainslie, J., Eckstein, I., and Ontanon, S. Fnet: Mixing tokens with fourier transforms. In ICML, 2021.
- [37]Liang, Y., Ge, C., Tong, Z., Song, Y., Wang, J., and Xie, P. Not all patches are what you need: Expediting vision transformers via token reorganizations. ArXiv, 2022a.arxiv.org/abs/2202.07800
- [38]Liang, Y., Ge, C., Tong, Z., Song, Y., Wang, J., and Xie, P. Not all patches are what you need: Expediting vision transformers via token reorganizations, 2022b. URL https://arxiv.org/abs/2202.07800.
- [39]Lin, Y., Zhang, T., Sun, P., Li, Z., and Zhou, S. Fq-vit: Post-training quantization for fully quantized vision transformer, 2023. URL https://arxiv.org/abs/2111.13824.
- [40]Liu, J., Li, L., Xiang, T., Wang, B., and Qian, Y. Tecra-llm: Token compression retrieval augmented large language model for inference cost reduction. ArXiv, 2023.
- [41]Long, S., Zhao, Z., Pi, J., Wang, S., and Wang, J. Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers, 2022. URL https://arxiv.org/abs/2211.11315.
- [42]Madhu, N. Note on measures for spectral flatness. Electronics Letters, 45:1195 – 1196, 12 2009. doi: 10.1049/el.2009.1977.DOI
- [43]Marin, D., Chang, J.-H. R., Ranjan, A., Prabhu, A., Rastegari, M., and Tuzel, O. Token pooling in vision transformers, 2021. URL https://arxiv.org/abs/2110.03860.
- [44]Pan, B., Panda, R., Jiang, Y., Wang, Z., Feris, R., and Oliva, A. Ia-red²: Interpretability-aware redundancy reduction for vision transformers. In Advances in NeurIPS, 2021.
- [45]Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., and Le, Q. V. Specaugment: A simple data augmentation method for automatic speech recognition. In Interspeech, Graz, Austria, 2019. arXiv:1904.08779.
- [46]Patro, B., Namboodiri, V., and Agneeswaran, V. Spectformer: Frequency and attention is what you need in a vision transformer. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 9543–9554, 02 2025. doi: 10.1109/WACV61041.2025.00924.
- [47]Piczak, K. J. ESC: Dataset for environmental sound classification. In 23rd ACM Multimedia, pp. 1015–1018. ACM, 2015.
- [48]Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. ICML, 2021.
- [50]Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsieh, C.-J. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems (NeurIPS), 2021b. URL https://github.com/raoyongming/DynamicViT.
- [51]Ren, S., Gao, Z., Hua, T., Xue, Z., Tian, Y., He, S., and Zhao, H. Co-advise: Cross inductive bias distillation. CoRR, abs/2106.12378, 2021.arxiv.org/abs/2106.12378
- [52]Renggli, C., Pinto, A. S., Houlsby, N., Mustafa, B., Puigcerver, J., and Riquelme, C. Learning to merge tokens in vision transformers. arXiv preprint arXiv:2202.12015, 2022.
- [53]Ristea, N.-C., Ionescu, R. T., and Khan, F. S. Septr: Separable transformer for audio spectrogram processing, 2022. URL https://arxiv.org/abs/2203.09581.
- [54]Ryoo, M., Piergiovanni, A., Arnab, A., Dehghani, M., and Angelova, A. Tokenlearner: Adaptive space-time tokenization for videos. In Advances in NeurIPS, 2021.
- [55]Ryoo, M. S., Piergiovanni, A., Arnab, A., Dehghani, M., and Angelova, A. Tokenlearner: What can 8 learned tokens do for images and videos?, 2022. URL https://arxiv.org/abs/2106.11297.
- [56]Seago, A., Holland, S., and Mulholland, P. Timbre space as synthesis space: Towards a navigation based approach to timbre specification. Proceedings of the Institute of Acoustics, 30, 01 2008.
- [57]Song, Z., Xu, Y., He, Z., Jiang, L., Jing, N., and Liang, X. Cp-vit: Cascade vision transformer pruning via progressive sparsity prediction. arXiv preprint arXiv:2203.04570, 2022.
- [58]Tamkin, A., Braverman, V., Jurafsky, D., and Re, C. Language through a prism: A spectral approach for multiscale language representations. In Advances in NeurIPS, 2020.arxiv.org/abs/2011.04823
- [59]Tang, Y., Han, K., Wang, Y., Xu, C., Guo, J., Xu, C., and Tao, D. Patch slimming for efficient vision transformers. In IEEE/CVF CVPR, 2022.arxiv.org/abs/2106.02852
- [60]Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transformers & distillation through attention. CoRR, abs/2012.12877, 2020.
- [62]Wang, P., Zheng, W., Chen, T., and Wang, Z. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. In ICLR, 2022a.arxiv.org/abs/2203.05962
- [63]Wang, Z., Luo, H., WANG, P., Ding, F., Wang, F., and Li, H. Vtc-lfc: Vision transformer compression via low-frequency component enhancement. In NeurIPS, 2022b.
- [64]Warden, P. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, April 2018.arxiv.org/abs/1804.03209
- [70]Yin, H., Vahdat, A., Alvarez, J., Mallya, A., Kautz, J., and Molchanov, P. Adavit: Adaptive tokens for efficient vision transformer, 2022. URL https://arxiv.org/abs/2112.07658.
- [71]Zeng, W., Jin, S., Liu, W., Qian, C., Luo, P., Ouyang, W., and Wang, X. Not all tokens are equal: Human-centric visual analysis via token clustering transformer. In IEEE/CVF CVPR, 2022.arxiv.org/abs/2204.08680
- [72]Zong, Z., Li, K., Song, G., Wang, Y., Qiao, Y., Leng, B., and Liu, Y. Self-slimmed vision transformer. In ECCV, 2022.






