betaFeedback

WorksComputer Science

TopoQuant: Topological Framework to Characterize Behavioral Shifts in Quantized Large Language Models

Abstract

Post-training quantization (PTQ) reduces the com putational cost of large language models (LLMs), but it can introduce internal structural and behavioral changes that are not captured by traditional metrics such as accuracy and per plexity. We present TopoQuant, a topological framework that uses topological data analysis (TDA) to quantify structural drift in self-attention weights and relate it to behavioral changes in quantized LLMs. Across four LLM families and multiple quantization levels, we show that topological drift remains stable at moderate compression levels but increases sharply below 4 bit precision, closely tracking declines in attention agreement. Our results reveal a strong relationship between topological drift and attention agreement, indicating that structural changes in model parameters closely reflect behavioral degradation under quantization. We further demonstrate that topological features can accurately predict attention consistency using lightweight machine learning models, achieving R2 scores of up to 0.85. Predicting attention agreement enables efficient assessment of be havioral consistency in quantized LLMs, allowing quantization induced degradation to be detected without running expensive downstream tasks. The results indicate that topological drift pro vides an efficient early-warning signal for quantization-induced degradation and can help identify safe compression limits without extensive downstream evaluation.

Introduction

Post-training quantization (PTQ), which reduces the precision of pre-trained large language model (LLM) parameters, has emerged as an effective approach for lowering memory consumption, inference latency, and computational costs during deployment [1, 2]. However, quantization often introduces performance degradation and behavioral drift [3]. Traditional evaluation metrics, such as accuracy, loss, and perplexity, provide only limited insights into how and why quantization affects the model behavior [3, 4].

Understanding these effects requires examining the internal structure of the model, rather than relying solely on downstream performance. Topological data analysis (TDA) offers a promising framework for this purpose by characterizing the shape, connectivity, and structural organization of high-dimensional data [5, 6]. Consequently, topological features may reveal quantization-induced changes that remain hidden from conventional statistical metrics.

Attention mechanisms play a central role in transformer-based LLMs by determining which tokens influence model predictions [7]. Prior studies have shown that attention behavior is sensitive to quantization artifacts and closely linked to model behavior [8, 9, 10]. Quantization-induced perturbations to the Query, Key, and Value projections can alter attention allocation patterns, even when the overall task accuracy remains stable. Therefore, measuring the attention agreement between full-precision and quantized models provides a sensitive indicator of behavioral drift that may not be detected using aggregate performance metrics alone.

Recent parameter-based quantization evaluation frameworks have attempted to analyze quantization effects directly from model parameters [3, 4]. However, several limitations remain. First, high accuracy can be misleading because quantized models may produce substantially different predictions while maintaining similar aggregate performance. Second, existing approaches primarily rely on aggregate statistical or coordinate-based representations that quantify numerical drift but fail to capture the underlying geometric and structural organization of model parameters. Third, current frameworks [11, 12] lack effective mechanisms for early detection of quantization-induced degradation. Finally, most approaches aggregate Query (QQ), Key (KK), and Value (VV) projections into a single global measure [4], making it difficult to identify which attention components are most affected by quantization.

To address these limitations, we propose TopoQuant, a topological framework for analyzing quantization-induced changes in large language models (LLMs). The framework combines three complementary perspectives: behavioral drift analysis, attention consistency analysis, and topological drift analysis. Attention consistency is quantified using Top-1 agreement and Top-3 Intersection-over-Union (IoU), whereas structural drift is characterized using TDA-based representations. By relating topological changes to attention behavior, TopoQuant provides an interpretable view of how quantization alters the model behavior and internal organization.

TopoQuant analyzes Query (QQ), Key (KK), and Value (VV) projection matrices independently, enabling targeted diagnosis of quantization-sensitive components. We focus on these projections because they directly govern attention formation and token interactions, whereas the Output (OO) projection primarily transforms attention outputs after the attention distribution has already been computed. This design allows the framework to identify structural changes before they become apparent in conventional downstream evaluation metrics.

This study investigates LLMs quantized from 8-bit to 2-bit precision and explores whether topological representations can characterize and predict quantization-induced behavioral changes. Our main contributions are:

  • We propose TopoQuant, a topology-aware framework for quantifying quantization-induced structural drift in LLM attention weights using persistence diagrams.

  • We introduce a multi-metric topological analysis pipeline that measures structural changes between base and quantized models across attention projections, blocks, and quantization levels.

  • We link topological drift to attention behavior through Top-1 agreement and Top-3 IoU-based attention consistency analysis.

  • We show that topological drift can identify quantization degradation and determine effective quantization stopping points without exhaustive model evaluation.

  • We develop lightweight machine learning models that predict attention agreement between the base and quantized model directly from topological drift features and LLM characteristics.

  • We evaluate TopoQuant across multiple LLM families and quantization schemes, showing its effectiveness in characterizing and predicting changes in quantization behavior.

PTQ methods and evaluation frameworks. PTQ is widely used to reduce the memory footprint and computational cost of LLMs while avoiding costly retraining. Representative methods include GPTQ [13], OmniQuant [14] and SliM-LLM [15]. Several studies have proposed evaluation frameworks to assess quantized models across performance, efficiency, robustness, and alignment dimensions [11, 12, 16]. While these studies provide valuable benchmarks, they primarily focus on task-level performance and do not investigate the structural changes introduced by quantization or their relationship with behavioral drift.

Limitations of perplexity and accuracy. Several studies have shown that conventional metrics such as accuracy and perplexity may fail to capture important behavioral differences between full-precision and quantized LLMs [17, 18]. Alternative measures, including KL divergence, prediction-flip rates [3], correctness agreement [4], Jensen–Shannon divergence, and total variation distance, have been proposed to quantify behavioral and distributional shifts [3, 4, 19, 20]. Although these approaches reveal behavioral changes that aggregate metrics often miss, they do not explicitly capture the structural transformations occurring within the model parameters.

Structural analysis using TDA. TDA has been increasingly applied to deep learning to analyze neural network complexity, representation dynamics, robustness, and interpretability [21, 22, 23, 24, 25]. Recently, TDA has been explored for evaluating quantized neural networks [26]. However, its use for studying quantization-induced behavioral drift and attention consistency in LLMs remains largely unexamined.

Despite these advances, existing approaches primarily evaluate quantization using task performance and output distributional-scale or conventional statistical properties. These perspectives provide limited insight into the multiscale structural changes occurring within attention weight matrices and how these changes relate to behavioral consistency.

Notation and Core Definitions

In this section, we formalize PTQ as an operator on transformer parameters and define two metric spaces: (1) structural drift in the weight space and (2) decision-level consistency in the output space.

Notation and Model Formalism

Let V\mathcal{V} be a vocabulary and X⊆V∗\mathcal{X} \subseteq\mathcal{V}^{*} be the set of token sequences. We consider a decoder-only transformer with NN blocks and hidden dimension dd, parameterized by θ∈RP\theta\in\mathbb{R}^{P}, where PP is the total parameter count. For an input prompt x∈Xx \in\mathcal{X} and target sequence y=(y1,…,yT)∈V∗y = (y_1,\ldots,y_T) \in\mathcal{V}^{*}, the model induces next-token conditional probability distributions:

pθ(y∣x)=∏t=1Tpθ(yt∣x,y<t).(1)p_{\theta}(y \mid x) = \prod_{t=1}^{T} p_{\theta}(y_t \mid x,y_{<t}). \tag*{(1)}

For each transformer block index i∈[N]:={1,…,N}i \in[N] := \{1,\ldots,N\}, let WiQ,WiK,WiV∈Rd×dW_i^Q, W_i^K, W_i^V \in\mathbb{R}^{d \times d} denote the self-attention projection matrices. We collect all attention parameters into the set:

Wθ:={WiL:i∈[N],L∈{Q,K,V}}.(2)\mathcal{W}_{\theta} := \{W_i^L : i \in[N], L \in\{Q,K,V\}\}. \tag*{(2)}

Post-Training Quantization (PTQ) Operators

We define a finite set of quantization configurations C\mathcal{C}, where each c∈Cc \in\mathcal{C} specifies a bit width b(c)∈{2,3,4,5,6,8}b(c) \in\{2,3,4,5,6,8\} and a quantization scheme (e.g., K-quantization). Let Qc\mathcal{Q}_c and Dc\mathcal{D}_c represent the quantization and dequantization mapping functions. We model post-training quantization as an operator Tc:RP→RPT_c : \mathbb{R}^{P} \to\mathbb{R}^{P} acting on parameter space:

Tc(θ):=Dc(Qc(θ))∈RP.(3)T_c(\theta) := \mathcal{D}_c(\mathcal{Q}_c(\theta)) \in\mathbb{R}^{P}. \tag*{(3)}

The resulting quantized-dequantized model parameterization is denoted as Tc(θ)T_c(\theta), and its corresponding parameter matrix for the attention projection layer L∈{Q,K,V}L \in\{Q,K,V\} at block i∈[N]i \in[N] is defined as W^iL,(c):=WiL(Tc(θ))\widehat{W}_{i}^{L,(c)} := W_{i}^{L}(T_c(\theta)).

For a given block ii and projection layer LL, the entry-wise weight error matrix is defined as:

ΔW,iL,(c):=WiL−W^iL,(c).(4)\Delta_{W,i}^{L,(c)} := W_i^L - \widehat{W}_i^{L,(c)}. \tag*{(4)}

When evaluating across all parameters in the concatenated parameter vector θ\theta, the overall parameter perturbation vector is denoted as Δθ(c):=Tc(θ)−θ\Delta_{\theta}^{(c)} := T_c(\theta)-\theta.

Weight-Space Topological Drift

To quantify the structural changes induced by quantization in the attention projection weights, we considered a family of topological distance measures evaluated independently for each transformer block and projection type. Let M=[Betti,Entropy,Landscape,PFD,PI,SWD,TSWD]\mathcal{M} = [\text{Betti}, \text{Entropy}, \text{Landscape}, \text{PFD}, \text{PI}, \text{SWD}, \text{TSWD}] denote the set of topological distance measures considered [?]. For M∈MM \in\mathcal{M}, let δM(⋅,⋅)\delta_M(\cdot,\cdot) denote the corresponding distance between the topological representations of the two weight matrices.

For transformer block i∈[N]i \in[N] and projection type L∈{Q,K,V}L \in\{Q,K,V\}, the topological distance induced by quantization configuration cc is defined as

δM,iL,(c)(θ):=δM(W^iL,(c),WiL).(5)\delta_{M,i}^{L,(c)}(\theta) := \delta_M\left(\widehat{W}_i^{L,(c)},W_i^L\right). \tag*{(5)}

The averaged topological distance for projection type LL is then defined by

δ‾ML,(c)(θ):=1N∑i=1NδM,iL,(c)(θ),(6)\overline{\delta}_{M}^{L,(c)}(\theta) := \frac{1}{N}\sum_{i=1}^{N}\delta_{M,i}^{L,(c)}(\theta), \tag*{(6)}

where M∈MM \in\mathcal{M}.

Behavioral Consistency Metrics

To evaluate the model behavior on downstream task datasets D={(x(s),y(s))}s=1S\mathcal{D} = \{(x^{(s)},y^{(s)})\}_{s=1}^{S}, let gθ(x)g_\theta(x) denote the model’s predicted output under a deterministic scoring functional.

For each instance s∈[S]s \in[S], we define binary correctness indicators for the base model M0M_0 (parameterized by θ\theta) and quantized model MqM_q (parameterized by Tc(θ)T_c(\theta)):

C0(x(s))=I(gθ(x(s))=y(s)),(7)C_0(x^{(s)}) = \mathbb{I}\left(g_\theta(x^{(s)})=y^{(s)}\right), \tag*{(7)}
Cq(x(s))=I(gTc(θ)(x(s))=y(s)).(8)C_q(x^{(s)}) = \mathbb{I}\left(g_{T_c(\theta)}(x^{(s)})=y^{(s)}\right). \tag*{(8)}

Standard task-level accuracies are given by the expectations Acc⁡(M0)=E[C0(x)]\operatorname{Acc}(M_0)=\mathbb{E}[C_0(x)] and Acc⁡(Mq)=E[Cq(x)]\operatorname{Acc}(M_q)=\mathbb{E}[C_q(x)]. However, aggregate accuracy completely ignores instance-level behavioral stability—that is, whether MqM_q preserves correct predictions on the exact same individual prompts relative to M0M_0.

Definition 1 (Top-1 Exact Match Token). Let x=(x1,…,xT)∈Xx=(x_1,\ldots,x_T)\in\mathcal{X} be an input prompt sequence of length T=∣x∣T=|x|, with query position index q=Tq=T. Let Ai(h)∈RT×TA_i^{(h)}\in\mathbb{R}^{T\times T} denote the attention matrix at transformer block i∈[N]i\in[N] for attention head h∈{1,…,H}h\in\{1,\ldots,H\}, where Ai,q,t(h)A_{i,q,t}^{(h)} represents the attention score assigned by the query token xqx_q to target token xtx_t.

The head-averaged attention weight vector for the query token xqx_q at block ii is defined as:

A‾i,q(t)≜1H∑h=1HAi,q,t(h),∀t∈{1,…,T}.(9)\overline{A}_{i,q}(t) \triangleq\frac{1}{H}\sum_{h=1}^{H}A_{i,q,t}^{(h)},\qquad\forall t\in\{1,\ldots,T\}. \tag*{(9)}

To restrict evaluation strictly to non-special context tokens, such as markers for the beginning or end of a sequence, let S⊂V\mathcal{S}\subset\mathcal{V} denote the set of special vocabulary tokens, and define the masked attention vector A~i,q(t)\widetilde{A}_{i,q}(t) by setting self-attention and special tokens to −∞-\infty:

A~i,q(t)≜{−∞,if t=q or xt∈S,A‾i,q(t),otherwise.(10)\widetilde{A}_{i,q}(t)\triangleq \begin{cases} -\infty, & \text{if }t=q\text{ or }x_t\in\mathcal{S},\\ \overline{A}_{i,q}(t), & \text{otherwise}. \end{cases} \tag*{(10)}

The Top-1 Exact Match token index t⋆∈{1,…,T−1}t^\star\in\{1,\ldots,T-1\} is defined as the context position attaining the maximum masked head-averaged attention score:

t⋆≜arg⁡max⁡t∈{1,…,T} A~i,q(t).(11)t^\star\triangleq\underset{t\in\{1,\ldots,T\}}{\arg\max}\ \widetilde{A}_{i,q}(t). \tag*{(11)}

Definition 2 (Top-3 IoU Attention Similarity). Let K=3K=3. For a given prompt sequence x(s)∈Dx^{(s)}\in\mathcal{D} at transformer block i∈[N]i\in[N], let T0(K)(s,i)⊆V\mathcal{T}_0^{(K)}(s,i)\subseteq\mathcal{V} denote the ordered set of top-KK attended non-special context tokens retrieved under the base parameterization θ\theta. Similarly, let Tq(K)(s,i)⊆V\mathcal{T}_q^{(K)}(s,i)\subseteq\mathcal{V} denote the corresponding set under the quantized parameterization Tc(θ)T_c(\theta).

Filtering out null or unassigned elements, let the sets of non-null extracted tokens be defined as:

U0≜{t∈T0(K)(s,i)∣t≠∅},\mathcal{U}_0\triangleq\left\{t\in\mathcal{T}_0^{(K)}(s,i)\mid t\ne\varnothing\right\},

and

Uq≜{t∈Tq(K)(s,i)∣t≠∅}.(12)\mathcal{U}_q\triangleq\left\{t\in\mathcal{T}_q^{(K)}(s,i)\mid t\ne\varnothing\right\}. \tag*{(12)}

The Top-3 Intersection over Union (IoU) similarity measure between the base model M0M_0 and the quantized model MqM_q at block ii for instance ss is defined using the Jaccard index:

IoU⁡K(M0,Mq;s,i)≜{1.0,if U0=Uq=∅,∣U0∩Uq∣∣U0∪Uq∣,otherwise.(13)\operatorname{IoU}_K(M_0,M_q;s,i)\triangleq \begin{cases} 1.0, & \text{if }\mathcal{U}_0=\mathcal{U}_q=\varnothing,\\ \frac{|\mathcal{U}_0\cap\mathcal{U}_q|}{|\mathcal{U}_0\cup\mathcal{U}_q|}, & \text{otherwise}. \end{cases} \tag*{(13)}

The aggregate Top-3 IoU similarity across all SS instances in dataset D\mathcal{D} for block ii is given by the expectation:

IoU⁡‾K(M0,Mq;i)≜1S∑s=1SIoU⁡K(M0,Mq;s,i).(14)\overline{\operatorname{IoU}}_K(M_0,M_q;i)\triangleq\frac{1}{S}\sum_{s=1}^{S}\operatorname{IoU}_K(M_0,M_q;s,i). \tag*{(14)}

TopoQuant Design

TopoQuant framework, illustrated in Figure 1, consists of four main stages: (1) model quantization and attention-weight extraction, (2) topological analysis and drift measurement, (3) attention agreement analysis, and (4) lightweight machine learning based prediction. At a high level, TopoQuant extracts self-attention weight matrices (Q, K, V) from base and quantized LLMs, constructs persistence diagrams using persistent homology to compute multi-metric topological drift across transformer blocks, and measures functional attention

Overview of the TopoQuant framework

Figure 1. Overview of the TopoQuant framework. The framework consists of four stages: model quantization and weight extraction, topological drift measurement, attention agreement analysis, and lightweight machine learning prediction. Topological features extracted from persistence diagrams are combined with model characteristics to predict attention consistency between base and quantized models.

consistency using Top-1 Exact Match and Top-3 IoU. These topological features are combined with model characteristics and passed to lightweight machine learning regressor to predict downstream attention behavioral shifts.

Model Quantization and Attention Weight Extraction

The first stage is to construct multiple quantized variants of a pre-trained LLM at different quantization settings. The full-precision model is called the base model. For a given transformer block ℓ\ell, the attention projections are characterized by the query, key, and value weight matrices, WQW_Q, WKW_K, WVW_V. The weight matrices are extracted from both the base (MOM_O) and each quantized variant (MqM_q). We extract weights across all blocks to systematically compare quantization changes at different depths. For each transformer block i∈[N]:={1,…,N}i \in[N] := \{1,\ldots,N\} and attention projection L∈{Q,K,V}L \in\{Q,K,V\}, the framework obtains a pair of matrices WL,base(i)W_{\mathrm{L,base}}^{(i)} and WL,quant(i)W_{\mathrm{L,quant}}^{(i)}, which serve as the input to the topological analysis.

Topological Analysis and Drift Measurement

TopoQuant characterizes the quantization-induced structural drift by measuring the distances between persistence diagrams generated from the base and quantized attention weight matrices. We constructed an appropriate representation and filtration to generate the corresponding persistence diagram (PD). Let PDbase(i,L)\mathrm{PD}_{\mathrm{base}}^{(i,L)} and PDquant(i,L)\mathrm{PD}_{\mathrm{quant}}^{(i,L)} denote the persistence diagrams obtained from the base and quantized models, respectively, for projection LL in transformer block ii. Then, the topological differences between the two models are measured using Dtopo(i,L)=dtopo(PDbase(i,L),PDquant(i,L))D_{\mathrm{topo}}^{(i,L)} = d_{\mathrm{topo}}\left(\mathrm{PD}_{\mathrm{base}}^{(i,L)}, \mathrm{PD}_{\mathrm{quant}}^{(i,L)}\right), where the distance represents the topological drift by quantization. We repeated this step across the transformer blocks, attention projections, quantization settings, and selected topological distances.

The framework measures the structural drift between the persistence diagrams of the base and quantized models using a diverse set of topological distance metrics. These measures capture the distinct geometric and statistical properties of the parameter manifolds: the Betti distance and Persistence Landscape measure the discretized L2L_2 norm between Betti curves and landscape functions, respectively; Persistence Entropy captures the overall structural complexity by computing the absolute difference between topological entropy values; the Persistence Fisher Distance (PFD) computes the Fisher–Rao distance between kernel density representations of the topological points; and Persistence Images (PI) convert diagrams into weighted two-dimensional Gaussian functions to calculate their Euclidean distance. Finally, the Sliced Wasserstein Distance (SWD) and Tree-Sliced Wasserstein Distance (TSWD) project persistence points onto one-dimensional lines and dyadic tree structures, respectively, to aggregate projection differences across multiple directions.

Attention Agreement Analysis

The third stage evaluates whether the structural changes identified in Phase 2 correspond to changes in the model’s functional attention behavior. Therefore, we processed the

same evaluation prompts independently with the base and quantized models and collected their attention maps for comparison. We considered two complementary measures of attention agreement.

Top-1 exact-match agreement. For each query token, the key token receiving the highest attention value is identified independently for the base and quantized models. tbase∗=arg⁡max⁡jAbase(i,j)t^*_{\mathrm{base}} = \arg\max_j A_{\mathrm{base}}(i,j) and tquant∗=arg⁡max⁡jAquant(i,j)t^*_{\mathrm{quant}} = \arg\max_j A_{\mathrm{quant}}(i,j), where A(i,j)A(i,j) denotes the attention value from the query token to the key token. The Top-1 agreement indicator is defined as

Mi={1if tbase∗=tquant∗,0otherwise.M_i = \begin{cases} 1 & \text{if } t^*_{\mathrm{base}} = t^*_{\mathrm{quant}},\\ 0 & \text{otherwise.} \end{cases}

Top-3 attention agreement. To provide a less restrictive measure of attention preservation, the three most strongly attended tokens are extracted from the base and quantized variant. Let Sbase(i)S_{\mathrm{base}}(i) and Squant(i)S_{\mathrm{quant}}(i) represent the corresponding Top-3 token sets for query token in block ii. Their agreement is quantified using intersection over union: IoUi=∣Sbase(i)∩Squant(i)∣∣Sbase(i)∪Squant(i)∣\mathrm{IoU}_i = \frac{\lvert S_{\mathrm{base}}(i) \cap S_{\mathrm{quant}}(i)\rvert}{\lvert S_{\mathrm{base}}(i) \cup S_{\mathrm{quant}}(i)\rvert}. The resulting agreement scores are then aggregated across the relevant query tokens, attention heads, blocks, and evaluation prompts to produce the average IoU agreement.

Lightweight Machine Learning Prediction

The final phase investigates whether topological drift measured from the attention weights can predict the average IoU agreement between the base and quantized variants for each block. Predicting attention agreement is valuable because it provides an early estimate of quantization-induced behavioral degradation without requiring expensive downstream evaluation. If attention agreement can be accurately predicted from structural features such as topological drift, practitioners can identify problematic quantization settings and select safe compression levels before performing extensive benchmarking.

We organize the topological distance drift from Step 2 and the LLM features (family, number of parameters, overall quantization level, quantization level of each projection, and block category [beginning, middle, late]) into feature vectors that capture the structural changes associated with each quantization configuration. We categorize LLM blocks into Early, Middle, and Late based on [27], which reveals a three-phase internal structure: early blocks align inputs into a shared semantic space, middle blocks perform task reasoning, and late blocks drive language-specific generation. A generic feature vector can be expressed as

x=[ffamily,fparams,fquant,fQ_Quant,fK_Quant,fV_Quant,flayer_category,Dtopo(1,Q),Dtopo(1,K),Dtopo(1,V),…,Dtopo(L,V)].\mathbf{x} = \left[f_{\mathrm{family}}, f_{\mathrm{params}}, f_{\mathrm{quant}}, f_{Q\_\mathrm{Quant}}, f_{K\_\mathrm{Quant}}, f_{V\_\mathrm{Quant}}, f_{\mathrm{layer\_category}}, D_{\mathrm{topo}}^{(1,Q)}, D_{\mathrm{topo}}^{(1,K)}, D_{\mathrm{topo}}^{(1,V)}, \ldots, D_{\mathrm{topo}}^{(L,V)}\right].

These feature vectors are paired with IoU obtained in phase 3, where the Top-3 IoU serves as the prediction target. The framework constructs feature configurations based on individual attention projection matrices ((Q, K, V), pairwise combinations (KQ, KV, and QV), and their joint representation (KQV) to study the effect of single and group projection in predicting the Top-3 IoU. For each feature configuration, we train and evaluate three lightweight machine learning algorithms (LinearRegression, RandomForest, and XGBoost). We compare the models’ predictive performance based on prediction error, and select the model with the lowest error as the best-performing configuration.

Evaluations

To evaluate TopoQuant, we addressed the following research questions:

  • RQ1: How do the topological structure of LLM weights and attention agreement change across quantization levels?

  • RQ2: Can TDA reliably determine the optimal stopping point in LLM quantization without fully retraining or re-evaluating the model?

  • RQ3: Can TDA detect the behavioral shift?

  • RQ4: RQ4: Which attention projection matrices are most informative for quantization analysis?

  • RQ5: What are the ablation study results for various TopoQuant components and hyperparameters?

Implementation

Models: We quantize the following four models: Llama-3.2-3B, Vicuna-7B-v1.5, Mistral-7B-v0.1, and Llama-3.1-8B into legacy quantization methods (Q8_0, Q5_0, Q4_0) and K-quantization methods (Q6_K, Q5_K, Q4_K, Q3_K, Q2_K) using llama.cpp [28]. We extract the attention matrices (Q, K, V) weights of each block of the base models and its quantized variants after dequantization.

V) weights of each block of the base models and its quantized variants after dequantization. Notably, K-quantization methods allocate higher precision to the value (VV) projection than to the other weights in the quantized configurations considered. Specifically, the VV projection uses Q6_K in both Q5_K and Q4_K, Q5_K in Q3_K, and Q4_K in Q2_K. Downstream task evaluations (accuracy and perplexity) were conducted on an NVIDIA Tesla H100 80GB PCIe GPU, while TopoQuant calculations were performed on a workstation equipped with an Intel Core i7-14700F CPU (2.10 GHz), 32 GB RAM, and an NVIDIA GeForce RTX 3090 GPU.

Persistence Diagram: We convert each attention weight matrix XX into an N×NN \times N pairwise cosine distance matrix DD. The distance matrix is then provided to Ripser [29] with a filtration threshold of 0.8. As the filtration radius rr increases, Ripser records the birth and death scales (rbirth,rdeath)(r_{\mathrm{birth}}, r_{\mathrm{death}}) of connected components (H0H_0). The resulting persistence diagram captures the evolution of connectivity within the attention weight space and is visualized using Persim [30] and serialized into .pkl files for subsequent analysis.

Topological Drift. We compute the topological distance metrics introduced in Section IV-B between the persistence diagrams of the base and quantized models across layers and attention projections (Q, K, V). We save the resulting distances, runtime, and layer metadata for benchmarking.

Datasets: We use zero-shot multiple-choice benchmarks including ARC (AI2 Reasoning Challenge) [31], HellaSwag [32], Winogrande [33]. ARC (AI2 Reasoning Challenge) [31] evaluates grade-school science question answering. HellaSwag [32] evaluates commonsense reasoning through context completion. WinoGrande [33] evaluates commonsense reasoning and pronoun resolution.

Attention Agreement. We evaluate base and quantized models on 500 samples from each dataset and extract block-wise self-attention scores. For each prompt, we analyze the attention distribution of the final query token while masking self-attention and special tokens. We identify the top-3 attended context tokens and compare quantized and base models using Top-1 Exact Match and Top-3 Intersection-over-Union (IoU) after aligning records by [model_name, block]. Item-level, block-level, and global summaries are then generated.

We use top-3 attention agreement for three reasons. First, prior studies show that only a small subset of tokens typically receives most of the attention mass in decoder-only LLMs [34, 35]. Second, top-1 matching is overly sensitive to minor ranking fluctuations that do not necessarily reflect meaningful behavioral changes. Third, larger sets (top-5 +) introduce additional noise. Therefore, top-3 provides a practical balance between stability and sensitivity when detecting quantization-induced attention drift.

Lightweight Machine Learning Prediction. We use the topological drift features and model metadata described in Section IV-D to predict IoU. The feature configurations include individual projections (Q, K, V), pairwise combinations (KQ, KV, QV), and the combined representation (KQV). Linear regression, random forest, and XGBoost were evaluated using 5-fold cross-validation.

RQ1: topological structure and attention agreement change across quantization levels: Both topological structure and attention agreement remain largely stable under moderate quantization before degrading sharply at sub-4-bit precision. As shown in Figure 2 and Table I, low structural drift corresponds closely to high attention agreement. At higher precisions (Q8_0 to Q5_K), SWD drift remains near zero while Top-1 Agreement and Top-3 IoU typically exceed 0.98. In contrast, aggressive quantization (Q3_K and Q2_K) produces large increases in topological drift accompanied by substantial declines in attention agreement, with Top-1 Agreement dropping to 0.61 and Top-3 IoU falling to 0.58 in some configurations. The degradation is concentrated in the Q and K projections, which exhibit substantially larger topological distortions than the V projection. The lower drift observed in the V projection may be partly influenced by the K-quantization schemes used in this study, which allocate higher precision to V than to other attention weights. Furthermore, mid-sized models such as Vicuna-7B and Mistral-7B preserve both structural and behavioral consistency more effectively than smaller models such as Llama-3B across all quantization levels.

Sliced Wasserstein Distance drift across quantization levels for Query, Key, and Value projection matrices

Figure 2. Sliced Wasserstein Distance (SWD) drift across quantization levels for Query (Q), Key (K), and Value (V) projection matrices.

MetricQuantLlama-3B ArcLlama-3B Hell.Llama-3B Wino.Vicuna-7B ArcVicuna-7B Hell.Vicuna-7B Wino.Mistral-7B ArcMistral-7B Hell.Mistral-7B Wino.Llama-8B ArcLlama-8B Hell.Llama-8B Wino.
Top-1 AgreementQ8_00.740.880.680.991.001.000.991.000.990.750.870.71
Top-1 AgreementQ6_K0.740.880.670.971.000.990.970.980.960.750.870.71
Top-1 AgreementQ5_00.730.880.680.940.990.980.930.960.920.730.860.68
Top-1 AgreementQ5_K0.740.880.670.960.990.990.960.970.950.750.860.71
Top-1 AgreementQ4_00.720.870.670.880.990.950.890.930.830.600.850.68
Top-1 AgreementQ4_K0.730.870.670.930.990.980.920.950.900.720.860.70
Top-1 AgreementQ3_K0.700.850.640.870.980.950.860.910.830.710.840.66
Top-1 AgreementQ2_K0.610.780.560.740.980.860.660.840.710.610.770.60
Top-3 IoUQ8_00.750.830.701.000.990.990.990.990.990.740.830.73
Top-3 IoUQ6_K0.750.830.700.990.980.970.970.980.960.730.830.73
Top-3 IoUQ5_00.740.820.710.970.960.930.940.950.930.720.830.72
Top-3 IoUQ5_K0.750.830.710.980.970.960.960.970.950.730.830.73
Top-3 IoUQ4_00.740.810.690.940.920.860.890.920.840.700.820.71
Top-3 IoUQ4_K0.740.820.700.970.950.930.930.940.910.720.820.72
Top-3 IoUQ3_K0.690.800.670.930.910.890.870.900.840.700.810.70
Top-3 IoUQ2_K0.610.730.590.860.830.730.660.820.720.580.750.63

Table I. Attention Agreement Metrics Across Quantization Configurations. Evaluation of Top-3 Intersection over Union (IoU) and Top-1 Agreement for ARC, HellaSwag, and Winogrande datasets across four LLM architectures.

RQ2: TDA as a reliable metric: The results suggest that TDA can identify practical quantization stopping points without exhaustive downstream evaluation. As shown in Figure 3, SWD exhibits a consistent inverse relationship with Top-3 IoU, where increasing structural drift is accompanied by declining attention agreement. Quantization levels from Q8_0 to Q5_K remain clustered in a low-drift, high-agreement region, whereas Q3_K and Q2_K move rapidly toward high-drift, low-agreement regions. This clear transition indicates that topological drift can serve as an effective indicator of quantization-induced degradation and help identify safe compression limits. Query (QQ) and Key (KK) projections exhibit the strongest correspondence between structural drift and attention degradation, while the Value (VV) projection shows consistently lower drift, potentially influenced by the higher precision assigned to V in the K-quantization schemes used in this study. Similar trends were observed across the other evaluated topological distance metrics, including Betti distance, persistence entropy, persistence landscapes, persistence Fisher distance (PFD), persistence images, and tree-sliced Wasserstein distance (TSWD). These findings suggest that TDA provides a lightweight alternative for monitoring quantization quality and identifying performance cliffs without requiring extensive downstream benchmarking.

Model trajectories across projections: the average SWD distance vs. Top-3 IoU similarity

Figure 3. Quantization degradation trajectories across attention projection matrices. Each curve tracks an LLM model across eight quantization level (ranging from high-precision q8_0 to extreme low-bit q2_k, indicated by marker shape). The xx-axis shows the average Sliced Wasserstein Distance (SWD) structural drift, while the yy-axis displays downstream behavioral preservation measured via Top-3 IoU Similarity on the ARC benchmark. The exponential divergence in Query (Q) and Key (K) projections highlights their structural vulnerability, validating SWD as an efficient proxy for tracking performance cliffs without full benchmark re-evaluation.

RQ3: TDA and the behavioral shift TDA integrated with LLM representations (the combined feature set) effectively predicts the behavioral shift (measured by IoU) across benchmark datasets. Based on the performance evaluation in Table II: (1) The combined feature representation achieves high predictive performance across all benchmark datasets, consistently reaching R2R^2 scores of up to 0.85 on HellaSwag, 0.83 on Arc, and 0.83 on Winogrande. (2) Both Random Forest and Linear Regression demonstrate superior predictive capabilities within the combined feature space compared to XGBoost. Random Forest consistently delivers robust scores across all matrix configurations (R2≈0.74R^2 \approx0.74–0.85), while Linear Regression achieves its strongest performance on HellaSwag and Winogrande (R2≈0.81R^2 \approx0.81–0.83). (3) High predictive power is maintained across individual matrices (Q, K, V), pairwise interactions (KQ, KV, QV), and the joint configuration (KQV), confirming that TopoQuant captures a reliable and stable signal for predicting attention agreement shifts.

Feature MatrixAlgorithmArc LLMArc TDAArc Comb.Hell. LLMHell. TDAHell. Comb.Wino. LLMWino. TDAWino. Comb.
Query (Q)LinearRegression0.680.260.760.530.270.830.750.220.81
Query (Q)XGBoost0.490.210.660.470.110.510.560.200.66
Query (Q)RandomForest0.670.330.830.720.350.850.740.310.83
Key (K)LinearRegression0.680.210.730.530.220.820.750.190.82
Key (K)XGBoost0.490.210.650.470.090.500.560.190.65
Key (K)RandomForest0.670.330.820.720.390.840.740.350.82
Value (V)LinearRegression0.68–0.750.53–0.830.75–0.15
Value (V)XGBoost0.490.210.600.470.030.470.560.180.60
Value (V)RandomForest0.670.330.750.720.110.740.740.290.75
Pairwise (KQ)LinearRegression0.680.280.760.530.290.830.750.260.82
Pairwise (KQ)XGBoost0.490.210.670.470.110.510.560.210.67
Pairwise (KQ)RandomForest0.670.330.820.720.390.840.740.370.82
Pairwise (KV)LinearRegression0.68–0.730.53–0.820.75–0.72
Pairwise (KV)XGBoost0.490.210.650.470.090.500.560.220.65
Pairwise (KV)RandomForest0.670.330.820.720.380.840.740.330.82
Pairwise (QV)LinearRegression0.68–0.760.53–0.830.75–0.69
Pairwise (QV)XGBoost0.490.210.670.470.110.510.560.250.67
Pairwise (QV)RandomForest0.670.330.820.720.360.840.740.340.82
Joint (KQV)LinearRegression0.680.310.750.53–0.720.75–0.73
Joint (KQV)XGBoost0.490.210.650.470.110.510.560.240.65
Joint (KQV)RandomForest0.670.310.820.720.410.850.740.320.82

Table II. Ablation Study of Feature Representations on Attention Agreement (R2R^2 for Top-3 IoU) across Benchmarks. Comparison across LLM features only, TDA features only, and combined (LLM + TDA) feature configurations. Top values per column are highlighted in green and lowest values in red.

RQ4:RQ4: attention projection matrices most informative for quantization analysis: Based on the combined feature evaluation (LLM + TDA) in Table II, the Q and K matrices, along with their pairwise interaction (KQ), emerge as the most informative indicators of quantization-induced attention degradation. Features derived from Q and K consistently achieve the highest predictive performance across all benchmarks. On HellaSwag, both Q and K reach peak R2R^2 scores of 0.85 (Random Forest) and 0.83 (Linear Regression), outperforming or matching all other configurations. Similarly, the KQ combination achieves strong predictive accuracy across ARC (R2=0.76R^2 = 0.76–0.82), HellaSwag (R2=0.84R^2 = 0.84), and Winogrande (R2=0.82R^2 = 0.82), indicating that structural changes within the attention routing mechanism are closely associated with attention agreement degradation. In contrast, configurations relying solely on the V projection generally exhibit lower predictive performance. For example, Linear Regression achieves an R2R^2 of only 0.15 on Winogrande using V alone, compared with 0.81 and 0.82 for Q and K, respectively. Although Random Forest improves V-based performance (R2R^2 = 0.74–0.75), it remains below the best-performing Q, K, and KQ based configurations. Combining all three matrices (KQV) maintains strong performance (R2R^2 up to 0.85), but does not meaningfully outperform Q, K, or KQ alone. These results suggest that structural changes within the Q and K projections are more strongly associated with attention degradation than those observed in the V projection, making them particularly useful for monitoring quantization-induced behavioral drift.

RQ5: ablations on feature representations and TopoQuant components

To isolate the individual predictive contributions of topological representation learning, we conduct a systematic ablation study comparing three distinct feature configurations: (1) Model Features Only (comprising 5 base configuration and layer-category indicators: quant, layer_category, q_quant, k_quant, and v_quant),

(2) TDA Features Only (comprising 21 topological persistence features extracted across the Q, K, and V weight matrices—specifically Betti curves, entropy, landscapes, persistence images, SWD and TSWD), and (3) Combined Features (5 + 21 = 26 total dimensions formed by concatenating the model and TDA feature spaces), as shown in Table II. The evaluation yields the following key insights regarding feature contributions across architectures and benchmarks: (1) Across all benchmarks and attention matrix configurations, the combined feature space consistently yields the highest predictive accuracy (R2R^2), outperforming both individual feature sets. On HellaSwag, the combined features raise R2R^2 to 0.85 (Random Forest across Q, K, and KQV), compared with 0.72 for LLM features only and 0.41 for standalone TDA features, confirming that topological metrics provide complementary signals that improve baseline model performance. (2) While LLM Features Only offered a moderate predictive baseline (R2≈0.67R^2 \approx0.67–0.75 on Arc and Winogrande using Random For est/Linear Regression), they failed to capture subtle behavioral shifts on complex benchmarks. Conversely, TDA Features Only yield modest standalone performance (R2≈0.09R^2 \approx0.09–0.410.41), and frequently struggle or produce negative/near-zero fits when evaluated independently on the V matrix under linear assumptions. (3) Non-linear ensemble methods (Random Forest) excelled at leveraging the combined feature space, maintaining stable R2R^2 scores (0.74–0.85) across all matrix views (Q, K, V, KQ, KV, QV, KQV). Linear Regression demonstrates strong performance when combining features on HellaSwag (R2=0.82R^2 = 0.82–0.83 for Q, K, KQ), but exhibits instability on V-dependent features alone, underscoring the importance of non-linear modeling for capturing complex topological dynamics.

TopoQuant overhead vs. downstream runtime: To demonstrate the runtime efficiency and light overhead of TopoQuant, Table III compares the runtime (in seconds) required to extract topological metrics—including persistence diagrams, topological distance calculations, and layer-wise attention agreement—against the execution time of the downstream ARC benchmark across various quantization schemes on Llama-3.1-8B.

QuantTopoQuantDownstream
Q8_08624619
Q6_K9374647
Q5_09314603
Q5_K9374627
Q4_09034709
Q4_K9144636
Q3_K12504708
Q2_K11094694

Table III. TopoQuant Overhead vs. Downstream Runtime: To demonstrate the effectiveness of our framework, we compare the time required to compute TopoQuant (persistence diagrams, topological distances, and attention agreement on the Arc dataset) against the time required for downstream task execution alone on Llama-3.1-8B.

Across all evaluated quantization levels (from 8-bit to 2-bit), the computational overhead introduced by TopoQuant remained a fraction of the total downstream execution duration. For high-to-medium precision variants (Q8_0 through Q4_K), TopoQuant computation requires between 862 and 937 seconds, accounting for approximately 16% to 17% of the downstream evaluation time (which ranges from 4,603 to 4,709 seconds). Even at extreme quantization levels (Q3_K and Q2_K), where the metric extraction time increases slightly—peaking at 1,250 s for Q3_K—the diagnostic overhead remains within 21.0%–26.5% of the downstream evaluation runtime.

Discussion

The empirical findings from the TopoQuant evaluation provide several key insights into the structural and functional mechanisms underlying PTQ in LLMs. In this section, we discuss the broader implications for LLM evaluation, interpretability, and quantization design.

Topological Drift as an Early Warning Signal: A central finding of this work is that conventional benchmark accuracy and perplexity can be highly misleading indicators of model integrity under quantization. As shown in [4] evaluation of the same LLMs we are studying, a quantized model can maintain perplexity and accuracy while simultaneously experiencing shifts in its internal attention allocation patterns—a phenomenon previously termed the “illusion of preservation”.

TopoQuant bridges this diagnostic gap by establishing that topological weight drift serves as an early, highly sensitive warning signal. Because topological metrics capture multi-scale structural deformations on the parameter manifold rather than aggregate statistical averages, they detect the onset of weight matrix breakdown well before downstream decision-making collapses. This capability lets practitioners set principled, data-free quantization stopping points (e.g., before the sub-4-bit exponential drift threshold) without running expensive, full-dataset evaluation benchmarks.

Asymmetric vulnerability and attention routing bottlenecks: Weight matrices responsible for attention routing (Q and K) exhibit substantially larger topological drift under aggressive compression than other attention components. This drift closely aligns with the observed reductions in Top-1 Agreement and Top-3 IoU. In contrast, V projections consistently exhibit lower structural drift across models and quantization levels. However, this observation should be interpreted with caution, as the K-quantization schemes used in this study allocate higher precision to the V projection than to other attention weights, which may partially contribute to its increased stability.

Computational Efficiency and Runtime Overhead: A critical requirement for structural diagnostics is that metric extraction must remain significantly cheaper than exhaustive downstream task evaluation. As demonstrated in Table III, TopoQuant introduces low computational overhead relative to full benchmark runs on Llama-3.1-8B. These results confirm that TopoQuant provides deep structural and topological diagnostics at a reasonable computational cost, enabling com-

prehensive quantization analysis without prohibitive runtime penalties relative to standard benchmark evaluations.

Predictive Power and Practical Deployment

Our lightweight machine learning experiments demonstrate that combining LLM-level features with multi-scale topological drift descriptors (LLM + TDA) yields highly accurate predictions of downstream attention agreement (R2R^2 scores up to 0.85). By relying on lightweight topological extraction via persistent homology and non-linear regressors (such as Random Forest), TopoQuant provides an efficient, parameter-space diagnostic pipeline. Model developers can inspect the structural stability of quantized checkpoints without requiring extensive downstream benchmark evaluation, reducing the computational burden associated with evaluating multiple quantization configurations.

Limitations and Future Directions

While TopoQuant provides a robust framework for structural drift analysis, certain limitations remain: (1) Current evaluations focused on self-attention projection matrices (Q, K, V). Extending persistent homology analysis to multi-layer perceptron (MLP) blocks and embedding could provide a more holistic view of full-network topological drift. (2) The current study evaluated standard uniform and K-quantization variants within llama.cpp. Future work will explore applying TopoQuant to non-uniform, outlier-preserving quantization methods such as AWQ and SmoothQuant. (3) Computing persistence diagrams on large weight matrices (d×dd \times d) incurs high computational complexity (O(d3)O(d^3) for exact filtrations). Although sliced variants (SWD, TSWD) mitigate this cost, scaling TDA to models exceeding 70B parameters requires further optimization, such as mini-batch topological sampling or low-rank projection filtrations. (4) The K-quantization formats evaluated in this study allocated different precisions to the Query, Key, and Value projections. Consequently, the lower drift observed in the Value projection may reflect both intrinsic robustness and reduced quantization severity. Future work should investigate projection-specific sensitivity under controlled quantization settings.

Conclusion

In this study, we introduced TopoQuant, a topological framework for characterizing and predicting quantization-induced structural and behavioral changes in large language models (LLMs). Moving beyond conventional metrics such as accuracy and perplexity, TopoQuant leverages persistent homology to quantify structural drift in self-attention weight matrices and relate these changes to attention behavior. Our evaluation across multiple LLM families and quantization levels demonstrates a strong relationship between topological drift and attention agreement. Topological drift remains limited at moderate quantization levels but increases sharply under aggressive compression, closely corresponding to declines in Top-1 Agreement and Top-3 IoU. These results indicate that topological representations provide a meaningful view of quantization-induced degradation that is not captured by traditional evaluation metrics alone. We further show that structural changes within the Query and Key projections are more strongly associated with attention degradation than those observed in the Value projection, making them particularly informative for quantization analysis. In addition, combining model-level characteristics with topological features enables lightweight machine learning models to predict attention agreement with R2R^2 scores of up to 0.85, reducing the need for extensive downstream benchmarking. Overall, TopoQuant provides a practical framework for monitoring quantization quality, identifying safe compression operating regions, and studying the relationship between structural and behavioral changes in quantized LLMs. We hope that this work encourages further exploration of topology-aware methods for efficient evaluation, diagnosis, and optimization of compressed foundation models.

Acknowledgment

This work was partially supported by (a) NSERC (Canada), (b) University of Manitoba, and (c) Red River College Polytechnic.

References

  1. [1]Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Trans. Assoc. Comput. Linguistics, 12:1556–1577, 2024.arxiv.org/abs/2308.07633
  2. [2]Baha Rababah, Yuzhang Shang, Carson K. Leung, Cuneyt Akcora, and Mubarak Shah. Post-training quantization for large language models: A survey. Available at SSRN 7518402, September 2026.DOI
  3. [3]Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee. Accuracy is not all you need. In Amir Globerson, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.arxiv.org/abs/2407.09141
  4. [4]Baha Rababah, Shahzeb Qamar, Lorenz Sprenberger, Rafet Sifa, Murat Kantarcioglu, Cuneyt Gurcan Akcora, and Carson K Leung. The illusion of equivalency: Statistical characterization of quantization effects in llms. arXiv preprint arXiv:2607.08734, 2026.arxiv.org/abs/2607.08734
  5. [5]Herbert Edelsbrunner and Dmitriy Morozov. Persistent homology: theory and practice. eScholarship, University of California, 2013.DOI
  6. [6]Gunnar E. Carlsson, Afra Zomorodian, Anne D. Collins, and Leonidas J. Guibas. Persistence barcodes for shapes. In Jean-Daniel Boissonnat and Pierre Alliez, editors, Second Eurographics Symposium on Geometry Processing, Nice, France, July 8-10, 2004, volume 71 of ACM International Conference Proceeding Series, pages 124–135. Eurographics Association, 2004.
  7. [7]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017.arxiv.org/abs/1706.03762
  8. [8]Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
  9. [9]Wen Cheng, Shichen Dong, Jiayu Qin, and Wei Wang. QAQ: quality adaptive quantization for LLM KV cache. In IEEE/CVF International Conference on Computer Vision, ICCV 2025 - Workshops, Honolulu, HI, USA, October 19-20, 2025, pages 2563–2571. IEEE, 2025.arxiv.org/abs/2403.04643
  10. [10]Mason Kadem and Rong Zheng. Interpreting transformers through attention head intervention. CoRR, abs/2601.04398, 2026.arxiv.org/abs/2601.04398
  11. [11]Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. A comprehensive evaluation of quantization strategies for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 12186–12215. Association for Computational Linguistics, 2024.arxiv.org/abs/2402.16775
  12. [12]Jiaqi Zhao, Ming Wang, Miao Zhang, Yuzhang Shang, Xuebo Liu, Yaowei Wang, Min Zhang, and Liqiang Nie. Benchmarking post-training quantization in llms: Comprehensive taxonomy, unified evaluation, and comparative analysis. CoRR, abs/2502.13178, 2025.arxiv.org/abs/2502.13178
  13. [13]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: accurate post-training quantization for generative pre-trained transformers. CoRR, abs/2210.17323, 2022.arxiv.org/abs/2210.17323
  14. [14]Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.arxiv.org/abs/2308.13137
  15. [15]Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojian Qi. Slim-llm: Salience-driven mixed-precision quantization for large language models. CoRR, abs/2405.14917, 2024.arxiv.org/abs/2405.14917
  16. [16]Eldar Kurtic, Alexandre Noll Marques, Shubhra Pandit, Mark Kurtz, and Dan Alistarh. ”give me BF16 or give me death”? accuracy-performance trade-offs in LLM quantization. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 26872–26886. Association for Computational Linguistics, 2025.
  17. [17]Ajay Kumar Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. Compressing llms: The truth is rarely pure and never simple. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.arxiv.org/abs/2310.01382
  18. [18]Björn Deiseroth, Max Meuer, Nikolas Gritsch, Constantin Eichenberg, Patrick Schramowski, Matthias Aßenmacher, and Kristian Kersting. Divergent token metrics: Measuring degradation to prune away LLM components - and optimize quantization. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 6764–6783. Association for Computational Linguistics, 2024.
  19. [19]Milos Nikolic, Ali Hadi Zadeh, Enrique Torres-Sánchez, and Andreas Moshovos. Displacement is not direction: Evaluating fidelity metrics for quantized LLM deployment. CoRR, abs/2606.19558, 2026.DOI
  20. [20]Shahzeb Qamar, Lorenz Sparrenberg, Christian Bauckhage, Baha Rababah, Carson Leung, Murat Kantarcıoğlu, Cuneyt Gurcan Akcora, and Rafet Sifa. Accuracy is not enough: A divergence-based approach to evaluate fidelity loss in quantized llms. arXiv preprint arXiv:2609.07664, 2026.arxiv.org/abs/2609.07664
  21. [21]Christoph D. Hofer, Roland Kwitt, Marc Niethammer, and Andreas Uhl. Deep learning with topological signatures. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1634–1644, 2017.arxiv.org/abs/1707.04041
  22. [22]Bastian Rieck, Matteo Togninalli, Christian Bock, Michael Moor, Max Horn, Thomas Gumbsch, and Karsten M. Borgwardt. Neural persistence: A complexity measure for deep neural networks using algebraic topology. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.DOI
  23. [23]Mathieu Carrière, Frédéric Chazal, Yuichi Ike, Théo Lacombe, Martin Royer, and Yuhei Umeda. Perslay: A neural network layer for persistence diagrams and new graph topological signatures. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pages 2786–2796. PMLR, 2020.arxiv.org/abs/1904.09378
  24. [24]Yuri Gardinazzi, Giada Panerai, Karthik Viswanathan, Alessio Ansuini, Alberto Cazzaniga, and Matteo Biagetti. Persistent topological features in large language models. CoRR, abs/2410.11042, 2024.arxiv.org/abs/2410.11042
  25. [25]Jatin Chauhan and Manohar Kaul. Bertops: Studying BERT representations under a topological lens. In International Joint Conference on Neural Networks, IJCNN 2022, Padua, Italy, July 18-23, 2022, pages 1–8. IEEE, 2022.arxiv.org/abs/2205.00953
  26. [26]Yugo Ogio, Naoki Tsubone, Yuki Minami, and Masato Ishikawa. A tda-based performance analysis for neural networks with low-bit weights. Artificial Life and Robotics, pages 1–10, 2025.DOI
  27. [27]J. Ben Tamo, Daniel Carlander-Reutefeldt, Jonathan Rubin, Oleg Poliannikov, Dezhi Hong, and Mingxian Wang. Linguamap: Which layers of llms speak your language and how to tune them? In Carl Vondrick, Bharath Hariharan, Colin Raffel, Lerrel Pinto, Diyi Yang, and Aleksandra Faust, editors, The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. proceedings.iclr.cc, 2026.
  28. [28]Georgi Gerganov. Llama.cpp. Online, 2024. Accessed: Nov. 8, 2025.
  29. [29]Ulrich Bauer. Ripser: efficient computation of Vietoris-Rips persistence barcodes. J. Appl. Comput. Topol., 5(3):391–423, 2021.DOI
  30. [30]Nathaniel Saul, Chris Tralie, and Scikit-TDA Contributors. Persim: Distances and representations of persistence diagrams. https://github.com/scikit-tda/persim, 2026. Part of the Scikit-TDA Python toolkit for Topological Data Analysis.
  31. [31]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
  32. [32]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.arxiv.org/abs/1905.07830
  33. [33]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019.arxiv.org/abs/1907.10641
  34. [34]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.arxiv.org/abs/2306.14048
  35. [35]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.arxiv.org/abs/2309.17453

Paper details

vlnc:W_sjn8es98
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents