betaFeedback

WorksComputer Science

TileQuant: Tile as the Precision-Allocation Unit for LLM Mixed-Precision Post-Training Quantization

Abstract

Post-training quantization (PTQ) reduces large language model (LLM) deployment costs, but preserving quality requires accounting for uneven weight sensitivity. Uniform PTQ uses one weight bit-width, importance-aware PTQ adjusts scales or reconstructs weights within that width, and column-wise mixed precision retains selected columns at higher precision. However, sensitivity varies within columns, so whole-column retention also protects less sensitive weights, while runtime outlier detection adds decoding overhead. We therefore introduce TileQuant, a training-free framework that assigns precision to hardware-aligned 2D weight tiles. Offline calibration provides input-activation-aware scoring from forward statistics for lower calibration cost and gradient-informed scoring from loss sensitivity for stronger quality preservation. Within each projection's retention budget, the highest-scoring tiles remain at 16-bit and the rest use 8-bit. The resulting map guides reconstruction into compact weight buffers, which the kernel reuses for native integer and 16-bit computation with local accumulation to reduce memory traffic. Experiments show that TileQuant reaches 2.66x the original model's generation throughput. Across language modeling, reasoning, long-context retrieval, and question answering, it preserves quality and surpasses quantized and original models on multiple tasks. These results support selecting precision units that capture local sensitivity while enabling efficient hardware execution. The implementation is: https://github.com/HanzhiZhang-Ulrica/TileQuant

Introduction

Efficient large language model (LLM) deployment requires reducing storage and inference costs while preserving quality in long-context understanding ([38, 2]), retrieval-augmented generation ([29, 34]), and multi-document reasoning ([6]), and

Post-training quantization (PTQ) reduces these costs by lowering numerical precision without retraining ([14] (Yao et al. 2022; Xiao et al. 2023)). However, quantization sensitivity varies across regions within each weight matrix, so a matrix-wide precision choice cannot selectively protect the most sensitive weights.

Figure 1 compares three PTQ designs. Uniform PTQ (Yao et al., 2022) assigns one weight bit-width across a matrix, while importance-aware PTQ ([36, 14] (Xiao et al. 2023)) adjusts channel scales or reconstructs weights using calibration data within that width. Protected-subset PTQ ([12, 27]) instead retains selected columns at higher precision. However, sensitivity varies even within columns, so whole-column retention also protects less sensitive weights. When selection relies on runtime activation outliers, repeated detection and processing further increase decoding latency ([12]). This motivates our research question: Which precision-allocation unit captures within-column sensitivity while supporting efficient tile computation?

Our key insight is to use hardware-aligned 2D weight tiles for local precision allocation. Each tile covers a local region along both matrix dimensions, allowing precision to vary within a column while preserving a regular tile for GPU matrix multiplication ([18, 10, 22]). However, turning these choices into efficient inference presents two coupled challenges. First, importance-based selection can retain tiles at different matrix locations, so their layout must combine compact storage with regular access to both precision formats. Second, prefill processes multiple prompt tokens, whereas decode processes one new token per sequence. The same stored layer must therefore exploit weight reuse in prefill and expose sufficient parallel work in decode while limiting intermediate memory traffic. Building on this insight, we introduce TileQuant, a training-free tile-level mixed-precision PTQ framework for LLM linear layers. Two offline calibration routes address different resource and quality needs: input-activation-aware (IAA) scoring combines forward activation statistics with weight magnitudes for lower calibration cost, while gradient-informed (GI) scoring uses loss gradients for stronger quality preservation. Each projection retains its highest-scoring tiles at 16-bit under a prescribed budget and quantizes the rest to 8-bit. The resulting decision map guides load-time reconstruction into separate compact buffers organized for dense tile loading. Prefill and decode reuse these buffers while the kernel adjusts parallel execution to each input shape. It rescales integer products and locally accumulates contributions from both precision paths, writing each completed output tile once to reduce memory traffic.

Figure 1. Comparison of PTQ precision-allocation units for LLMs. Colors denote precision: blue indicates 16-bit, and green indicates 8-bit. Uniform PTQ uses one weight bit-width. Importance-aware PTQ adjusts scales or reconstructs weights at the same bit-width, with green shades illustrating calibrated scaling. Protected-subset PTQ illustrates channel-wise 16-bit retention. TileQuant assigns precision to 2D tiles by local sensitivity.

On Llama-3.2-3B, TileQuant reaches 2.66×2.66 \times the generation throughput of the original 16-bit model while preserving quality across language modeling, reasoning, long-context retrieval, and question answering. For example, GI with 20% tile retention exceeds Original in 26 of the 27 displayed LV-Eval dataset–length settings. Together, these results show that local precision retention supports broad model capabilities alongside faster end-to-end generation.

We summarize our contributions as follows:

  1. Tile as the Precision-Allocation Unit. We assign 16-bit or 8-bit precision to hardware-aligned weight tiles, allowing selected regions within a column to retain higher precision.

  1. Calibration-Driven Tile Decision Maps. We develop IAA and GI calibration routes that prioritize tiles under per-layer retention budgets, with distinct cost and quality profiles.

  1. Executable Mixed-Precision Reconstruction. We convert tile decision maps into executable mixed-precision linear layers through compact metadata and load-time reconstruction, with dense tile computation and local accumulation reducing memory traffic.

Related Work

We review uniform PTQ, importance-aware PTQ, and protected-subset PTQ through their choices of weight bit-width, calibration criterion, and high-precision retention. They describe how weights are represented, how quantization error is controlled, and where additional precision is allocated.

Uniform PTQ applies a single weight bit-width to all weights within a chosen unit, producing a homogeneous low-precision representation for the target tensor ([15, 26] (Zhao et al. 2025; Sharify et al. 2024)). Representative examples include rounding and reconstruction methods ([8, 25, 16] (Yao et al. 2022)) and scaling- or transformation-based methods ([37, 5, 39] (Xiao et al. 2023; Pan et al. 2023; Wei et al. 2022b; Shao et al. 2023; Sun et al. 2024)) that reduce quantization error in weights and activations. A homogeneous weight format simplifies deployment, while its shared bit-width couples the precision assigned to sensitive and less sensitive regions.

Importance-aware PTQ addresses these sensitivity differences by estimating which weights or channels are sensitive to quantization error and adjusting scaling factors or reconstruction objectives accordingly [33, 3] (Zhang et al. 2025). Calibration-based methods leverage activation or sensitivity statistics to pinpoint salient weights [36, 24, 23] (Xue et al. 2025), while reconstruction-based approaches optimize the quantized tensor to better match original outputs [41, 32, 14] (Wei et al. 2022a; Wei et al. 2025; Yvinec et al. 2023). These approaches show that quantization quality depends on where error occurs, and they provide useful signals for protecting important model components. However, in fixed-bit formulations, the resulting tensor retains a single weight bit-width within the target unit.

Protected-subset PTQ further introduces non-uniform precision by preserving selected substructures in higher precision while quantizing the remaining weights [12, 4, 13, 19] (Shang et al. 2023; Yu et al. 2024). Methods protect groups [21, 31, 17, 28], channels [9] (Nrusimha et al. 2024), or columns and output features [27] (Zheng et al. 2024; Wang et al. 2024; Yuan et al. 2023). These methods demonstrate that non-uniform precision can recover accuracy under quantization [31] (Saxena et al. 2024; Xie et al. 2025; Xiao et al. 2025). Their protected units are usually tied to predefined structural axes or sparse outlier subsets, making them effective for axis-aligned or outlier-driven protection. However, whole-structure retention also protects less sensitive weights, while sparse protection adds indexing and sparse-computation costs. Mixed precision also supports fine-tuning (Zhou et al., 2024) and KV-cache compression [35].

Preliminaries

We first review quantized linear projections and then introduce their tile-wise decomposition to support local precision allocation.

PTQ for Transformer Linear Layers

During LLM inference, a linear layer computes Y=XWTY = XW^{\mathsf{T}}, where X∈RN×KX \in\mathbb{R}^{N \times K}, W∈RM×KW \in\mathbb{R}^{M \times K}, and Y∈RN×MY \in\mathbb{R}^{N \times M}. Here, NN is the number of activation rows, and KK and MM are the input and output feature dimensions. PTQ reduces storage by replacing 16-bit weights with lower-precision formats.

For a positive quantization scale Δ\Delta, the 8-bit integer weights are W^=clip⁡[−127,127](⌊W/Δ⌉)\widehat{W} = \operatorname{clip}_{[-127,127]}(\lfloor W/\Delta\rceil), with W^^≈W\widehat{\widehat{W}} \approx W. Here, ⌊⋅⌉\lfloor\cdot\rceil rounds to the nearest integer, and clipping keeps the result within the quantization range. When activations are also quantized, the projection is approximated as Y≈(ΔXΔW)(X^W^T)Y \approx(\Delta_X\Delta_W)(\widehat{X}\widehat{W}^{\mathsf{T}}) for shared operand scales ΔX\Delta_X and ΔW\Delta_W. With tile-wise scales, each tile product is rescaled before accumulation.

Tile-Wise Linear Projection

To support local precision allocation, we partition WW into a regular grid of fixed-size 2D tiles after any necessary padding. Let W(r,c)W^{(r,c)} denote a tile, where r=1,…,ntrr = 1,\ldots,n_{\mathrm{tr}} indexes output-feature tiles and c=1,…,ntcc = 1,\ldots,n_{\mathrm{tc}} indexes input-feature tiles. Their index set is denoted by T(W)\mathcal{T}(W).

Since each weight-tile column uses the same input-feature range, we partition XX into corresponding tiles X(c)X^{(c)}. The output-feature tile Y(r)Y^{(r)} is then computed as

Y(r)=∑c=1ntcX(c)(W(r,c))T,r=1,…,ntr.Y^{(r)} = \sum_{c=1}^{n_{\mathrm{tc}}} X^{(c)}\left(W^{(r,c)}\right)^{\mathsf{T}}, \qquad r = 1,\ldots,n_{\mathrm{tr}}.

Each term is a dense tile product, allowing different weight tiles to use different precisions while contributing to the same output tile.

We define F(W)\mathcal{F}(W) and Q(W)\mathcal{Q}(W) as the tile-index sets assigned 16-bit and 8-bit precision, respectively. They partition T(W)\mathcal{T}(W), satisfying F(W)∪Q(W)=T(W)\mathcal{F}(W) \cup\mathcal{Q}(W) = \mathcal{T}(W) and F(W)∩Q(W)=∅\mathcal{F}(W) \cap\mathcal{Q}(W) = \varnothing. Section § 4 determines these assignments from calibration scores.

TileQuant

TileQuant Overview

Building on the tile-wise projection in § 3, TileQuant converts attention and feed-forward linear projections into mixed-precision layers. The method must identify local regions worth retaining at 16-bit and organize their computation with the remaining 8-bit weights. Figure 2 shows how calibration choices reach the deployed layer.

Figure 2. TileQuant framework for calibration-driven mixed-precision linear layers. The input-activation-aware route (top left) combines forward activation statistics with weight magnitudes, while the gradient-informed route (bottom left) accumulates weight-gradient importance over calibration batches. Per-layer top-kk selection converts either tile-score map into a 16-bit/8-bit decision map. Retained tile IDs and grid metadata guide load-time reconstruction (right). The reconstructed layer supplies the selected weights to two arithmetic paths, whose tile products are combined through output-local 32-bit floating-point accumulation.

Offline calibration assigns an importance score to every tile using input-activation-aware (IAA) statistics or gradient-informed (GI) sensitivity. A common rule retains the highest-scoring tiles within each projection’s budget and records their locations in a binary decision map (Section 4.2). During loading, this map identifies the regions to retain and those to quantize. The resulting weight buffers and scales supply the mixed-precision kernel (§ 4.3), so the same offline choices govern arithmetic across inference inputs.

Calibration-Driven Tile Map Construction

Calibration ranks local weight regions for each projection’s 16-bit retention budget. With pretrained weights fixed, IAA scores activation-weighted contribution magnitudes, while GI scores first-order loss sensitivity on the same calibration data. These scores provide retention priorities, and the prescribed budget determines how many tiles are selected. Both routes therefore produce score maps for the common selection rule below.

Input-activation-aware route. IAA estimates tile importance from contribution magnitudes observed in forward passes. A weight’s contribution depends on both its magnitude and the activations multiplied by it. We therefore combine weight magnitudes with input-channel activation statistics.

For each input channel jj, let aj=E[∣xj∣]a_j = \mathbb{E}[|x_j|] denote the mean absolute activation magnitude gathered from the calibration data. We define the entry-level and aggregated tile-level importance as

Iijact=∣Wij∣⋅aj,Stact=∑(i,j)∈t∣Wij∣⋅ajI_{ij}^{\mathrm{act}} = |W_{ij}| \cdot a_j,\qquad S_t^{\mathrm{act}} = \sum_{(i,j)\in t} |W_{ij}| \cdot a_j

where t∈T(W)t \in\mathcal{T}(W) identifies a weight tile. The sum prioritizes regions containing large weights connected to channels with large mean absolute inputs. During calibration, IAA collects these channel means through forward passes and per-channel sums and counts, providing importance scores with a small persistent statistics buffer.

Gradient-informed route. For loss-sensitive selection, GI relates weight importance to the model’s next-token prediction objective. For calibration batch bb, a forward pass computes the next-token cross-entropy loss Lb\mathcal{L}_b, and backward computation obtains gradients with respect to the target weights. The entry-level importance is Iij,bgrad=∣Wij∂Lb∂Wij∣I_{ij,b}^{\mathrm{grad}} = \left|W_{ij}\frac{\partial\mathcal{L}_b}{\partial W_{ij}}\right|. The batch-level and accumulated tile scores are

St,bgrad=∑(i,j)∈tIij,bgrad,Stgrad=∑bSt,bgrad.S_{t,b}^{\mathrm{grad}} = \sum_{(i,j)\in t} I_{ij,b}^{\mathrm{grad}},\qquad S_t^{\mathrm{grad}} = \sum_b S_{t,b}^{\mathrm{grad}}.

The weight-gradient product measures first-order sensitivity to a relative weight change under the prediction objective. Taking magnitudes before summation prevents positive and negative terms from canceling within a tile or across calibration inputs. The accumulated score therefore ranks tiles by loss sensitivity over the calibration data. Appendix § 9.§ 2 derives this interpretation, and both routes use the precision-selection rule below.

Per-layer precision selection. The resulting scores determine retention priority, and a prescribed ratio controls the retained capacity. Here, each layer denotes one target linear projection. Given a target 16-bit retention ratio ρ∈(0,1]\rho\in(0,1], that projection retains k=max⁡(1,⌈ρ∣T(W)∣⌉)k = \max(1,\lceil\rho|\mathcal{T}(W)|\rceil) tiles. The kk highest-scoring tiles form F(W)\mathcal{F}(W), and the others form Q(W)\mathcal{Q}(W) for 8-bit quantization.

For a fixed tile grid and ρ\rho, IAA and GI retain the same number of 16-bit weights, while their scores determine the retained locations. Since scoring precedes budget selection, one completed score map can support several retention ratios. Each resulting decision map records the chosen precision assignments, whose retained indices guide load-time reconstruction.

Mixed-Precision Reconstruction and Execution

The decision map specifies which regions receive each precision. To turn this allocation into storage and execution benefits, the reconstructed layer keeps the selected weight formats in GPU memory and supplies them directly to their arithmetic paths. We first describe how the map becomes a stored layer, then explain how its tile products are computed and combined with a single writeback per output tile.

Load-time reconstruction. For each target projection, TileQuant stores retained tile IDs and tile-grid metadata. During checkpoint loading, these IDs locate the 16-bit weights, and their complement identifies the weights to quantize to 8-bit. The loader packs the two precision sets into separate contiguous buffers, grouping tiles by output-tile row. This layout places the weights contributing to the same output tile together for traversal by the kernel. Weight scales accompany the integer tiles, and non-target tensors keep their original format.

The scales recover the magnitudes represented by integer weights, and the tile IDs identify their corresponding input-feature tiles. IAA also uses its collected channel statistics to select weight scales, as detailed in Appendix § 10.§ 2. These weights, scales, and indices remain resident across inference calls, so each call begins by preparing activations.

Tile-wise arithmetic. Native 8-bit integer general matrix multiplication (INT8 GEMM) requires both operands in integer form. Therefore, the runtime quantizes each input-feature tile X(c)X^{(c)} to X^(c)\widehat{X}^{(c)} using a scale ΔX(c)\Delta_X^{(c)} shared across its activation rows. Every 8-bit weight tile in column cc reuses this prepared tile, together with its stored weights W^(r,c)\widehat{W}^{(r,c)} and scale ΔW(r,c)\Delta_W^{(r,c)}. The retained tiles multiply the current 16-bit inputs directly. Thus, the saved map determines weight precision, and input preparation supplies the operands for both paths.

For weight-tile row rr, let Qr(W)Q_r(W) and Fr(W)F_r(W) denote the corresponding column-tile indices in the two sets. Within the current activation-row tile, the mixed-precision projection approximates the original output as

Y(r)≈∑c∈Qr(W)ΔX(c)ΔW(r,c)X^(c)(W^(r,c))T+∑c∈Fr(W)X(c)(W(r,c))T.Y^{(r)} \approx\sum_{c \in Q_r(W)} \Delta_X^{(c)} \Delta_W^{(r,c)} \widehat{X}^{(c)} \left(\widehat{W}^{(r,c)}\right)^{\mathsf{T}} + \sum_{c \in F_r(W)} X^{(c)} \left(W^{(r,c)}\right)^{\mathsf{T}}.

Each integer tile product first accumulates in 32-bit integers and is then converted to floating point for rescaling. The activation and weight scales are applied before summation because they can differ across tiles. This restores all products to the output’s numerical units, allowing both precision paths to update a common accumulator.

Output-stationary execution. The tile-wise projection sums contributions from multiple weight tiles into each output tile. To reduce memory traffic during this sum, our Triton kernel accumulates successive tile products in registers. Each output tile covers a group of activation rows and the output features associated with one weight-tile row. The kernel loads the corresponding inputs and weights as dense tiles for Tensor Core multiplication (Guide, [18]; Choquette et al., [10]; Kerr et al., [22]). Products from both precision paths enter a 32-bit floating-point (FP32) accumulator, allowing each program to combine its assigned tile contributions within its registers.

Output tiles also provide independent work for parallel execution. When enough tiles are available, one program completes each tile. With fewer output tiles, the kernel distributes the weight tiles contributing to one output tile across multiple programs, exposing additional parallel work along the input dimension. Each program accumulates its assigned products locally, and their partial sums are then combined within the same kernel. Both schedules write each completed output tile once, preserving local accumulation while adapting the available parallel work to the output shape. Appendix E.3 provides the indexing, scheduling, and partial-sum reduction details.

Prefill and decode. The same stored layer must support the different input shapes encountered during generation. Prefill processes multiple prompt-token rows through GEMM, allowing a loaded weight tile to contribute to several rows within an output tile. During decode, each sequence contributes one new activation row, giving general matrix-vector multiplication (GEMV) for a single sequence or a small GEMM for a batch.

Accordingly, row tiling and launch parameters adapt to the number of activation rows, while both phases reuse the precision map, packed weights, scales, and tile indices. The mapping from weight-tile rows to output features remains unchanged, and the runtime selects the parallel schedule for the current input shape. This organization keeps importance estimation and precision selection outside token generation while reusing the same mixed-precision representation throughout the request.

Experiments

Experimental Setup

We evaluate efficiency and model quality on Llama-3.2-3B-Instruct (AI, [1]). Appendix § 12.§ 2 extends the system measurements, while Appendices § 12.§ 3 and F.4 report additional quality results.

Datasets and Metrics. To assess general capabilities, we report WikiText-2 word perplexity (Merity et al., [40]) and accuracy on HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., [20]), ARC-Challenge (Clark et al., [11]), LAMBADA (Paperno et al., 2016), PIQA (Bisk et al., [7]), and WinoGrande (Sakaguchi et al., 2021). For long contexts, LongEval (Li et al., [30]) measures line-retrieval exact-match accuracy over 2.7K–43.4K tokens, while LV-Eval (Yuan et al., 2024) evaluates question answering across 9 datasets. We report LV-Eval at 16K, 32K, and 64K; Appendix § 12.§ 4 includes 128K and 256K.

Configuration and Hardware. Experiments use NVIDIA A100 GPUs with 40 GB memory. TileQuant employs 64 × 64 weight tiles, retaining selected tiles at 16-bit and quantizing the rest to INT8. The main tables report 1% and 20% retention, while LongEval and Appendix § 12 additionally cover 5% and 10%.

Quality Comparisons. We compare against the original BF16 model and 5 quantized baselines: Uniform W8, Uniform W8A8, GPTQ (Frantar et al., [14]), LLM.int8 (Dettmers et al., [12]), and SmoothQuant (Xiao et al., 2023). Uniform W8 uses channel-wise 8-bit weights with 16-bit activations, whereas W8A8 quantizes activations per token. GPTQ uses 8-bit weights and 16-bit activations, while SmoothQuant uses simulated W8A8 quantization. To ensure consistent quality comparisons, methods share the checkpoint, prompts, decoding settings, and evaluation protocol within each benchmark. Configurations and measurement details are provided in Appendix § 12.§ 1.

Inference Efficiency

We measure complete-request generation throughput to evaluate the practical speed benefit of mixed-precision execution. BF16 denotes the original bfloat16 model with scaled dot-product attention (SDPA), while Torch uses Hugging Face half-precision (FP16) generation with eager attention. Both GI1_1 and GI20_{20} exceed the displayed baselines across all 64 settings on Llama-3.2-3B in Figure 3. With a 128-token prompt and 512 generated tokens at batch size 1, GI1_1 reaches 154.98 tokens/s, 2.66× BF16 and 5.87× GPTQ throughput. GI20_{20} retains 96.7–98.6% of GI1_1 throughput across the grid with 20% of target tiles in BF16. Separate geometry controls report lower prefill and decode latency for tiles than for full-column and compact-strip implementations at matched retention (Appendix § 13.§ 2).

Figure 3. Generation-throughput speedup relative to BF16 on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8, and columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. Throughput includes the complete request and is normalized to BF16 at the same request shape. Values use the median of five measured runs after two warm-up runs. OOM denotes an out-of-memory run.

The relative benefit varies with the balance between prompt processing and token generation. For long prompts, longer continuations strengthen the throughput advantage: with 16K input tokens at batch size 8, GI20_{20} improves from 1.07× to 1.91× BF16 throughput as output length increases from 128 to 1024 tokens. Longer continuations spread prompt-processing cost over more generated tokens, giving repeated decoding a larger role in the complete request. During decoding, TileQuant reuses its calibrated precision map and stored tiles in mixed-precision arithmetic.

Findings callout

Figure 18.

FINDINGS Tile execution speeds up tested generation workloads, preserving nearly all GI throughput from 1% to 20% retention.

General Tasks

Having established generation speedups, we evaluate quality under the same tile-retention settings. GI20_{20} leads five of the seven benchmarks in Table 1. Its LAMBADA accuracy is 65.78%, compared with 64.43% for GPTQ and 64.23% for Original, while WikiText-2 perplexity is 12.403 versus 12.487 for Original. The same loss-guided maps therefore support language modeling and several downstream tasks beyond the calibration inputs.

BenchmarkMetricOriginalOursUniformGPTQSmoothQLLM.int8
GI1_1IAA1_1GI20_{20}IAA20_{20}W8W8A8
WikiText-2 ↓\downarrowPPL12.48712.51812.53712.40312.43812.45312.63012.51512.60912.639
WinoGrande ↑\uparrowAcc.71.4371.3870.9871.4671.3070.8870.6471.4371.5971.11
PIQA ↑\uparrowAcc.77.3176.6177.0977.4277.1576.6176.5576.8276.5576.82
LAMBADA ↑\uparrowAcc.64.2365.6265.0765.7865.4964.4163.9664.4363.9864.31
ARC-Challenge ↑\uparrowAcc.52.5652.8252.3053.6952.8852.3051.7952.7351.7151.88
HellaSwag ↑\uparrowAcc.73.6573.2573.0972.9572.5073.5973.5373.5573.4773.48
MMLU ↑\uparrowAcc.60.4760.6460.7861.1060.5460.5160.8060.4660.4860.89

Table 1. General language-modeling and reasoning results on Llama-3.2-3B. GI and IAA denote gradient-informed and input-activation-aware tile retention, with subscripts indicating the percentage of tiles retained in floating point. ↑/↓\uparrow/\downarrow indicate higher/lower is better. Accuracy is reported in percent, while WikiText-2 uses word perplexity. Bold marks the best result per row. Relative to Original (gray), green indicates equal or better performance and blue indicates worse performance.

With only 1% tile retention, GI1_1 and IAA1_1 achieve perplexities of 12.518 and 12.537, respectively, within 0.05 of Original while 99% of target tiles use 8-bit weights. Appendix § 12.§ 3 reports complete retention sweeps, and Appendix § 13.§ 1 compares IAA, magnitude-based, and random retention at equal high-precision capacity.

Findings callout

Figure 19.

FINDINGS 1% tile retention preserves near-original language-modeling quality; GI improves accuracy on several downstream tasks.

Long Context Tasks

Beyond general benchmarks, we examine whether the same offline-selected tiles preserve evidence retrieval and question answering as inputs grow.

Long-context Retrieval. LongEval requires locating a queried line and returning its content. Figure 4 shows that GI generally follows Original closely across the length sweep, with GI20_{20} improving accuracy at multiple longer-input settings. At 43.4K tokens, GI20_{20} reaches 51% accuracy, compared with 47% for Original and 44% for GPTQ. At matched retention, the GI and IAA comparison further shows how the choice of calibration route affects retrieval quality.

Line-level retrieval plots for TileQuant and baseline configurations

Figure 4. Line-level retrieval on Llama-3.2-3B across 2.7K–43.4K prompt tokens. Lines (left axis) show exact-match accuracy, while stacked bar segments (right axis) show each configuration’s accuracy difference from Original. Top: TileQuant with GI maps (left) and IAA maps (right), both retaining 1%, 5%, 10%, or 20% of tiles in floating point (keep01, keep05, keep10, keep20). Bottom: uniform INT8 weight-only, uniform W8A8, GPTQ, SmoothQuant, and LLM.int8 (left); representative comparison of the strongest TileQuant and baseline configurations (right).

FINDINGS A single calibrated map supports retrieval across lengths. GI20_{20} exceeds Original and GPTQ at the longest tested context.

Long-context Question Answering. Beyond line retrieval, LV-Eval tests question answering from long input contexts across nine datasets, identified in Table 2. We use keyword-recall-based F1, with blacklist-based F1 for CMRC and blacklist-based ROUGE-L for DuReader.

DatasetLenOriginalOurs GI1_1Ours IAA1_1Ours GI20_{20}Ours IAA20_{20}Uniform W8Uniform W8A8GPTQSmoothQLLM.int8
①16k21.4123.8224.2124.0623.8721.1521.3921.5320.1520
32k16.8618.1618.2318.3818.2317.9517.0517.3916.0616.31
64k7.58.4888.598.047.466.937.587.246.47
②16k16.4616.4517.3716.6416.516.5417.3316.0717.4717.88
32k17.116.4816.8816.3917.0316.616.8815.8316.6416.37
64k15.5215.414.7215.7615.4714.5514.9715.0414.7714.84
③16k24.4925.2924.2225.5624.3724.3320.7124.7423.1523.52
32k19.2318.4317.2819.4819.1517.8219.1218.3520.7417.7
64k15.8613.6117.0216.115.9615.8914.2415.0815.7315.82
④16k11.2314.0814.0514.2114.0911.1711.9910.7611.7412.39
32k11.3912.6810.1712.8311.3410.8411.610.7211.1712.17
64k3.455.225.35.275.244.294.434.043.934.36
⑤16k22.3924.1224.0424.3724.1122.9824.0122.3723.0122.68
32k23.0224.3224.6324.6124.4123.6123.623.1223.5122.39
64k15.4915.615.6715.8415.716.6816.4915.5417.0815.24
⑥16k17.8819.7719.6519.9719.7118.6119.2118.6218.8518.26
32k15.7716.1315.7716.3315.8315.9116.1315.316.3916.01
64k11.7712.4812.0112.6612.0712.4511.5211.6312.3911.97
⑦16k52.6151.8653.7352.8952.6752.6451.6452.8453.1952.69
32k37.7537.7237.9738.0737.8536.5236.6436.9835.5738.03
64k24.3126.2223.726.5924.2424.6424.1424.6926.2425.66
⑧16k33.8134.2836.0134.5634.3433.9834.132.6935.7534.64
32k25.9527.5125.0827.8325.8525.1426.0325.8426.525.74
64k21.2420.2221.2721.5621.3821.3920.0121.5820.8921.42
⑨16k19.0523.3817.9923.5918.9620.9119.4118.8420.222.97
32k13.4815.9515.3716.1215.4214.7715.6214.1918.0214.35
64k13.7113.8511.6114.0613.6713.5113.7213.9514.4912.57

Table 2. Long-context question answering results for Llama-3.2-3B on LV-Eval at the 16K, 32K, and 64K length levels. Dataset IDs 1–9 denote CMRC, DuReader, HotpotWikiQA, LIC, LooGLE-CR, LooGLE-MIR, LooGLE-SD, English and Chinese MultiFieldQA, respectively. Cell colors are relative to Original, with green for higher scores and blue for lower scores.

GI20_{20} exceeds Original in 26 of the 27 displayed dataset–length settings. On LooGLE-SD at 64K, it scores 26.59, compared with 24.69 for GPTQ and 24.31 for Original. The improvements across datasets and lengths show that a map calibrated once remains useful across long-context tasks.

The forward-only route preserves strong performance with limited retention. On HotpotWikiQA at 64K, IAA1_1 scores 17.02, compared with 15.86 for Original and 15.89 for Uniform W8. Appendix F.4 reports more models and length levels.

FINDINGS Both GI and IAA maintain question-answering quality across datasets and input lengths with maps calibrated once.

Conclusion

TileQuant uses 2D tiles to connect local precision allocation with executable mixed-precision inference. Calibration scores determine which weight regions retain 16-bit precision, and the resulting maps guide compact storage and dense tile computation. Experiments show faster generation alongside strong language-modeling, reasoning, retrieval, and question-answering quality. These results suggest aligning precision-allocation granularity with hardware-aligned weight tiles for GPU computation. This alignment allows high precision to follow local sensitivity while preserving regular memory access and dense computation.

Limitations

TileQuant studies training-free mixed-precision inference with 8-bit weights and retained 16-bit tiles. Precision maps are calibrated offline and reused across inference inputs. Input- or task-adaptive allocation would require an additional runtime decision mechanism. Extending the framework to training requires corresponding treatment of gradients, optimizer states, and forward-backward numerical consistency.

CategorySymbolDefinition
ProjectionX∈RN×KX \in\mathbb{R}^{N \times K}, W∈RM×KW \in\mathbb{R}^{M \times K}Input activations and projection weights
Y=XW⊤Y = XW^{\top}Projection output
NN, KK, MMActivation rows, input features, output features
xx, ii, jj, nnActivation row; output-feature, input-feature, row indices
Tilingrr, cc; ntrn_{\mathrm{tr}}, ntcn_{\mathrm{tc}}One-based tile coordinates; tile-row and tile-column counts
bMb_M, bKb_K; M′M^{\prime}, K′K^{\prime}Weight-tile dimensions; padded weight dimensions
T(W)\mathcal{T}(W), tt; id⁡(r,c)\operatorname{id}(r,c)Tile-index set, tile index t=(r,c)t=(r,c); zero-based stored ID
W(r,c)=WtW^{(r,c)} = W_tWeight tile at row rr and column cc
X(c)X^{(c)}, Y(r)Y^{(r)}Input- and output-feature tiles over the activation rows
Calibrationθ\theta, PP, DDModel parameters, target projections, calibration batches
bb; XbX_b, NbN_bCalibration batch index; its activations and row count
zb,τz_{b,\tau}, TbT_b, pθp_{\theta}Token at position τ\tau, token count, predictive distribution
LbL_b, gij,bg_{ij,b}Batch loss and its gradient ∂Lb/∂Wij\partial L_b/\partial W_{ij}
aja_jMean absolute activation of input feature jj
IijactI^{\mathrm{act}}_{ij}, Iij,bgradI^{\mathrm{grad}}_{ij,b}IAA entry score; GI entry score for batch bb
St,bgradS^{\mathrm{grad}}_{t,b}GI score of tile tt in batch bb
StactS^{\mathrm{act}}_t, StgradS^{\mathrm{grad}}_tCompleted IAA and GI tile scores
ϵ\epsilon, EijE_{ij}Relative weight change; unit-entry matrix at (i,j)(i,j)
AllocationT=∣T(W)∣T = |\mathcal{T}(W)|, StS_tTile count; completed score of either calibration route
ρ\rho, kkTarget 16-bit retention ratio; selected tile count
F(W)\mathcal{F}(W), Q(W)\mathcal{Q}(W)16-bit and 8-bit tile-index sets
Fr(W)\mathcal{F}_r(W), Qr(W)\mathcal{Q}_r(W)Retained and quantized column-tile indices in row rr
QuantizationQ(V;Δ)Q(V;\Delta)Signed 8-bit quantizer for array VV with positive scale Δ\Delta
X^\widehat{X}, W^\widehat{W}; X^(c)\widehat{X}^{(c)}, W^(r,c)\widehat{W}^{(r,c)}Integer operands and their input-feature or weight tiles
ΔX\Delta_X, ΔW\Delta_W; ΔX(c)\Delta_X^{(c)}, ΔW(r,c)\Delta_W^{(r,c)}Shared operand scales; per-input-feature and per-weight scales
mcm_cAbsolute activation maximum over input-feature group cc
Δ0,t\Delta_{0,t}, Δα,t\Delta_{\alpha,t}, α\alphaAbsmax weight scale, candidate scale, candidate multiplier
Et(Δ)E_t(\Delta)Activation-weighted weight-tile reconstruction error
StorageT8=∣Q(W)∣T_8 = |\mathcal{Q}(W)|, T16=∣F(W)∣T_{16} = |\mathcal{F}(W)|Quantized and retained weight-tile counts
ρreal\rho_{\mathrm{real}}, SpackS_{\mathrm{pack}}Realized fraction T16/TT_{16}/T; packed storage in bytes
ExecutionB\mathcal{B}, bNb_N, NcallN_{\mathrm{call}}Output-tile activation rows, row-tile size, GEMM-call rows
XB(c)X^{(c)}_{\mathcal{B}}, X^B(c)\widehat{X}^{(c)}_{\mathcal{B}}16-bit and integer input tiles restricted to rows B\mathcal{B}
DB,r,cD_{\mathcal{B},r,c}, AAINT32 tile product; FP32 register accumulator
pp, Rp(r)R_p(r), hp,rh_{p,r}Precision (p∈{8,16}p \in\{8,16\}), row pointer, row tile count
uu, SS, ssPacked-buffer position, split count, zero-based split index
δr\delta_rOutput-tile rows preceding a grouped projection
EvaluationBB, PP, OO, TtotalT_{\mathrm{total}}Request batch size, prompt tokens, output tokens, request time
R1\mathcal{R}_1, R2\mathcal{R}_2; J(R1,R2)\mathcal{J}(\mathcal{R}_1,\mathcal{R}_2)Retained sets from two draws; their Jaccard similarity

Table 3. Notation used in the main text and Appendix of TileQuant. Each calibration batch contains one record.

Notation and Reading Guide

Table 3 summarizes the notation used for tile-wise precision allocation, calibration, and mixed-precision execution. Appendix B details calibration, and Appendix C explains the importance scores and per-layer selection. Appendix D describes numerical representation and load-time reconstruction, followed by GPU execution in Appendix E. Appendix F follows the main-text evaluation sequence, beginning with the detailed generation protocol and inference-efficiency measurements in Appendix F.1 and Appendix F.2. Extended general-task results appear in Appendix F.3, followed by long-context retrieval and question answering in Appendix § 12.§ 4. Appendix § 13 examines calibration and allocation choices through controls.

Within a weight-tile row, rr identifies the output-feature tile, while cc traverses the corresponding input-feature tiles. The retained and quantized tile sets therefore determine which 16-bit and 8-bit products contribute to the same output tile.

Calibration Procedure

Calibration converts model observations into one importance map per target weight matrix. This section describes the observations collected by input-activation-aware (IAA) and gradient-informed (GI) scoring. The resulting maps enter the common selection rule in Appendix § 9.§ 3, while IAA channel statistics also guide the weight-scale construction in Appendix § 10.§ 2.

Calibration Inputs and Target Projections

The calibration builder uses 8,192 document-summary records from the Multi-News training split in dataset order. Each document is placed in a summarization prompt, followed by its reference summary. The prompt and summary are tokenized separately with limits of 4,096 and 256 tokens, respectively. The tokenizer’s beginning-of-sequence token is prepended when available, and its end-of-sequence token is appended to the summary. Each calibration batch bb contains one concatenated sequence. Both routes therefore process the same records in dataset order.

The target set P\mathcal{P} contains the query, key, value, and attention-output projections, together with the gate, up, and down projections in each decoder layer. Calibration runs the pretrained model in evaluation mode with key/value caching disabled. Its parameters remain fixed throughout observation collection. IAA collects the inputs to these projections, while GI differentiates the mean next-token cross-entropy over the concatenated sequence. Thus, the two routes share the model and records while extracting different information for local precision placement.

Input-Activation Statistics

IAA needs the mean absolute input magnitude of each channel. For a target projection, let Xb∈RNb×KX_b \in\mathbb{R}^{N_b \times K} contain the activation rows observed in calibration batch bb, and let D\mathcal{D} index these batches. Input hooks accumulate channel sums and the number of observed rows, giving

aj=∑b∈D∑n=1Nb∣Xb,nj∣∑b∈DNb.(1)a_j = \frac{\sum_{b \in\mathcal{D}} \sum_{n=1}^{N_b} \lvert X_{b,nj} \rvert}{\sum_{b \in\mathcal{D}} N_b}. \tag*{(1)}

Weighting by the observed row count makes every activation row contribute equally, including when records have different lengths.

Per-record channel reductions use 32-bit floating point (FP32), and their running totals use 64-bit floating point (FP64). The collected means are stored in FP32 and combined with fixed weights:

Stact=∑(i,j)∈t∣Wij∣aj.(2)S_t^{\mathrm{act}} = \sum_{(i,j)\in t} \lvert W_{ij} \rvert a_j. \tag*{(2)}

Each input hook immediately reduces its observations to channel statistics, so the persistent calibration state consists of sums and counts. The final tile scores determine retention priority, and the channel means accompany the IAA decision artifacts for load-time scale selection.

Gradient-Informed Score Accumulation

GI identifies weights whose relative changes influence prediction loss. For calibration batch bb with sequence (zb,1,…,zb,Tb)(z_{b,1}, \ldots, z_{b,T_b}), the loss is

Lb=−1Tb−1∑τ=1Tb−1log⁡pθ(zb,τ+1∣zb,1:τ).(3)\mathcal{L}_b = -\frac{1}{T_b - 1}\sum_{\tau=1}^{T_b-1} \log p_\theta(z_{b,\tau+1} \mid z_{b,1:\tau}). \tag*{(3)}

The shifted prediction targets cover the concatenated prompt and summary. Backpropagation gives gij,b=∂Lb/∂Wijg_{ij,b} = \partial\mathcal{L}_b/\partial W_{ij}, so the entry score is Iij,bgrad=∣Wijgij,b∣I_{ij,b}^{\mathrm{grad}} = \lvert W_{ij}g_{ij,b}\rvert.

The collector forms the absolute weight-gradient products in FP32, reduces them within each tile, and adds the resulting tile scores across records:

St,bgrad=∑(i,j)∈tIij,bgrad,S_{t,b}^{\mathrm{grad}} = \sum_{(i,j)\in t} I_{ij,b}^{\mathrm{grad}},
Stgrad=∑b∈DSt,bgrad.(4)S_t^{\mathrm{grad}} = \sum_{b\in\mathcal{D}} S_{t,b}^{\mathrm{grad}}. \tag*{(4)}

Taking magnitudes before aggregation prevents cancellation between positive and negative weight-gradient terms within a tile or across records. Gradients are cleared before the next record, while accumulated FP32 tile maps remain on the host. The completed maps enter the common retention rule, as summarized in Algorithm 1.

Algorithm 1 Collecting calibration statistics and tile scores

Require: Fixed pretrained model θ\theta, target projections P\mathcal{P}, ordered one-record batches D\mathcal{D}, route ∈{IAA,GI}\in\{\mathrm{IAA},\mathrm{GI}\}

Ensure: One tile-score map per target projection; channel means for IAA

  1. if route is IAA then

  2. Initialize a channel-sum vector and row count for each projection in P\mathcal{P}

  3. for all calibration batches b∈Db \in\mathcal{D} do

  4. Run the model with gradient recording disabled

  5. Input hooks add channel-wise absolute activation sums and row counts

  6. end for

  7. for all projections in P\mathcal{P} do

  8. Compute aja_j using Eq. (1)

  9. Compute StactS_t^{\mathrm{act}} using Eq. (2); save channel means

  10. end for

  11. else

  12. Initialize each target projection’s tile-score map to zero

  13. for all calibration batches b∈Db \in\mathcal{D} do

  14. Clear parameter gradients and evaluate Lb\mathcal{L}_b using Eq. (3)

  15. Backpropagate Lb\mathcal{L}_b through the fixed model

  16. for all target weights WW and tiles t∈T(W)t \in\mathcal{T}(W) do

  17. Stgrad←Stgrad+∑(i,j)∈t∣Wijgij,b∣S_t^{\mathrm{grad}} \leftarrow S_t^{\mathrm{grad}} + \sum_{(i,j)\in t}|W_{ij}g_{ij,b}|

  18. end for

  19. end for

  20. Clear parameter gradients

  21. end if

  22. Save the tile-score maps for the retention rule in Appendix § 9.§ 3

Score Interpretation and Precision Selection

The collected statistics provide two complementary views of local weight importance. We first relate IAA to activation-weighted contributions and GI to first-order loss sensitivity. We then show how a fixed per-layer retention count turns either score into a precision map.

Activation-Weighted Magnitudes

For a fixed weight WijW_{ij} and an observed activation row xx, the corresponding multiply contribution is WijxjW_{ij}x_j. Taking its magnitude and averaging over calibration rows gives ∣Wij∣E[∣xj∣]|W_{ij}|\mathbb{E}[|x_j|]. Therefore, the total expected magnitude of the multiply contributions inside tile tt is

E[∑(i,j)∈t∣Wijxj∣]=∑(i,j)∈t∣Wij∣E[∣xj∣]=Stact.(5)\mathbb{E}\left[\sum_{(i,j)\in t}|W_{ij}x_j|\right] = \sum_{(i,j)\in t}|W_{ij}|\mathbb{E}[|x_j|] = S_t^{\mathrm{act}}. \tag*{(5)}

The identity gives the IAA score a direct interpretation as the sum of expected absolute multiplication contributions within a tile. At a fixed weight magnitude, a larger mean absolute input increases the score. Summing these contributions lets IAA rank local regions using the channel statistics collected during forward computation.

First-Order Weight Sensitivity

GI connects local weight importance to the full model’s prediction loss. Consider a relative change Wij↦(1+ϵ)WijW_{ij}\mapsto(1+\epsilon)W_{ij}, with all other parameters fixed. Let EijE_{ij} be a matrix with one at (i,j)(i,j) and zeros elsewhere. At a differentiable calibration state,

ddϵLb(W+ϵWijEij)∣ϵ=0=Wij∂Lb∂Wij.(6)\left.\frac{\mathrm{d}}{\mathrm{d}\epsilon}\mathcal{L}_b(W+\epsilon W_{ij}E_{ij})\right|_{\epsilon=0} = W_{ij}\frac{\partial\mathcal{L}_b}{\partial W_{ij}}. \tag*{(6)}

Thus, ∣Wijgij,b∣|W_{ij}g_{ij,b}| measures the magnitude of the first-order loss response to a relative change of that weight. Summing these magnitudes within a tile and across calibration records yields Eq. (4).

The gradient carries the effect of the target projection through downstream model computation. Consequently, GI prioritizes regions that are important under the prediction objective, while IAA prioritizes regions with large activation-weighted contributions. Both interpretations lead to the same capacity-constrained selection below.

Per-Layer Retention and Decision Maps

Here, a layer denotes one target linear projection with weight matrix WW. Let T=∣T(W)∣T=|\mathcal{T}(W)| and let StS_t be the completed IAA or GI score for tile tt. For a retention ratio ρ∈(0,1]\rho\in(0,1], the selected count is

k=max⁡(1,⌈ρT⌉).(7)k=\max(1,\lceil\rho T\rceil). \tag*{(7)}

The count is applied independently to each target projection. Equal-sized tiles make kk a direct control over the number of stored 16-bit weight elements on that tile grid. The zero-retention execution control sets F(W)=∅\mathcal{F}(W)=\varnothing and assigns every target tile to Q(W)\mathcal{Q}(W).

At fixed scores and kk, retaining the highest-scoring tiles solves

max⁡F(W)⊆T(W), ∣F(W)∣=k∑t∈F(W)St.(8)\max_{\mathcal{F}(W)\subseteq\mathcal{T}(W),\ |\mathcal{F}(W)|=k} \sum_{t\in\mathcal{F}(W)} S_t. \tag*{(8)}

To see this, consider a feasible set that retains a lower-scoring tile while excluding a higher-scoring tile. Exchanging the two increases its retained score. Repeating these exchanges produces a top-kk set, which attains the maximum. Equal scores permit equally valued selections under this objective.

The implementation identifies the top-kk positions through an array-partition operation and saves the concrete selected tile IDs. Their complement defines Q(W)\mathcal{Q}(W), and the pair (F(W),Q(W))(\mathcal{F}(W), \mathcal{Q}(W)) determines the layer’s binary precision map. Multiple retention ratios can therefore be constructed from the same completed score map. The saved IDs preserve each chosen allocation through loading and inference, using the representation described next.

Numerical Representation and Load-Time Reconstruction

The precision map identifies locations, and reconstruction associates those locations with values, scales, and addresses that the kernel can use. We first define tile coordinates and integer quantization, then describe how the decision archive becomes packed weight buffers. Execution of the mixed-precision modules follows Appendix § 11.

Tile Coordinates and Padding

We retain the convention X∈RN×KX \in\mathbb{R}^{N\times K} and W∈RM×KW \in\mathbb{R}^{M\times K}, with NN activation rows and MM output features. Let bMb_M and bKb_K be the weight-tile dimensions. The deployed configuration uses bM=bK=64b_M=b_K=64. Zero padding completes the weight grid to

M′=bM⌈M/bM⌉,K′=bK⌈K/bK⌉,M' = b_M\left\lceil M/b_M\right\rceil,\qquad K' = b_K\left\lceil K/b_K\right\rceil,

with ntr=M′/bMn_{\mathrm{tr}}=M'/b_M and ntc=K′/bKn_{\mathrm{tc}}=K'/b_K. {#eq:9}

With one-based tile coordinates t=(r,c)t=(r,c), the zero-based stored identifier is

id⁡(r,c)=(r−1)ntc+(c−1).(9)\operatorname{id}(r,c)=(r-1)n_{\mathrm{tc}}+(c-1). \tag*{(9)}

Consequently, integer division and remainder recover the output-tile row and input-tile column. Boundary tiles contain zeros outside the original matrix. The kernel masks activation accesses beyond KK and output writes beyond MM, preserving the original linear-layer interface while processing complete weight tiles.

Integer Quantization and Weight Scales

Each quantized tile needs integer values and a scale that restores their numerical magnitude. For an array VV and positive scale Δ\Delta, define

Q(V;Δ)=clip⁡[−127,127](round⁡even(V/Δ)),(10)Q(V;\Delta)=\operatorname{clip}_{[-127,127]}\left(\operatorname{round}_{\mathrm{even}}(V/\Delta)\right), \tag*{(10)}

where nearest-integer rounding resolves ties to even. The zero point is zero; reconstruction gives ΔQ(V;Δ)\Delta Q(V;\Delta). For a weight tile Wt=W(r,c)W_t=W^{(r,c)}, the base scale is

Δ0,t={max⁡(i,j)∈t∣Wij∣/127,Wt≠0,1,otherwise.(11)\Delta_{0,t} = \begin{cases} \max_{(i,j)\in t}|W_{ij}|/127, & W_t\ne0,\\ 1, & \text{otherwise}. \end{cases} \tag*{(11)}

The GI path uses this absmax scale. The maximum is obtained from the original 16-bit values, and the scale is computed and stored in FP32.

IAA additionally reuses its channel means to select scales for the tiles assigned to 8-bit. For each such tile, it tests Δα,t=max⁡{αΔ0,t,10−12}\Delta_{\alpha,t}=\max\{\alpha\Delta_{0,t},10^{-12}\} for α∈{0.75,1.00,1.25}\alpha\in\{0.75,1.00,1.25\} and minimizes the activation-weighted reconstruction error

Et(Δ)=1bMbK∑(i,j)∈tmax⁡{aj,10−8}(ΔQ(Wt;Δ)ij−Wij)2(12)E_t(\Delta)=\frac{1}{b_Mb_K}\sum_{(i,j)\in t}\max\{a_j,10^{-8}\}\left(\Delta Q(W_t;\Delta)_{ij}-W_{ij}\right)^2 \tag*{(12)}

Reducing the scale gives finer quantization steps and a narrower representable range. The weighted error therefore accounts for rounding and clipping while emphasizing channels with larger observed inputs. Candidates are evaluated in FP32 in the listed order, with the first retained in a tie.

The chosen scale Δ^W(r,c)\widehat{\Delta}^{(r,c)}_W accompanies the signed INT8 tile W^(r,c)\widehat{W}^{(r,c)} throughout execution. Retained tiles preserve their deployment values in bfloat16 (BF16) or half precision (FP16). Thus, calibration determines the protected locations, while scale construction supplies the numerical representation of the remaining weights.

Decision Archives and Packed Tile Layout

The decision archive records each target projection’s name, tile-grid shape, and retained IDs. It stores IDs as unsigned 16-bit integers for grids containing at most 216−12^{16}-1 tiles and as unsigned 32-bit integers for larger grids. Loading recovers F(W)\mathcal{F}(W) from these IDs and constructs Q(W)\mathcal{Q}(W) from the complementary grid positions.

For GPU execution, the two precision sets occupy separate contiguous tile buffers. Each buffer is sorted by the identifier in Eq. (9), so tiles belonging to the same output-tile row are contiguous and ordered by input-tile column. Let Rp(r)R_p(r) count the precision-pp tiles in rows preceding rr, for p∈{8,16}p \in\{8, 16\}. Row rr occupies positions

[Rp(r),Rp(r+1)).(13)[R_p(r), R_p(r+1)). \tag*{(13)}

At buffer position uu, the kernel reads the weight tile and, for the integer path, its scale. The tile ID supplies the corresponding input-feature tile. The kernel can therefore traverse weights contiguously while using their IDs to select activations.

The packed layer stores T8=∣Q(W)∣T_8 = |\mathcal{Q}(W)| integer tiles of shape [T8,bM,bK][T_8, b_M, b_K] and T16=∣F(W)∣T_{16} = |\mathcal{F}(W)| retained tiles of shape [T16,bM,bK][T_{16}, b_M, b_K]. Integer tiles use INT8; retained tiles and the optional bias of shape [M][M] use BF16 or FP16. An FP32 scale array of shape [T8][T_8] restores weight magnitudes.

Each precision set has INT32 tile-ID and row-order arrays of length T8T_8 or T16T_{16}. IDs identify the corresponding input-tile columns, while row-order indices record the packing permutations. Two INT32 row-pointer arrays of shape [ntr+1][n_{\mathrm{tr}}+1] delimit each output-tile row’s contiguous range.

The packed representation separates numerical payload from indexing. With realized retained fraction ρreal=T16/(T8+T16)\rho_{\mathrm{real}} = T_{16}/(T_8 + T_{16}), the weight values require 8+8ρreal8 + 8\rho_{\mathrm{real}} bits per padded weight element. Including these scales and indexing arrays, the packed tensor storage before bias is

Spack=bMbK(T8+2T16)+4T8+8(T8+T16)+8(ntr+1) bytes.(14)S_{\mathrm{pack}} = b_M b_K(T_8 + 2T_{16}) + 4T_8 + 8(T_8 + T_{16}) + 8(n_{\mathrm{tr}} + 1)\ \text{bytes}. \tag*{(14)}

The terms account for weight values, integer-tile scales, tile IDs plus row-order indices, and the two row-pointer arrays. Activation buffers, outputs, and execution workspace have input-dependent lifetimes and are described separately in Appendix § 11.

Loading the Mixed-Precision Model

The loader constructs the deployed representation directly from the pretrained checkpoint and decision archive. It initializes the model structure, checks each target projection’s saved grid against its weight shape, and reads checkpoint tensors on the host. As each target weight becomes available, the loader gathers the retained tiles, quantizes the complementary tiles using Appendix § 10.§ 2, builds their row schedules, and installs the resulting mixed-precision module on the GPU.

The original biases and non-target tensors retain their roles in the model. Compatible projection modules are subsequently grouped using the procedure in Appendix § 11.§ 4. After loading, the selected weights, scales, and indices remain resident across requests. Runtime work therefore begins with preparing the current activations and consuming the already constructed tile representation.

Mixed-Precision GPU Execution

The packed layer supplies persistent weights and their numerical interpretation. Execution must prepare the current input, evaluate both precision paths, and combine their contributions into the model’s output format. We describe this data flow first, then explain how output scheduling, shared projection inputs, and repeated-generation state reuse the same selected model.

Activation Preparation

The integer path requires 8-bit activations, while retained tiles use the current 16-bit input. The runtime therefore preserves XX and prepares an integer copy grouped by the weight tiles’ input-feature ranges. For column group cc, let

mc=max⁡1≤n≤N,  j∈c∣Xnj∣.(15)m_c = \max_{1 \le n \le N,\; j \in c} |X_{nj}|. \tag*{(15)}

Here, j∈cj \in c denotes the original input features covered by that tile column. The maximum covers all activation rows of the current projection input, so one scale is shared across its output-tile rows.

For a nonzero group, the fused GPU quantizer computes mc/127m_c/127 in FP32, rounds the result to the input’s 16-bit datatype, and stores that value in FP32 as ΔX(c)\Delta_X^{(c)}. An all-zero group uses scale one. Division and nearest-integer rounding then produce

X^(c)=Q(X(c);ΔX(c)),(16)\widehat{X}^{(c)} = Q\left(X^{(c)}; \Delta_X^{(c)}\right), \tag*{(16)}

using the integer range in Eq. (10). The prepared integer values and scales supply every quantized weight tile consuming those input features.

For inputs with at most 128 rows, maximum reduction and quantization are fused. Larger inputs first compute the shared scales and then quantize row tiles. Above 4,096 rows, the runtime applies quantization and mixed GEMM in chunks of at most 4,096 rows using scales computed over the complete input. Each chunk writes directly into its assigned rows of one output tensor. Consequently, row chunking bounds the temporary integer-activation allocation while retaining the same scale definition.

Tile Products and Scaled Accumulation

Let B\mathcal{B} denote the activation rows assigned to an output tile, and let rr identify its output-feature tile. The integer path reads X^B(c)\widehat{X}_{\mathcal{B}}^{(c)} and the stored W^(r,c)\widehat{W}^{(r,c)} as signed INT8 operands for Tensor Core matrix multiplication. Their tile product accumulates in a signed 32-bit integer (INT32) result,

DB,r,c=X^B(c)(W^(r,c))T.(17)D_{\mathcal{B},r,c}=\widehat{X}_{\mathcal{B}}^{(c)}\left(\widehat{W}^{(r,c)}\right)^{\mathsf{T}}. \tag*{(17)}

The kernel converts this result to FP32 and multiplies it by ΔX(c)ΔW(r,c)\Delta_{X}^{(c)}\Delta_{W}^{(r,c)} before adding it to the output accumulator. Applying the scales per tile allows differently scaled integer products to contribute in the same numerical units.

The retained path instead multiplies XB(c)X_{\mathcal{B}}^{(c)} by the stored 16-bit weights, with FP32 accumulation. Within each assigned range, the kernel evaluates integer tiles in ascending input-column order, followed by retained tiles in ascending input-column order. Both paths consequently share a local partial sum while preserving their operand formats.

Once all contributions are combined, the result is cast to the deployment datatype. The original bias is added in that datatype when present, and valid output elements are stored. The module restores the input’s leading batch and sequence dimensions, completing the linear-layer interface.

Output-Tile Scheduling

Prefill provides multiple activation rows that can reuse a loaded weight tile, whereas decode provides fewer rows and benefits from additional parallel work across the weight grid. The kernel therefore chooses its activation-row tile size from the current call’s shape, while keeping the weight tiles and precision assignments unchanged. Let NcallN_{\mathrm{call}} be the rows passed to one GEMM invocation, including a row chunk when chunking is active.

Each output tile spans 64 output features. The kernel uses 16 activation rows per tile for Ncall≤16N_{\mathrm{call}} \le16 and 32 rows for 16<Ncall≤3216 < N_{\mathrm{call}} \le32. For Ncall>64N_{\mathrm{call}} > 64 with ntr≥4ntcn_{\mathrm{tr}} \ge4n_{\mathrm{tc}}, it uses 128 rows; all remaining shapes use 64 rows. Every configuration uses four warps and three loop-pipeline stages.

The split count balances the number of parallel programs against the tile work assigned to each. With activation-row tile size bNb_N, the grid contains ntr⌈Ncall/bN⌉n_{\mathrm{tr}}\lceil N_{\mathrm{call}}/b_N\rceil output tiles before splitting. The target is four times the GPU’s streaming-multiprocessor count. Once the output grid reaches this target, one program handles each tile. Otherwise, the kernel rounds the target-to-output-tile ratio up to a power of two and caps the split count SS at 16. It then halves SS while S>1S > 1 and ⌈ntc/S⌉<4\lceil n_{\mathrm{tc}}/S\rceil< 4, limiting splitting when few input tiles are available.

Each split receives a contiguous range from both precision-specific tile lists. For precision pp, let hp,r=Rp(r+1)−Rp(r)h_{p,r}=R_p(r+1)-R_p(r) be its tile count in row rr. Split s∈{0,…,S−1}s \in\{0,\ldots,S-1\} processes the buffer positions in

[Rp(r)+⌊hp,rsS⌋, Rp(r)+⌊hp,r(s+1)S⌋).(18)\left[ R_p(r)+\left\lfloor\frac{h_{p,r}s}{S}\right\rfloor,\, R_p(r)+\left\lfloor\frac{h_{p,r}(s+1)}{S}\right\rfloor \right). \tag*{(18)}

These ranges cover each stored tile exactly once. Sequential positions provide contiguous weight addresses, and the three-stage loop pipeline schedules tile loads alongside multiplication.

For S=1S=1, the program keeps its complete FP32 sum in registers and writes the final output tile. For S>1S>1, each program publishes its FP32 partial to a reusable workspace and increments an integer completion counter with acquire-release ordering. The last arriving program loads the partials in ascending split order, combines them inside the same kernel, and writes the completed output once. The fixed combination order makes the accumulation order independent of program completion order for a given schedule.

The workspace has active FP32 partial slices of shape [S,Ncall,M][S,N_{\mathrm{call}},M] and one INT32 counter per output tile. Buffers are cached per device and CUDA stream, and the completing program resets its counter for reuse. Algorithm 2 summarizes the arithmetic and completion steps.

Shared Projection Inputs

Query, key, and value projections consume the same attention input, while gate and up projections consume the same feed-forward input. For members sharing input width and tile geometry, with tile-aligned output widths and compatible device and bias configuration, the loader stacks their packed weight rows along the output dimension. A member preceded by δr\delta r output-tile rows receives a tile-ID offset of δr ntc\delta r\,n_{\mathrm{tc}}, following Eq. (9). Its row pointers are offset by the preceding tile count in the corresponding precision buffer. Weight values and scales stay with their original tiles.

Algorithm 2 Mixed-precision projection with output-tile scheduling

Require: Current input XX; packed weight tiles, scales, IDs, and row pointers; optional bias Ensure: Output in the deployment datatype with the original feature dimensions

  1. Prepare X^\hat{X} and {ΔX(c)}c\{\Delta X^{(c)}\}_c as in Appendix § 11.§ 1

  2. Select row-tile size and split count SS; obtain workspace with reusable zeroed counters

  3. for all output tiles (B,r)(B,r) and splits ss in parallel do

  4. Initialize register accumulator A←0A \leftarrow0 in FP32

  5. for all positions uu in the assigned 8-bit range from Eq. (18) do

  6. Recover input-tile column cc from its stored ID

  7. Compute INT32 product DB,r,cD_{B,r,c} using Eq. (17)

  8. A←A+FP32⁡(DB,r,c)(ΔX(c)ΔW(r,c))A \leftarrow A + \operatorname{FP32}(D_{B,r,c})(\Delta X^{(c)} \Delta W^{(r,c)})

  9. end for

  10. for all positions uu in the assigned 16-bit range from Eq. (18) do

  11. Recover cc and accumulate Xβ(c)(W(r,c))TX_{\beta}^{(c)}(W^{(r,c)})^{\mathsf{T}} into AA

  12. end for

  13. if S=1S=1 then

  14. Cast AA, add bias when present, and write valid output elements

  15. else

  16. Publish AA to the workspace slot for (s,B,r)(s,B,r)

  17. Atomically increment the output tile’s integer completion counter

  18. if this program is the last to arrive then

  19. Combine FP32 workspace partials in split order 0,…,S−10,\ldots,S-1

  20. Cast the result, add bias when present, and write valid output elements

  21. Reset the completion counter to zero

  22. end if

  23. end if

  24. end for

When the first projection in a group is invoked, the runtime prepares the shared input once and computes the combined output. Subsequent projections read their feature slices from that result. The group therefore shares activation preparation and one mixed-GEMM invocation while preserving each projection’s precision choices and output features. With row chunking, all grouped projections follow the same chunk sequence.

The runtime checks input identity and layout before reusing a grouped output and releases that output after every projection has consumed its slice. Separately invoked projections also reuse prepared integer activations through an inference-only, per-stream cache for compatible inputs of at least 32 rows. Both mechanisms reuse input preparation across projections sharing the same activations.

Repeated Generation and Runtime Reuse

Successive token steps invoke the same projection operations with new inputs, allowing the runtime to reuse model data and execution state. Weight buffers, scales, and row schedules remain resident. Compiled launcher handles are cached for repeated argument specializations, and device- and stream-local workspaces serve successive split reductions. Reusing these launchers and buffers reduces recurring setup across generation steps.

For fixed request shapes, the runtime prepares CUDA Graphs for prefill and one-token decode. It allocates the key/value cache, input-token storage, position state, and generated-token buffer before replay. Prefill initializes the cache and produces the first greedy token. Each decode replay consumes the current token, updates the cache, selects the next token on the GPU, and advances the GPU-resident position. Attention uses the valid cache prefix indicated by the current position.

Prefill capture is used for requests with at most 16,384 prompt activation rows across the batch. Larger prefills execute eagerly into the same preallocated cache, while the repeated decode body remains captured. This reserves graph-owned activation storage for the phases in which replay is used. Across both paths, the runtime continues to consume the same selected mixed-precision weights, while input values, cache contents, and token positions evolve with generation.

Additional Experiments

This appendix extends the main-text evaluation in the same order: inference efficiency, general tasks, and long-context tasks. We first detail the generation protocol and examine throughput, decoding latency, energy, and GPU memory. We then report complete quality results across model scales, retention ratios, and context lengths.

Experimental Setup

Generation workloads. End-to-end efficiency is evaluated on Llama-3.2-3B-Instruct using an NVIDIA A100 GPU with 40 GB memory. Each model instance runs on one GPU, with four independent processes per node assigned separate GPUs and corresponding CPU/NUMA resources. We vary batch size B∈{1,2,4,8}B \in\{1, 2, 4, 8\}, prompt length P∈{128,2048,8192,16384}P \in\{128, 2048, 8192, 16384\}, and generated length O∈{128,256,512,1024}O \in\{128, 256, 512, 1024\}, producing 64 request configurations per method. All methods receive identical synthetic token sequences constructed from a repeated text pattern. Greedy decoding generates exactly OO tokens for each request, even if an end-of-sequence token is produced.

Execution configurations. The BF16 reference uses the original model with scaled dot-product attention (SDPA). Torch FP16 uses Hugging Face generation with eager attention.

Uniform 8-bit weight-only quantization uses torchaos’s Int8WeightOnlyConfig through Hugging Face TorchAoConfig, while uniform 8-bit weight-and-activation quantization uses Int8DynamicActivationInt8WeightConfig with cuBLAS INT8 GEMM. GPTQ uses GPTQ-Model with 8-bit weights, group size 128, and the torch backend. LLM.int8 uses bitsandbytes with an outlier threshold of 6.0. For TileQuant, both GI and IAA are evaluated at 1%, 5%, 10%, and 20% retention. A pure 8-bit configuration using the same TileQuant execution implementation provides the zero-retention control.

Measurement protocol. Each configuration uses two warm-up runs followed by five measured runs. We report median timing measurements and maximum observed GPU memory, retaining out-ofmemory (OOM) and execution-failure statuses. Time to first token (TTFT) measures the interval from request submission to the first generated token, while time per output token (TPOT) averages the generation time after the first token. Output throughput is BO/TtotalBO/T_{\mathrm{total}}, where TtotalT_{\mathrm{total}} is the complete request duration. Energy is measured through NVIDIA Management Library (NVML) accumulated GPU energy and normalized to joules per 1,000 generated tokens, including idle power during the measured request. Throughput, latency, and energy comparisons use these definitions.

Inference Efficiency

The main text establishes the generation-throughput advantage of TileQuant across request shapes. Here, we examine how the two calibration routes and their retention budgets affect throughput, per-token latency, and energy. These measurements use the same 64 request configurations, allowing each metric to be compared under identical generated workloads.

Generation throughput. Figure 5, Figure 6, and Figure 7 expand the main throughput comparison into separate GI, IAA, and baseline groups. The GI and IAA panels include all four retention ratios, showing how additional 16-bit capacity affects complete-model generation. The baseline panels provide the corresponding executable-method comparisons across batch sizes, prompt lengths, and output lengths.

Figure 5. Generation-throughput speedup relative to BF16 on Llama-3.2-3B for gradient-informed (GI) TileQuant variants. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length and compares GI retention ratios; OOM denotes an out-of-memory run.

Figure 6. Generation-throughput speedup relative to BF16 on Llama-3.2-3B for input-activation-aware (IAA) TileQuant variants. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length and compares IAA retention ratios; OOM denotes an out-of-memory run.

Figure 7. Generation-throughput speedup relative to BF16 on Llama-3.2-3B for baseline inference methods. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel varies output length and compares the baseline methods; OOM denotes an out-of-memory run.

Figure 16. Line-level retrieval for Qwen2.5-7B across prompt lengths from 3.2K to 52.9K tokens. Line plots (left y-axis) report exact-match accuracy, while bar plots (right y-axis) show accuracy differences relative to the original baseline. Top-left: TileQuant with gradient-informed maps, where keep01, keep05, keep10, and keep20 retain 1%, 5%, 10%, and 20% of tiles in the deployment floating-point dtype. Top-right: TileQuant with input-activation-aware maps under the same keep ratios. Bottom-left: baseline PTQ methods, including uniform INT8 weight-only quantization, uniform W8A8 quantization, GPTQ, SmoothQuant, and LLM.int8. Bottom-right: representative cross-family comparison between the strongest TileQuant configurations and strongest baseline methods.

The throughput profiles show that increasing the retained fraction produces a modest change in execution efficiency. With batch size 1, a 128-token prompt, and 512 generated tokens, GI1_1 and GI20_{20} achieve 154.98 and 150.17 tokens/s, respectively. At batch size 8 with 16K prompt tokens and 1024 generated tokens, they achieve 306.20 and 301.93 tokens/s. IA exhibits closely matched throughput at the corresponding retention ratios. These results show that the common tile-native execution structure accommodates different calibration maps and retained capacities with efficient generation.

Time per output token. Complete-request throughput includes both prompt processing and repeated decoding. We therefore examine time per output token (TPOT) after the first generated token. Figure 8, Figure 9, Figure 10, and Figure 11 report TPOT speedup relative to BF16, defined as TPOTBF16/TPOTmethod\mathrm{TPOT}_{\mathrm{BF16}}/\mathrm{TPOT}_{\mathrm{method}} at matched request shapes. Higher values indicate lower decoding latency. Both GI1_1 and GI20_{20} exceed the BF16 reference throughout the displayed grid, showing that the generation gains extend to repeated decoding.

Figure 8. Time-per-output-token (TPOT) comparison on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares inference methods across output lengths of 128, 256, 512, and 1024 tokens; OOM denotes an out-of-memory run.

Figure 9. Time-per-output-token (TPOT) comparison for gradient-informed (GI) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares GI retention configurations across output lengths; OOM denotes an out-of-memory run.

Figure 10. Time-per-output-token (TPOT) comparison for input-activation-aware (IAA) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares IAA retention ratios across output lengths; OOM denotes an out-of-memory run.

Figure 11. Time-per-output-token (TPOT) comparison for baseline inference methods on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares the baseline configurations across output lengths; OOM denotes an out-of-memory run.

Figure 17. Line-level retrieval accuracy for Qwen2.5-14B on LongEval across prompt lengths from 3.2K to 52.9K tokens. Line plots (left y-axis) report exact-match retrieval accuracy, while bar plots (right y-axis) show accuracy differences relative to the Original floating-point model.

Generation energy. We further measure GPU energy per 1,000 generated tokens over the complete request, including prompt processing and decoding. Figure 12, Figure 13, Figure 14, and Figure 15 report energy efficiency relative to BF16, defined as EBF16/EmethodE_{\mathrm{BF16}}/E_{\mathrm{method}} for the same request shape. Higher values therefore indicate lower energy consumption per generated token. Across the displayed grid, GI1_1 achieves higher throughput and lower measured energy consumption than BF16.

Figure 12. Generation-energy comparison on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares inference methods across output lengths of 128, 256, 512, and 1024 tokens; OOM denotes an out-of-memory run.

Figure 13. Generation-energy comparison for gradient-informed (GI) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares GI retention configurations across output lengths; OOM denotes an out-of-memory run.

Figure 14. Generation-energy comparison for input-activation-aware (IAA) TileQuant variants on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares IAA retention configurations across output lengths; OOM denotes an out-of-memory run.

Figure 15. Generation-energy comparison for baseline inference methods on Llama-3.2-3B. Rows vary batch size over 1, 2, 4, and 8; columns vary prompt length over 128, 2K, 8K, and 16K tokens. Each panel compares the baseline configurations across output lengths; OOM denotes an out-of-memory run.

GPU Memory Results

We further examine how the retained-tile budget controls the memory footprint of reconstructed models across three model scales. These measurements cover checkpoint storage, post-load VRAM, peak VRAM, and prefill peak VRAM under the model-scale evaluation protocol. The end-to-end generation measurements follow the separate workload protocol in Appendix § 12.§ 1.

Table 4 reports prefill peak VRAM across prompt lengths from 1K to 32K. Table 5, Table 6, and Table 7 provide the complete memory profiles for Llama-3.2-3B, Qwen2.5-7B, and Qwen2.5-14B.

ModelLenGI1_1GI5_5GI10_{10}GI20_{20}
Llama-3.2-3B1K3.683.913.964.31
Llama-3.2-3B4K4.644.824.875.09
Llama-3.2-3B8K5.265.445.505.71
Llama-3.2-3B16K6.897.077.127.34
Llama-3.2-3B32K10.6410.8210.8711.09
Qwen2.5-7B1K8.568.829.269.72
Qwen2.5-7B4K9.689.9410.3810.84
Qwen2.5-7B8K10.1210.3710.8211.27
Qwen2.5-7B16K12.4312.6913.1313.58
Qwen2.5-7B32K15.9016.1516.6017.05
Qwen2.5-14B1K16.1216.7417.3518.29
Qwen2.5-14B4K17.6518.2718.8819.82
Qwen2.5-14B8K18.4919.3219.7220.66
Qwen2.5-14B16K21.6522.4822.8823.82
Qwen2.5-14B32K27.9627.9529.1930.13

Table 4. Prefill peak VRAM (GB) across prompt lengths for Llama-3.2-3B, Qwen2.5-7B, and Qwen2.5-14B. GI denotes gradient-informed tile retention; subscripts give the target 16-bit tile-retention percentage.

MetricOriginalOursUniformGPTQLLM.int8
GI1_1IAA1_1GI5_5IAA5_5GI10_{10}IAA10_{10}GI20_{20}IAA20_{20}W8W8A8
Storage5.983.393.393.503.503.633.633.893.893.363.363.423.36
Post-load VRAM5.993.513.513.643.643.693.694.024.025.995.993.553.46
Peak VRAM20.1516.1416.1416.3216.3216.3816.3816.5916.5916.7521.0017.5919.15
Prefill Peak VRAM12.6410.6410.6410.8210.8210.8710.8711.0911.0910.2412.2410.0911.40

Table 5. GPU memory comparison on the Llama-3.2-3B model. We compare the original model, representative PTQ baselines, and tile-level mixed-precision configurations across retained floating-point tile ratios. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. Values are in GB. Prefill peak VRAM is measured at a prompt length of 32K.

MetricOriginalOursUniformGPTQLLM.int8
GI1_1IAA1_1GI5_5IAA5_5GI10_{10}IAA10_{10}GI20_{20}IAA20_{20}W8W8A8
Storage14.198.188.188.428.428.738.739.339.338.118.118.258.11
Post-load VRAM14.198.258.258.528.528.978.979.409.4013.2013.208.358.44
Peak VRAM28.2722.8322.8323.0923.0923.5423.5423.9923.9924.8238.7022.3426.86
Prefill Peak VRAM21.3315.9015.9016.1516.1516.6016.6017.0517.0517.8824.8215.4017.61

Table 6. GPU memory comparison on the Qwen2.5-7B model. We compare the original model, representative PTQ baselines, and tile-level mixed-precision configurations across retained floating-point tile ratios. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. Values are in GB. Prefill peak VRAM is measured at a prompt length of 32K.

MetricOriginalOursUniformGPTQLLM.int8
GI1_1IAA1_1GI5_5IAA5_5GI10_{10}IAA10_{10}GI20_{20}IAA20_{20}W8W8A8
Storage27.5115.3515.3515.8515.8516.4616.4617.6917.6915.2115.2115.5015.21
Post-load VRAM27.5115.6715.6716.2616.2616.9016.9017.8517.8527.5227.5215.7515.95
Peak VRAM40.1027.9627.9927.9527.9529.1929.1930.1330.1338.4734.2827.6828.92
Prefill Peak VRAM40.1027.9627.9927.9527.9529.1929.1930.1330.1329.2234.2827.6828.92

Table 7. GPU memory comparison on the Qwen2.5-14B model. We compare the original model, representative PTQ baselines, and tile-level mixed-precision configurations across retained floating-point tile ratios. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. Values are in GB. Prefill peak VRAM is measured at a prompt length of 32K.

The results show that the retention ratio provides direct control over the persistent memory footprint. On Llama-3.2-3B, increasing retention from 1% to 20% changes post-load memory from 3.51 to 4.02 GB, compared with 5.99 GB for Original. The corresponding values are 8.25 to 9.40 GB on Qwen2.5-7B and 15.67 to 17.85 GB on Qwen2.5-14B. GI and IAA have identical or closely matched memory usage at the same retention ratio because the number and format of retained weights remain the same, while their selected locations differ.

Prefill measurements further show how the compact representation carries into longer inputs. At 32K prompt tokens, GI20_{20} uses 11.09 GB on Llama-3.2-3B, 17.05 GB on Qwen2.5-7B, and 30.13 GB on Qwen2.5-14B, compared with 12.64, 21.33, and 40.10 GB for their respective original models. The retention ratio controls persistent weight storage, while longer prompts additionally require activation and attention memory. Together with the quality evaluations, these measurements support selecting the retained capacity and calibration route jointly for deployment memory and accuracy.

General Tasks

After evaluating execution efficiency, we extend the general-task quality analysis to three model scales and the complete set of retention ratios. Tables 8, 9, and 10 report WikiText-2 perplexity and six downstream benchmarks for Llama-3.2-3B, Qwen2.5-7B, and Qwen2.5-14B. Both GI and IAA are evaluated with 1%, 5%, 10%, and 20% of target weight tiles retained at 16-bit, extending the main text’s 1% and 20% configurations.

Across the three models, the full retention sweep shows that locally preserving a limited number of sensitive tiles maintains strong general-task performance. On Qwen2.5-7B, GI20_{20} achieves 9.353 WikiText-2 perplexity, compared with 9.407 for Original. On Qwen2.5-14B, the corresponding values are 6.611 and 6.661. The improvements extend to selected downstream tasks: GI10_{10} reaches 75.55% MMLU accuracy on Qwen2.5-7B versus 74.23% for Original, while GI20_{20} reaches 86.49% HellaSwag accuracy on Qwen2.5-14B versus 85.07% for Original. These results extend the quality preservation observed on Llama-3.2-3B to larger models using the same tile allocation rule.

The two calibration routes also provide different quality profiles at the same retained capacity. On Qwen2.5-14B at 20% retention, IAA reaches 80.67% WinoGrande accuracy, compared with 80.42% for GI and 79.16% for Original. On HellaSwag, GI reaches 86.49%, compared with 86.06% for IAA and 85.07% for Original. These task-dependent rankings show how calibration affects quality with retained capacity and mixed-precision execution held fixed.

Table 8: General benchmark results for Llama-3.2-3B across language modeling and downstream reasoning tasks. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. Arrows next to benchmark names indicate the preferred metric direction: ↓ means lower is better and ↑ means higher is better. Accuracy values are reported as percentages, while WikiText-2 is reported with word perplexity. Bold values indicate the best result in each row. Cell colors are relative to the row-wise Original model under the corresponding metric direction: green for better than or equal to Original, blue for worse than Original, and gray for the Original column.

BenchmarkMetricOriginalGI₁IAA₁GI₅IAA₅GI₁₀IAA₁₀GI₂₀IAA₂₀W8W8A8GPTQSmoothQLLM.int8
WikiText-2 ↓PPL12.48712.51812.53712.47312.53112.45612.49412.40312.43812.45312.63012.51512.60912.639
WinoGrande ↑Acc.71.4371.3870.9872.0170.6770.9871.3071.4671.3070.8870.6471.4371.5971.11
PIQA ↑Acc.77.3176.6177.0977.0476.6176.9376.6177.4277.1576.6176.5576.8276.5576.82
LAMBADA ↑Acc.64.2365.6265.0765.4965.5865.6465.6665.7865.4964.4163.9664.4363.9864.31
ARC-Challenge ↑Acc.52.5652.8252.3052.8252.4753.5652.7153.6952.8852.3051.7952.7351.7151.88
HellaSwag ↑Acc.73.6573.2573.0973.2272.7273.1072.5072.9572.5073.5973.5373.5573.4773.48
MMLU ↑Acc.60.4760.6460.7860.9060.7460.8960.7161.1060.5460.5160.8060.4660.4860.89

Table 8.

Table 9: General benchmark results for Qwen2.5-7B across language modeling and downstream reasoning tasks. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. Arrows next to benchmark names indicate the preferred metric direction: ↓ means lower is better and ↑ means higher is better. Accuracy values are reported as percentages, while WikiText-2 is reported with word perplexity. Bold values indicate the best result in each row. Cell colors are relative to the row-wise Original model under the corresponding metric direction: green for better than or equal to Original, blue for worse than Original, and gray for the Original column.

BenchmarkMetricOriginalGI₁IAA₁GI₅IAA₅GI₁₀IAA₁₀GI₂₀IAA₂₀W8W8A8GPTQSmoothQLLM.int8
WikiText-2 ↓PPL9.4079.5429.5769.4669.5199.4349.5029.3539.4089.4209.7199.4139.7539.464
WinoGrande ↑Acc.75.5375.4075.5175.1775.1475.2474.8075.3275.8075.3074.5975.2274.6675.37
PIQA ↑Acc.80.5880.4480.3280.3380.8880.5581.0181.1580.6680.5879.0580.3078.8480.20
LAMBADA ↑Acc.69.4969.1469.8969.3769.1169.5669.0769.8969.2269.4767.3069.3864.4169.71
ARC-Challenge ↑Acc.67.0666.9066.4767.3366.8167.3367.7567.5867.5868.8967.1566.8965.9667.06
HellaSwag ↑Acc.81.3981.1081.2481.4981.2481.3581.3081.1581.5481.2680.8581.4080.3881.33
MMLU ↑Acc.74.2375.3775.0975.2675.1975.5575.2275.3975.5175.2274.7374.2373.5675.29

Table 9.

Table 10: General benchmark results for Qwen2.5-14B across language modeling and downstream reasoning tasks. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. Arrows next to benchmark names indicate the preferred metric direction: ↓ means lower is better and ↑ means higher is better. Accuracy values are reported as percentages, while WikiText-2 is reported with word perplexity. Bold values indicate the best result in each row. Cell colors are relative to the row-wise Original model under the corresponding metric direction: green for better than or equal to Original, blue for worse than Original, and gray for the Original column.

BenchmarkMetricOriginalGI₁IAA₁GI₅IAA₅GI₁₀IAA₁₀GI₂₀IAA₂₀W8W8A8GPTQSmoothQLLM.int8
WikiText-2 ↓PPL6.6616.7766.7746.7256.7406.6856.6956.6116.6376.6677.7216.6616.9286.751
WinoGrande ↑Acc.79.1679.2879.8080.4579.7379.7179.8080.4280.6779.4880.5879.4879.4879.16
PIQA ↑Acc.81.7781.8381.8481.5981.3181.4881.3581.9781.6881.8381.8381.6681.3481.56
LAMBADA ↑Acc.73.1072.7672.7572.9372.7972.9772.9473.0672.7972.4672.0772.9572.2172.99
ARC-Challenge ↑Acc.72.7071.1671.4271.5971.3371.7671.5071.2571.0072.2571.6772.6171.1671.93
HellaSwag ↑Acc.85.0784.9484.7885.0584.4286.3885.1986.4986.0685.2885.0685.0384.8085.24
MMLU ↑Acc.79.7780.3980.6780.5780.7880.9280.6680.8180.8180.5680.4680.4080.4280.51

Table 10.

Long Context Tasks

We further examine whether calibration-derived local precision choices preserve retrieval and question-answering capabilities as context length increases. Following the main text, LongEval evaluates evidence localization across input lengths, while LV-Eval extends the analysis to question answering over multiple documents and datasets.

Long-context retrieval. The main text evaluates Llama-3.2-3B on LongEval from 2.7K to 43.4K tokens. Figures 16 and 17 extend the same retrieval analysis to Qwen2.5-7B and Qwen2.5-14B, covering input lengths from 3.2K to 52.9K tokens. Each figure compares GI and IAA retention sweeps with the quantization baselines, using exact-match accuracy to measure successful line retrieval.

The Qwen results extend the retrieval evaluation to larger models and prompts up to 52.9K tokens. Within each model, the retention sweep examines how high-precision capacity affects line retrieval, while the matched-budget GI and IAA comparison evaluates the two calibration routes. Each resulting map is reused throughout the length sweep, connecting offline calibration to retrieval under changing input lengths.

Long-context question answering. LV-Eval extends the retrieval analysis to answering questions from long input contexts. Tables Table 11, Table 12, and Table 13 report complete results for Llama-3.2-3B, Qwen2.5-7B, and Qwen2.5-14B at five context-length levels: 16K, 32K, 64K, 128K, and 256K. Each model includes all four retention ratios for IAA and GI.

DatasetLenOriginalOurs GI₁Ours IAA₁Ours GI₅Ours IAA₅Ours GI₁₀Ours IAA₁₀Ours GI₂₀Ours IAA₂₀Uniform W8Uniform W8A8GPTQSmoothQLLM.int8
①16k21.4123.8224.2124.0123.8223.9223.7624.0623.8721.1521.3921.5320.1520
32k16.8618.1618.2318.3418.1918.2618.1318.3818.2317.9517.0517.3916.0616.31
64k7.58.4888.588.038.548.028.598.047.466.937.587.246.47
128k3.084.113.634.163.644.143.644.163.653.233.993.163.573.58
256k2.372.682.522.722.532.712.532.732.542.432.612.382.432.47
②16k16.4616.4517.3716.616.4616.5416.4216.6416.516.5417.3316.0717.4717.88
32k17.116.4816.8816.3517.0216.2916.1716.3917.0316.616.8815.8316.6416.37
64k15.5215.414.7215.7215.4615.6415.4315.7615.4714.5514.9715.0414.7714.84
128k11.11111.411.2711.1311.211.0811.311.1511.611.0411.1511.2610.92
256k7.436.667.077.567.417.517.47.587.426.876.66.886.616.6
③16k24.4925.2924.2225.5124.3625.4124.3325.5624.3724.3320.7124.7423.1523.52
32k19.2318.4317.2819.4319.1419.3519.1119.4819.1517.8219.1218.3520.7417.7
64k15.8613.6117.0216.0615.9215.9815.8616.115.9615.8914.2415.0815.7315.82
128k16.1513.4116.7815.8215.6815.7415.6215.8615.7214.2114.0314.8516.1215.58
256k10.449.4611.1210.6210.4910.5510.4410.6510.5210.319.9310.659.5710.66
④16k11.2314.0814.0514.1814.0814.1314.0514.2114.0911.1711.9910.7611.7412.39
32k11.3912.6810.1712.811.3312.7511.3212.8311.3410.8411.610.7211.1712.17
64k3.455.225.35.265.235.255.225.275.244.294.434.043.934.36
128k3.433.483.973.533.493.513.473.543.53.073.753.063.453.37
256k2.372.52.922.542.512.532.52.552.522.372.12.52.712.25
⑤16k22.3924.1224.0424.3224.124.2324.0624.3724.1122.9824.0122.3723.0122.68
32k23.0224.3224.6324.5624.3624.4624.2924.6124.4123.6123.623.1223.5122.39
64k15.4915.615.6715.815.6615.7215.615.8415.716.6816.4915.5417.0815.24
128k14.9813.9913.7515.2114.9315.1214.915.2414.9515.0614.5315.214.2915.47
256k10.4710.3411.9510.6510.5210.5810.4710.6810.5510.6211.5810.8111.4611.09
⑥16k17.8819.7719.6519.9319.719.8619.6819.9719.7118.6119.2118.6218.8518.26
32k15.7716.1315.7716.2915.8216.2215.816.3315.8315.9116.1315.316.3916.01
64k11.7712.4812.0112.6312.0612.5712.0412.6612.0712.4511.5211.6312.3911.97
128k7.527.917.848.027.887.987.868.047.897.557.917.316.897.12
256k6.135.616.836.246.166.26.136.256.176.186.846.257.036.24
⑦16k52.6151.8653.7352.8352.6152.7352.5552.8952.6752.6451.6452.8453.1952.69
32k37.7537.7237.9738.0137.7937.937.7138.0737.8536.5236.6436.9835.5738.03
64k24.3126.2223.726.5324.2126.4124.1826.5924.2424.6424.1424.6926.2425.66
128k14.6116.1616.9716.3816.216.316.116.4216.2314.8315.715.216.6914.92
256k10.9310.9310.7411.1210.911.0510.8811.1510.9210.5610.6510.8710.8311.24
⑧16k33.8134.2836.0134.534.2834.434.2234.5634.3433.9834.132.6935.7534.64
32k25.9527.5125.0827.7725.8327.6625.827.8325.8525.1426.0325.8426.525.74
64k21.2420.2221.2721.5121.3221.421.2421.5621.3821.3920.0121.5820.8921.42
128k12.2112.6514.0812.8412.6812.7612.6312.8612.7114.1312.9513.0413.512.98
256k7.657.977.218.17.638.057.628.127.647.998.018.248.047.73
⑨16k19.0523.3817.9923.5518.9523.4718.9323.5918.9620.9119.4118.8420.222.97
32k13.4815.9515.3716.0915.4116.0315.416.1215.4214.7715.6214.1918.0214.35
64k13.7113.8511.6114.0313.6513.9513.6314.0613.6713.5113.7213.9514.4912.57
128k6.68.446.798.546.828.56.818.566.836.936.946.966.987.31
256k4.785.354.845.434.875.44.865.454.875.734.965.375.515.48

Table 11. Full long-context question answering results for Llama-3.2-3B on LV-Eval across 9 datasets and context lengths from 16K to 256K. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. We follow the LV-Eval metric protocol: keyword-recall F1 by default, blacklist-based F1 for cmrc_mixup, and blacklist-based ROUGE-L for dureader_mixup. Cell colors are relative to the row-wise Original reference: green and blue indicate higher and lower scores, respectively, while gray marks Original and values within the neutral comparison range.

DatasetLenOriginalOurs GI₁Ours IAA₁Ours GI₅Ours IAA₅Ours GI₁₀Ours IAA₁₀Ours GI₂₀Ours IAA₂₀Uniform W8Uniform W8A8GPTQSmoothQLLM.int8
①16k24.622.3822.1823.423.3323.8523.8124.6324.6623.9924.2623.3322.6323.35
①32k23.1919.1720.1821.6621.622.3222.3523.4623.4922.5122.1621.9121.0123.41
①64k11.8311.1310.0611.8811.8512.2212.1912.8112.812.6311.7712.7611.7912.12
①128k7.416.55.696.976.957.187.167.547.567.516.877.336.127.46
①256k4.013.723.844.174.164.314.324.564.574.364.134.323.984.54
②16k20.8917.0517.8219.2219.1619.8419.8720.9420.9220.1320.1620.2220.2419.85
②32k19.0616.7917.4718.8218.7619.2619.2120.0520.0319.3719.9918.3518.3918.89
②64k17.5716.3818.0118.2818.2718.4118.4118.6118.6316.916.5217.1518.5516.4
②128k14.6612.9513.1714.1514.1114.5814.6115.3615.3315.0314.3515.2713.2614.48
②256k8.698.08.318.558.538.668.668.858.848.798.68.588.278.74
③16k5.064.534.574.094.995.225.235.575.564.935.4755.565.39
③32k3.42.722.42.822.873.253.153.533.513.2233.33.523.22
③64k2.612.472.442.362.8333.293.282.812.932.712.693.28
③128k2.211.881.972.122.122.192.22.312.32.172.022.172.092.3
③256k1.421.521.281.531.521.531.521.531.511.361.31.411.471.42
④16k18.8918.8618.0419.4919.4419.7719.7220.2720.2519.5720.219.7720.2218.68
④32k20.5418.4117.7720.0820.0219.4719.4219.9719.9519.2819.919.4720.1318.4
④64k13.813.0511.5413.713.6613.9813.9514.4814.4914.4313.314.1513.7313.68
④128k5.395.075.45.715.695.855.856.096.085.246.065.454.865.5
④256k2.973.472.743.483.473.483.483.493.492.913.463.323.242.9
⑤16k14.6713.7413.8214.3514.3214.5914.615.0114.9914.9713.9214.4114.7914.26
⑤32k13.2113.1312.2413.3813.3413.4913.4613.6813.6712.7713.2113.1912.9813.64
⑤64k10.6910.1710.0510.5910.5610.7810.7511.111.1111.0610.9610.8410.6610.9
⑤128k10.529.519.7910.1410.1210.310.3110.5910.5610.2210.1210.110.1510.15
⑤256k9.878.638.759.289.259.529.5310.059.949.419.449.448.419.47
⑥16k10.4110.139.3710.5510.5210.7410.7111.0511.071111.0410.8310.3910.84
⑥32k9.28.347.989.099.069.429.399.99109.328.989.519.979.7
⑥64k7.156.976.67.37.287.447.437.77.697.217.477.367.337.67
⑥128k5.965.465.385.725.75.845.826.046.055.725.756.0165.78
⑥256k5.595.355.275.665.645.85.786.036.045.525.715.8365.77
⑦16k26.321.8422.3624.6124.5425.625.6527.3727.3425.8125.1927.325.4725.67
⑦32k19.8417.8717.8119.0518.9919.5719.5820.520.4719.8419.3119.7220.4319.35
⑦64k13.2312.2112.3612.8812.8513.1213.1313.5113.5313.4412.5313.4613.4712.29
⑦128k9.558.858.619.559.539.869.8310.4210.49.5810.069.8210.369.4
⑦256k7.147.496.787.67.587.657.637.747.737.497.697.117.27.06
⑧16k18.0516.6216.9717.5117.4717.7517.7618.1618.1817.4916.518.1318.0117.96
⑧32k14.2412.5712.914.314.2614.9314.9616.0116.0314.4914.4213.7215.9814.43
⑧64k12.111.6911.7711.9711.9512.0512.0612.2212.211.9311.6411.4511.6312.16
⑧128k9.869.669.8710.410.3710.6310.6411.0311.059.6210.879.9710.999.48
⑧256k8.528.68.749.189.169.389.399.729.739.679.048.569.679.01
⑨16k14.5715.2914.9115.315.2615.3115.2815.3315.3114.6914.6515.1113.6514
⑨32k13.5113.8813.6914.0914.0514.1814.1514.3514.3313.1114.1613.7712.8714.3
⑨64k9.989.9410.5610.810.7910.9110.9211.0911.1110.949.8810.8211.0610.74
⑨128k7.157.177.617.637.637.637.637.657.646.476.926.877.267.58
⑨256k5.214.464.995.345.325.495.55.775.764.94.114.495.735.03

Table 12. Full long-context question answering results for Qwen2.5-7B on LV-Eval across 9 datasets and context lengths from 16K to 256K. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. We follow the LV-Eval metric protocol: keyword-recall F1 by default, blacklist-based F1 for cmrc_mixup, and blacklist-based ROUGE-L for dureader_mixup. Cell colors are relative to the row-wise Original reference: green and blue indicate higher and lower scores, respectively, while gray marks Original and values within the neutral comparison range.

DatasetLenOriginalGI₁IAA₁GI₅IAA₅GI₁₀IAA₁₀GI₂₀IAA₂₀W8W8A8GPTQSmoothQLLM.int8
①16k27.6226.6125.9628.2628.1829.2529.1730.2830.228.1128.7727.7926.6930.12
32k26.8425.5124.7926.5726.4927.2127.1327.8727.7927.1927.6826.7525.5925.84
64k12.8212.0113.2314.314.2614.9514.915.6115.5712.1915.511.4812.5712.8
128k8.867.257.638.248.28.68.578.988.948.78.888.638.18.57
256k4.414.954.515.195.175.335.315.485.464.615.424.674.884.82
②16k20.5919.419.2219.9919.9320.3520.2920.7220.6620.4519.1320.3720.219.91
32k20.217.4719.2220.1520.0920.7120.6521.2921.2320.2619.0220.0620.0620.36
64k19.0917.1317.2418.2518.1918.8518.7919.4819.4218.9818.2719.3118.3918.13
128k15.1714.5914.3915.3415.2715.7915.7216.2616.1915.4414.8315.3915.9816.08
256k8.2610.489.5110.5310.4910.5610.5210.5910.558.668.768.238.218.72
③16k4.214.033.544.214.24.274.234.374.354.234.334.344.244.09
32k2.762.532.93.353.223.363.223.373.362.833.182.783.333.07
64k2.542.32.192.412.392.522.392.62.592.482.582.52.482.45
128k2.22.362.012.122.122.382.32.392.382.32.042.242.182.32
256k1.621.941.751.781.781.961.921.971.961.681.851.691.761.59
④16k21.1822.622.6323.2123.1523.5623.523.9323.8622.0723.7921.0723.1622.73
32k24.5122.3221.5223.3923.3124.0323.9524.724.6221.7423.4220.7522.8122.75
64k13.213.5314.0314.4214.3714.6514.614.8914.8413.2814.4912.7714.7714
128k6.117.16.657.137.117.157.127.177.146.576.946.256.216.62
256k4.624.894.565.075.055.175.155.285.264.674.624.625.224.65
⑤16k13.9914.0513.5214.2114.1714.3114.2614.4114.3614.3214.213.6913.6914.29
32k14.1812.913.3213.7613.7114.0213.9714.2914.2413.7213.6213.5913.9913.28
64k10.2610.2610.8510.9610.9311.0210.9911.0911.0610.2710.1410.6310.3611
128k10.8910.210.0310.5910.5410.8210.7711.0611.0110.129.9910.5610.2910.36
256k8.568.728.858.98.888.938.98.968.938.718.758.838.338.19
⑥16k11.5610.4410.3610.9810.9411.311.2611.6311.611.1210.0310.869.910.55
32k9.849.359.349.69.579.769.739.919.889.738.919.398.749.07
64k7.537.527.727.977.958.128.18.288.258.217.947.818.177.73
128k6.295.675.836.076.046.216.196.366.346.195.6965.956.06
256k4.944.914.955.135.115.245.225.365.334.885.115.025.295.07
⑦16k28.8523.0522.822.0121.9927.3422.6829.0128.9327.3925.3428.6225.9424.8
32k24.017.8218.2521.1421.0622.8722.7924.6624.5923.5822.8224.482222.3
64k14.7412.5913.4914.2514.214.714.6515.1715.1215.0414.9914.4213.7914.32
128k10.938.948.6910.1610.1110.910.8511.6611.611111.5310.810.089.79
256k8.037.57.587.957.928.178.148.418.377.658.268.37.197.35
⑧16k16.4317.316.7417.3517.2917.3817.3217.4117.3516.116.5516.416.8317.1
32k14.9212.7614.3914.6814.6314.8514.8115.0314.9914.1614.3514.7313.3814.7
64k11.9511.411.9812.0512.0212.0812.0512.1312.0911.7312.0211.7211.4511.76
128k9.759.339.399.749.699.959.910.1610.1210.059.729.729.749.93
256k7.277.737.868.168.138.348.318.538.497.668.437.898.077.71
⑨16k16.915.6518.5419.2319.1719.6419.5920.0720.0116.4115.91619.9617.9
32k16.314.8515.015.9615.9116.5416.4917.1417.0915.7313.5515.3417.0216.72
64k14.2714.0713.5914.6314.5814.9714.9215.3215.2714.213.3514.2514.3614.56
128k7.547.596.838.518.489.069.039.649.68.679.138.579.558.51
256k5.565.496.356.386.376.46.386.426.45.556.135.515.365.17

Table 13. Full long-context question answering results for Qwen2.5-14B on LV-Eval across 9 datasets and context lengths from 16K to 256K. GI and IAA denote Gradient-informed and Input-activation-aware tile-retention policies, with subscripts indicating the retained floating-point tile ratio in percent. We follow the LV-Eval metric protocol: keyword-recall F1 by default, blacklist-based F1 for cmrc_mixup, and blacklist-based ROUGE-L for dureader_mixup. Cell colors are relative to the row-wise Original reference: green and blue indicate higher and lower scores, respectively, while gray marks Original and values within the neutral comparison range.

The nine datasets follow the numbering in Table 2: ① cmrc_mixup, ② dureader_mixup, ③ hotpotwikiqa_mixup, ④ lic_mixup, ⑤ loogle_CR_mixup, ⑥ loogle_MIR_mixup, ⑦

loogle_SD_mixup, ⑧ multifieldqa_en_mixup, and ⑨ multifieldqa_zh_mixup. We follow the LV-Eval protocol, reporting keyword-recall-based F1 by default, blacklist-based F1 for cmrc_mixup, and blacklist-based ROUGE-L for dureader_mixup.

The 128K and 256K LV-Eval levels extend the main-text question-answering evaluation to longer contexts. On Qwen2.5-14B, GI₂₀ scores 11.66 on LooGLE-SD at 128K, compared with 10.93 for Original. On Qwen2.5-7B, it scores 9.72 on English MultiFieldQA at 256K, compared with 8.52 for Original. Each model reuses its calibrated maps across these length levels, showing that the selected local regions remain useful for answering questions in longer contexts.

Across the full LV-Eval evaluation, the retention ratio sets the available 16-bit capacity, and the calibration route determines its allocation and the associated quantization parameters. The 20% configurations frequently achieve strong results across datasets and lengths, while lower-retention variants provide additional operating points with smaller weight representations. Each model uses its own calibration-derived maps, which are reused across

that model’s retrieval and question-answering tasks and input lengths.

Calibration and Allocation Analysis

We examine how retained locations, allocation geometry, and calibration choices contribute to Tile-Quant. The first three studies isolate tile selection, allocation geometry, and tile size under matched high-precision budgets on Qwen2.5-7B. We then evaluate calibration cost, the reproducibility of retained sets, and the differences between GI and IAA maps. For the selection, geometry, and tile-size controls, matched-control perplexity (PPL) is evaluated on a fixed subset of 62 WikiText-2 test documents using a common evaluation wrapper. Full-benchmark results are in Appendix § 12.§ 3.

Matched-Budget Tile Selection

To isolate the value of retained locations, we fix the 64×6464 \times64 tile grid and compare IAA, magnitude-based selection, and random selection. Within each budget, all three rules retain the same number of high-precision weights: 326,516,736 at 5% and 1,305,608,192 at 20%. Random selection uses five independent draws. Table 14 reports the resulting matched-control PPL and retained saliency mass, the percentage of calibration saliency covered by the retained weights.

Table 14: Matched-budget tile-selection controls on Qwen2.5-7B with 64×6464 \times64 tiles. Matched-control PPL uses the fixed subset of 62 WikiText-2 test documents. The target-linear payload is shared by all selection rules within each budget. Random PPL is reported as mean ±\pm sample standard deviation over five draws. Bold marks the best displayed value in each row.

Retention (%)Target-linear (GiB)MetricRandomMagnitudeIAA
56.39Retained saliency mass (%) ↑\uparrow5.005.347.05
Matched-control PPL ↓\downarrow9.92±0.039.92 \pm0.039.939.85
207.30Retained saliency mass (%) ↑\uparrow20.0020.7623.90
Matched-control PPL ↓\downarrow9.82±0.039.82 \pm0.039.789.7650

Table 14.

At 5% retention, IAA achieves 9.85 PPL, compared with 9.93 for magnitude selection and 9.92 for random selection, while covering 7.05% of the calibration saliency. At 20%, IAA further achieves 9.7650 PPL, compared with 9.78 and 9.82 for the two controls. Since all methods retain the same high-precision capacity within each comparison, these results show that activation-aware scores effectively prioritize useful local weight regions. We next examine how the geometry of these regions affects their representation and execution.

Allocation Geometry and Execution

Selection scores determine priority, while the allocation unit determines which weights share a precision decision. We therefore compare 1×641 \times64 and 64×164 \times1 strips, full rows, full columns, and 64×6464 \times64 tiles at a matched 20% retention target. Table 15 reports the payload, metadata, retained saliency mass, and matched-control PPL for these alternatives.

Table 15: Allocation-geometry controls on Qwen2.5-7B at a matched 20% high-precision retention target. Total payload denotes the stored representation measured in this control. Retained saliency mass is reported as a fraction, and PPL uses the fixed 62-document subset. Bold marks the best displayed value in each row.

MetricUnitFull column1×641 \times6464×164 \times1Full row64×6464 \times64
Total payload ↓\downarrowGiB7.29957.37447.37447.29977.2984
Metadata ↓\downarrowMiB0.4377.2177.210.530.61
Retained saliency mass ↑\uparrowFraction0.310.260.320.230.24
Matched-control PPL ↓\downarrowPPL9.729.879.719.989.7650

Table 15.

At 20% retention, 64×6464 \times64 tiles achieve 9.7650 matched-control PPL, close to 9.72 for full columns and 9.71 for 64×164 \times1 strips. Their metadata occupies 0.61 MiB, compared with 77.21 MiB for the compact strips. Thus, 2D tiles preserve similar language-modeling quality in this control with substantially less metadata than fine-grained strips.

The execution measurements below test how these representations affect prefill and decoding. We further compare executable full-column, compact-strip, and tile implementations using the same IAA entry scores and 20% retention target. These geometry measurements use an A100-40GB GPU, with 10 warm-up requests followed by 30 timed requests in each of three fresh processes. Table 16 reports prefill latency, decode TPOT, and post-load memory for this dedicated control.

Table 16: Geometry-specific execution on Qwen2.5-7B with IAA scores and 20% 16-bit retention. Native tiles are 64×6464 \times64. FP-intersected tiles contain retained weights; partially mixed tiles contain both precisions. B1 and B4 denote batch sizes 1 and 4; TPOT is time per output token. Bold marks the minimum in each row.

MetricUnitFull column64×164 \times1 compact strip64×6464 \times64 tile
Metadata ↓\downarrowMiB0.43677.2080.611
FP-intersected native tiles ↓\downarrow%99.5099.5320.01
Partially mixed native tiles ↓\downarrowCount1,581,5521,583,8600
Prefill B1/8K ↓\downarrowms1548.574183.851211.46
Decode B1 TPOT ↓\downarrowtoken57.1258.6740.08
Decode B4 TPOT ↓\downarrowtoken58.0662.9339.73
Post-load memory ↓\downarrowGiB10.57411.6749.377

Table 16.

Relative to full-column allocation, the tile implementation reduces 8K prefill latency by 21.8%, B1 decode TPOT by 29.8%, and B4 decode TPOT by 31.6%, with approximately 1.20 GiB less post-load memory. Together with the quality comparison, these measurements support the hardware-aligned tile as an effective allocation and execution unit among those evaluated.

Tile-Size Sensitivity

Within the 2D design, tile size controls the resolution of precision placement and the amount of selection metadata. We compare 32×3232 \times32, 64×6464 \times64, and 128×128128 \times128 tiles while retaining exactly 66,977,792 high-precision weight elements in every configuration, corresponding to approximately 1.026% retention. Table 17 therefore evaluates size changes at equal retained capacity.

Table 17: Exact-budget tile-size sensitivity on Qwen2.5-7B. Each configuration retains 66,977,792 high-precision weight elements. Map time measures decision-map construction. PPL is the matched-control value, MMLU accuracy is a percentage, and LongEval accuracy is a fraction at 52.9K tokens. Bold marks the best displayed value in each row.

MetricUnit32×3232 \times3264×6464 \times64128×128128 \times128
Target-linear payload ↓\downarrowGiB6.14576.14546.1454
Metadata ↓\downarrowBytes246,84835,84011,312
Map-construction time ↓\downarrows5.483.162.16
Matched-control PPL ↓\downarrowPPL9.93689.93199.9322
MMLU ↑\uparrow%73.8974.0673.80
LongEval at 52.9K ↑\uparrowAccuracy0.400.500.30

Table 17.

The 64×6464 \times64 configuration achieves the highest MMLU accuracy and LongEval accuracy among the three evaluated sizes, reaching 74.06% and 0.50, respectively. Compared with 32×3232 \times32 tiles, it requires 85.5% less metadata and 42.3% less map-construction time. These results support the selected tile dimensions for local precision allocation and compact metadata construction.

Calibration Cost

The two scoring routes share the selection and execution pipeline, while collecting different calibration signals. To quantify their offline resource requirements, we compare wall-clock time and peak memory at matched sample counts of 128, 256, and 512 (Table 18). These cost measurements use the stated sample counts, while the full quality evaluation uses maps calibrated with 8,192 examples.

Table 18: Offline calibration time and peak GPU memory for GI and IAA at three sample counts. Speedup is the ratio of GI time to IAA time at the same sample count. Bold marks the lower-cost route in each pair.

SamplesTime (s) ↓\downarrow GITime (s) ↓\downarrow IAAIAA speedup (×\times)Peak memory (GiB) ↓\downarrow GIPeak memory (GiB) ↓\downarrow IAA
128112.436.83.0525.419.24
256218.377.02.8325.419.24
512407.1132.63.0725.419.24

Table 18.

IAA is 2.83–3.07×2.83\text{--}3.07\times faster across these sample counts, and its peak memory is 9.24 GiB versus 25.41 GiB for GI, a reduction of 63.6%. This difference follows the information collected by each route: IAA uses forward statistics, while GI additionally computes gradients of the prediction loss. The shared pipeline therefore supports lower-cost calibration alongside loss-sensitive selection.

Retained-Set Stability

We examine whether changing the calibration records changes which tiles receive high precision. Calibration is repeated across three independent draws for each route at 1% and 20% retention. For retained sets R1R_1 and R2R_2 from two draws, Jaccard similarity is J(R1,R2)=∣R1∩R2∣/∣R1∪R2∣J(R_1, R_2) = \lvert R_1 \cap R_2\rvert/\lvert R_1 \cup R_2\rvert. Table 19 reports agreement between full-model retained sets and within individual projections, measuring the repeatability of local precision selection.

Table 19: Retained-set stability across three independent calibration draws at 1% and 20% retention. Values report Jaccard similarity between selected tile sets. Bold marks the larger value in each pair, including ties.

Statistic1% retention GI1% retention IAA20% retention GI20% retention IAA
Pairwise Jaccard mean ↑\uparrow0.980.990.990.99
Pairwise Jaccard minimum ↑\uparrow0.970.990.990.99
Per-layer Jaccard median ↑\uparrow0.970.990.990.99
Per-layer Jaccard minimum ↑\uparrow0.930.970.970.99

Table 19.

The mean pairwise Jaccard is 0.98 for GI at 1% retention and 0.99 for the other three configurations. Layer-level minima range from 0.93 to 0.99, showing that the agreement extends to individual layers. Both routes therefore produce reproducible retained locations under the evaluated sampling changes. We next compare GI with IAA to distinguish this within-route stability from differences between the two scoring signals.

GI/IAA Retained-Set Overlap

GI and IAA use the same capacity rule but evaluate importance through different signals. Table 20 compares their retained sets across three models and four retention ratios. Jaccard measures intersection relative to union, while shared/retained measures the fraction of either equally sized retained set selected by both routes. The chance-normalized column compares Jaccard with the expected overlap of random maps at the same retention ratio.

ModelRetention (%)JaccardJaccard / chance (×)Shared / retained (%)
Llama-3.2-3B10.048.608.29
Llama-3.2-3B50.093.4516.25
Llama-3.2-3B100.132.5223.39
Llama-3.2-3B200.221.9435.44
Qwen2.5-7B10.1529.1925.58
Qwen2.5-7B50.2911.2044.63
Qwen2.5-7B100.366.8753.11
Qwen2.5-7B200.423.7859.14
Qwen2.5-14B10.1325.6722.85
Qwen2.5-14B50.228.7736.71
Qwen2.5-14B100.295.4744.71
Qwen2.5-14B200.373.3053.62

Table 20. Overlap between GI and IAA retained sets across models and retention ratios. Jaccard is reported as a fraction, shared/retained as a percentage, and Jaccard/chance as the reported ratio to random-map overlap. These columns describe agreement between selection routes across models.

Agreement exceeds the reported chance reference by 1.94–29.19×1.94\text{--}29.19\times. At 20% retention, the two routes share 35.44% of retained tiles on Llama3.2-3B, 59.14% on Qwen2.5-7B, and 53.62% on Qwen2.5-14B. The maps therefore contain shared selections alongside locations preferred by each scoring signal. These overlaps describe the difference in retained locations, while the quality and calibration-cost experiments evaluate the practical value of each route.

To locate differences between the two retained maps, Table 21 groups the 20% retention results on Qwen2.5-7B by projection family across 28 decoder layers. Both routes retain 318,752 tiles, with 188,508 shared selections, corresponding to 41.98% Jaccard similarity and 59.14% shared/retained overlap. Retained saliency mass measures the fraction of importance scores covered by selected regions in each projection family.

ProjectionGIIAAJaccard (%)Shared / retained (%)
q_proj32.3725.3323.8938.03
k_proj28.6226.1331.1245.67
v_proj27.8624.0722.3032.66
o_proj29.3529.7058.5171.97
up_proj22.7521.9439.1255.83
gate_proj24.0325.5949.7466.13
down_proj22.5822.0941.4058.11

Table 21. Projection-level saliency coverage and GI/IAA retained-set overlap on Qwen2.5-7B at 20% retention. All values are percentages. The GI and IAA columns report retained saliency mass, while the final two columns describe agreement between their retained sets. These statistics characterize selected regions by projection family and scoring route.

Agreement is highest in the attention-output and gate projections, while the query and value projections show less overlap. The retained budget is identical for the two routes within each projection, so these differences concern the locations receiving high precision. Projection-level comparisons therefore identify where the two calibration routes make the most different local selections.

References

  1. [1]Meta AI. 2024. Llama 3.2 model card.
  2. [2]Chenxin An, Shanshan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. L-level: Instituting standardized evaluation for long context language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14388–14411.arxiv.org/abs/2307.11088
  3. [3]Yamato Arai and Yuma Ichikawa. 2025. Quantization error propagation: Revisiting layer-wise post-training quantization. arXiv preprint arXiv:2504.09629.arxiv.org/abs/2504.09629
  4. [4]Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. 2024a. Quik: Towards end-to-end 4-bit inference on generative large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3355–3371.
  5. [5]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024b. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213–100240.arxiv.org/abs/2404.00456
  6. [6]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 3119–3137.arxiv.org/abs/2308.14508
  7. [7]Yonatan Bisk, Rowan Zellers, Jiafeng Gao, Yejin Choi, et al. 2020. Piq: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.arxiv.org/abs/1911.11641
  8. [8]Jerry Chee, Arturs Backurs, Rainie Heck, Li Zhang, Janardhan Kulkarni, Thomas Rothvoss, and Sivakant Gopi. 2025. Discquant: A quantization method for neural networks inspired by discrepancy theory. arXiv preprint arXiv:2501.06417.arxiv.org/abs/2501.06417
  9. [9]Zihan Chen, Bike Xie, Jundong Li, and Cong Shen. 2024. Channel-wise mixed-precision quantization for large language models. arXiv preprint arXiv:2410.13056.arxiv.org/abs/2410.13056
  10. [10]Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021. Nvidia a100 tensor core gpu: Performance and innovation. IEEE Micro, 41(2):29–35.DOI
  11. [11]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.arxiv.org/abs/1803.05457
  12. [12]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318–30332.
  13. [13]Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. 2023. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078.arxiv.org/abs/2306.03078
  14. [14]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.arxiv.org/abs/2210.17323
  15. [15]Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2022. A survey of quantization methods for efficient neural network inference. In Low-power computer vision, pages 291–326. Chapman and Hall/CRC.arxiv.org/abs/2103.13630
  16. [16]Zhuocheng Gong, Jiahao Liu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. 2024. What makes quantization for large language model hard? an empirical study from the lens of perturbation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18082–18089.DOI
  17. [17]Ziyi Guan, Hantao Huang, Yupeng Su, Hong Huang, Ngai Wong, and Hao Yu. 2024. Aptq: Attention-aware post-training mixed-precision quantization for large language models. In Proceedings of the 61st ACM/IEEE Design Automation Conference, pages 1–6.DOI
  18. [18]Design Guide. 2020. Cuda c++ programming guide. NVIDIA, July.
  19. [19]Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. 2023. Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–15.DOI
  20. [20]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multilingual language understanding. arXiv preprint arXiv:2009.03300.
  21. [21]Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xianglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and Xiaojian Qi. 2024. Slim-llm: Salience-driven mixed-precision quantization for large language models. arXiv preprint arXiv:2405.14917.arxiv.org/abs/2405.14917
  22. [22]Andrew Kerr, Duane Merrill, Julien Demouth, and John Tran. 2017. Cutlass: Fast linear algebra in cuda c++. NVIDIA Developer Blog, 2:10.
  23. [23]Junhan Kim, Ho-young Kim, Eulrang Cho, Chungman Lee, Joonyoung Kim, and Yongkwoon Jeon. 2024. Boa: Attention-aware post-training quantization without backpropagation. arXiv preprint arXiv:2406.13474.arxiv.org/abs/2406.13474
  24. [24]Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2023. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629.arxiv.org/abs/2306.07629
  25. [25]Alex Kogan. 2025. Is (selective) round-to-nearest quantization all you need? arXiv preprint arXiv:2505.15909.arxiv.org/abs/2505.15909
  26. [26]Jiedong Lang, Zehao Guo, and Shuyu Huang. 2024. A comprehensive study on quantization techniques for large language models. In 2024 4th International conference on artificial intelligence, robotics, and communication (ICAIRC), pages 224–231. IEEE.arxiv.org/abs/2411.02530
  27. [27]Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjin Kim, and Eunhyeok Park. 2024. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13355–13364.
  28. [28]Sangjun Lee, Seung-taek Woo, Jun-gyu Jin, Changhun Lee, and Eunhyeok Park. 2025. Amq: Enabling automl for mixed-precision weight-only quantization of large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35520–35538.arxiv.org/abs/2509.12019
  29. [29]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474.arxiv.org/abs/2005.11401
  30. [30]Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xue zhe Ma, and Hao Zhang. 2023a. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
  31. [31]Shiyao Li, Xuefei Ning, Ke Hong, Tengxuan Liu, Luning Wang, Xiuhong Li, Kai Zhong, Guohao Dai, Huazhong Yang, and Yu Wang. 2023b. Llm-mq: Mixed-precision quantization for efficient llm deployment. In The Efficient Natural Language and Speech Processing Workshop with NeurIPS, volume 9, page 3.
  32. [32]Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426.arxiv.org/abs/2102.05426
  33. [33]Yuhang Li and Priyadarshini Panda. 2024. Tesseraq: Ultra low-bit llm post-training quantization with block reconstruction. arXiv preprint arXiv:2410.19103.arxiv.org/abs/2410.19103
  34. [34]Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 881–893.arxiv.org/abs/2407.16833
  35. [35]Chengxi Liao and Zeyi Wen. 2026. Channel-aware mixed-precision quantization for efficient long-context inference. In The Fourteenth International Conference on Learning Representations.
  36. [36]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87–100.DOI
  37. [37]Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. 2023. Qllm: Accurate and efficient low-bitwidth quantization for large language models. arXiv preprint arXiv:2310.08041.arxiv.org/abs/2310.08041
  38. [38]Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12:157–173.arxiv.org/abs/2307.03172
  39. [39]Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024b. Spiquant: Llm quantization with learned rotations. arXiv preprint arXiv:2405.16406.arxiv.org/abs/2405.16406
  40. [40]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.arxiv.org/abs/1609.07843
  41. [41]Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quantization. In International conference on machine learning, pages 7197–7206. PMLR.arxiv.org/abs/2004.10568

Paper details

vlnc:W_ttr9666e
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents