betaFeedback

WorksComputer Science

Query-Guided RoPE Modulation for Lost-in-the-Middle Mitigation in Long-Context LLMs

Abstract

Long-context language models often neglect information in the middle of the input, a phenomenon known as "lost-in-the-middle". Part of this gap may stem from the long-range decay of rotary position embedding (RoPE), which weakens the positional signal between distant tokens. Existing input compression deletes low-importance tokens, and attention reweighting acts only after RoPE has weakened this signal. RoPE-level methods modify positions directly but apply the same remapping to every query. To address these limitations, we propose QGPM (Query-Guided Positional Modulation), a training-free method that reshapes RoPE positions using query relevance from the model's own attention heads. QGPM builds a broad relevance map for lower layers and a sharp one for upper layers. Low-relevance tokens advance by less than one position on fast-rotating planes, pulling relevant evidence toward the query. Slow-rotating planes keep the original layout, with every token in place. Experiments on NoLiMa and RULER with multiple context lengths show that QGPM improves over base models and strong baselines, with the largest gains on retrieval tasks, and generalizes across multiple backbones. Our code is available at https://github.com/iZHpy/QGPM.

Introduction

Long-context language models now support context windows of hundreds of thousands of tokens [1, 5, 26]. Whether these models actually use the information in such windows, however, remains an open question. Models often attend to the beginning and end of the input while neglecting the middle, a phenomenon known as “lost-in-the-middle” [15]. Benchmarks designed to probe context utilization [17, 8] show that accuracy drops sharply as context grows, especially when the relevant evidence shares little surface overlap with the query. This paper therefore aims to improve how models use the information already in the context window, rather than to extend the window further.

Part of this utilization gap may stem from rotary position embedding (RoPE) [23], the position encoding adopted by most modern LLMs. RoPE represents position as rotations in many two-dimensional planes, each with a distinct frequency. Over a long distance, the fast-rotating planes complete many full turns and no longer carry a consistent distance signal. Evidence far from the query is thus matched through fewer planes than nearby text. This long-range decay depends on the positional distance between tokens. Assigning relevant evidence a position closer to the query can therefore mitigate the decay without changing the input text.

Existing methods that target lost-in-the-middle fall short of intervening on RoPE in a query-aware manner. Input compression methods [11, 19, 13] delete low-importance tokens and thus discard graded relevance information. Attention reweighting methods (Zhang et al., 2024a,b; Guardieiro et al., 2025) strengthen attention to selected tokens, but only after RoPE has weakened the positional signal between distant tokens. RoPE-level methods [32, 12, 2] modify position encoding directly, yet apply the same remapping regardless of the query. A separate line of work shows that a few attention heads already attend to query-relevant spans in long inputs [24, 30]. Such attention signals have been used to select KV cache entries [14], filter input tokens [22], or re-rank documents [30], but always by deleting or reordering content. In short, no existing method uses the model’s own query relevance to reshape RoPE positions while keeping every token in place.

We propose QGPM (Query-Guided Positional Modulation), a training-free method that fills this gap. QGPM reads the attention from the query to each context sentence in a few such heads. The resulting relevance ranking is shaped into a broad map and a sharp map. Using the two maps, QGPM lets low-relevance tokens advance by less than one position on the fast-rotating planes, so relevant sentences move closer to the query. In contrast, the slow-rotating planes keep the original layout of the document. The broad map is applied to the lower layers and the sharp map to the upper layers. QGPM deletes or reorders no token and updates no model weight.

Our main contributions are as follows:

  • We propose QGPM, which converts query relevance read from the model’s own attention heads into token-level and frequency-specific RoPE positions. The same context thus receives different positions under different queries, which makes the evidence needed by each query easier for the model to retrieve.

  • We design a modulation scheme that is aware of both frequency and depth. Low-relevance text is compressed only on fast-rotating planes, and separate relevance maps guide the lower and upper layers.

  • Comprehensive experiments on multiple LLM backbones and long-context benchmarks show that QGPM achieves superior overall performance over strong baselines across a range of context lengths.

Related Work

Mitigating Lost-in-the-Middle

Inference-time methods for lost-in-the-middle ([15]) mostly edit the input or the attention weights without touching position encoding. Input compression methods such as Selective Context ([13]), LongLLMLingua ([11]), and LLMLingua-2 ([19]) delete tokens judged unimportant. TAG ([18]) instead inserts tags that mark salient content. Attention reweighting methods such as PASTA ([28]), AutoPASTA ([29]), and InstABoost ((Guardieiro et al., 2025)) amplify attention to selected tokens, and Hsieh et al. [9] calibrate the positional bias in attention. Deletion loses information, and tagging leaves the long-range decay untouched. Attention-side methods can only rescale a positional signal that RoPE has already weakened.

RoPE and Position Modification

Most RoPE modifications extend the context window by rescaling position indices, as in Position Interpolation ([3]), YaRN ([20]), and LongRoPE ([4]). Fewer methods improve utilization within the window. Ms-PoE ([32]) rescales positions with head-specific ratios, and Self-Extend ([12]) maps distant positions back into the trained range by grouping. STRING ([2]) reuses well-trained positions for long distances, and Position Engineering ([7]) inserts placeholder positions tuned for each task. All of these remappings are fixed before the query is seen, whereas QGPM derives the remapping from the query.

Attention as a Relevance Signal

Attention itself provides a reliable relevance signal in long inputs. A sparse set of retrieval heads copies answers from long contexts ([24]), and QRHead ([30]) uses query-focused heads to re-rank documents. Attention sorting ([21]) re-orders documents by attention, while SnapKV ([14]) and GemFilter ([22]) use attention to select KV cache entries and input tokens, respectively. All of these methods use the signal to select or reorder content. QGPM instead uses a similar query-to-context signal to modulate RoPE positions, so every token stays in the input.

Preliminaries

Rotary Position Embedding

RoPE ([23]) splits each query and key vector of dimension dd into d/2d/2 two-dimensional planes. Plane mm rotates with angular frequency

ωm=θ−2m/d,m=0,1,…,d/2−1,(1)\omega_m = \theta^{-2m/d}, \qquad m = 0,1,\ldots,d/2 - 1, \tag*{(1)}

where θ\theta is a base constant, typically between 10410^4 and 10810^8. Plane m=0m = 0 rotates fastest. A token at position pp is rotated by R(p)=diag⁡(R2(ω0p),…,R2(ωd/2−1p))R(p) = \operatorname{diag}(R_2(\omega_0p),\ldots,R_2(\omega_{d/2-1}p)), where R2(ϕ)R_2(\phi) is a planar rotation by angle ϕ\phi. For a query vector qiq_i at position pip_i and a key vector kjk_j at position pjp_j, the identity R(pi)⊤R(pj)=R(pj−pi)R(p_i)^\top R(p_j) = R(p_j-p_i) gives

logit⁡(i,j)=qi⊤R(−Δp)kj=∑m=0d/2−1[amcos⁡(ωmΔp)+bmsin⁡(ωmΔp)],(2)\begin{aligned} \operatorname{logit}(i,j) &= q_i^\top R(-\Delta p) k_j \\ &= \sum_{m=0}^{d/2-1} \left[a_m \cos(\omega_m\Delta p) + b_m \sin(\omega_m\Delta p)\right], \tag*{(2)} \end{aligned}

where Δp=pi−pj\Delta p = p_i-p_j, am=qi,2mkj,2m+qi,2m+1kj,2m+1a_m = q_{i,2m}k_{j,2m} + q_{i,2m+1}k_{j,2m+1}, and bm=qi,2mkj,2m+1−qi,2m+1kj,2mb_m = q_{i,2m}k_{j,2m+1} - q_{i,2m+1}k_{j,2m}. The logit thus depends on position only through Δp\Delta p, and each plane adds one oscillating term.

Frequency Planes and Long-Range Decay

Plane mm completes one full turn every λm=2π/ωm\lambda_m = 2\pi/\omega_m positions, and λm\lambda_m grows exponentially with mm. At a distance Δp\Delta p, every plane with λm<Δp\lambda_m < \Delta p has turned at least once. The terms of these planes in Equation (2) no longer change consistently with distance and tend to average out across nearby keys. Only planes with λm>Δp\lambda_m > \Delta p keep a coherent distance signal, namely the planes above the boundary index

m(Δp)=dln⁡(Δp/2π)2ln⁡θ.(3)m(\Delta p) = \frac{d\ln(\Delta p/2\pi)}{2\ln\theta}. \tag*{(3)}

Since m(Δp)m(\Delta p) grows with Δp\Delta p, distant evidence is matched through fewer planes than nearby text. For example, with θ=107\theta= 10^7 and d=128d = 128, about 53 of 64 planes stay coherent at 100 tokens, but only about 30 at 32K tokens. We refer to this effect as the long-range decay of RoPE.

The same view suggests how to counter the decay. Shrinking the distance between the query and relevant evidence from Δp\Delta p to Δp′\Delta p' restores coherence on every plane with Δp′<λm<Δp\Delta p' < \lambda_m < \Delta p. Slow planes with wavelengths beyond the context length stay coherent at any distance and keep the coarse layout of the document. These planes need no change. Fast planes also encode local word order, so changes on fast planes should be limited to text that is irrelevant to the query.

Method: QGPM

QGPM runs the backbone twice on each input (Figure 1). A plain prefill first measures how strongly the query attends to each context sentence (§ 4.1). The resulting ranking is shaped into a broad and a sharp relevance map (§ 4.2). In the answering pass, each map sets plane-specific strides (§ 4.3), and the accumulated strides replace the original positions in RoPE (§ 4.4).

Figure 1. Overview of QGPM. (a) A plain prefill reads the attention from the query to each context sentence in a few selected heads. The fused sentence ranking is shaped into a broad relevance map and a sharp score map. Context before the first relevant sentence is protected and keeps a score of 1. (b) Each map sets a positional stride for every token on every frequency plane. Low-relevance tokens advance by less than one position on fast planes. Slow planes, relevant sentences, and the query keep unit stride. Cumulative sums give the remapped positions. (c) In the answering pass, the lower layers use the broad map and the upper layers use the sharp map. Decoding continues in unit steps from the last position of each map.

Relevance from Attention Heads

QGPM reads relevance from a small set H\mathcal{H} of attention heads. The heads are selected once per backbone by how well each head ranks annotated evidence on a held-out set (Appendix § 9). Let Q\mathcal{Q} denote the query rows, namely the question tokens and the last few prompt tokens (Appendix § 10). For head hh and context token jj, the relevance signal is the largest RoPE logit from any query row:

ℓh(j)=max⁡r∈Q(R(pr)qh,r)⊤(R(pj)kh,j)d.(4)\ell_h(j)=\max_{r\in\mathcal{Q}}\frac{\left(R(p_r)q_{h,r}\right)^{\top}\left(R(p_j)k_{h,j}\right)}{\sqrt{d}}. \tag*{(4)}

QGPM splits the context into sentences S1,…,SnS_1,\ldots,S_n and scores each sentence by the strongest token in the sentence, ℓh(Si)=max⁡j∈Siℓh(j)\ell_h(S_i)=\max_{j\in S_i}\ell_h(j). Each head is standardized over sentences, and the heads are averaged:

z~i=1∣H∣∑h∈Hℓh(Si)−μhσh,(5)\tilde{z}_i=\frac{1}{|\mathcal{H}|}\sum_{h\in\mathcal{H}}\frac{\ell_h(S_i)-\mu_h}{\sigma_h}, \tag*{(5)}

where μh\mu_h and σh\sigma_h are the mean and standard deviation of ℓh(Si)\ell_h(S_i) over sentences.

Broad and Sharp Relevance Maps

The scale of ziz_i varies across inputs, so QGPM keeps only the order of the sentences. Let ρ(i)\rho(i) be the rank of SiS_i by ziz_i, with ρ(i)=0\rho(i)=0 for the most relevant sentence. The score of SiS_i is

fi=σ((K−0.5−ρ(i))/T),si=slow+(1−slow)f~i,(6)\begin{aligned} f_i&=\sigma\left((K-0.5-\rho(i))/T\right),\\ s_i&=s_{\mathrm{low}}+(1-s_{\mathrm{low}})\tilde{f}_i, \tag*{(6)} \end{aligned}

where σ(⋅)\sigma(\cdot) is the sigmoid function and f~i\tilde{f}_i rescales fif_i to [0,1][0,1] over sentences. The top sentence thus receives a score of 1, and the bottom sentence receives slows_{\mathrm{low}}. The center KK sets how many sentences stay near 1, and the temperature TT sets how quickly the scores fall off. Setting T=0T=0 gives a hard top-KK cut.

QGPM shapes the same ranking twice. The broad map uses a larger KK and a softer TT, so several candidate sentences stay relevant. The sharp map uses a smaller KK and a harder TT, so only the top sentences stay relevant. Each token inherits the score of the enclosing sentence. Tokens outside the context, such as the instruction and the question, always receive a score of 1.

We call a sentence relevant if the sentence score is at least (1+slow)/2(1+s_{\mathrm{low}})/2. In each map, all context tokens before the first relevant sentence also receive a score of 1 and form a protected prefix. Under causal attention, the representation of the evidence is built from the text before the evidence. Keeping this prefix intact preserves the encoding of the evidence. Compression is then limited to the text between the evidence and the query, which is the span that sets the distance between the two.

Algorithm 1 QGPM inference
Require: context sentences S1,…,SnS_1,\ldots,S_n, heads H\mathcal{H}, query rows Q\mathcal{Q}
Require: hyperparameters (Kb,Tb)(K_b,T_b), (Ks,Ts)(K_s,T_s), slows_{\mathrm{low}}, LaL_a, τm\tau_m, ϵ\epsilon
Pass 1: relevance scoring
1: Run a plain prefill up to the deepest layer in H\mathcal{H}
2: Compute ℓh(j)\ell_h(j) and ziz_i by Eqs. (4) and (5)
3: Rank the sentences by ziz_i to obtain ρ(i)\rho(i)
4: for each map g∈{broad,sharp}g\in\{\mathrm{broad},\mathrm{sharp}\} do
5: Score sentences by Eq. (6) with (Kg,Tg)(K_g,T_g)
6: Set scores of non-context tokens and the protected prefix to 1
7: Compute strides by Eq. (7) with m∗=m(La)m^*=m(L_a)
8: Accumulate positions pm,j′p'_{m,j} by Eq. (8)
9: end for
Pass 2: answering
10: Prefill with the broad map below layer ℓs\ell_s and the sharp map from ℓs\ell_s on
11: Decode in unit steps from pm,N−1′p'_{m,N-1} of each map

Table Algorithm 1.

Frequency-Aware Strides

Following § 3, QGPM changes only the fast planes. Given an anchor length LaL_a proportional to the context length, the boundary plane is m∗=m(La)m^*=m(L_a) from Equation (3). Planes with m<m∗m<m^* complete at least one turn within LaL_a tokens and count as fast. The remaining planes count as slow. A sigmoid gate of width τm\tau_m separates the two groups. For a map with token scores s(j)s(j), the stride δm(j)\delta_m(j) of token jj on plane mm is

gm=σ((m−m∗)/τm),δm(j)=max⁡(s(j)+(1−s(j))gm,ϵ),(7)\begin{aligned} g_m&=\sigma\left((m-m^*)/\tau_m\right),\\ \delta_m(j)&=\max\left(s(j)+(1-s(j))g_m,\epsilon\right), \tag*{(7)} \end{aligned}

where ϵ\epsilon is a stride floor. On slow planes gm≈1g_m\approx1, so every token keeps unit stride. On fast planes gm≈0g_m\approx0, so the stride falls to about max⁡(s(j),ϵ)\max(s(j),\epsilon). Relevant sentences, whose scores are close to 1, thus keep near-unit stride on every plane, and only low-relevance tokens on fast planes are compressed.

The floor ϵ\epsilon keeps fast-plane increments close to the range seen in pretraining. With a small floor, about 1/ϵ1/\epsilon consecutive low-relevance tokens share one position on each fast plane, so the local order of these tokens is largely lost. We hypothesize that this blur de-emphasizes distractor text without deleting the text, while the slow planes still place every token correctly within the document.

Remapped Positions and Layer Schedule

Remapped positions accumulate the strides:

pm,j′=∑k=1jδm(k).(8)p'_{m,j} = \sum_{k=1}^{j} \delta_m(k). \tag*{(8)}

Every stride is at least ϵ>0\epsilon> 0, so positions stay strictly increasing. Token jj is rotated by ωmpm,j′\omega_m p'_{m,j} on plane mm, and the logit follows Equation (2) with Δp\Delta p replaced by the plane-specific offset Δpm,ij′=pm,i′−pm,j′\Delta p'_{m,ij} = p'_{m,i} - p'_{m,j}. Because positions accumulate, unit stride does not mean an unchanged position. On fast planes, every token after a compressed span moves toward the span, which brings relevant sentences closer to the query. Distances within the query are preserved exactly, and distances within relevant sentences are preserved approximately.

QGPM computes one set of positions for each map. Layers below ℓs=⌊nlayer/2⌋\ell_s = \lfloor n_{\mathrm{layer}}/2 \rfloor use the broad map, and the remaining layers use the sharp map. Each layer stores the KV cache in the coordinates of the corresponding map. We hypothesize a division of labor across depth. Lower layers build token representations from local context, so text blurred there cannot be recovered later. The broad map therefore keeps several candidate sentences intact in these layers. The selected heads lie mostly in the middle and upper layers, where the sharp map leaves fewer competing sentences near the query.

During decoding, the kk-th generated token takes position pm,N−1′+kp'_{m,N-1} + k on each plane of each map, where NN is the prompt length. Decoding thus continues in unit steps from the last prompt position. Compared with standard inference, QGPM adds one plain prefill that stops at the deepest selected layer. The per-token cost of decoding stays equal to that of standard RoPE (Appendix § 12). Algorithm 1 summarizes the full procedure, and Appendix § 11 describes all hyperparameters.

Experiments

We organize the experiments around three research questions.

  • RQ1: Does QGPM outperform existing inference-time methods across backbones, tasks, and context lengths?

  • RQ2: Is modulating RoPE positions more effective than selecting or reordering content by relevance?

  • RQ3: How close does the relevance read from attention heads come to gold evidence?

RQ2 and RQ3 examine the two design choices of QGPM. RQ2 concerns how the relevance is used, and RQ3 concerns where the relevance comes from.

Experimental Setup

Models. We evaluate four instruction-tuned backbones from three model families: Qwen2.5-7B-Instruct-1M ([27]), Qwen3-4B-Instruct-2507 ([25]), Ministral-8B-Instruct-2410 ([16]), and Granite-3.3-8B-Instruct ([10]). The RoPE bases of the four backbones range from 5×1065 \times10^6 to 10810^8. Except for Ministral-8B at 64K,3 all inputs fit within the native context window of each backbone, and we apply no length extrapolation.

Benchmarks and metrics. NoLiMa ([17]) tests retrieval when the evidence shares little lexical overlap with the question. We evaluate NoLiMa at 4K, 8K, 16K, and 32K tokens. From RULER ([8]), we use two question answering tasks, single-hop QA-1 and multi-hop QA-2, and two retrieval tasks, NIAH-MK and NIAH-MV. We evaluate the RULER tasks at 4K, 8K, 16K, 32K, and 64K tokens. NIAH-MK hides one target key among many similar keys, and NIAH-MV asks for all values of one key. We report accuracy (%) on every task. Correctness is judged by whether the response contains the required gold answers (Appendix § 10).

Baselines. We compare QGPM with the unmodified backbone, denoted Vanilla, and with four inference-time methods. LLM Lingua-2 ([19]) represents input compression. Ms-PoE ([32]), Self-Extend ([12]), and STRING ([2]) modify RoPE positions without using the query. All baselines use the default settings recommended in the original papers (Appendix § 10).

Implementation details. All methods use greedy decoding and run with PyTorch and HuggingFace Transformers on NVIDIA A100 GPUs. Appendix § 9 describes head selection, and Appendix § 11 describes the hyperparameters for each backbone and dataset.

Main Results (RQ1)

Overall performance. Table 1 (PDF p. 6) reports the accuracy averaged over context lengths. QGPM achieves the best five-task average on all four backbones and improves on Vanilla by 2.4 to 8.9 points. QGPM is also the only method whose five-task average never falls below Vanilla. Among the baselines, STRING exceeds Vanilla on three backbones and Self-Extend on one. Ms-PoE and LLMLingua-2 exceed Vanilla on none.

Qwen2.5-7BQwen3-4B
MethodNoLiMaQA-1QA-2MKMVAvg.NoLiMaQA-1QA-2MKMVAvg.
Vanilla28.579.657.094.383.568.616.177.556.699.696.069.2
LLMLingua-219.751.040.02.265.335.610.449.136.62.075.934.8
Ms-PoE32.977.252.036.868.553.516.675.455.388.284.764.0
Self-Extend29.579.357.578.882.665.518.176.557.599.892.268.8
STRING33.179.457.892.376.667.822.978.860.099.393.070.8
QGPM46.180.458.099.493.475.425.180.156.299.696.871.6
Ministral-8B†Granite-3.3-8B
MethodNoLiMaQA-1QA-2MKMVAvg.NoLiMaQA-1QA-2MKMVAvg.
Vanilla11.573.559.089.491.465.018.678.662.766.071.159.4
LLMLingua-26.248.240.83.678.035.314.150.143.10.248.431.2
Ms-PoE9.068.553.032.481.248.818.574.160.261.514.145.7
Self-Extend11.172.858.894.692.866.016.677.361.956.871.856.9
STRING17.173.359.194.790.867.025.978.862.973.575.863.4
QGPM24.173.659.095.293.469.135.678.863.185.178.968.3

Table 1 (PDF p. 6). Accuracy (%) averaged over context lengths, from 4K to 32K for NoLiMa and from 4K to 64K for RULER. MK and MV denote NIAH-MK and NIAH-MV, and Avg. is the mean over the five tasks. Bold marks the best result for each backbone. †Includes 64K, which exceeds the configured window of Ministral-8B. Appendix § 13 gives the results at each length.

Retrieval tasks. The largest gains appear on retrieval. On NoLiMa, QGPM is the best method at every context length on every backbone. The average gain over Vanilla ranges from 9.0 to 17.6 points. On NIAH, the gains are large where the backbone degrades. QGPM raises NIAH-MK on Granite from 66.0 to 85.1 and NIAH-MV on Qwen2.5 from 83.5 to 93.4. Qwen3 already reaches at least 96.0 on both NIAH tasks, and QGPM stays within one point of Vanilla on this backbone.

Question answering. On RULER QA, QGPM performs on par with Vanilla. The average difference ranges from −0.4 to +2.6 points, and the only drop occurs on QA-2 for Qwen3, where STRING performs best. § 5.4 shows that even annotated evidence helps little on QA-1.

Robustness across settings. Table 2 compares QGPM with each method in every setting of backbone, task, and context length. QGPM wins in at least 71 of the 96 settings against every method. QGPM never loses on NoLiMa and never loses to STRING on NIAH. Most losses occur on QA. Against Vanilla, QGPM loses in 17 settings by at most 3.2 points, and 14 of these settings are QA.

QGPM vs.NoLiMaNIAHQAAll
Vanilla16/0/031/6/324/2/1471/8/17
LLMLingua-216/0/040/0/040/0/096/0/0
Ms-PoE16/0/040/0/038/1/194/1/1
Self-Extend16/0/034/3/326/2/1276/5/15
STRING16/0/038/2/022/2/1676/4/16

Table 2. Wins, ties, and losses (W/T/L) of QGPM against each method over settings of backbone, task, and context length. The four columns cover 16, 40, 40, and 96 settings.

Baseline behavior. LLMLingua-2 falls below Vanilla on every task and nearly fails on NIAH-MK, with averages from 0.2 to 3.6. Deleting tokens likely breaks the exact keys that NIAH-MK must match. The query-agnostic RoPE methods are unstable on retrieval. Ms-PoE averages 14.1 on NIAH-MV for Granite, against 71.1 for Vanilla.

Self-Extend drops to 58.0 on NIAH-MK at 64K for Qwen2.5, where Vanilla keeps 91.4. A fixed remapping treats evidence and distractors alike, which may explain these drops. STRING is the strongest baseline, yet the five-task average of STRING still falls below Vanilla on Qwen2.5.

Effect of context length. Figure 2 shows the accuracy of every task at each context length. On NoLiMa, QGPM stays above every baseline at every length. The margin over the strongest baseline peaks at 4K or 8K. At 32K, the margin shrinks to between 0.3 and 2.4 points on Qwen2.5, Qwen3, and Ministral. Granite still keeps a margin of 8.9 points. § 5.4 examines the smaller margin at 32K. On NIAH-MK, the gain over Vanilla grows with length on backbones that degrade. On Granite, the gain rises from 2.8 points at 4K to 41.2 points at 64K. On NIAH-MV, QGPM improves on Vanilla at every length on Qwen2.5, by 7.8 to 14.2 points. On QA, QGPM tracks Vanilla closely, and small gains appear mainly at 64K, where QGPM exceeds Vanilla in seven of eight settings.

Figure 2. Accuracy (%) of all five tasks at each context length. Gray lines show LLMLingua-2, Ms-PoE, and Self-Extend. The shaded region marks 64K for Ministral-8B, which exceeds the configured window of this backbone.

Modulation versus Input Editing (RQ2)

We compare QGPM with two variants that edit the input according to relevance. Both variants split the context into chunks and score each chunk against the question with Qwen3-Reranker-4B (Zhang et al., 2025b). Top-K keeps the five highest-scoring chunks and drops the rest. Reorder keeps all chunks and sorts the chunks by relevance, so that the most relevant chunk lies closest to the query. QGPM instead keeps every token in place and needs no external ranker. The tokenizer of the reranker is not fully compatible with those of Ministral-8B and Granite-3.3-8B, so we run this comparison on Qwen2.5-7B only.

Table 3 reports the averages, and Figure 3 shows each context length. On NoLiMa, both variants outperform Vanilla and STRING, with 41.1 points for Top-K and 39.9 points for Reorder. Query relevance thus helps even without positional modulation. QGPM reaches 46.1 points and stays above both variants at every length, by 0.4 to 10.2 points. On NIAH-MV, QGPM also leads with 93.4 points, against 92.2 for Reorder and 91.1 for Top-K. The margin over Top-K reaches 5.2 points at 64K, likely because the five kept chunks do not always cover every value of the key. Top-K loses any evidence that the ranker misses, and Reorder breaks the original order of the document. QGPM avoids both risks because every token stays in the input. On QA-1, the three methods perform on par, and Top-K is 0.2 points above QGPM. Unlike NoLiMa, QA-1 does not minimize the lexical overlap between question and evidence, so the ranker likely keeps the evidence and deletion costs little.

Qwen2.5-7BNoLiMa
MethodNoLiMaQA-1MVMinistralGranite
Vanilla28.579.683.511.518.6
STRING33.179.476.617.125.9
Top-K41.180.691.1n/an/a
Reorder39.980.192.2n/an/a
QGPM46.180.493.424.135.6
Oracle47.381.693.9n/an/a

Table 3. Average accuracy (%) of QGPM, input editing, and Oracle. MV denotes NIAH-MV, and n/a marks settings that are not run. Top-K and Reorder rely on Qwen3-Reranker-4B, whose tokenizer is not fully compatible with those of Ministral-8B and Granite-3.3-8B. Oracle (gray) uses evidence positions marked by hand, which are costly to obtain for every backbone. Bold marks the best result other than Oracle.

Figure 3. Accuracy (%) of QGPM, input editing, and Oracle at each context length on Qwen2.5-7B. Top-K and Reorder share one color as the two input editing variants. Oracle (dashed) uses annotated evidence and serves as a reference.

Quality of Head Relevance (RQ3)

Oracle replaces the relevance read from attention heads with evidence positions marked by hand from the benchmark labels. All other steps stay unchanged, including the two maps, the protected prefix, and the frequency gate. Any gap between QGPM and Oracle therefore comes from the relevance signal alone. Marking the evidence by hand is costly, so we run Oracle on Qwen2.5-7B only.

On all three tasks of Qwen2.5, QGPM comes within 1.2 points of Oracle (Table 3). The relevance read from attention heads is thus close to annotated evidence. QGPM even exceeds Oracle at some lengths, such as 8K on NoLiMa and 64K on NIAH-MV (Figure 3). Oracle is therefore a reference rather than a strict upper bound.

On NoLiMa and QA-1, the remaining gap concentrates at the longest length. On NoLiMa, Oracle improves on Vanilla by 17.6 to 21.2 points at every length. Moving correctly identified evidence closer to the query thus helps even at 32K. QGPM trails Oracle by at most 2.0 points up to 16K, but the gap grows to 5.7 points at 32K. At this length, the relevance signal rather than the positional modulation limits QGPM. The smaller NoLiMa margin at 32K in § 5.2 likely has the same cause. On QA-1, Oracle gains only 2.0 points over Vanilla on average, so position is not the main bottleneck for this task. The exception is again the longest length, where Oracle gains 8.0 points at 64K and QGPM gains 3.2 points.

Conclusion

We presented QGPM, a training-free method that reshapes RoPE positions using query relevance read from the model’s own attention heads. QGPM compresses low-relevance text only on fast-rotating planes and guides the lower and upper layers with a broad and a sharp relevance map. Every token stays in the input, and no model weight changes. Across four backbones, QGPM achieves the best average accuracy, with the largest gains on retrieval tasks. On question answering, where position matters less, QGPM matches the backbone. Modulating positions also outperforms input editing on retrieval, and the head relevance comes close to annotated evidence. The remaining gap appears at the longest contexts, so better relevance estimation for very long inputs is a promising next step.

Limitations

Access to model internals. QGPM reads the query and key vectors of selected heads and replaces the RoPE tables of the backbone. The method therefore applies to open-weight models but not to models served only through an API. Most open long-context models use RoPE, so this requirement still covers a broad range of backbones.

Extra prefill. QGPM adds one plain prefill that stops at the deepest selected layer, which increases the time to the first token. The decoding cost and the size of the KV cache stay unchanged. The context keys in this first pass do not depend on the question. Several questions over the same document could therefore share a cached prefix. A shallower head set would also shorten the first pass.

Ethics Statement

This work proposes a training-free inference method that changes how a long-context language model encodes positions. QGPM introduces no new model, training data, or alignment procedure, so the safety and factuality of each backbone remain those of the released model. All experiments use public benchmarks, NoLiMa and RULER, which consist of synthetic or existing question answering instances without personal information. The backbones and the reranker used by the input editing baselines are publicly released models. QGPM requires no fine-tuning, so the computational cost of this work is modest. We are not aware of risks specific to QGPM beyond those of the underlying models.

Head Selection

QGPM reads relevance from a fixed set H\mathcal{H} of attention heads, selected once for each backbone. Every (layer, head) pair of the backbone is a candidate. The calibration set is a fixed subset of NoLiMa.

BackboneLayersHeadsθ\thetaWindow
Qwen2.5-7B2828 / 410710^{7}1,010,000
Qwen3-4B3632 / 85×1065 \times10^{6}262,144
Ministral-8B3632 / 810810^{8}32,768
Granite-3.3-8B4032 / 810710^{7}131,072

caption: Backbone configurations from the HuggingFace releases. Heads lists query heads and key-value heads. Window is the configured maximum number of positions.

:::

samples whose evidence spans are known. The calibration samples are disjoint from the evaluation samples.

For each calibration sample and candidate head, we compute the sentence signal ℓh(Si)\ell_h(S_i) of Equation (4) with the same query rows as at inference. The context sentences are then ranked by this signal. The quality of a head is the mean reciprocal rank of the evidence sentence over the calibration samples, with ties counted at the expected rank. The ∣H∣|\mathcal{H}| heads with the highest quality form the head set. Calibration only selects heads and updates no weight.

Three details keep the signal faithful to the attention of the model. Under grouped-query attention, head hh reads the key head of the same group. On Qwen3, the vectors are taken after the query and key normalization of the model. Tokens that contain a newline character are excluded from the signal, because the line break around a NoLiMa needle is a formatting artifact that some shallow heads attend to.

The selected heads lie mostly in the middle and upper layers of each backbone, which motivates the layer schedule in § 4.4.

Implementation Details

Backbones. Table 4 lists the configuration of the four backbones. All four use a head dimension of d=128d = 128 and unscaled RoPE, so the boundary plane m∗m^{*} follows Equation (3) directly.

Prompts and query rows. Each prompt contains a system prompt, a task instruction, the context, the question, and an answer prefix. Only the context is scored, and every other token receives a score of 1. On NoLiMa, the context is a book excerpt with one inserted needle. On RULER, the context is the haystack, and the question is the text from the last Question: onward. The query rows QQ are the question tokens together with the last few tokens of the prompt. All methods read exactly the same prompts and needle placements.

Sentence segmentation. The context is split into sentences after “.”, “!”, and “?” followed by whitespace, with common abbreviations protected. A token belongs to the sentence that contains the first character of the token, and the whitespace between two sentences belongs to the next sentence. Tokens are aligned to sentences through the character offsets of the backbone tokenizer.

Position tables. The remapped positions and the cosine and sine tables are computed in FP32 and then cast to the model precision. QGPM builds the tables once per prefill for each map. A forward pre-hook passes the tables of the assigned map to the attention of each decoder layer, so the KV cache of each layer stays in the coordinates of that map. With every score equal to 1, the patched model reproduces the unmodified backbone, which we verify for each backbone.

Decoding and hardware. All methods use greedy decoding with the same limit on new tokens for each dataset. The experiments run with PyTorch and HuggingFace Transformers on NVIDIA A100 GPUs.

Metrics. Each response is scored as correct or incorrect, and accuracy is the percentage of correct responses. On NoLiMa, a response is correct if the response contains the gold answer. On RULER QA, a response must contain the gold answer, and a question with several required answers needs every answer. NIAH-MK requires the value of the target key, and NIAH-MV requires every value of the key. Each RULER task has 500 samples at each length.

Baselines. All baselines use the default settings recommended in the original papers.

Input editing. Top-K and Reorder split the context into sentence-aligned chunks and score each chunk against the question with Qwen3-Reranker-4B. Top-K keeps the five highest-scoring chunks and drops the rest. Reorder keeps all chunks and sorts the chunks by score, so that the most relevant chunk ends right before the question. Both variants run with standard RoPE. The tokenizer of the reranker is not fully compatible with those of Ministral-8B and Granite-3.3-8B, so both variants run on Qwen2.5-7B only.

Oracle. Oracle marks the evidence in each context by hand according to the benchmark labels, such as the needle in NoLiMa and the gold documents and inserted values in RULER. Oracle marks the gold spans at the token level, whereas the attention scorer of QGPM ranks whole sentences. The two maps, the protected prefix, the frequency gate, and the stride floor stay the same as in QGPM. Because these settings act on both, Oracle is a reference rather than a strict upper bound. Marking evidence by hand is costly, so we run Oracle on Qwen2.5-7B only.

Hyperparameters

QGPM sets the following hyperparameters for each backbone and dataset. Two settings are shared by all runs. The split layer is ℓs=⌊nlayer/2⌋\ell_s = \lfloor n_{\mathrm{layer}}/2 \rfloor, and the score of the top sentence is 1. The remaining settings play the following roles.

  • The broad map uses (Kb,Tb)(K_b, T_b) and the sharp map uses (Ks,Ts)(K_s, T_s), with Kb>KsK_b > K_s and Tb>TsT_b > T_s. KsK_s is at least the number of evidence spans that a task requires.

  • The minimum score slows_{\mathrm{low}} and the stride floor ϵ\epsilon set the strength of compression. A low-relevance token advances by max⁡(slow,ϵ)\max(s_{\mathrm{low}}, \epsilon) on a fast plane.

  • The anchor length is La=c⋅LL_a = c \cdot L, where LL is the context length and cc is set for each dataset. For the same LaL_a, a larger θ\theta gives a smaller m∗m^*, so fewer planes are compressed.

  • The gate width τm\tau_m is a small value in units of plane index.

On NIAH-MK and NIAH-MV, KsK_s is chosen on development samples for each task and fixed before the evaluation samples are run.

Computational Cost

QGPM changes the cost of inference in three places.

  • The first pass is a plain prefill that stops right

after the key projection of the deepest layer in H\mathcal{H}. This pass costs at most one plain prefill and becomes cheaper with a shallower head set.

  • The answering pass is a full prefill with the remapped tables. The attention kernels stay unchanged, so this pass costs the same as a plain prefill. Building the position tables takes O(Nd/2)O(Nd/2) time and memory for each map.

  • Decoding keeps the cost per step and the KV cache size of standard inference. Each step adds one vector of d/2d/2 angles for each map.

Full Results

Tables Table 5 and Table 6 report the accuracy of every method at each context length, and Table 7 reports the comparison with input editing and Oracle at each length. The averages over lengths match Table 1 (PDF p. 6) and Table 3.

TaskMethodQwen2.5-7B 4K8K16K32K64KAvg.Qwen3-4B 4K8K16K32K64KAvg.
NoLiMaVanilla36.732.927.217.2–28.530.619.410.34.0–16.1
NoLiMaLLMLingua-229.818.517.413.2–19.723.210.75.02.8–10.4
NoLiMaMs-PoE41.236.632.221.7–32.932.219.99.54.6–16.6
NoLiMaSelf-Extend36.231.727.122.8–29.530.719.913.98.0–18.1
NoLiMaSTRING38.034.433.126.7–33.135.629.518.77.9–22.9
NoLiMaQGPM56.554.943.829.1–46.138.033.220.98.3–25.1
QA-1Vanilla86.482.480.877.071.479.685.280.077.075.469.877.5
QA-1LLMLingua-256.252.850.249.446.651.053.249.850.248.443.849.1
QA-1Ms-PoE83.079.077.475.871.077.284.277.876.074.664.475.4
QA-1Self-Extend86.479.879.677.473.279.385.279.875.675.666.276.5
QA-1STRING85.082.077.278.274.479.484.280.475.677.476.678.8
QA-1QGPM87.480.881.677.474.680.484.680.481.878.475.480.1
QA-2Vanilla64.058.856.855.650.057.062.060.257.052.651.456.6
QA-2LLMLingua-241.038.841.640.837.640.037.038.037.235.235.636.6
QA-2Ms-PoE58.454.851.451.843.852.058.061.055.052.650.055.3
QA-2Self-Extend64.059.858.454.251.257.562.059.459.453.653.057.5
QA-2STRING61.060.459.056.252.257.860.063.462.057.257.660.0
QA-2QGPM61.460.859.056.652.458.060.659.256.652.652.256.2
NIAH-MKVanilla100.097.292.890.291.494.399.899.8100.099.499.299.6
NIAH-MKLLMLingua-21.82.22.42.42.22.22.22.02.01.62.02.0
NIAH-MKMs-PoE50.851.639.027.415.236.895.098.494.888.065.088.2
NIAH-MKSelf-Extend100.096.078.661.258.078.899.8100.099.899.899.899.8
NIAH-MKSTRING98.496.295.084.887.292.399.499.699.299.698.699.3
NIAH-MKQGPM100.099.898.699.299.299.499.899.8100.099.698.899.6
NIAH-MVVanilla85.082.482.286.881.283.593.895.492.299.299.696.0
NIAH-MVLLMLingua-269.865.866.863.660.665.376.676.479.074.872.675.9
NIAH-MVMs-PoE79.256.262.871.073.468.579.285.281.486.091.884.7
NIAH-MVSelf-Extend85.282.484.077.284.282.694.091.688.893.293.692.2
NIAH-MVSTRING79.877.276.475.274.276.694.291.689.692.896.693.0
NIAH-MVQGPM93.490.292.095.895.493.496.496.293.498.899.496.8

Table 5. Accuracy (%) at each context length on Qwen2.5-7B and Qwen3-4B. NoLiMa is evaluated up to 32K, and Avg. is the mean over the evaluated lengths. Bold marks the best result for each backbone, task, and length.

TaskMethodMinistral-8B 4K8K16K32K64K†Avg.Granite-3.3-8B 4K8K16K32K64KAvg.
NoLiMaVanilla25.813.04.52.6–11.531.821.712.97.9–18.6
LLMLingua-215.96.11.90.8–6.222.816.910.36.5–14.1
Ms-PoE19.89.63.62.9–9.031.619.814.48.3–18.5
Self-Extend25.811.54.42.5–11.129.018.712.06.5–16.6
STRING32.919.810.65.1–17.140.330.020.812.4–25.9
QGPM46.332.612.35.4–24.152.539.828.921.3–35.6
QA-1Vanilla80.077.075.072.063.673.583.681.078.679.070.678.6
LLMLingua-250.851.046.846.845.448.253.848.851.249.847.050.1
Ms-PoE77.273.069.266.256.868.582.077.474.874.461.874.1
Self-Extend80.076.274.071.862.072.885.078.477.076.070.077.3
STRING79.676.075.672.263.073.382.080.081.080.270.878.8
QGPM80.475.476.871.863.873.687.277.881.476.871.078.8
QA-2Vanilla66.062.861.459.445.659.066.464.263.461.857.662.7
LLMLingua-241.242.643.637.638.840.845.042.644.841.841.443.1
Ms-PoE58.660.655.250.040.653.065.062.860.262.051.060.2
Self-Extend63.863.661.459.246.258.867.265.460.860.455.661.9
STRING65.062.862.259.246.459.167.466.264.461.255.462.9
QGPM64.462.460.859.448.259.067.667.263.662.654.663.1
NIAH-MKVanilla100.0100.099.698.649.089.495.489.479.651.214.266.0
LLMLingua-23.23.43.63.64.23.60.20.40.00.40.00.2
Ms-PoE50.250.430.620.410.632.489.285.474.842.016.261.5
Self-Extend100.099.899.698.275.294.695.287.461.429.011.256.8
STRING100.099.899.898.275.894.792.089.883.263.639.073.5
QGPM100.0100.0100.099.676.495.298.298.296.677.055.485.1
NIAH-MVVanilla98.498.496.693.669.891.482.674.268.868.861.071.1
LLMLingua-283.282.282.275.267.478.064.456.252.446.622.648.4
Ms-PoE81.691.687.880.664.281.220.49.823.613.03.814.1
Self-Extend98.496.896.694.677.892.883.078.275.064.858.071.8
STRING95.892.696.494.075.290.884.276.280.668.869.475.8
QGPM98.898.696.894.878.293.484.479.081.280.070.078.9

Table 6. Accuracy (%) at each context length on Ministral-8B and Granite-3.3-8B. NoLiMa is evaluated up to 32K, and Avg. is the mean over the evaluated lengths. Bold marks the best result for each backbone, task, and length. †Exceeds the configured window of Ministral-8B.

TaskMethod4K8K16K32K64KAvg.
NoLiMaVanilla36.732.927.217.2–28.5
STRING38.034.433.126.7–33.1
Top-K47.950.737.228.7–41.1
Reorder50.444.738.226.4–39.9
QGPM56.554.943.829.1–46.1
Oracle57.950.845.834.8–47.3
QA-1Vanilla86.482.480.877.071.479.6
STRING85.082.077.278.274.479.4
Top-K86.282.281.079.674.080.6
Reorder84.881.281.279.274.080.1
QGPM87.480.881.677.474.680.4
Oracle85.881.683.078.079.481.6
NIAH-MVVanilla85.082.482.286.881.283.5
STRING79.877.276.475.274.276.6
Top-K92.290.290.692.490.291.1
Reorder92.890.291.293.293.892.2
QGPM93.490.292.095.895.493.4
Oracle93.692.692.896.094.693.9

Table 7. Accuracy (%) of QGPM, input editing, and Oracle at each context length on Qwen2.5-7B. Oracle (gray) is shown for reference. Bold marks the best result other than Oracle.

References

  1. [1]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.arxiv.org/abs/2303.08774
  2. [2]Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shanshan Gong, Yao Luo, Jingjing Xu, and Linpeng Kong. 2025. Who does the effective context length of LLMs fall short? In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.18745
  3. [3]Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595.arxiv.org/abs/2306.15595
  4. [4]Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. LongRoPE: Extending LLM context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753.arxiv.org/abs/2402.13753
  5. [5]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.arxiv.org/abs/2407.21783
  6. [7]Zhiyuan He, Huiqiang Jiang, Zilong Wang, Yuqing Yang, Luna K. Qiu, and Lili Qiu. 2024. Position engineering: Boosting large language models through positional information manipulation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7333–7345.arxiv.org/abs/2404.11216
  7. [8]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024a. RULER: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654.arxiv.org/abs/2404.06654
  8. [9]Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T. Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. 2024b. Found in the middle: Calibrating positional attention bias improves long context utilization. In Findings of the Association for Computational Linguistics: ACL 2024.arxiv.org/abs/2406.16008
  9. [10]IBM. 2025. IBM Granite 3.3: Speech recognition, refined reasoning, and RAG LoRAs. Blog post, April 16, 2025.
  10. [11]Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1658–1677, Bangkok, Thailand. Association for Computational Linguistics.
  11. [12]Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. LLM maybe LongLM: Self-extend LLM context window without tuning. arXiv preprint arXiv:2401.01325.arxiv.org/abs/2401.01325
  12. [13]Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353. Singapore. Association for Computational Linguistics.
  13. [14]Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM knows what you are looking for before generation. In Advances in Neural Information Processing Systems, volume 37.arxiv.org/abs/2404.14469
  14. [15]Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran-jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.arxiv.org/abs/2307.03172
  15. [16]Mistral AI. 2024. Un Mistral, des Mistraux. Blog post, October 16, 2024.
  16. [17]Ali Modarressi, Hanieh Dalmalseleh, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schütze. 2025. NoLiMa: Long-context evaluation beyond literal matching. arXiv preprint arXiv:2502.05167.arxiv.org/abs/2502.05167
  17. [18]Anwesan Pal, Karen Hovsepian, Tinghao Guo, Mengnan Zhao, Somendra Tripathi, Nikos Kanakaris, George Mihaila, and Sumit Nigam. 2025. Tagging-augmented generation: Assisting language models in finding intricate knowledge in long contexts. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2209–2220. Association for Computational Linguistics.arxiv.org/abs/2510.22956
  18. [19]Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Association for Computational Linguistics: ACL 2024, pages 963–981, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2403.12968
  19. [20]Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2309.00071
  20. [21]Alexander Peysakhovich and Adam Lerer. 2023. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427.arxiv.org/abs/2310.01427
  21. [22]Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, and Shafiq Joty. 2026. Discovering the gems in early layers: Accelerating long-context LLMs with 1000x input token reduction. In Findings of the Association for Computational Linguistics: ACL 2026.
  22. [23]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.arxiv.org/abs/2104.09864
  23. [24]Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2025. Retrieval head mechanisms explain long-context factuality. In The Thirteenth International Conference on Learning Representations.
  24. [25]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenghen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.arxiv.org/abs/2505.09388
  25. [26]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, and 1 others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.arxiv.org/abs/2412.15115
  26. [27]An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, and 9 others. 2025b. Qwen2.5-1M technical report. arXiv preprint arXiv:2501.15383.arxiv.org/abs/2501.15383
  27. [28]Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. 2024a. Tell your model where to attend: Post-hoc attention steering for LLMs. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2311.02262
  28. [29]Qingru Zhang, Xiaodong Yu, Chandan Singh, Xiaodong Liu, Liyuan Liu, Jianfeng Gao, Tuo Zhao, Dan Roth, and Hao Cheng. 2024b. Model tells itself where to attend: Faithfulness meets automatic attention steering. arXiv preprint arXiv:2409.10790.arxiv.org/abs/2409.10790
  29. [30]Wuwei Zhang, Fangcong Yin, Howard Yen, Danqi Chen, and Xi Ye. 2025a. Query-focused retrieval heads improve long-context reasoning and re-ranking. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.arxiv.org/abs/2506.09944
  30. [32]Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024c. Found in the middle: How language models use long contexts better via plug-and-play positional encoding. In Advances in Neural Information Processing Systems, volume 37, pages 60755–60775.arxiv.org/abs/2403.04797

Paper details

vlnc:W_7gmpvgcv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 13

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents