In this paper
AbstractIntroductionRelated workEvaluationResultsDiscussion and limitationsConclusionProtocol detailsReader evaluationEvidence preservation and compressionAnalysis of evidence preservationRecall-compression tradeoffsReferencesWorksComputer Science
CoRA-Bench: Evaluating Evidence Preservation by Context Selectors Under Token Budgets
Abstract
Context selection can reduce input length during long-input inference but can remove evidence required to answer a query. We introduce CoRA-Bench, a benchmark for evaluating evidence preservation under shared token budgets. We compare head and tail truncation, BM25, dense retrieval, dense retrieval with reranking, LLMLingua-2, and Provence across BABILong, LOONG Financial Spotlight, a custom NoLiMa diagnostic, and LongMemEval. Evidence-preservation results reveal task-dependent tradeoffs: BM25 achieves 0.921 supporting-fact recall on BABILong at 128K, dense retrieval and dense reranking each achieve 0.908 needle recall on NoLiMa at 64K, and dense retrieval achieves 0.993 answer-bearing-turn recall on LongMemEval at 64K. Evidence, token retention, and complementary reader evaluations use the same selected contexts. Reader evaluations use dataset-native answer metrics and three runs on the same questions, with paired bootstrap confidence intervals. At a 32K budget, BM25 achieves 46.20% BABILong answer accuracy and dense retrieval achieves 71.13% LongMemEval accuracy. The results show that method rankings vary across evidence types and between evidence recall and downstream accuracy.
Introduction
Performance on long-context tasks depends on evidence position, even within supported context windows [18]. Processing full reports or conversation histories also increases input length when much of the content may be irrelevant. Context selection reduces this input but risks discarding supporting evidence.
We compare selectors under shared token budgets, measuring evidence recall, token retention, and downstream answer accuracy on the same reader-visible contexts.
Our contribution is a common implementation and evaluation protocol for six practical selectors across four complementary long context settings. BABILong provides explicit supporting facts in long book text [13]. LOONG contributes evidence localization in multi-document financial contexts [34]. NoLiMa tests retrieval with minimal literal overlap [20]. LongMemEval supplies answer-bearing turns in multi-session dialogue [Wu et al., 2025]. Across these settings, CoRA-Bench provides common token accounting and budget enforcement, occurrence-aware evidence tracing, and direct comparison at 32K, 64K, and, where source length permits, 128K tokens.
We evaluate downstream answers using Gemini 2.5 Flash-Lite with dataset-native scoring and paired bootstrap confidence intervals (Appendix § 8).
Related work
Long context benchmarks cover position sensitivity, broad task collections, controlled synthetic tests, multi-document reasoning, latent retrieval, and conversational memory [18, 1, 2, 9, 13, 34, 20, 19, 32] (Li et al. 2024; Wu et al. 2025). This diversity motivates evaluating selectors across evidence regimes instead of treating long context retrieval as one task.
Context reduction spans lexical search and reproducible retrieval toolkits [29, 17], dense passage retrieval [12], retrieval augmented generation and neural reranking [15, 22, 38], and prompt or retrieved-context compression [10, 11, 26, 37, 4]. CoRA-Bench contributes the controlled evaluation layer needed to compare these heterogeneous approaches under common token budgets and evidence definitions.
Learned representation compression includes gist tokens [21], AutoCompressors [3], and the In-context Autoencoder [6]. Perception Compressor combines question-guided retrieval with token pruning [30]; COMI allocates compression across groups and merges tokens using marginal information gain [31]. These approaches broaden the compression landscape. Our evaluation focuses on selectors that emit text for a shared reader.
Architectural and systems advances have improved the feasibility of long-context inference [33, 5, 14, 7]. We examine evidence preservation under token budgets using a fixed downstream reader.
Evaluation
Figure 1 summarizes the protocol. Given a query , source context , budget , and selector , we construct
Figure 1. CoRA-Bench: shared token budgets and dataset-specific evidence scoring across four retrieval regimes. Evidence and reader evaluation use the same final context, with overlapping retrieval spans merged; reader inference is repeated with 3 seeds. The LM-filter bypass denotes Threshold , uncapped; its ranked, capped arms follow budget enforcement without a score threshold.
| Dataset | Selection stress | N | Source tokens | Configuration |
| BABILong [13] | Sparse supporting facts | 500 | 217.7K to 253.7K | qa1 to qa5, 100 examples each, generated at 256K with gold support occurrences |
| LOONG [34] | Financial line localization | 75 | 45.2K to 326.4K | English Financial Spotlight, level 1; 2 to 9 documents; 74/69/50 evidence-evaluable at 32K/64K/128K |
| NoLiMa [20] | Minimal lexical overlap | 870 | 128.57K to 128.74K | 58 question and needle pairs, 3 depths, 5 fixed haystacks; repeated measures |
| LongMemEval [Wu et al., 2025] | Conversational retrieval | 500 | 98.4K to 107.1K | Cleaned small split; 470 answerable and 30 abstention examples |
Table 1. Evaluated data. Lengths use cl100k_base. NoLiMa is a custom repeated-measures diagnostic, not the official benchmark aggregate. LongMemEval is treated as static context selection, not online memory management.
Here, denotes the cl100k_base tokenizer from tiktoken [24]. We use it for chunking and token accounting, although it may differ from model-native tokenizers. The 32K, 64K, and 128K budgets (K = 1,000) cap selected context, excluding query and instruction overhead. We report macro-averaged compression ratios and question-level evidence recall over evidence-evaluable examples. Extractive methods use source-occurrence provenance and text-modifying methods require verbatim evidence survival. Full context has a recall of one by construction.
Budgets and cohorts. BABILong and NoLiMa use the same examples at every reported budget. LongMemEval sources are shorter than 128K, so only 32K and 64K are informative. For LOONG, we follow the existing run rule that evaluates only sources longer than the budget, giving 75, 70, and 51 examples for reader accuracy and token ratios at 32K, 64K, and 128K. Evidence-evaluable counts are 74, 69, and 50; the missing candidate annotation is excluded from recall, not treated as a selection failure. LOONG values should therefore be read within a budget; changes across budgets also reflect cohort changes.
Methods. All retrievers use 512 token chunks with 64 token overlap and return selected chunks in source order. Head and tail truncation is a query independent positional baseline. BM25 uses Okapi scoring [29]. Dense retrieval uses Qwen3-Embedding-0.6B, and Dense plus Rerank applies Qwen3-Reranker-0.6B to candidate pools of up to 100, 133, and 258 chunks at 32K, 64K, and 128K [39]. LLMlingua-2 performs query-agnostic token compression [26]. Provenance reranks and prunes sentences within chunks at a threshold 0.1, without forced title retention [4]. The neural selectors use PyTorch and the Transformers ecosystem [28, 35]. BM25, dense retrieval, and dense reranking merge overlapping source-token spans, accept a strict prefix of ranked chunks under the output budget, and restore source order. Reranking can underfill the budget when its candidate pool is exhausted. Evidence, token, and reader scores use these same final outputs. Final emitted text is re-counted under and must satisfy . Text-modifying methods are evaluated on their actual emitted text. Financial tables retain their materialized source representation, without table-specific preservation or whitespace normalization.
Dataset specific evidence. BABILong uses occurrence-aware recall of gold supporting facts. LOONG uses automatic candidate-line survival, unless independently reviewed support spans are supplied. These candidate lines are not human-annotated gold evidence. This diagnostic is distinct from LOONG’s published document coverage Recall@ and from its judge based answer score. NoLiMa uses verbatim survival of the inserted needle. LongMemEval uses recall of annotated answer bearing turns; its 30 abstention questions have no positive evidence and are excluded from evidence recall.
Results
Evidence preservation. Rankings depend on the retrieval regime. BM25 leads BABILong supporting-fact recall at all three budgets (0.838, 0.868, 0.921). At 64K, dense retrieval and dense reranking each achieve 0.908 NoLiMa needle recall versus 0.467 for BM25. Dense reranking retains 0.979 and 0.991 of LongMemEval answer-bearing turns at 32K and 64K; dense retrieval reaches 0.993 at 64K. These findings favor lexical retrieval under explicit surface cues and semantic retrieval under limited lexical overlap (Appendix § 9).
Reader accuracy. Table 2 reports three-run averages at 32K. On BABILong, BM25 reaches 46.20% and Provence 48.13%, versus 31.13% for full context. Paired gains are 15.07 percentage points for BM25 (95% CI: 10.87 to 19.60) and 17.00 for Provence (12.47 to 21.47). Provence minus BM25 is 1.93 points (−1.73 to 5.73), which does not establish superiority. These contrasts are exploratory and unadjusted for multiple testing. Dense retrieval reaches 71.13% on LongMemEval versus 63.20% for full context. On LOONG, head and tail reaches 78.67% Perfect Rate versus 72.44% for full context. Perfect Rate requires a judge score of exactly 100. On the custom NoLiMa diagnostic, Provence reaches 27.43% versus 0.57% for full context. Appendix § 8 reports all budgets, judge settings, and 95% paired-bootstrap intervals.
| Method | BABILong | LOONG | NoLiMa | LongMemEval |
| Full context | 31.13 | 72.44 | 0.57 | 63.20 |
| Empty context | 15.00 | 0.00 | 0.00 | 10.27 |
| Head and tail | 21.40 | 78.67 | 0.00 | 25.27 |
| BM25 | 46.20 | 69.33 | 3.10 | 68.13 |
| Dense | 30.27 | 69.33 | 4.60 | 71.13 |
| Dense+Rerank | 32.20 | 75.56 | 5.29 | 70.60 |
| LLMLingua-2 | 12.07 | 31.56 | 0.57 | 43.00 |
| Provence | 48.13 | 35.11 | 27.43 | 66.60 |
Table 2. Reader accuracy (%) at a 32K context budget under overlap-union packing, averaged over seeds 5768, 78516, and 944601. LOONG uses Perfect Rate; NoLiMa is the custom diagnostic. Full and empty controls use all questions without a context budget. Confidence intervals and all budgets are reported in Appendix § 8.
Discussion and limitations
Results are conditional on the evaluated cohorts, packing policy, and Gemini 2.5 Flash-Lite reader. Three seeds characterize this reader configuration. LOONG candidate lines are automatically identified; verbatim scoring can underestimate semantic retention, and automated answer metrics can err. NoLiMa is a custom diagnostic, and LongMemEval evaluates static selection. Shared token budgets differ from native model costs; recorded selection compute is not end-to-end latency. Appendix § 8 details scoring limitations, output caps, uncertainty, and validation.
Conclusion
CoRA-BENCH evaluates evidence preservation and downstream answer quality on the same emitted contexts under shared context budgets. Evidence preservation favors lexical retrieval when surface cues are explicit and dense methods when evidence requires latent associations or conversational retrieval. The three-seed reader results yield different rankings: Provence has the highest 32K point estimates on BABILong and the custom NoLiMa diagnostic, while dense retrieval leads on LongMemEval. Differences between evidence recall and reader accuracy support reporting both metrics, particularly for text-modifying methods.
Protocol details
Packing and provenance. Evidence recall, token retention, and reader accuracy are computed on the same reader-visible contexts. BM25, dense retrieval, and dense reranking merge source-token spans to avoid duplicated overlap, preserve ranked-prefix selection and source order, and enforce the budget on the emitted text. Reranking uses candidates. The fixed reranking candidate pool may leave part of the token budget unused after overlap removal. Selection records retain chunk identifiers, source offsets, token counts, and request identities. Evidence is mapped to source occurrences rather than a document-level flag. For methods that alter chunk text, provenance identifiers are ignored during scoring and the labeled text must remain verbatim.
LM filter. For LOONG, the LM-filter baseline uses 512-token chunks with 64-token overlap. Gemini 3 Flash Preview [8] scores batches of 20 full chunks from 0 to 100 for relevance to the question, at temperature 0, seed 5768, and thinkingLevel=MINIMAL. This setting does not imply zero reasoning tokens. The Threshold , uncapped arm keeps only chunks scoring at least 50, without filling a token budget. The ranked, capped arms sort all chunks by score and pack a strict prefix up to 32K, 64K, or 128K, without applying the threshold. Both arms merge overlapping selected source spans and emit the retained text in source order, so shared regions appear only once. They reuse the same chunk relevance scores. Reader inference uses the common three-seed configuration with concurrency 4 and selection is scored once.
Selection compute. GPU-based selectors ran on Google Cloud g2-standard-8 instances with one NVIDIA L4 (24 GB) each, in us-central1-{a,b,c}, using PyTorch 2.9 and CUDA 12.9 [23]. Dense retrieval, reranking, LLMLingua-2, and Provence use GPU inference; BM25 and head-and-tail truncation are CPU-only. LM filtering and downstream inference use the Gemini API.
We measure selector compute-block wall time using time.perf_counter. The measurements total approximately 48.5 L4 instance-hours: BABILong 17.5, NoLiMa 18.0, LongMemEval 9.4, and LOONG 3.6. Table 3 provides the per-method breakdown. Provence on NoLiMa required the most compute (11.5 L4 instance-hours), distributed across eight L4s.
Table 3: Historical per-method selection compute under chunk-concatenation packing, in L4 instance wall-clock hours (recorded via time.perf_counter around each selector’s main compute block; this is instance wall time, not measured GPU utilization). CPU-only baselines and API-based LM filtering are excluded. LM-filter API costs are reported separately in the text.
| Method | BABILong | LOONG | NoLiMa | LongMemEval | Total |
| Dense | 2.5 | 0.3 | 1.2 | 1.1 | 5.1 |
| Dense+Rerank | 3.8 | 0.5 | 2.4 | 1.8 | 8.5 |
| LLMLingua-2 | 4.6 | 0.4 | 2.9 | 1.4 | 9.3 |
| Provence | 6.6 | 2.4 | 11.5 | 5.1 | 25.6 |
| Total | 17.5 | 3.6 | 18.0 | 9.4 | 48.5 |
Table 3.
For LM filtering, estimated API costs were $9.80 for selection, $3.98 for reader inference, and $1.28 for judging. The recorded compute times exclude query encoding for dense retrieval, model loading, and downstream inference. They use the chunk-concatenation selection implementation and do not measure the complete overlap-union pipeline.
LM-filter API timing. The extension recorded 1,481 successful selector calls, 813 reader calls, and 248 distinct judge calls. Mean recorded request completion times were 1.57, 1.33, and 2.22 seconds, respectively. These per-request measurements include retries. They do not measure end-to-end latency. Provider-native usage under the recorded list prices determines the API cost estimates above.
Dataset licenses. BABILong code is Apache 2.0; its PG19 source is Apache 2.0 and bAbI is BSD. LOONG is Apache 2.0. The NoLiMa evaluation code and needle set use the Adobe Research License, which permits noncommercial research use; the five official haystacks retain the source-specific terms listed by NoLiMa. LongMemEval-cleaned is MIT. We used these assets for noncommercial research and do not redistribute upstream data.
Method and model licenses. The rank-bm25 implementation is Apache 2.0. Qwen3-Embedding-0.6B and Qwen3-Reranker-0.6B are Apache 2.0. The LLMLingua software is MIT and the LLMLingua-2 MeetingBank checkpoint is Apache 2.0. The Provence checkpoint is CC BY-NC-ND 4.0 and was used only for noncommercial evaluation without modification or redistribution. tiktoken is MIT, PyTorch is BSD 3-Clause, and Transformers and Sentence Transformers are Apache 2.0. Gemini 3 Flash Preview was accessed through the official API under the Google Gemini API terms. Head and tail truncation is implemented locally and uses no external model asset.
Implementation and reproducibility. The implementation records dataset checksums, model identifiers, selection provenance, reader settings, scoring rules, and request identities. Configuration files and tests specify the packing, budget enforcement, and evaluation procedures. Source datasets and model weights remain subject to their owners’ licenses.
Reader evaluation
Repetitions and frozen cohorts. Each selected context and control is evaluated on the same questions at seeds 5768, 78516, and 944601. BABILong data generation uses seed 42, independently of reader inference. The reader temperature is set to 0. Deterministic selectors are evaluated once per configuration, with reproducibility verified across three seeds on 30 prespecified questions per dataset. Selection equality is required across these checks. Identical reader requests may share a response only within a seed. Identical judge requests may share a judgment. Hence, the reported uncertainty excludes judge-repeat variability.
| Component | Configuration |
| Reader | gemini-2.5-flash-lite; temperature 0; thinking budget 0; output cap 1,024; concurrency 8 |
| Reader repetitions | Same questions; seeds 5768, 78516, 944601 |
| BABILong metric | Pinned official task-label answer scorer and qa1-qa5 prompts |
| NoLiMa metric | Released case-sensitive contains scorer on the frozen custom diagnostic |
| LongMemEval judge | gpt-4o-2024-08-06; official question-type/abstention rubric; temperature 0; 10 output tokens |
| LOONG judge | gpt-5.5-2026-04-23; original rubric with an adapted judge model; reasoning disabled; 512 output tokens |
| LOONG metrics | Perfect Rate: score exactly 100; mean rubric score reported separately |
| Intervals | 2,000 paired bootstrap draws after averaging seeds within each question; BABILong task-stratified; NoLiMa needle-family-clustered |
| Latency pass | 30 matched questions per cell at reader seed 5768; selector controls use all 3 seeds; online concurrency 8 |
Table 4. Reader and evaluation settings. The same questions are evaluated at 3 seeds; dataset-native scoring rules are retained, with an adapted judge model for LOONG.
Reference alignment and scoring. BABILong uses the released task-label first-sentence scorer. LOONG uses the published grading rubric and numeric parser with GPT-5.5 as an adapted judge in place of the reference GPT-4 evaluator. LongMemEval uses its question-type and abstention prompts with the reference GPT-4o snapshot. All 500 questions enter answer accuracy, including 30 abstentions. NoLiMa retains the released case-sensitive contains rule on our custom cohort.
Intervals and seed variation. For each question we first average correctness over the 3 reader runs, then resample paired questions using 2,000 bootstrap draws and report the 2.5th and 97.5th percentiles. BABILong draws are stratified by task. NoLiMa draws keep related variants, depths, and haystacks together within a needle family (ten clusters). The same draws support paired method differences, reported as exploratory and unadjusted for multiple comparisons. Reader seeds are averaged within question, not treated as independent observations. Tables additionally report the minimum and maximum of the 3 seed accuracies; those ranges are not confidence intervals. Figure 2 visualizes the 32K comparison. Configurations without complete 3-seed scoring are excluded from inferential comparisons.
Figure 2. Reader accuracy at 32K with Gemini 2.5 Flash-Lite, averaged over seeds 5768, 78516, and 944601. Points show mean accuracy. Horizontal bars are 95% paired-bootstrap intervals after averaging reader runs within question. BABILong is task-stratified and NoLiMa is needle-family-clustered. Full/empty controls have no budget cap. LOONG reports Perfect Rate; other panels report their dataset-specific answer metrics. Evidence and token-retention measurements in Figure 3 use these same emitted contexts.
Coverage and validation. All 68 configurations in the completed six-method matrix and controls have complete 3-seed scoring: 91,518 logical answer outcomes and 73,272 distinct saved reader receipts. The separate LM-filter extension adds four LOONG configurations and 813 fully judged reader outcomes from new inference on all 75 sources. All 28,709 chunks have valid scores; all selections, evidence scores, reader requests, and native judge parses were reconstructed from saved artifacts. We validated source and request identities, emitted contexts, budget compliance, and scoring outputs across the primary matrix, with additional sampled receipt reconstruction. The tables report scoring coverage, output-cap counts, and complementary LOONG mean rubric scores.
Scope and limitations. LOONG uses Financial Spotlight level 1 and budget-specific cohorts. NoLiMa is a custom repeated-measures diagnostic with ten needle-family clusters, rather than the official aggregate or its base-length normalization. LongMemEval evaluates static context selection, rather than online memory management. Automatically identified LOONG candidate lines may be ambiguous, and verbatim scoring can underestimate semantic retention by text-modifying methods. String matching may accept unsupported answer mentions, automated judges may err, and source/reference unit ambiguities can affect financial scores. We retain each dataset’s stated scoring rules.
Reader evaluations use Gemini 2.5 Flash-Lite with thinking disabled and a 1,024-token output cap. We report output-cap hits and quantify variation across three seeds for this reader configuration. Confidence intervals reflect sampling variability within the evaluated cohorts, with the stratification and clustering described above. The evidence tables report fixed-cohort point estimates. The selection-compute totals in Appendix § 7 exclude query encoding and cannot be interpreted as end-to-end serving latency.
Future evaluations could extend this protocol to online memory systems that update stored information across interactions [27, 40, 25].
Token accounting and tables. Budgets count the selected context using cl100k_base, excluding task prompt overhead. Provider-native input/output usage determines reader API cost, and the native reader window is checked separately. Chunk length in the shared tokenizer is not assumed to equal native model length; the completed audit covers 671,634 source chunks for Provence preprocessing and embedding/reranker input-length limits. LLMingua-2’s internal segmentation is outside this audit. Provence windows cover long passages, while its ranking score uses the first native block. We retain raw materialized table text and apply no table-specific preservation, routing, or whitespace repair.
Hardware and timing scope. The reader evaluation uses an NVIDIA L4 (24 GB) per g2-standard-8 GPU worker and e2-standard-4 CPU execution. A complete standalone selection measurement includes chunking, query encoding, inference, and packing, with CUDA synchronization and fresh data caches. Incremental timings that reuse work across budgets are not standalone latency estimates. Provider-native input and output usage determines API cost.
| Method | Budget | Coverage | Accuracy [95% CI] | Seed min-max | Cap hits |
| Full context | n/a | 500/500 | 31.13 [27.33, 35.07] | 30.80 to 31.60 | 0 |
| Empty context | n/a | 500/500 | 15.00 [12.00, 18.00] | 14.20 to 15.40 | 0 |
| Head and tail | 32K | 500/500 | 21.40 [18.00, 24.80] | 21.40 to 21.40 | 0 |
| Head and tail | 64K | 500/500 | 30.13 [26.33, 33.87] | 30.00 to 30.20 | 0 |
| Head and tail | 128K | 500/500 | 32.47 [28.53, 36.40] | 32.40 to 32.60 | 0 |
| BM25 | 32K | 500/500 | 46.20 [42.60, 49.80] | 46.20 to 46.20 | 0 |
| BM25 | 64K | 500/500 | 45.47 [41.67, 49.00] | 45.00 to 46.00 | 0 |
| BM25 | 128K | 500/500 | 39.47 [35.60, 43.47] | 39.40 to 39.60 | 0 |
| Dense | 32K | 500/500 | 30.27 [26.60, 34.07] | 30.20 to 30.40 | 0 |
| Dense | 64K | 500/500 | 32.53 [28.73, 36.33] | 32.00 to 33.00 | 0 |
| Dense | 128K | 500/500 | 32.73 [28.87, 36.47] | 32.00 to 33.20 | 0 |
| Dense+Rerank | 32K | 500/500 | 32.20 [28.40, 35.80] | 32.20 to 32.20 | 0 |
| Dense+Rerank | 64K | 500/500 | 32.20 [28.33, 35.93] | 31.80 to 32.60 | 0 |
| Dense+Rerank | 128K | 500/500 | 33.47 [29.47, 37.33] | 33.00 to 34.00 | 0 |
| LLMLingua-2 | 32K | 500/500 | 12.07 [9.40, 14.87] | 12.00 to 12.20 | 0 |
| LLMLingua-2 | 64K | 500/500 | 14.27 [11.27, 17.20] | 14.20 to 14.40 | 0 |
| LLMLingua-2 | 128K | 500/500 | 16.60 [13.53, 19.93] | 16.40 to 16.80 | 0 |
| Provence | 32K | 500/500 | 48.13 [45.20, 50.93] | 48.00 to 48.20 | 0 |
| Provence | 64K | 500/500 | 48.13 [45.20, 50.93] | 48.00 to 48.20 | 0 |
| Provence | 128K | 500/500 | 48.13 [45.20, 50.93] | 48.00 to 48.20 | 0 |
Table 5. BABILong: reader evaluation under overlap-union packing. Accuracy is a percentage; brackets are 95% paired-bootstrap intervals after averaging three reader runs within each question (BABILong: task-stratified; NoLiMa: needle-family-clustered). The seed range is descriptive, not a confidence interval. Coverage counts questions scored at all three seeds; cap hits count outputs reaching the output-token limit.
Table 7: NoLiMa custom 870-observation diagnostic: reader evaluation under overlap-union packing. Accuracy is a percentage; brackets are 95% paired-bootstrap intervals after averaging three reader runs within each question (BABILong: task-stratified; NoLiMa: needle-family-clustered). The seed range is descriptive, not a confidence interval. Coverage counts questions scored at all three seeds; cap hits count outputs reaching the output-token limit.
| Method | Budget | Coverage | Accuracy [95% CI] | Seed min-max | Cap hits |
| Full context | n/a | 870/870 | 0.57 [0.00, 1.48] | 0.57 to 0.57 | 0 |
| Empty context | n/a | 870/870 | 0.00 [0.00, 0.00] | 0.00 to 0.00 | 0 |
| Head and tail | 32K | 870/870 | 0.00 [0.00, 0.00] | 0.00 to 0.00 | 0 |
| Head and tail | 64K | 870/870 | 0.27 [0.00, 0.88] | 0.23 to 0.34 | 0 |
| BM25 | 32K | 870/870 | 3.10 [0.65, 6.42] | 3.10 to 3.10 | 0 |
| BM25 | 64K | 870/870 | 1.69 [0.17, 4.11] | 1.61 to 1.72 | 0 |
| Dense | 32K | 870/870 | 4.60 [1.81, 7.45] | 4.60 to 4.60 | 0 |
| Dense | 64K | 870/870 | 2.64 [0.66, 5.57] | 2.64 to 2.64 | 0 |
| Dense+Rerank | 32K | 870/870 | 5.29 [2.29, 9.14] | 5.17 to 5.40 | 0 |
| Dense+Rerank | 64K | 870/870 | 2.84 [0.88, 5.61] | 2.76 to 2.87 | 0 |
| LLMLingua-2 | 32K | 870/870 | 0.57 [0.00, 2.08] | 0.57 to 0.57 | 0 |
| LLMLingua-2 | 64K | 870/870 | 0.34 [0.00, 0.87] | 0.34 to 0.34 | 0 |
| Provence | 32K | 870/870 | 27.43 [17.65, 43.08] | 27.36 to 27.47 | 0 |
| Provence | 64K | 870/870 | 27.43 [17.65, 43.08] | 27.36 to 27.47 | 0 |
Table 7.
Table 8: LongMemEval: reader evaluation under overlap-union packing. Accuracy includes all 500 questions, including 30 abstentions, and is a percentage; brackets are 95% paired-bootstrap intervals after averaging three reader runs within each question (BABILong: task-stratified; NoLiMa: needle-family-clustered). The seed range is descriptive, not a confidence interval. Coverage counts questions scored at all three seeds; cap hits count outputs reaching the output-token limit.
| Method | Budget | Coverage | Accuracy [95% CI] | Seed min-max | Cap hits |
| Full context | n/a | 500/500 | 63.20 [59.00, 67.13] | 62.80 to 63.80 | 156 |
| Empty context | n/a | 500/500 | 10.27 [7.53, 13.00] | 10.20 to 10.40 | 3 |
| Head and tail | 32K | 500/500 | 25.27 [21.53, 29.07] | 25.00 to 25.40 | 97 |
| Head and tail | 64K | 500/500 | 44.13 [39.93, 48.20] | 44.00 to 44.20 | 121 |
| BM25 | 32K | 500/500 | 68.13 [64.00, 72.07] | 67.00 to 68.80 | 115 |
| BM25 | 64K | 500/500 | 67.87 [63.93, 71.93] | 67.60 to 68.20 | 118 |
| Dense | 32K | 500/500 | 71.13 [67.07, 75.00] | 71.00 to 71.40 | 128 |
| Dense | 64K | 500/500 | 68.40 [64.33, 72.27] | 67.20 to 69.60 | 118 |
| Dense+Rerank | 32K | 500/500 | 70.60 [66.33, 74.60] | 70.20 to 71.00 | 110 |
| Dense+Rerank | 64K | 500/500 | 67.27 [63.33, 71.20] | 67.00 to 67.60 | 143 |
| LLMLingua-2 | 32K | 500/500 | 43.00 [39.00, 47.40] | 43.00 to 43.00 | 429 |
| LLMLingua-2 | 64K | 500/500 | 56.33 [52.07, 60.67] | 56.00 to 57.00 | 259 |
| Provence | 32K | 500/500 | 66.60 [62.60, 70.60] | 66.60 to 66.60 | 51 |
| Provence | 64K | 500/500 | 66.60 [62.60, 70.60] | 66.60 to 66.60 | 51 |
Table 8.
Table 9: LOONG complementary mean rubric scores (0 to 100 scale as returned by the native parser) with the adapted GPT-5.5 judge at each budget. Full and empty controls use all 75 questions. The LM-filter ranked arms apply no threshold; Threshold , uncapped has mean score 81.15 on all 75 questions. Scores outside the requested 1 to 100 rubric range are preserved and flagged, not clipped.
| Method | 32K | 64K | 128K |
| Full context | 88.75 | 88.75 | 88.75 |
| Empty context | 6.60 | 6.60 | 6.60 |
| Head and tail | 91.21 | 90.39 | 87.19 |
| BM25 | 81.11 | 85.78 | 79.48 |
| Dense | 84.39 | 88.72 | 81.73 |
| Dense+Rerank | 89.98 | 88.67 | 82.45 |
| LLMLingua-2 | 44.52 | 61.06 | 78.81 |
| Provence | 57.80 | 57.71 | 55.69 |
| LM filter (ranked) | 88.72 | 88.66 | 84.50 |
Table 9.
Evidence preservation and compression
Table 10–Table 13 report evidence recall and compression ratio on the same emitted contexts used in Appendix § 8. Both metrics are macro question-level means, with evidence evaluated only where annotations are available. Lower compression ratios indicate fewer transmitted tokens. Text-modifying methods use verbatim evidence survival; extractive methods use source occurrence or span provenance.
| Method | Evidence recall 32K | Evidence recall 64K | Evidence recall 128K | Compression ratio 32K | Compression ratio 64K | Compression ratio 128K |
| Head and tail | 0.238 | 0.446 | 0.659 | 0.130 | 0.260 | 0.519 |
| BM25 | 0.838 | 0.868 | 0.921 | 0.129 | 0.259 | 0.518 |
| Dense | 0.329 | 0.481 | 0.727 | 0.129 | 0.259 | 0.519 |
| Dense+Rerank | 0.395 | 0.483 | 0.714 | 0.129 | 0.258 | 0.491 |
| LLMLingua-2 | 0.000 | 0.000 | 0.003 | 0.105 | 0.216 | 0.453 |
| Provence | 0.750 | 0.750 | 0.750 | 0.014 | 0.014 | 0.014 |
Table 10. BABILong macro supporting-fact recall and macro compression ratio. Each cell uses the same 500 examples, all longer than 128K tokens.
| Method | Evidence recall 32K | Evidence recall 64K | Compression ratio 32K | Compression ratio 64K |
| Head and tail | 0.301 | 0.645 | 0.305 | 0.609 |
| BM25 | 0.940 | 0.976 | 0.302 | 0.607 |
| Dense | 0.967 | 0.993 | 0.302 | 0.607 |
| Dense+Rerank | 0.979 | 0.991 | 0.302 | 0.587 |
| LLMLingua-2 | 0.000 | 0.000 | 0.286 | 0.600 |
| Provence | 0.036 | 0.036 | 0.049 | 0.049 |
Table 13. LongMemEval macro answer-bearing turn recall over 470 answerable examples and macro compression ratio over all 500 examples, including 30 abstentions. All methods have complete coverage.
| Method | Evidence recall 32K | Evidence recall 64K | Evidence recall 128K | Compression ratio 32K | Compression ratio 64K | Compression ratio 128K |
| Head and tail | 0.973 | 0.986 | 1.000 | 0.241 | 0.427 | 0.653 |
| BM25 | 0.932 | 0.942 | 0.980 | 0.239 | 0.426 | 0.652 |
| Dense | 0.946 | 0.986 | 0.980 | 0.240 | 0.425 | 0.652 |
| Dense+Rerank | 0.973 | 0.986 | 0.980 | 0.240 | 0.417 | 0.609 |
| LLMLingua-2 | 0.122 | 0.188 | 0.220 | 0.226 | 0.389 | 0.585 |
| Provence | 0.676 | 0.667 | 0.640 | 0.017 | 0.017 | 0.017 |
| LM filter (ranked) | 1.000 | 1.000 | 1.000 | 0.240 | 0.425 | 0.652 |
Table 11. LOONG macro automatically matched candidate-line survival and macro compression ratio. Evidence denominators are 74/69/50; reader and token denominators are 75/70/51 at 32K/64K/128K. LM filter (ranked) applies no score threshold. Threshold ≥ 50, uncapped is reported separately: survival 0.932 () and compression ratio 0.037 ().
| Method | Evidence recall 32K | Evidence recall 64K | Compression ratio 32K | Compression ratio 64K |
| Head and tail | 0.000 | 0.333 | 0.249 | 0.498 |
| BM25 | 0.264 | 0.467 | 0.247 | 0.496 |
| Dense | 0.641 | 0.908 | 0.247 | 0.496 |
| Dense+Rerank | 0.683 | 0.908 | 0.247 | 0.495 |
| LLMLingua-2 | 0.000 | 0.000 | 0.210 | 0.450 |
| Provence | 0.332 | 0.332 | 0.005 | 0.005 |
Table 12. Custom NoLiMa normalized verbatim needle survival and macro compression ratio over 870 repeated observations.
Analysis of evidence preservation
Method rankings depend on the retrieval regime. On BABILong, BM25 has the highest evidence recall at every budget, reaching 0.838, 0.868, and 0.921. Dense retrieval improves from 0.329 to 0.727 as the budget grows, while reranking raises the 32K result to 0.395. On the custom NoLiMa diagnostic, where literal overlap is intentionally limited, the ordering changes. Dense and Dense+Rerank reach 0.641 and 0.683 at 32K, compared with 0.264 for BM25. At 64K, dense and Dense+Rerank both reach 0.908. These results support a conditional conclusion: lexical retrieval is effective when benchmark templates expose entity overlap, while semantic retrieval is more robust to a lexical gap.
Retrieval transfers to static dialogue selection. On LongMemEval, Dense+Rerank retains 0.979 and 0.991 of answer-bearing turns at 32K and 64K. Dense retrieval reaches 0.967 and 0.993, and BM25 reaches 0.940 and 0.976. Head and tail truncation is substantially lower at 0.301 and 0.645. At 32K, reranking therefore preserves nearly all annotated conversational evidence while transmitting about 0.302 of the original tokens. Dense retrieval has the highest evidence-recall point estimate at 64K.
Compression exposes distinct notions of preservation. On BABILong, Provence retains 0.750 of supporting facts while transmitting approximately 0.014 of the original tokens. Its complete-turn recall on LongMemEval is 0.036 under verbatim scoring, while LLMLingua-2 rarely preserves the complete labeled strings on BABILong, NoLiMa, or LongMemEval. Because both methods transform text, this result identifies a measurable gap between exact evidence survival and possible semantic sufficiency. Reader accuracy provides a complementary measure of semantic utility, on the same emitted contexts, subject to the reader configuration.
LOONG and the LM filter diagnostic. On LOONG, head and tail truncation retains 0.973, 0.986, and 1.000 of automatically matched candidate lines. Dense+Rerank reaches 0.973, 0.986, and 0.980, while BM25 reaches 0.932, 0.942, and 0.980. LLMLingua-2 ranges from 0.122 to 0.220, and Provence ranges from 0.640 to 0.676 while using about 0.017 of source tokens. The LM-filter Threshold , uncapped arm achieves 0.932 candidate-line survival, retains 0.037 of source tokens, and reaches 54.67% reader Perfect Rate (mean rubric score 81.15/100). The ranked, capped arms achieve 1.000 candidate-line survival at all three budgets, retain 0.240, 0.425, and 0.652 of source tokens, and reach 74.67%, 75.71%, and 63.40% Perfect Rate. Their larger outputs reflect ranking without the threshold, rather than a cap applied to the threshold-selected context. Accuracy and uncertainty appear in Table 6; cross-budget differences also reflect the eligible cohorts.
Table 6: LOONG: reader evaluation under overlap-union packing. The LM-filter threshold arm keeps scores without a cap; its ranked arms use no score threshold. Accuracy is LOONG Perfect Rate with the adapted GPT-5.5 judge; eligible cohorts are 75/70/51 at 32K/64K/128K; brackets are 95% paired-bootstrap intervals after averaging three reader runs within each question (BABILong: task-stratified; NoLiMa: needle-family-clustered). The seed range is descriptive, not a confidence interval. Coverage counts questions scored at all three seeds; cap hits count outputs reaching the output-token limit.
| Method | Budget / policy | Coverage | Accuracy [95% CI] | Seed range | Cap hits |
| Full context | n/a | 75/75 | 72.44 [61.78, 82.22] | 72.00 to 73.33 | 0 |
| Empty context | n/a | 75/75 | 0.00 [0.00, 0.00] | 0.00 to 0.00 | 0 |
| Head and tail | 32K | 75/75 | 78.67 [69.78, 87.12] | 77.33 to 80.00 | 0 |
| Head and tail | 64K | 70/70 | 80.00 [70.48, 88.57] | 78.57 to 81.43 | 0 |
| Head and tail | 128K | 51/51 | 67.32 [54.90, 79.08] | 64.71 to 70.59 | 0 |
| BM25 | 32K | 75/75 | 69.33 [59.11, 79.56] | 68.00 to 70.67 | 0 |
| BM25 | 64K | 70/70 | 73.81 [62.86, 82.86] | 72.86 to 74.29 | 0 |
| BM25 | 128K | 51/51 | 60.13 [46.41, 73.86] | 58.82 to 60.78 | 0 |
| Dense | 32K | 75/75 | 69.33 [59.10, 79.56] | 69.33 to 69.33 | 0 |
| Dense | 64K | 70/70 | 71.90 [61.43, 81.90] | 71.43 to 72.86 | 0 |
| Dense | 128K | 51/51 | 62.09 [49.02, 75.16] | 58.82 to 64.71 | 0 |
| Dense+Rerank | 32K | 75/75 | 75.56 [65.33, 84.89] | 74.67 to 76.00 | 0 |
| Dense+Rerank | 64K | 70/70 | 70.00 [58.57, 81.43] | 70.00 to 70.00 | 0 |
| Dense+Rerank | 128K | 51/51 | 60.78 [47.71, 74.51] | 58.82 to 62.75 | 0 |
| LLMLingua-2 | 32K | 75/75 | 31.56 [21.78, 42.22] | 30.67 to 32.00 | 1 |
| LLMLingua-2 | 64K | 70/70 | 42.86 [31.43, 54.29] | 42.86 to 42.86 | 0 |
| LLMLingua-2 | 128K | 51/51 | 58.17 [45.10, 71.90] | 56.86 to 58.82 | 0 |
| Provence | 32K | 75/75 | 35.11 [24.89, 46.67] | 34.67 to 36.00 | 0 |
| Provence | 64K | 70/70 | 36.19 [25.71, 47.62] | 35.71 to 37.14 | 0 |
| Provence | 128K | 51/51 | 35.29 [21.57, 49.02] | 35.29 to 35.29 | 0 |
| LM filter | Threshold , uncapped | 75/75 | 54.67 [44.00, 65.33] | 54.67 to 54.67 | 0 |
| LM filter (ranked) | 32K | 75/75 | 74.67 [65.33, 84.00] | 74.67 to 74.67 | 0 |
| LM filter (ranked) | 64K | 70/70 | 75.71 [65.71, 85.71] | 75.71 to 75.71 | 0 |
| LM filter (ranked) | 128K | 51/51 | 63.40 [50.98, 76.47] | 62.75 to 64.71 | 0 |
Table 6.
Recall-compression tradeoffs
Figure 3 shows evidence recall against retained context for the six non-LLM selectors. LOONG LM-filter results are reported in the accompanying analysis.
Figure 3. Evidence recall versus retained source tokens across four datasets. Both axes show percentages, with a logarithmic token axis. Marker shapes indicate token budgets with hollow markers denoting verbatim evidence scoring and ringed markers denoting full context. Dashed lines show nondominated envelopes for fixed-cohort datasets. LOONG uses different cohorts across budgets (74/69/50 evidence-evaluable examples). Insets enlarge clustered results, with horizontally offset markers for clarity. Provence is shown once where results coincide across budgets. LM-filter results are excluded. All values are point estimates without confidence intervals.
References
- [1]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.172. URL https://aclanthology.org/2024.acl-long.172/.DOI
- [2]Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association of Computational Linguistics (Volume 1: Long Papers), pages 3639–3664, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.183. URL https://aclanthology.org/2025.acl-long.183/.DOI
- [3]Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts, 2023. URL https://arxiv.org/abs/2305.14788.
- [4]Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, and Stéphane Clinchant. Provence: Efficient and robust context pruning for retrieval-augmented generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=TDy5Ih78b4.arxiv.org/abs/2501.16214
- [5]Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, volume 2024, pages 35549–35562, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/98ed250b203d1ac6b24bbcf263e3d4a7-Abstract-Conference.html.arxiv.org/abs/2307.08691
- [6]Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model, 2024. URL https://arxiv.org/abs/2307.06945.
- [7]Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/abs/2403.05530.
- [8]Google. Gemini 3 flash preview. Gemini API documentation, 2025. URL https://ai.google.dev/gemini-api/docs/models/gemini-3-flash-preview.
- [9]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=kIoBbc76Sy.arxiv.org/abs/2404.06654
- [10]Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of EMNLP, pages 13358–13376, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.825. URL https://aclanthology.org/2023.emnlp-main.825/.DOI
- [11]Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Proceedings of ACL, pages 1658–1677. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.91. URL https://aclanthology.org/2024.acl-long.91/.DOI
- [12]Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP, pages 6769–6781. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-main.550. URL https://aclanthology.org/2020.emnlp-main.550/.DOI
- [13]Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack. In Advances in Neural Information Processing Systems, volume 37, pages 106519–106554. Curran Associates, Inc., 2024. doi: 10.52202/079017-3381. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/c0d62e70dbc659cc9bd44cbcf1cb652f-Abstract-Datasets_and_Benchmarks_Track.html.DOI
- [14]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626. Association for Computing Machinery, 2023. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006.3613165.
- [15]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc269df3b755da0-Abstract.html.arxiv.org/abs/2005.11401
- [17]Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2356–2362. Association for Computing Machinery, 2021. doi: 10.1145/3404835.3463238. URL https://doi.org/10.1145/3404835.3463238.
- [18]Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl_a_00638.DOI
- [19]Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. In Proceedings of ACL, pages 13851–13870. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.747. URL https://aclanthology.org/2024.acl-long.747/.DOI
- [20]Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schütze. NoLiMa: Long-context evaluation beyond literal matching. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 44554–44570. PMLR, 2025. URL https://proceedings.mlr.press/v267/modarressi25a.html.arxiv.org/abs/2502.05167
- [21]Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens, 2024. URL https://arxiv.org/abs/2304.08467.
- [22]Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085, 2019. URL https://arxiv.org/abs/1901.04085.
- [23]NVIDIA Corporation. CUDA Toolkit Documentation, Release 12.9. NVIDIA CUDA Toolkit Documentation, 2025. URL https://docs.nvidia.com/cuda/archive/12.9.0/. Accessed 2026-09-14.
- [24]OpenAI and Shantanu Jain. tiktoken: A fast BPE tokenizer for use with openai’s models. Software repository, 2022. URL https://github.com/openai/tiktoken.
- [25]Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems, 2023. URL https://arxiv.org/abs/2310.08560.
- [26]Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Association for Computational Linguistics: ACL 2024, pages 963–981, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.57. URL https://aclanthology.org/2024.findings-acl.57/.DOI
- [27]Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, pages 1–22, New York, NY, USA, 2023. Association for Computing Machinery. doi: 10.1145/3586183.3606763. URL https://doi.org/10.1145/3586183.3606763.
- [28]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alikhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, pages 8024–8035. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/hash/bcbce4ef7f92f2bf9f701277740-Abstract.html.arxiv.org/abs/1912.01703
- [29]Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009. doi: 10.1561/1500000019. URL https://doi.org/10.1561/1500000019.
- [30]Jiwei Tang, Jin Xu, Tingwei Lu, Zhicheng Zhang, Yiming Zhao, Lin Hai, and Hai-Tao Zheng. Perception compressor: A training-free prompt compression framework in long context scenarios, 2025. URL https://arxiv.org/abs/2409.19272.
- [31]Jiwei Tang, Shilei Liu, Zhicheng Zhang, Yujin Yuan, Libin Zheng, Wenbo Su, and Bo Zheng. Comi: Coarse-to-fine context compression via marginal information gain, 2026. URL https://arxiv.org/abs/2602.01719.
- [32]Nikita Tatarinov, Vidhyakshaya Kannan, Haricharana Srinivasa, Arnav Raj, Harpreet Singh Anand, Varun Singh, Aditya Luthra, Ravi J J Jade, Agam Shah, and Sudheer Chava. KG-MuLOA: A framework for KG-based multi-level QA extraction and long-context LLM evaluation. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3323–3359, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.151. URL https://aclanthology.org/2026.acl-long.151/.DOI
- [33]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.arxiv.org/abs/1706.03762
- [34]Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. Leave no document behind: Benchmarking long-context LLMs with extended multi-doc QA. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5627–5646. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.emnlp-main.322. URL https://aclanthology.org/2024.emnlp-main.322/.DOI
- [35]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of EMNLP: System Demonstrations, pages 38–45. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6/.DOI
- [37]Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmented LMs with context compression and selective augmentation. In International Conference on Learning Representations, volume 2024, pages 43478–43502, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/bda88ed2892f5e61c9a9bf215c566913-Abstract-Conference.html.arxiv.org/abs/2310.04408
- [38]Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. RankRAG: Unifying context ranking with retrieval-augmented generation in LLMs. In Advances in Neural Information Processing Systems, volume 37, pages 121156–121184. Curran Associates, Inc., 2024. doi: 10.52202/079017-3850. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/db93ccb6cf392f352570dd5af0a223d3-Paper-Conference.pdf.DOI
- [39]Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 Embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025. URL https://arxiv.org/abs/2506.05176.
- [40]Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. MemoryBank: Enhancing large language models with long-term memory. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17): 19724–19731, 2024. doi: 10.1609/aaai.v38i17.29946.DOI


