betaFeedback

WorksComputer Science

Embeddings vs. Grep: Rethinking the Dense Advantage in Code Search

Abstract

Code-search benchmarks show dense encoders well ahead of BM25, yet coding agents do well with grep. What does this dense advantage measure? Existing benchmarks cannot tell: they use curated queries, old code and a handicapped BM25. To find out, we build Repo-Premium, a realistic repository-level code-search benchmark. It ranks files in real repositories, with curated and raw queries in Python, JavaScript, TypeScript, Go, Rust and Java, pull requests from 2022 to 2026, and a preregistered test pool over 9191 repositories. Our experiments trace the advantage to two sources: evaluation choices that favor dense retrieval, such as splitting files into chunks for BM25 and counting edited tests as relevant, and queries that lack the code's rare identifiers. Expanding the query with LLM-written identifiers (generate-then-grep) helps BM25 far more than dense retrieval, letting it match dense retrieval in every Python pool, even on pull requests opened after the LLM was released. No method we tried predicts which retriever will win on a given query. From these results we derive three guidelines: measure dense gains against a fair lexical baseline, fuse dense retrieval with BM25, and add the code's words to queries that lack them.

https://github.com/hq-bench/repo-premium

https://huggingface.co/datasets/hq-bench/repo-premium

Introduction

Coding agents are tested on ever newer repository tasks ([50, 18]), but an end-to-end score does not say whether search, reasoning or editing failed. Retrieval benchmarks test search alone. On them, code-specialized encoders beat BM25 by wide nDCG@10 margins ([25, 27]). Yet one practitioner report found that embeddings were not the bottleneck for a SWE-bench agent that searched with grep and find ([5]). Teams build embedding indexes because of benchmark gaps, so the conflict matters. What does the dense advantage buy, and where does it come from? Existing evidence cannot settle this. Embedding benchmarks mostly pair curated or docstring-style queries with old code ([19, 35]); agent reports use raw issue or pull-request (PR) text ([21]). Lexical baselines often reuse the dense retriever’s chunking.

Weak baselines are a known source of false gains in neural ranking ([28, 49, 22]), but their effect in repository code search has not been measured. Gold labels also differ. SWE-bench’s gold patches contain no test files ([21]). A PR’s list of changed files does, and those tests count as relevant unless they are removed. Our own first version kept them. Tests can be fair targets elsewhere, so we report both label rules. Finally, benchmark code predates the encoders’ training cutoffs, so the advantage could also reflect contamination ([33, 39]).

To answer this, we build REPOPREMIUM, a realistic repository-level code-search benchmark over real repositories with file-level labels. Repository localization pipelines rank files first, and file labels come straight from the merged change (§ 3). Within each repository we fix the code snapshot, the label rule and the protocol, and vary only where the query comes from and when the change was made. The core has 1,265 queries over ten SWE-bench Verified repositories: curated problem statements ([36]) and raw GitHub PR text from 2022, 2023 and 2026.1 Each main analysis is thus repeated across query sources and years. An extension set adds five languages, nine held-out repositories and a 91-repository preregistered test pool (Table 1). The two preregistered tests are scored on the labels as released (edited tests included), with source-file results alongside. We use “grep” for lexical search and measure it with ranked BM25, so generate-then-grep is BM25 on an expanded query; only the agent evaluation (§ 5.7) lets agents call grep directly on 96 tasks.

PoolQuery textCoverage dateLanguagesReposFilesQueries
REPOPREMIUM-CORE
CUR-PRESWE-bench issue2013–23Python1010,200472
RAW-POSTPR title + body2026Python99,619288
RAW-PRE’23PR title + body2023Python1010,469138
RAW-PRE’22PR title + body2022Python1010,200367
total1,265
REPOPREMIUM-EXT
CROSS-LANGUAGEPR title + body2022–23JS, TS, Go, Rust437,712458
MULTI-SWE-BENCHCurated issue–TS/JS, Rust326,000134
MATCHED REPL.PR + LLM rewriteBoth sides5 languages36154,041360×2
HELD-OUT PYTHONPR title + body≥2025-04Python93,702139
DEEPSWETask spec–5 languages8831,288109
total1,560

Table 1. Query pools. REPOPREMIUM-CORE carries the main results over one shared corpus; REPOPREMIUM-EXT tests whether they hold elsewhere, each pool on its own repositories. Coverage date: the period the pool’s queries come from (PR merge dates; for CUR-PRE, SWE-bench issue dates); the encoders’ training cutoffs fall in 2024–25, so REPOPREMIUM-CORE pools lie clearly before (CUR-PRE, RAW-PRE’23, RAW-PRE’22) or after (RAW-POST). Benchmark pools (–) keep their released tasks; the matched replication samples both sides of the cutoff. Files: indexed files summed over a pool’s repositories (RAW-PRE’23 uses a 2024-Q1 snapshot; the matched replication takes each repository’s median over its tasks’ base commits). Queries: queries that pass the filters; the matched replication scores 351 of its 360 tasks (Section O). 5 languages: matched replication, Python, Java, Go, Rust and JS/TS repositories sampled as one stratum; DeepSWE, Python, JavaScript, TypeScript, Go and Rust. Routing and fusion were developed on the 472 CUR-PRE and 132 original RAW-POST queries (604) and tested once on a preregistered lockbox: DeepSWE and the TypeScript, Go and Rust PRs (539 queries).

We call dense retrieval’s nDCG@10 lead over BM25 the dense premium and ask three questions. How large is it under one consistent protocol? Which queries carry it? Should a system route each query to a retriever or fix one setup? Hybrid retrieval is known to help ([41]), and LLM query expansion helps both BM25 and dense retrieval ([42, 12]). We use them as controlled interventions, asking which retriever expansion helps and why, and test our advice on preregistered held-out repositories.

We evaluate thirteen dense encoders, plus SPLADE ([11]) and ColBERTv2 ([40]). Our experiments identify where the premium comes from:

F1 Evaluation choices inflate the dense advantage; a smaller lead survives on raw queries. Indexing whole files for BM25 cuts the curated premium from +0.121 to +0.034; counting edited tests as relevant doubles the RAW-POST premium (Figure 1, Section 5.1).

Figure 1. Dense retrieval’s advantage over BM25 (nDCG@10, canonical encoder jina-code-embeddings-0.5b, source-file labels) in the four Python pools, against three versions of BM25: untuned (the dense retriever’s chunking, c=1,000c = 1{,}000), tuned (whole files), and tuned with LLM-written anchors added to the query (generate-then-grep). Tuning BM25 shrinks the premium to +0.03–+0.13; adding anchors removes it in every pool, with BM25 significantly ahead on the curated pool. Means with repository-clustered 95% CIs (Table 2).

(Figure 1, Section 5.1). Against whole-file BM25 the encoder still leads the raw pools, mostly through Recall@10; on curated queries its interval covers zero.

F2 Adding the code’s words closes what remains. The premium sits on queries that lack the code’s rare words, and LLM-written identifiers help BM25 far more than the encoder. From the same expanded query the encoder gains a fraction of what BM25 gains, and prose stripped of identifiers changes nothing. BM25 then matches or beats dense retrieval in every Python pool, even on PRs newer than the LLM. The LLM may know the repository, so this replaces the premium but does not explain it (§ 5.2 and § 5.4).

F3 The two retrievers win different queries, and nothing we tried predicts which. Classifiers, LLM judges and a preregistered router all fail. A fixed fusion of the two beats either retriever alone and held on 539 preregistered lockbox queries (§ 5.3 and § 5.6).

From these findings we derive three guidelines for code search:

G1 Measure dense gains against a fair lexical baseline. Index whole files for BM25, strip templated PR text, and state which edited files count as relevant before comparing against dense retrieval.

G2 Keep dense retrieval and fuse it with BM25. A fixed rank fusion beats dense alone on development data and on 539 lockbox queries, and no router beat the fusion. In agents, the LLM matters more than the search tool (§ 5.7).

G3 When a query names code, add the code’s words before searching. One LLM call that drafts the missing identifiers lets whole-file BM25 match an embedding index. The call recurs per query, so it suits repositories searched rarely (Section E).

Related Work

Agent suites such as SWE-bench (Jimenez et al., [21]) and DeepSWE (Huang et al., [18]) run the whole agent loop. Code-retrieval benchmarks such as ColR (Li et al., [27]) pair docstring- or web-style queries with old code. None of them varies query mining and training-cutoff date together. Their labels also differ: SWE-bench gold patches contain no tests, while a PR’s file list keeps every edited file. It is well known that lexical and dense retrievers fail on different queries and that fusing them helps ([7, 41]). Our τ\tau is the highest inverse document frequency (IDF) among tokens shared by the query and the first source file the change edited. We compute IDF over the repository’s chunks. It adapts the MaxIDF predictor [15] but needs the gold file; its proxy τ^\hat{\tau} uses the query alone. LLM query expansion helps BM25 [42] and, through generated pseudo-documents, dense retrieval [12]. We use it as a controlled intervention, not as a proposed method. The closest work, SWE-Explore [51], also isolates retrieval: it scores the code regions that successful agents read, under a line budget, and its dense baseline uses static word embeddings. We score the files the change edited, use current code encoders, and vary the lexical baseline, the label rule and the query source. Section A gives further references.

RepoPremium Construction

Task and Data

Task. REPOPREMIUM is file retrieval inside one repository. Input: a natural-language description of a code change (an issue statement or a PR’s title and body) and a pinned snapshot of its repository. Output: a ranking of the snapshot’s files. Relevant: the non-test source files the described change edited. Test files stay in the ranking but count as non-relevant, and every retriever ranks the same files for the same query (an example in Figure 2).

Example of the same change worded two ways, comparing curated and raw queries and BM25 and dense retrieval rankings

Figure 2. The same change worded two ways, from the matched replication (Jina-code-0.5b against BM25 at c=1,000c=1{,}000). The curated rewrite states the behavior in the code’s terms, and BM25 ranks the gold file first. In the raw PR title, the words DOC and docs (highlighted) pull BM25 toward documentation files, while dense retrieval still ranks the gold file first. Gold sets are the source files the change edited (Step 3).

Two sets: REPOPREMIUM-CORE and REPOPREMIUM-EXT. The data has two sets (Table 1). REPOPREMIUM-CORE carries the main results. Its four query pools differ in how the query was written and when the change was made, but share ten Python repositories, one code snapshot per repository (except RAW-PRE’23, Step 6) and one label rule. The pools are curated statements from before the encoders’ cutoffs (CUR-PRE) and raw PR text from three windows: after the cutoffs (RAW-POST), 2023 (RAW-PRE’23) and 2022 (RAW-PRE’22). Because the code is shared, differences between pools come from the queries. The pools still describe different changes, so these comparisons are descriptive. The core is Python-only because SWE-bench Verified, the source of the curated queries, is Python-only, and the raw pools must use the same repositories.

REPOPREMIUM-EXT (extension) checks whether the core results hold elsewhere. Each pool uses its own repositories. The set has raw PRs in JavaScript, TypeScript, Go and Rust; curated

Multi-SWE-bench statements outside Python; nine held-out Python repositories; and DeepSWE tasks written by benchmark authors. A matched replication also presents each of 360 changes twice, as the raw PR and as an LLM rewrite, so only the wording changes. All pools use the same label rule, and their raw PR text goes through the same filters and stripper. DeepSWE and the TypeScript, Go and Rust PRs form the preregistered lockbox, on which we tested routing and fusion once.

Scale. REPOPREMIUM-CORE holds 1,265 queries over 10,200 indexed files (from 35 to 2,752 per repository; Table 6). REPOPREMIUM-EXT adds 1,560 queries, counting both views of each matched task, from five more languages (JavaScript, TypeScript, Go, Rust and Java), nine held-out Python repositories and 88 DeepSWE repositories (Table 1).

RepoSnapshotCorpus filesCUR-PRERAW-POSTRAW-PRE’23RAW-PRE’22
django2023-072,752226451024
sympy2023-021,51172573369
scikit-learn2023-0592018361552
astropy2023-0689921141528
matplotlib2023-0889534131544
sphinx2023-075814401731
pytest2022-102441934723
xarray2022-1217422301546
pylint2023-072,189832835
requests2022-0135827315
TOTAL10,200472288138367
MEAN EDITED FILES (AS MINED)1.152.963.222.69
MEAN GOLD FILES (SCORED)1.151.171.361.21
QUERIES EDITING A TEST0%63%59%58%
MEAN QUERY LENGTH (CH.)1,606662472493

Table 6. Per-repository corpus and query counts after quality filtering. Snapshot: creation month of the SWE-bench Verified instance whose base commit pins the corpus (RAW-PRE’23 uses a 2024-Q1 snapshot). Query length is in characters after stripping.

Why rank whole files, not functions or lines? Repository-level work starts by choosing which files to edit: an engineer or an agent opens a few files and edits within them. Localization pipelines rank files before functions or lines [44, 6]. SWE-bench’s retrieval baseline ranks whole files with BM25 [21]. Bug localization has long ranked files against bug reports [53]. Finer-grained work ranks functions or line regions [38, 51]. File labels also come straight from the merged change, so no annotator judges relevance. Scoring is still fine-grained: the encoder embeds 1,000-character chunks, and a file takes its best chunk’s score. The task is not easy either: repositories hold up to 2,752 files, and no pipeline we ran exceeds 0.7 nDCG@10.

Construction Pipeline

Six steps build the REPOPREMIUM-CORE pools (Figure 3). The REPOPREMIUM-EXT pools reuse the label rule and, for raw PR text, the filters and stripper (Steps 3–5), but index their own repositories; the appendix details each pool and its results.

Figure 3. Construction pipeline of the four REPOPREMIUM-CORE pools (Steps 1–6 of § 3.2); the REPOPREMIUM-EXT pools of Table 1 reuse Steps 3–5 on their own repositories. Curated SWE-bench Verified statements (blue) and raw merged-PR text (orange) share the label rules, preprocessing and one code snapshot per repository (RAW-PRE’23 uses a 2024-Q1 snapshot). Text filters apply to raw PRs only. Of 2,733 candidates, 1,265 remain in the four core pools.

Step 1: Repositories. The core uses ten of the twelve SWE-bench Verified repositories; the other two have too few merged PRs. RAW-POST covers nine, as sphinx merged too few in 2026.

Step 2: Queries. CUR-PRE takes each SWE-bench Verified issue statement. The raw pools take the title and body of merged PRs from each date window, most recent first (up to 100 per repository, 40 for RAW-PRE’23).

Step 3: Gold labels. The gold set is the indexed source files the change edited, each with binary relevance. CUR-PRE reads them from the SWE-bench gold patch, which has no tests. For raw PRs we drop edited tests and unindexed files (documentation, changelogs) from the changed-file list, matching SWE-bench’s oracle files. Because this choice moves the premium, we also score with tests included (Section B.3).

Step 4: Quality filter. A query is kept if it has one to three gold files and every one exists at the snapshot; a change that creates its source file is dropped rather than relabeled. Raw PRs must also describe a code change (at least 100 characters, at most 20 edited files, not docs-, CI- or test-only). Of 2,733 candidates, 1,265 remain (Table 5): changes that edit a few existing source files.

PoolCandidates<100 ch.>20 filesDocsCITestsNo codeNo source>3 filesMissingKept
CUR-PRE497––––––01015472
RAW-POST89719219821074949243255288
RAW-PRE’234001094414917221082138
RAW-PRE’229392512790632548124115367
TOTAL2,73355250213219911194691871,265

Table 5. Quality-filter funnel for the four core pools, ten repositories. Text rules apply to raw PRs only. The gold rule counts scored gold files (indexed, non-test): no source means that no scored gold file remains, and missing means that at least one is absent at the pinned snapshot (RAW-PRE’23: the 2024-Q1 snapshot).

Step 5: Preprocessing. A regex stripper removes PR templates, checklists and HTML comments that otherwise swamp BM25 (Sections D.1 and J).

Step 6: Corpus. Each repository is pinned at the base commit of its most recent SWE-bench Verified instance, and every .py file is one document; RAW-PRE’23 uses a 2024-Q1 snapshot. RAW-POST’s queries and changes therefore postdate the encoders, but the indexed code does not. Rescoring each task at its own base commit moves no tuned premium by more than 0.025 (Section B.2). Further construction details are in Section B.

Release. We release the queries, file-level labels with a test-file flag, pinned commits, the stripper and all evaluation harnesses (Section G).

Experimental Setup

Retrievers. We compare BM25 (BM25Okapi over chunks), a literal IDF-weighted grep baseline (reported separately as “grep” in appendix tables), and thirteen sentence-transformers encoder checkpoints. The encoders come from six families: jina-code-embeddings, Qwen3-Embedding, C2LLM, F2LLM, E5 and gte-Qwen2. All run without instruction prefixes. Unless noted, the canonical encoder is jina-code-embeddings-0.5b (Section C).

Chunking and the lexical baseline. Documents are chunked at c=1,000c = 1{,}000 characters with a 10-line overlap, and chunk scores are max-pooled to a file score. Chunk size matters: dense encoders truncate long inputs, while BM25 prefers large chunks. We use c=1,000c = 1{,}000 as the common protocol. The tuned lexical baseline is BM25 over whole files, which is BM25’s best setting on the raw pools and within 0.002 of its best (c=10,000c = 10{,}000) on CUR-PRE (Section 5.1).

Metric. The per-query premium is Δ(q)=nDCG⁡@10dense(q)−nDCG⁡@10BM25(q)\Delta(q) = \operatorname{nDCG}@10_{\mathrm{dense}}(q) - \operatorname{nDCG}@10_{\mathrm{BM25}}(q), scored per file with binary relevance against source-file gold sets. We average it over queries and also report Recall@10 and MRR@10.

Query distinctiveness. For a stripped query with token set qq and the token set gg of its first gold file (the first indexed, non-test file in the change’s list),

τ(q)=max⁡t∈q∩gIDF⁡(t),(1)\tau(q) = \max_{t \in q \cap g} \operatorname{IDF}(t), \tag*{(1)}

with IDF over the repository’s chunk corpus: the MaxIDF predictor of Hauff et al. [15] restricted to shared tokens. It needs the gold file, so it is an offline statistic. A deployable proxy τ~\tilde{\tau} is the mean IDF of the stripped query’s five rarest tokens, computed from the query alone.

Statistics. Intervals are 95% bootstraps, resampling repositories for comparisons between pools. Where we say an intervention does not raise the premium, its 90% interval lies below a preset +0.05 nDCG@10 margin (Section C).

Query sets. Pool-level results use the full raw pools of Table 1. Analyses that needed further retrieval runs use 461 raw queries chosen before we unified the gold rule. These are the thirteen-encoder sweep, HyDE, learned sparse and late-interaction retrievers, the reranker, anchor ablation, per-commit snapshots, the label audit, the τ\tau analyses and the robustness battery. On this subset every canonical ranking is identical (Section B.1). Routing and fusion keep the development pool they were configured on.

Results and Analysis

How Large Is the Premium?

Chunking decides the sign; whole-file BM25 removes most of the gap. On django, dense retrieval over whole files loses to BM25 (Δ=−0.153\Delta= -0.153) because the encoders truncate long files; chunking at c=1,000c = 1{,}000 flips this to +0.119. BM25, in contrast, gains from larger windows. Against whole-file BM25 the premium falls to +0.034 on CUR-PRE and +0.059 to +0.125 on the raw pools (Table 2). No encoder keeps a significant premium on CUR-PRE or RAW-POST, and seven of thirteen fall significantly below zero on CUR-PRE (Ta ble 15). BM25’s setting was picked on the evaluation queries, so these are lower bounds for dense retrieval.

POOLDENSEQUERY+ANCH.PREM.AFTER
CUR-PRE0.5320.4980.613+0.034−0.081
RAW-POST0.5970.5390.651+0.059−0.054
RAW-PRE’230.5440.4550.603+0.089−0.059
RAW-PRE’220.5720.4470.635+0.125−0.062

Table 2. Generation against a tuned lexical baseline (nDCG@10, source-file labels). Both BM25 arms score whole files; dense is the canonical encoder at c=1,000c = 1{,}000. “Premium” is dense minus BM25 on the original query, “after” dense minus BM25 on the query plus generated anchors; n=472,288,138,367n = 472, 288, 138, 367. Bold: the repository-clustered 95% interval excludes zero. Generate-then-grep is never behind dense retrieval in point estimate and is significantly ahead on the curated pool.

Counting test files as relevant inflates the premium. Dense retrievers rank a test file first far more often than BM25 (59% against 34% of curated queries). Adding edited tests to the gold sets therefore raises the tuned premium from +0.034 to +0.165 on CUR-PRE and from +0.059 to +0.136 on RAW-POST (Section B.3). With either rule applied to every pool, the curated and raw pools are indistinguishable. Mixing the rules, as happens when SWE-bench gold patches are scored next to PR file lists, widens the RAW-POST-minus-CUR-PRE gap from +0.025 to +0.102.

What survives is mostly coverage. Dense retrieval places more gold files in its top 10 (Recall@10 +0.10 to +0.19) but ranks its first hit significantly higher only on RAW-PRE’22 (Table 11). Outside Python, a modest premium remains on TypeScript and Rust but not on Go (single repositories; Section B.6).

CELLNDCG@10 VS BM25NDCG@10 VS GEN.RECALL@10 VS BM25RECALL@10 VS GEN.MRR@10 VS BM25MRR@10 VS GEN.
CUR-PRE+0.034−0.081+0.099−0.001+0.017−0.106
RAW-POST+0.059−0.054+0.106+0.014+0.036−0.084
RAW-PRE’23+0.089−0.059+0.161−0.037+0.062−0.064
RAW-PRE’22+0.125−0.062+0.188−0.031+0.108−0.071

Table 11. The dense advantage under three metrics: the canonical encoder (c=1,000c = 1{,}000) minus BM25 over whole files on the original query (“vs BM25”) and on the query plus generated anchors (“vs gen.”), source-file labels, full pools. Against plain BM25 the advantage lies mostly in Recall@10 (significant in the three raw pools); MRR@10 differs significantly only on RAW-PRE’22. Against generate-then-grep it disappears on every metric. CIs are in the released analysis files.

Much of the reported premium thus comes from the setup. Against whole-file BM25 the curated premium nearly vanishes (+0.121 → +0.034), and the raw premiums shrink by a third to over a half. Counting test files as relevant inflates it again. What survives is mostly wider coverage.

Supplying the Missing Evidence

If the premium reflects lexical evidence a query lacks, supplying that evidence should help BM25 more than dense retrieval.

Generate-then-grep catches up with dense retrieval. A generator (qwen35-122B, temperature 0, one frozen prompt) reads the stripped query, but never gold paths or patches. It writes identifiers, file and module hints, APIs, error terms and a code skeleton, and we prepend this text to the query for BM25. With BM25 over whole files in both arms, generation adds 0.11–0.19 nDCG@10 to BM25, and generate-then-grep is never behind the canonical encoder. It leads significantly on CUR-PRE, by 0.081. Its lead on the raw pools is 0.054–0.062, but those intervals cover zero, so a small dense lead remains possible (Table 2). Whole files are not needed: with BM25 at the encoder’s 1,000-character chunks, the premium after expansion is −0.052 to +0.012.

The added words help BM25 far more than the encoder. When the encoder embeds the expanded query, it gains +0.03 to +0.07, against +0.12 to +0.21 for BM25; the difference excludes zero in every pool (Table 12). Identifiers, APIs and error terms alone close the premium; prose search phrases do not (Section B.5). Pseudo-relevance feedback (RM3, default setting; Lavrenko and Croft, [26]) even lowers BM25.

CELLBM25, WHOLE FILES QUERYBM25, WHOLE FILES +ANCH.BM25, WHOLE FILES RM3DENSE QUERYDENSE +ANCH.GAIN GAP [95% CI]
CUR-PRE0.4980.6130.4290.5320.566+0.081 [+0.029, +0.128]
RAW-POST0.5390.6640.4970.6130.639+0.099 [+0.047, +0.130]
RAW-PRE’230.3990.5530.3870.5230.595+0.082 [+0.006, +0.129]
RAW-PRE’220.4630.6710.4390.5720.640+0.140 [+0.087, +0.221]

Table 12. Controls for the generative arm (nDCG@10). RM3: pseudo-relevance feedback on the same whole-file BM25 (10 documents, 10 terms, weight 0.5, untuned). Dense “+anch.”: the canonical encoder embeds the expanded query (HyDE-style). Gain gap: BM25’s gain from the anchors minus dense retrieval’s, with repository-clustered 95% CIs; source-file labels. Loader guards: RM3’s first pass reproduces whole-file BM25 exactly; dense on the original query reproduces the canonical lists for all but one RAW-PRE’23 query. Raw pools: original queries (n=132,72,257n = 132, 72, 257; § 4), so values differ from the full-pool Table 2.

The result holds after the generator’s release and against other dense arms. On held-out PRs opened after the generator’s release, generate-then-grep still beats dense retrieval by 0.094 (Section E). Memorized changes do not explain this, though knowledge of the repositories might. Qwen3-Embedding-8B keeps a larger premium on two of the three raw pools, but is again never ahead after generation. SPLADE (Formal et al., [11]) and ColBERTv2 (Santhanam et al., [40]) stay below the canonical encoder (Table 12, Table 10). Smaller and weight-matched generators close two thirds to 80% of the premium, so the direction holds but the size depends on the generator.

CELLQWEN3-EMB-8B PREM.QWEN3-EMB-8B AFTER GEN.QWEN3-EMB-8B GAPSPLADE QUERYSPLADE +ANCH.COLBERTV2 QUERYCOLBERTV2 +ANCH.
CUR-PRE+0.017−0.099+0.0700.3900.4360.3970.428
RAW-POST+0.054−0.072+0.0640.5400.5570.4220.570
RAW-PRE’23+0.120−0.034+0.0760.5230.5820.4540.542
RAW-PRE’22+0.159−0.049+0.1620.4890.5470.4210.541

Table 10. Generality checks (nDCG@10). Left: the generative-arm analysis repeated with Qwen3-Embedding-8B (premium against whole-file BM25; residual against whole-file BM25 on the query plus anchors; gap = BM25’s gain from the anchors minus the encoder’s). Its original-query lists reproduce the cached canonical runs exactly on CUR-PRE and RAW-POST. On the full raw pools, Qwen3-Emb-8B’s tuned premium is +0.037, +0.117 and +0.146 (RAW-POST, RAW-PRE’23, RAW-PRE’22) and its residual after generation −0.075, −0.031 and −0.041. Right: off-the-shelf SPLADE (splade-cocondenser-ensembledistil) and ColBERTv2 (exhaustive MaxSim, queries up to 256 tokens) on c=1,000c = 1{,}000 chunks max-pooled per file, on the original and expanded query; raw pools: original queries (132 RAW-POST, 72 RAW-PRE’23, 257 RAW-PRE’22; § 4), so values differ from the full-pool Table 2. For reference, on these queries whole-file BM25 scores 0.498, 0.539, 0.399, 0.463 and the canonical encoder 0.532, 0.613, 0.523, 0.572. Source-file labels.

When an LLM writes the missing identifiers into the query, BM25 is never behind dense retrieval in any pool (point estimates), and the same text helps the encoder far less. Generated identifiers can replace the premium. Since the generator may know the repository, this does not show where the premium comes from.

Fuse Once Instead of Routing Per Query

Fusion beats either retriever and a reranker. Reciprocal rank fusion of dense retrieval and whole-file BM25 raises nDCG@10 by +0.060 over dense alone on the development pool; fusing with generate-then-grep instead, at one LLM call per query, raises it by +0.089. A listwise reranker (jina-reranker-v3) over the same candidates does no better than fusion.

A preregistered router misses its criterion; the fixed fusions hold. A router picks dense retrieval, BM25 or their fusion for each query, and the oracle headroom is large. We built about 30 candidate routers on query, list-agreement and score features. Only one had a development gain that excluded zero. We froze it and scored it once on a lockbox that no earlier analysis had touched (539 queries, 91 repositories). It gained +0.005 over always using the fusion, against +0.077 for the oracle, and its interval covered zero, so it missed its preregistered criterion. Both fixed fusions were configured on development data only. On the lockbox they beat dense retrieval by +0.019 and +0.045 on the labels as released, and by +0.027 and +0.062 with source-file labels. They still beat it when repositories are weighted equally (Section F).

A fixed fusion is therefore the robust choice: it beats either retriever alone and held up on the lockbox, and per-query routing did not detectably beat it (Table 3).

PipelinePer queryDevLockbox
BM25, whole filesnone0.5070.432
Denseembed0.5500.533
RRF(Dense, BM25)embed0.6090.551
Reranker+rerank0.574—
Router (frozen)embed—0.557
Generate-then-grepLLM0.6250.534
RRF(Dense, Gen.)LLM+embed0.6380.578

Table 3. Fixed pipelines by per-query cost (nDCG@10). Dev: the development pool D (CUR-PRE and the original RAW-POST queries; 604 queries, 10 repositories), source-file labels; lockbox: DeepSWE and the TypeScript, Go and Rust pools (539 queries, 91 repositories), scored once as preregistered on the labels as released. Dense is jina-code-0.5b; the reranker reorders the union of the dense and BM25 lists; the router chooses per query among dense, BM25 and their RRF. Per query: work beyond BM25 (embed: one query embedding; rerank: jina-reranker-v3; LLM: one generation call). —: not run (the router is fit on dev; the reranker was not preregistered). Bold: highest score. BM25’s chunk size was chosen on dev. Differences quoted in the text are computed before rounding.

Which Queries Carry the Premium

The premium shrinks as the shared tokens get rarer. On the 604 CUR-PRE and original RAW-POST queries, the premium falls from +0.217 in the lowest τ\tau quartile to +0.050 in the highest (Figure 4). Against whole-file BM25, it falls from +0.125 to −0.046. Per query the link is weak (r=−0.124r = -0.124), so the claim is about group means. The slope is negative in 37 of 38 slices (mining regimes, thirteen encoders, five languages), but we chose τ\tau on those slices. A preregistered test on 29 held-out repositories had the same sign, with an interval covering zero (Section N).

Figure 4. The dense-over-BM25 premium Δ\Delta decays with query distinctiveness τ\tau (dense at c=1,000c = 1{,}000 against BM25 at c=1,000c = 1{,}000; source-file labels). Decile means with query-bootstrap 95% CIs over the n=604n = 604 curated and original raw queries; dashed: linear fit. Quartile means fall monotonically; deciles show the noise around that trend. Dotted verticals mark the exemplar queries of Section M.1.

Removing anchors hurts both retrievers. We replace the five rarest tokens a query shares with its first gold file by synthetic strings, and compare against an IDF-matched placebo. If dense retrieval did not rely on these tokens, the premium should rise. It does not on curated (CUR-PRE) or raw (RAW-POST) Python queries (a curated rise above 0.05 is excluded). It does rise on TypeScript, and on Rust without a resolved interval. Instead both retrievers collapse, and the encoder loses a third to almost half of its nDCG@10. Most replaced tokens are ordinary words, so the edit may also remove task meaning (Section E).

The premium thus sits on queries whose rarest token shared with the code is still common. τ\tau shows where the premium is, but it is not a lever: removing the rare tokens lowers both retrievers and, on Python queries, leaves the premium unchanged.

Query Source and Vintage

Pools differ little once labeled alike. At c=1,000c = 1{,}000 every pool shows a premium, +0.121 curated against +0.128 to +0.191 raw (Table 4); the RAW-POST-minus-curated gap of +0.031 has a repository-block interval covering zero. Keeping the raw pools’ edited tests as gold more than doubles that gap, so most of the pooled contrast came from the label rule.

PoolQueryDatennΔ\Delta sourceΔ\Delta raw tests kept
CUR-PRECURATEDPRE472+0.121+0.121
RAW-PRE’23RAW2023138+0.128+0.192
RAW-PRE’22RAW2022367+0.191+0.245
RAW-POSTRAW2026288+0.152+0.192

Table 4. Premium at c=1,000c = 1{,}000 by pool under source-file labels and with the raw pools’ edited test files kept as gold (indexed files only). Curated labels are unchanged in this comparison, since its gold patch has no tests. Keeping tests raises every raw pool by 0.04–0.06.

Raw wording raises the premium within a task. A 36-repository replication pairs each raw PR with an LLM rewrite of the same change, against one gold set and corpus. The raw view’s premium is larger by +0.091 for jina-code-0.5b and +0.111 for Qwen3-Embedding-0.6B, both intervals excluding zero (Section O). A date effect of similar size cannot be excluded there. DeepSWE task descriptions, which describe work to do rather than a finished change, show no premium distinguishable from zero (Section R).

Vintage shows no consistent effect. The 2022, 2023 and 2026 raw pools lie within 0.063 of each other, with no ordering by date. On 11 further Python repositories, 2022 and 2026 PRs differ by +0.019, not the smaller post-cutoff premium that contamination would predict. With so few repositories, modest effects of wording or date cannot be excluded.

Wording therefore moves the premium when the task is held fixed; pool origin and date show no consistent effect once labels match.

Predicting the Winning Retriever

The headroom is real. BM25 strictly beats dense retrieval on 24% of CUR-PRE and 21% of RAW-POST queries, and a per-query oracle would add about 0.09 nDCG@10 over dense alone.

No classifier or LLM judge we tried finds those queries. We tried classifiers on structural query features, logistic regression on query embeddings and two zero-shot LLM judges (qwen35-122B, gemma-4-31B-it). All reach AUROC 0.47–0.54, with every interval covering 0.5 (Table 19). Once tuned, each falls back to (nearly) always choosing dense; untuned, the judges pick keyword search for most queries and lose over 0.1 nDCG@10. The RAW-POST intervals are wide (n=132n = 132), so a modest signal on raw queries cannot be excluded (Section P).

PREDICTORAUROCROUTED − DENSE
QUERY FEATURES (GBT)0.510.000
QUERY EMBEDDING (LOGISTIC)0.52−0.004
qwen35-122B JUDGE0.540.000
gemma-4-31B JUDGE0.51−0.002

Table 19. Predicting per query whether BM25 beats dense retrieval (CUR-PRE+RAW-POST, 604 queries, leave one repository out). Pooled AUROC on non-tied queries; every interval covers 0.5. “Routed − dense”: nDCG@10 change from switching to BM25 above a threshold tuned on the training repositories. Query features: seven structural features plus τ^\hat{\tau}, an identifier flag and length; embedding: jina-code-0.5b. Other feature sets and models score 0.47–0.51.

BM25 thus wins about a quarter of queries, but nothing we tried can tell which ones from the query.

The Agent Boundary

Swapping the search tool changes little; swapping the LLM changes a lot. With the LLM fixed, a SWE-agent on the held-out pool (96 tasks) scores within about 0.04 exact-match whether it gets an unrestricted shell with grep, a lexical index, a dense index or a τ^\hat{\tau}-router. Exact-match means the files its final diff edits equal the gold set. Across four setups of three LLMs on a 30-task subset, exact-match ranges from 0.03 to 0.30. No tests are run, so this measures localization, not fix correctness. All runs use clean environments with the merged fix unreachable (Sections Q and Q.2).

For agents, then, the choice of LLM matters far more for localization than the search tool it calls.

Discussion and Conclusion

Reconciling the debate. Benchmark gaps and agent experience can both be right. Against BM25 with the dense retriever’s chunking, or with edited tests counted as relevant, dense encoders lead by 0.1–0.2 nDCG@10. Against whole-file BM25 on source-file labels they lead by little, mostly in coverage. Where they still lead, the query names no rare identifier. A language model can add those identifiers, and grep-style search then matches the encoder. An agent that picks its own search terms does the same. Fusing the two retrievers still beats either alone (Table 3).

Guidelines. 🟢 G1 Measure dense gains against a fair lexical baseline: whole files for BM25, templated PR text stripped, and one stated rule for which edited files count as relevant, applied to every query source. Report Recall@10 and an anchor-augmented BM25 arm next to the dense number. 🟢 G2 Keep dense retrieval and fuse it with whole-file BM25; per-query routing did not beat the fusion. If the caller is an agent, spend on the model before the search endpoint. 🟢 G3 When a query lacks the code’s vocabulary, add LLM-written anchors before searching. The call recurs per query, so a dense index is cheaper for repositories searched often (Section E).

We release REPOPREMIUM with source and test labels, the PR-boilerplate stripper, and the generation and agent-evaluation harnesses.

Limitations

The core covers ten Python repositories and the cross-language PR pools one repository per language, so modest effects cannot be excluded. BM25 was tuned on the evaluation queries, generation results depend on the generator, and labels are the files one change edited. Function- and line-level targets are untested.

Extended Related Work

Benchmarks. Agent suites refresh or harden SWE-bench [50, 9, 18, 34] but score the whole agent loop, so they cannot attribute a failure to retrieval. Code-retrieval benchmarks [19, 17, 31, 27, 45] use docstring, web or competition queries rather than repository changes. SWE-Explore [51] labels the line regions that successful agents read. Its explanation of why lexical search fails on issue text matches what τ\tau measures: the issue names no rare identifier of the edited module. Our encoders descend from code pretraining [10, 14, 13, 43]; the canonical one is jina-code [25].

Retrieval methods. Fusion, dense, late-interaction, learned-sparse and cascaded pipelines combine lexical and dense signals [7, 23, 24, 40, 11, 30, 29, 3]; we ask when the dense signal adds value and whether that can be predicted per query. τ\tau builds on pre-retrieval query-performance prediction [8, 16, 15, 4, 2] (Zhou and Croft 2007). LLM query expansion helps BM25 and dense retrieval [20, 32, 42, 12]; we use it as a controlled intervention against pseudo-relevance feedback [26]. LLM systems outside code also rewrite a user request into search queries before retrieving, for example over a knowledge base and a web search engine [48]. Rerankers trained on general web data transfer imperfectly to new domains [46]; our reranker baseline (jina-reranker-v3) is such a general-purpose model.

Embeddings beyond code. A survey of omnimodal embeddings [52] organizes cross-modal unification around coverage, comparability and compositionality. Domain-specific retrieval also requires careful evaluation: ZooClaw-FashionSigLIP2 [47] studies adaptation to fashion retrieval and structural biases in benchmark ground truth. These broader settings motivate examining how benchmark construction shapes measured retrieval gains; our experiments isolate this question for repository code search.

Practice and contamination. The claim we test is that embeddings were not the bottleneck for a grep-driven SWE-bench agent [5]; code-graph systems argue for structural retrieval instead [1]. Contamination work [33, 39] motivates our post-cutoff pools and the exposure test of generation.

Benchmark Construction Details

The construction keeps the pools comparable. Every pool uses one corpus snapshot per repository and one label rule (the indexed, non-test files a change edited), and every raw pool uses the same text filters. The checks below show that the snapshot, unlabeled relevant files and BM25’s tokenizer do not drive the results, but the test-file rule does.

Construction Details by Step

These notes add to Steps 1–6 of § 3.2 only what the main text omits.

Repositories. SWE-bench Verified’s 500 instances cover twelve repositories; seaborn and flask are too small for raw pools, leaving ten with enough merged PRs in every window. The core is Python because SWE-bench Verified is; Multi-SWE-bench enters as an REPOPREMIUM-Ext pool (§ 18), since its Python split repackages SWE-bench Verified.

Mining and labels. Raw PRs come from the GitHub search API, most recent first, capped at 100 per repository (40 for RAW-PRE’23); the query is the PR title and body. Gold files come from the SWE-bench gold patch (CUR-PRE) or the PR’s changed-file list (raw pools). The gold patch has no tests. PR file lists, by contrast, include tests, documentation, changelogs and configuration files, so scored gold sets keep only indexed, non-test files (Section B.3). The release flags each edited file as source, test or non-indexed. In the raw pools, tests are 29–33% of edited files and appear in 58–63% of queries. Undefined documentation, changelogs and templates are 29–32% and appear in about half of queries. RAW-PRE’22 was added because RAW-PRE’23, mined at 40 PRs per repository, is small.

Filtering. Filters apply in the order of Table 5. The gold rule asks for one to three scored gold files, all present at the snapshot, so changes that add their source file are excluded. Edited tests and documentation do not count toward the cap and need not exist. The cap keeps the task a localization problem. The text rule drops bodies under 100 characters, PRs touching more than 20 files and docs-, CI- or test-only changes. Paths under src/X map to X when only the latter exists. Analyses that needed new retrieval runs use the 461 raw queries of the original, stricter rule. The queries added by the unified rule have a lower premium at c=1,000c = 1{,}000, but no detectable difference in the tuned premium or the generation comparison. sphinx yields no RAW-POST query: it merged ten PRs in the 2026 window.

Preprocessing. Raw PR descriptions carry ticket headers, checklists, AI-disclosure sections, checkbox lines and HTML comments; unstripped, BM25 on django RAW-POST collapses to nDCG@10=0.010. The stripper (Section J) shortens raw queries by about 40% and curated statements by about 4%.

Corpus. Each repository is pinned at the base commit of its most recent SWE-bench Verified instance, so every CUR-PRE base commit precedes the snapshot. The corpus is every .py file, tests included (Table 6). RAW-PRE’23 uses a 2024-Q1 snapshot. As a result, most curated fixes are already in the snapshot, and the RAW-POST changes come about three years after the indexed code. “Post-cutoff” therefore describes the query and the change but not the code. Section B.2 tests whether this matters.

Per-Task Pre-Change Snapshots

The shared snapshot does not drive the premium. We re-score 933 core queries at each task’s own base commit (834 commits). Retrieval is otherwise identical, and a guard reproduces the shared-snapshot results exactly. No tuned premium moves by more than 0.025, and the raw-minus-curated gap is unchanged (Table 7). The curated premium rises slightly, so a snapshot that already contains the fix favors BM25 a little on curated queries.

PoolnnShared snapshotOwn base commitΔ\Delta
CUR-PRE472+0.034 [−0.024, +0.117]+0.056 [+0.024, +0.116]+0.022
RAW-POST132+0.074 [−0.027, +0.220]+0.098 [−0.003, +0.239]+0.024
RAW-PRE’2372+0.124 [−0.053, +0.309]+0.102 [−0.062, +0.347]−0.022
RAW-PRE’22255+0.112 [+0.028, +0.240]+0.125 [+0.033, +0.259]+0.013

Table 7. Tuned premium (dense minus whole-file BM25, nDCG@10) with each query scored at the shared snapshot and at its own pre-change commit, same queries, repository-clustered 95%95\% intervals. RAW-PRE’23’s shared corpus is its 2024-Q1 snapshot; two RAW-PRE’22 tasks whose gold is absent at the base commit are dropped.

Label Definition and the Test-File Bias

Counting edited test files as relevant rewards a habit of dense retrievers, not localization. If one rule is used for all pools, either source files only or all edited files, the curated and raw premiums are indistinguishable. Only mixing the two separates them (Table 8).

BM25LabelsCUR-PRERAW-POSTGap [95% CI]
c=1,000c = 1{,}000Source files+0.121+0.175+0.054 [−0.045, +0.198]
All edited+0.191+0.204+0.013 [−0.051, +0.122]
Mixed (as mined)+0.121+0.204+0.083
TunedSource files+0.034+0.074+0.040 [−0.059, +0.176]
All edited+0.165+0.152−0.013 [−0.122, +0.096]
Mixed (as mined)+0.034+0.152+0.118

Table 8. Premium (canonical encoder minus BM25, nDCG@10) under three label rules, on the original RAW-POST queries (§ 4); gaps are RAW-POST minus CUR-PRE, repository-block intervals. On the full raw pools the RAW-POST tuned premium is +0.059 (source files) and +0.136 (all edited files).

Rule. A file is a test if its path contains a test directory (test, tests, testing, __tests__, . . .) or its name matches a test pattern (test_*.py, conftest.py, *_test.go, *.spec.ts, . . .); it is indexed if the pool’s corpus indexes its extension. Every analysis is re-scored from stored top-10 lists with these files removed from the gold set; retrieved tests then count as non-relevant.

Dense retrievers rank tests first far more often. Dense retrieval puts a test file first for 59% of curated and 28–35% of raw queries, against 34% and 12–16% for tuned BM25. Word overlap and file names do not explain this: those tests share fewer query words than the source files dense retrieval ranks, and few are named after a gold module. Reproduction code in the query strengthens the preference, but it remains when the query has no code.

Beyond code. Many relevance labels are harvested from records made for other purposes, such as a change’s file list, a ticket’s links or a clinical note’s codes. The harvesting rule then decides what counts as relevant, and retriever families may differ in how easily they find the extra items. A retrieval comparison is only as consistent as its label rule.

Audit of Unlabeled Retrieved Files

Missing labels lower both retrievers’ scores about equally, so they do not explain the premium. An LLM judge (qwen35-122B) read each change description together with the top-3 non-gold files from the canonical encoder and from whole-file BM25. One gold file per query was mixed in without its label. We used all 132 original RAW-POST queries and 132 sampled CUR-PRE queries. The judge marked 75–78% of gold files as files to edit, but only 5–6% of the non-test misses, with no difference between the retrievers. The dense retriever’s apparent surplus of editable misses comes from its preference for tests (Section B.3).

Which Part of the Expansion Matters

Generation works through the identifiers it writes. We append one group of generated fields at a time (Table 9). Identifiers, APIs and error terms close the premium with or without file paths, and the code skeleton nearly does. Prose search phrases help only through the identifiers they contain; with those removed, the premium stays where it was.

DENSE − BM25 AFTER ADDING TO THE QUERY
POOLQUERY ONLYALL FIELDSIDENT.+APIS+ERRORSIDENT. + PATHSCODE SKELETONPROSE PHRASESPROSE, NO IDENTIFIERS
CUR-PRE+0.034−0.081−0.040−0.039−0.047−0.020+0.027
RAW-POST+0.074−0.052−0.006−0.009+0.018+0.036+0.086
RAW-PRE’23+0.124−0.030+0.007+0.006−0.008+0.019+0.128
RAW-PRE’22+0.109−0.099−0.045−0.028−0.001+0.016+0.109

Table 9. Dense premium (canonical encoder minus whole-file BM25, nDCG@10, source-file labels) after appending one group of generated fields to the query. Original queries (n=472,132,72,257n = 472, 132, 72, 257), so values differ from the full-pool Table 2 and from Table 7 (255 RAW-PRE’22 tasks). Repository-clustered intervals are in the released analysis file.

Identifier-Splitting BM25

Splitting snake- and camel-case identifiers does not change the conclusion. It helps BM25 at c=1,000c = 1{,}000, but in Python whole files help more. Against BM25’s best configuration the premium is at most about +0.09 in every pool, Python or not (TypeScript already used splitting).

Held-Out Pool

The held-out pool tests the findings on repositories we did not use during construction: nine Python repositories outside the core and outside SWE-bench Verified. We mined PRs merged on or after 2025-04-01 and applied the RAW-POST filter (756 mined PRs, 238 usable, 139 with a source gold file). On the 139 scored queries the premium is +0.068. It is positive on six of nine repositories and falls across τ\tau quartiles as in the core pools. The pool also hosts the agent evaluation (Section Q).

Methods Details

BM25 is BM25Okapi (rank_bm25) over chunks or whole files, with chunk scores max-pooled to the file. Dense retrieval uses sentence-transformers encoders from six families and thirteen checkpoints: jina-code-embeddings-{0.5b, 1.5b}, Qwen3-Embedding-{0.6B, 4B, 8B}, C2LLM-{0.5B, 7B}, F2LLM-{0.6B, 4B}, e5-{base, large}-v2 and gte-Qwen2-{1.5B, 7B}-instruct. Each chunk is encoded once, similarity is cosine and chunk scores are max-pooled to files. All encoders run without instruction prefixes. This uniform protocol penalizes models that depend on instructions, gte-Qwen2 most. Per query we report nDCG@10 and Recall@10 at file granularity, the premium Δ(q)=nDCG@10dense(q)−nDCG@10BM25(q)\Delta(q) = \mathrm{nDCG@10}_{\mathrm{dense}}(q) - \mathrm{nDCG@10}_{\mathrm{BM25}}(q) and τ(q)\tau(q) (§ 5.4).

Statistics. Bootstraps use 10,000 resamples and fixed seeds. Per-query comparisons use a paired bootstrap over per-query nDCG@10. Cross-slice analyses resample repositories, then tasks, and keep all encoders’ results for a task together. For gaps between pools, repository-block intervals are primary; query-level intervals are conditional on the 19 repositories pooled. An interval that includes zero is not evidence of no effect; outside the preset ±0.05 margin we report attenuation, not equivalence.

Extended Findings

Three results support the main text. BM25 is usable on raw text only after PR templates are stripped. The τ\tau-effect is weak per query but large across query groups, and it is exploratory. Under one label definition, the query pools are hard to tell apart.

PR-Boilerplate Stripping Recovers BM25

Templates break BM25 but not dense retrieval, so queries taken from PRs or issues must be stripped. On raw RAW-POST queries stripping roughly halves the premium (+0.369 to +0.175), entirely through BM25 gains. Without stripping, BM25 on django returns the same template-matching files (global_settings.py, docs/conf.py) for almost every query. The repeated headers, checklists and HTML comments dominate BM25’s IDF-weighted query, while the encoder largely ignores them. A hand audit of eight stripped queries confirms that titles and substantive prose survive.

The τ\tau-Effect in Detail

τ\tau predicts how large the premium is for a group of queries, not for a single query, and the evidence for it is exploratory.

τ\tau is MaxIDF on the tokens a query shares with its target. τ(q)\tau(q) (equation (1)) is the MaxIDF predictor of Hauff et al. [15] restricted to tokens shared with the first indexed, non-test gold file. We chose it among four variants because it transfers best across slices (Table 13); that choice was made on the same slices. Its query-only analogue τ~\tilde{\tau} carries little per-query signal.

τ\tau / QPP VARIANTPEARSON rrSPEARMAN ρ\rhoQ1Q_1–Q4Q_4MONOTONE?
τMaxIDF\tau_{\mathrm{MaxIDF}} (CHOSEN)−0.124−0.105+0.166YES
τmean5\tau_{\mathrm{mean5}}−0.050−0.080+0.099NO
τJaccard\tau_{\mathrm{Jaccard}}−0.113−0.120+0.136YES
τmean−all\tau_{\mathrm{mean-all}}−0.083−0.068+0.076YES
Additional baselines (gold-file count requires labels)
∣q∣|q| (QUERY LENGTH IN TOKENS)+0.040+0.045−0.026NO
ngoldn_{\mathrm{gold}} (GOLD-FILE COUNT)+0.046+0.079–NO

Table 13. Alternative τ\tau definitions and query-side baselines on the pooled CUR-PRE and original RAW-POST queries (n=604n = 604, source-file labels). Q1Q_1–Q4Q_4: the drop in mean premium from the lowest to the highest quartile of each variant. MaxIDF has the largest gap; whole-query predictors carry no comparable signal.

The premium falls with τ\tau in both cells. Pooled quartile means fall monotonically, from +0.217 to +0.050; the raw cell decays more steeply, and the curated cell has one interior wobble.

The association is weak per query but large per group. The τ\tau quartile explains about 2% of per-query variance, as much as knowing the repository; most of that variance is metric noise. Yet the drop from the lowest to the highest quartile is as large as the whole spread of the premium across encoders. τ\tau thus shows where the premium is but cannot route single queries (Section P).

Query Source and Vintage in Detail

Under one label rule the core pools are not distinguishable, vintage has no consistent effect, and the raw-PR premium recurs in other languages.

The pooled curated–raw gap was mostly a label contrast. With source-file labels the rawminus-curated gap is +0.031 and not significant under repository-block resampling; keeping the raw pools’ edited tests more than doubles it (Section B.3). The pools also differ in repositories, query length and gold-set size, so even the residual gap is descriptive. Wording does matter when the task is held fixed (Section O).

Vintage has no consistent effect. On 11 further Python repositories, we score PRs from 2022 and from 2026 against one pinned snapshot per repository. The date contrast has both signs across repositories and pools; pooled, it is +0.019. Contamination would predict a smaller post-cutoff premium, which we do not see, though a modest effect is not excluded.

The raw-PR premium recurs outside Python. At c=1,000c = 1{,}000 raw PRs keep a premium in JavaScript, TypeScript, Go and Rust; against the strongest BM25 we ran it remains on TypeScript and Rust but not on Go (Section B.6). Each language is a single repository.

Generation and Anchor Ablation

This appendix gives details for the two query interventions of § 5.2 and § 5.4; the tables below hold the numbers.

Setup. One generator (qwen35-122B, temperature 0, one frozen prompt) reads only the stripped query, never gold paths or patch content. It writes identifiers, file and module hints, APIs, error terms and a code skeleton, which we prepend to the query for BM25. A control arm reproduces the cached BM25 exactly.

Generation does more than tuning BM25. The anchors help BM25 more than any chunk size does. Over whole files, generate-then-grep matches the canonical encoder on every raw pool and beats it on curated queries (Table 2). The residuals cover zero but are not shown to be equivalent to zero. We therefore say that the premium is removed, but not that the two retrievers are equal.

The anchors carry evidence, not just more text. Corpus-based expansion (RM3) lowers BM25 rather than raising it. Embedding the expanded query helps the encoder far less than it helps BM25, and SPLADE and ColBERTv2 gain less from the same anchors (Table 12, Table 10). What helps is specific identifiers that a lexical index can match exactly.

A stronger encoder does not change the picture. Qwen3-Embedding-8B keeps a larger premium over the tuned BM25 on the raw pools, but generate-then-grep again matches or beats it in every pool (Table 10).

The size of the effect depends on the generator. Across five generators from 4B to 122B the direction always holds, but only the largest turns the curated premium negative. gemma-4-31B-it and deepseek-v4-flash close about 80% of it; Qwen3-8B closes a little over half against the canonical encoder and about two thirds against the encoder built from its own weights.

Memorization does not explain it. Generation also removes the premium on post-cutoff PRs that appear in no benchmark, and the effect survives when every file reference is stripped from the anchors. On held-out PRs opened after the generator’s release (116 queries, seven repositories), generate-then-grep still beats dense retrieval, with an interval that excludes zero. Naming the gold file is not needed either. The generator names it more often for curated than for raw queries, yet Raw-Pre’22 closes its whole premium although the gold file is named for only half its queries.

Generation is cheaper for rarely searched repositories. One Qwen3-8B call takes about 1.5 s per query. The dense index needs a one-off encoding pass per repository (about four minutes for django). Generation is therefore cheaper below roughly 158 queries per repository. Failed generations fall back to the original query, which biases against generation.

Removing anchors hurts both retrievers. We replace the five rarest tokens a query shares with its gold file by synthetic strings of the same length and case pattern. The placebo replaces the same number of equally rare tokens absent from the gold file, and a guard arm reproduces the cached baseline exactly. If dense retrieval matched meaning without those tokens, its premium should rise. On curated and raw Python queries it does not: both retrievers lose a third to almost half of their nDCG@10, by similar amounts. The premium does rise on TypeScript, and on Rust without a resolved interval. Most replaced tokens are ordinary words rather than identifiers. Our attempt at meaning-preserving substitutes failed its fidelity check. The edit thus rules out matching that ignores these tokens, but it cannot tell the loss of task meaning. The evidence for the mechanism therefore comes from adding anchors. An earlier version based on LLM rewrites (DESC) removes too much of the query to be informative.

Decision Analysis: Protocol and Lockbox

The preregistered router failed on the lockbox, while both fixed fusions, configured on development data alone, held.

What was frozen. The actions are dense retrieval (jina-code-0.5b, c=1,000c = 1{,}000), whole-file BM25 (the best pooled chunk size on development data) and their RRF (k=60k = 60). Development is CUR-PRE and RAW-POST (10 repositories), evaluated leave one repository out. The router uses gradient-boosted regressors over 48 query, list-agreement and score features; it serves RRF unless another action is predicted to beat it. Of about 30 candidates, it was the only one whose development gain excluded zero. We froze it with its hash and a written criterion: on the pooled lockbox, the lower 95% bound of routed minus always-RRF must be positive. An earlier router had already failed the same kind of test on 29 held-out repositories.

The router fails; the fusions hold. The lockbox (539 DeepSWE and cross-language TypeScript, Go and Rust queries, 91 repositories) was scored once on the labels as released. The router does not beat always fusing, and the lockbox is not reused for routing. Both fusions beat dense retrieval, and re-scoring with source-file labels changes no conclusion.

The fusion gain is not carried by three repositories. 80% of lockbox queries come from the TypeScript, Go and Rust slices. With the 91 repositories weighted equally (post hoc), both fusions keep their gain over dense retrieval, even with those three slices dropped. The generation tier’s extra gain over plain fusion does not survive; it comes from the three single-repository slices.

Data Release and Personal Information

REPOPREMIUM redistributes only public GitHub content (problem statements, PR descriptions and code at pinned SHAs) under the source repositories’ licenses. Released queries pass the same boilerplate stripper used for evaluation, and a release-time scrubber removes every email address. @-mentions are kept because they are lexical tokens; most are bots, package scopes or decorators. We redistribute no commit histories or non-public data, and make no attempt to link or profile individuals. Reported numbers were computed on the unredacted text; BM25 rankings differ for a few released queries. The use of LLMs in this work is described in Section H.

Reproducibility and Use of LLMs

Reproducibility. All pools are built from public GitHub data at pinned SHAs (Table 1). The code and data releases (links on the first page) contain the query and qrel files, the PR-boilerplate stripper, the cached per-query results, and the scripts that produce every table and figure. It also contains the agent-evaluation harness: the clean-environment builder, PATH shims and bypass auditor (Section Q.2). Bootstrap intervals use 10,000 resamples with fixed seeds; all encoders are open-weight checkpoints pinned by revision.

Use of large language models. LLMs are part of the experiments. They drive the agent evaluation (Section Q) and generate the query anchors of § 5.2, an experimental condition that never enters the corpus or the qrels. They also write the DESC rewrites (Section K) and produce and check the curated views of the matched replication (Section O). No LLM generated, labeled or filtered the queries or qrels of the four core pools. Coding assistants were used to implement analysis tooling, run experiments and draft revisions; the authors reviewed and verified all results and text.

Chunk-Size and -Definition Ablations

Chunk size decides the baseline. BM25 keeps improving up to whole files while the encoder does not, so the common c=1,000c = 1{,}000 protocol favors dense retrieval. Against BM25 at its own best size, no encoder keeps a significant premium. Among τ\tau variants, MaxIDF is the most informative; whole-query predictors carry no signal.

BM25 needs large windows; the encoder does not. BM25 rises with chunk size: it levels off from c=4,000c = 4{,}000 on CUR-PRE and peaks at whole files on RAW-POST. The canonical encoder stays flat because it already truncates long inputs (Table 14). Only the canonical encoder was swept.

cc (CHARS)1,0001,5002,0003,0004,0006,00010,000FILE
BM25 NDCG@10 (CUR-PRE)0.4110.4270.4430.4650.4880.4940.5000.498
DENSE NDCG@10 (CUR-PRE)0.5320.5220.5310.530––––
BM25 NDCG@10 (RAW-POST)0.4380.4660.4690.4750.4610.4850.4960.539

Table 14. Chunk-window sweep (nDCG@10, source-file labels; CUR-PRE n=472n = 472, RAW-POST n=132n = 132). “File” scores each file as one window. Dense is the canonical encoder, swept on CUR-PRE only. Bold: BM25’s best size per pool.

Rebasing removes every significant premium. Scored against BM25 at its best size, no encoder of thirteen keeps a significantly positive premium on either pool, and seven fall significantly below zero on CUR-PRE (Table 15). BM25’s size was picked on the evaluation queries, so these rebased premiums are lower bounds for dense retrieval.

EncoderCur-Pre (curated) @1,000Cur-Pre (curated) tunedCur-Pre (curated) repo CIRaw-Post (raw) @1,000Raw-Post (raw) tunedRaw-Post (raw) repo CI
Jina-Code-1.5B+0.124+0.035[−0.026, +0.075]+0.192+0.091[−0.009, +0.230]
Jina-Code-0.5B (canonical)+0.121+0.032[−0.037, +0.103]+0.175+0.074[−0.027, +0.219]
Qwen3-Emb-8B+0.103+0.015[−0.021, +0.085]+0.155+0.054[−0.023, +0.174]
E5-Base-V2+0.058−0.030[−0.105, +0.014]+0.084−0.017[−0.072, +0.079]
Qwen3-Emb-4B+0.051−0.038[−0.070, +0.018]+0.105+0.004[−0.075, +0.127]
GTE-Qwen2-7B+0.049−0.040[−0.135, +0.003]+0.029−0.072[−0.170, +0.056]
F2LLM-0.6B+0.041−0.048[−0.095, −0.006]+0.091−0.010[−0.068, +0.090]
E5-Large-V2+0.035−0.054[−0.124, −0.017]+0.109+0.009[−0.069, +0.101]
F2LLM-4B+0.016−0.073[−0.135, −0.015]+0.063−0.038[−0.114, +0.070]
Qwen3-Emb-0.6B+0.008−0.081[−0.112, −0.023]+0.100−0.001[−0.063, +0.089]
C2LLM-7B−0.042−0.130[−0.202, −0.072]+0.012−0.089[−0.188, +0.026]
C2LLM-0.5B−0.043−0.132[−0.178, −0.106]−0.025−0.126[−0.225, +0.012]
GTE-Qwen2-1.5B−0.166−0.254[−0.314, −0.227]−0.078−0.178[−0.291, −0.031]

Table 15. Premium against BM25 at c=1,000c = 1{,}000 and at its own best chunk size (“tuned”: c=10,000c = 10{,}000 on Cur-Pre, whole files on Raw-Post); dense at c=1,000c = 1{,}000 for every encoder; repository-block 95% CIs on the tuned premium; source-file labels; Raw-Post: the 132 original queries (§ 4). Against a tuned BM25 no encoder keeps a significantly positive premium on either pool.

MaxIDF is the most informative τ\tau. Of four shared-token variants, MaxIDF has the largest gap between the lowest and highest τ\tau quartile and transfers best across slices (Section N). Query length and gold-set size carry no signal, so what matters is restricting τ\tau to tokens shared with the target file (Table 13).

The PR-Boilerplate Stripper

The stripper processes the raw PR description line by line. It deletes HTML comments left by the PR template and drops entire markdown sections whose header matches a boilerplate name (“Trac ticket number”, “AI Assistance Disclosure”, “Checklist”, . . .). It removes the #### prefix from kept headers so they read as prose. It also drops checkbox lines (- [ ] / - [x]) and collapses runs of blank lines. A real Raw-Post example, verbatim from the released stripper:

Example (django PR 21426; 1,730 → 193 chars, 89% reduction). Raw (excerpt):

Fixed #37130 -- Skipped DB cache deletion
when culling offset is zero.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding
Trac ticket number. -->
<!-- Or delete the line and write "N/A -
typo" for typo fixes. -->

ticket-37130

#### Branch description
Added a condition to bypass the deletion
query when [...]

#### AI Assistance Disclosure (REQUIRED)
<!-- Please select exactly ONE of the
following: -->
- [x] **No AI tools were used** in
preparing this PR.
- [ ] **If AI tools were used**, I have
disclosed which ones [...]

#### Checklist
- [x] This PR follows the [contribution
guidelines](https://[...])

Stripped (complete):

Fixed #37130 -- Skipped DB cache deletion
when culling offset is zero.

Branch description
Added a condition to bypass the deletion
query when the calculated culling offset
evaluates to zero.

The Behavior-Only Probe (Desc)

DESC holds 125 queries (87 from CUR-PRE, 38 from RAW-POST, all ten repositories). For each, gemma-4-31B-it (temperature 0) rewrote the source query as a one- or two-sentence description of the behavior, with every identifier, path, error string and ticket reference removed. Gold files come from the source query. Because a full description shrinks to a sentence or two, DESC removes much more than anchors. We use it only as one slice of the decay analysis (Section N) and as an anchor-removal test that also loses other query content (Section E).

Curated Multilingual Queries

Curated issue-symptom statements keep the dense premium but lose its τ\tau-dependence: they share rare tokens with the target file as often as raw PRs do, yet those tokens rarely anchor BM25. The τ\tau-effect is therefore specific to lexically anchored queries.

Setup. We score 134 human-verified Multi-SWE-bench issue statements on vuejs/core, mui/material-ui and tokio-rs/tokio against the same corpora, with two encoders.

The premium stays; the slope disappears. The premium is +0.03 to +0.18, but its pooled slope on τ\tau is flat (+0.002, against −0.164 for the raw language slices).

Rare shared tokens no longer anchor BM25. These queries have the same τ\tau as raw PRs. Yet three quarters return no grep hit in the top ten, against 40% for raw PRs, and BM25 scores far lower (Table 16).

RepoQuery typeτ\tauGrepBM25Grep= 0
Same-repo comparison (identical corpus)
vuejs/coreRaw (PR)6.250.3970.51243%
vuejs/coreCurated (Issue)6.130.1040.13084%
tokioRaw (PR)6.370.3860.56934%
tokioCurated (Issue)7.440.4200.68038%
Pooled
Raw (4 languages)Raw (PR)7.050.3250.45340%
Multi-SWE-BenchCurated (Issue)6.930.1370.25076%

Table 16. Curated issue-symptom queries carry the same τ\tau as raw PR queries but retrieve far worse lexically. grep and BM25 are nDCG@10; grep= 0 is the share of queries with no grep hit in the top ten; jina-code-0.5b. τ=4.8\tau=4.8: “Skipped DB cache deletion when culling offset is zero” (raw PR), BM25 0.00 vs. dense 1.00. The exemplars are extreme by construction; the claim of Section N is about the mean curve.

Samples are small, query type is confounded with language, and tokio does not show the gap. We read this as a limit on scope, not a falsification.

Per-Repository Results and Example Queries

The premium is not driven by one repository: dense retrieval leads BM25 at c=1,000c = 1{,}000 in every repository of CUR-PRE but requests, and in all nine repositories of RAW-POST (Table 17). Sizes range from near zero to over 0.3.

REPOnnBM25DENSEΔ\Delta
CUR-PRE (curated, pre-cutoff)
astropy210.4900.563+0.073
django2260.4100.528+0.119
matplotlib340.3780.593+0.215
pylint80.2910.443+0.152
pytest190.2930.423+0.130
requests80.5430.534−0.009
scikit-learn180.5610.586+0.025
sphinx440.3040.489+0.185
sympy720.4680.532+0.064
xarray220.4040.615+0.210
RAW-POST (raw PRs, post-cutoff)
astropy140.4050.495+0.089
django450.3570.572+0.215
matplotlib130.3290.690+0.361
pylint320.4500.642+0.192
pytest340.5860.634+0.048
requests270.3740.579+0.205
scikit-learn360.3480.595+0.247
sympy570.5750.588+0.013
xarray300.4160.592+0.176

Table 17. Per-repository nDCG@10 for BM25, dense retrieval and the premium (jina-code-0.5b, c=1,000c = 1{,}000, source-file labels). Dense retrieval leads in every repository but one curated requests slice of eight queries.

Exemplar Queries of fig:decay

The dotted verticals in Figure 4 mark three real queries, from rare-token to common-token. (A), τ=9.8\tau=9.8: “simplify_regexp doesn’t replace trailing groups”, BM25 1.00 vs. dense 0.33. (B), τ=8.5\tau=8.5: “index refactor: more _coord_names than _variables on Dataset”, BM25 0.61 vs. dense 0.83. (C),

τ=4.8\tau=4.8: “Skipped DB cache deletion when culling offset is zero” (raw PR), BM25 0.00 vs. dense 1.00. The exemplars are extreme by construction; the claim of Section N is about the mean curve.

Premium Decay across Slices (Exploratory)

Across 38 evaluation slices the premium falls with τ\tau in all but one, and one common slope fits them. Because the predictor and its normalization were chosen on these slices, this is exploratory; a preregistered test on held-out repositories had the predicted sign but missed its criterion.

Model and slices. We fit the premium as one common function of τ\tau’s within-repository percentile (IDF is not comparable across corpora) plus a per-slice offset. The slices cover the five Python mining regimes, all thirteen encoders on Cur-Pre and Raw-Post, the held-out pool and four other languages (9,339 query-slice observations).

One slope fits all slices. The per-slice slope is negative in 37 of 38 slices. Encoders sharing a pool are not independent, so the faier count is nine of ten distinct query pools; the exception, Raw-Pre’23, is small and near zero. The common slope is −0.165 nDCG@10 across the full percentile range; intervals resample repositories, then tasks, so encoders sharing a task move together. Dropping any one repository barely changes the slope. A homogeneity test does not reject a single slope. Per-slice slopes are noisy, so we make the claim for the pooled data only (Figure 5).

Figure 5. The premium-decay regularity (exploratory). (a) Quintile means of Δ\Delta against the within-repository τ\tau percentile for the 38 evaluation slices (thin lines, colored by slice family), each centered on its own mean; black: pooled mean with its 95% band. (b) Per-slice slopes with bootstrap intervals; 37 of 38 are negative, and the dashed line marks the common slope (−0.165-0.165). Source-file labels.

The preregistered held-out test was not met. We applied the frozen predictor to the 29 repositories of the matched replication (Section O) that no slice used. The slope is negative in all seven sub-slices (−0.109 pooled), but its interval includes zero. With about ten tasks per repository, the within-repository percentile is too coarse to resolve a slope of this size.

A Matched Replication at Scale

Holding the change fixed, raw PR wording raises the premium over a curated rewrite of the same change, for two encoders from different families. A date effect of similar size cannot be excluded, and the curated premium itself depends on the encoder.

Design. We mine 360 PRs from 36 repositories in Go, Java, Python, Rust and TypeScript/JavaScript, five before and five after the encoders’ cutoffs per repository; 351 keep an indexed, non-test gold file and are scored. Each task is queried twice, against one gold set and one corpus at its base commit: once with the raw PR text and once with a curated rewrite. qwen35-122B wrote the rewrite from the PR text without seeing gold paths or patches, and a separate call to the same model checked its fidelity. The curated view is thus an LLM rewrite, not human curation. A repository-disjoint development/test split was frozen before analysis.

Wording moves the premium within a task. Raw wording raises the premium for both jina-code-0.5b and Qwen3-Embedding-0.6B, in about 70% of repositories and on the held-out test split (Table 18). Both views share one gold set, so the label rule cannot create this contrast. This is our strongest evidence that wording moves the premium; the pooled ten-repository comparison no longer separates curated from raw pools once labels are consistent (Section D.3).

JINA-CODE-0.5BQWEN3-EMB-0.6B
WORDING EFFECT+0.091 [+0.024, +0.163]+0.111 [+0.049, +0.176]
TEST SPLIT+0.090 [+0.011, +0.172]+0.119 [+0.043, +0.201]
REPOS POSITIVE26/3625/36
PREMIUM, CURATED VIEW+0.088 [+0.029, +0.145]−0.015 [−0.077, +0.048]
PREMIUM, RAW VIEW+0.178 [+0.109, +0.245]+0.096 [+0.026, +0.166]
DATE EFFECT, CURATED−0.044 [−0.147, +0.058]−0.048 [−0.156, +0.055]
DATE EFFECT, RAW−0.028 [−0.140, +0.081]−0.048 [−0.168, +0.068]
|WORDING| − |DATE|, CURATED+0.046 [−0.075, +0.127]+0.063 [−0.064, +0.143]

Table 18. Matched scale-up (351 of the 360 tasks; the other nine have no indexed, non-test gold file; 36 repositories, five languages; nDCG@10, source-file labels). Brackets: hierarchical bootstrap 95% CIs resampling repositories, then tasks. The date effect is the within-repository post-minus-pre contrast.

Only the raw premium survives a change of encoder. The canonical encoder keeps a premium on both views, but the second encoder’s curated premium is indistinguishable from zero, as it is on never-mined DeepSWE descriptions (Section R). This is what we expect if the premium is a property of the query.

Date is not separated from wording. The post-minus-pre contrast points the way contamination predicts and is about half the size of the wording effect. Its interval includes zero, and a direct comparison of the two sizes does not separate them. By language, all ten estimates point the same way, but only Python (and, for the second encoder, TypeScript/JavaScript and Rust) excludes zero on its own. We therefore do not claim a five-language replication.

When BM25 Wins

BM25 wins a sizable minority of queries, but nothing we tried could tell which ones from the query, so none of the predictors we tested could use this headroom.

The headroom is real. BM25 strictly beats dense retrieval on about a quarter of CUR-PRE and a fifth of RAW-POST queries, and picking the better list per query would add about 0.09 nDCG@10 over dense alone. The headroom is in the ranking order, not in coverage: the union of both top-10 lists finds few gold files that dense retrieval misses.

The winner cannot be read from the query. We tried classifiers on seven structural query features (with or without τ^\hat{\tau} and length), a classifier on the query’s embedding, and two zero-shot LLM judges. Under leave-one-repository-out evaluation, none predicts BM25 wins better than chance. Tuned on the training repositories, each falls back to (nearly) always choosing dense (Table 19). Untuned, the judges pick keyword search for most queries and lose over 0.1 nDCG@10. Even τ\tau, which uses the gold file, separates BM25 wins only weakly. The RAW-POST sample is small, so a modest signal on raw queries cannot be excluded.

Agent-Driven End-to-End Evaluation

Retrieval quality does not decide agent localization. For a SWE-agent on the held-out pool, the choice of LLM moves exact-match far more than the choice of search tool. Our first version of this evaluation was invalid because the merged fix was reachable through git; all numbers here come from the clean rerun.

Search Tools at a Fixed LLM

At a fixed LLM the search tool barely matters: four very different tools land within about 0.04 exact-match of each other, and no difference between them is resolved.

Setup. We draw stratified samples from the held-out pool (Section B.7), balanced across its nine repositories: 30 queries for the model comparison and a nesting draw of 96 for the search-tool comparison. The agent is mini-swe-agent ([37]) in its raw-shell configuration (temperature 0, 30 steps, 900 s per query). It sees the raw problem statement and is scored on the files its final diff touches against the gold files: precision, recall, any-hit (at least one gold file) and exact-match (the gold set exactly).

The model matters most. Across four open-weight configurations (Table 20), exact-match ranges from 0.033 (gemma-4-31B-it) to 0.300 (kimi-k3). Turning off qwen35-122B’s reasoning phase changes exact-match by an amount we cannot resolve, so the data do not show that thinking helps. The spread across families mixes family with scale: it shows that the served model matters, but not why. Low-recall trajectories share one failure: the agent commits to a wrong file early and keeps editing it.

Q.1 Search Tools at a Fixed LLM

At a fixed LLM the search tool barely matters: four very different tools land within about 0.04 exact-match of each other, and no difference between them is resolved.

Design. We fix the LLM (qwen35-122B, thinking on) and replace the shell’s search with one tool, repo_search. Its backend is BM25, the canonical dense encoder, or a router that uses τ^\hat{\tau} to pick one of the two per query. The restriction is enforced: find, grep, rg and git grep are shadowed on PATH, and attempts around them are logged.

No tool wins. The unrestricted shell scores lowest in point estimate, but no paired difference ex-

METRICQWEN35-122B (THINK)QWEN35-122B (NO-THINK)KIMI-K3 (THINK)GEMMA-4-31B-IT
MEAN PRECISION0.6130.5610.6000.350
MEAN RECALL0.3530.2660.5100.468
ANY-HIT RATE0.6330.6000.6330.733
EXACT-MATCH RATE0.1330.1000.3000.033

caption: Model comparison on the 30-query sample of the held-out pool (clean rerun; fresh single-commit environments, no network). The no-think column runs the same weights with the reasoning phase disabled; kimi-k3 replaces the withdrawn kimi-k2.6 of the superseded run and is reported in thinking mode only.

:::

cludes zero. The ordering of the three tools also changes between the 30- and 96-query samples (Table 21). At 30 steps, what limits the agent is which file the LLM commits to, not how it found candidates. This does not show that agents write their own anchors; that argument rests on § 5.2 and Section R.

METRICRAW SHELLBM25 INDEXDENSEROUTER
MEAN PRECISION0.5170.4750.5190.541
MEAN RECALL0.3590.3850.3700.392
ANY-HIT RATE0.5730.6040.6150.625
EXACT-MATCH RATE0.1350.1670.1460.177

Table 21. Search-tool comparison at a fixed LLM (qwen35-122B, thinking on), 96 queries, clean rerun. Only the search affordance varies; the tool restriction is enforced on PATH.

The τ\tau-effect reaches the agent on any-hit only. A preregistered test predicted that the dense tool beats the BM25 tool on queries with few rare anchors (bottom τ^\hat{\tau} quartile) and not on the top quartile. On any-hit it holds: dense is well ahead at the bottom and behind at the top, so the flat pooled comparison hides a large swing. On exact-match the contrast is unresolved: finding some gold file more often does not mean reproducing the whole gold set.

Task-Environment Contamination

Our first version of this evaluation leaked the answer, and the leak reversed its main result. We report it because the mistake is easy to make in any benchmark built from merged changes.

What went wrong. Tasks were materialized with git worktree add at the base commit. This pins the files but shares the object store, so the merged fix stayed reachable through git log --all or git show. In 8 of 30 shell trajectories the agent read a later commit covering the whole gold set. The tool restriction was also only a request in the prompt, which the agent ignored in most trajectories.

The fix, and what it changed. Each task is now exported with git archive into a fresh repository with one synthetic commit and no remotes, checked per query. The tool restriction is enforced, and the shell has no network access; this also closes the route by which one query had fetched its own pull-request diff. Raw-shell exact-match fell from 0.333 to 0.133, while the BM25 tool, which had nothing to gain from the leak, stayed at 0.167. The apparent advantage of the shell was therefore the leak.

Limitations. The model comparison uses 30 queries, so only its largest differences resolve. Scoring is file-level: any-hit is a lenient ceiling and no execution-based check is run. The search tools return 20 chunk hits that are not deduplicated to files.

Never-Mined Queries (DeepSWE)

DeepSWE task descriptions are written to specify work, not mined from a completed change. On them the premium does not transfer and BM25 wins far more often, which fits the view that the premium depends on how queries are written.

Setup. 109 of DeepSWE’s 113 tasks pass our qrel audit (88 repositories, five languages: Python, JavaScript, TypeScript, Go and Rust). The gold files are those of one reference patch, so we measure reference-edit localization rather than exhaustive relevance. The retrieval harness is that of § 4.

The premium does not transfer. The canonical encoder’s premium, +0.121 and +0.152 on our curated and raw pools, is not distinguishable from zero here, and Qwen3-Embedding-0.6B falls below BM25 (Table 22). This is the sharpest limit on the premium we found, and it matches the within-task wording effect of Section O.

ENCODERDENSEBM25GREPDENSE−BM25BM25 WINS
JINA-CODE-0.5B0.4810.4450.435+0.036 [−0.030, +0.100]37.6%
QWEN3-EMB-0.6B0.3670.4450.435−0.078 [−0.144, −0.013]52.3%

Table 22. Retrieval on DeepSWE’s original task descriptions (nDCG@10; 68 tasks in 56 repositories whose corpus is identical across the two encoder runs, so the lexical columns are shared). Brackets: repository-clustered paired bootstrap 95% CIs.

Lexical rescue is a property of the query. BM25 strictly wins 38–52% of queries, against about a fifth to a quarter on our pools, and the two encoders’ BM25-win sets largely coincide.

Corpus size does not explain it. If anything the premium shrinks as corpora grow, though no size tercile resolves.

References

  1. [1]Ibrahim Abdelaziz, Julian Dolby, Jamie P. McCusker, and Kavitha Srinivas. 2021. A toolkit for generating code knowledge graphs. In K-CAP ’21: Knowledge Capture Conference, Virtual Event, USA, December 2-3, 2021, pages 137–144. ACM.arxiv.org/abs/2002.09440
  2. [2]Negar Arabzadeh, Maryam Khodabakhsh, and Ebrahim Bagheri. 2021. BERT-QPP: contextualized pre-trained transformers for query performance prediction. In CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, pages 2857–2861. ACM.DOI
  3. [3]Stefano Campese, Ivano Lauriola, and Alessandro Moschitti. 2024. Pre-training methods for question reranking. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 - Volume 2: Short Papers, St. Julian’s, Malta, March 17-22, 2024, pages 469–476. Association for Computational Linguistics.DOI
  4. [4]David Carmel and Elad Yom-Tov. 2010. Estimating the query difficulty for information retrieval. In Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010, Geneva, Switzerland, July 19-23, 2010, page 911. ACM.
  5. [5]Tongfei Chen and Colin Flaherty. 2025. #1 open-source agent on SWE-Bench Verified by combining Claude 3.7 and O1. Augment Code blog. Accessed 2026-10-14.
  6. [6]Zhaoling Chen, Robert Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor K. Prasanna, Arman Cohan, and Xingyao Wang. 2025. Locagent: Graph-guided LLM agents for code localization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 8697–8727. Association for Computational Linguistics.arxiv.org/abs/2503.09089
  7. [7]Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19-23, 2009, pages 758–759. ACM.DOI
  8. [8]Stephen Cronen-Townsend, Yun Zhou, and W. Bruce Croft. 2002. Predicting query performance. In SIGIR 2002: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 11-15, 2002, Tampere, Finland, pages 299–306. ACM.
  9. [9]Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 2025. Swe-bench pro: Can AI agents solve long-horizon software engineering tasks? CoRR, abs/2509.16941.arxiv.org/abs/2509.16941
  10. [10]Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1536–1547. Association for Computational Linguistics.arxiv.org/abs/2002.08155
  11. [11]Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: sparse lexical and expansion model for first stage ranking. In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 2288–2292. ACM.arxiv.org/abs/2107.05720
  12. [12]Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 1762–1777. Association for Computational Linguistics.arxiv.org/abs/2212.10496
  13. [13]Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7212–7225. Association for Computational Linguistics.arxiv.org/abs/2203.03850
  14. [14]Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. Graphcodebert: Pre-training code representations with data flow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769–6781. Association for Computational Linguistics.
  15. [15]Claudia Hauff, Djoerd Hiemstra, and Franciska de Jong. 2008. A survey of pre-retrieval query performance predictors. In Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM 2008, Napa Valley, California, USA, October 26-30, 2008, pages 1419–1420. ACM.DOI
  16. [16]Ben He and Iadh Ounis. 2004. Inferring query performance using pre-retrieval predictors. In String Processing and Information Retrieval, 11th International Conference, SPIRE 2004, Padova, Italy, October 5-8, 2004, Proceedings, volume 3246 of Lecture Notes in Computer Science, pages 43–54. Springer.DOI
  17. [17]Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. 2021. Cosqa: 20, 000+ web queries for code search and question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACLIJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5690–5700. Association for Computational Linguistics.
  18. [18]Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. 2026. Deepswe: Measuring frontier coding agents on original, long-horizon engineering tasks. CoRR, abs/2607.07946.arxiv.org/abs/2607.07946
  19. [19]Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Code-searchnet challenge: Evaluating the state of semantic code search. CoRR, abs/1909.09436.arxiv.org/abs/1909.09436
  20. [20]Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query expansion by prompting large language models. CoRR, abs/2305.03653.arxiv.org/abs/2305.03653
  21. [21]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.arxiv.org/abs/2310.06770
  22. [22]Chris Kamphuis, Arjen P. de Vries, Leonid Boytsov, and Jimmy Lin. 2020. Which BM25 do you mean? A large-scale reproducibility study of scoring variants. In Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part II, volume 12036 of Lecture Notes in Computer Science, pages 28–34. Springer.
  23. [23]Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769–6781. Association for Computational Linguistics.arxiv.org/abs/2004.04906
  24. [24]Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 39–48. ACM.arxiv.org/abs/2004.12832
  25. [25]Daria Kryvosheieva, Saba Sturua, Michael Günther, Scott Martens, and Han Xiao. 2025. Efficient code embeddings from code generation models. CoRR, abs/2508.21290.arxiv.org/abs/2508.21290
  26. [26]Victor Lavrenko and W. Bruce Croft. 2001. Relevance-based language models. In SIGIR 2001: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, September 9-13, 2001, New Orleans, Louisiana, USA, pages 120–127. ACM.
  27. [27]Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Hao Zhang, Xinyi Dai, Yasheng Wang, and Ruiming Tang. 2025. Coir: A comprehensive benchmark for code information retrieval models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 22074–22091. Association for Computational Linguistics.arxiv.org/abs/2407.02883
  28. [28]Jimmy Lin. 2018. The neural hype and comparisons against weak baselines. SIGIR Forum, 52(2):40–51.DOI
  29. [29]Jimmy Lin and Xueguang Ma. 2021. A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. CoRR, abs/2106.14807.
  30. [30]Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 2356–2362. ACM.
  31. [31]Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, and 3 others. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  32. [32]Iain Mackie, Shubham Chatterjee, and Jeffrey Dalton. 2023. Generative relevance feedback with large language models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pages 2026–2031. ACM.
  33. [33]Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 157–165. Association for Computational Linguistics.arxiv.org/abs/2203.08242
  34. [34]Mike A. Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, Estefany Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Di Lu, Orfeas Menis-Mastromichalakis, Zhiwei Xu, and 65 others. 2026. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. proceedings.iclr.cc.
  35. [35]Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. MTEB: massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 2006–2029. Association for Computational Linguistics.
  36. [36]OpenAI. 2024. Introducing SWE-bench Verified. OpenAI research note. Human-verified subset of SWE-bench.
  37. [37]Princeton NLP. 2025. mini-swe-agent: a minimal, hackable SWE-agent. https://github.com/SWE-agent/mini-swe-agent. Version 2.4.2.
  38. [38]Revanth Gangi Reddy, Tarun Suresh, JaeHyeok Doo, Ye Liu, Xuan-Phi Nguyen, Yingbo Zhou, Semih Yavuz, Caiming Xiong, Heng Ji, and Shafiq Joty. 2026. Swerank: Software issue localization with code ranking. In The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. proceedings.iclr.cc.arxiv.org/abs/2505.07849
  39. [39]Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Extaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, volume EMNLP 2023 of Findings of ACL, pages 10776–10787. Association for Computational Linguistics.
  40. [40]Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 3715–3734. Association for Computational Linguistics.
  41. [41]Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  42. [42]Liang Wang, Nan Yang, and Furu Wei. 2023a. Query2doc: Query expansion with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 9414–9423. Association for Computational Linguistics.arxiv.org/abs/2303.07678
  43. [43]Yue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023b. Codet5+: Open code large language models for code understanding and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 1069–1088. Association for Computational Linguistics.
  44. [44]Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. CoRR, abs/2407.01489.arxiv.org/abs/2407.01489
  45. [45]Siqiao Xue, Zihan Liao, Jin Qin, Ziyin Zhang, Yixiang Mu, Fan Zhou, and Hang Yu. 2026a. Beyond retrieval: A multitask benchmark and model for code search. CoRR, abs/2605.04615.arxiv.org/abs/2605.04615
  46. [46]Siqiao Xue, Shuxuan Liu, and Ning Hu. 2026b. Zoowork-shopranker: An open, preference-aligned e-commerce reranker. CoRR, abs/2609.31002.arxiv.org/abs/2609.31002
  47. [47]Siqiao Xue and Chunkxue Xu. 2026. ZooClaw-FashionSigLIP2: Distilled fine-tuning for robust fashion retrieval. CoRR, abs/2606.27708.arxiv.org/abs/2606.27708
  48. [48]Siqiao Xue, Fan Zhou, Yi Xu, Hongyu Zhao, Shuo Xie, Qingyang Dai, Caigao Jiang, James Y. Zhang, Jun Zhou, Dacheng Xiu, and Hongyuan Mei. 2023. Weaverbird: Empowering financial decision-making with large language model, knowledge base, and search engine. CoRR, abs/2308.05361.arxiv.org/abs/2308.05361
  49. [49]Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically examining the "neural hype": Weak baselines and the additivity of effectiveness gains from neural ranking models. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 1129–1132. ACM.
  50. [50]Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nalippogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. 2025. Swe-bench goes live! In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.
  51. [51]Shaoqiu Zhang, Yuhang Wang, Jialiang Liang, Yuling Shi, Wenhao Zeng, Maoquan Wang, Shilin He, Ningyuan Xu, Siyu Ye, Kai Cai, and Xiaodong Gu. 2026a. Swe-explore: Benchmarking how coding agents explore repositories. CoRR, abs/2606.07297.arxiv.org/abs/2606.07297
  52. [52]Ziyin Zhang, Yuyu Guo, Ying Zhang, Siqiao Xue, Zihan Liao, Jin Qin, Wenjie Yang, Siyuan Yang, Shijie Li, Cheng Chen, Hao Wu, Zhaoyang Zhu, Ruipeng Li, Zhaokun Jiang, Lei Lei, Rui Wang, Hang Yu, and Rui Wang. 2026b. From fragmentation to fusion: A survey and roadmap for omnimodal embeddings. CoRR, abs/2611.00172.arxiv.org/abs/2611.00172
  53. [53]Jian Zhou, Hongyu Zhang, and David Lo. 2012. Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports. In 34th International Conference on Software Engineering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland, pages 14–24. IEEE Computer Society.

Paper details

vlnc:W_uc4u86vu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents