WorksComputer Science
Beyond Document Relevance: Evaluating Citation-Set Structure in Citation Recommendation
Abstract
When authors of a scientific paper cite several papers at a single point, they are often not aiming to cite the exact same claim repeatedly. Citation sets include a high degree of author judgement — they are often chosen together to cover a conceptual space, reference both a problem and its main methods, or pair a method with a result it is usually compared to. Despite this, citation recommendation, as it is usually posed, ignores this set-level structure, and instead scores each target in isolation to maximize its independent semantic similarity to the citing sentence. We show that in the case of recommending a citation set, this may be the wrong approach. We introduce LOCUS, a controlled evaluation that determines whether or not a retriever can tell which of a paper’s own citations belong at one sentence rather than another one in the text, and that looks beyond the use of topic-level similarity. A simple, training-free co-citation reranker built from past citation structure improves Mean Reciprocal Rank (MRR) by +0.104 to +0.124 across four retrievers. The gain largely comes from 61.3% of cases where the candidate and an already-known citation are linked in the citation graph, while using citations from other locations in the same paper hurts performance. In corpus-wide retrieval with a self-retrieved first citation, SPECTER2 still improves Recall@20 by +0.075. Further, once one citation is known, the retriever is 2.6× more effective at recovering the remaining members at Recall@1.
Introduction
Citation recommendation and citation sets
Citations very often come in sets of two or more. In arXiv papers in Computer Science, Physics, Mathematics, and more [8], 40% of citing sentences cite two or more papers at once. These sets’ structures reflect a wide array of human epistemic judgements, from grouping the dominant approaches to an open problem, to pairing a formal theoretical claim with its subsequent empirical validations [12]. In effect, citation sets often support a claim by providing disparate sources of evidence for it [10].
Despite the fact that such sets are prevalent, local citation recommendation (LCR), which treats citation as isolated retrieval of the correct paper, is the most prominent task in this field. In standard LCR, the text surrounding a citation marker (citance) serves as a query, and a single ground-truth cited paper serves as the target. By construing this as an isolated task, LCR implicitly assumes that different members of a citation set are semantically interchangeable. As such, many retrieval methods designed for LCR rely on coarse notions of similarity via dense embedding models trained on text and paper-level citation graphs. These models rank candidates by cosine similarity and are very good at capturing topical relevance. But within a single citation set, topical relevance is a given – papers cited in the same bracket are far more topically similar than papers cited merely in the same paper [5]. So the real task for citation sets is less about finding papers on the right topic, and more about choosing among them, which is a finer task than what coarse embedding models have resolution for.
This gap in ability for LCR over citation sets matters because an increasing proportion of researchers rely on automated tools to find and check their citations, and every recommendation that these tools make contributes a small act of attribution. While differences in how retrieval systems compose citation sets as compared to humans might make little difference at the individual level, at the scale of a complete literature they may become structurally important [6]. For citation sets, current retrieval systems favor semantic similarity more strongly than human authors, whose choices reflect a wider range of judgements. Because researchers often discover new works via citations, widespread adoption of these systems risks homogenizing knowledge attribution and artificially narrowing science’s intellectual scope [18, 14].
In this paper, we ask how much of the missing set structure a simple co-citation reranker, that uses information about what papers have been cited together in sets in the past, can recover. We find that BM25, a simple bag-of-words lexical similarity model [15], outperforms each dense embedding model we test, suggesting that topical smoothing in these models might obscure more fine-grained lexical distinctions necessary to distinguish complementary papers on the same topic. Second, the co-citation reranker boosts performance across all of our baseline retrieval methods, and part of this gain comes from calibrating the diversity of the suggested sets to better match that of human-constructed sets. Finally, a sequential analysis shows that citation-set information is most strongly concentrated in the first correct anchor for a set, making this decision a key bottleneck.2
The citation set completion task
Traditionally, citation retrieval involves the following task: given the citance () a retriever scores candidates () by their textual or semantic compatibility, . This can be thought of as a discovery task, and because it involves optimizing the individual textual similarity between the context and the paper it cites, citation-trained encoders such as SPECTER2 [16] and SciNCL [13] are often used. However, it leaves open the task of set completion. Namely, once any citation at a given location is observed, can we identify which other papers belong with it? That is, given and any existing citations at that location, score candidates: (Figure 1).

Figure 1. Document retrieval formulates the problem as . Once any citation at the location is observed, the set allows us to make a second conditional available at query time.
Structure and judgement in citation sets
When researchers group multiple citations into a single set, the structure of that set reflects specific judgements about how evidence should be assembled. We can see these differences clearly by looking at examples from two functional extremes. Some citation sets are corroborating, like the following example:
Recent papers that report state-of-the-art results in that arena suggest angular margin losses [9, 10, 11] as a better measure of choice. Here the citations are SphereFace, CosFace, and ArcFace respectively, representing three variants of a single idea. Each member of this citation set reiterates the same point that the citance makes. By contrast, consider the following complementary citation set:
The transverse field is not the only possible way of controlling the adiabatic state and we can find many other choices in principle. In fact, it has been recognized that “nonstoquastic” effect improves the performance [9, 10, 11, 12].
Here each citation contributes a different aspect of the claim’s support. [9] is the original proposal of the idea, [10] is a theory paper that defines the key concept the claim depends on, [11] is a theoretical analysis proving the improvement, and [12] is a set of computer experiments that demonstrate it in practice3.
These distinct roles manifest clearly in the vector spaces of the embeddings of these citation sets. When each cited work’s title and abstract are embedded with SPECTER2 [4, 16], the citations in the corroborating example are in the 98th percentile of cosine similarity among all gold (i.e., human) citation sets. In contrast, the complementary citations example falls in the 1.9th percentile of the same measure. In fact, across our corpus, for a given citation set of size , SPECTER2’s top- retrieved sets are more self-similar than human-constructed citation sets (see Figure 2). Their mean pairwise cosine similarity is +0.014 higher (95% CI [+0.013, +0.014]), which, for scale, is roughly a fifth of the gap between human and entirely random sets. This difference is bigger in introduction sections (+0.015 [+0.013, +0.017]), where authors often synthesize and compare the literature, than methods sections (+0.010 [+0.007, +0.013], ). It is easy to see, then, why a similarity-maximizing objective will often fail to capture the kinds of judgement that produce complementary, or otherwise multifaceted citation sets.

Figure 2. Retriever-recommended citation sets have higher mean pairwise cosine similarity than human-constructed citation sets. Error bars show 95% CIs.
Related Work: Document Retrieval and Citation-Set Retrieval
Local citation recommendation. LCR retrieves a missing reference from the text surrounding a citation placeholder [9]. Modern systems are often two-stage: a prefetcher narrows an otherwise large corpus, and a cross-encoder reranks the narrowed set. We build LOCUS on the arXiv split of Gu et al. [8], whose hierarchical-attention encoder (HAtten) with SciBERT-based [1] reranking is one of our base retrievers. Recent generative approaches [2] are not in scope for this work, but they still use the model of filling one placeholder at a time.
Citation-trained representations. SPECTER [4], SPECTER2 [16] and SciNCL [13] learn document representations from a combination of title and abstract text, and the citation graph. This is precisely what makes these models an interesting base to test here — they already encode some citation signal, so a residual gain over them is not evidence that encoders ignore citations. We add BM25 [15] as a lexical reference point. It is untuned at , .
Co-citation and set-aware retrieval. Co-citation has been used in the past as a measure of document relatedness since the work of Small [17]. However, co-citation is often construed as papers cited in the reference paper. Here we use a more fine-grained measure of co-citation, which is papers that appear in the same citation set. We score this with positive pointwise mutual information [3] with the context-distribution smoothing of Levy et al. [11] (this is the standard correction for PMI’s instability on rare pairs). We chose this simple scorer to ensure that the empirical differences we find are attributable to citation set structure rather than to a given reranking model’s capacity. The learned alternative here would be a graph embedding, so we also compare our results to node2vec [7].
LOCUS
Problem formulation
Let be the citation context, defined as a masked text window surrounding a citation marker. Contexts from the same citing paper are grouped into citation locations based on textual overlap. Let denote the set of distinct papers cited at the location containing . For the target paper associated with , let denote the other papers cited at that location, excluding the target. Conventional retrieval scores a candidate paper using only the query context, , whereas set-conditioned retrieval additionally incorporates the observed anchors, . LOCUS evaluates whether this additional citation-set information improves retrieval.
Location and anchor construction
We build on the arXiv dataset from Gu et al. [8], with splits disjoint by citing paper. Contexts are grouped into locations within each citing paper using 8-word-shingle Jaccard clustering (). The threshold was fixed before any test scoring; Appendix A Appendix § 10 rebuilds the benchmark at from 0.3 to 0.7 and finds the headline gain insensitive to it.
We construct anchors through two non-nested routes. The clustering route uses citations targeted by other contexts assigned to the same location as , while the marker route resolves additional inline citation markers appearing within . We take the union of the two routes, excluding the target associated with . The two routes yield 69,874 and 70,357 anchored contexts, with 78,334 in their union.
These inferred locations are also supported by human annotation, as the union of the two routes recovers 97.7% of same-sentence citation pairs (Appendix A). Additional route and proximity controls show that the headline result does not depend on any single anchor-construction mechanism (Appendix A).
Evaluation subset
The test set contains 104,401 citation contexts, which are grouped into 55,714 citation locations, of which 40% have multiple citations. We consider a citation’s “anchor” to be other papers cited at the same location, and we divide the evaluation into hard and easy pools/contexts, where easy pools are those in which at least half of a target’s title words appear in the context window. Our headline evaluation uses hard contexts with ten-candidate pools and at least one anchor. Appendix B gives a more detailed composition of the test split and its null controls.
Controlled probes
We then build two separate probes. These probes are designed to isolate whether citation-set structure adds information beyond document-level relevance, rather than to reproduce end-to-end deployment. In LOCUS-Rank, a system is given a citation context and 10 candidate papers, of which the paper actually cited is present and the 9 others are drawn from the same citing paper just at different locations. Therefore every candidate is something the author cited, and topical relevance alone is less able to distinguish the candidates, making the citing context uniquely important. In LOCUS-Swap, a system is given two citing contexts and two cited papers, and must choose which paper goes with which context.
The fact that we use distractors from within a given paper means LOCUS has a simple analytic floor. For LOCUS-Swap, any scorer independent of the query context assigns the same two candidate scores under either context assignment. Exchanging the two query contexts therefore flips the preferred assignment, giving expected accuracy 0.5. On LOCUS-Rank, a constant scorer assigns equal scores to all candidates. Under uniform tie-breaking, the target’s expected reciprocal rank is . With , this gives .
Set-Conditioned Retrieval
A co-citation reranker
where denotes the standardized base-retriever score. For LOCUS-Rank, we standardize scores using a -score over the 10 candidates in each pool. LOCUS-Swap instead uses global standardization, with and estimated on validation and applied unchanged to test; a two-candidate pool -score would collapse the base scores to exactly . Consequently, is scale-dependent and its values should not be compared directly across Rank and Swap. All reranker hyperparameters are selected on validation data only. For the headline Rank experiment, validation selects , and ; sequential completion reuses these settings without further tuning.
, where following context-distribution smoothing [11]. Here, counts citation locations containing both papers, is the corresponding marginal count, and . We define a co-citation event as one (citing paper, location), with each unordered citation pair counted once per location. To prevent information leakage, the co-citation graph is constructed using only the training split, comprising 374,691 papers and 3,436,198 directed non-zero entries, with no measured overlap with the 20,845 held-out papers.
Evaluation protocol
The primary metric, declared before any test-set scoring, is MRR on LOCUS-Rank. The preregistered primary comparison uses SPECTER2 as the base retriever on the headline set (hard slices); the remaining retrievers serve as robustness checks. Result is evaluated using a paired paper-level bootstrap 95% CI excluding 0. Every reported metric is a mean of per-citing-paper means, because contexts from one paper share a reference list and are not independent; all intervals use paired paper-level cluster bootstraps with 1000 resamples, and and the smoothing parameters are selected using validation data only. Compute resources and runtimes are reported in Appendix § 11.
Results
LOCUS-Rank
Across all bases, the structural (co-citation) intervention produces a substantial and consistent improvement in MRR (Table 1). Its gains over base retrievers range from +0.104 to +0.124, despite differences in those retrievers’ architectures and training objectives. This consistency indicates that citation-set structure provides information complementary to the base retrieval signals.
| Base | Base | Hybrid | MRR (95% CI) | R@1 |
| SPECTER2 | 0.497 | 0.621 | +0.124 [+0.116, +0.132] | +0.156 |
| SciNCL | 0.490 | 0.615 | +0.124 [+0.116, +0.133] | +0.158 |
| HAtten | 0.480 | 0.597 | +0.117 [+0.110, +0.124] | +0.151 |
| BM25 (lexical) | 0.522 | 0.626 | +0.104 [+0.096, +0.112] | +0.128 |
Table 1. LOCUS-Rank on test headline pools; chance floor 0.29.
BM25 has the best base MRR (0.522), but the smallest structural gain (+0.104). Although we filter title overlap, we do not suppress abstract overlap. This is why the lexical baseline remains well above the null floor on the hard slice; nevertheless, the citation-set signal is still useful.
The aggregate gain depends strongly on coverage of the co-citation graph. In 61.3% of pools, the graph captures the target relation and improves MRR by +0.316. In 20.4%, it instead captures a distractor relation and reduces MRR by −0.245. In the remaining 18.3%, it captures neither and has no effect (Appendix § 11). Nevertheless, each base benefits from the same co-citation evidence when it is available. This suggests that the textual retrievers do not fully encode this relation. Further controls show that these gains depend on location-specific citation evidence rather than arbitrary graph connections. Random and degree-matched anchors provide no improvement (MRR 0.496 and 0.493 vs. 0.497 for the baseline), while anchors from other locations in the same paper reduce MRR by 0.099. The gains are also substantially larger on the hard slice than the easy slice (+0.124 vs. +0.022). All of the controls mentioned are provided in Appendix § 11.
Beyond simply improving the MRR, the co-citation reranker alters the semantic composition of the retrieved sets, bringing them closer to human behavior. As we noted previously, base retrievers tend to produce very uniform sets, with an average pairwise cosine similarity of 0.944. By leveraging structural signals, the reranker drops this similarity down to 0.933, bridging much of the gap toward the more diverse citations found in human ground truth sets (0.928).
LOCUS-Swap
LOCUS-Swap is scored on the same frozen pools and reported here (Table 2). Gains are smaller than on Rank, ranging from +0.022 to +0.033, but every base improves. The co-citation scorer produces the same Swap decision at 0.643 regardless of the base model used and doesn’t beat any of the base models, showing that it is not a replacement for semantic or lexical retrieval—it provides complementary information.
| Base | Base | Co-cite | Hybrid | Accuracy (95% CI) | ||
| SPECTER2 | 0.0302 | 0.2 | 0.816 | 0.643 | 0.844 | +0.028 [+0.024, +0.031] |
| SciNCL | 0.0560 | 0.2 | 0.801 | 0.643 | 0.834 | +0.033 [+0.029, +0.037] |
| HAtten | 0.1570 | 0.1 | 0.831 | 0.643 | 0.853 | +0.022 [+0.019, +0.026] |
| BM25 (lexical) | 29.868 | 0.1 | 0.834 | 0.643 | 0.859 | +0.025 [+0.022, +0.028] |
Table 2. LOCUS-Swap accuracy on test swap pairs; chance floor exactly 0.50. BM25’s is on a different scale because its scores are unbounded where a cosine is not.
We note that HAtten differs in architecture and dimension and, unlike the other bases, builds queries from three paragraphs that include the citing paper’s title and abstract. Its weak score in Rank but high performance in Swap is due to this input asymmetry rather than a disagreement between the probes. We also evaluate graph structure using node2vec as a control; it does not improve LOCUS-Swap, with validation selecting and larger positive weights reducing accuracy. Details are reported in Appendix § 11.
Corpus-wide retrieval
LOCUS-Rank and LOCUS-Swap are controlled probes: they restrict the candidate set and, for set-conditioned retrieval, assume that one citation at the location is already known. To test whether the observed gain depends on either source of privileged information, we evaluate sequential citation-set completion under the full combination of candidate scope and first-anchor source.
Starting from , a self-seeded system retrieves and conditions each subsequent step on the citations retrieved so far. Table 3 compares oracle-prefetched versus corpus-wide candidates and revealed versus self-retrieved first anchors.
| Setting | Candidates | First anchor | Semantic | Hybrid | R@20 (95% CI) |
| A | oracle prefetch | revealed citation | 0.563 | 0.716 | +0.153 [+0.148, +0.158] |
| B | oracle prefetch | self-retrieved | 0.264 | 0.386 | +0.122 [+0.116, +0.128] |
| C | corpus-wide | revealed citation | 0.522 | 0.627 | +0.105 [+0.101, +0.110] |
| D | corpus-wide | self-retrieved | 0.194 | 0.269 | +0.075 [+0.070, +0.080] |
Table 3. Sequential set completion under a removal of privileged information.
Removing either source of privileged information reduces but does not eliminate the benefit of set conditioning. With oracle-prefetched candidates but a self-retrieved first citation (B), the reranker improves Recall@20 by +0.122 (46% relative). With corpus-wide candidates but a revealed first citation (C), the gain remains +0.105 (20% relative). Setting D removes both: the first citation is self-retrieved and candidates come from cosine retrieval over the full embedding store, using the top 2,000 candidates per location. First-stage recall over 67,492 gold citations is 0.313 at and 0.684 at , leaving 31.6% of gold citations unreachable by the reranker. Even under this candidate-recall ceiling, set conditioning improves Recall@20 by +0.075 (39% relative). Thus, the gain does not depend on either an oracle candidate pool or a revealed first citation.
What the Evaluation Reveals
The first-anchor bottleneck
The main bottleneck in sequential completion is obtaining the first correct citation. A correct first pick occurs at 9.39% of locations (2,091 of 22,274) and yields Recall@20 of 0.723, compared with 0.333 after an incorrect first pick. Excluding the revealed citation from evaluation further strengthens this pattern, with one correct citation yielding a gain in Recall@1 under oracle prefetch and under corpus-wide retrieval (Table 4). Additional correct anchors provide smaller gains. This diminishing marginal benefit follows directly from the averaging in Eq. (2); Appendix § 12 derives the influence of an individual anchor.
| Base | Hybrid | (95% CI) | |||
| oracle prefetch | 1 | 0.048 | 0.132 | +0.084 [+0.079, +0.090] | |
| oracle prefetch | 2 | 0.077 | 0.206 | +0.129 [+0.122, +0.135] | |
| oracle prefetch | 20 | 0.265 | 0.518 | +0.253 [+0.244, +0.262] | |
| corpus-wide | 1 | 0.035 | 0.089 | +0.055 [+0.051, +0.059] | |
| corpus-wide | 2 | 0.058 | 0.144 | +0.086 [+0.081, +0.091] | |
| corpus-wide | 20 | 0.193 | 0.368 | +0.175 [+0.167, +0.183] |
Table 4. Recall over citations still to be found after one true citation is revealed. Corpus-wide retrieval has a candidate-recall ceiling of 0.684.
The 9.39% figure is a per-location hit rate, whereas Recall@1 is 0.0426 because each location can contain multiple gold citations, with an average gold-set size of 3.03. The same pattern persists under corpus-wide retrieval, indicating that it is not an artifact of oracle prefetching. The first citation therefore acts as an information-bearing entry point into the citation set. This makes first-anchor quality a key limitation of sequential completion and exposes a problem not captured by standard document-retrieval framing: informative anchor selection.
What makes an anchor informative?
Anchors are not interchangeable. Figure 3 shows that their usefulness is primarily determined by co-citation evidence rather than topical similarity. Across cosine-similarity octiles, the highest-PPMI band has positive influence ranging from to . Variation across PPMI bands is substantially larger than variation across cosine similarity, suggesting that anchor usefulness is more associated with co-citation evidence than with topical similarity.

Figure 3. Mean rank influence over all interventions, binned by embedding similarity (cosine octiles) and co-citation evidence.
Based on this pattern, we have identified three groups of interventions. Informative anchors have PPMI and whose removal worsens the target’s rank; they account for 8.4% of interventions and have mean influence . Redundant anchors also have PPMI , but their removal does not change rank; they account for 31.7%. Finally, non-co-cited anchors have PPMI and account for 59.8% of interventions, with mean influence .
Limitations and Open Problems
Citation sets reflect observed author choices rather than complete evidence sets. LOCUS therefore evaluates recovery of citation-set structure, not the completeness or scientific correctness of the underlying evidence. A system may reproduce citation conventions without recovering every scientifically appropriate reference, which this evaluation would not detect. 24.97% of test contexts have no resolvable anchor and are excluded from the headline set, so LOCUS applies only to locations with multiple resolvable citations. Furthermore, the effect size depends on the 61.3% coverage of this particular corpus. A different corpus or training graph could produce a very different aggregate number.
The evaluation is further limited to a single arXiv split, so conclusions may not generalize to other corpora, languages, or scientific domains. Citation locations are also inferred using 8-shingle Jaccard clustering with , rather than obtained from ground-truth annotations; Appendix § 10 shows the result is insensitive to the threshold, and that it does not rest on either inference route.
Our experiments show that sequential retrieval is strongly dependent on the first informative anchor and that anchors differ substantially in predictive value, but they do not establish how to identify the most informative anchor. Developing an effective first-anchor selection strategy is therefore a natural direction for future work.
Conclusion
Scientific citation recommendation is typically framed as independent document retrieval, but LOCUS shows that citations already observed alongside a query context provide additional query-time information. A simple co-citation reranker consistently improves retrieval across the base systems, while same-paper controls show that anchors from the target location are more useful than anchors from elsewhere in the citing paper. Sequential evaluation further shows that most of the benefit comes from the first correct anchor.
These findings suggest that citation relationships should be treated as structured evidence available at retrieval time, rather than only as properties of individual documents. The remaining bottleneck is selecting that first informative anchor: our results identify its importance, but do not address how to select it optimally. This provides a concrete direction for future work on citation-set retrieval and sequential scientific search.
Acknowledgments and Disclosure of Funding
Compute for this work (AWS GPU instances) was provided by Valency. A.L., J.S.B. and K.R. are affiliated with Valency, a company that develops tools for searching the scientific literature. The authors declare no other competing interests.
Data
Sensitivity to the clustering threshold We rebuilt all components downstream of clustering at . For each threshold, and were re-selected on a validation split rebuilt using the same clustering threshold. As shown in Table 5, finer clustering () increases the test set from 55,714 to 62,839 locations, whereas coarser clustering () reduces it to 48,747. The fraction of marker anchors falling outside their context’s cluster varies substantially, from 5.1% to 33.1%. In contrast, because the marker route itself does not depend on , the fraction of contexts with anchors remains relatively stable at 73.9–76.0% as does the mean anchor-set size at 3.91–4.27.
| locations | Rank pools | headline | anchored | marker outside cluster | |||
| 0.3 | 48,747 | 53,117 | 34,010 | 4.27 | 76.0% | 5.1% | 0.75, 4 |
| 0.4 | 52,177 | 53,783 | 34,429 | 4.07 | 75.5% | 10.2% | 0.75, 3 |
| 0.5 | 55,714 | 54,020 | 34,577 | 3.96 | 75.0% | 17.0% | 0.75, 3 |
| 0.6 | 59,391 | 54,145 | 34,654 | 3.92 | 74.5% | 25.2% | 0.75, 4 |
| 0.7 | 62,839 | 54,169 | 34,667 | 3.91 | 73.9% | 33.1% | 0.5, 3 |
Table 5. Effect of clustering threshold on dataset composition, anchor statistics, and hyperparameters.
Comparisons across thresholds require care because changing affects context eligibility, distractor composition, and the inferred anchors. Table 6 therefore reports four complementary comparisons. A full rebuild, using each threshold’s own headline pools, gives a stable gain of +0.122 to +0.125. Restricting to the 33,916 contexts eligible at every threshold (fixed cohort) gives a similar +0.122 to +0.126. Restricting further to the 29,345 contexts whose ten candidates are unchanged across all thresholds (unchanged pools) gives +0.127 to +0.131; 29,276 of these also retain identical anchors, leaving mainly the rebuilt training graph and hyperparameters to vary. Finally, swapped anchors holds the pools, graph, and hyperparameters fixed for 31,142 contexts and changes only the anchors inferred at each threshold. Although 1.5–3.7% of these contexts receive a different anchor set, the gain remains essentially unchanged at +0.127.
| full rebuild | fixed cohort | unchanged pools | swapped anchors | |
| 0.3 | +0.12235 [+0.114, +0.130] | +0.12244 [+0.114, +0.130] | +0.12775 [+0.119, +0.136] | +0.12699 [+0.119, +0.135] |
| 0.4 | +0.12179 [+0.114, +0.130] | +0.12246 [+0.114, +0.130] | +0.12727 [+0.119, +0.135] | +0.12741 [+0.119, +0.136] |
| 0.5 | +0.12382 [+0.116, +0.132] | +0.12432 [+0.116, +0.132] | +0.12909 [+0.121, +0.137] | +0.12745 [+0.119, +0.136] |
| 0.6 | +0.12526 [+0.117, +0.133] | +0.12629 [+0.118, +0.134] | +0.13071 [+0.122, +0.139] | +0.12749 [+0.119, +0.136] |
| 0.7 | +0.12512 [+0.117, +0.133] | +0.12641 [+0.118, +0.134] | +0.13078 [+0.123, +0.139] | +0.12746 [+0.119, +0.136] |
Table 6. Sensitivity of the SPECTER2 headline gain to clustering threshold under four controlled comparisons. Values are MRR averaged over citing papers with paired paper-level 95% CIs.
Annotation Validation. We validated the inferred citation locations and anchors on 400 hand-labelled context–candidate pairs. Agreement between annotators was substantial (). The inferred anchors align well with sentence-level citation structure: the clustering route achieves 0.852 precision and 0.935 recall, while the union of clustering and marker routes recovers 97.7% of same-sentence pairs. Under the stricter same-citation-group definition, recall remains near complete but precision falls to 0.47–0.55, indicating that the dominant error is over-inclusion rather than missed anchors. Importantly, removing the human-rejected anchors does not reduce the reranking gain. The gain changed from +0.155 to +0.158 under the citation-group definition and from +0.047 to +0.048 under the sentence definition.
| Site | Anchors | Precision (95% CI) | Recall (95% CI) | F1 |
| Same citation group | ||||
| clustering route | 0.553 [0.521, 0.593] | 0.998 [0.998, 0.998] | 0.712 | |
| marker route | 0.472 [0.444, 0.510] | 0.909 [0.890, 0.927] | 0.621 | |
| union | 0.469 [0.442, 0.503] | 1.000 [1.000, 1.000] | 0.639 | |
| Same sentence | ||||
| clustering route | 0.852 [0.833, 0.870] | 0.935 [0.909, 0.950] | 0.891 | |
| marker route | 0.759 [0.745, 0.772] | 0.889 [0.863, 0.906] | 0.819 | |
| union | 0.754 [0.739, 0.770] | 0.977 [0.951, 0.993] | 0.851 |
Table 7. The inferred anchors against 400 hand-labelled pairs from the test split.
Where the anchors come from The anchor construction can fail through either over- or under-merging. On the test split, 87.5% of contexts align their masked window to the raw text, and of the 467,679 inline markers they contain, 52.2% resolve to a distinct in-corpus paper. Of the remainder, 39.0% refer to works outside the arXiv corpus and cannot be resolved by construction, 6.0% are implausible numbers such as years, and 2.7% point back to the target itself. Table 8 shows that rescoring the same 34,577 headline pools gives a gain of +0.122 with marker anchors only and +0.130 with clustering anchors only, compared with +0.124 for their union. Thus, either route is sufficient to recover the effect.
| Anchors | pools | anchored | papers | Base | Hybrid | MRR (95% CI) |
| By inference route | ||||||
| union | 34,577 | 34,577 | 1,968 | 0.497 | 0.621 | +0.124 [+0.116, +0.132] |
| marker route | 34,577 | 34,505 | 1,968 | 0.497 | 0.619 | +0.122 [+0.114, +0.131] |
| clustering route | 34,577 | 31,121 | 1,968 | 0.497 | 0.627 | +0.130 [+0.123, +0.138] |
| By anchor proximity | ||||||
| same citation group | 22,218 | 22,218 | 1,820 | 0.489 | 0.679 | +0.190 [+0.180, +0.199] |
| same sentence | 14,432 | 14,432 | 1,588 | 0.491 | 0.570 | +0.079 [+0.069, +0.091] |
| another sentence | 14,724 | 14,724 | 1,676 | 0.513 | 0.523 | +0.010 [+0.000, +0.019] |
| outside the window | 1,954 | 1,954 | 402 | 0.482 | 0.568 | +0.086 [+0.061, +0.114] |
Table 8. The SPECTER2 headline gain by where the anchors come from.
The useful signal is also concentrated in the most local anchors. Among headline pools, the closest available anchor is in the target’s citation set for 64.3%, elsewhere in the same sentence for 21.3%, in another sentence for 14.3%, and only outside the window for 0.2%. Using only these tiers, the gain is +0.190 for same-group anchors, +0.079 for same-sentence anchors, and +0.010 for anchors from another sentence. Moreover, on the 22,218 pools with a same-group anchor, adding progressively more distant anchors reduces the gain from +0.190 to +0.179 and then +0.165. This pattern suggests that less local anchors dilute rather than create the effect, making over-merging an unlikely explanation for the result.
Method details
Composition and null controls. Table 9 gives the composition of the test split and its null controls. The constant rank scorer yields an exact expected floor of , while popularity shifts strictly off the floor.
| Test | Val | Null control (test) | Value | Required | |
| Contexts | 104,401 | 112,779 | Swap, constant | 0.50 | Exactly 0.5 |
| Citing papers | 9,208 | 11,637 | Swap, popularity | 0.50 | Exactly 0.5 |
| Locations | 55,714 | 62,466 | Rank, constant | 0.29 | Exactly |
| Anchor coverage | 75.03% | 72.23% | Rank, popularity | 0.35 | Strictly off floor |
| Rank pools | 54,020 | 47,946 | |||
| Swap pairs | 27,905 | 33,203 | |||
| Hard fraction | 78.21% | 78.44% | |||
| Headline pools | 34,577 | 30,177 |
Table 9. LOCUS composition and null controls.
Compute resources and runtimes. Embeddings were computed on a single NVIDIA A10G GPU (AWS ml.g5.xlarge) and reproduced on an Apple M1 laptop, where the full pipeline runs in about 14 hours. Scoring, reranking, and sequential-completion experiments were CPU-only, with the longest individual run taking under 90 minutes.
Graph-coverage. The strata partition the 34,577 pools, but not the citing papers, since a paper may contribute pools to multiple strata (Table 10). Therefore, the paper-level stratum means do not recombine to the overall paper-level mean by weighting them with pool counts. Averaged over pools instead, the rows recombine exactly: 0.498 → 0.627 overall (+0.129 [+0.122, +0.136]), +0.292 on the seen pools and −0.129 on the unseen; those intervals still resample citing papers, taking the mean over every pool the resampled papers contribute.
| Stratum | Pools | Share | Base | Hybrid | MRR (95% CI) |
| PPMI(, ) > 0 | 21,188 | 61.3% | 0.489 | 0.805 | +0.316 [+0.30579, +0.32628] |
| PPMI(, ) = 0 | 13,389 | 38.7% | 0.504 | 0.362 | −0.142 [-0.14914, -0.13490] |
| – distractor is seen | 7,061 | 20.4% | 0.478 | 0.232 | −0.245 [-0.255, -0.236] |
| – nothing is seen | 6,328 | 18.3% | 0.532 | 0.532 | 0.000 |
| All headline pools | 34,577 | 100% | 0.497 | 0.621 | +0.124 [+0.116, +0.132] |
Table 10. LOCUS-Rank headline pools stratified by training-graph coverage of the (target, anchor) relation.
Additional controls. On the hard slice, the reranker improves MRR by +0.124 (95% CI [+0.116, +0.132]), compared with +0.022 ([+0.010, +0.033]) on the easy slice (Table 11). The gap between the two gains is +0.102 ([+0.089, +0.114]), a ratio of 5.7× ([3.8, 11.9]); both intervals come from one paper-level bootstrap in which each resample of citing papers is scored on both slices. A same-paper location control shows that anchors from the target location are substantially more useful than anchors from elsewhere in the same citing paper. Holding the citing paper fixed while replacing the true anchors with citations from another location reverses the gain by 0.099. The effect is also not explained by arbitrary or popular anchors. Random anchors leave MRR essentially unchanged (0.496 versus 0.497), while degree-matched random anchors reduce it slightly to 0.493. Moreover, making the anchor set directly available as text by appending the sibling titles to the query raises the base MRR to 0.554, but structural reranking still contributes an additional +0.079. Finally, the form of the structural signal matters. Raw co-citation counts are the strongest alternative to PPMI but remain weaker because they don’t measure relative associations. Replacing PPMI with node2vec on the same train-only graph, with and selected on validation, retains only a +0.045 gain, approximately 36% of the gain obtained with PPMI.
| Anchor / query condition | Base | Hybrid | MRR (95% CI) | |
| Hardness | Easy | 0.680 | 0.702 | +0.022 [+0.010, +0.033] |
| Hard | 0.497 | 0.621 | +0.124 [+0.116, +0.132] | |
| Anchor provenance | other location | 0.497 | 0.397 | −0.099 [-0.105, -0.093] |
| Random | 0.497 | 0.496 | −0.000 | |
| Degree-matched random | 0.497 | 0.493 | −0.004 | |
| Robustness | PPMI | 0.497 | 0.621 | +0.124 [+0.116, +0.132] |
| Raw co-count | 0.497 | 0.608 | +0.111 [+0.103, +0.119] | |
| node2vec | 0.497 | 0.542 | +0.045 [+0.038, +0.054] | |
| Sibling titles | 0.554 | 0.633 | +0.079 [+0.071, +0.086] |
Table 11. Controls on SPECTER2 (LOCUS-Rank MRR).
The node2vec control. node2vec does not improve LOCUS-Swap. Using the validation-selected embedding, validation selects . Accuracy is 0.81593 at , 0.81550 at , 0.81568 at , and 0.81356 at . The small changes at are within bootstrap noise, while from onward every interval relative to excludes zero and performance continues to deteriorate. Thus, there is no evidence that a positive node2vec weight helps: small weights are indistinguishable from the zero-weight baseline and larger weights significantly hurt. This conclusion is consistent across all five tested values. The comparison with PPMI is also not specific to Swap: on LOCUS-Rank, node2vec improves MRR by only +0.04519, compared with +0.12382 for PPMI, and in sequential completion it provides no measurable gain from a true seed and reduces recall in both self-seeded settings.
Discussion
Anchor influence. For target with anchor set we withhold one anchor at a time and measure on both score and rank. With , and the mean over ,
so an anchor moves the target only to the extent it is more co-cited with it than the average anchor is; average association buys zero influence. The factor damps everything as the set grows, which the data confirms: mean rank influence falls from +0.084 at to +0.030 at 2, +0.016 at 3 and 0.000 at 9.
References
- [1]Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. arXiv:1903.10676.arxiv.org/abs/1903.10676
- [2]Ege Yiğit Çelik and Selma Tekir. CiteBART: Learning to generate citations for local citation recommendation. arXiv preprint arXiv:2412.17534, 2024.arxiv.org/abs/2412.17534
- [3]Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22–29, 1990.
- [4]Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020. arXiv:2004.07180.arxiv.org/abs/2004.07180
- [5]Giovanni Colavizza, Kevin W. Boyack, Nees Jan van Eck, and Ludo Waltman. The closer the better: Similarity of publication pairs at different co-citation levels, 2017. URL https://arxiv.org/abs/1707.03076.
- [6]James A. Evans. Electronic Publication and the Narrowing of Science and Scholarship. Science, 321(5887):395–399, July 2008. doi: 10.1126/science.1150473.DOI
- [7]Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks, 2016. URL https://arxiv.org/abs/1607.00653.
- [8]Nianlong Gu, Yingqiang Gao, and Richard H. R. Hahnloser. Local citation recommendation with hierarchical-attention text encoder and SciBERT-based reranking. In Advances in Information Retrieval (ECIR 2022), pages 274–288, Cham, 2022. Springer International Publishing. doi: 10.1007/978-3-030-99736-6_19.DOI
- [9]Qi He, Jian Pei, Daniel Kifer, Prasenjit Mitra, and C. Lee Giles. Context-aware citation recommendation. In Proceedings of the 19th International Conference on World Wide Web (WWW), pages 421–430, 2010.DOI
- [10]Bruno Latour. Science in Action: How to Follow Scientists and Engineers Through Society, chapter 1, pages 21–62. Harvard University Press. ISBN 978-0-674-79291-3.
- [11]Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211–225, 2015.DOI
- [12]M. H. MacRoberts and Barbara R. MacRoberts. Problems of citation analysis. Scientometrics, 36(3):435–444, July 1996. ISSN 1588-2861. doi: 10.1007/BF02129604.DOI
- [13]Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp, and Georg Rehm. Neighborhood contrastive learning for scientific document representations with citation embeddings. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. arXiv:2202.06671.arxiv.org/abs/2202.06671
- [14]Raj K. Pan, Alexander M. Petersen, Fabio Pammolli, and Santo Fortunato. The memory of science: Inflation, myopia, and the knowledge network. Journal of Informetrics, 12(3):656–678, August 2018. ISSN 1751-1577. doi: 10.1016/j.joi.2018.06.005.DOI
- [15]Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009.
- [16]Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. SciRepEval: A multi-format benchmark for scientific document representations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5548–5566, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.338. URL https://aclanthology.org/2023.emnlp-main.338/.DOI
- [17]Henry Small. Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for Information Science, 24(4):265–269, 1973.DOI
- [18]Attila Varga. The narrowing of literature use and the restricted mobility of papers in the sciences. Proceedings of the National Academy of Sciences, 119(17):e2117488119, April 2022. doi: 10.1073/pnas.2117488119.DOI