betaFeedback

WorksComputer Science

SciToT: Multimodal Tip-of-the-Tongue Retrieval for Scientific Papers

Abstract

Researchers frequently encounter difficulties re-finding scientific papers when they remember aspects of the content but cannot recall reliable identifiers, such as titles or author names. This type of re-finding task, known as tip-of-the-tongue (ToT) retrieval, is particularly challenging for scientific documents, where recollections may involve textual information and visual elements, including figures, diagrams, and plots. Existing ToT retrieval benchmarks do not address scientific search. We introduce a multimodal benchmark for scientific ToT retrieval, comprising 108,512 open-access scientific documents across five domains and 11,499 synthetically generated queries. To construct the benchmark, we propose a scalable query elicitation pipeline that independently extracts textual and visual memory cues from source documents and uses large language models to generate queries resembling incomplete human recollections. We characterize scientific ToT information needs through an empirical analysis of publicly available re-finding requests and use the outcomes to assess the generated queries using human annotations and automated quality control. The benchmark includes lexical, dense, late-interaction, and visually grounded retrieval baselines. We release the dataset, extracted document representations, generated memory cues, evaluation splits, and accompanying generation and retrieval pipelines under an MIT license to facilitate research on multimodal scientific paper re-finding.

Introduction

Tip-of-the-tongue (ToT) retrieval is a known-item search task in which users attempt to re-find an item that they have previously encountered, while not having access to exact identifiers, such as document titles or author names [8, 15, 41]. ToT queries are more complex and longer than conventional keyword-based search queries, and they typically contain details about the context in which a user engaged with a work [59]. Furthermore, such queries may contain uncertainties, relative comparisons or false memories [8, 41]. This makes ToT retrieval a difficult task for traditional retrieval systems [7, 41, 57]. Importantly, it is a highly relevant task to resolve, as it is associated with a high level of frustration in users and can lead to decreases in productivity [32].

Scientific literature search is a critical task for researchers, used for collecting related work, checking previously proposed methods, and known-item search [2, 56]. Known patterns from ToT queries persist in scientific literature search: expanded queries often contain imprecise or incorrect information, relying on attributes from remembered documents [54, 55]. Scientific literature relies on figures for supporting text, which means effective representations of visuals are essential for understanding [48, 63, 85]. In ToT retrieval, the visual component is highly relevant, as clues uttered in ToT queries rely on remembered elements, which are often visual in nature [8, 16]. Our analysis of real-world ToT queries reveals that visual elements are frequent, and can be highly distinctive elements. Hence, evaluating ToT retrieval for scientific literature should account for remembered clues that stem from visual content.

Despite the prevalence of ToT style query failures in scientific literature search, no datasets exist to evaluate systems on scientific ToT queries. Prior work in ToT has mostly focused on casual [33] domains such as movies, music, and games [15, 18, 46], which can be sourced from community question-answering platforms [59]. Importantly, for many domains, including the scientific domain, no such platforms exist with sufficient coverage for dataset construction, due to confidentiality, privacy, or data scarcity [41, 59].

In this work, we address the lack of evaluation resources for scientific ToT retrieval by introducing a scalable approach for generating multimodal recollection queries, an example of which is shown in Figure 1. We extend the large language model (LLM)-based query elicitation framework proposed by He et al. [41] to scientific documents and incorporate both visual and textual information. We parse PDF documents into visual and textual elements, and generate potential clues. We then generate a ToT query based on these clues. We validate the resulting resource and establish retrieval baselines to support future research. We release the dataset, dataset construction code, retrieval code, and baseline implementations under an MIT license.

Extracted figure showing a three-dimensional plot with purple and green intersecting regions and a black line; textual memory cues: Debiasing of word embeddings; Discussion of intersectional biases; Comparison of multiple bias mitigation methods; · · ·; visual memory cues: Three-dimensional plot; Intersecting purple and green regions; Black line passing through their intersection; · · ·; generated tip-of-the-tongue query: I’m trying to track down a paper I came across a while back. It involved debiasing of word embeddings and discussed intersectional biases, comparing multiple bias mitigation methods. One figure that stood out was a 3D plot with intersecting purple and green shaded regions, and a black line passing through their intersection. Does this ring a bell?

Figure 1. Example of the proposed tip-of-the-tongue query generation pipeline including figure, descriptive cues, and generated query. The generated query is lightly shortened for presentation; only some cues are included.

Related Work

We position our dataset within existing tip-of-the-tongue, multimodal, and scientific retrieval tasks. We also discuss methods for synthetic query elicitation to motivate our approach. Table 1 compares currently available scientific and ToT benchmarks, illustrating that our benchmark (SciToT) addresses a gap, and generally aligns with previous work in terms of scale.

Benchmark#Docs#QQuery GenerationToTSci.Vis.
MS-ToT [8]69,1321,000Human (collected)✓––
TREC-ToT ’25 [7]6.4M622Mixed (collected + generated)✓––
Multilingual ToT [40]3.42M†20,000LLM (generated)✓––
BLUR [70]–573Human (collected)✓–✓
ToT2MeM [16]82.5K*82.5K*Human (collected)✓–✓
SPIQA [63]26,156265,996Mixed (collected + generated)–✓✓
arXivDoc [48]8,210547LLM (generated, human-verified)–✓✓
SciToT (Ours)108,51211,499LLM (generated, human-verified)✓✓✓

Table 1. Comparison of ToT and scientific retrieval benchmarks. Query counts describe aggregated query set. *Paired video subset. †Combined Chinese, Japanese, and Korean corpora. A dash under #Docs indicates no fixed retrieval corpus. #Q is short for #Queries, Sci. for Scientific, Vis. for Visual.

Tip-of-the-tongue Retrieval

Arguello et al. [8] propose ToT retrieval as a known-item search task without reliable identifiers; they rely on conversational posts from social media. Subsequent work constructs ToT datasets for several domains. Initial work relies strongly on domains for which datasets can be collected from online interactions, including video-games [46], movies and books [15, 57, 59], and music [14]. Later work has increased the scale and diversity of benchmarks and methods, e.g., through a TREC task [4, 5, 7]. Sky et al. [70] introduce a domain-diverse and partially multilingual ToT corpus. He et al. [40] introduce a large-scale multilingual corpus for ToT Queries. Bhattacharyya et al. [16] introduce a large-scale collection of multimodal ToT queries including visual and textual elements from social media interactions, with a specific pairing of visual elements to textual descriptions. These works have shown that ToT evaluation can be expanded to domains where this information need is prevalent. Our contribution is the structured generation of ToT queries from complex multimodal documents, and the first ToT benchmark for the domain of scientific document retrieval.

Scientific Literature Search

Known-item search is especially prevalent in scientific literature search, with resources reporting up to 57% of queries as directly looking for a known item [42, 56, 69, 83]. Such queries are highly prone to failure, as users often cannot recall a direct identifier, resulting in many queries receiving zero results, or not retrieving the intended document [13, 69]. Schultheiß et al. [69] report that users often rely on query expansion when they cannot recall a query. Importantly, people are highly effective at remembering and recognizing images [22], with a strong capacity for remembering visually distinctive items [44, 52]. When remembering visual features of scientific papers, users are highly likely to recall semantic details of images when describing them in addition to structural elements [21, 23, 53, 60]. Our benchmark builds on these key insights, which are backed by our own empirical analysis, and provides a method to generate synthetic scientific search queries that are aligned with real-world queries in style and composition.

Query Elicitation

Query elicitation generates synthetic queries from ground-truth documents [64], supporting retrieval model training [61, 62] and evaluation [9, 39, 41]. Early approaches employed statistical term-selection methods for query generation [10, 12, 31, 51]. Large language models (LLMs) have significantly increased the possibilities for synthetic dataset generation for information retrieval, generating labels [34, 75], queries [3, 65], or test collections [64]. Recent frameworks generate queries from target documents [20, 45], guide generation through task-specific prompts [29], and construct synthetic test collections [64]. He et al. [41] introduce a framework for query elicitation for ToT simulated evaluation; by fine-tuning instructions based on human-elicited queries they reach high rank correlation as well as high linguistic similarity. Later work has applied this framework for multilingual ToT-query elicitation [40] and for evaluation [6, 7]. Structured LLM-based query elicitation is an effective method for generating reliable and effective queries, generalizing across domains, which suggests that these methods can be effectively applied for the generation of scientific ToT queries. What we add to this work is a structured method for query generation based on distinct elements for scientific documents, by extracting clues from figures and text and combining these in a single query.

Scientific Tip-of-the-Tongue Information Needs

We follow the definition of a ToT information need by Arguello et al. [8] as “an item identification task where the searcher has previously experienced or consumed the item, but cannot recall a reliable identifier.” We consider two requirements for successful scientific ToT query elicitation: (i) alignment of the query with ToT information needs, assessed through an empirical characterization of re-finding requests in the scientific domain (§ 3.1). (ii) alignment of the query with the target document, which we evaluate through human annotation as well as LLM-based filtering (§ 4.5 and § 4.6).

Empirical Characterization of Scientific ToT Queries

Previous work in ToT retrieval [8, 40, 41] and studies of scientific literature search [54, 55, 69] identify characteristics of queries such as verbosity and uncertainty. However, to the best of our knowledge, no prior work has empirically characterized scientific ToT queries in practice. We annotate a sample of publicly available re-finding requests collected from CQA platforms. The annotation codebook is constructed using a hybrid deductive–inductive procedure [37]. An initial set of codes was derived from prior work on ToT retrieval [8]. We then applied and refined these codes while examining the collected queries, introducing additional categories when recurring forms of recollection are not captured by the initial scheme following Bolotova et al. [19]. The final set of codes was determined through discussion by the authors. Named-entity associations were excluded from codes to remain consistent with prior ToT and scientific re-finding work, which typically excludes these. Annotations were made by two annotators with scientific backgrounds, reporting annotator agreement with Cohen’s κ\kappa.

Table 2 summarizes our annotations, together with examples and their frequencies within the collected sample of N=51N = 51 queries. Our coding splits document-based (visual and structural) elements, which require the analysis of the target document, from user-based elements (Contextual and Social/ Other), which are user-specific, and may be generated without document knowledge [41].

CodeDefinitionFreq.κ\kappa
Textual
Research topicBroad scientific subject or problem studied93.1%0.85
Specific ClaimSpecific proposition, argument, or conclusion35.3%0.74
MethodMethod, model, algorithm, or analysis used38.2%0.71
Document Struct.General structure of information in the document3.9%1.00
TerminologyRemembered technical term, phrase, or wording19.6%0.75
MotivationPractical use case or reason for the work4.9%0.79
ComparisonRelation to another method, paper, or concept5.9%0.65
Exclusion CriteriaSomething the target is known not to be or contain9.8%0.78
Result/FindingGeneral outcome or observation of the target28.4%0.95
DatasetDataset, corpus, sample, or participants used6.9%0.55
Visual
PresentMemory that a visual element was present15.7%1.00
Graph TypeDescribes the type of graph or visual11.8%1.00
StructuralStructure, colours, or arrangement of a visual7.8%0.73
Semantic DetailsMeaning of elements within a remembered visual8.8%0.88
Contextual
Temporal{sup}*When the searcher previously encountered the target34.3%0.96
Academic ActivityAcademic context in which the target was found4.9%0.79
Physical RelationRemembered physical or spatial relation41.2%0.52
Social/Other
Uncertainty{sup}*Explicit doubt or alternative recollection74.5%0.59
Previous Search{sup}*Earlier attempts to locate or identify the target30.4%0.68
Failed Recall{sup}*Explicit inability to remember a detail26.5%0.65
Social{sup}*Politeness, thanks, apology, or request for help85.3%0.77
Emotion{sup}*Emotional reaction to retrieval failure7.8%0.73

Table 2. Codebook for characteristics in scientific tip-of-the-tongue queries. Freq. describes the annotated queries containing each characteristic. Cohen’s κ\kappa measures inter-annotator agreement. Asterisks ∗\ast denote codes reused from [8].

Dataset Construction

In this section, we discuss the construction of the SciToT benchmark, following the pipeline shown in Figure 2. We extract visual and textual elements from scientific documents, generate clues and queries, and validate and filter the resulting queries through human annotation and an LLM judge.

Pipeline for multimodal ToT query elicitation and validation for scientific documents

Figure 2. Pipeline for multimodal ToT query elicitation and validation for scientific documents. Numerical annotations refer to the sections discussing each element.

Document Collection

We construct our document collection from three sources and five scientific domains: Computational Linguistics from the ACL Anthology,1 Medicine and Biology from PubMed Central (PMC),2 and Engineering and Physics from arXiv.3 We include only documents explicitly released under CC0 or CC BY licenses, permitting redistribution and reuse of documents, with CC BY requiring appropriate attribution [27, 28]. The final collection consists of five sets of 20K+ documents, resulting in a final corpus size of 100K+ documents, and allows for evaluation across a range of different domains and visualizations.

Visual and Textual Element Extraction

To generate clues and enable textual and visual retrieval methods, we extract visual and textual elements from PDF documents. Content extraction from documents is a substantial field of research [43, 50, 58, 76, 79, 80]. Since content extraction is not the focus of this work, we rely on MinerU [76], which satisfies the requirements of being open-source, has a low compute cost and is fine-tuned for scientific documents. For each document, we extract the text into markdown, and extract figures and tables into separate images. Qualitative inspection of extracted documents indicated that textual content generally preserved the source material and that figures and tables were generally correctly identified.

Memory Clue Extraction

To facilitate query generation, we generate potentially remembered clues from the target document. Memory clues are extracted from both images and text. To ensure that clues are only related to a single modality and element, we separate clue extraction for text and images, following previous work on query generation [72].

For textual clues, we apply Qwen3-4B [82], providing it with instructions and the text of the paper, truncated to the token limit, following He et al. [41]. The model is instructed to output JSON-formatted text with a set of unstylized derived clues that are non-specific, and to exclude named entities or directly identifying information. For visual clues, we apply Qwen3-VL-4B-Instruct [11]. The model is provided with a single image and is instructed to derive clues about the structural appearance as well as high-level semantic elements.

ToT Query Elicitation

Our ToT query elicitation generates a query from the set of clues of the target document. We apply Phi-4 14B [1], using a different model provider from cue extraction to reduce single-provider bias and stylistic monotonicity [17, 49, 71, 84]. We generate queries based on both modalities, selecting the visual clues of at most 15 figures to stay within token limits. We closely follow the prompt style of He et al. [41], providing the model with general role-play instructions, the clue set, a set of strict requirements (musts), and optional requirements (coulds). The prompt was adjusted to align with the scientific re-finding task, instructing the model to apply the clues exactly, and to follow the clue style elements described in § 3.1. We exclude instruction of the model to provide temporal or physical contextual clues, as these would require further complex user simulation, without adding a significant new challenge for retrieval. For visual elements, we instruct the model to focus on figures that stand out to it, as these are most likely to be remembered [44, 52]. All prompts are made available in our code repository.

Human Annotation

To assess the quality of generated ToT queries, we conduct a manual evaluation of 100 queries sampled from the test set with 20 queries per scientific domain. Two independent annotators with research backgrounds, neither involved in the development of the benchmark, evaluate all 100 query–document pairs. Annotators are presented with the generated query and the full source document and independently assess two dimensions: relevance, measuring the correspondence between the query and its source document, and well-formedness, assessing the clarity and interpretability of the query. Both dimensions are evaluated on three-point scales (0–2), corresponding to non-relevant, partially relevant, and highly relevant for relevance, and not well-formed, partially well-formed, and well-formed for well-formedness, aligning with previous work [47, 66, 77]. Annotators received written guidelines containing label definitions and illustrative examples and completed a calibration round using a separate set of examples covering all annotation categories during a one-hour training session. We describe annotator judgments and report the label distributions and inter-annotator agreement using weighted Cohen’s κ\kappa for each dimension. Annotator instructions are made available in our code repository.

Filtering

We systematically verify the alignment of the target documents with the generated query. We do so by applying LLM-as-a-Judge to verify whether the query components are supported by the document, and if the final query aligns with the document in general. We apply Gemma4-12B-IT [74], aligning with previous patterns of applying a different model family for evaluation [30], as well as using a model with long-context capacity to ensure sufficient encoding capacity [73]. The judge is supplied with the figures and full text of the target document, as well as the target query. The judge is instructed to assess each remembered component conservatively, and provide judgments with justifications; following this, it is instructed to take all components into account, and grade the target query on the same three-point scale as used by annotators. In filtering documents, we follow the approach of Weller et al. [81], discarding documents that do not align with the target. We discard all 0 relevance score elements for the training and test set, and report on the agreement with annotators.

Dataset Validation

Validation of generated queries is carried out following the two components described in § 3. We first describe accuracy statistics for the Query dataset derived from human relevance annotations and LLM-as-a-Judge filtering, and then detail the statistical and qualitative alignment with tip-of-the-tongue queries.

Query-Document Alignment

Table 3 summarizes human evaluations of 100 test queries. For relevance, 92.5% of annotations were annotated as highly relevant, although inter-annotator agreement was limited, potentially reflecting the known sensitivity of Cohen’s κ\kappa to highly skewed label distributions [24, 36]. For well-formedness, 94.5% of annotations received the highest score, but no meaningful chance-corrected agreement was observed. We attribute this to differences in annotators’ preferences regarding query formulation, and limited difference in query formulation quality.

Relevance (κ=0.210\kappa= 0.210)
ScoreFreq. (%)
00.0
17.5
292.5
Well-formedness (κ=−0.034\kappa= -0.034)
ScoreFreq. (%)
00.0
15.5
294.5

Table 3. Distribution of human annotation scores for relevance and well-formedness. Frequencies are averaged across 2 annotators. Linear-weighted Cohen’s κ\kappa measures inter-annotator agreement on paired items.

In Table 4 we show the LLM judge results, and agreement with human annotators. Overall, we observe the judge is slightly more lenient than human annotators, and assigns a score of 2 more often. We do find that it is better aligned with human judges than inter-human agreement, with higher agreement for both annotators. Overall, the automated judgments should primarily be interpreted as a filtering mechanism rather than a fine-grained measure of query quality. Across the full dataset, the judge identifies two non-relevant queries, which were manually confirmed and removed. The relatively modest agreement with human annotations limits the conclusions that can be drawn about the reliability of this filtering process. These findings support the overall quality of the generated queries while highlighting differences in evaluation judgments.

Annotated Set (κˉ=0.362\bar{\kappa} = 0.362)Full Dataset
ScoreFreq. (%)ScoreFreq.Freq. (%)
00.0020.02
13.013693.21
297.0211,13096.77

Table 4. Distribution of LLM-assigned relevance scores for the annotated test subset and all saved LLM judgements. Linear-weighted Cohen’s κˉ\bar{\kappa} is averaged across the two LLM–annotator comparisons (0.421 and 0.302).

ToT Query Alignment

We assess alignment between generated and real-world ToT queries using the annotation procedure described in § 3.1, applying the same codebook and annotators to N=50N = 50 generated queries. Figure 3 compares code frequencies across three groups separated by dashed lines. The rightmost group contains contextual elements deliberately excluded from generation, as they require modelling user history beyond paper content, as noted in § 4.4. The middle group comprises visual elements, which are overrepresented by design to emphasize multimodal retrieval. The leftmost group contains elements moderated in frequency by the generation prompt, where we find high agreement, with an average 17.8% difference in frequency. A notable difference includes that we observe fewer specific claims, likely because the selected sample of real-world queries included many mathematics-related requests for proofs. We also find more references to methods, document structure, and applications, which can be explained by the tendency of LLM-extracted clues to emphasize these aspects. Overall, generated queries reproduce much of the diversity and relative frequency of characteristics observed in real-world ToT queries.

Figure 3. Distribution of code annotations found in online queries compared to those found in generated queries. Results are grouped into naturally generated, explicitly oversampled, and explicitly excluded items.

Retrieval Benchmark

Scientific ToT retrieval inherits several challenges from traditional ToT retrieval, including incomplete recollections and uncertain descriptions [8, 41]. These challenges are compounded in scientific literature search, where relevant information may be confined to a small portion of a lengthy document. To facilitate future work and provide insights into the challenges associated with our dataset, we provide implementations and evaluations of several baseline methods.

We evaluate scientific ToT retrieval as a known-item search task, where the objective is to retrieve the single target document from a collection based on an incomplete recollection query. Each scientific domain is evaluated independently, allowing future work to investigate both domain-specific retrieval effectiveness and cross-domain generalization. Since each query has a single target document, we report MRR@10, Recall@1, Recall@10, and Recall@100.

Retrieval Baselines

We evaluate pretrained retrieval baselines across four configurations: textual, visual, page-level, and multimodal retrieval. For textual retrieval, we consider BM25 [67] for lexical matching, E5-small-v2 [78] and Qwen3-Emb-0.6B [82] for dense semantic retrieval, and ColBERTv2 [68] for token-level late interaction. For visual and page-level retrieval, we evaluate OpenCLIP [26] and Qwen3-VL [11]. For page-level retrieval, we also apply ColQwen2.5, a variant of the ColPali late interaction method for VLM retrieval [35]. We additionally evaluate Qwen3-VL for multimodal retrieval using both extracted text and figures. All exact model versions and implementation details can be found in our public repository.

All methods are evaluated without fine-tuning. Textual methods use extracted paper text, truncated to satisfy model input limits. OpenCLIP encodes individual figures or pages, using maximum cosine similarity for relevance scores.

For Qwen3-VL we jointly encode all available inputs for each configuration, truncated to the token limit, producing a single document-level embedding to allow for in-document context. ColQwen2.5 is designed as a page-level method, to retain this architecture, we apply individual page embeddings and use the max-similarity in the ranking stage [35]. To reduce the size of the index to a feasible size for our storage budgets, we apply built-in hierarchical token pooling.

Benchmark Results

In Table 5 we show the retrieval effectiveness results. For textual methods, we observe that Qwen3-Emb substantially outperforms traditional retrieval baselines. Textual baselines BM25, E5, and ColBERTv2 achieve relatively low effectiveness (MRR@10 of 0.115–0.128), compared to 0.710 for Qwen3-Emb using text alone. These results may be explained by noise in queries, limited context length (512 tokens for E5, and 32 query tokens for ColBERTv2). For visual methods, we find poor performance for OpenCLIP, indicating that visual information alone is insufficient without representations capable of capturing more complex semantic relationships. Notably, we find that Qwen3-VL performs relatively well using only image elements, indicating that visual query descriptions can be a useful retrieval signal in this task. Page-level retrieval with Qwen3-VL achieves the highest effectiveness (MRR@10 of 0.793). Furthermore, including both text and visual modalities also improves retrieval effectiveness for Qwen3-VL. For ColQwen2.5, we find that it performs worse than Qwen3-VL, and requires significantly more index storage, which aligns with retrieval results reported in previous work [48]. We do note that Qwen3-based models are used for both clue extraction and retrieval; it is possible that representational biases may favor these retrievers, although the query generation model is different.

MethodModalityMRR@10R@1R@10R@100Index Size
BM25Text0.1230.0950.1940.3710.113
E5Text0.1280.0900.2280.4510.033
ColBERTv2Text0.1150.0980.1500.1970.144
Qwen3-EmbText0.7100.6530.8250.9140.089
OpenCLIPVisual0.0350.0200.0780.1881.093
Qwen3-VLVisual0.6370.5690.7750.9030.177
OpenCLIPPage0.0100.0050.0240.0910.557
Qwen3-VLPage0.7930.7460.8810.9320.178
ColQwen2.5Page0.6380.5840.7630.85878.982
Qwen3-VLText + Visual0.7500.6990.8430.8990.178

Table 5. Retrieval effectiveness across methods and modalities. Each cell is an unweighted mean over available results from the five retrieval datasets; missing values are excluded. Index size (decimal GB) is averaged over datasets.

Our findings highlight that traditional retrieval methods can struggle with scientific ToT retrieval, and reveal opportunities for developing efficient yet finegrained multimodal retrieval methods that align uncertain memory cues with localized document content. Furthermore, data augmentation methods such as query rewriting, document summarization, or image description extraction may lead to improvements for methods which are limited by context length.

Resource Availability and Intended Use

We release SciToT on Hugging Face with a persistent DOI, including source documents, extracted textual and visual content, memory cues, queries, judgments, and evaluation splits.4 The accompanying datasheet [38] documents the resource’s composition, collection and preparation procedures, intended uses, and maintenance policy. Furthermore, we release the dataset generation pipeline5 and retrieval implementations,6 which support further development and dataset downloading. All generated resources and code are released under MIT licenses, while source documents retain their original CC0 or CC BY licenses.

SciToT supports the development and evaluation of scientific ToT retrieval methods, particularly multimodal retrieval from noisy and incomplete recollections. The generation pipeline also enables the construction of related benchmarks in other tasks or domains. The resource is intended for permanent availability, with updates managed through the public repositories.

Conclusion

We have introduced SciToT, a benchmark for scientific ToT retrieval, comprising over 100K scientific documents across five domains. We have proposed a scalable query elicitation pipeline that combines textual and visual memory cues to generate queries reflecting incomplete recollections of scientific papers. Human evaluation has demonstrated high query–document alignment and well-formedness, while an empirical comparison with real-world scientific re-finding requests showed that the generated queries capture many of their characteristics.

Our retrieval experiments highlighted the challenges of scientific ToT retrieval. Conventional lexical and dense retrieval baselines achieved limited effectiveness, but more expressive document representations improved retrieval performance, highlighting the importance of matching remembered information to localized content within scientific documents and capturing visual elements.

The synthetic nature of the queries in the SciToT benchmark and their deliberate emphasis on visual clues may limit their representativeness of naturally occurring requests. Users of SciToT should be mindful of these limitations when training or fine-tuning their retrieval models. We encourage future work on expanding the coverage and resources of SciToT, in terms of both ToT query (sub)types and additional scientific domains. We also call for research on generalizability across these two dimensions.

References

  1. [1]Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R.J., Javaheripi, M., Kauffmann, P., et al.: Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024)arxiv.org/abs/2412.08905
  2. [2]Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., Gao, T.: Lit-Search: A retrieval benchmark for scientific literature search. arXiv preprint arXiv:2407.18940 (2025)arxiv.org/abs/2407.18940
  3. [3]Alaofi, M., Gallagher, L., Sanderson, M., Scholer, F., Thomas, P.: Can generative LLMs create query variants for test collections? An exploratory study. In: Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, pp. 1869–1873 (2023)DOI
  4. [4]Arguello, J., Bhargav, S., Diaz, F., Kanoulas, E., Mitra, B.: Overview of the TREC 2023 tip-of-the-tongue track. In: TREC 2023 Proceedings (2024)
  5. [5]Arguello, J., Bhargav, S., Diaz, F., Kim, T.E., He, Y., Kanoulas, E., Mitra, B.: Overview of the TREC 2024 tip-of-the-tongue track. In: TREC 2024 Proceedings (2025)
  6. [6]Arguello, J., Diaz, F., Fröbe, M., He, X., Kim, T.E., Mitra, B.: NTCIR-19 Tip-of-the-Tongue (ToT) Shared Task. https://ntcir-tot.github.io/ (2026), NTCIR-19: The 19th NTCIR Conference on Evaluation of Information Access Technologies
  7. [7]Arguello, J., Diaz, F., Fröbe, M., Kim, T.E., Mitra, B.: Overview of the TREC 2025 tip-of-the-tongue track. arXiv preprint arXiv:2601.20671 (2026)arxiv.org/abs/2601.20671
  8. [8]Arguello, J., Ferguson, A., Fine, E., Mitra, B., Zamani, H., Diaz, F.: Tip of the tongue known-item retrieval: A case study in movie identification. In: Proceedings of the 2021 Conference on Human Information Interaction and Retrieval, pp. 5–14 (2021)
  9. [9]Azzopardi, L., de Rijke, M.: Automatic construction of known-item finding test beds. In: 29th Annual International ACM SIGIR Conference on Research & Development on Information Retrieval, pp. 603–604, ACM (2006)DOI
  10. [10]Azzopardi, L., de Rijke, M., Balog, K.: Building simulated queries for known-item topics: An analysis using six European languages. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 455–462 (2007)
  11. [11]Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)arxiv.org/abs/2511.21631
  12. [12]Balog, K., Azzopardi, L., Kamps, J., de Rijke, M.: Overview of webclef 2006. In: Workshop of the Cross-Language Evaluation Forum for European Languages, pp. 803–819, Springer (2006)DOI
  13. [13]Behnert, C., Lewandowski, D.: Known-item searches resulting in zero hits: considerations for discovery systems. The Journal of Academic Librarianship 43(2), 128–134 (2017)DOI
  14. [14]Bhargav, S., Schuth, A., Hauff, C.: When the music stops: Tip-of-the-tongue retrieval for music. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2506–2510 (2023)DOI
  15. [15]Bhargav, S., Sidiropoulos, G., Kanoulas, E.: ‘It’s on the tip of my tongue’ A new dataset for known-item retrieval. In: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 48–56 (2022)DOI
  16. [16]Bhattacharyya, S., Singla, Y.K., Yarram, S., Singh, S., Si, H., Wang, J.Z.: Unsupervised memorability modeling from tip-of-the-tongue retrieval queries. In: 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4234–4241, IEEE (2026)arxiv.org/abs/2511.20854
  17. [17]Bitton, Y., Bitton, E., Nisan, S.: Detecting stylistic fingerprints of large language models. arXiv preprint arXiv:2503.01659 (2025)arxiv.org/abs/2503.01659
  18. [18]Bogers, T., Gäde, M., Hall, M., Koolen, M., Petras, V., Skov, M.: Exploring the zero-shot known-item retrieval capabilities of LLMs for casual leisure information needs. In: Proceedings of the 2025 ACM SIGIR Conference on Human Information Interaction and Retrieval, pp. 316–325 (2025)DOI
  19. [19]Bolotova, V., Blinov, V., Scholer, F., Croft, W.B., Sanderson, M.: A non-factoid question-answering taxonomy. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1196–1207 (2022)DOI
  20. [20]Bonifacio, L., Abonizio, H., Fadaee, M., Nogueira, R.: Inpars: Data augmentation for information retrieval using large language models. arXiv preprint arXiv:2202.05144 (2022)arxiv.org/abs/2202.05144
  21. [21]Borkin, M.A., Bylinski, Z., Kim, N.W., Bainbridge, C.M., Yeh, C.S., Borkin, D., Pfister, H., Oliva, A.: Beyond memorability: Visualization recognition and recall. IEEE Transactions on Visualization and Computer Graphics 22(1), 519–528 (2015)DOI
  22. [22]Brady, T.F., Konkle, T., Alvarez, G.A., Oliva, A.: Visual long-term memory has a massive storage capacity for object details. Proceedings of the National Academy of Sciences 105(38), 14325–14329 (2008)DOI
  23. [23]Brady, T.F., Konkle, T., Gill, J., Oliva, A., Alvarez, G.A.: Visual long-term memory has the same limit on fidelity as visual working memory. Psychological Science 24(6), 981–990 (2013)DOI
  24. [24]Byrt, T., Bishop, J., Carlin, J.B.: Bias, prevalence and kappa. Journal of Clinical Epidemiology 46(5), 423–429 (1993)pubmed.ncbi.nlm.nih.gov/8501467
  25. [26]Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., Jitsev, J.: Reproducible scaling laws for contrastive language-image learning. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2829, IEEE (2023)DOI
  26. [27]Creative Commons: Attribution 4.0 international (CC BY 4.0). https://creativecommons.org/licenses/by/4.0/ (n.d.), accessed: 2026-09-18
  27. [28]Creative Commons: CC0 1.0 universal. https://creativecommons.org/publicdomain/zero/1.0/ (n.d.), accessed: 2026-09-18
  28. [29]Dai, Z., Zhao, V.Y., Ma, J., Luan, Y., Ni, J., Lu, J., Bakalov, A., Guu, K., Hall, K.B., Chang, M.W.: Promptgator: Few-shot dense retrieval from 8 examples. arXiv preprint arXiv:2209.11755 (2022)arxiv.org/abs/2209.11755
  29. [30]Dietz, L., Zendel, O., Bailey, P., Clarke, C.L., Cotterill, E., Dalton, J., Hasibi, F., Sanderson, M., Craswell, N.: Principles and guidelines for the use of llm judges. In: Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR), pp. 218–229 (2025)DOI
  30. [31]Elsweiler, D., Losada, D.E., Toucedo, J.C., Fernandez, R.T.: Seeding simulated queries with user-study data for personal search evaluation. In: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pp. 25–34 (2011)
  31. [32]Elsweiler, D., Ruthven, I., Jones, C.: Towards memory supporting personal information management tools. Journal of the American Society for Information Science and Technology 58(7), 924–946 (2007)DOI
  32. [33]Elsweiler, D., Wilson, M.L., Kirkegaard Lunn, B.: Understanding casual-leisure information behaviour. In: Proceedings of the Third Symposium on Information Interaction in Context (2010)
  33. [34]Faggioli, G., Dietz, L., Clarke, C.L., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., Stein, B., et al.: Perspectives on large language models for relevance judgment. In: Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval, pp. 39–50 (2023)DOI
  34. [35]Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot, C., Colombo, P.: Colpali: Efficient document retrieval with vision language models. In: International Conference on Learning Representations, vol. 2025, pp. 61424–61449 (2025)arxiv.org/abs/2407.01449
  35. [36]Feinstein, A.R., Cicchetti, D.V.: High agreement but low Kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology 43(6), 543–549 (1990)pubmed.ncbi.nlm.nih.gov/2348207
  36. [37]Fereday, J., Muir-Cochrane, E.: Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International Journal of Qualitative Methods 5(1), 80–92 (2006)
  37. [38]Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wallach, H., III, H.D., Crawford, K.: Datasheets for datasets. Commun. ACM 64(12), 86–92 (Nov 2021)arxiv.org/abs/1803.09010
  38. [39]Hagen, M., Wä gner, D., Stein, B.: A corpus of realistic known-item topics with associated web pages in the ClueWeb09. In: European Conference on Information Retrieval, pp. 513–525, Springer (2015)DOI
  39. [40]He, X., Kim, T.E., Fröbe, M., Arguello, J., Mitra, B., Diaz, F.: Multilingual and domain-agnostic tip-of-the-tongue query generation for simulated evaluation. In: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3228–3234 (2026)
  40. [41]He, Y., Kim, T.E., Diaz, F., Arguello, J., Mitra, B.: Tip of the tongue query elicitation for simulated evaluation. In: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3398–3407 (2025)arxiv.org/abs/2502.17776
  41. [42]Herskovic, J.R., Tanaka, L.Y., Hersh, W., Bernstam, E.V.: A day in the life of PubMed: Analysis of a typical day’s query log. Journal of the American Medical Informatics Association 14(2), 212–220 (2007)pubmed.ncbi.nlm.nih.gov/17213501
  42. [43]Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L., Zhang, B., Zhang, J., Jin, Q., Huang, F., Zhou, J.: mPLUG-DocOwl 1.5: Unified structure learning for OCR-free document understanding. In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 3096–3120 (2024)arxiv.org/abs/2403.12895
  43. [44]Isola, P., Xiao, J., Torralba, A., Oliva, A.: What makes an image memorable? In: Proceedings of the 2011 IEEE Computer Vision and Pattern Recognition (CVPR 2011), pp. 145–152 (2011)
  44. [45]Jeronymo, V., Bonifacio, L., Abonizio, H., Fadaee, M., Lotufo, R., Zavrel, J., Nogueira, R.: Inpars-v2: Large language models as efficient dataset generators for information retrieval. arXiv preprint arXiv:2301.01820 (2023)arxiv.org/abs/2301.01820
  45. [46]Jørgensen, I.K.H., Bogers, T.: “kinda like the sims... but with ghosts?”: A qualitative analysis of video game re-finding requests on reddit. In: Proceedings of the 15th International Conference on the Foundations of Digital Games, pp. 1–4 (2020)DOI
  46. [47]Kato, M.P., Ohshima, H., Liu, Y.H., Chen, H.L.: A test collection for ad-hoc dataset retrieval. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2450–2456 (2021)DOI
  47. [48]Khalighinejad, G., Thirukovalluru, R., Oh, A.H., Dhingra, B.: Document-as-image representations fall short for scientific retrieval. arXiv preprint arXiv:2604.18508 (2026)arxiv.org/abs/2604.18508
  48. [49]Kim, E., Garg, A., Peng, K., Garg, N.: Correlated errors in large language models. arXiv preprint arXiv:2506.07962 (2025)arxiv.org/abs/2506.07962
  49. [50]Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., Park, S.: OCR-free document understanding transformer. In: European Conference on Computer Vision, pp. 498–517, Springer (2022)arxiv.org/abs/2111.15664
  50. [51]Kim, J., Croft, W.B.: Retrieval experiments using pseudo-desktop collections. In: Proceedings of the 18th ACM Conference on Information and Knowledge Management, pp. 1297–1306 (2009)DOI
  51. [52]Konkle, T., Brady, T.F., Alvarez, G.A., Oliva, A.: Conceptual distinctiveness supports detailed visual long-term memory for real-world objects. Journal of Experimental Psychology: General 139(3), 558 (2010)DOI
  52. [53]Kramer, M.A., Hebart, M.N., Baker, C.I., Bainbridge, W.A.: The features underlying the memorability of objects. Science Advances 9(17), eadd2981 (2023)DOI
  53. [54]Lee, J.H., Renear, A.: How incorrect information delivers correct search results: A pragmatic analysis of queries. Proceedings of the American Society for Information Science and Technology 44(1), 1–6 (2007)DOI
  54. [55]Lee, J.H., Renear, A., Smith, L.C.: Known-item search: Variations on a concept. In: Proceedings of the American Society for Information Science and Technology, vol. 43, pp. 1–17, Wiley Online Library (2006)DOI
  55. [56]Li, X., Schijnbaers, B.J., de Rijke, M.: Investigating queries and search failures in academic search. Information Processing & Management 53(3), 666–683 (2017)DOI
  56. [57]Lin, K., Lo, K., Gonzalez, J., Klein, D.: Decomposing complex queries for tip-of-the-tongue retrieval. In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5521–5533 (2023)arxiv.org/abs/2305.15053
  57. [58]Lv, T., Huang, Y., Chen, J., Zhao, Y., Jia, Y., Cui, L., Ma, S., Chang, Y., Huang, S., Wang, W., et al.: Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419 (2023)arxiv.org/abs/2309.11419
  58. [59]Meier, F., Bogers, T., Gäde, M., Edbrup Thomsen, L.: Towards understanding complex known-item requests on Reddit. In: Proceedings of the 32nd ACM Conference on Hypertext and Social Media, pp. 143–154 (2021)DOI
  59. [60]Moscovitch, C., Kapur, S., Köhler, S., Houle, S.: Distinct neural correlates of visual long-term memory for spatial location and object identity: a positron emission tomography study in humans. Proceedings of the National Academy of Sciences 92(9), 3721–3725 (1995)pubmed.ncbi.nlm.nih.gov/7731972
  60. [61]Ogilvie, P., Callan, J.: Combining document representations for known-item search. In: Proceedings of the 26th annual international ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 143–150 (2003)
  61. [62]Phanse, R., Liu, G.K.M., Cohan, A.: Yale NLP at TREC 2024: Tip-of-the-tongue track. In: TREC (2024)
  62. [63]Pramanick, S., Chellappa, R., Venugopalan, S.: SPIQA: A dataset for multimodal question answering on scientific papers. Advances in Neural Information Processing Systems 37, 118807–118833 (2024)arxiv.org/abs/2407.09413
  63. [64]Rahmani, H.A., Craswell, N., Yılmaz, E., Mitra, B., Campos, D.: Synthetic test collections for retrieval evaluation. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2647–2651 (2024)arxiv.org/abs/2405.07767
  64. [65]Rajapakse, T.C., de Rijke, M.: Improving the generalizability of the dense passage retriever using generated datasets. In: European Conference on Information Retrieval, pp. 94–109, Springer (2023)DOI
  65. [66]Roberts, K., Alam, T., Bedrick, S., Demner-Fushman, D., Lo, K., Soboroff, I., Voorhees, E., Wang, L.L., Hersh, W.R.: Searching for scientific evidence in a pandemic: An overview of TREC-COVID. Journal of Biomedical Informatics 121, 103865 (2021)arxiv.org/abs/2104.09632
  66. [67]Robertson, S.E., Walker, S., Jones, S., Hancock-Beaulieu, M., Gatford, M.: Okapi at TREC-3. In: Text Retrieval Conference (1994)
  67. [68]Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., Zaharia, M.: ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3715–3734 (2022)
  68. [69]Schultheiß, S., Linhart, A., Behnert, C., Rulik, I., Lewandowski, D.: Known-item searches and search tactics in library search systems: results from four transaction log analysis studies. The Journal of Academic Librarianship 46(5), 102202 (2020)DOI
  69. [70]Sky, C.W., Deshpande, D.G., Muresan, S., Kannappan, A., Qian, R.: Browsing lost uninformed recollections: A benchmark for tip-of-the-tongue search and reasoning. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8317–8331 (2025)
  70. [71]Spiliopoulou, E., Fogliato, R., Burnsky, H., Soliman, T., Ma, J., Horwood, G., Ballesteros, M.: Play favorites: A statistical method to measure self-bias in LLM-as-a-judge. arXiv preprint arXiv:2508.06709 (2025)arxiv.org/abs/2508.06709
  71. [72]Talmor, A., Yoran, O., Catav, A., Lahav, D., Wang, Y., Asai, A., Ilharco, G., Hajishirzi, H., Berant, J.: MultiModalQA: Complex question answering over text, tables and images. arXiv preprint arXiv:2104.06039 (2021)arxiv.org/abs/2104.06039
  72. [73]Tan, S., Zhuang, S., Montgomery, K., Tang, W., Cuadron, A., Wang, C., Popa, R., Stoica, I.: Judgebench: A benchmark for evaluating LLM-based judges. In: International Conference on Learning Representations, vol. 2025, pp. 63277–63303 (2025)arxiv.org/abs/2410.12784
  73. [74]Team, G., Abd, S.E., Aggarwal, V., Algayres, R., Andreev, A., Bachem, O., Ballantyne, I., Brick, C., Čárbune, V., Casbon, M., et al.: Gemma 4 technical report. arXiv preprint arXiv:2607.02770 (2026)
  74. [75]Thomas, P., Spielman, S., Craswell, N., Mitra, B.: Large language models can accurately predict searcher preferences. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1930–1940 (2024)arxiv.org/abs/2309.10621
  75. [76]Wang, B., Xu, C., Zhao, X., Ouyang, L., Wu, F., Zhao, Z., Xu, R., Liu, K., Qu, Y., Shang, F., et al.: Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839 (2024)arxiv.org/abs/2409.18839
  76. [77]Wang, J.A., Wang, K., Wang, X., Naidu, P., Bergen, L., Paturi, R.: Scientific document retrieval using multi-level aspect-based queries. Advances in Neural Information Processing Systems 36, 38404–38419 (2023)DOI
  77. [78]Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., Wei, F.: Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 (2022)arxiv.org/abs/2212.03533
  78. [79]Wei, H., Kong, L., Chen, J., Zhao, L., Ge, Z., Yang, J., Sun, J., Han, C., Zhang, X.: Vary: Scaling up the vision vocabulary for large vision-language model. In: European Conference on Computer Vision, pp. 408–424, Springer (2024)DOI
  79. [80]Wei, H., Kong, L., Chen, J., Zhao, L., Ge, Z., Yu, E., Sun, J., Han, C., Zhang, X.: Small language model meets with reinforced vision vocabulary. arXiv preprint arXiv:2401.12503 (2024)arxiv.org/abs/2401.12503
  80. [81]Weller, O., Chang, B., MacAvaney, S., Lo, K., Cohan, A., Durme, B., Lawrie, D., Soldaini, L.: FollowIR: Evaluating and teaching information retrieval models to follow instructions. arXiv preprint arXiv:2403.15246 (2024)arxiv.org/abs/2403.15246
  81. [82]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)arxiv.org/abs/2505.09388
  82. [83]Yeganova, L., Kim, W., Comeau, D.C., Wilbur, W.J., Lu, Z.: A field sensor: computing the composition and intent of PubMed queries. Database 2018, bay052 (2018)DOI
  83. [84]Zhang, X., Zhai, J., Ma, S., Bao, Q., Jiang, W., Wang, Q., Shen, C., Liu, Y.: The invisible hand: Unveiling provider bias in large language models for code generation. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 21376–21403 (2025)arxiv.org/abs/2501.07849
  84. [85]Zhong, X., Tan, Z., Gao, S., Li, J., Shen, J., Ji, J., Tang, J., Chiu, B.: SMSMO: Learning to generate multimodal summary for scientific papers. Knowledge-Based Systems 310, 112908 (2025)DOI

Paper details

vlnc:W_gxga2nrw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents