WorksComputer Science
GEMS-RAG: Graph-grounded Engineering Multimodal Standards Retrieval-Augmented Generation
Abstract
Engineering standards are challenging knowledge sources for large language models (LLMs) because they are hierarchical, multimodal, and densely connected through figures, tables, cross-references, and normative rules. Existing retrieval-augmented generation (RAG) methods often rely on unstructured text retrieval and struggle with questions requiring joint reasoning over textual provisions, visual specifications, and document structure. We propose GEMS-RAG (Graph-grounded Engineering Multimodal Standards Retrieval-Augmented Generation), a framework for grounded reasoning over engineering standards. GEMS-RAG combines hierarchical parsing, fine-grained visual entity extraction, multimodal knowledge graph construction, and graph-grounded hybrid retrieval. Instead of treating figures as monolithic images, it decomposes them into meaningful visual entities linked to textual provisions, engineering concepts, and regulatory rules. During inference, dense, sparse, and visual retrieval are combined with graph expansion to identify relevant evidence across text, figures, and document structure. We evaluate GEMS-RAG on the Manual on Uniform Traffic Control Devices (MUTCD), a large engineering standard containing egulations, technical drawings, and standardized traffic-control devices. Results show that graph-grounded multimodal retrieval improves retrieval effectiveness and question-answering performance over text-only, multimodal, and graph-based RAG baselines. Beyond MUTCD, GEMS-RAG offers a general framework for reasoning over engineering codes, technical manuals, regulatory documents, and scientific standards in safety-critical applications.
main.pdf
Code — https://github.com/hannanazad/gemsrag.git
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in language understanding, reasoning, and question answering. However, their effectiveness remains fundamentally constrained by the knowledge encoded during pretraining [18]. Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for grounding LLMs in external knowledge by retrieving relevant documents at inference time and incorporating them into the generation process [21, 22, 9, 37]. While recent RAG systems have achieved substantial success on web documents [35], scientific literature [20], and domain knowledge bases [19], they remain poorly suited for regulatory and engineering standards documents that exhibit rich hierarchical organization, multimodal content, and normative reasoning requirements.
Engineering standards documents constitute a particularly challenging class of knowledge sources for retrieval-augmented generation. Unlike conventional text corpora, they combine hierarchical organization, normative language, technical specifications, figures, tables, design examples, and dense cross-references. For example, the Manual on Uniform Traffic Control Devices (MUTCD) [12] contains textual regulations, engineering guidance, standardized traffic-control devices, technical drawings, sign specifications, warrants, and cross-referenced provisions. In such documents, critical knowledge is rarely contained in a single passage; instead, correct interpretation often requires connecting textual rules with visual specifications, document structure, and regulatory context [10, 30].
Conventional RAG systems face three key limitations when applied to engineering standards documents such as the MUTCD. First, many retrieval pipelines represent documents as collections of independent text chunks, which can weaken the hierarchical context that determines the scope and interpretation of regulatory provisions. Second, even when multimodal content is considered, figures and diagrams are often indexed only at the page, caption, or image level, limiting access to the fine-grained visual semantics embedded in engineering symbols, standardized devices, and technical drawings. Third, standard dense or sparse retrieval does not explicitly model cross-references, regulatory dependencies, or relationships among sections, figures, tables, devices, and rule types. As a result, existing systems may produce incomplete, decontextualized, or inconsistent answers when standards-compliant reasoning requires joint evidence from text, visuals, and document structure.
To address these limitations, we propose GEMS-RAG, a Graph-grounded Engineering Multimodal Standards Retrieval-Augmented Generation framework for engineering standards documents which combines hierarchical document parsing, multimodal figure decomposition, dense and sparse retrieval, and graph-based knowledge organization within a unified retrieval architecture. Instead of treating figures as auxiliary content, we explicitly extract and index individual visual entities, such as traffic signs, plaques, and signal illustrations, and connect them to their corresponding textual provisions through a document knowledge graph. The resulting system enables retrieval over both textual and visual semantics while preserving the structural relationships that govern interpretation of the standard.
More specifically, we construct a multimodal knowledge graph that captures relationships among sections, figures, traffic control devices, regulatory concepts, and normative statements. During retrieval, graph-based expansion supplements traditional dense retrieval by identifying structurally related content, allowing the system to recover relevant provisions even when they are distributed across multiple sections and modalities. Retrieved textual and visual evidence are then jointly provided to a vision-language reasoning model for grounded response generation.
The contributions of this paper are threefold:
We introduce a standards-aware hierarchical multimodal framework that represents engineering standards as structured collections of text, figures, tables, visual entities, cross-references, and rule types rather than isolated chunks.
We propose a graph-grounded hybrid retrieval mechanism integrating hierarchical parsing, textual and visual retrieval, graph expansion, and rule-aware ranking to support reasoning across sections, devices, figures, and normative relationships.
We develop a multimodal RAG benchmark and system for the MUTCD, as an example, showing that fine-grained visual grounding, preserved document structure, and normative rule modeling improve retrieval and question answering over complex engineering standards.
Related Work
Retrieval-Augmented Generation for Domain-Specific Knowledge
RAG has become a widely adopted approach for grounding large language models in external knowledge. By retrieving relevant evidence at inference time, RAG reduces reliance on parametric memory and improves factuality in question answering over open-domain documents, scientific literature, and domain-specific knowledge bases [17]. Recent advances have further improved retrieval quality through hybrid dense–sparse retrieval, reranking, iterative retrieval, self-reflection, and adaptive retrieval planning [9, 32, 37]. These methods have demonstrated strong potential for knowledge-intensive reasoning tasks. Most existing RAG systems assume that knowledge can be represented as independent textual chunks. This assumption is inadequate for engineering standards, where meaning is distributed across section hierarchies, cross-references, tables, figures, captions, device drawings, and normative statements. A locally relevant passage may therefore be insufficient without its structural location, visual specifications, and related provisions. Consequently, text-centered RAG may retrieve relevant fragments yet still produce noncompliant answers. This limitation motivates a standards-aware architecture that preserves document hierarchy and regulatory context throughout retrieval and generation.
Multimodal Understanding of Engineering Standards Documents
Multimodal retrieval and vision-language models (Mathew, Karatzas, and Jawahar 2021; Xia et al. 2025; Li et al. 2025; Kuai et al. 2026b) provide an important foundation for reasoning over visual and textual information. Shared image–text embedding models and recent multimodal foundation models have shown strong capabilities in visual question answering, document understanding, chart interpretation, and image-grounded generation. These developments have motivated multimodal RAG systems that retrieve images, tables, document pages, or visual regions alongside textual evidence (Yu et al. 2025). Engineering standards present a specialized form of multimodality. Their visuals often encode formal concepts, standardized devices, geometric layouts, operational constraints, and regulatory specifications. For example, traffic signs, plaques, lane-use arrows, signal faces, and pavement markings are regulated entities with defined meanings, usage conditions, and governing provisions. Treating a figure as a monolithic image or relying only on its caption therefore obscures the fine-grained semantics required for grounded reasoning. Standards-centric multimodal RAG must explicitly extract, index, and link visual entities to their textual and regulatory context.
Graph-Grounded Retrieval and Structural Reasoning
Knowledge graph-augmented retrieval offers a natural mechanism for representing relationships among entities, concepts, documents, and rules (Procko and Ochoa 2024; Park et al. 2026; Hsiao et al. 2026; Luo et al. 2026; Zhu et al. 2025; Wu et al. 2026). Graph-based RAG methods can expand retrieved evidence through neighboring nodes, connect semantically related concepts, and support multi-hop reasoning beyond lexical similarity. These capabilities are especially relevant for engineering standards, where correct interpretation often depends on relationships among sections, figures, tables, definitions, cross-references, and rule types. Nevertheless, most existing graph-RAG approaches build graphs from textual entities, citations, or domain ontologies. They rarely treat engineering visuals as first-class entities or jointly represent visual objects, document hierarchy, regulatory concepts, and normative statements. Likewise, document intelligence methods can parse layouts, tables, and figures, but usually serve only as preprocessing rather than supporting graph-grounded retrieval and reasoning.
To address these limitations, this paper proposes GEMS-RAG, a graph-grounded multimodal RAG framework for engineering standards. GEMS-RAG preserves document hierarchy, decomposes figures into fine-grained visual entities, and constructs a multimodal knowledge graph linking sections, figures, traffic-control devices, regulatory concepts, and normative statements. By combining dense, sparse, and visual retrieval with graph expansion, it supports grounded reasoning across the textual, visual, and structural semantics of engineering standards. As shown in Figure 1, GEMS-RAG parses standards by their native hierarchy, extracts captioned figures and tables, links textual and visual elements in a knowledge graph, retrieves multimodal evidence, and adaptively gates visual evidence per query.

Figure 1. Framework of GEMS-RAG
Methodology
Problem Formulation
Let denote a standards document containing textual passages, figures, tables, and cross-references. We represent each document element as , where denotes the textual content, denotes the visual content, and contains structural metadata, including the page number, section identifier, rule type, and figure or table identifier. Given a user query , the system retrieves a supporting evidence set
and generates an answer
The generated answer must ground its claims in the retrieved evidence and cite identifiers that resolve to valid sections, figures, or tables in the underlying document. Visual elements are included only when the query requires visual evidence.
Hierarchical Document Parsing
Many engineering standards organize their content into parts, chapters, sections, and typed paragraphs. GEMS-RAG preserves this native structure rather than treating the document as an unstructured sequence of pages.
Hierarchical Document Parsing
Each textual chunk is assigned a hierarchy label
where , , and denote the part, chapter, section, and normative rule type of chunk , respectively. Text is segmented by paragraph within each section, with each chunk retaining its heading, page number, and related figure, table, and section references. The parser also normalizes identifier variants to ensure consistent cross-reference matching.
Caption-Anchored Figure and Table Extraction
GEMS-RAG uses detected captions as spatial anchors to extract figures and tables from each page. For a caption on a page of height , the corresponding visual region is defined by
where denotes the next caption below , and define the relevant caption boundaries, and and provide small spatial margins. The extraction procedure adapts to variations in page layout, including multi-column pages, horizontally aligned visuals, and figures distributed across multiple panels or pages. Spatial cues from caption position, column boundaries, and neighboring document elements are used to isolate the corresponding visual content. Visual components that share the same document identifier are subsequently consolidated into a single canonical entity during graph construction. A secondary recovery procedure is applied when an identified figure or table is not successfully extracted during the initial pass. This procedure relaxes the spatial constraints to improve coverage of small or irregularly formatted visual elements. Extraction completeness is verified through a document-wide consistency check between referenced visual identifiers and the figures and tables recovered by the pipeline. Identifiers corresponding to external documents are excluded from this comparison. Any unresolved references are flagged for additional extraction or review before ingestion is completed.
Multimodal Knowledge Graph Construction
We organize the extracted document elements in a multimodal knowledge graph . The node set represents the document hierarchy, textual chunks, figures, tables, and traffic control devices:
A single canonical node is created for each figure identifier and stores the images associated with all of its sheets. The edge-type set is
The cites relation includes chunk-to-figure, chunk-to-table, and chunk-to-section references extracted from the chunk text.
Figures are associated with sections using three complementary forms of evidence. Let denote the set of sections whose chunks cite figure , and let denote the chapter encoded in the identifier of . The primary anchor section is defined as
where \lvert \operatorname{cites}(s,f) \rvert is the number of citations to figure within section . If no citing section belongs to the chapter encoded in the figure identifier, the system falls back to the section with the largest number of citations to that figure. Each figure is linked to its primary section, referencing sections, parent chapter, and depicted traffic control devices or concepts. These relations preserve the document hierarchy and each visual element’s semantic role in the knowledge graph.
Multimodal Indexing
Textual and visual evidence are encoded using modality-specific representation models. For each textual chunk , we compute a dense embedding and a learned sparse representation:
where both representations are produced using BGE-M3.
For a figure or table crop , we compute a token-level multi-vector visual representation:
where denotes the number of visual tokens and denotes the embedding dimension. We use ColQwen2 ([11]) to encode both individual visual crops and full-page renderings. The resulting representations are stored in vector indexes, while document metadata and cross-modal relations remain in the knowledge graph.
Graph-Grounded Hybrid Retrieval
For textual retrieval, dense and sparse rankings are combined using reciprocal rank fusion:
where is a rank-smoothing constant. The dense ranking is induced by the cosine similarity cores , where is the dense representation of query .
Explicit figure, table, section, and device identifiers in the query are matched to a set of graph nodes . Each retrieved candidate then receives a graph-proximity score
where denotes the shortest-path distance in .
Normative Rule Modeling
The MUTCD expresses provisions at four levels of normative authority. We assign each textual chunk a rule type
and define a rule-type weight satisfying
We also introduce a hierarchical relevance prior
which rewards candidates whose part, chapter, or section is explicitly referenced in the query. The fused first-stage retrieval score is
where are fusion coefficients. This formulation prioritizes mandatory provisions over advisory or explanatory text when the candidates are otherwise similarly relevant.
A cross-encoder subsequently re-ranks the highest-scoring candidates:
The resulting order determines the final textual evidence set, denoted by .
Query-Adaptive Visual Evidence Gating
Attaching figures to answers that do not require visual evidence can reduce precision and increase unnecessary context. GEMS-RAG therefore employs a query-adaptive router that determines whether visual evidence should enter the generation stage:
where activates visual retrieval and suppresses all visual inputs.
Deterministic rules are evaluated first. An explicit figure or table identifier forces , as does a visual expression such as “show,” “appearance,” or “look like.” In contrast, a definitional or permission-oriented question without visual cues forces . Queries not resolved by the deterministic rules receive a visual-need score
where represents lexical signals associated with device codes, visual nouns, and placement verbs. The set contains the sections associated with the highest-ranked textual chunks. The second term measures the proportion of those sections that cite at least one figure.
The router activates visual retrieval when
where is a decision threshold. The system records the rules and signals responsible for every routing decision. When , neither figure crops nor page images are passed to the generator, and no images are included in the resulting answer.
Visual Evidence Retrieval and Filtering
When , visual candidates are obtained through two complementary retrieval paths.
Graph-linked retrieval. The first path collects figures and tables cited by the highest-ranked textual chunks. These candidates are ordered according to the rank of their citing chunks.
Direct visual retrieval. The second path directly searches the visual crop index using late interaction. For a visual candidate , its relevance score is
where is the embedding of the -th query token and is the -th visual token representation of candidate .
The candidates returned by the two paths are merged and evaluated by a vision-language filtering model:
where is the maximum number of retained visual entities. The filter is instructed to retain only visual evidence that a reader must inspect to answer the query, with a conservative preference for excluding marginally relevant images. The filtering model may be smaller than the final answer-generation model. When a retained figure contains multiple sheets, all sheets associated with its canonical graph node are included.
Multimodal Answer Generation
The final evidence set is
where denotes the textual evidence, denotes the filtered figure and table crops, and denotes the retrieved full-page renderings.
The multimodal generator receives the query, the textual evidence with section identifiers and rule types, and the gated visual evidence:
Each textual chunk retains its section identifier and normative rule type, while each visual entity retains its canonical identifier and anchor section. After generation, every cited section, figure, and table is verified through an exact lookup in . Citations that cannot be resolved to valid graph entities are rejected.
Retrieval Procedure
For a query , GEMS-RAG extracts figure, table, section, and traffic-control-device identifiers and maps them to knowledge-graph nodes. It retrieves text using dense and learned sparse search, fuses rankings through reciprocal rank fusion, and scores candidates using graph proximity, rule-type weights, and hierarchy priors in Eq. (14). The cross-encoder in Eq. (15) then produces the textual evidence set . The visual-evidence router determines and, when , retrieves graph-linked and directly matched figures and tables for vision-language filtering. Finally, the framework assembles textual evidence, visual crops, captions, and page renderings into the generation context and produces an answer with graph-verified section, figure, table, and rule-type citations.
Experimental Evaluation
This section evaluates GEMS-RAG on retrieval and question answering using the 11th Edition of the MUTCD. We assess its ability to retrieve sufficient textual and visual evidence, support grounded standards reasoning, and remain robust across evidence modalities and difficulty levels. Table 1 summarizes the resulting knowledge base, while Figure 2 presents an exemplar subgraph centered on the STOP sign.

Figure 2. Knowledge graph for the R1-1 STOP sign example.
Evaluation Tasks and Results
We constructed MUTCD-QA-150 as a benchmark of 150 questions spanning textual, tabular, visual, and mixed-evidence reasoning. For each question, we manually locked the gold answer and the required MUTCD sections, source pages, tables, figures, and visual crops; 85 questions specifically required figure or table evidence. Using these fixed annotations and a common retrieval stack, we evaluated 12 vision-language models (OpenAI 2026; Alibaba Cloud 2026c; Google 2026c,a,b; Anthropic 2025, 2026a,b; Alibaba Cloud 2026e,d,a,b on evidence retrieval, visual filtering, and question answering. Retrieval performance was consistent across models (19.35–19.63/25), with Recall@5 of 86.7%–88.0% and evidence sufficiency of 93.3%. Larger differences emerged in model-controlled visual filtering, where exact visual-crop hit rates ranged from 75.3% to 83.5% and crop precision ranged from 62.7% to 79.8%, given in Figure 3. The shared retrieval stack produced stable evidence coverage across models, while the selected vision-language model primarily affected visual-crop selection. Context precision remained below 50%, indicating that irrelevant evidence was still frequently passed to the answer-generation stage.
Figure 3. Retrieval and visual-evidence performance.
For answer evaluation, GPT-5.6 ([26]) served as a constrained, evidence-grounded LLM judge. Judgments with confidence above 50% were accepted directly; lower-confidence cases were manually verified. It compared each model response with the locked gold answer and MUTCD evidence and applied a predefined rubric covering correctness, completeness, faithfulness, citation localization, and visual or table reasoning. Strict full credit required perfect correctness, completeness, faithfulness, and citation scores. As shown in Figure 4, overall normalized scores ranged from 85.08 to 88.08, with Claude Fable 5 achieving the highest provisional score and an 80.0% full-credit rate. GPT-5.6 supported benchmark construction, evidence review, and rubric-based scoring, whereas retrieval metrics and final composite scores were calculated using fixed rule-based scripts.
Figure 4. Question-answering performance.
Based on the cross-model comparison, we selected Claude Fable 5 for detailed analysis because it achieved the highest provisional overall and strict full-credit scores while balancing correctness, faithfulness, citation localization, and visual reasoning. We then assessed its consistency across evidence modalities, answerability, and M-SDI difficulty levels, defined by localization, modality integration, reasoning, answer composition, and normative precision. As shown in Figure 5, the model demonstrated strong performance across all evidence types. It achieved full credit on 90.0% of tabular and 83.3% of figure-based questions. Mixed-evidence questions yielded 91.2% correctness, 89.5% completeness, and a 76.7% full-credit rate, while purely textual questions achieved high correctness (93.5%) and faithfulness (99.3%) alongside a 75.0% full-credit rate. Notably, most point deductions stemmed from omitted definitions, exceptions, quantities, thresholds, or citations, rather than fundamentally incorrect conclusions. Performance also remained stable across M-SDI difficulty levels. For answerable questions (those fully supported by MUTCD evidence), correctness reached 88.2% on easy, 92.3% on medium, and 89.2% on hard queries. Furthermore, the model reliably handled unanswerable queries (those lacking sufficient, or in-scope support), maintaining near-perfect performance across all difficulty tiers.

Figure 5. Claude Fable 5 performance by evidence modality and by M-SDI difficulty and answerability.
Performance Comparisons
To evaluate GEMS-RAG, we benchmarked it against three established baselines (i.e., BM25 (Robertson and Zaragoza [29]), GraphRAG (Procko and Ochoa [28]), and PaperQA2 (Skarlinski et al. [31])) across three language models. We first assessed retrieval by comparing the top 10 figures and evidence items returned by each method against GPT-5.6-Pro generated gold annotations. As shown in Figure 6, GEMS-RAG achieves the highest Recall@10, MRR@10, and nDCG@10, whereas GraphRAG substantially underperforms BM25. As shown in Figure 7, GEMS-RAG improves the overall score by more than 10% across all models, with larger gains in correctness and full credit. Under the strict full-credit criterion, it exceeds the strongest baseline by approximately 48%–50% points, demonstrating its ability to generate correct, complete, and well-grounded answers.

Figure 7. Question answering performance

Figure 6. Retrieval performance comparison
Ablation Study
We assessed the contribution of three GEMS-RAG components through controlled ablations using the full Claude Fable 5 configuration as the reference.
As shown in Table 2, removing the VLM figure filter or hierarchy prior slightly increased retrieval scores but reduced generation, reliability, and overall performance. Removing the reranker caused the largest degradation, lowering the overall score by 4.65 points, which highlights its central role in selecting answer-relevant evidence.
| Condition | Retr. (/25) | Gen. (/60) | Reliab. (/12) | Overall (/100) |
| GEMS-RAG | 19.35 | 55.18 | 10.90 | 88.08 |
| Without VLM filter | 19.59 (+0.24) | 55.12 (-0.06) | 10.70 (-0.20) | 88.06 (-0.02) |
| Without hierarchy prior | 19.59 (+0.24) | 55.10 (-0.08) | 10.54 (-0.36) | 87.86 (-0.22) |
| Without reranker | 17.78 (-1.57) | 52.77 (-2.41) | 10.38 (-0.52) | 83.43 (-4.65) |
Table 2. Ablation results. Best results are bolded and second-best results are underlined. Unit: percent.
Conclusion
This paper presented GEMS-RAG, a graph-grounded multimodal RAG framework for engineering standards. Unlike conventional pipelines that treat standards as independent text chunks, GEMS-RAG preserves document hierarchy, decomposes figures into fine-grained visual entities, and links provisions, figures, devices, rule types, and cross-references in a multimodal knowledge graph. By combining dense, sparse, and visual retrieval with graph expansion and rule-aware ranking, it enables standards-grounded question answering sensitive to visual specifications and regulatory structure.
Using the MUTCD as a representative engineering standards corpus, we constructed a multimodal knowledge base containing textual sections, figures, tables, visual entities, and graph relations. The experimental evaluation demonstrates that explicitly modeling multimodal and structural relationships can improve evidence retrieval, visual grounding, and downstream question answering compared with text-only, multimodal, and graph-based RAG baselines. The case study also illustrates that GEMS-RAG can retrieve not only the relevant textual provisions, but also associated visual entities, rule categories, and graph-connected context, leading to more complete and interpretable responses. Beyond the MUTCD, further work will apply GEMS-RAG to other engineering codes, design manuals and scientific standards.
Appendix
Evaluation Metrics
Let denote the evaluation set. For each question , let denote the locked gold evidence and let denote the ordered list of the top- retrieved text chunks. A chunk is relevant when its page, section, table reference, or figure reference matches the locked evidence for the question. We define when is relevant and otherwise.
Retrieval Metrics
Recall at 5: The main model and ablation evaluations use a question-level retrieval hit. A question receives credit when at least one relevant chunk appears within the first five ranks:
Mean reciprocal rank: Let
be the rank of the first relevant item. The reciprocal rank is zero when no relevant item appears among the top- results. Thus,
Normalized discounted cumulative gain: The discounted cumulative gain for question is
Let denote the number of relevant chunks recovered in the logged top- list. The ideal DCG places those recovered relevant chunks first:
The normalized score is
Context precision: Context precision measures the fraction of the logged text context that matches the locked evidence:
Evidence sufficiency: Let denote the top four retrieved page images and let denote the locked source pages. Let denote the locked visual entities and let denote the visual entities supplied to the answer model. The saved retrieval package is sufficient when it contains at least one relevant text chunk, one required page among the top four, or one exact required visual entity:
The main model and ablation evaluations report Recall@5, MRR over the six logged text ranks, nDCG@6, and context precision@6. The separate RAG baseline comparison reports Recall@10, MRR@10, and nDCG@10 using the same binary relevance definition.
Visual-Evidence Metrics
Let denote the questions with at least one locked figure or table identifier. For each , let be the set of locked visual entities and let be the unique visual entities supplied to the answer model.
Exact visual-crop hit rate: A hit occurs when at least one retained entity exactly matches a locked visual entity:
Visual-crop recall: Visual-crop recall measures the fraction of required visual entities recovered for each visual question and then averages across the visual subset:
Visual-crop precision: Visual-crop precision is calculated for each visual question and then averaged. A question with no retained visual entity receives zero precision:
Required-page hit rate: Let be the first page images and let be the locked source pages. For , the required-page hit rate is
Modality-routing accuracy: Let when locked visual evidence is required and otherwise. Given the router output , routing accuracy is
The visual metrics are reported as diagnostics. Exact visual hit and visual precision do not receive separate points in the 25-point retrieval score.
Question-Answering Metrics
For answer , GPT-5.6 assigns normalized scores for correctness, completeness, faithfulness, and citation localization. Let
denote the mean score for dimension . Reported percentages equal .
Visual reasoning is calculated as mean correctness over questions whose primary modality is table, figure, or mixed:
The relevance and concision term is held at for the completed model comparison. The 60-point generation score is
Strict full credit: An answer receives full credit only when correctness, completeness, faithfulness, and citation localization all equal 1:
Scores by modality, answerability, or difficulty are calculated using the same equations after restricting to the corresponding question subset.
Reliability Metrics
Let denote the 30 questions whose locked annotations mark them as unanswerable. Let equal 1 when answer receives strict full credit and 0 otherwise. Unanswerable success is
Let denote the figure-router confidence recorded for question . The Brier score is
For ten equal-width confidence bins , expected calibration error is
The calibration contribution and the measurable 12-point reliability score are
and
The confidence value comes from the visual router rather than the final answer model or GPT-5.6. The no-router ablation has no comparable confidence value, so its Brier score, ECE, calibration points, 12-point reliability score, and 97-point full score are not defined.
Composite Scores
The 25-point retrieval score is
Repeatability contributes up to 3 points in the intended 100-point framework. That component remains pending. The measured score therefore uses 97 available points:
The reported provisional score normalizes the measured total to 100:
The intended final score is
where is the pending repeatability component. Latency is reported separately through mean, median, and 90th-percentile response time. It does not contribute to the composite score.
For the no-router ablation, the common score excludes calibration and repeatability and uses a 91-point denominator:
MUTCD Structural Difficulty Index
For each question , the MUTCD Structural Difficulty Index (M-SDI) evaluates five dimensions: evidence localization, modality integration, reasoning operations, answer composition, and normative precision. Let denote the score for dimension . The total difficulty score is
Questions are categorized as
Improvement over Baselines
For GEMS-RAG score and baseline score , the relative improvement is
For percentage-valued metrics such as full-credit rate, the percentage-point improvement over the strongest baseline is
For the ablation study, we drew 800 paired bootstrap samples of the 150 question identifiers. Each sample used the same resampled questions for the original RAG and the ablated condition. We recomputed the score difference for each sample and reported the 2.5th and 97.5th percentiles. These intervals measure uncertainty from the benchmark question sample. They do not measure variation across repeated model calls.
Reproducibility Assets
The benchmark assets were frozen before model execution. The runtime question file contained only the question identifier and question text. A separate locked gold file contained the gold answer, required sections, source pages, figures, tables, answerability labels, and M-SDI annotations. The answering models did not load the gold file. This separation prevented answer leakage and kept the generation runs independent from the evaluation annotations.
The benchmark contained 150 questions, with 120 answerable and 30 unanswerable questions. The primary-modality distribution was 60 text, 30 table, 30 figure, and 30 mixed questions. The development split contained 30 questions and the test split contained 120 questions. The development split supported prompt and retrieval configuration. The test split remained fixed for final reporting.
We retained the source document, question-only benchmark, locked annotations, model registry, runtime manifests, answer files, retrieval files, error logs, item-level scores, ablation outputs, and per-response rationale records. Each model run wrote separate JSONL files for answers and retrieval traces together with a manifest that recorded the model identifier, run identifier, runner version, and execution settings. These files provide the audit trail used to reproduce the reported tables and figures.
Version control recorded the implementation state used for the experiments. The updated ablation archives identify harness version ablations_v3 and Git commit ba26b07031398fb25efca8ba793a42c6ebb4ae03. The main benchmark records also identify runner version 1.1.2 and preserve the exact provider model identifiers. Recording both the run manifest and source revision distinguishes changes in model behavior from changes in the evaluation code.
Implementation Hyperparameters
The values in Tables 1 and 2 come from the saved configuration, run manifests, provider adapter, and ablation metadata. Parameters that the code did not set explicitly are identified as provider defaults.
Retry, Resume, and Failure Handling
The runner wrote each completed record to disk before moving to the next question. Resume mode skipped valid records that already existed and continued from missing or failed question identifiers. The runner allowed at most five provider attempts and placed a 2.0-second interval between requests. For Claude, the provider adapter used a 16,000-token output cap and performed one truncation-specific retry with twice the limit, capped at 32,000 tokens.
We reran records only for operational failures, missing records, corrupted outputs, provider errors, or materially incomplete retrieval logging. We did not rerun an answer because it was factually incorrect or received a low score. When a retry succeeded, the evaluator retained the first valid response for the question and preserved the original failure record for audit. This rule avoided selecting a more favorable answer after observing model quality.
Role of GPT-5.6 and the Evaluated Language Models
The authors used GPT-5.6 to review benchmark-runner code, provider adapters, runtime validation checks, scoring scripts, plotting code, and technical documentation. The authors executed the code, inspected saved records, selected the final settings, and resolved failures before accepting an output. Model-generated code or prose did not enter the reported results without execution and record-level verification.
During evaluation, GPT-5.6 supported benchmark construction, evidence review, and application of the locked answer-quality rubric. It did not generate the retrieval ranks, latency values, calibration statistics, or composite scores. Fixed scripts calculated those quantities from the saved records. The evaluated vision-language models received only the user question and retrieved evidence. They did not receive the locked gold answer file.
The cross-model study used twelve vision-language configurations under the same benchmark and scoring framework. The selected model also controlled visual filtering when the filter-model override was null. Claude Fable 5 produced the answers for the detailed modality, difficulty, and ablation analyses after it achieved the highest provisional score in the cross-model comparison. The ablation study kept Claude Fable 5 and the few-shot prompt fixed while removing one retrieval component at a time.
Answer-Rationale Audit and Evaluator Confidence
We created one grading rationale for each of the 900 ablation responses and each of the 150 original-RAG baseline responses. Each record retained the four official item scores, the strict full-credit flag, the model answer, the locked gold answer, the retrieved evidence metrics, and the rationale provenance. The rationale package distinguishes an explicit adjudication note from a structured reconstruction of an existing grade.
| Setting | Value |
| Text representation | BAAI/bge-m3, used for dense and learned sparse representations |
| Visual representation | vidore/colqwen2-v0.1 |
| Cross-encoder reranker | mixedbread-ai/mxbai-rerank-large-v2 |
| Page and figure rendering | 180 dpi for pages, 220 dpi for figure and table crops |
| Text retrieval depths | 30 dense, 30 sparse, 30 fused, 40 after graph expansion, 6 after reranking |
| Visual retrieval depths | 10 candidate entities, 6 direct visual candidates, at most 4 retained entities, 4 page images |
| Caption fallback | Disabled |
| Question router | Enabled, soft threshold 0.5, VLM tie break disabled |
| VLM figure filter | Enabled. For the final Claude and Gemini runs, the filter followed the selected answer model because the filter override was null |
| Fusion weights | Dense 1.00, sparse 0.60, hierarchy 0.20, graph proximity 0.40, rule type 0.30 |
| Rule-type multipliers | Standard 1.20, Guidance 1.00, Option 0.90, Support 0.70 |
| ColQwen2 image settings | Maximum 768 image patches, binary quantization enabled |
| Maximum text from each chunk | 1400 characters in the answer prompt |
Table 1 (PDF p. 13). Core indexing and retrieval hyperparameters.
| Setting | Value |
| Detailed-analysis and ablation model | Claude Fable 5, alias frontier_claude |
| Answer prompt | Few-shot, 3 examples |
| Visual-filter prompt | Zero-shot |
| Claude output-token cap | 16,000 tokens, including adaptive-thinking tokens and visible answer tokens |
| Truncation handling | One retry with a multiplier of 2.0, capped at 32,000 tokens for Claude |
| Temperature and top-p | Not set explicitly in the adapter, so the provider defaults applied |
| Benchmark runner | Version 1.1.2 |
| Random seed and replicates | Seed 20260710, 1 replicate per model configuration |
| Retry policy | Maximum 5 attempts, 2.0 seconds between requests, resume enabled |
| Ablation controls | Claude Fable 5 and the few-shot prompt remained fixed. Each run removed one component |
| Paired bootstrap | 800 paired resamples of the 150 question identifiers |
Table 2 (PDF p. 13). Generation, benchmark-runner, and evaluation hyperparameters.
Programmatic Consistency Checks
Table 3 summarizes checks performed over all 1,050 records. These checks validate the internal representation of the official grades. They do not establish that every underlying item grade is correct against the MUTCD.
The token-overlap field is diagnostic rather than authoritative. It listed a missing or underdeveloped element in 216 full-credit records, primarily when an unanswerable response used a refusal phrasing that did not lexically match the gold wording. It listed no missing element in 173 partial-credit records, where the deduction instead came from an adjudication note, faithfulness, or citation localization. The official four scores and the adjudication notes remain the source of truth.
A direct recheck found 20 records with citation score 1.00 and an empty regex-detected citation field. All 20 were unanswerable responses involving a false premise, unsupported precision, or out-of-scope request. The accepted citation-localization score reflected correct handling of the unsupported request, while the citation detector searched only for explicit section, figure, table, page, or legal-reference patterns. We therefore treat these records as a detector limitation rather than a contradiction in the official grade.
Evaluator Confidence
GPT-5.6 gave a strict self-assessment of 82% confidence in the complete ablation evaluation. This value reflected high confidence in deterministic runtime and retrieval calculations and lower confidence in rubric-based item judgments. The value is a judgmental assessment, not a model probability and not a sampling-based confidence interval.
The authors then supplied the 1,050 rationale records to Claude Fable 5 for a secondary model audit. Table 5 checked score arithmetic, aggregate agreement, provenance rules, repaired-record allocation, and a small set of MUTCD content examples. It assigned 85% confidence with a reported evaluator-confidence interval of [78%, 90%]. Table 5 placed the mechanical consistency of the package near the top of its scale because the score and aggregate checks matched. It reduced its overall assessment because most rationales reconstruct or restate existing grades and therefore do not independently validate the substantive MUTCD judgment.
The reported evaluator-confidence summary is
This interval is a judgmental interval supplied by a secondary language-model auditor. It is not a frequentist confidence interval, a Bayesian credible interval, or the paired bootstrap interval used for ablation score differences. Claude Fable 5 also generated the evaluated answers, so this secondary audit does not replace an independent MUTCD-informed human annotator.
| Check | Verified result |
| Strict full-credit flag compared with all four dimensions equal to 1.00 | 0 mismatches in 1,050 records |
| Descriptive mean compared with the mean of correctness, completeness, faithfulness, and citation | 0 mismatches in 1,050 records |
| Per-condition correctness, completeness, faithfulness, citation, and full-credit aggregates compared with the version 4 report | Exact match for all seven conditions |
| Strict full-credit counts | 120, 114, 112, 114, 112, 109, and 103 for the baseline and A1 through A6 |
| Rationale provenance | 895 structured reconstructions, 48 explicit adjudication notes, and 107 reused baseline notes |
| Reused-note criterion | 0 violations. Each reused note had the same question identifier and identical four official scores as the baseline |
| Explicit-note allocation | 30 original-RAG partial-credit records and 18 repaired A1 records |
| Repaired A1 records | Exactly T001 through T018 |
Table 3. Programmatic checks of the per-response rationale package.
The rationale package assigns fixed confidence values by provenance tier: 0.90 for explicit adjudication notes, 0.88 for reconstructed full-credit records, 0.82 for the 18 repaired A1 responses reviewed by GPT-5.6, and 0.76 for reconstructed partial-credit rationales. These are design constants used to describe confidence in the explanation. They are not empirical probabilities. A second MUTCD-informed annotator should review a stratified sample, report agreement, and adjudicate disagreements before the item-level judgments are treated as independently validated.
Sample MUTCD-150 Questions and Locked Gold Answers
This section provides 10 representative examples for each reporting category: answerable, unanswerable, easy, medium, and hard. The five displayed sets do not repeat question identifiers. Answerability and M-SDI difficulty remain independent benchmark labels, so each entry retains both labels. We selected examples to cover text, table, figure, and mixed evidence where the category permits. The question wording, locked gold answers, evidence identifiers, answerability labels, and M-SDI scores below are copied from the locked gold file. The answering models did not receive this file during execution.
Answerable Questions
Answerable Questions: 10 Locked Examples
T002 Text, Easy, Answerable, M-SDI 1/10
Question: Does the applicability of the MUTCD to a facility open to public travel depend on whether the facility is publicly or privately owned or on the source of its funding?
Locked Gold Answer: No. Applicability is independent of public or private ownership or jurisdiction and independent of whether funding is Federal, State, local, or private.
Locked Evidence: Section 1A.01, PDF page 42.
T005 Text, Easy, Answerable, M-SDI 3/10
Question: How do the MUTCD headings Standard, Guidance, Option, and Support differ, including the verbs typically associated with them?
Locked Gold Answer: A Standard is mandatory or prohibitive and typically uses ’shall.’ Guidance is recommended practice with justified deviations allowed and typically uses ’should.’ An Option is permissive and typically uses ’may.’ Support is informational and conveys no mandate, recommendation, permission, prohibition, or enforceable condition.
Locked Evidence: Section 1C.01, PDF page 54.
T025 Text, Medium, Answerable, M-SDI 5/10
Question: What must a Speed Limit sign display, in what increments, and may it be used to show an advisory speed for a roadway condition?
Locked Gold Answer: It must display the speed limit established by law, ordinance, regulation, or an authorized agency based on an engineering study, in multiples of 5 mph. It should not be used for an advisory speed; an advisory speed plaque below an appropriate warning sign should be used instead.
Locked Evidence: Section 2B.21, PDF pages 126, 127.
TB001 Table, Easy, Answerable, M-SDI 3/10
Question: According to Table 1B-1, what is the compliance date for implementing and continuing to use a method designed to maintain longitudinal pavement-marking retroreflectivity?
Locked Gold Answer: September 6, 2026.
Locked Evidence: Section 1B.03, Table 1B-1, PDF page 46.
TB002 Table, Medium, Answerable, M-SDI 4/10
Question: In Table 1D-1, what standard abbreviations are permitted for ’Pedestrian(s)’?
Locked Gold Answer: PED or PEDS.
Locked Evidence: Section 1D.08, Table 1D-1, PDF page 76.
TB024 Table, Medium, Answerable, M-SDI 5/10
Question: What minimum off-roadway sizes does Table 9A-1 give for a STOP sign, a YIELD sign, and a Crossbuck sign on a bicycle facility?
Locked Gold Answer: STOP: 18 x 18 inches; YIELD: 18 x 18 x 18 inches; Crossbuck: 24 x 4.5 inches.
Locked Evidence: Section 9A.03, Table 9A-1, PDF page 1090.
F008 Figure, Medium, Answerable, M-SDI 5/10
Question: From Figure 2B-1, identify the shapes of the STOP and YIELD signs and the color scheme of the ALL-WAY plaque.
Locked Gold Answer: STOP is an octagon; YIELD is a downward-pointing equilateral triangle; the ALL-WAY plaque has white legend and border on a red background.
Locked Evidence: Sections 2B.04, 2B.05, Figure 2B-1, PDF page 115.
F012 Figure, Medium, Answerable, M-SDI 5/10
Question: What four crosswalk marking patterns are illustrated in Figure 3C-1?
Locked Gold Answer: Transverse, longitudinal bar, ladder, and bar pair.
Locked Evidence: Section 3C.03, Figure 3C-1, PDF page 631.
M001 Mixed, Hard, Answerable, M-SDI 7/10
Question: A provision appears under a Guidance heading and uses the word ’should.’ Is it mandatory, and when may a deviation be appropriate?
Locked Gold Answer: It is recommended practice, not a mandatory Standard. A deviation may be appropriate when engineering judgment or an engineering study indicates that the deviation is suitable for the specific situation.
Locked Evidence: Sections 1A.04, 1C.01, PDF pages 43, 54.
M019 Mixed, Hard, Answerable, M-SDI 9/10
Question: For a 12-foot lateral offset and a 35-mph speed, calculate using Table 6B-4. Then state the minimum merging, shifting, and shoulder taper lengths from Table 6B-3.
Locked Gold Answer: For 40 mph or less, feet. Minimum merging taper: 245 feet; shifting taper: 122.5 feet; shoulder taper: about 80.9 feet (0.33L, commonly rounded appropriately in design).
Locked Evidence: Section 6B.08, Tables 6B-3, 6B-4, PDF page 816.
Unanswerable Questions
Unanswerable Questions: 10 Locked Examples
T049 Text, Medium, Unanswerable, M-SDI 5/10, Unsupported Precision
Question: According to the MUTCD, what exact percentage reduction in crashes is guaranteed by installing a STOP sign or all-way stop control?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 2B.04, 2B.12-2B.17, PDF pages 115, 120.
T052 Text, Medium, Unanswerable, M-SDI 4/10, False Presupposition
Question: What single mandatory signal-cycle length must be used at every urban signalized intersection in the United States?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 4A.01-4A.06, 4B.01-4B.05, PDF pages 684, 691.
T054 Text, Medium, Unanswerable, M-SDI 5/10, Out Of Scope
Question: How many licensed traffic engineers must a road agency employ per 100 centerline miles to comply with the MUTCD?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 1A.04, 1D.03, PDF pages 43, 73.
T059 Text, Easy, Unanswerable, M-SDI 3/10, Contradicted Premise
Question: What minimum miles-per-gallon rating must a vehicle achieve to qualify as an inherently low-emission vehicle under the MUTCD?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Section 1C.02, PDF pages 60, 62.
TB027 Table, Easy, Unanswerable, M-SDI 3/10, Unsupported Precision
Question: What RGB values are listed in Table 2A-2 for fluorescent yellow-green?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Section 2A.06, Table 2A-2, PDF page 85.
TB029 Table, Medium, Unanswerable, M-SDI 6/10, Out Of Scope
Question: Which table lists State-by-State monetary fines for violating a No Turn on Red sign?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Section 2B.60, Table 2B-1, PDF page 113.
F026 Figure, Easy, Unanswerable, M-SDI 3/10, Absent Information
Question: What foundation depth is required for the sign posts drawn in Figure 2A-2?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 2A.13, 2A.18, Figure 2A-2, PDF page 92.
F029 Figure, Easy, Unanswerable, M-SDI 3/10, Unsupported Precision
Question: What exact crossing angle, in degrees, is shown for the railroad tracks in Figure 8E-1?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 8E.01, 8E.02, Figure 8E-1, PDF page 1076.
M028 Mixed, Medium, Unanswerable, M-SDI 6/10, Absent Information
Question: What were the sample size, participant demographics, and statistical significance values of the human-factors study underlying each treatment in Figure 2A-1?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Section 2A.11, Figure 2A-1, PDF pages 89, 90.
M030 Mixed, Hard, Unanswerable, M-SDI 7/10, Out Of Scope
Question: What exact dollar penalty and criminal classification apply when an agency violates a MUTCD Standard?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 1B.01, 1D.02, PDF pages 45, 72.
Easy Questions
Easy Questions: 10 Locked Examples
T001 Text, Easy, Answerable, M-SDI 2/10
Question: What is the purpose of the MUTCD, and what four objectives are used to achieve that purpose?
Locked Gold Answer: The MUTCD establishes uniform national criteria for traffic control devices on streets, highways, pedestrian and bicycle facilities, and site roadways open to public travel. Its objectives are to promote safety, inclusion, and mobility; promote efficiency through national uniformity in device meaning and appearance; promote consistency in use, installation, and operation; and provide basic decision-making principles for traffic engineers.
Locked Evidence: Section 1A.01, PDF page 42.
T003 Text, Easy, Answerable, M-SDI 1/10
Question: Are islands, curbs, speed humps, raised roadway surfaces, and rumble strips considered traffic control devices under the MUTCD?
Locked Gold Answer: No. Infrastructure elements that physically restrict travel paths or speeds, including islands, curbs, speed humps, other raised surfaces, and transverse or longitudinal rumble strips, are not traffic control devices.
Locked Evidence: Section 1A.02, PDF page 42.
T014 Text, Easy, Answerable, M-SDI 2/10
Question: What general meanings does the MUTCD assign to fluorescent pink, orange, and purple?
Locked Gold Answer: Fluorescent pink is for incident management, orange is for temporary traffic control, and purple is restricted to vehicles with registered electronic toll collection accounts.
Locked Evidence: Section 1D.05, PDF page 74.
T019 Text, Easy, Answerable, M-SDI 2/10
Question: What are the minimum mounting heights for roadside signs in rural areas and in business, commercial, or residential areas where parking, bicyclist, or pedestrian activity is likely?
Locked Gold Answer: The minimum is 5 feet in rural areas and 7 feet in business, commercial, or residential areas where such activity or obstruction is likely, measured as specified from the near pavement edge or curb.
Locked Evidence: Section 2A.15, PDF page 99.
TB011 Table, Easy, Answerable, M-SDI 3/10
Question: At a posted or 85th-percentile speed of 60 mph, what advance placement distance does Table 2C-3 show for deceleration to an advisory speed of 30 mph?
Locked Gold Answer: 200 feet.
Locked Evidence: Section 2C.04, Table 2C-3, PDF page 193.
TB014 Table, Easy, Answerable, M-SDI 3/10
Question: What minimum passing sight distance does Table 3B-1 specify for a speed of 55 mph?
Locked Gold Answer: 900 feet.
Locked Evidence: Section 3B.03, Table 3B-1, PDF page 584.
TB019 Table, Easy, Answerable, M-SDI 3/10
Question: What minimum sight distance for signal visibility does Table 4D-2 list at an 85th-percentile speed of 45 mph?
Locked Gold Answer: 460 feet.
Locked Evidence: Section 4D.06, Table 4D-2, PDF page 711.
F001 Figure, Easy, Answerable, M-SDI 3/10
Question: In Figure 1B-1, what happens immediately after FHWA approves a request to experiment?
Locked Gold Answer: The requesting jurisdiction installs the experimental traffic control device.
Locked Evidence: Section 1B.05, Figure 1B-1, PDF page 48.
F009 Figure, Easy, Answerable, M-SDI 3/10
Question: Which sign is placed above the other in every paired assembly shown in Figure 2B-17?
Locked Gold Answer: The DO NOT ENTER sign is above the WRONG WAY sign.
Locked Evidence: Section 2B.48, Figure 2B-17, PDF page 156.
F024 Figure, Easy, Answerable, M-SDI 3/10
Question: In Figure 9E-12, what minimum distance separates the advance motor-vehicle stop line from the intersection stop line across the bicycle box?
Locked Gold Answer: 10 feet minimum.
Locked Evidence: Section 9E.12, Figure 9E-12, PDF page 1147.
Medium Questions
Medium Questions: 10 Locked Examples
T009 Text, Medium, Answerable, M-SDI 4/10
Question: When a non-compliant traffic control device is replaced because it is damaged, missing, or no longer serviceable, what is the general rule and the limited replacement-in-kind exception?
Locked Gold Answer: The replacement must generally be a compliant device. A non-compliant device may be replaced in kind only when engineering judgment finds that a lone compliant device among adjacent non-compliant devices would confuse users and/or the planned replacement schedule for the whole series will achieve timely compliance.
Locked Evidence: Section 1B.03, PDF page 47.
T012 Text, Medium, Answerable, M-SDI 5/10
Question: What public-domain condition applies to a new traffic control device proposed for experimentation or interim approval?
Locked Gold Answer: The traffic-control message, design, or application must be in the public domain and not protected by patent, trademark, or copyright. Legally binding statements are required from the relevant agency/developer/supplier for experimentation. Protection of internal components or manufacturing methods can still be allowed.
Locked Evidence: Sections 1B.05, 1D.06, PDF pages 50, 74, 75.
T035 Text, Medium, Answerable, M-SDI 5/10
Question: Does satisfying a traffic signal warrant automatically require installation of a traffic control signal?
Locked Gold Answer: No. A warrant is a threshold that triggers analysis; it is not conclusive justification. Installation must be based on an engineering study considering safety, operation, alternatives, and the site. A signal can also be installed where no warrant is met if an engineering study documents why it is the best solution.
Locked Evidence: Sections 4B.04, 4C.01, PDF pages 691, 694.
T046 Text, Medium, Answerable, M-SDI 4/10
Question: When two-way traffic must share one lane for a limited distance, what coordination rule and control methods apply?
Locked Gold Answer: Movements from both ends must be coordinated, except for a short, low-volume, mutually visible work zone that can self-regulate. Methods include flaggers, flag transfer, a pilot car, temporary traffic signals, or stop/yield control.
Locked Evidence: Section 6E.01, PDF page 828.
TB003 Table, Medium, Answerable, M-SDI 4/10
Question: In Table 1D-2, how is ’Road Work’ abbreviated on a portable changeable message sign, and what may follow it?
Locked Gold Answer: RD WK; it may be followed by AHEAD or a distance, such as ’RD WK 1 MILE.’
Locked Evidence: Section 1D.08, Table 1D-2, PDF page 77.
TB007 Table, Medium, Answerable, M-SDI 5/10
Question: For white-on-red signs, what minimum maintained retroreflectivity values and contrast criterion are shown in Table 2A-5?
Locked Gold Answer: White at least 35 cd/lx/m², red at least 7 cd/lx/m², and a white-to-red contrast ratio of at least 3:1.
Locked Evidence: Section 2A.22, Table 2A-5, PDF page 104.
TB018 Table, Medium, Answerable, M-SDI 4/10
Question: For a four-leg intersection, what one-year crash thresholds are listed in Table 4C-2 for all-severity angle/pedestrian crashes and for fatal-and-injury angle/pedestrian crashes?
Locked Gold Answer: Five all-severity angle and pedestrian crashes, or three fatal-and-injury angle and pedestrian crashes.
Locked Evidence: Section 4C.08, Table 4C-2, PDF page 702.
F003 Figure, Medium, Answerable, M-SDI 4/10
Question: What conspicuity treatment is labeled ’E’ in Figure 2A-1?
Locked Gold Answer: A vertical retroreflective strip on the sign support.
Locked Evidence: Section 2A.11, Figure 2A-1, PDF page 89.
F017 Figure, Medium, Answerable, M-SDI 5/10
Question: What four main areas of a temporary traffic control zone are shown in Figure 6B-1, in order of travel?
Locked Gold Answer: Advance warning area, transition area, activity area, and termination area.
Locked Evidence: Sections 6B.03–6B.07, Figure 6B-1, PDF page 813.
M004 Mixed, Medium, Answerable, M-SDI 6/10
Question: A portable CMS designer proposes ’DLY’ to mean DELAY. Using Section 1D.08 and Table 1D-3, is this acceptable and why?
Locked Gold Answer: No. DLY is explicitly prohibited because it can be misinterpreted as DAILY.
Locked Evidence: Section 1D.08, Table 1D-3, PDF pages 75, 78.
Hard Questions
Hard Questions: 10 Locked Examples
1. M002 Mixed, Hard, Answerable, M-SDI 7/10
Question: Using Figure 2A-1 and Section 2A.11, describe the requirements for a retroreflective strip used on a sign support to enhance conspicuity.
Locked Gold Answer: The strip must be at least 2 inches wide, extend from the sign down the full support to within 2 feet above the near edge of the roadway, match the sign background color except that YIELD and DO NOT ENTER strips are red, and contain no legend or other information.
Locked Evidence: Section 2A.11, Figure 2A-1, PDF pages 89, 90.
2. M003 Mixed, Hard, Answerable, M-SDI 8/10
Question: For a basic rural roadside sign, combine Figure 2A-2 with Sections 2A.15 and 2A.16 to state the normal minimum mounting height and lateral offset.
Locked Gold Answer: The minimum mounting height is 5 feet. The normal minimum lateral offset is 12 feet from the edge of the traveled way, or 6 feet from the edge of a shoulder wider than 6 feet.
Locked Evidence: Sections 2A.15, 2A.16, Figure 2A-2, PDF pages 92, 99, 101.
3. M005 Mixed, Hard, Answerable, M-SDI 7/10
Question: At an all-way stop, what plaque arrangement and appearance are required by Section 2B.04 and Figure 2B-1, and may a 4-WAY plaque be substituted?
Locked Gold Answer: An ALL-WAY (R1-3P) plaque must be mounted below each STOP sign. It has a white legend and border on a red background. A 4-WAY or other numeric-way plaque may not be substituted.
Locked Evidence: Section 2B.04, Figure 2B-1, PDF page 115.
4. M007 Mixed, Hard, Answerable, M-SDI 8/10
Question: If an agency is using an approved retroreflectivity-management method but a few individual signs temporarily fall below Table 2A-5, is the agency automatically out of compliance? Include the white-on-red thresholds relevant to STOP-type signs.
Locked Gold Answer: Not automatically. The agency complies by having and using a method designed to maintain the minimums, even if some individual signs fall below them at a particular moment. For white on red, the table gives white at least 35, red at least 7 cd/lx/m², and at least a 3:1 white-to-red contrast ratio.
Locked Evidence: Section 2A.22, Table 2A-5, PDF pages 103, 104.
5. M008 Mixed, Hard, Answerable, M-SDI 8/10
Question: On a multi-lane roadway posted at 35 mph, may the single-lane regulatory-sign size be used, and what size must a STOP sign still use?
Locked Gold Answer: For regulatory signs other than STOP, the single-lane size may be used at 35 mph or less. The STOP-sign exception remains: the multi-lane STOP size is 36 x 36 inches.
Locked Evidence: Section 2B.03, Table 2B-1, PDF pages 108, 109.
6. M010 Mixed, Hard, Answerable, M-SDI 8/10
Question: For a curve with an advisory speed of 40 mph and a radius within 401 to 700 feet, what Chevron Alignment sign spacing and placement are indicated?
Locked Gold Answer: Typical spacing is 120 feet. The signs must be on the outside of the curve, approximately at right angles to approaching traffic, with a minimum mounting height of 4 feet.
Locked Evidence: Section 2C.08, Table 2C-5, PDF page 199.
7. M012 Mixed, Hard, Answerable, M-SDI 9/10
Question: For Warrant 1 Condition A with at least two major-street lanes and one minor-street lane, what volumes must exist and over what time basis, and does meeting them compel signal installation?
Locked Gold Answer: For each of the same 8 selected hours, 600 vph must exist on the major street (both approaches total) and 150 vph on the critical minor approach (one direction) at the 100-percent thresholds. Meeting the warrant does not compel installation; an engineering study must determine whether a signal is justified.
Locked Evidence: Sections 4C.01, 4C.02, Table 4C-1, PDF page 694.
8. M015 Mixed, Hard, Answerable, M-SDI 7/10
Question: Using Section 4I.06 and Figure 4I-4, describe the pedestrian interval order, minimum buffer interval, and walking speed normally used for clearance timing.
Locked Gold Answer: The order is WALK, flashing DONT WALK pedestrian change interval, then steady DONT WALK buffer interval. The buffer interval is at least 2 seconds before conflicting traffic is released, and clearance is normally evaluated using 3.5 feet per second.
Locked Evidence: Section 4I.06, Figure 4I-4, PDF pages 764, 765.
9. M021 Mixed, Hard, Answerable, M-SDI 7/10
Question: For a Type C arrow board in Figure 6L-3, state its minimum size, legibility distance, minimum number of elements, and required categories of display modes.
Locked Gold Answer: Minimum size 96 x 48 inches, minimum legibility distance 1 mile, and at least 15 elements. It must provide at least one directional merge mode, the flashing double-arrow mode, and at least one caution mode.
Locked Evidence: Section 6L.06, Figure 6L-3, PDF page 875.
10. M024 Mixed, Hard, Answerable, M-SDI 7/10
Question: Using Section 9E.12 and Figure 9E-12, state the minimum bicycle-box length, required pavement symbol, turn-on-red rule, and green-pavement coverage rule.
Locked Gold Answer: The bicycle box must be at least 10 feet long, include at least one bicycle symbol, prohibit turns on red from its lane, and, if green pavement is used, fill the entire limits of the box.
Locked Evidence: Section 9E.12, Figure 9E-12, PDF pages 1146, 1147.
Apppendix.pdf
Evaluation Metrics
Let denote the evaluation set. For each question , let denote the locked gold evidence and let denote the ordered list of the top- retrieved text chunks. A chunk is relevant when its page, section, table reference, or figure reference matches the locked evidence for the question. We define when is relevant and otherwise.
Retrieval Metrics
Recall at 5: The main model and ablation evaluations use a question-level retrieval hit. A question receives credit when at least one relevant chunk appears within the first five ranks:
Mean reciprocal rank: Let
be the rank of the first relevant item. The reciprocal rank is zero when no relevant item appears among the top- results. Thus,
Normalized discounted cumulative gain: The discounted cumulative gain for question is
Let denote the number of relevant chunks recovered in the logged top- list. The ideal DCG places those recovered relevant chunks first:
The normalized score is
Context precision: Context precision measures the fraction of the logged text context that matches the locked evidence:
Evidence sufficiency: Let denote the top four retrieved page images and let denote the locked source pages. Let denote the locked visual entities and let denote the visual entities supplied to the answer model. The saved retrieval package is sufficient when it contains at least one relevant text chunk, one required page among the top four, or one exact required visual entity:
The main model and ablation evaluations report Recall@5, MRR over the six logged text ranks, nDCG@6, and context precision@6. The separate RAG baseline comparison reports Recall@10, MRR@10, and nDCG@10 using the same binary relevance definition.
Visual-Evidence Metrics
Let denote the questions with at least one locked figure or table identifier. For each , let be the set of locked visual entities and let be the unique visual entities supplied to the answer model.
Exact visual-crop hit rate: A hit occurs when at least one retained entity exactly matches a locked visual entity:
Visual-crop recall: Visual-crop recall measures the fraction of required visual entities recovered for each visual question and then averages across the visual subset:
Visual-crop precision: Visual-crop precision is calculated for each visual question and then averaged. A question with no retained visual entity receives zero precision:
Required-page hit rate: Let be the first page images and let be the locked source pages. For , the required-page hit rate is
Modality-routing accuracy: Let when locked visual evidence is required and otherwise. Given the router output , routing accuracy is
The visual metrics are reported as diagnostics. Exact visual hit and visual precision do not receive separate points in the 25-point retrieval score.
Question-Answering Metrics
For answer , GPT-5.6 assigns normalized scores for correctness, completeness, faithfulness, and citation localization. Let
denote the mean score for dimension . Reported percentages equal .
Visual reasoning is calculated as mean correctness over questions whose primary modality is table, figure, or mixed:
The relevance and concision term is held at for the completed model comparison. The 60-point generation score is
Strict full credit: An answer receives full credit only when correctness, completeness, faithfulness, and citation localization all equal 1:
Scores by modality, answerability, or difficulty are calculated using the same equations after restricting to the corresponding question subset.
Reliability Metrics
Let denote the 30 questions whose locked annotations mark them as unanswerable. Let equal 1 when answer receives strict full credit and 0 otherwise. Unanswerable success is
Let denote the figure-router confidence recorded for question . The Brier score is
For ten equal-width confidence bins , expected calibration error is
The calibration contribution and the measurable 12-point reliability score are
and
The confidence value comes from the visual router rather than the final answer model or GPT-5.6. The no-router ablation has no comparable confidence value, so its Brier score, ECE, calibration points, 12-point reliability score, and 97-point full score are not defined.
Composite Scores
The 25-point retrieval score is
Repeatability contributes up to 3 points in the intended 100-point framework. That component remains pending. The measured score therefore uses 97 available points:
The reported provisional score normalizes the measured total to 100:
The intended final score is
where is the pending repeatability component. Latency is reported separately through mean, median, and 90th-percentile response time. It does not contribute to the composite score.
For the no-router ablation, the common score excludes calibration and repeatability and uses a 91-point denominator:
MUTCD Structural Difficulty Index
For each question , the MUTCD Structural Difficulty Index (M-SDI) evaluates five dimensions: evidence localization, modality integration, reasoning operations, answer composition, and normative precision. Let denote the score for dimension . The total difficulty score is
Questions are categorized as
Improvement over Baselines
For GEMS-RAG score and baseline score , the relative improvement is
For percentage-valued metrics such as full-credit rate, the percentage-point improvement over the strongest baseline is
For the ablation study, we drew 800 paired bootstrap samples of the 150 question identifiers. Each sample used the same resampled questions for the original RAG and the ablated condition. We recomputed the score difference for each sample and reported the 2.5th and 97.5th percentiles. These intervals measure uncertainty from the benchmark question sample. They do not measure variation across repeated model calls.
Reproducibility Assets
The benchmark assets were frozen before model execution. The runtime question file contained only the question identifier and question text. A separate locked gold file contained the gold answer, required sections, source pages, figures, tables, answerability labels, and M-SDI annotations. The answering models did not load the gold file. This separation prevented answer leakage and kept the generation runs independent from the evaluation annotations.
The benchmark contained 150 questions, with 120 answerable and 30 unanswerable questions. The primary-modality distribution was 60 text, 30 table, 30 figure, and 30 mixed questions. The development split contained 30 questions and the test split contained 120 questions. The development split supported prompt and retrieval configuration. The test split remained fixed for final reporting.
We retained the source document, question-only benchmark, locked annotations, model registry, runtime manifests, answer files, retrieval files, error logs, item-level scores, ablation outputs, and per-response rationale records. Each model run wrote separate JSONL files for answers and retrieval traces together with a manifest that recorded the model identifier, run identifier, runner version, and execution settings. These files provide the audit trail used to reproduce the reported tables and figures.
Version control recorded the implementation state used for the experiments. The updated ablation archives identify harness version ablations_v3 and Git commit ba26b07031398fb25efca8ba793a42c6ebb4ae03. The main benchmark records also identify runner version 1.1.2 and preserve the exact provider model identifiers. Recording both the run manifest and source revision distinguishes changes in model behavior from changes in the evaluation code.
Implementation Hyperparameters
The values in Tables 1 and 2 come from the saved configuration, run manifests, provider adapter, and ablation metadata. Parameters that the code did not set explicitly are identified as provider defaults.
Retry, Resume, and Failure Handling
The runner wrote each completed record to disk before moving to the next question. Resume mode skipped valid records that already existed and continued from missing or failed question identifiers. The runner allowed at most five provider attempts and placed a 2.0-second interval between requests. For Claude, the provider adapter used a 16,000-token output cap and performed one truncation-specific retry with twice the limit, capped at 32,000 tokens.
We reran records only for operational failures, missing records, corrupted outputs, provider errors, or materially incomplete retrieval logging. We did not rerun an answer because it was factually incorrect or received a low score. When a retry succeeded, the evaluator retained the first valid response for the question and preserved the original failure record for audit. This rule avoided selecting a more favorable answer after observing model quality.
Role of GPT-5.6 and the Evaluated Language Models
The authors used GPT-5.6 to review benchmark-runner code, provider adapters, runtime validation checks, scoring scripts, plotting code, and technical documentation. The authors executed the code, inspected saved records, selected the final settings, and resolved failures before accepting an output. Model-generated code or prose did not enter the reported results without execution and record-level verification.
During evaluation, GPT-5.6 supported benchmark construction, evidence review, and application of the locked answer-quality rubric. It did not generate the retrieval ranks, latency values, calibration statistics, or composite scores. Fixed scripts calculated those quantities from the saved records. The evaluated vision-language models received only the user question and retrieved evidence. They did not receive the locked gold answer file.
The cross-model study used twelve vision-language configurations under the same benchmark and scoring framework. The selected model also controlled visual filtering when the filter-model override was null. Claude Fable 5 produced the answers for the detailed modality, difficulty, and ablation analyses after it achieved the highest provisional score in the cross-model comparison. The ablation study kept Claude Fable 5 and the few-shot prompt fixed while removing one retrieval component at a time.
Answer-Rationale Audit and Evaluator Confidence
We created one grading rationale for each of the 900 ablation responses and each of the 150 original-RAG baseline responses. Each record retained the four official item scores, the strict full-credit flag, the model answer, the locked gold answer, the retrieved evidence metrics, and the rationale provenance. The rationale package distinguishes an explicit adjudication note from a structured reconstruction of an existing grade.
| Setting | Value |
| Text representation | BAAI/bge-m3, used for dense and learned sparse representations |
| Visual representation | vidore/colqwen2-v0.1 |
| Cross-encoder reranker | mixedbread-ai/mxbai-rerank-large-v2 |
| Page and figure rendering | 180 dpi for pages, 220 dpi for figure and table crops |
| Text retrieval depths | 30 dense, 30 sparse, 30 fused, 40 after graph expansion, 6 after reranking |
| Visual retrieval depths | 10 candidate entities, 6 direct visual candidates, at most 4 retained entities, 4 page images |
| Caption fallback | Disabled |
| Question router | Enabled, soft threshold 0.5, VLM tie break disabled |
| VLM figure filter | Enabled. For the final Claude and Gemini runs, the filter followed the selected answer model because the filter override was null |
| Fusion weights | Dense 1.00, sparse 0.60, hierarchy 0.20, graph proximity 0.40, rule type 0.30 |
| Rule-type multipliers | Standard 1.20, Guidance 1.00, Option 0.90, Support 0.70 |
| ColQwen2 image settings | Maximum 768 image patches, binary quantization enabled |
| Maximum text from each chunk | 1400 characters in the answer prompt |
Table 1 (PDF p. 22). Core indexing and retrieval hyperparameters.
| Setting | Value |
| Detailed-analysis and ablation model | Claude Fable 5, alias frontier_claude |
| Answer prompt | Few-shot, 3 examples |
| Visual-filter prompt | Zero-shot |
| Claude output-token cap | 16,000 tokens, including adaptive-thinking tokens and visible answer tokens |
| Truncation handling | One retry with a multiplier of 2.0, capped at 32,000 tokens for Claude |
| Temperature and top-p | Not set explicitly in the adapter, so the provider defaults applied |
| Benchmark runner | Version 1.1.2 |
| Random seed and replicates | Seed 20260710, 1 replicate per model configuration |
| Retry policy | Maximum 5 attempts, 2.0 seconds between requests, resume enabled |
| Ablation controls | Claude Fable 5 and the few-shot prompt remained fixed. Each run removed one component |
| Paired bootstrap | 800 paired resamples of the 150 question identifiers |
Table 2 (PDF p. 22). Generation, benchmark-runner, and evaluation hyperparameters.
Programmatic Consistency Checks
Table 3 summarizes checks performed over all 1,050 records. These checks validate the internal representation of the official grades. They do not establish that every underlying item grade is correct against the MUTCD.
The token-overlap field is diagnostic rather than authoritative. It listed a missing or underdeveloped element in 216 full-credit records, primarily when an unanswerable response used a refusal phrasing that did not lexically match the gold wording. It listed no missing element in 173 partial-credit records, where the deduction instead came from an adjudication note, faithfulness, or citation localization. The official four scores and the adjudication notes remain the source of truth.
A direct recheck found 20 records with citation score 1.00 and an empty regex-detected citation field. All 20 were unanswerable responses involving a false premise, unsupported precision, or out-of-scope request. The accepted citation-localization score reflected correct handling of the unsupported request, while the citation detector searched only for explicit section, figure, table, page, or legal-reference patterns. We therefore treat these records as a detector limitation rather than a contradiction in the official grade.
Evaluator Confidence
GPT-5.6 gave a strict self-assessment of 82% confidence in the complete ablation evaluation. This value reflected high confidence in deterministic runtime and retrieval calculations and lower confidence in rubric-based item judgments. The value is a judgmental assessment, not a model probability and not a sampling-based confidence interval.
The authors then supplied the 1,050 rationale records to Claude Fable 5 for a secondary model audit. Fable 5 checked score arithmetic, aggregate agreement, provenance rules, repaired-record allocation, and a small set of MUTCD content examples. It assigned 85% confidence with a reported evaluator-confidence interval of [78%, 90%]. Fable 5 placed the mechanical consistency of the package near the top of its scale because the score and aggregate checks matched. It reduced its overall assessment because most rationales reconstruct or restate existing grades and therefore do not independently validate the substantive MUTCD judgment.
The reported evaluator-confidence summary is
This interval is a judgmental interval supplied by a secondary language-model auditor. It is not a frequentist confidence interval, a Bayesian credible interval, or the paired bootstrap interval used for ablation score differences. Claude Fable 5 also generated the evaluated answers, so this secondary audit does not replace an independent MUTCD-informed human annotator.
| Check | Verified result |
| Strict full-credit flag compared with all four dimensions equal to 1.00 | 0 mismatches in 1,050 records |
| Descriptive mean compared with the mean of correctness, completeness, faithfulness, and citation | 0 mismatches in 1,050 records |
| Per-condition correctness, completeness, faithfulness, citation, and full-credit aggregates compared with the version 4 report | Exact match for all seven conditions |
| Strict full-credit counts | 120, 114, 112, 114, 112, 109, and 103 for the baseline and A1 through A6 |
| Rationale provenance | 895 structured reconstructions, 48 explicit adjudication notes, and 107 reused baseline notes |
| Reused-note criterion | 0 violations. Each reused note had the same question identifier and identical four official scores as the baseline |
| Explicit-note allocation | 30 original-RAG partial-credit records and 18 repaired A1 records |
| Repaired A1 records | Exactly T001 through T018 |
Table 3 (PDF p. 23). Programmatic checks of the per-response rationale package.
The rationale package assigns fixed confidence values by provenance tier: 0.90 for explicit adjudication notes, 0.88 for reconstructed full-credit records, 0.82 for the 18 repaired A1 responses reviewed by GPT-5.6, and 0.76 for reconstructed partial-credit rationales. These are design constants used to describe confidence in the explanation. They are not empirical probabilities. A second MUTCD-informed annotator should review a stratified sample, report agreement, and adjudicate disagreements before the item-level judgments are treated as independently validated.
Sample MUTCD-150 Questions and Locked Gold Answers
This section provides 10 representative examples for each reporting category: answerable, unanswerable, easy, medium, and hard. The five displayed sets do not repeat question identifiers. Answerability and M-SDI difficulty remain independent benchmark labels, so each entry retains both labels. We selected examples to cover text, table, figure, and mixed evidence where the category permits. The question wording, locked gold answers, evidence identifiers, answerability labels, and M-SDI scores below are copied from the locked gold file. The answering models did not receive this file during execution.
Answerable Questions
| Answerable Questions: 10 Locked Examples |
| 1. T002 — Text, Easy, Answerable, M-SDI 1/10 |
| Question: Does the applicability of the MUTCD to a facility open to public travel depend on whether the facility is publicly or privately owned or on the source of its funding? |
| Locked Gold Answer: No. Applicability is independent of public or private ownership or jurisdiction and independent of whether funding is Federal, State, local, or private. |
| Locked Evidence: Section 1A.01, PDF page 42. |
Table 4.
| 2. T005 — Text, Easy, Answerable, M-SDI 3/10 |
| Question: How do the MUTCD headings Standard, Guidance, Option, and Support differ, including the verbs typically associated with them? |
| Locked Gold Answer: A Standard is mandatory or prohibitive and typically uses ’shall.’ Guidance is recommended practice with justified deviations allowed and typically uses ’should.’ An Option is permissive and typically uses ’may.’ Support is informational and conveys no mandate, recommendation, permission, prohibition, or enforceable condition. |
| Locked Evidence: Section 1C.01, PDF page 54. |
Table 5.
| 3. T025 — Text, Medium, Answerable, M-SDI 5/10 |
| Question: What must a Speed Limit sign display, in what increments, and may it be used to show an advisory speed for a roadway condition? |
| Locked Gold Answer: It must display the speed limit established by law, ordinance, regulation, or an authorized agency based on an engineering study, in multiples of 5 mph. It should not be used for an advisory speed; an advisory speed plaque below an appropriate warning sign should be used instead. |
| Locked Evidence: Section 2B.21, PDF pages 126, 127. |
Table 6.
| 4. TB001 — Table, Easy, Answerable, M-SDI 3/10 |
| Question: According to Table 1B-1, what is the compliance date for implementing and continuing to use a method designed to maintain longitudinal pavement-marking retroreflectivity? |
| Locked Gold Answer: September 6, 2026. |
| Locked Evidence: Section 1B.03, Table 1B-1, PDF page 46. |
Table TB001.
| Component | Count |
| Sections | 678 |
| Text Chunks | 5705 |
| Figures | 485 |
| Tables | 68 |
| Knowledge Graph Nodes | 8240 |
| Knowledge Graph Edges | 18245 |
Table 1. MUTCD dataset statistics.
Table TB002.
TB024 Table, Medium, Answerable, M-SDI 5/10
Question: What minimum off-roadway sizes does Table 9A-1 give for a STOP sign, a YIELD sign, and a Crossbuck sign on a bicycle facility?
Locked Gold Answer: STOP: 18 x 18 inches; YIELD: 18 x 18 x 18 inches; Crossbuck: 24 x 4.5 inches.
Locked Evidence: Section 9A.03, Table 9A-1, PDF page 1090.
F008 Figure, Medium, Answerable, M-SDI 5/10
Question: From Figure 2B-1, identify the shapes of the STOP and YIELD signs and the color scheme of the ALL-WAY plaque.
Locked Gold Answer: STOP is an octagon; YIELD is a downward-pointing equilateral triangle; the ALL-WAY plaque has white legend and border on a red background.
Locked Evidence: Sections 2B.04, 2B.05, Figure 2B-1, PDF page 115.
F012 Figure, Medium, Answerable, M-SDI 5/10
Question: What four crosswalk marking patterns are illustrated in Figure 3C-1?
Locked Gold Answer: Transverse, longitudinal bar, ladder, and bar pair.
Locked Evidence: Section 3C.03, Figure 3C-1, PDF page 631.
M001 Mixed, Hard, Answerable, M-SDI 7/10
Question: A provision appears under a Guidance heading and uses the word ’should.’ Is it mandatory, and when may a deviation be appropriate?
Locked Gold Answer: It is recommended practice, not a mandatory Standard. A deviation may be appropriate when engineering judgment or an engineering study indicates that the deviation is suitable for the specific situation.
Locked Evidence: Sections 1A.04, 1C.01, PDF pages 43, 54.
M019 Mixed, Hard, Answerable, M-SDI 9/10
Question: For a 12-foot lateral offset and a 35-mph speed, calculate L using Table 6B-4. Then state the minimum merging, shifting, and shoulder taper lengths from Table 6B-3.
Locked Gold Answer: For 40 mph or less, L = = = 245 feet. Minimum merging taper: 245 feet; shifting taper: 122.5 feet; shoulder taper: about 80.9 feet (0.33L, commonly rounded appropriately in design).
Locked Evidence: Section 6B.08, Tables 6B-3, 6B-4, PDF page 816.
Unanswerable Questions
Unanswerable Questions: 10 Locked Examples
T049 Text, Medium, Unanswerable, M-SDI 5/10, Unsupported Precision
Question: According to the MUTCD, what exact percentage reduction in crashes is guaranteed by installing a STOP sign or all-way stop control?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 2B.04, 2B.12-2B.17, PDF pages 115, 120.
T052 Text, Medium, Unanswerable, M-SDI 4/10, False Presupposition
Question: What single mandatory signal-cycle length must be used at every urban signalized intersection in the United States?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 4A.01-4A.06, 4B.01-4B.05, PDF pages 684, 691.
T054 Text, Medium, Unanswerable, M-SDI 5/10, Out Of Scope
Question: How many licensed traffic engineers must a road agency employ per 100 centerline miles to comply with the MUTCD?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 1A.04, 1D.03, PDF pages 43, 73.
T059 Text, Easy, Unanswerable, M-SDI 3/10, Contradicted Premise
Question: What minimum miles-per-gallon rating must a vehicle achieve to qualify as an inherently low-emission vehicle under the MUTCD?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Section 1C.02, PDF pages 60, 62.
TB027 Table, Easy, Unanswerable, M-SDI 3/10, Unsupported Precision
Question: What RGB values are listed in Table 2A-2 for fluorescent yellow-green?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Section 2A.06, Table 2A-2, PDF page 85.
TB029 Table, Medium, Unanswerable, M-SDI 6/10, Out Of Scope
Question: Which table lists State-by-State monetary fines for violating a No Turn on Red sign?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Section 2B.60, Table 2B-1, PDF page 113.
F026 Figure, Easy, Unanswerable, M-SDI 3/10, Absent Information
Question: What foundation depth is required for the sign posts drawn in Figure 2A-2?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 2A.13, 2A.18, Figure 2A-2, PDF page 92.
F029 Figure, Easy, Unanswerable, M-SDI 3/10, Unsupported Precision
Question: What exact crossing angle, in degrees, is shown for the railroad tracks in Figure 8E-1?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 8E.01, 8E.02, Figure 8E-1, PDF page 1076.
M028 Mixed, Medium, Unanswerable, M-SDI 6/10, Absent Information
Question: What were the sample size, participant demographics, and statistical significance values of the human-factors study underlying each treatment in Figure 2A-1?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Section 2A.11, Figure 2A-1, PDF pages 89, 90.
M030 Mixed, Hard, Unanswerable, M-SDI 7/10, Out Of Scope
Question: What exact dollar penalty and criminal classification apply when an agency violates a MUTCD Standard?
Locked Gold Answer: The MUTCD does not specify this information. A correct response should state that the requested value or requirement cannot be determined from the manual rather than inventing an answer.
Locked Evidence: Sections 1B.01, 1D.02, PDF pages 45, 72.
Easy Questions
Easy Questions: 10 Locked Examples
T001 Text, Easy, Answerable, M-SDI 2/10
Question: What is the purpose of the MUTCD, and what four objectives are used to achieve that purpose?
Locked Gold Answer: The MUTCD establishes uniform national criteria for traffic control devices on streets, highways, pedestrian and bicycle facilities, and site roadways open to public travel. Its objectives are to promote safety, inclusion, and mobility; promote efficiency through national uniformity in device meaning and appearance; promote consistency in use, installation, and operation; and provide basic decision-making principles for traffic engineers.
Locked Evidence: Section 1A.01, PDF page 42.
T003 Text, Easy, Answerable, M-SDI 1/10
Question: Are islands, curbs, speed humps, raised roadway surfaces, and rumble strips considered traffic control devices under the MUTCD?
Locked Gold Answer: No. Infrastructure elements that physically restrict travel paths or speeds, including islands, curbs, speed humps, other raised surfaces, and transverse or longitudinal rumble strips, are not traffic control devices.
Locked Evidence: Section 1A.02, PDF page 42.
T014 Text, Easy, Answerable, M-SDI 2/10
Question: What general meanings does the MUTCD assign to fluorescent pink, orange, and purple?
Locked Gold Answer: Fluorescent pink is for incident management, orange is for temporary traffic control, and purple is restricted to vehicles with registered electronic toll collection accounts.
Locked Evidence: Section 1D.05, PDF page 74.
T019 Text, Easy, Answerable, M-SDI 2/10
Question: What are the minimum mounting heights for roadside signs in rural areas and in business, commercial, or residential areas where parking, bicyclist, or pedestrian activity is likely?
Locked Gold Answer: The minimum is 5 feet in rural areas and 7 feet in business, commercial, or residential areas where such activity or obstruction is likely, measured as specified from the near pavement edge or curb.
Locked Evidence: Section 2A.15, PDF page 99.
TB011 Table, Easy, Answerable, M-SDI 3/10
Question: At a posted or 85th-percentile speed of 60 mph, what advance placement distance does Table 2C-3 show for deceleration to an advisory speed of 30 mph?
Locked Gold Answer: 200 feet.
Locked Evidence: Section 2C.04, Table 2C-3, PDF page 193.
TB014 Table, Easy, Answerable, M-SDI 3/10
Question: What minimum passing sight distance does Table 3B-1 specify for a speed of 55 mph?
Locked Gold Answer: 900 feet.
Locked Evidence: Section 3B.03, Table 3B-1, PDF page 584.
TB019 Table, Easy, Answerable, M-SDI 3/10
Question: What minimum sight distance for signal visibility does Table 4D-2 list at an 85th-percentile speed of 45 mph?
Locked Gold Answer: 460 feet.
Locked Evidence: Section 4D.06, Table 4D-2, PDF page 711.
F001 Figure, Easy, Answerable, M-SDI 3/10
Question: In Figure 1B-1, what happens immediately after FHWA approves a request to experiment?
Locked Gold Answer: The requesting jurisdiction installs the experimental traffic control device.
Locked Evidence: Section 1B.05, Figure 1B-1, PDF page 48.
F009 Figure, Easy, Answerable, M-SDI 3/10
Question: Which sign is placed above the other in every paired assembly shown in Figure 2B-17?
Locked Gold Answer: The DO NOT ENTER sign is above the WRONG WAY sign.
Locked Evidence: Section 2B.48, Figure 2B-17, PDF page 156.
F024 Figure, Easy, Answerable, M-SDI 3/10
Question: In Figure 9E-12, what minimum distance separates the advance motor-vehicle stop line from the intersection stop line across the bicycle box?
Locked Gold Answer: 10 feet minimum.
Locked Evidence: Section 9E.12, Figure 9E-12, PDF page 1147.
Medium Questions
Medium Questions: 10 Locked Examples
T009 Text, Medium, Answerable, M-SDI 4/10
Question: When a non-compliant traffic control device is replaced because it is damaged, missing, or no longer serviceable, what is the general rule and the limited replacement-in-kind exception?
Locked Gold Answer: The replacement must generally be a compliant device. A non-compliant device may be replaced in kind only when engineering judgment finds that a lone compliant device among adjacent non-compliant devices would confuse users and/or the planned replacement schedule for the whole series will achieve timely compliance.
Locked Evidence: Section 1B.03, PDF page 47.
T012 Text, Medium, Answerable, M-SDI 5/10
Question: What public-domain condition applies to a new traffic control device proposed for experimentation or interim approval?
Locked Gold Answer: The traffic-control message, design, or application must be in the public domain and not protected by patent, trademark, or copyright. Legally binding statements are required from the relevant agency/developer/supplier for experimentation. Protection of internal components or manufacturing methods can still be allowed.
Locked Evidence: Sections 1B.05, 1D.06, PDF pages 50, 74, 75.
T035 Text, Medium, Answerable, M-SDI 5/10
Question: Does satisfying a traffic signal warrant automatically require installation of a traffic control signal?
Locked Gold Answer: No. A warrant is a threshold that triggers analysis; it is not conclusive justification. Installation must be based on an engineering study considering safety, operation, alternatives, and the site. A signal can also be installed where no warrant is met if an engineering study documents why it is the best solution.
Locked Evidence: Sections 4B.04, 4C.01, PDF pages 691, 694.
T046 Text, Medium, Answerable, M-SDI 4/10
Question: When two-way traffic must share one lane for a limited distance, what coordination rule and control methods apply?
Locked Gold Answer: Movements from both ends must be coordinated, except for a short, low-volume, mutually visible work zone that can self-regulate. Methods include flaggers, flag transfer, a pilot car, temporary traffic signals, or stop/yield control.
Locked Evidence: Section 6E.01, PDF page 828.
TB003 Table, Medium, Answerable, M-SDI 4/10
Question: In Table 1D-2, how is ’Road Work’ abbreviated on a portable changeable message sign, and what may follow it?
Locked Gold Answer: RD WK; it may be followed by AHEAD or a distance, such as ’RD WK 1 MILE.’
Locked Evidence: Section 1D.08, Table 1D-2, PDF page 77.
TB007 Table, Medium, Answerable, M-SDI 5/10
Question: For white-on-red signs, what minimum maintained retroreflectivity values and contrast criterion are shown in Table 2A-5?
Locked Gold Answer: White at least 35 cd/lx/m², red at least 7 cd/lx/m², and a white-to-red contrast ratio of at least 3:1.
Locked Evidence: Section 2A.22, Table 2A-5, PDF page 104.
TB018 Table, Medium, Answerable, M-SDI 4/10
Question: For a four-leg intersection, what one-year crash thresholds are listed in Table 4C-2 for all-severity angle/pedestrian crashes and for fatal-and-injury angle/pedestrian crashes?
Locked Gold Answer: Five all-severity angle and pedestrian crashes, or three fatal-and-injury angle and pedestrian crashes.
Locked Evidence: Section 4C.08, Table 4C-2, PDF page 702.
F003 Figure, Medium, Answerable, M-SDI 4/10
Question: What conspicuity treatment is labeled ’E’ in Figure 2A-1?
Locked Gold Answer: A vertical retroreflective strip on the sign support.
Locked Evidence: Section 2A.11, Figure 2A-1, PDF page 89.
F017 Figure, Medium, Answerable, M-SDI 5/10
Question: What four main areas of a temporary traffic control zone are shown in Figure 6B-1, in order of travel?
Locked Gold Answer: Advance warning area, transition area, activity area, and termination area.
Locked Evidence: Section 6B.03-6B.07, Figure 6B-1, PDF page 813.
M004 Mixed, Medium, Answerable, M-SDI 6/10
Question: A portable CMS designer proposes ’DLY’ to mean DELAY. Using Section 1D.08 and Table 1D-3, is this acceptable and why?
Locked Gold Answer: No. DLY is explicitly prohibited because it can be misinterpreted as DAILY.
Locked Evidence: Section 1D.08, Table 1D-3, PDF pages 75, 78.
Hard Questions
Hard Questions: 10 Locked Examples
M002 Mixed, Hard, Answerable, M-SDI 7/10
Question: Using Figure 2A-1 and Section 2A.11, describe the requirements for a retroreflective strip used on a sign support to enhance conspicuity.
Locked Gold Answer: The strip must be at least 2 inches wide, extend from the sign down the full support to within 2 feet above the near edge of the roadway, match the sign background color except that YIELD and DO NOT ENTER strips are red, and contain no legend or other information.
Locked Evidence: Section 2A.11, Figure 2A-1, PDF pages 89, 90.
M003 Mixed, Hard, Answerable, M-SDI 8/10
Question: For a basic rural roadside sign, combine Figure 2A-2 with Sections 2A.15 and 2A.16 to state the normal minimum mounting height and lateral offset.
Locked Gold Answer: The minimum mounting height is 5 feet. The normal minimum lateral offset is 12 feet from the edge of the traveled way, or 6 feet from the edge of a shoulder wider than 6 feet.
Locked Evidence: Sections 2A.15, 2A.16, Figure 2A-2, PDF pages 92, 99, 101.
M005 Mixed, Hard, Answerable, M-SDI 7/10
Question: At an all-way stop, what plaque arrangement and appearance are required by Section 2B.04 and Figure 2B-1, and may a 4-WAY plaque be substituted?
Locked Gold Answer: An ALL-WAY (R1-3P) plaque must be mounted below each STOP sign. It has a white legend and border on a red background. A 4-WAY or other numeric-way plaque may not be substituted.
Locked Evidence: Section 2B.04, Figure 2B-1, PDF page 115.
M007 Mixed, Hard, Answerable, M-SDI 8/10
Question: If an agency is using an approved retroreflectivity-management method but a few individual signs temporarily fall below Table 2A-5, is the agency automatically out of compliance? Include the white-on-red thresholds relevant to STOP-type signs.
Locked Gold Answer: Not automatically. The agency complies by having and using a method designed to maintain the minimums, even if some individual signs fall below them at a particular moment. For white on red, the table gives white at least 35, red at least 7 cd/lx/m², and at least a 3:1 white-to-red contrast ratio.
Locked Evidence: Section 2A.22, Table 2A-5, PDF pages 103, 104.
M008 Mixed, Hard, Answerable, M-SDI 8/10
Question: On a multi-lane roadway posted at 35 mph, may the single-lane regulatory-sign size be used, and what size must a STOP sign still use?
Locked Gold Answer: For regulatory signs other than STOP, the single-lane size may be used at 35 mph or less. The STOP-sign exception remains: the multi-lane STOP size is 36 x 36 inches.
Locked Evidence: Section 2B.03, Table 2B-1, PDF pages 108, 109.
M010 Mixed, Hard, Answerable, M-SDI 8/10
Question: For a curve with an advisory speed of 40 mph and a radius within 401 to 700 feet, what Chevron Alignment sign spacing and placement are indicated?
Locked Gold Answer: Typical spacing is 120 feet. The signs must be on the outside of the curve, approximately at right angles to approaching traffic, with a minimum mounting height of 4 feet.
Locked Evidence: Section 2C.08, Table 2C-5, PDF page 199.
M012 Mixed, Hard, Answerable, M-SDI 9/10
Question: For Warrant 1 Condition A with at least two major-street lanes and one minor-street lane, what volumes must exist and over what time basis, and does meeting them compel signal installation?
Locked Gold Answer: For each of the same 8 selected hours, 600 vph must exist on the major street (both approaches total) and 150 vph on the critical minor approach (one direction) at the 100-percent thresholds. Meeting the warrant does not compel installation; an engineering study must determine whether a signal is justified.
Locked Evidence: Sections 4C.01, 4C.02, Table 4C-1, PDF page 694.
M015 Mixed, Hard, Answerable, M-SDI 7/10
Question: Using Section 4I.06 and Figure 4I-4, describe the pedestrian interval order, minimum buffer interval, and walking speed normally used for clearance timing.
Locked Gold Answer: The order is WALK, flashing DON'T WALK pedestrian change interval, then steady DON'T WALK buffer interval. The buffer interval is at least 2 seconds before conflicting traffic is released, and clearance is normally evaluated using 3.5 feet per second.
Locked Evidence: Section 4I.06, Figure 4I-4, PDF pages 764, 765.
M021 Mixed, Hard, Answerable, M-SDI 7/10
Question: For a Type C arrow board in Figure 6L-3, state its minimum size, legibility distance, minimum number of elements, and required categories of display modes.
Locked Gold Answer: Minimum size 96 x 48 inches, minimum legibility distance 1 mile, and at least 15 elements. It must provide at least one directional merge mode, the flashing double-arrow mode, and at least one caution mode.
Locked Evidence: Section 6L.06, Figure 6L-3, PDF page 875.
M024 Mixed, Hard, Answerable, M-SDI 7/10
Question: Using Section 9E.12 and Figure 9E-12, state the minimum bicycle-box length, required pavement symbol, turn-on-red rule, and green-pavement coverage rule.
Locked Gold Answer: The bicycle box must be at least 10 feet long, include at least one bicycle symbol, prohibit turns on red from its lane, and, if green pavement is used, fill the entire limits of the box.
Locked Evidence: Section 9E.12, Figure 9E-12, PDF pages 1146, 1147.
References
- [9]Asai, A.; Wu, Z.; Wang, Y.; Sil, A.; and Hajishirzi, H. 2023. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ArXiv:2310.11511 [cs].arxiv.org/abs/2310.11511
- [10]Doris, A. C.; Grandi, D.; Tomich, R.; Alam, M. F.; Ataei, M.; Cheong, H.; and Ahmed, F. 2025. Designqa: A multimodal benchmark for evaluating large language models’ understanding of engineering documentation. Journal of Computing and Information Science in Engineering, 25(2): 021009.arxiv.org/abs/2404.07917
- [11]Faysse, M.; Sibille, H.; Wu, T.; Omrani, B.; Viaud, G.; Hude-lot, C.; and Colombo, P. 2025. Copali: Efficient document retrieval with vision language models. In International Conference on Learning Representations, volume 2025, 61424–61449.arxiv.org/abs/2407.01449
- [12]Federal Highway Administration. 2025. Manual on Uniform Traffic Control Devices for Streets and Highways. U.S. Department of Transportation, Federal Highway Administration, Washington, DC, 11th ed., revision 1 edition.
- [17]Karpukhin, V.; Oğuz, B.; Min, S.; Lewis, P.; Wu, L.; Edunov, S.; Chen, D.; and Yih, W.-t. 2020. Dense Passage Retrieval for Open-Domain Question Answering. ArXiv:2004.04906 [cs].arxiv.org/abs/2004.04906
- [18]Kuai, C.; Jiang, J.; Zhu, Z.; Wang, H.; Wu, K.; Li, Z.; Zhang, Y.; Liu, C.; Tu, Z.; Fan, Z.; et al. 2026a. How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles. arXiv preprint arXiv:2604.07650.
- [19]Kuai, C.; Wu, C.; Zhou, Y.; Wang, B.; Yang, T.; Tu, Z.; Li, Z.; and Zhang, Y. 2026b. Cyportqa: Benchmarking multimodal large language models for cyclone preparedness in port operation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 38781–38789.
- [20]Lála, J.; O’Donoghue, O.; Shtedritski, A.; Cox, S.; Rodrigues, S. G.; and White, A. D. 2023. Paperqa: Retrieval-augmented generative agent for scientific research. arXiv preprint arXiv:2312.07559.arxiv.org/abs/2312.07559
- [21]Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.-t.; Rocktäschel, T.; et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 9459–9474.arxiv.org/abs/2005.11401
- [22]Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.-t.; Rocktäschel, T.; Riedel, S.; and Kiela, D. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. ArXiv:2005.11401 [cs].arxiv.org/abs/2005.11401
- [26]OpenAI. 2026. GPT-5.6 in ChatGPT. https://help.openai.com/en/articles/20001354-gpt-56-in-chatgpt. Accessed: July 27, 2026.
- [28]Procko, T. T.; and Ochoa, O. 2024. Graph retrieval-augmented generation for large language models: A survey. In 2024 Conference on AI, science, engineering, and technology (AIxSET), 166–169. IEEE.DOI
- [29]Robertson, S.; and Zaragoza, H. 2009. The probabilistic relevance framework: BM25 and beyond, volume 4. Now Publishers Inc.
- [30]Shi, Q.; Yang, Y.; Di, J.; Li, L.; Liu, J.; Hu, Y.; Xiao, J.; and Hu, Z. 2026. Trust-Aware KG-RAG: a framework for trustworthy question answering for engineering equipment. Advanced Engineering Informatics, 74: 104801.DOI
- [31]Skarlinski, M. D.; Cox, S.; Laurent, J. M.; Braza, J. D.; Hinks, M.; Hammerling, M. J.; Ponnapati, M.; Rodrigues, S. G.; and White, A. D. 2024. Language agents achieve su-perhuman synthesis of scientific knowledge. arXiv preprint arXiv:2409.13740.
- [32]Wang, J.; Chen, M.; Hu, B.; Yang, D.; Liu, Z.; Shen, Y.; Wei, P.; Zhang, Z.; Gu, J.; Zhou, J.; Pan, J. Z.; Zhang, W.; and Chen, H. 2024. Generating to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs. In Findings of the Association for Computational Linguistics: EMNLP 2024, 7813–7835. Miami, Florida, USA: Association for Computational Linguistics.arxiv.org/abs/2406.14282
- [35]Yang, X.; Sun, K.; Xin, H.; Sun, Y.; Bhalla, N.; Chen, X.; Choudhary, S.; Gui, R. D.; Jiang, Z. W.; Jiang, Z.; et al. 2024. Crag-comprehensive rag benchmark. Advances in Neural Information Processing Systems, 37: 10470–10490.arxiv.org/abs/2406.04744
- [37]Zhai, W. 2024. Self-adaptive Multimodal Retrieval-Augmented Generation.arxiv.org/abs/2410.11321

