From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines

CCS Concepts: • Software and its engineering → Software testing and debugging; Software defect analysis; Software maintenance tools.

ACM Reference Format:

Hira Naveed, Scott Barnett, Stefanus Kurniawan, John Grundy, Chetan Arora, Hourieh Khalajzadeh, and Omar Haggag. 2026. From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines. ACM Trans. Softw. Eng. Methodol. 1, 1 (November 2026), 44 pages. https://doi.org/XXXXXXX.XXXXXXX

Introduction

Retrieval-Augmented Generation (RAG) is widely adopted for domain-specific question answering, as it enhances the responses of large language models (LLMs) with query-relevant context retrieved from external knowledge sources [7, 10, 28, 37]. It enables organizations to leverage proprietary data without retraining models and reduces the risk of

Authors’ Contact Information: Hira Naveed, [email protected], Monash University, Australia; Scott Barnett, Deakin University, Australia, [email protected]; Stefanus Kurniawan, Deakin University, Australia; John Grundy, Monash University, Australia; Chetan Arora, Monash University, Australia; Hourieh Khalajzadeh, Deakin University, Australia; Omar Haggag, Monash University, Australia.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

© 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. hallucination. While these advantages are substantial, developing and maintaining RAG pipelines is not trivial. A RAG pipeline is not a single model but a sequence of interdependent stages, typically comprising query rewriting, embedding, retrieval, reranking, and answer generation, many of which are realized through calls to third-party services such as hosted language models, embedding models, vector databases, and reranking APIs [9].

Faults can arise at any of these stages, and their effects may propagate downstream, making it difficult to identify their origins [9, 35, 58]. For example, a rewriter may add a constraint that the user never asked for, which leads the retriever to irrelevant chunks, and a generator may still produce a correct answer from incomplete context. Such effects are far from rare. In our study of 1,500 runs across five open-source RAG pipelines (§ 4), 37% of the runs contained at least one faulty stage, and more than a third of these runs still produced a correct answer. Furthermore, among the 357 runs with incorrect answers, 71% have at least one faulty stage before the generator. Detecting these faults is currently a manual and iterative process in which developers inspect logs and adjust parameters to debug RAG pipelines [10, 35]. This approach is error-prone and expensive in terms of time and resources. For example, it may require re-indexing all documents after parameter changes and then repeating the process several times [35].

Automated fault localization for RAG pipelines faces two main challenges. First, logs need to be converted into a form that can be analyzed. RAG pipelines are built from different providers, request formats, and call sequences; therefore, an automated approach must identify which call belongs to which stage, and what each stage received and returned, without manual configuration or code instrumentation. Second, faults need to be identified without a failure signal. Traditional fault localization starts from a known failure, such as a failing test, and searches for its cause [63]. However, a RAG pipeline provides no such signal, since reference answers are not available in production and not all faults change the final answer. Each stage therefore needs to be assessed on its own, without reference answers. Existing RAG evaluation frameworks measure how well documents are retrieved and how accurately responses are generated, but they often rely on ground truth and do not cover all stages of the pipeline [15, 20, 45, 55]. Observability and tracing tools record each call but leave its analysis to the developer [1, 14], while general-purpose coding agents can analyze logs but are not designed to locate faults across RAG stages. These gaps highlight the need for an automated, reference-free approach to locate faults across the RAG pipeline.

In this study, we present RAGLog, a reference-free fault localization approach for debugging, evaluating, and monitoring RAG pipelines. RAGLog uses call-level execution logs, which contain one record for each API call that a pipeline stage makes to an external service, such as a language model, an embedding model, or a vector database. These logs can be collected through existing tracing tools, API gateways, or the logging proxy we provide with RAGLog. The approach first transforms execution logs into a common representation by classifying each call into a RAG stage, using rules for common APIs and an LLM for unfamiliar ones, and mapping each call to a common RAG schema. Then, a reference-free detector decides for every stage of every run if that stage is faulty. The detector combines rule-based and similarity-based measurements with targeted checks performed by a small LLM. A decision function learned from labeled runs combines these measurements into a verdict for each stage. Since the approach uses a common schema, a single detector works for multiple RAG pipelines.

We evaluate RAGLog on five open-source RAG pipelines, which use models and services from seven providers, and three public datasets. In addition to 1,500 natural runs containing 670 faults, we evaluate RAGLog on 480 injected runs containing 419 faults. To evaluate RAGLog on faults with a known source, we developed a fault injector that modifies the input or output of a single stage while the pipeline runs, so that all later stages react to the fault as they would in practice. All development and evaluation runs, together with their stage-level fault labels and full traces, are released as RAGLog Faults, a dataset of 3,134 runs.

To the best of our knowledge, no existing approach locates faults across all RAG pipeline stages without reference answers. Therefore, we compare RAGLog with two general-purpose coding agents, Claude Code and Codex, which are given the same logs, reflecting how a developer without a dedicated tool would approach the task. On natural runs, RAGLog, with Claude Haiku 5.5 as its underlying LLM, achieves a pooled F1 score of 0.73, compared to 0.58 for Claude Code and 0.52 for Codex, as the agents flag fewer stages and miss more than half of the faulty ones. On injected runs, the agents perform similarly, with pooled F1 scores of 0.80 and 0.82 compared to 0.84 for RAGLog. When an injected fault took effect, RAGLog flagged the correct stage in 90% of the runs, compared to 80% and 87% for the agents. RAGLog also generalizes to previously unseen pipelines, outperforming both agents on natural runs, and automatically transforming logs of new pipelines into the common schema. Furthermore, RAGLog achieves these results at a fraction of Codex’s cost and at a lower overall cost than Claude Code.

The main contributions of this work include:

• Reference-free, cross-stage fault localization. RAGLog identifies faulty stages of a RAG pipeline using execution logs alone, without requiring reference answers or code instrumentation. By combining evidence across stages, it can detect faults that propagate to downstream stages or occur even when the final answer is correct. This is designed to support debugging failing runs and monitoring pipelines in production, where reference answers are often unavailable.

• Infrastructure for observing, debugging, and evaluating RAG pipelines. We provide a logging proxy, a RAG schema, and an automated mechanism for transforming execution logs from different providers into this schema without any manual configuration. We also introduce a live fault injector that modifies the input or output of a pipeline stage during execution, allowing later stages to respond to the injected fault as they would to a real fault. Together, these components enable developers to trace pipeline execution, evaluate how faults affect downstream stages, and assess changes in pipeline behavior, without modifying application code.

• Empirical evidence on effectiveness, generalization, and cost. We evaluate RAGLog across five open-source RAG pipelines and three public datasets, using 1,500 natural runs and 480 injected runs, and compare it against two general-purpose coding agents. The evaluation examines fault-localization effectiveness, generalization to unseen pipelines, the contribution of individual components, and analysis cost and time. The results also reveal that many injected faults are absorbed by later stages and that faulty stages do not necessarily lead to incorrect final answers.

• Dataset and tool for debugging, evaluating, and monitoring RAG pipelines. We release RAGLog Faults, a public dataset of 3,134 runs across five RAG pipelines, with stage-level fault labels, execution traces, and labeling judgments. We also provide an open-source RAGLog tool with a dashboard for debugging, evaluating, and monitoring RAG pipelines.

The remainder of this paper is structured as follows: § 2 presents a motivating example and introduces the stages of a RAG pipeline. § 3 explains our approach, and § 4 describes the dataset. § 5 describes the experimental setup, § 6 summarizes the experimental results, and § 7 presents the RAGLog tool. § 8 discusses the implications of the findings, and § 9 discusses the threats to validity. Related work is reviewed in § 10, followed by conclusions and directions for future work in § 11.

Motivating Example

Consider Alex, an AI engineer tasked with developing a question-answering chatbot for employees at a large enterprise. Alex decides to build a RAG pipeline since the questions will be based on a corpus of proprietary documents, and LLMs cannot correctly answer domain-specific questions based on their training data alone. The pipeline will take the user query as input, find query-relevant document chunks from the corpus, and produce a response based on the query and these chunks. To achieve this, Alex’s pipeline will consist of the following stages: 1) a rewriter that reformulates the user query to improve retrieval, 2) an embedder that converts the query into a vector, 3) a retriever that retrieves the chunks most similar to this vector from a vector database, 4) a reranker that ranks the retrieved chunks by their relevance to the query and passes the top-ranked ones on, and 5) a generator that produces the response from the query and the passed chunks. The rewriter and reranker are optional, and not every pipeline uses them. Alex builds the RAG pipeline shown in Figure 1 by leveraging third-party APIs for all stages, as APIs scale well in production and do not require additional infrastructure, development, or maintenance overhead.

To validate the pipeline, Alex asks several questions; while some return correct results, many others produce partially or completely incorrect responses. Alex realizes that the pipeline has several potentially faulty stages. We consider a stage faulty when its output deviates from the expected behavior for a given input, such as when the retriever fails to retrieve chunks containing the information needed to answer a question or when the generator produces claims that are not supported by the retrieved context. A fault in one stage may propagate downstream, so the stage where a wrong answer becomes visible is often not the stage that caused it. For example, an employee asks “How many days of parental leave are new employees entitled to?”, and the chatbot replies that the documents do not contain this information. Alex suspects the retriever, but the logs show that the generator received a chunk of the leave policy stating the number of days and ignored it; the fault lies in the generator, not in retrieval. In another case, an employee asks “What does GDPR stand for?”, and the chatbot answers correctly. However, the retriever did not return the section of the document that defines the term, and the generator answered from its own training data instead. The answer is correct, but the retriever’s fault remains hidden, and it will affect any question whose answer the generator cannot know, such as questions about the company’s own procedures. Locating such faults requires inspecting every stage of every run, which Alex can only do by reading the logs of each API call, a laborious and error-prone process.

The absence of reference answers further complicates fault localization. Before deploying the chatbot, Alex wants to evaluate it on questions from the company documents. This requires domain experts to write reference answers, which is expensive and must be repeated whenever the documents change [66]. Although synthetic test sets can reduce this cost [20], they still require validation from domain experts. Even if such domain-specific evaluation is conducted, in production the chatbot will answer real employee queries, for which reference answers will not be available. Alex’s RAG pipeline is built using third-party APIs, so internal instrumentation is also not possible. As a result, the only way she can learn about production faults is through log inspection or user reports of incorrect answers.

Therefore, she starts exploring existing solutions that can locate faults in RAG pipelines, summarized in Table 1. RAG evaluation frameworks [20, 54, 55] score retrieval and generation quality, but do not cover the other stages, rely on reference answers for several of their metrics, and do not determine which stage is at fault. Test-generation approaches [58, 66] reduce the cost of reference answers by generating question-answer pairs from the documents, which reveal pipeline failures but do not identify the root cause. Recent diagnostic approaches go further and attribute faults to stages of the RAG pipeline, yet they all have certain limitations: RAGEC classifier [36] requires reference answers, RAG-Critic [19] distinguishes only retrieval from generation errors in responses it judges as incorrect, and Doctor-RAG [29] analyzes only agentic runs already known to have failed. Hallucination detectors such as MetaRAG [59] work without reference answers, but examine only the generator. Interactive debugging tools [14, 35] make the behavior of a pipeline visible, but leave its analysis to the developer. Observability platforms [5, 33, 34] trace every stage of a pipeline and offer automated evaluators for retrieval and generation quality, but they score stages individually rather than determining which stage caused a fault, and they usually require the pipeline’s code to be instrumented. Many of these approaches rely on large LLM judges or several LLM calls per answer, which makes analyzing all pipeline runs costly.

Approach	All RAG stages	Locates the faulty stage	No reference answers needed	No known failure needed	No code changes needed	Fully Automated
RAGAS [20]	✗	✗	◐	✓	✗	✓
ARES [55]	✗	✗	◐	✓	✗	✓
RAGChecker [54]	✗	◐	✗	✓	✗	✓
RAGEval [66]	✗	✗	◐	✓	✗	✓
RAGProbe [58]	✗	✗	◐	✓	✓	✓
RAGEC classifier [36]	◐	✓	✗	◐	✗	✓
RAG-Critic [19]	✗	◐	✓	✓	✗	✓
Doctor-RAG [29]	◐	✓	✓	✗	✗	✓
MetaRAG [59]	✗	✗	✓	✓	◐	✓
Raggy [35]	◐	✗	✓	✓	✗	✗
RAGTrace [14]	◐	✗	✓	✓	✗	✗
Langfuse [34]	✓	✗	✓	✓	◐	◐
LangSmith [33]	✓	✗	✓	✓	◐	◐
Arize Phoenix [5]	✓	✗	✓	✓	◐	◐

This motivating example highlights the following features that Alex requires for locating faults in her RAG pipeline:

• Automated fault localization across all stages of the RAG pipeline, including faults that do not change the final answer.

• Independence from reference answers, enabling pipeline evaluation on domain-specific questions and monitoring in production environments.

• Integration with RAG pipelines without any changes to the source code.

• Low cost and resource usage, enabling every run to be analyzed in production.

Approach

To address the challenges described in § 2, we present RAGLog, a reference-free fault localization approach for RAG pipelines. RAGLog takes the execution logs of a RAG pipeline as input and decides, for every stage of every run, whether that stage is faulty. Throughout this paper, fault localization covers both detecting that a run contains a fault and identifying the stage that is faulty. Figure 2 shows an overview of RAGLog, which consists of three components. First, each logged API call is classified into the RAG stage defined in our schema. Second, the classified calls are transformed into instances of the RAG schema, one instance per run, using a mapping that is generated once for each pipeline. Third, stage checks based on an LLM, an embedding model, and rules measure each stage in the schema instance, and an ensemble decision of stage-specific fault classifiers turns these measurements into the fault locations of the run. We first describe the RAG schema and then each of the components in detail below.

RAG Schema

The RAG schema serves as a template that captures the architecture of a typical RAG pipeline, including its stages and the inputs and outputs of each stage. The schema is populated using execution logs, with each instance representing a single run including all API calls made by the pipeline to answer one question. These populated instances are later used for fault localization. We include five stages, shown in Table 2: 1) rewriter, 2) embedder, 3) retriever, 4) reranker, and 5) generator:

Stage	Inputs	Outputs
Rewriter (optional)	System prompt & user query	Refined user query
Embedder	User query	Query embedding
Retriever	Query embedding	Retrieved chunks
Reranker (optional)	Query & retrieved chunks	Reranked chunks
Generator	System prompt, user query & retrieved chunks	Query response

• Rewriter: reformulates the user query to improve its effectiveness in subsequent stages. It takes a system prompt describing the task and the user query, and produces a refined query that is optimized for downstream stages. The rewriter is optional.

• Embedder: converts the user query into a semantic vector representation for retrieval from a vector database. It takes the user query, or the refined query when a rewriter is present, and produces the corresponding query embedding.

• Retriever: performs a similarity search over the vector database using the query embedding produced by the embedder. It returns the retrieved chunks, a ranked set of document chunks that may contain query-relevant information.

• Reranker: refines the ordering of the retrieved chunks by estimating their relevance to the query. It takes the query together with the retrieved chunks and produces the reranked chunks, from which the most relevant ones are passed on to the generator. This stage is also optional.

• Generator: uses an LLM to answer the user query based on the retrieved chunks. It takes a system prompt containing generation instructions, the user query, and the retrieved chunks (or reranked chunks, if a reranker is present), and produces the final response.

API Call Classification

This component classifies raw API call logs according to RAG pipeline stages defined in our schema (§ 3.1). Since raw logs do not explicitly identify the stage associated with each API call, they cannot be used directly for fault localization. We first pair corresponding request and response logs using API call IDs, and then classify each pair with a hybrid strategy that combines efficient rule-based classification for known APIs with an LLM fallback for unseen ones. The classification also filters out irrelevant API calls, such as health checks. An example is shown in Listing 1.

Listing 1. A snippet of a reranker call logged before and after classification

{"id": "4f2c91",
  "method": "POST",
  "url": "https://api.cohere.com/v2/rerank",
  "model_name": "rerank-english-v3.0",
  "raw_request": {"query": "parental leave days new employees",
                 "documents": ["Annual leave accrues ...",
                               "New employees are entitled to 90 days ...", ...]},
  "raw_response": {"results": [{"index": 1, "relevance_score": 0.93},
                               {"index": 0, "relevance_score": 0.41}, ...]}}

// The same call after classification
{"id": "4f2c91", ...,
  "stage": "reranker",
  "classifier_used": "rule",
  "classification_reasoning": "RULE: URL matches: /rerank"}

Rule-based Classification

Rule-based Classification. The logs are first processed by a rule-based classifier that classifies each request-response pair as one of the five stages defined in our schema. The classification is performed by identifying patterns such as keywords, URLs, model names, and request and response formats. For instance, we detect embedder calls by examining the model name and URL for embedding-related keywords such as ’embedding’, ’embed’, ’encode’, and ’text-embedding’. The classifier also considers the position and content of each call within a run to distinguish between similar stages, such as the rewriter and generator, which typically use the same LLM endpoint. While the rule-based classifier can detect most stages, it is not robust to API changes or unseen patterns. When a request-response pair does not match any predefined rule, classification falls back to the LLM-based classifier.

3.2.2 LLM-based Classification. The LLM-based classifier sequentially performs binary classification on the request-response pair for each of the five stages. Each prompt provided to the LLM describes the characteristics of the stage, and contains the URL, model name, and request-response pair. The prompt also includes positive and negative examples to improve classification performance. Since the order of calls carries information, the prompt also contains the stage of the previous call. Once the LLM identifies a stage, the remaining stages are not checked. If a request–response pair matches no stage in either classifier, it is marked as irrelevant. We use Claude Haiku 5.5 for API call classification.

Log-to-Schema Transformation

This component transforms the classified call logs from the previous step into instances of the RAG schema, providing a structured format that can later be used for fault localization. A mapping generator identifies how fields in each call map to fields in the RAG schema, then uses this mapping to transform the calls into schema instances. The mapping is created only once for a given RAG pipeline; subsequent calls are transformed into schema instances using the existing mappings. Similar to API call classification, we use a hybrid strategy for mapping generation with a rule-based generation followed by an LLM fallback. Listing 2 shows an example of rule-based mapping generation.

Listing 2. Mapping generation for the call in Listing 1: a predefined rule matches the call and becomes the mapping for the reranker

// Predefined rule for Cohere's rerank format
{"stage": "reranker",
  "inputs": [{"input_key": "query",      "target": "userQuery"},
             {"input_key": "documents",  "target": "retrievedChunks"}],
  "outputs": [{"output_key": "results[*]", "target": "rerankedChunks"}]}
// query, documents and results exist in the call: the rule matches


// Mapping stored for this RAG pipeline
{"stage": "reranker",
  "inputs": {"userQuery": "<query>",
             "retrievedChunks": ["<documents>"]},
  "outputs": {"rerankedChunks": "<results[*]>"}}

3.3.1 Rule-based Mapping Generation. A mapping describes which fields of the request and response correspond to the inputs and outputs of that stage in the schema. For example, the mapping of a reranker call maps the query and documents fields of the request to the query and retrieved chunks of the schema, and the results field of the response to the reranked chunks in the schema. For each stage in the schema, we define several such rules to support different API providers. If a mapping does not exist for a RAG pipeline, the rule-based mapping generator creates one by matching the calls from each stage against predefined rules for common API formats. For example, a call classified as the embedder stage would be matched with mapping rules for common embedding providers such as OpenAI, Nomic AI, Cohere, and BGE. The rules are extensive enough to cover most log formats; however, if a match is not found, the mapping generation is handled by the LLM-based mapping generator.

LLM-based Mapping Generation

We prompt an LLM to generate mappings between a classified call and its corresponding schema subset. The prompt contains the request and response of the call, the schema fields of the stage, a description of the mapping task, and positive and negative examples. The LLM returns, for every schema field, the path of the corresponding field in the call. The mapping is stored as a JSON file for each pipeline. We use Claude Haiku 5.5 for mapping generation.

Log Transformation

This step uses the created mappings to transform classified call logs into schema instances. A JSON-to-JSON mapper applies the mapping to each call, extracts the values at the mapped paths, and populates the corresponding fields of the schema instance. An example is shown in Listing 3. RAGLog then groups the mapped calls into individual runs, each representing a single question and ending with the generator call that produces the answer. Each run forms a separate schema instance used for fault localization. To account for stages that may not generate API calls, RAGLog reconstructs missing information where possible. For example, some pipelines use an in-process vector database for retrieval, leaving no retriever API call in the logs. In such cases, the retriever’s output is reconstructed from the reranker’s input, which contains the retrieved chunks in their original retrieval order, allowing the retriever to be assessed even without an API call.

Listing 3. RAG schema instance for the reranker stage generated by transforming the API call in Listing 1 using the mapping in Listing 2

{"stage": "reranker",
 "inputs": {"userQuery": "parental leave days new employees",
            "retrievedChunks": ["Annual leave accrues ...",
                                "New employees are entitled to 90 days ...", ...]},
 "outputs": {"rerankedChunks": [{"index": 1, "relevance_score": 0.93},
                                {"index": 0, "relevance_score": 0.41}, ...]}}

Fault Localization

This component locates faults in a RAG pipeline using a populated instance of the RAG schema. It determines whether each stage in the instance is faulty by examining its inputs and outputs, as well as those of other stages. Without reference answers, this requires examining how information flows through the pipeline. For example, whether the retrieved chunks contain the information needed to answer the question, whether the reranker preserves that information, and whether the generator uses it to produce the answer. A capable reasoning model such as Opus 5.5 [4] could perform this analysis, but its cost makes it impractical for analyzing every run in production. Smaller LLMs, such as GPT-5 Mini [47], are less reliable when asked to identify faulty stages directly. Similarly, embedding similarities and rules are computationally inexpensive but can only detect certain types of faults. RAGLog therefore divides fault localization into two parts. First, three types of checks assess specific properties of each stage: rule-based checks, embedding-based checks, and LLM-based checks. Second, an ensemble decision combines the measurements collected across the entire run to determine which stage is faulty using a fault classifier trained on labeled runs for each stage.

Stage Checks

Each stage check produces one or more numeric measurements, such as a similarity, a count, or a confidence. Table 3 lists the checks and the stages they measure.

Check	Stage	What it measures	Output
Rule-based
Rewriter rules	Rewriter	Question’s names, numbers, and dates missing from every rewrite; names and numbers added by the rewrites; content words lost; reversed intent; changed question type; number of rewrites	Shares, counts, and flags
Embedder rules	Embedder	Question’s names missing from the embedder input; misspelled words; added words; length relative to the question; corrupted text	Shares, counts, ratio, and flag
Embedding-based
Query drift	Rewriter	Similarity of each rewrite to the question	Similarities
Query drift	Embedder	Similarity of the embedder input to the question and its rewrites	Similarities
Retrieval coverage	Retriever	Similarity of the best retrieved chunk to the question; rank of the most similar chunk; agreement of the retrieval order with the similarity order	Similarity, relative rank, rank correlation
Reranking order	Reranker	Agreement of the reranked order with the similarity order; position of the most similar chunk; similarity gap between the top-ranked and the most similar chunk	Rank correlation, relative position, similarity gap
Answer grounding	Generator	Similarity of the answer to its context; answer given with an empty context; refusal despite a relevant chunk	Similarities and flag
LLM-based
Graded stage check	Rewriter	Three fault types: a rewrite drops a constraint, adds a constraint, or changes an entity	Confidence (0–100) per fault type
Graded stage check	Embedder	One fault type: the embedder input is not a faithful rendering of the question	Confidence (0–100)
Graded stage check	Retriever	Three fault types: no retrieved chunk answers the question, the chunks cover only part of it, or most chunks are irrelevant	Confidence (0–100) per fault type
Graded stage check	Reranker	Two fault types: a good chunk was excluded, or weak chunks were passed on	Confidence (0–100) per fault type
Graded generator check	Generator	Six fault types: the answer is unsupported, off the question, incomplete, should have abstained, wrongly refuses, or misstates a number	Confidence (0–100) per fault type
Generator verification	Generator	Six fault types defined by explicit rules, such as information about a different entity or a missing item that the question asks for, and any fault	Confidence (0–100) per fault type
Evidence flow check	Retriever	Pieces of information the question needs that no passed or excluded chunk contains, or contains only partly	Yes, partly, or no per piece; shares of pieces
Evidence flow check	Reranker	Needed pieces that only chunks excluded by the reranker contain	Yes, partly, or no per piece; shares of pieces
Evidence flow check	Generator	Needed pieces that were passed on but that the answer omits, states differently, or says are unknown; the answer’s unsupported statements; refusal	Answer treatment per piece; shares and counts
Statement-level generator check	Generator	Statements of the answer that its context does not state, classified by materiality; omitted asked items, wrong refusal, answer to a different question	Count per category; confidence (0–100) per fault type

Together, these checks target the fault types identified in the RAG error taxonomy RAGEC [36] that are observable in RAG pipeline execution logs. These faults are also documented in other studies, including: failures to retrieve relevant chunks or the retrieval of irrelevant chunks [9, 19, 29, 35, 54]; relevant chunks being ranked too low to reach the generator or being obscured by irrelevant chunks [9, 16, 35, 40]; generated answers containing claims unsupported by or inconsistent with the retrieved context [19, 44, 54, 59]; and answers that are incomplete, fail to address the question, or disregard the document context, such as answering when the available context is insufficient and the system should abstain [9, 13, 19, 30, 35]. In addition, the checks target two fault types not covered by RAGEC: query rewrites that alter the original question [29, 38, 42] and embeddings that fail to faithfully represent the query text. Some checks compare a stage’s output with the original question, so they can also fire when an earlier stage caused the problem; the fault classifiers combine the checks of all stages to decide which stage is responsible.

Rule-based checks. The rule-based checks assess the rewriter and embedder, where faults often involve individual words, such as a missing year or a misspelled name, that embedding similarities may fail to detect. For the rewriter, the checks extract names, numbers, and dates from the original question and measure how many are missing from the rewrites, as well as any names or numbers introduced by the rewrites. The checks also identify changes in intent, such as an added negation, and changes in question type, such as a how question becoming a what question. For the embedder, the checks compare the embedder input with the original question and its rewrites to identify missing names, words differing by one or two edits from the original, added words, changes in relative text length, and corrupted text.

Embedding-based checks. The embedding-based checks use a small local embedding model (all-MiniLM-L6-v2 [53] [56]) to measure semantic similarities across pipeline stages. For the rewriter and embedder, they measure how much the rewrites and embedder input deviate from the original question. For the retriever and reranker, they assess the similarity between the question and the retrieved chunks, the rank of the most similar chunk, and how closely the retrieval and reranking orders align with question–chunk similarity, using Spearman’s rank correlation. For the generator, the checks measure the similarity between the generated answer and its passed document chunks, detect whether an answer was produced with an empty context, and identify cases where the generator refuses to answer despite having a relevant chunk in its context.

LLM-based checks. Many faults can only be recognized by understanding the question, the chunks, and the answer. For example, a retrieved chunk may refer to a different person with the same name, or an answer may introduce a date that is not supported by any chunk. RAGLog uses five targeted LLM-based checks to identify such faults. Each check asks a small LLM (by default Claude Haiku 5.5; § 5.2 evaluates variants of RAGLog with two other LLMs) to assess specific aspects of a run using only the available execution data, without relying on external knowledge or reference answers. The LLM returns its findings in a predefined JSON format. Most checks request confidence scores between 0 and 100 rather than binary judgments, allowing the ensemble decision mechanism to account for uncertainty.

• The graded stage check examines the user query, rewritten query, embedder input, retrieved chunks in their final order, and the chunks passed to the generator. It requests confidence scores for nine fault types associated with the rewriter, embedder, retriever, and reranker stages, as listed in Table 3. Stages that are absent from the pipeline are explicitly marked as not present.

• The graded generator check examines the user query, the chunks passed to the generator, and the generated answer. It requests confidence scores for six generator fault types, including unsupported claims and incorrect refusals.

• The generator verification assesses similar generator fault types but provides explicit criteria for each. For example, leaving out optional details does not make an answer incomplete, while information concerning a different entity, case, or time period cannot be used to support the answer.

• The evidence flow check examines how information passes through the retriever, reranker, and generator. The LLM first identifies the individual pieces of information needed to answer the question. For each piece, it determines whether the information appears in the chunks passed to the generator or in retrieved chunks excluded by the reranker, using yes, partly, or no judgments. It also determines whether the generated answer includes the information, omits it, states something different, or indicates that it is unknown. RAGLog uses these judgments to derive stage-specific measurements, such as the proportion of required information missing from all retrieved chunks. Information absent from the retrieved chunks indicates a potential retriever fault; information present only in excluded chunks indicates a potential reranker fault; and information passed to the generator but omitted or misstated in the answer indicates a potential generator fault. When many chunks are retrieved, the check includes all chunks passed to the generator and a shortlist of excluded chunks most relevant to the question, selected using a local cross-encoder (ms-marco-MiniLM-L-6-v2 [53, 57]).

• The statement-level generator check identifies statements in the generated answer that are not supported by the provided chunks and classifies them into categories, such as contradictions, unsupported answers to the requested facts, added names or numbers, and harmless elaborations or hedged remarks. Only the first three categories are considered faults, avoiding false alarms for paraphrasing or general explanations. RAGLog records the number of statements in each category. The LLM also gives a confidence for each of four fault types: the answer states a fact that the chunks do not support, omits an item that the question asks for, wrongly refuses, or answers a different question.

The five LLM-based checks run independently and are executed in parallel with the rule-based and embedding-based checks. As a result, the overall analysis latency is primarily determined by the slowest LLM call.

3.4.2 Ensemble Decision. The ensemble decision turns the stage check measurements of a run into a verdict for each stage. Using fixed rules for this process has two limitations. First, measurements vary in scale and reliability, and their importance may differ across stages. Second, evidence of a fault in one stage may appear in the inputs or outputs of another stage. Hence, RAGLog uses a separate fault classifier for each stage, which uses measurements from all stages of the run to estimate the probability that the stage is faulty.

All fault classifiers combine three types of models: logistic regression, which provides a simple regularized model when relatively few faulty examples are available; one gradient-boosted tree model [31], which captures interactions between measurements; and an extremely randomized trees model [21], whose randomized splits give smoother, lower-variance probabilities. Their predictions are combined using a weighted average to estimate the probability that stage $s$ is faulty given the measurements $x$:

where logistic regression contributes one third of the prediction, the gradient-boosted model contributes one half, and extremely randomized trees contribute one sixth. The ensemble weights were fixed during development, and alternatives, including equal weights, changed the pooled F1 score by no more than approximately 0.03.

To account for the lower frequency of faulty stages, all models use class weights inversely proportional to class frequency. Missing measurements are handled natively by the gradient-boosted models and replaced with median values for the other models. To limit overfitting given the relatively small number of faulty runs per stage in the training data, we use regularized logistic regression ($C = 0.5$), shallow trees (maximum depth of three, 100 iterations, a learning rate of 0.05, and at least ten runs per leaf) and 300 extremely randomized trees with at least three runs per leaf.

A stage is classified as faulty when its predicted probability $p_s(x)$ reaches a stage-specific threshold $\tau_s$. To limit false alarms, we select the threshold that maximizes the F1 score for each stage while keeping its false-positive rate at or below 10%. Thresholds are selected using out-of-fold predictions rather than predictions on the data used to train the models. Specifically, the training runs are divided into three folds across three different partitions, with all runs associated with the same question kept in the same fold. Each stage is assessed only in runs where that stage is present. Finally, RAGLog clears a retriever flag when the evidence flow check finds the needed information only in chunks that the reranker excluded, attributing the fault to the reranker instead.

The stage-specific fault classifiers and their thresholds are trained once, offline, using runs with stage-level fault labels. Since all measurements are derived from the RAG schema instance, RAGLog can be applied to new pipelines without additional labeling or training. In this study, we train the classifiers using the development runs of the RAGLog Faults dataset (§ 4). For each stage in a run, RAGLog reports its fault classification, predicted probability $p_s(x)$, and plain-language explanations derived from the LLM-based checks.

The RAGLog Faults Dataset

Evaluating fault localization in RAG pipelines requires runs of real pipelines in which every stage is labeled as faulty or not. Existing RAG benchmarks provide questions, documents, and reference answers, but no labels for the stages of a pipeline, and existing studies of RAG failures describe faults without releasing the runs in which they occurred [9, 35]. We therefore built RAGLog Faults, a dataset of 3,134 runs of five open-source RAG pipelines on three public question-answering datasets, in which every observable stage of every run is labeled. Each run contains the question, its reference answer and evidence, the answer of the pipeline, the full trace of the run, the stage-level fault labels with their fault types, and the judgments from which the labels were derived. The dataset contains natural runs with faults that the pipelines produced on their own, and injected runs, in which we introduced faults into individual stages while the pipelines were running. The runs are divided into development and test sets, with no questions shared between them. The development runs were used to develop and train RAGLog, while the test runs were used exclusively for evaluation. This section describes the dataset construction and characteristics in detail.

RAG Pipelines and Source Datasets

We use five open-source RAG pipelines: RAGFlow [27], LightRAG [25], Quivr [52], Onyx [46], and Verba [62]. These pipelines were selected based on three criteria: community adoption as measured by GitHub stars, architectural diversity covering different combinations of stages and models, and support for container deployment with Docker [17]. Table 4 lists the stages of each pipeline and the models used. Two pipelines perform query rewriting: Quivr refines the user query, while Onyx generates multiple keyword queries. Three pipelines, RAGFlow, LightRAG, and Quivr, rerank the retrieved chunks. All five pipelines embed the query, retrieve relevant chunks, and generate answers. However, RAGFlow and Quivr search their vector databases in-process, without making retriever API calls. For these two pipelines, the retrieved chunks are reconstructed from the input of the reranker (§ 3.3).

Pipeline	Rewriter	Embedder	Retriever	Reranker	Generator
RAGFlow	–	Cohere embed-english-v3.0	in-process*	Cohere rerank-english-v3.0	GPT-4.1 nano
LightRAG	–	BGE-M3	Qdrant	Jina reranker v2	GPT-4o
Quivr	GPT-4o	OpenAI text-embedding-ada-002	in-process*	Cohere rerank-multilingual-v3.0	GPT-4o
Onyx	Gemini 3.5 Flash	Nomic embed-text-v1	Vespa	–	Gemini 3.5 Flash
Verba	–	Nomic embed-text	Weaviate	–	Llama 3

To make the pipelines comparable, we harmonized their settings where their design allowed: chunks of about 512 tokens with an overlap of 50 tokens, six chunks passed to the generator, up to 64 candidates for the reranker, and no chat history between questions.

We use three public question-answering datasets with documents, questions, reference answers, and their evidence from different domains. DragonBall, part of the RAGEval benchmark [66], contains generated documents from finance, law, and medical domains with questions of several types, such as factual, multi-hop, and summary questions. Natural Questions (NQ) [32] contains real questions asked on the Google search engine, each paired with a Wikipedia article. MS MARCO [8] contains real questions asked on the Bing search engine with web passages. For DragonBall, we use its entire English corpus of 108 documents, over which all of its questions are asked. For NQ and MS MARCO, we built a corpus of 200 documents each that contains the documents referenced by the sampled questions. From all three datasets, we sampled 150 questions, 50 for development and 100 for testing, with no overlap between the two; the DragonBall test questions are stratified by question type and domain.

Collecting Execution Logs

To collect execution logs without changing the code of the RAG pipelines, we built a logging proxy based on mitmproxy [43], an interactive man-in-the-middle proxy that can record and modify HTTP and HTTPS traffic. The proxy is added as a service to the Docker Compose [18] file of a pipeline, and the pipeline sends its outbound requests through it by setting the standard proxy environment variables and trusting the proxy’s certificate. For every API call, the proxy records the URL, the HTTP method, the request and response content, the status code, and timestamps. A second instance of the proxy runs in front of the pipeline’s own API and records each question that the pipeline receives and the answer it returns to the user. These records mark where the run of each question begins and ends, and record the original question and the final answer. Verba queries its Weaviate vector database over gRPC [23], which mitmproxy cannot forward, so we log these calls with a small pass-through gRPC logger in front of Weaviate. The proxy forwards every call unchanged, except when faults are injected (§ 4.3), and it sends each call to RAGLog, which classifies it, transforms it into the RAG schema, and assembles the runs (§ 3.2 and § 3.3).

For each pipeline and dataset, a launcher sends the questions one at a time through the proxy and checks the completed session: every question has exactly one run with an answer from a generator call, the expected stages are logged, and the logged settings match the harmonized settings. Sessions that failed this check were repeated before any run was labeled.

Fault Injection

Natural faults are spread unevenly over the stages; in our test runs, embedder faults are rare, and rewriter and reranker faults occur only in the pipelines that have these stages. To evaluate fault localization for every stage with a known fault, and to study how faults propagate, we built a fault injector that introduces a fault into a single stage of a running pipeline.

Live injection

The injector works through the logging proxy. In fault mode, the proxy passes each call to RAGLog before forwarding it. RAGLog classifies the call, maps it to its schema component, and if the call belongs to the stage selected for injection, it changes the input or output of that component. The change is then written back into the request or response in the API’s own format, and the proxy forwards the modified call. Since the fault is introduced while the pipeline runs, every later stage receives the faulty input and reacts to it as it would in practice, for example by retrieving other chunks after a faulty rewrite or by answering from a context that lacks the evidence. Faults can only be injected into stages whose calls pass through the proxy; we therefore do not inject retriever faults into RAGFlow and Quivr, whose retrieval runs in-process, or into Verba, whose retriever is logged by the separate gRPC logger. Reranker faults are injected only into RAGFlow and Quivr, since LightRAG’s reranker passes on every chunk it ranks.

Stage	Fault types	Pipelines	Target
Rewriter	A wrong year is added; a date or qualifier is dropped; the entity is replaced by a similar one	Quivr, Onyx	Response
Embedder	The query is cut to its first half; the key entity is misspelled	All five	Request
Retriever	The evidence chunks are not returned	LightRAG, Onyx	Response
Reranker	The evidence is ranked below the cut-off; one of several evidence chunks is ranked below the cut-off	RAGFlow, Quivr	Response
Generator	The evidence is placed in the middle of the context among passages about similar entities	All five	Request

4.3.2 Fault types. Table 5 lists the injected fault types. We selected fault types that are reported in other RAG failure studies [9, 35, 38] and resemble the faults observed in our natural runs. Faults injected into the retriever, reranker, and generator use the question’s reference evidence to target the information needed to answer it. For example, a retriever fault removes chunks containing the required evidence from the retrieved set, while a reranker fault moves these chunks just below the cut-off, so they are retrieved but not passed to the generator. For the generator, the fault places the relevant evidence in the middle of the context, surrounded by passages about similar entities retrieved by the same pipeline for other questions [40]. The development runs also contain further fault types that we explored and did not use for evaluation because they rarely affected the answer.

4.3.3 Injected test set. The injected test set only uses questions on which the pipeline’s original run was fault-free. We aimed to include 30 runs for each of the 16 pipeline–stage pairs where faults could be injected, covering all three datasets and different fault types. To construct this set, we injected faults into 634 runs and subsequently labeled them using our judges. However, an injection does not always result in an observable fault, as the pipeline may absorb its effects. For example, a reranker may demote relevant evidence without excluding it from the generator’s context, or the generator may ignore distracting passages. The judges, who were unaware of whether or where faults had been injected, confirmed faults at the targeted stage in 345 of the 634 runs. From these labeled runs, we selected a final test set of 480 runs, with 30 runs per pipeline–stage pair, prioritizing runs in which the judges confirmed faults at the injected stage.

Labeling

We consider a stage faulty when it introduces a fault rather than just passing on an earlier one, regardless of its effect on the final answer. The retriever and reranker are assessed against the evidence needed for the original question, meaning that an earlier pipeline fault may also cause these stages to be labeled faulty; this happens in 4 natural and at most 32 injected test runs. Since labeling every stage of 3,134 runs by hand was not feasible, we used two LLM judges from different providers, GPT-5.1 and Claude Sonnet 5.5, to detect faulty stages. Each judge receives the question, reference answer and evidence, and the full execution trace, with chunks containing the reference evidence marked. For each stage, the judge records factual findings, such as which pieces of information needed to answer the question are present in the retrieved chunks and which statements in the generated answer are not supported by the context. The judge also identifies fault types based on the RAG error taxonomy RAGEC [36], which we extend with two additional fault types for the rewriter and embedder.

The stage labels are then determined from these findings using fixed rules. The rewriter is labeled faulty if it adds or removes a constraint from the original question, or changes its intent. The embedder is faulty if the text passed for embedding alters the query, for example, by omitting names or constraints, introducing misspellings, or adding unrelated text. The retriever is faulty if none of the retrieved chunks contains information needed to answer the question. The reranker is faulty if the required information was retrieved but excluded from the chunks passed to the generator. Finally, the generator is faulty if its answer contains unsupported claims, omits explicitly requested information, incorrectly refuses to answer, fails to abstain when the question is unanswerable, or answers a different question. Where both judges agree, their label is final. Where they disagree, a third model, Claude Opus 5.5 [4], adjudicates. Of the 1,500 natural test runs, 1,133 were labeled by agreement and 367 by adjudication. Of the 480 injected test runs, 299 were labeled by agreement and 181 by adjudication.

The judges’ labels serve as the ground truth for training and evaluating RAGLog, but they are never an input to it. RAGLog sees only the trace of a run, without the reference answer and evidence. For an injected run, the injected stage is labeled faulty only if the judges found a fault there; the injection itself records only where a fault was introduced.

Splits, Statistics, and Release

Table 6 summarizes the runs of RAGLog Faults. The development runs consist of 750 natural runs, one for each of the 50 development questions of each dataset on each pipeline, and 250 injected runs on DragonBall questions. The test runs consist of 1,500 natural runs, one for each of the 100 test questions of each dataset on each pipeline, and the 634 labeled injected runs, of which 480 form the final injected test set.

		Runs			Faulty stage / runs with the stage
		All	Faulty†	Correct*	Rewriter	Embedder	Retriever	Reranker	Generator
Development	Natural	750	264	544	36/300	4/750	118/750	25/450	141/750
Development	Injected	250	188	123	25/146	56/248	75/248	41/149	43/250
Test	Natural	1,500	551	1,143	42/600	10/1,500	227/1,500	62/900	329/1,500
Test	Injected	480	333	347	77/240	159/480	67/479	36/299	80/480

The natural test runs show how common hidden faults are: 551 of the 1,500 runs (37%) contain at least one faulty stage, and 202 of these runs still produce a correct answer. Faults are concentrated in the generator (329 runs) and the retriever (227 runs), and 252 of the 357 incorrect answers (71%) are caused by a stage before the generator. Embedder faults are rare (10 runs), which is one reason for the injected test set, in which each stage has the same number of injections per pipeline. Among the 634 injected runs that were labeled, almost all rewriter and embedder injections produced a fault at the injected stage, while many retriever, reranker, and generator injections were absorbed. To evaluate localization on known faults while keeping the stages balanced, the final injected test set contains 480 runs (30 runs per pipeline–stage pair), prioritizing runs in which the judges confirmed a fault at the injected stage.

We release RAGLog Faults on Hugging Face.³ It is organized into three subsets, one per source dataset, each distributed under the terms of its source dataset: CC BY-NC-SA 4.0 for DragonBall, CC BY-SA 4.0 for NQ (whose source is CC BY-SA 3.0), and the MS MARCO terms for non-commercial research. Our annotations are released under CC BY 4.0, and access requires agreeing to the terms of the source datasets. Each subset contains the runs with their labels and fault types, the traces, both judges’ findings and reasons, and a record of every injection with its fault type.

Experimental Setup

Research Questions

We answer the following research questions in our evaluation of RAGLog:

RQ1: How effective is RAGLog at detecting and localizing natural faults in RAG pipelines? This question evaluates RAGLog on naturally occurring faults and compares its effectiveness with two general-purpose coding agents, Claude Code and Codex, using the same execution logs.

RQ2: How effective is RAGLog at detecting and localizing injected faults in RAG pipelines? This question evaluates RAGLog on faults injected into individual pipeline stages and compares its effectiveness with the baseline agents. It also examines how injected faults manifest and propagate through the pipeline.

RQ3: How do the key components of RAGLog contribute to its effectiveness? This question examines the contribution of individual components and groups of components to RAGLog’s effectiveness.

RQ4: How efficient is RAGLog at detecting and localizing faults in RAG pipelines? This question compares RAGLog with the baseline agents in terms of API cost, token consumption, and analysis time.

RAGLog Variants

We evaluate three variants of RAGLog, each using a different LLM for the LLM-based checks: Claude Haiku 5.5 [3], GPT-5 Mini [47], and Gemini 3.1 Flash-Lite [22]. We denote each variant by its LLM, for example RAGLog$_{\mathrm{Haiku\ 5.5}}$. Claude Haiku 5.5 is RAGLog’s default LLM; where we report a single result for RAGLog without a subscript, it refers to RAGLog$_{\mathrm{Haiku\ 5.5}}$. We selected smaller LLMs to keep the cost and latency of fault localization low, making RAGLog suitable for use in production. For each variant, the fault classifiers and their thresholds were trained using the development runs of the RAGLog Faults dataset. The variants were then evaluated on both the natural and injected test sets. All local computations, including the rule-based checks, embedding-based checks, and fault classifiers, were performed on a laptop with an Apple M1 Pro processor and 16 GB of memory, using Python 3.13 and scikit-learn 1.8.

Baselines

To the best of our knowledge, no existing approach localizes faults across all stages of a RAG pipeline without reference answers. Existing evaluation frameworks either require reference answers or assess only the retriever and the generator, while observability platforms record execution traces but leave their analysis to the developer. The diagnostic approaches closest to ours also have similar limitations. RAG-Critic [19] provides a critic model that assesses generated responses and attributes faults only to retrieval or generation as a whole, without separating the stages before generation into the rewriter, embedder, retriever, and reranker. Furthermore, it reports faults only when it judges the final answer is incorrect, leaving faults in runs with correct answers undetected. The automatic fault classification of RAGEC [36] requires reference answers. Doctor-RAG [29] diagnoses agentic RAG runs already known to have failed, and MetaRAG [59] detects hallucinations in the generated answers. A developer without a dedicated tool would therefore most likely hand the logs to a general-purpose coding agent and ask it to find the faulty stages. We use two such agents as baselines: Claude Code [2] with Claude Haiku 5.5, the same model as RAGLog’s checks, and Codex [48] with GPT-5.5 at medium reasoning effort, a stronger model. Using the same LLM as one of RAGLog’s variants helps isolate the contribution of RAGLog’s design, while the stronger model tests whether greater reasoning capability can close the performance gap.

Each agent receives the raw proxy logs of a pipeline session containing runs from one dataset, along with a fixed task description. The description defines the five pipeline stages and the criteria for considering each stage faulty, using the same fault definitions as the judges. It also explains the log format and instructs the agent to provide a fault verdict for every stage present in each run. The agents receive no reference answers, access to source documents, or examples from our dataset, and the task description was not adjusted based on test results. Given the size of the logs, the agents write and execute their own scripts to separate the session into individual runs and extract relevant information. Each session is analyzed in a fresh, isolated working directory outside the RAGLog project, ensuring that the agent has access only to the logs and task description. For injected runs, the logs contain the API calls as observed by the pipeline after fault injection, without revealing whether or where a fault was injected. We evaluate the agents’ verdicts using the same ground-truth labels and scoring rules as RAGLog.

Evaluation Metrics

We evaluate fault-localization effectiveness at both the stage and run levels. At the stage level, each stage present in a run is treated as a binary decision: faulty or not faulty. We compare predictions against the ground-truth labels assigned by the judges and report precision, recall, F1 score, and false-positive rate (FPR) for each stage. Precision measures the proportion of flagged stages that are faulty, while recall measures the proportion of faulty stages correctly identified. F1 is the harmonic mean of precision and recall, and FPR measures the proportion of fault-free stage instances incorrectly flagged. To summarize effectiveness across stages, we report macro-F1 and pooled F1. Macro-F1 averages the F1 scores of the stages that have at least one fault in the evaluated runs, giving each stage equal weight. However, faults are unevenly distributed across stages. For example, the natural test runs contain only 10 embedder faults compared with 329 generator faults, making macro-F1 sensitive to a small number of predictions for rarely faulty stages. We therefore use pooled F1 as our primary metric, calculated by aggregating predictions across all stage instances rather than averaging stage-level scores.

To estimate how much pooled F1 depends on the particular test questions, we compute 95% bootstrap confidence intervals. We draw the test questions with replacement 2,000 times, keep all runs of each sampled question together, and recompute pooled F1 on each sample. To compare RAGLog with a baseline, we compute the interval of the difference between their scores on the same samples; a difference is significant when this interval does not include zero.

We additionally measure run-level detection to assess whether a system identifies faulty runs, regardless of the stage it flags. A run is considered faulty if the judges label at least one stage as faulty, and is flagged if the system predicts a fault in any stage. We report the same four metrics at this level. Along with fault localization, we evaluate the accuracy of RAGLog’s API classification and log-to-schema transformation. Classification accuracy is the proportion of calls assigned to the correct RAG stage, including those correctly identified as irrelevant. Since the rule-based classifier may leave some calls undecided, we also report its coverage (the proportion of calls classified) and accuracy on the classified calls. For log-to-schema transformation, accuracy is the proportion of stage calls whose traces contain the expected values, while coverage is the proportion successfully mapped by the transformation rules.

Finally, we evaluate efficiency through token usage, API cost, and execution time. For RAGLog, we record the number of LLM calls, input and output tokens, API cost based on providers’ list prices, and analysis time for each run. Since the five LLM calls execute in parallel with local checks, the total analysis time is determined by the slowest operation. For the agent baselines, we record tokens, cost, and wall-clock time for each session, then divide these totals by the number of questions in that session.

Evaluation Procedure

RQ1. We evaluate RAGLog and both agents on the 1,500 natural test runs in our dataset (§ 4.5) and compare their effectiveness per stage and per dataset. To assess how well RAGLog generalizes to pipelines it was not trained on, we additionally perform a leave-one-pipeline-out evaluation. For each pipeline, we train the fault classifiers on the development runs of the other four pipelines and apply them to the test runs of the held-out pipeline.

RQ2. We evaluate RAGLog and both agents on the final injected test set of 480 runs from our dataset (§ 4.5) and compare their effectiveness as in RQ1. To examine how injected faults manifest and propagate through the pipeline, we also analyze all 480 labeled injected runs. For each stage and fault type, we examine how often the judges confirmed a fault at the targeted stage, how often the fault was absorbed, and how often it propagated to later stages or affected the final answer.

RQ3. We evaluate the three components of RAGLog, corresponding to its three steps. Since fault localization depends on API call classification and log-to-schema transformation, these components cannot be removed and are instead evaluated directly on familiar and unfamiliar APIs. For fault localization, we assess the contribution of its individual parts through an ablation study.

API call classification. We use two sets of labeled API calls. The first, which we call familiar APIs, contains all recorded calls from the five evaluation pipelines (RAGFlow, LightRAG, Quivr, Onyx, and Verba). The second, unfamiliar APIs, contains calls to providers and endpoints not used by any of these pipelines, covering embedding services, vector databases, rerankers, and LLM APIs, as well as calls unrelated to RAG stages, such as ingestion, setup, title generation, and routing. These calls follow the providers’ documented API formats or were collected from live API requests to reflect realistic traffic. Each call is labeled with its corresponding stage or as irrelevant. For both sets, we evaluate three classification methods: rules alone, the LLM alone, and the hybrid approach, where the LLM handles calls not recognized by the rules. We measure classification accuracy for each method, the proportion of calls covered by the rules, and the number of LLM calls required by the combined approach.

Log-to-schema transformation. We use the same two sets of API calls. For both sets, we generate mappings for individual stage calls and compare the extracted values, including queries, retrieved chunks, rankings, context, and answers, against manually established reference values. We evaluate the transformation accuracy of three methods: rules alone, the LLM alone, and the hybrid approach, where the LLM generates mappings for calls not covered by the rules. We evaluate the LLM-based components of both API call classification and log-to-schema transformation using Claude Haiku 5.5 and GPT-4.1 mini.

Fault localization. We assess the contributions of the stage checks and ensemble decision through an ablation study, modifying one component at a time.

• Stage checks. We remove each group of checks in Table 3 (rule-based, embedding-based, and LLM-based), and then each of the five LLM-based checks individually.

• Ensemble decision. We examine four design choices: replacing the ensemble with each of its three model types individually, varying the model weights, restricting each stage classifier to measurements from its own stage to assess the benefit of cross-stage information, and removing the false-alarm cap, the 10% limit on the false-positive rate used to select decision thresholds.

For each configuration, we retrain the fault classifiers on the development runs and evaluate their effectiveness on the test runs.

RQ4. We compare RAGLog and both agents in terms of API cost, token usage, and analysis time per question across the natural and injected test runs.

Results

RQ1: How effective is RAGLog at detecting and localizing natural faults in RAG pipelines?

Tables 7 to 9 show the effectiveness of the three variants of RAGLog and the two agent baselines on a test set of 1,500 natural runs. We report precision, recall, and F1 score for each stage (Table 7), false-positive rates, macro-F1, and pooled F1 (Table 8), and run-level detection (Table 9). Overall, RAGLog$_{\text{Haiku 5.5}}$ achieves a pooled F1 score of 0.73, compared to 0.58 for Claude Code and 0.52 for Codex, and a macro-F1 score of 0.56, compared to 0.44 and 0.34, respectively. RAGLog also outperforms the agents with its other variants, achieving pooled F1 scores of 0.69 for RAGLog$_{\text{GPT-5 Mini}}$ and 0.67 for RAGLog$_{\text{Gemini 3.1 Flash-Lite}}$. These improvements are statistically significant, with 95% bootstrap confidence intervals for the pooled F1 difference of [0.10, 0.19] against Claude Code and [0.16, 0.25] against Codex for RAGLog$_{\text{Haiku 5.5}}$. The other two variants also significantly outperform both agents. Since RAGLog$_{\text{Haiku 5.5}}$ and Claude Code use the same LLM, the difference between their performance cannot be attributed to the underlying model alone, suggesting that RAGLog’s design contributes to its improved performance.

Dataset	System	Rewriter P	Rewriter R	Rewriter F1	Embedder P	Embedder R	Embedder F1	Retriever P	Retriever R	Retriever F1	Reranker P	Reranker R	Reranker F1	Generator P	Generator R	Generator F1
DragonBall	Claude CodeHaiku 5.5	1.00	0.07	0.13	0.00	0.00	0.00	0.91	0.84	0.87	0.83	0.62	0.71	0.87	0.49	0.63
	CodexGPT-5.5	0.05	0.07	0.06	0.00	0.00	0.00	0.88	0.84	0.86	0.82	0.57	0.68	0.87	0.41	0.56
	RAGLogGemini 3.1 Flash-Lite	0.57	0.93	0.70	0.25	0.11	0.15	0.75	0.91	0.83	0.88	0.55	0.68	0.66	0.66	0.66
	RAGLogGPT-5 Mini	0.54	0.93	0.68	0.25	0.11	0.15	0.77	0.90	0.83	0.66	0.78	0.71	0.77	0.68	0.72
	RAGLogHaiku 5.5	0.62	0.93	0.74	0.00	0.00	0.00	0.77	0.92	0.84	0.77	0.75	0.76	0.68	0.81	0.74
NQ	Claude CodeHaiku 5.5	1.00	0.29	0.45	0.00	0.00	0.00	0.93	0.37	0.53	0.86	0.40	0.55	0.63	0.41	0.50
	CodexGPT-5.5	0.00	0.00	0.00	0.00	0.00	0.00	0.61	0.57	0.59	0.75	0.20	0.32	0.73	0.19	0.30
	RAGLogGemini 3.1 Flash-Lite	0.27	0.35	0.31	0.00	0.00	0.00	0.66	0.71	0.68	0.69	0.60	0.64	0.68	0.60	0.63
	RAGLogGPT-5 Mini	0.42	0.65	0.51	0.00	0.00	0.00	0.58	0.63	0.60	0.47	0.60	0.53	0.77	0.54	0.63
	RAGLogHaiku 5.5	0.37	0.59	0.45	0.00	0.00	0.00	0.68	0.80	0.74	0.55	0.80	0.65	0.75	0.71	0.73
MS MARCO	Claude CodeHaiku 5.5	0.60	0.27	0.38	–	–	–	0.64	0.35	0.45	0.00	0.00	0.00	0.64	0.21	0.31
	CodexGPT-5.5	0.00	0.00	0.00	–	–	–	0.55	0.35	0.42	0.00	0.00	0.00	0.83	0.16	0.27
	RAGLogGemini 3.1 Flash-Lite	0.25	0.45	0.32	–	–	–	0.57	0.75	0.65	0.75	0.86	0.80	0.67	0.57	0.62
	RAGLogGPT-5 Mini	0.22	0.36	0.28	–	–	–	0.58	0.75	0.66	0.40	0.86	0.55	0.81	0.59	0.68
	RAGLogHaiku 5.5	0.31	0.82	0.45	–	–	–	0.61	0.77	0.68	0.55	0.86	0.67	0.75	0.73	0.74
All	Claude CodeHaiku 5.5	0.82	0.21	0.34	0.00	0.00	0.00	0.87	0.65	0.75	0.84	0.50	0.63	0.72	0.36	0.48
	CodexGPT-5.5	0.05	0.02	0.03	0.00	0.00	0.00	0.78	0.69	0.73	0.81	0.42	0.55	0.82	0.25	0.39
	RAGLogGemini 3.1 Flash-Lite	0.37	0.57	0.45	0.25	0.10	0.14	0.70	0.85	0.76	0.80	0.60	0.69	0.67	0.61	0.64
	RAGLogGPT-5 Mini	0.41	0.67	0.51	0.25	0.10	0.14	0.70	0.82	0.76	0.57	0.74	0.64	0.78	0.60	0.68
	RAGLogHaiku 5.5	0.42	0.76	0.54	0.00	0.00	0.00	0.72	0.87	0.79	0.67	0.77	0.72	0.72	0.75	0.74

Dataset	System	Rewriter	Embedder	Retriever	Reranker	Generator	Pooled	Macro	Pooled
DragonBall	Claude Code$_{\mathrm{Haiku\ 5.5}}$	0.0	0.0	3.1	1.9	2.1	1.4	0.47	0.74
	Codex$_{\mathrm{GPT-5.5}}$	9.7	0.0	4.4	1.9	1.8	2.7	0.43	0.68
	RAGLog$_{\mathrm{Gemini\ 3.1\ Flash-Lite}}$	5.4	0.6	11.7	1.2	9.8	5.7	0.60	0.73
	RAGLog$_{\mathrm{GPT-5\ Mini}}$	5.9	0.6	10.3	6.2	5.7	5.3	0.62	0.76
	RAGLog$_{\mathrm{Haiku\ 5.5}}$	4.3	0.8	10.6	3.5	10.8	6.0	0.62	0.77
NQ	Claude Code$_{\mathrm{Haiku\ 5.5}}$	0.0	0.0	0.2	0.4	6.1	1.4	0.41	0.50
	Codex$_{\mathrm{GPT-5.5}}$	0.0	0.0	2.8	0.4	1.8	1.2	0.24	0.37
	RAGLog$_{\mathrm{Gemini\ 3.1\ Flash-Lite}}$	8.7	0.0	2.8	1.4	7.1	3.3	0.45	0.61
	RAGLog$_{\mathrm{GPT-5\ Mini}}$	8.2	0.0	3.4	3.5	4.1	3.1	0.45	0.60
	RAGLog$_{\mathrm{Haiku\ 5.5}}$	9.3	0.0	2.8	3.5	5.9	3.5	0.51	0.69
MS MARCO	Claude Code$_{\mathrm{Haiku\ 5.5}}$	1.1	0.0	2.2	0.0	3.7	1.4	0.28	0.35
	Codex$_{\mathrm{GPT-5.5}}$	0.0	0.0	3.3	0.0	1.1	1.1	0.17	0.30
	RAGLog$_{\mathrm{Gemini\ 3.1\ Flash-Lite}}$	7.9	0.0	6.5	0.7	9.1	4.4	0.60	0.61
	RAGLog$_{\mathrm{GPT-5\ Mini}}$	7.4	0.0	6.2	3.1	4.5	3.8	0.54	0.63
	RAGLog$_{\mathrm{Haiku\ 5.5}}$	10.6	0.2	5.8	1.7	7.7	4.5	0.63	0.69
All	Claude Code$_{\mathrm{Haiku\ 5.5}}$	0.4	0.0	1.7	0.7	4.0	1.4	0.44	0.58
	Codex$_{\mathrm{GPT-5.5}}$	3.2	0.0	3.5	0.7	1.6	1.6	0.34	0.52
	RAGLog$_{\mathrm{Gemini\ 3.1\ Flash-Lite}}$	7.3	0.2	6.6	1.1	8.6	4.5	0.54	0.67
	RAGLog$_{\mathrm{GPT-5\ Mini}}$	7.2	0.2	6.4	4.2	4.8	4.0	0.55	0.69
	RAGLog$_{\mathrm{Haiku\ 5.5}}$	8.1	0.3	6.0	2.9	8.1	4.6	0.56	0.73

System	DragonBall P	DragonBall R	DragonBall F1	DragonBall FPR	NQ P	NQ R	NQ F1	NQ FPR	MS MARCO P	MS MARCO R	MS MARCO F1	MS MARCO FPR	All P	All R	All F1	All FPR
Claude Code$_{\mathrm{Haiku\ 5.5}}$	0.93	0.71	0.80	5.2	0.74	0.45	0.56	6.4	0.78	0.33	0.47	4.4	0.85	0.53	0.66	5.4
CodexGPT-5.5	0.88	0.70	0.78	10.0	0.88	0.30	0.45	1.7	0.86	0.27	0.41	2.1	0.87	0.47	0.61	4.0
RAGLog$_{\mathrm{Gemini\ 3.1\ Flash-Lite}}$	0.84	0.87	0.85	16.5	0.70	0.67	0.68	11.1	0.70	0.70	0.70	14.1	0.77	0.77	0.77	13.6
RAGLog$_{\mathrm{GPT-5\ Mini}}$	0.85	0.87	0.86	15.7	0.76	0.64	0.69	7.8	0.79	0.72	0.75	9.1	0.81	0.77	0.79	10.3
RAGLog$_{\mathrm{Haiku\ 5.5}}$	0.83	0.90	0.87	18.1	0.74	0.76	0.75	10.6	0.74	0.81	0.77	13.5	0.78	0.84	0.81	13.6

The main difference between RAGLog and the agents is recall. The agents flag relatively few stages, but most of their predictions are correct. Claude Code achieves a pooled precision of 0.80 and Codex 0.76, with false-positive rates of 1.4% and 1.6%, respectively. However, their pooled recall is only 0.46 and 0.40 (Figure 3). In contrast, RAGLog$_{\text{Haiku 5.5}}$ detects 78% of faulty stages, with a precision of 0.68 and a false-positive rate of 4.6%. RAGLog therefore trades a higher false-alarm rate for substantially higher recall.

Stage-level effectiveness.

As shown in Table 7, the largest differences occur at the generator and rewriter stages. For the generator, the most frequently faulty stage (329 runs), RAGLog$_{\text{Haiku 5.5}}$ achieves an F1 score of 0.74, compared to 0.48 for Claude Code and 0.39 for Codex. The agents’ generator recall is relatively low, at 0.36 and 0.25, respectively. Generator faults, such as introducing unsupported facts or omitting information requested by the question, require comparing the generated answer against its context. RAGLog performs these checks explicitly for every run. For the

rewriter, RAGLog$_{\mathrm{Haiku\ 5.5}}$ achieves an F1 score of 0.54, compared to 0.34 for Claude Code and 0.03 for Codex, which rarely identifies faulty rewrites. Retriever performance is relatively similar across systems, with F1 scores ranging from 0.73 to 0.79. For the reranker, RAGLog achieves F1 scores of 0.64–0.72 across its variants, compared to 0.63 for Claude Code and 0.55 for Codex. Natural embedder faults are rare (10 runs) and difficult for all systems. Neither agent nor RAGLog$_{\mathrm{Haiku\ 5.5}}$ detects any, while RAGLog$_{\mathrm{GPT-5\ Mini}}$ and RAGLog$_{\mathrm{Gemini\ 3.1\ Flash-Lite}}$ each detect one (F1 score of 0.14).

Effectiveness across datasets

RAGLog’s advantage varies across datasets (Table 8). On DragonBall, which contains generated questions and documents, the gap is smallest: RAGLog$_{\mathrm{Haiku\ 5.5}}$ achieves a pooled F1 score of 0.77, compared to 0.74 for Claude Code, which performs better at retrieval. On NQ and MS MARCO, which contain real-world questions and documents, Claude Code’s pooled F1 drops to 0.50 and 0.35, respectively, while RAGLog$_{\mathrm{Haiku\ 5.5}}$ achieves 0.69 on both. One possible explanation is that RAGLog applies the same targeted stage checks across different document types, whereas the agents must independently extract and interpret relevant information from the logs.

Run-level detection

Table 9 shows that at run level, RAGLog$_{\mathrm{Haiku\ 5.5}}$ achieves an F1 score of 0.81, compared to 0.66 for Claude Code and 0.61 for Codex. It identifies at least one faulty stage in 84% of runs containing a fault, compared to 53% for Claude Code and 47% for Codex. This indicates that RAGLog is more effective at detecting whether a run contains a fault than at identifying every faulty stage correctly. However, its higher run-level recall comes with a false-positive rate of 13.6%, compared to 5.4% and 4.0% for the agents.

Effect of the LLM

Among the three LLMs, Claude Haiku 5.5 achieves the highest stage-level pooled F1 (0.73) and run-level F1 (0.81), respectively. This is followed by GPT-5 Mini at 0.69 stage-level and 0.79 run-level and Gemini

3.1 Flash-Lite at 0.67 stage-level and 0.77 run-level. Their differences are particularly visible at the generator stage. RAGLog$_{\mathrm{GPT-5\ Mini}}$ achieves the highest precision (0.78), but lower recall (0.60), while RAGLog$_{\mathrm{Haiku\ 5.5}}$ achieves a more balanced precision of 0.72 and recall of 0.75.

Generalization to unseen pipelines

To evaluate generalization to unseen pipelines, we train RAGLog’s fault classifiers on development runs from four pipelines and evaluate them on the test runs of the fifth. On pipelines that were not seen during training, the pooled F1 score of RAGLog$_{\mathrm{Haiku\ 5.5}}$ decreases from 0.73 to 0.67 on natural runs, while its false-positive rate remains similar (4.6% and 4.3%). However, its pooled F1 remains higher than those of Claude Code (0.58) and Codex (0.52). The other variants also experience decreases, reaching 0.61 for RAGLog$_{\mathrm{GPT-5\ Mini}}$ and 0.60 for RAGLog$_{\mathrm{Gemini\ 3.1\ Flash-Lite}}$, which is still higher than both agents. These results suggest that RAGLog can generalize to pipelines not represented in its training data, although its effectiveness decreases. RAGLog can be trained on data from more pipelines in the future.

Answer to RQ1: On natural runs, RAGLog$_{\mathrm{Haiku\ 5.5}}$ localizes faults more effectively than both agent baselines, with a pooled F1 score of 0.73 compared to 0.58 for Claude Code and 0.52 for Codex. Its advantage comes primarily from substantially higher recall, particularly at the generator and rewriter stages and on the real-world questions of NQ and MS MARCO, while the agents are competitive on DragonBall. Among the three LLMs, Claude Haiku 5.5 performs best overall, with the highest pooled F1 score. All three variants also perform better than both agents on pipelines excluded from their training data (0.60–0.67).

RQ2: How effective is RAGLog at detecting and localizing injected faults in RAG pipelines?

Table 10 to Table 12 show the effectiveness of RAGLog and the two agent baselines on a test set of 480 injected runs. We report stage-level precision, recall, and F1 score in Table 10, false-positive rates, macro-F1, pooled F1, and detection of confirmed injected faults in Table 11, and run-level detection in Table 12. In contrast to natural runs, the agents are competitive on injected runs. RAGLog$_{\mathrm{Haiku\ 5.5}}$ and RAGLog$_{\mathrm{GPT-5\ Mini}}$ achieve a pooled F1 score of 0.84, followed by Codex and RAGLog$_{\mathrm{Gemini\ 3.1\ Flash-Lite}}$ at 0.82 and Claude Code at 0.80. However, these differences are small: only RAGLog$_{\mathrm{Haiku\ 5.5}}$ and RAGLog$_{\mathrm{GPT-5\ Mini}}$ significantly outperform Claude Code, both by 0.04, while no RAGLog variant significantly outperforms Codex. For macro-F1, RAGLog$_{\mathrm{Haiku\ 5.5}}$ achieves the highest score of 0.81, compared to 0.77 for Codex and 0.74 for Claude Code.

Dataset	System	Rewriter P	Rewriter R	Rewriter F1	Embedder P	Embedder R	Embedder F1	Retriever P	Retriever R	Retriever F1	Reranker P	Reranker R	Reranker F1	Generator P	Generator R	Generator F1
DragonBall	Claude Code Haiku 5.5	1.00	0.77	0.87	1.00	0.88	0.94	0.88	0.30	0.45	0.95	0.86	0.90	0.81	0.39	0.53
	Codex GPT-5.5	1.00	0.77	0.87	1.00	0.92	0.96	0.88	0.65	0.75	1.00	0.71	0.83	0.88	0.45	0.60
	RAGLog Gemini 3.1 Flash-Lite	1.00	0.96	0.98	1.00	0.87	0.93	0.44	0.52	0.48	0.90	0.90	0.90	0.64	0.76	0.69
	RAGLog GPT-5 Mini	0.93	1.00	0.96	1.00	0.90	0.95	0.52	0.52	0.52	0.75	0.86	0.80	0.72	0.70	0.71
	RAGLog Haiku 5.5	0.96	1.00	0.98	0.94	0.90	0.92	0.58	0.65	0.61	0.77	0.95	0.85	0.69	0.76	0.72
NQ	Claude Code Haiku 5.5	1.00	0.80	0.89	1.00	0.82	0.90	1.00	0.44	0.61	0.67	0.67	0.67	0.83	0.53	0.65
	Codex GPT-5.5	0.91	0.84	0.88	1.00	0.91	0.95	1.00	0.62	0.77	1.00	0.33	0.50	0.75	0.32	0.44
	RAGLog Gemini 3.1 Flash-Lite	0.88	0.88	0.88	1.00	0.91	0.95	0.91	0.62	0.74	0.75	1.00	0.86	0.48	0.58	0.52
	RAGLog GPT-5 Mini	0.78	0.84	0.81	1.00	0.93	0.96	1.00	0.81	0.90	0.60	1.00	0.75	0.67	0.63	0.65
	RAGLog Haiku 5.5	0.80	0.80	0.80	1.00	0.87	0.93	0.86	0.75	0.80	0.38	1.00	0.55	0.64	0.74	0.68
MS MARCO	Claude Code Haiku 5.5	1.00	0.81	0.89	1.00	0.98	0.99	0.82	0.50	0.62	0.80	0.33	0.47	0.83	0.36	0.50
	Codex GPT-5.5	1.00	0.85	0.92	1.00	0.98	0.99	0.79	0.68	0.73	0.75	0.50	0.60	0.60	0.32	0.42
	RAGLog Gemini 3.1 Flash-Lite	0.91	0.77	0.83	1.00	0.96	0.98	0.76	0.89	0.82	0.86	0.50	0.63	0.56	0.54	0.55
	RAGLog GPT-5 Mini	0.91	0.77	0.83	1.00	0.98	0.99	0.87	0.96	0.92	0.67	0.67	0.67	0.58	0.54	0.56
	RAGLog Haiku 5.5	0.81	0.81	0.81	1.00	0.96	0.98	0.85	1.00	0.92	0.69	0.92	0.79	0.57	0.75	0.65
All	Claude Code Haiku 5.5	1.00	0.79	0.88	1.00	0.89	0.94	0.88	0.42	0.57	0.89	0.67	0.76	0.82	0.41	0.55
	Codex GPT-5.5	0.97	0.82	0.89	1.00	0.94	0.97	0.86	0.66	0.75	0.92	0.61	0.73	0.75	0.38	0.50
	RAGLog Gemini 3.1 Flash-Lite	0.93	0.87	0.90	1.00	0.91	0.95	0.66	0.70	0.68	0.88	0.78	0.82	0.57	0.64	0.60
	RAGLog GPT-5 Mini	0.87	0.87	0.87	1.00	0.94	0.97	0.78	0.78	0.78	0.71	0.81	0.75	0.66	0.62	0.64
	RAGLog Haiku 5.5	0.86	0.87	0.86	0.98	0.91	0.94	0.75	0.82	0.79	0.68	0.94	0.79	0.63	0.75	0.69

Dataset	System	Rewriter	Embedder	Retriever	Reranker	Generator	Pooled	Macro	Pooled	Conf (%)
DragonBall	Claude CodeHaiku 5.5	0.0	0.0	0.8	1.4	2.5	1.1	0.74	0.79	79
	CodexGPT-5.5	0.0	0.0	1.5	0.0	1.7	0.8	0.80	0.83	88
	RAGLogGemini 3.1 Flash-Lite	0.0	0.0	11.5	2.9	11.8	6.5	0.80	0.81	84
	RAGLogGPT-5 Mini	3.5	0.0	8.5	8.7	7.6	5.9	0.79	0.82	85
	RAGLogHaiku 5.5	1.8	3.0	8.5	8.7	9.2	6.7	0.82	0.83	88
NQ	Claude CodeHaiku 5.5	0.0	0.0	0.0	1.0	1.5	0.6	0.74	0.82	83
	CodexGPT-5.5	4.1	0.0	0.0	0.0	1.5	0.8	0.71	0.84	89
	RAGLogGemini 3.1 Flash-Lite	6.1	0.0	0.7	1.0	9.0	3.3	0.79	0.83	87
	RAGLogGPT-5 Mini	12.2	0.0	0.0	2.1	4.5	2.7	0.81	0.86	93
	RAGLogHaiku 5.5	10.2	0.0	1.4	5.2	6.0	3.8	0.75	0.83	89
MS MARCO	Claude CodeHaiku 5.5	0.0	0.0	2.1	1.0	1.4	1.1	0.70	0.79	79
	CodexGPT-5.5	0.0	0.0	3.5	2.0	4.2	2.3	0.73	0.80	85
	RAGLogGemini 3.1 Flash-Lite	3.5	0.0	5.6	1.0	8.5	4.1	0.76	0.81	86
	RAGLogGPT-5 Mini	3.5	0.0	2.8	4.1	7.7	3.8	0.79	0.84	90
	RAGLogHaiku 5.5	8.8	0.0	3.5	5.1	11.3	5.6	0.83	0.85	94
All	Claude CodeHaiku 5.5	0.0	0.0	1.0	1.1	1.8	0.9	0.74	0.80	80
	CodexGPT-5.5	1.2	0.0	1.7	0.8	2.5	1.4	0.77	0.82	87
	RAGLogGemini 3.1 Flash-Lite	3.1	0.0	5.8	1.5	9.6	4.6	0.79	0.82	86
	RAGLogGPT-5 Mini	6.1	0.0	3.6	4.6	6.6	4.1	0.80	0.84	89
	RAGLogHaiku 5.5	6.7	0.9	4.4	6.1	8.9	5.3	0.81	0.84	90

System	DragonBall P	DragonBall R	DragonBall F1	DragonBall FPR	NQ P	NQ R	NQ F1	NQ FPR	MS MARCO P	MS MARCO R	MS MARCO F1	MS MARCO FPR	All P	All R	All F1	All FPR
Claude CodeHaiku 5.5	0.99	0.80	0.88	2.9	0.99	0.82	0.89	1.5	1.00	0.82	0.90	0.0	0.99	0.81	0.89	1.4
CodexGPT-5.5	1.00	0.90	0.95	0.0	1.00	0.89	0.94	0.0	0.99	0.86	0.92	2.1	1.00	0.88	0.94	0.7
RAGLogGemini 3.1 Flash-Lite	0.96	0.92	0.94	14.3	0.91	0.95	0.93	13.8	0.94	0.89	0.92	14.9	0.94	0.92	0.93	14.3
RAGLogGPT-5 Mini	0.96	0.90	0.93	11.4	0.96	0.96	0.96	6.2	0.95	0.91	0.93	12.8	0.96	0.92	0.94	9.5
RAGLogHaiku 5.5	0.95	0.96	0.95	17.1	0.91	0.93	0.92	12.3	0.94	0.96	0.95	14.9	0.94	0.95	0.94	14.3

As with natural runs, the main difference is the trade-off between precision and recall. Claude Code and Codex achieve pooled precision scores of 0.95 and 0.94, respectively, with false-positive rates of 0.9% and 1.4%. However, their pooled recall is 0.69 and 0.74, as shown in Figure 3. In contrast, RAGLog$_{\mathrm{Haiku\ 5.5}}$ achieves a recall of 0.86, with a precision of 0.81 and a false-positive rate of 5.3%. Thus, RAGLog identifies more faulty stages but also produces more false positives. The agents perform considerably better on injected runs than on natural runs, possibly because some injected changes, such as an incorrect year in a rewritten query or a truncated embedding query, are more easily identifiable in execution logs.

Stage-level effectiveness

6.2.1 Stage-level effectiveness. Rewriter and embedder faults are detected effectively by all systems, with F1 scores ranging from 0.86 to 0.97 across the overall results (Table 10). The differences are more pronounced at later stages. For the generator, RAGLog Haiku 5.5 achieves an F1 score of 0.69, compared to 0.55 for Claude Code and 0.50 for Codex. The agents detect less than half of generator faults, with recall scores of 0.41 and 0.38, respectively, while RAGLog achieves 0.75. At the retriever, RAGLog achieves F1 scores of 0.68–0.79, compared to 0.75 for Codex and 0.57 for Claude Code. At the reranker, RAGLog achieves F1 scores of 0.75–0.82, compared to 0.76 for Claude Code and 0.73 for Codex; RAGLog Haiku 5.5 combines a reranker recall of 0.94 with a precision of 0.68.

6.2.2 Effectiveness across datasets. Performance varies across the three datasets (Table 11). On DragonBall, Codex and RAGLog Haiku 5.5 achieve the highest pooled F1 score of 0.83. On NQ, RAGLog GPT-5 Mini achieves the highest pooled F1 score of 0.86, compared to 0.84 for Codex and 0.83 for RAGLog Haiku 5.5. On MS MARCO, RAGLog performs better, with RAGLog Haiku 5.5 achieving a pooled F1 score of 0.85, compared to 0.79 for Claude Code and 0.80 for Codex.

6.2.3 Detection of confirmed injected faults. Among the 322 test runs in which the judges confirmed a fault at the injected stage, RAGLog Haiku 5.5 correctly flags that stage in 90% of runs, compared to 87% for Codex and 80% for Claude Code (Table 11). The other RAGLog variants achieve detection rates of 89% for RAGLog GPT-5 Mini and 86% for

RAGLogGemini 3.1 Flash-Lite. These results show that RAGLog is particularly effective at identifying the stage where an injected fault occurred, even when its overall pooled F1 is similar to that of the agents.

Run-level detection

Of the 480 injected test runs, 333 contained at least one stage-level fault, while 322 had a confirmed fault at the targeted injection stage. At the run level, all systems detect most runs containing faults (Table 12). Codex, RAGLogHaiku 5.5, and RAGLogGPT-5 Mini achieve the highest F1 score of 0.94, followed by RAGLogGemini 3.1 Flash-Lite at 0.93 and Claude Code at 0.89. RAGLogHaiku 5.5 achieves the highest recall of 0.95, compared to 0.88 for Codex and 0.81 for Claude Code. However, it also has a substantially higher false-positive rate of 14.3%, compared to 0.7% and 1.4% for the agents. Since every run in this test set underwent fault injection, these false positives occur in runs where judges found no stage-level fault despite the injection. RAGLogGPT-5 Mini offers a better balance in this setting, achieving the same F1 score of 0.94 with a lower false-positive rate of 9.5%.

Manifestation and propagation of faults

6.2.5 Manifestation and propagation of faults. Table 13 examines how injected faults manifest and propagate across the 480 test runs. Whether an injection produces a confirmed fault depends strongly on the targeted stage. All rewriter and embedder injections produce faults at their respective stages, compared to 83% of retriever injections, 48% of reranker injections, and 22% of generator injections. Many injections at later stages are absorbed without producing a fault under the judges’ criteria. For example, demoting an evidence chunk at the reranker may still leave sufficient evidence in the selected context, while generators may produce correct answers even when relevant evidence is mixed with passages about similar entities. The consequences of confirmed faults also vary by stage. Retriever and reranker faults frequently affect the final answer, with incorrect answers in 49 of 50 and 26 of 29 confirmed cases, respectively. In contrast, many rewriter and embedder faults leave the final answer correct. These cases demonstrate why evaluating only final-answer correctness can miss faults in earlier stages. They also show that faults do not necessarily propagate through the entire pipeline, even when a stage produces an incorrect output.

Stage	Fault type	Runs	Confirmed	Later stage faulty	Wrong answer
Rewriter	A wrong year is added	43	43 (100%)	14	7
	The entity is replaced by a similar one	16	16 (100%)	10	10
	A date or qualifier is dropped	1	1 (100%)	1	0
Embedder	The query is cut to its first half	76	76 (100%)	18	16
	The key entity is misspelled	74	74 (100%)	3	3
Retriever	The evidence chunks are not returned	60	50 (83%)	8	49
Reranker	The evidence is ranked below the cut-off	30	20 (67%)	4	17
	One of several evidence chunks is ranked below the cut-off	30	9 (30%)	4	9
Generator	The evidence is placed among similar-entity passages	150	33 (22%)	–	17
Total		480	322 (67%)		128

RQ3: How do the key components of RAGLog contribute to its effectiveness?

API call classification

6.3.1 API call classification. Table 14 shows the effectiveness of the rule-based, LLM-based, and hybrid approaches. On familiar APIs from the five evaluated pipelines, the rules correctly classify all 80,778 calls without requiring the LLM. With the rules disabled, Claude Haiku 5.5 correctly classifies all 51 sampled calls, compared to 47 for GPT-4.1 mini. On 158 unfamiliar API calls, the rules classify 151, of which 149 are correct. The hybrid correctly classifies 156 calls with either model, compared to 151 and 144 for the LLM alone, while sending only seven calls to the LLM. Overall, the

Step	Set	Calls	Rules alone: Coverage	Rules alone: Correct	LLM alone *: Haiku 5.5	LLM alone *: GPT-4.1 mini	Hybrid: Haiku 5.5	Hybrid: GPT-4.1 mini
Classification	Familiar APIs	80,778	100	100	100	92	100	100
	Unfamiliar APIs	158	96	99	96	91	99	99
Transformation	Familiar APIs	18	94	100	100	100	100	100
	Unfamiliar APIs	124	94	99	98	97	98	98

• The LLM-only classification was evaluated on a sample of 51 calls.

Coverage: percentage of calls handled by the rules; Correct: percentage of covered calls that are handled correctly.

hybrid approach provides the best balance of accuracy and efficiency, combining reliable rule-based classification with selective LLM assistance.

Log-to-schema transformation

When the mappings for the five pipelines are rebuilt from scratch, both the hybrid and LLM-only approaches correctly reproduce all 18 stage traces. The rules independently map 17 stages, with only Verba’s retriever requiring LLM assistance. On 124 unfamiliar API stage calls, the rules map 116, of which 115 are correct. The hybrid produces correct traces for 121 calls with Claude Haiku 5.5 and 122 with GPT-4.1 mini. The remaining errors involve unusual response formats, such as ranking results identified by IDs rather than positions. Together, these results show that rules handle most classification and transformation tasks, while LLMs resolve cases the rules cannot handle.

{sup}†` Statistically significant difference from RAGLogHaiku 5.5: the 95% bootstrap confidence interval of the difference excludes zero.

Fault localization

Table 15 shows the contribution of individual fault-localization components, marking differences that are statistically significant. LLM-based checks contribute the most: removing them reduces pooled F1 by 0.275 on natural runs and 0.170 on injected runs. Embedding-based checks are particularly important for injected faults (−0.111), while rule-based checks contribute modestly on both test sets (−0.016 and −0.020), significantly only on injected runs. Among individual checks, removing the evidence flow check reduces pooled F1 by 0.027 on natural and 0.025 on injected runs, highlighting the value of relating evidence across stages; it is the only individual check with a significant effect on both sets. The graded stage check also contributes on both sets, significantly only on injected runs. In contrast, the three generator checks show overlapping contributions: removing any one of them changes pooled F1 by at most 0.013, which is not significant, and removing the statement-level generator check, the most expensive one, reduces pooled F1 by only 0.001 on natural and 0.006 on injected runs.

Configuration	Natural Pooled F1	Injected Pooled F1	Natural FPR (%)
RAGLogHaiku 5.5	0.728	0.837	4.6
Stage checks
Without rule-based checks	0.712 (−0.016)	0.817 (−0.020){sup}†`	4.6
Without embedding-based checks	0.723 (−0.005)	0.726 (−0.111){sup}†`	4.5
Without LLM-based checks	0.453 (−0.275){sup}†`	0.667 (−0.170){sup}†`	6.6
Without graded stage check	0.717 (−0.011)	0.813 (−0.024){sup}†`	4.9
Without graded generator check	0.725 (−0.003)	0.824 (−0.013)	3.5
Without generator verification	0.719 (−0.009)	0.837 (+0.000)	5.0
Without evidence flow check	0.701 (−0.027){sup}†`	0.812 (−0.025){sup}†`	4.3
Without statement-level generator check	0.727 (−0.001)	0.831 (−0.006)	4.1
Ensemble decision
Logistic regression only	0.685 (−0.043){sup}†`	0.786 (−0.051){sup}†`	4.3
Gradient-boosted trees only	0.692 (−0.036){sup}†`	0.810 (−0.027){sup}†`	4.2
Extremely randomized trees only	0.714 (−0.014)	0.816 (−0.021)	3.0
Equal weights	0.722 (−0.006)	0.840 (+0.003)	4.1
Weights 1/2, 1/4, 1/4	0.716 (−0.012)	0.825 (−0.012)	4.7
Weights 1/5, 3/5, 1/5	0.696 (−0.032){sup}†`	0.839 (+0.002)	4.1
Own-stage measurements only	0.716 (−0.012)	0.818 (−0.019)	3.2
Without the false-alarm cap	0.728 (+0.000)	0.837 (+0.000)	4.6

Replacing the ensemble with logistic regression alone significantly reduces pooled F1 by 0.043 on natural runs and 0.051 on injected runs. Gradient-boosted trees alone also perform significantly worse on both sets (−0.036 on natural and −0.027 on injected runs), whereas extremely randomized trees alone perform similarly to the ensemble (−0.014 and −0.021, not significant). Most alternative weightings have little impact, changing pooled F1 by at most 0.012; only counting logistic regression once (weights 1/5, 3/5, 1/5) reduces natural pooled F1 by 0.032. Restricting classifiers to their own-stage measurements reduces pooled F1 by 0.012 on natural and 0.019 on injected runs; both differences point to a benefit of cross-stage measurements, although neither is statistically significant. Removing the false-alarm cap has no effect, as the selected thresholds already satisfy its constraint. Overall, beyond the LLM-based and embedding-based checks, the components provide modest observed benefits rather than evidence that each one is necessary.

Answer to RQ3: The hybrid classification and transformation approaches achieve high accuracy by handling most calls through rules and using LLMs only when those rules cannot resolve a call. For fault localization, LLM-based checks contribute the most, followed by embedding-based checks, particularly for injected faults. The evidence flow check, which relates evidence across stages, provides a smaller but significant improvement, and the ensemble significantly outperforms logistic regression and gradient-boosted trees alone, whereas giving each classifier the measurements of other stages shows a modest benefit that is not statistically significant.

RQ4: How efficient is RAGLog at detecting and localizing faults in RAG pipelines?

Figure 4 compares the cost, execution time, and token usage per question, while Figure 5 relates cost to pooled F1. Cost and token counts use logarithmic scales. For RAGLog, cost and tokens are averaged per run and time is the median; for the agents, the reported values are session totals divided by the number of questions.

Cost and effectiveness

On natural runs, all three RAGLog variants outperform both agents in pooled F1 while costing less (Figure 5). RAGLog$\mathrm{_{Haiku\ 5.5}}$ achieves the highest F1 (0.73) at US$0.0049 per question, compared to 0.58 at US$0.0108 for Claude Code and 0.52 at US$0.0365 for Codex. On injected runs, the systems achieve similar F1 scores (0.80–0.84), but RAGLog is substantially cheaper than Codex. RAGLog$\mathrm{_{Haiku\ 5.5}}$ costs US$0.0051 per question, matching Claude Code and costing approximately twenty times less than Codex. Across both test sets, RAGLog$\mathrm{_{Haiku\ 5.5}}$ achieves the best overall cost-effectiveness, with a pooled F1 of 0.77 at US$0.0050 per question, compared to 0.67 at US$0.0094 for Claude Code and 0.65 at US$0.0535 for Codex. RAGLog’s cost is also predictable because each run requires five LLM calls, while the agents’ usage depends on their analysis. Classification and transformation add little cost, as rules handle all calls in the evaluated pipelines.

Tokens

RAGLog uses 22,000–40,000 input tokens and 700–2,400 output tokens per question. Claude Code uses substantially more input tokens (318,000 on natural runs) because it examines raw logs containing requests, responses,

and embedding vectors. However, 87–93% of the agents’ input tokens are cached, reducing their effective cost. Codex uses 36,000–88,000 input tokens per question, with its higher monetary cost also reflecting GPT-5.5’s API pricing.

Time

RAGLog$_{\text{Gemini 3.1 Flash-Lite}}$ is the fastest variant, analyzing each run in a median of 2.0 seconds on both test sets, while achieving pooled F1 scores of 0.67 on natural runs and 0.82 on injected runs. This makes it an attractive option when latency is prioritized over maximum effectiveness. RAGLog$_{\text{Haiku 5.5}}$ takes approximately 4.6–4.7 seconds, offering higher effectiveness at modest additional latency, while RAGLog$_{\text{GPT-5 Mini}}$ takes 4.3–6.7 seconds. RAGLog’s five LLM calls execute in parallel, with local computation taking approximately 0.25 seconds. In contrast, the agents require 5.6–19.0 seconds per question and analyze complete sessions rather than individual runs. RAGLog can therefore provide fault-localization results shortly after each run completes, making it more suitable for monitoring pipelines in production.

Answer to RQ4: RAGLog offers a better cost-effectiveness trade-off than the agent baselines, particularly on natural runs, where all variants achieve higher pooled F1 at lower cost. RAGLog$_{\text{Haiku 5.5}}$ provides the highest overall effectiveness at approximately US$0.005 per question, while RAGLog$_{\text{Gemini 3.1 Flash-Lite}}$ offers the lowest latency (2.0 seconds) with competitive effectiveness. RAGLog’s predictable cost and per-run analysis further support its suitability for continuous monitoring.

The RAGLog Tool

We implemented RAGLog as an open-source tool⁴ for debugging, evaluating, and monitoring RAG pipelines. This section describes how developers connect their pipelines to RAGLog, inspect its findings, and configure the analysis for different use cases.

Architecture and Deployment

RAGLog consists of three services (Figure 6): a logging proxy, a server, and a web dashboard. The logging proxy captures outbound API calls from a RAG pipeline (§ 4.2). The RAGLog server classifies these calls, transforms them into RAG schema instances, groups them into individual runs, and applies the fault classifiers. The dashboard presents the analysis results and provides controls for configuring and evaluating the pipeline. The services can run together using Docker Compose or as separate processes on a developer’s machine. To connect a pipeline, developers route its outbound HTTP traffic through the proxy using standard proxy settings and configure its HTTP clients to trust the proxy’s certificate. This requires no changes to the pipeline’s source code and works for containerized, local, and server-based deployments. The server analysis runs asynchronously, so it does not delay the pipeline’s responses.

RAGLog also supports offline analysis of recorded data. It accepts API call logs from the proxy or a simple call-record format, which undergo the same processing as live traffic. Alternatively, developers can provide tabular runs containing questions, retrieved chunks, generator context, and answers for direct fault localization. RAGLog also accepts OpenTelemetry traces with OpenInference attributes [6, 49], mapping their embedding, retrieval, reranking, and LLM spans to the corresponding RAG stages. These options allow RAGLog to analyze previously recorded runs and traces from existing instrumentation without routing traffic through its proxy.

Connecting a Pipeline

When a new pipeline is connected, RAGLog automatically generates the mapping between its API calls and the RAG schema. To help developers verify this mapping, the dashboard provides a setup check that shows which stages were captured, their API endpoints, and the information extracted from the latest run. For example, it displays the embedding input, the number of retrieved chunks, and a preview of the generated answer. It also identifies unclassified calls and provides possible explanations for missing stages, such as stages that are not present in the pipeline. If a stage is captured but its information is not extracted correctly, developers can adjust its mapping directly in the dashboard. RAGLog retains the previous mapping alongside the updated version, allowing developers to inspect their changes.

Inspecting Runs

Once a pipeline is connected, the dashboard provides an overview of its analyzed runs (Figure 7a). It summarizes how often faults are detected, which stages are most frequently flagged, and the time and cost of analysis. Results are organized into sessions, each representing a period or configuration of a pipeline, such as a test execution or software release. Developers can inspect fault rates across sessions and navigate to individual runs. The runs page lists each question alongside its fault verdict, pipeline response time, and RAGLog analysis time.

For each run, RAGLog presents the fault verdict for every stage, its predicted fault probability, and explanation. The dashboard also displays the complete execution trace, including rewritten queries, embedding inputs, retrieved chunks and their scores, generator context, and final answer. These details help developers understand why a stage was flagged and trace the fault to the relevant evidence. For example, Figure 8 shows a LightRAG run with a retriever fault. The pipeline responds that it lacks sufficient information, while RAGLog identifies the retriever as faulty because none of the retrieved chunks contains the requested cash flow figure. RAGLog can also identify faults that do not affect the final answer.

Configuring the Analysis

Developers can adjust RAGLog’s analysis according to their accuracy, latency, and cost requirements (Figure 7b). The main setting is the LLM used for the LLM-based checks. RAGLog supports Claude Haiku 5.5 (the default), GPT-5 Mini, and Gemini 3.1 Flash-Lite, costing approximately US$0.005, US$0.009, and US$0.007 per question, respectively. Each LLM can also be used in an economy mode that omits the statement-level generator check, the most expensive of the five LLM calls. This reduces the cost of an analysis by about 20%, while keeping pooled F1 within 0.01 of the full configuration in cross-validation. It barely shortens the analysis, because the LLM calls run in parallel and the evidence flow check is usually the slowest. For faster analysis, developers can choose Gemini 3.1 Flash-Lite, which takes about two seconds per run (§ 6.12). The released tool uses the configuration evaluated in § 6, but its classifiers are trained on all labeled runs in RAGLog Faults rather than only the development set. For Claude Haiku 5.5, this increases pooled F1 from 0.755 to 0.780 in grouped cross-validation.

RAGLog also allows developers to control when analysis takes place. Runs are always recorded, but fault detection can be performed after every run, after a specified number of runs, or at regular intervals. Immediate analysis supports debugging and monitoring, while batch analysis reduces the load in production. Developers can also set an analysis budget, after which RAGLog stops issuing LLM calls.

To control sensitivity, developers can adjust the fault-classification threshold for each stage. When labeled runs are available, RAGLog suggests thresholds that maximize stage-level F1 or keep the false-positive rate below a specified target, such as 5%. The dashboard shows the expected precision, recall, and false-positive rate before a threshold is applied. For pipelines retrieving many chunks, developers can also limit analysis to the highest-ranked candidates. Chunks passed to the generator are always retained, but this setting may miss relevant chunks outside the limit, trading some retrieval and reranking fault detection for faster analysis.

Testing and Evaluating Pipelines

Beyond monitoring individual runs, RAGLog supports fault injection and comparative evaluation. The dashboard integrates the live fault injector described in § 4.3, allowing developers to select a stage, a fault type, and the number of questions to affect. Faults are injected into subsequent pipeline runs, so downstream stages respond to the modified outputs. The dashboard identifies injected runs and reports how often RAGLog detects the affected stage. This allows developers to examine how faults propagate through their pipelines and assess the effectiveness of fault localization. Fault types that remove or demote relevant evidence require questions with reference passages and are therefore limited to suitable evaluation datasets.

For evaluation, developers can upload reference answers, enabling RAGLog to assess final-answer correctness alongside stage-level faults at a small additional cost (about US$0.0001 per run in our tests). Developers can also manually label faulty stages, which RAGLog uses when suggesting detection thresholds. Finally, RAGLog supports comparisons across pipeline sessions. The comparison page displays results from two sessions side by side, such as before and after a pipeline update, and highlights questions whose verdicts have changed.

Discussion

The Importance of Stage-Level Fault Localization

Our findings highlight the limitations of evaluating RAG pipelines solely through their final answers. Among the 551 naturally faulty runs in our test set, 202 (37%) still produced correct answers (§ 4.5). Such runs would be considered successful by answer-level evaluation, even though faults occurred in intermediate stages. For example, a retriever may omit relevant evidence or a reranker may discard a useful chunk without affecting the answer to a particular question. These faults can leave the pipeline more vulnerable to failure on other questions, even when their immediate effect is not visible in the output.

RAGLog detects many of these hidden faults. Among the 202 faulty runs with correct answers, RAGLogHaiku 5.5 correctly identifies at least one faulty stage in 144 runs (71%), compared with 59 (29%) for Claude Code and 32 (16%) for Codex. This is stricter than run-level detection (Table 9), which counts a run as detected whenever any stage is flagged, regardless of whether that stage is actually faulty. For faulty runs that produce incorrect answers, the gap is smaller, with 85% of runs detected by RAGLog, compared with 63% and 58% for Claude Code and Codex, respectively. These findings suggest that the agents may rely more heavily on visible answer failures when identifying faults, whereas RAGLog evaluates intermediate stages regardless of final-answer correctness. Stage-level analysis therefore provides information that answer-level evaluation alone cannot capture [60].

The ablation results further suggest that stage-level fault localization benefits from considering evidence across the pipeline. The evidence flow check, which follows the needed information from retrieval to the answer, is the only individual check whose removal significantly reduces pooled $F_1$ on both test sets, and restricting the classifiers to their own stage’s measurements also reduces pooled $F_1$ on both sets, although not significantly (Section 6.3). This is consistent with the interconnected behavior of RAG stages: the output of one stage becomes the input to another, and evidence from downstream stages can help assess whether an earlier stage behaved correctly. These findings support evaluating not only individual stage outputs but also how information flows between stages.

Stage-level verdicts help developers identify where to begin debugging a failed run. In the natural test runs, 71% of incorrect answers originated from stages before the generator, although answer-level evaluation and hallucination detectors focus on the final output. Among the 349 faulty runs with incorrect answers, the earliest stage flagged by RAGLogHaiku 5.5 matched the earliest faulty stage in 78% of runs, compared with 59% for Claude Code and 55% for Codex. RAGLog can therefore help developers focus their debugging efforts on where faults originate, such as query rewriting, retrieval, or reranking, rather than making changes to the generator that may not address the underlying problem.

Combining Rules, LLMs, and Learned Classifiers

RAGLog’s component evaluations highlight the benefits of assigning different analysis tasks to techniques suited to their characteristics (Section 6.3). API call classification and log-to-schema transformation are largely structural tasks, for which rules are often sufficient. The rules correctly classify all calls from the five evaluated pipelines and 149 of 158 calls from unfamiliar APIs. The remaining cases are primarily calls whose role depends on their instructions rather than their endpoint. For example, the same chat API may be used for query rewriting or another generation task. RAGLog uses an LLM to resolve such cases instead of applying it to every call. The hybrid approach correctly classifies 156 of 158 unfamiliar API calls while minimizing reliance on LLMs and their associated costs and variability.

Fault localization presents a different challenge because many faults require semantic judgments about questions, retrieved evidence, and generated answers. Accordingly, removing LLM-based checks produces the largest decline in fault-localization effectiveness of the three evaluated groups of stage checks. However, the LLMs do not make the final fault decisions independently. Their outputs are combined with rule-based and embedding-based measurements through learned classifiers, which also incorporate evidence from other stages. The ablations show smaller, partly significant contributions from these additional measurements and the ensemble compared with the LLM-based checks. Together, the results favor a hybrid design over relying exclusively on deterministic checks or unconstrained LLM judgments.

The choice of LLM for the checks also appears relatively flexible. On natural runs, pooled F1 ranges from 0.67 with Gemini 3.1 Flash-Lite to 0.73 with Claude Haiku 5.5, while the three variants achieve pooled F1 scores of 0.82–0.84 on injected runs (§ 6.1 and § 6.6). Since each LLM-based check performs a narrowly defined assessment and the classifiers are trained on its outputs, these results suggest that RAGLog’s effectiveness depends not only on the underlying LLM but also on how its judgments are combined with other measurements. Developers can therefore consider cost, latency, and provider constraints when selecting an LLM, although a different model from the ones provided requires the fault classifiers to be retrained and its effectiveness to be evaluated.

The ablation results also identify opportunities to reduce analysis overhead. Three of the five LLM-based checks assess the generator, and their contributions overlap: removing any one changes pooled F1 by less than 0.02 (Section 6.3). Omitting the statement-level generator check reduces cost by about 20%, with little difference in effectiveness in the evaluated configurations. The released tool offers this configuration as a lower-cost economy mode. We still retain all five checks, since, among all combinations of the three generator checks, this configuration achieves the highest pooled F1 on natural runs, and the combinations differ by at most 0.02 in cross-validation on the development runs.

Specialized Fault Localization and General-Purpose Agents

The comparison with Claude Code and Codex highlights different trade-offs between specialized fault localization and general-purpose log analysis. On natural runs, the agents have low false-positive rates but miss many faulty stages. Their pooled recalls are 0.46 and 0.40, respectively, compared with 0.78 for RAGLog$_{\mathrm{Haiku\ 5.5}}$, which achieves a pooled F1 of 0.73 against 0.58 and 0.52 for the agents (§ 6.1). RAGLog achieves substantially higher recall while maintaining a relatively low pooled false-positive rate of 4.6%, compared with approximately 1.5% for the agents. This reflects a trade-off between detecting more faults and generating additional false alarms. Developers can adjust stage-specific thresholds to balance these priorities according to their monitoring requirements. At run level, this trade-off is larger: RAGLog flags about 14% of fault-free runs, compared with about 5% for the agents, because a run is flagged when any of its stages is flagged. In monitoring, these false alarms cost a developer the time to inspect a run; stricter thresholds can reduce them at the cost of missing more faults.

The approaches also differ in their analysis overhead. General-purpose agents must interpret the log format, identify individual runs, and extract relevant information before investigating faults. Some of this work is repeated for each session, making the cost sensitive to how logs are divided. For example, Codex costs approximately US$1.83 per session for natural logs containing 50 questions and US$1.07 for injected logs containing about 10 questions. Its average cost per question consequently increases from US$0.037 to US$0.107 (§ 6.12). RAGLog instead uses rules and previously generated mappings to process calls, followed by a fixed set of checks for each run. With Claude Haiku 5.5, its analysis costs approximately US$0.005 per run and takes around five seconds. These results make RAGLog better suited to repeated analysis, particularly when runs arrive continuously rather than in large batches.

RAGLog also provides structured, repeatable stage-level verdicts, making it suitable for regular monitoring and regression testing. General-purpose agents can undertake more open-ended investigations, including exploring additional log details and generating scripts to examine suspected problems. A practical workflow could therefore use RAGLog to identify potentially faulty stages and reserve agent-based investigation for selected runs requiring deeper diagnosis. This division could reduce repeated analysis overhead while retaining the flexibility of agents for cases that structured checks cannot fully explain.

Generalization and Pipeline Adaptation

Although RAGLog uses a RAG schema to structure logs from different providers into a common representation, its effectiveness decreases on pipelines excluded from training. For RAGLog$_{\mathrm{Haiku}\ 5.5}$, pooled F1 on natural runs falls from 0.73 to 0.67 in the unseen-pipeline evaluation, with similar decreases for the other variants (§ 6.1). This reveals an important distinction between processing unfamiliar pipeline logs and accurately identifying faults in those pipelines. RAGLog can classify and transform most unfamiliar API calls, but the learned fault classifiers may still encounter stage behaviors and measurement patterns that are different from those represented in their training data.

Increasing the diversity of labeled runs is one way to address this limitation. Training on additional data already provides some benefit within the evaluated pipelines: when the tool’s classifiers are trained using all labeled runs, grouped cross-validation pooled F1 increases from 0.755 to 0.780 for Claude Haiku 5.5 (§ 7). Since RAGLog represents runs using a schema instance, additional pipelines can contribute training examples without requiring changes to the fault-localization architecture. However, the extent to which further training improves performance on unseen pipelines is left for future work.

These results show that RAGLog transfers to pipelines it was not trained on under our evaluation conditions with open-source pipelines, harmonized settings, benchmark questions, and recorded sessions. They do not establish its effectiveness in production, where queries come from real users, document collections change, and pipelines run continuously; validating RAGLog in a production deployment remains future work.

Implications for Fault Injection and Evaluation

Our fault-injection experiments demonstrate why injecting a disturbance should not automatically be treated as evidence of a fault. Among the 480 labeled injected runs, only 83% of retriever injections, 48% of reranker injections, and 22% of generator injections produced a fault at the targeted stage (§ 6.6). Some disturbances were absorbed because alternative evidence remained available or the modified output did not violate the stage’s expected behavior. Therefore, treating every injected run as faulty would introduce incorrect ground-truth labels and could penalize a detector for correctly leaving an unaffected stage unflagged.

The distinction is also important when interpreting RAGLog’s predictions. RAGLog flags 14% of injected runs that the judges labeled fault-free, indicating that some checks respond to disturbances even when those disturbances are not confirmed faults. This highlights the difficulty of differentiating unusual stage behavior from behavior that is actually faulty. Fault-injection benchmarks should therefore verify whether the intended fault manifested, rather than equating the injection operation with its outcome. They should also distinguish faults at the injected stage from faults that propagate to downstream stages. Our live fault injector and stage-level labeling procedure support this distinction, allowing fault-localization effectiveness to be evaluated against observed pipeline behavior rather than injection intent.

Contributions of the Schema and the Detector

RAGLog differs from the agent baselines in two ways: it transforms raw logs into structured RAG schema instances and applies targeted stage checks and learned fault classifiers to these instances. To examine the contribution of each, we conducted a pilot experiment in which the agents received RAG schema instances generated by RAGLog instead of execution logs. The task description was adapted to explain the schema fields, while the stage definitions and fault criteria remained unchanged. We selected five test sessions, three containing natural runs (DragonBall on LightRAG, NQ on Onyx, and MS MARCO on RAGFlow) and two containing injected runs (DragonBall on Quivr, MS MARCO on Onyx). These sessions comprised 291 runs, including 150 with faults, and were used to evaluate fault-localization effectiveness (Table 16).

System	Input Structure	P	R	Pooled F1	Cost (US$) | Input tokens | Output tokens
RAGLog$_{\text{Haiku 5.5}}$	Raw logs	0.77	0.86	0.81	0.0050	39k	2.3k
Claude Code$_{\text{Haiku 5.5}}$	Raw logs	0.83	0.63	0.72	0.0070	171k	3.6k
	RAG schema	0.91	0.66	0.77	0.0022	57k	1.6k
Codex$_{\text{GPT-5.5}}$	Raw logs	0.91	0.66	0.77	0.044	50k	0.3k
	RAG schema	0.78	0.71	0.75	0.026	26k	0.2k

The schema substantially reduces agent resource usage but has a smaller and less consistent effect on fault-localization effectiveness. Claude Code improves mainly on RAGFlow, where in-process retrieval and preprocessed reranker inputs make the raw logs difficult to interpret. By making the reranker’s inputs and outputs explicit, the schema improves its reranker F1 from 0.00 to 0.67, while its results on the other sessions only change slightly. Codex, in contrast, achieves higher recall but lower precision with schema instances, resulting in a lower pooled F1.

Despite these improvements, structured inputs do not close the recall gap with RAGLog, particularly for generator faults. RAGLog detects 74% of generator faults, compared with 15% for Claude Code and 36% for Codex when given schema instances. RAGLog systematically applies targeted checks to every run, whereas the agents do not always perform the same detailed analysis, even when given structured inputs. This suggests that RAGLog’s effectiveness cannot be attributed to the schema alone; the transformation facilitates log interpretation and reduces analysis cost, while the checks and learned classifiers provide systematic fault localization. However, these findings are preliminary, as the pilot experiment covered only five sessions.

Threats to Validity

Internal Validity

A primary threat to internal validity is the accuracy of the ground truth. Manually labeling every stage of 3,134 runs was not feasible, so we rely on two LLM judges, which may produce incorrect labels. To mitigate this threat, we use judges from different providers, provide reference answers and evidence, derive stage labels from their factual findings using fixed rules, and employ a third LLM to adjudicate disagreements. Two judges share providers with RAGLog’s underlying LLMs (OpenAI and Anthropic), potentially favoring those variants. To reduce this risk, the judges have access to reference evidence, use different prompts and different models, and their labels must agree or undergo adjudication. For the development runs, which are used only for training, we manually reviewed 64 runs with correct answers that the judges labeled as generator faults solely because of unsupported statements. Some involved harmless elaborations, common knowledge, or hedged inferences. Following fixed criteria, we retained a fault when a statement contradicted the retrieved chunks, provided an unsupported answer to the question, or introduced a specific name, number, date, or place. We corrected 19 labels (2% of development runs). These corrections may affect the learned classifiers, but the test labels remained unchanged. We did not validate the judges’ test labels against independent human annotation, as our manual review covered only development runs. Effectiveness scores are therefore based on these labels, which may contain errors. Overall, the judges agreed on 94% of stage decisions (Cohen’s $\kappa$ = 0.72), although agreement was lower for the embedder ($\kappa$ = 0.27, with only 10 embedder faults), and the rewriter ($\kappa$ = 0.47).

LLM nondeterminism also affects classification, transformation, fault checks, judges, and agent baselines. RAGLog mitigates its impact through deterministic rules, which correctly handle all API calls in the evaluated pipelines and 149 of 158 unfamiliar calls, and through fault classifiers that combine rule-based, embedding-based, and LLM-based measurements. However, variability in LLM outputs may affect the results. Agent effectiveness may also depend on task instructions. We mitigate this threat by providing them with the same fault definition as the judges, describing the log format, and avoiding prompt tuning on test results.

External Validity

We evaluate five open-source RAG pipelines with different architectures, models, and retrieval backends, using three English-language datasets: DragonBall (finance, law, and medicine), NQ (Wikipedia articles), and MS MARCO (web passages). These pipelines and datasets do not represent all production settings, limiting external validity. Furthermore, the leave-one-pipeline-out evaluation shows reduced effectiveness on unseen pipelines (§ 6.1), suggesting that generalization may have limits.

RAGLog covers observable stages after data ingestion. It supports HTTP-based API calls and Weaviate’s gRPC calls through a dedicated logger, but not other binary protocols, such as PostgreSQL’s protocol [50]. Although HTTP-based communication is common in RAG pipelines, unsupported protocols may limit applicability. Agentic RAG pipelines, which dynamically select and repeat stages, are also not supported. Effectiveness may further depend on the underlying LLM. We evaluated three LLMs from different providers, but other or newer models may behave differently. Additionally, the local cross-encoder used to select excluded chunks for LLM-based checks was trained on MS MARCO, which is also an evaluation dataset. However, it is only used in pipelines retrieving many chunks (Quivr and RAGFlow) at similar frequencies across all three datasets (37–40% of runs), reducing the likelihood of a dataset-specific advantage.

Construct Validity

Our definition of a faulty stage considers deviations from the expected output for a given input, even when the final answer remains correct. Alternative definitions could produce different labels. We therefore also report run-level detection, which does not depend on the identified faulty stage. Furthermore, because the judges and RAGLog use the same classified and transformed traces, errors in either step could affect both labels and measurements without being detected. We address this by evaluating API classification and log-to-schema transformation separately in RQ3.

Since injected runs may not fully represent naturally occurring faults, we evaluate natural faults separately and count injected stages as faulty only when confirmed by the judges. We also replaced 36 NQ and MS MARCO questions whose reference answers were independently judged by two LLMs to be unsupported by their source documents.

The agents’ measured efficiency depends on session size because they repeat log exploration and script generation for each session. Natural run sessions contain 50 questions, compared to approximately 10 for injected runs, contributing to Codex’s cost difference (US$0.037 versus US$0.107 per question). We therefore report efficiency separately for both test sets. Larger natural run sessions favor the agents, yet all RAGLog variants remain less expensive. Analyzing individual runs, as required for immediate monitoring, could further increase agent costs. Finally, monetary estimates depend on provider pricing at the time of evaluation and may change in the future.

Conclusion Validity

Some stages have far fewer faults than others, making their individual F1 scores less reliable. For example, natural test runs contain only 10 embedder faults compared to 329 generator faults. As a result, even a few cases from rare stages can have a large impact on F1 scores and macro-F1. We therefore focus on pooled F1, which aggregates decisions across all stages.

To account for variation in the selected test questions, we report bootstrap confidence intervals for comparisons with the agents (§ 6). We also assessed variability in LLM outputs by repeating the LLM-based checks of RAGLog$_{\mathrm{Haiku}\ 5.5}$ twice on 150 natural test runs (10 per pipeline and dataset). Across the three runs, 95–97% of stage verdicts were consistent, and pooled F1 varied by only about 0.02, which is much less than RAGLog’s advantage over the agents on natural runs. We did not repeat the agent evaluations due to their higher cost, so their run-to-run variability remains unknown. Finally, fault-classifier thresholds were selected on development runs to keep each stage’s false-positive rate below 10%. Although pooled stage-level false-positive rates remain below 10% for every test dataset, individual stages sometimes exceed this threshold, such as the generator of RAGLog$_{\mathrm{Haiku}\ 5.5}$ on natural DragonBall runs (10.8%). Thus, the development-set constraint does not guarantee the same bound for every stage and dataset.

Related Work

Evaluation and debugging of RAG pipelines

Several techniques exist for improving the quality of RAG pipelines. Raggy [35] is a real-time interactive visual debugging tool for the indexing, retrieval, and generation steps of RAG pipelines. RAGAS [20], ARES [55], RAGChecker [54], RAGEval [66], and RAGProbe [58] are automated evaluation frameworks for RAG pipelines that generate data or use references to assess the quality of retrieval and generation with metrics such as correctness, faithfulness, and relevance. FROAV [39] is a RAG observability and agent verification framework with a visual interface for prototyping, integrated evaluation metrics, and correlation between automated and human judgments. Another interactive evaluation system is RAGTrace [14], which focuses on retrieval and generation in RAG pipelines through a multi-level analysis, custom metrics, and cross-component interactions to identify potential failures. Observability platforms such as Langfuse [34], LangSmith [33], and Phoenix [5] record a trace of every stage and offer evaluators that score individual outputs, but they require instrumentation of the pipeline and leave the analysis of the traces to the developer. Most of these techniques require instrumentation or reference answers and focus on the retrieval and generation stages, and none of them automatically localizes faults to individual pipeline stages. They are well suited for development but may not scale to production environments, where reference answers are unavailable and every run has to be analyzed at a low cost. In contrast, RAGLog locates faults in any stage of a RAG pipeline from its execution logs, without code instrumentation or reference answers, and at a cost that allows its use in production.

Fault detection and localization in RAG pipelines

The closest to our work are approaches that detect or localize faults in RAG pipelines [19, 29, 36, 59, 60]. Tanhäei [60] proposes a multi-layer quality-gating framework for detecting silent failures across retrieval, context construction, generation, and runtime behavior. The framework demonstrates that answer-level evaluation alone misses deeper evidence and provenance defects under controlled fault injection. However, it is designed as a pre-release validation gate: its composite score depends on reference evidence sets, reference document identifiers, answerability labels, and citation mappings that are unavailable in live deployments, in addition to instrumented runtime metadata not exposed by standard RAG APIs. RAGLog instead detects and localizes faults without reference data, from API call logs. Leung et al. [36] introduce RAGEC, which defines 16 fault types across the chunking, retrieval, reranking, and generation stages and automatically classifies the stage and fault type responsible for an incorrect response. RAGEC first compares the generated response against a reference answer and documents to identify incorrect responses. For each identified fault, it then uses reference chunks and intermediate pipeline outputs to determine the earliest stage that exhibits erroneous behavior. RAGEC therefore supports the post-hoc classification of known faults when reference data are available. In contrast, RAGLog determines whether a fault has occurred and localizes the faulty stage without reference answers or reference evidence at run time, and without knowing that an answer is incorrect.

RAG-Critic [19] trains a critic model, based on a hierarchical taxonomy of RAG faults, that evaluates a response and attributes its faults to retrieval or generation, and uses this feedback to correct the response. It operates without reference answers, but distinguishes only retrieval from generation faults, without separating the steps before generation, and only reports faults where it judges the final answer to be incorrect. Doctor-RAG [29] diagnoses agentic RAG runs that are known to have failed, identifies the earliest failing step of the trajectory, and repairs the run from that step. AgenticRAG-FP [51] is a benchmark for causal failure attribution in agentic RAG, which injects a fault at a known retrieval hop and evaluates whether diagnostic methods identify that hop; it attributes failures to hops of a multi-hop trajectory rather than to the stages of a pipeline. Sok et al. [59] propose MetaRAG, a reference-free, black-box approach for detecting hallucinations in RAG-generated answers. MetaRAG decomposes an answer into atomic factoids, generates synonym and antonym mutations, and verifies these mutations against the retrieved context. Although MetaRAG shares RAGLog’s reference-free setting, it focuses exclusively on the grounding of the generator’s answer, and it cannot determine whether an earlier rewriter, embedder, retriever, or reranker produced faulty context. RAGLog instead analyzes the information flow across five pipeline stages to detect and localize faults in all of them, including faults that propagate to later stages and faults that leave the final answer correct.

Fault localization in LLM and agentic systems

Fault localization techniques also exist for general LLM and agentic systems. Notable ones among these are RAFFLES [65] and CHIEF [61]. RAFFLES locates faults in LLM systems with multiple components by using an LLM judge to evaluate all components and custom evaluators to assess the quality of the faults it detects and the rationale provided by the judge. CHIEF organizes a complex execution trace into a structured causal map and then traces backward through it to locate the root cause of a failure. AgentSight [64] is an instrumentation-free observability framework for agentic systems. It intercepts TLS-encrypted LLM traffic to extract semantic intent, monitors system behavior through kernel events, and links these two streams across processes to detect prompt injections, resource-wasting reasoning loops, and coordination bottlenecks in multi-agent systems. Chan et al. [11] discuss categories and potential implementations of visibility into AI agents, including agent identifiers, real-time monitoring, and activity logging. More generally, log analysis is a well-established technique for debugging and fault localization in software systems [12, 24]. These techniques, however, do not understand the roles of the stages of a RAG pipeline; they cannot check whether the information that a question needs was retrieved, passed on by the reranker, and used by the generator, which is where most RAG faults become visible.

Fault injection and datasets

Fault injection is an established technique for evaluating the robustness of software and deep-learning systems [26, 41]. For RAG pipelines, RAGProbe [58] generates question scenarios to expose failures, while Tanhai [60] injects controlled faults to demonstrate that answer-level evaluation can miss faults in earlier stages. In contrast, our fault injector modifies the input or output of a single stage during pipeline execution by intercepting API calls, without changing the pipeline’s code. This allows subsequent stages to process the modified data, enabling us to examine how faults propagate or are absorbed. Existing RAG datasets primarily label answer correctness or hallucinations [44, 54], rather than identifying the faulty pipeline stages. RAGLog Faults addresses this gap by providing stage-level fault labels for both natural and injected runs across five real RAG pipelines.

Conclusion and Future Work

We presented RAGLog, a reference-free approach for detecting and localizing faults across RAG pipeline stages using execution logs, without code instrumentation. RAGLog combines rules and LLMs to classify API calls and transform logs into a RAG schema instance. It then uses rule-based, embedding-based, and LLM-based checks with an ensemble of learned classifiers to identify faulty stages, allowing pipelines from different providers to be analyzed without manual configuration.

To evaluate RAGLog, we developed a logging proxy and a live fault injector. We also collected RAGLog Faults, a dataset of 3,134 runs from five open-source RAG pipelines (LightRAG, Onyx, Quivr, RAGFlow, and Verba) across three public datasets (DragonBall, Natural Questions, and MS MARCO), with stage-level fault labels. We compared RAGLog against two general-purpose coding agents, Claude Code and Codex, which analyzed the same execution logs. With Claude Haiku 5.5 as its underlying LLM, RAGLog achieved a pooled F1 of 0.73 on natural runs, outperforming Claude Code (0.58) and Codex (0.52). On injected runs, all three achieved comparable effectiveness (0.84, 0.80, and 0.82, respectively). RAGLog also achieved overall lower cost and latency than both agents, even when compared with Claude Code using the same underlying LLM. RAGLog with Claude Haiku 5.5 analyzed each run in approximately five seconds at US$0.005.

Beyond these comparisons, our results provide insight into RAGLog’s effectiveness and the importance of stage-level fault localization. More than one-third of naturally faulty runs still produced correct answers, showing why evaluating only the final answer is not sufficient. RAGLog addresses this by combining measurements across stages. The ablation study confirms the importance of LLM-based checks and suggests a smaller contribution from cross-stage information. Its fault-localization effectiveness also remained relatively consistent across three variants using LLMs from different providers. Meanwhile, the hybrid classification and transformation approach handled most API calls through rules, limiting LLM assistance to unfamiliar cases. We release RAGLog as an open-source tool with a dashboard for debugging, evaluation, and monitoring, together with the RAGLog Faults dataset.

In the future, we plan to expand RAGLog’s training data across additional pipelines, providers, and domains. We plan to extend the fault injector to support more diverse fault types, providing broader training and evaluation data. We also plan to include human annotations for a sample of runs to assess agreement with the LLM labels. In addition, we plan to consolidate overlapping generator checks to reduce analysis costs and extend RAGLog to data ingestion stages and agentic RAG pipelines.

Acknowledgments

Naveed is supported by a Faculty of IT Post-graduate scholarship. Grundy and Haggag are supported by ARC Laureate Fellowship FL190100035. Haggag is also supported by a National Intelligence Post-doctoral Fellowship. This work is also partly supported by ARC Discovery Project DP200100020. ChatGPT and Claude were used to assist with plotting scripts and academic writing. Claude was also used to assist with portions of the RAGLog implementation and evaluation scripts. The authors reviewed the AI-assisted outputs and take full responsibility for the content of this work.

Data Availability

The RAGLog Faults dataset is available at https://doi.org/10.57967/hf/11214. The RAGLog Tool is available at https://github.com/hiraa221/raglog. A complete replication package is available at https://doi.org/10.5281/zenodo.23792939. Received 31 October 2026