betaFeedback

WorksComputer Science

Decision Models as Judges for Hallucination Detection

Abstract

Large language models are widely used as judges of model outputs (LLM-as-a-Judge), but they are costly and can hallucinate or be overconfident. Decision models are a potential alternative. They return a predefined verdict with a probability for each label instead of free-form text and are much cheaper and faster. We test this promise on hallucination detection. We build binary benchmarks for five hallucination types (9,852 items from six datasets) and compare five decision models with the generative judge gpt-6-sol. Performance depends strongly on the type: the best decision model matches gpt-6-sol in balanced accuracy on context inconsistency and comes within 3 percentage points on factual errors, but trails it by 4-8 points on logical and instruction inconsistency. Decision-model probabilities are overconfident, though less so than the GPT-6 models, and separate correct from incorrect decisions only moderately. We also find that a confidence cascade of decision models improves on the best single model by 1.9-3.3 points on three types and matches or exceeds gpt-6-sol at 1-3% of its cost. Overall, decision models are a practical low-cost judge for some hallucination types but need to be validated before use.

Introduction

Large language models (LLMs) are widely used as judges of model outputs, an approach often called LLM-as-a-Judge [39] (Gu et al. 2026). They rate open-ended responses [32, 2], check whether responses follow instructions [33, 38], and detect hallucinated content [18, 29, 37, 22]. However, LLM judges are not fully reliable themselves. They can hallucinate, produce incorrect or unsupported judgments,

and be overconfident in their judgments [16, 30].

Recently released decision models offer a different kind of judge. Models such as jev (TypeSafe AI, 2026), d1 [1], and Perplexity Decider [3] do not generate free-form assessments but choose among predefined verdicts and return a probability for each. This design has two potential advantages. First, providers present these probabilities as a measure of certainty, so reliable probabilities can show when a judgment should not be trusted (TypeSafe AI, 2026). Second, decision models are much cheaper and faster than generative LLMs, which matters for large-scale evaluation [19, 26, 13].

Whether these advantages hold is still unclear. Recent studies of decision models as judges each examine a single model [19, 26], leaving several questions open. First, judge performance varies across tasks [36], but how consistently decision models perform across error types is unknown. Second, whether their probabilities are calibrated and separate correct judgments from errors has merely been tested [34]. Third, multi-judge studies [32, 21] have not tested ensembles composed solely of decision models for hallucination detection.

We address these gaps in the context of hallucination detection. First, judge reliability matters most here, since a judge that hallucinates while checking for hallucinations defeats its purpose. Second, hallucination types require different evidence (world knowledge, sources, the response itself, or the prompt), so they test whether decision models can generalize. We ask three research questions:

  • RQ1. How accurately do decision models detect different hallucination types?

  • RQ2. How reliable is their confidence?

  • RQ3. Can ensembles of decision models improve detection accuracy?

We evaluate five decision models against the generative LLM gpt-6-sol on 9,852 samples from six datasets based on five hallucination types ([12]). Our results show that decision models can match gpt-6-sol on some hallucination types at a small fraction of its cost, and that most of them are only mildly overconfident. Confidence cascades of decision models further improve the accuracy of hallucination detection. Overall, our results show that decision models are a practical low-cost judge for some hallucination types but should be validated before use.

Related Work

LLM-as-a-Judge and Decision Models. LLM-as-a-Judge is widely used for automated evaluation. LLM judges can align with human evaluations ([23]) but also show biases such as position and verbosity bias ([39, 20]), and because they generate text, they add cost and latency. In contrast, decision models provide a non-generative alternative by directly selecting from predefined labels and returning their associated probabilities. Recent work evaluates jev on HaluEval factuality and final-answer adjudication ([19]), as well as on rubric criteria ([26]), and reports lower cost and latency than generative LLM judges. Ibrahim and Zaki [13] compare several decision models on social-science annotation.

Hallucination detection. Detecting a hallucination requires evidence depending on its type ([12]). FAVABench and HADTest annotate factual errors ([24, 37]), RAGTruth annotates unsupported or contradictory content in retrieval-grounded responses ([25]), ContraDoc and ROSCOE cover contradictions within documents and reasoning chains ([17, 9]), and IF-RewardBench covers instruction following ([33]). AggreFact shows that factuality detector performance varies across datasets, summarizers, and error types ([28]).

Confidence reliability. Returning probabilities does not make a judge’s confidence reliable. Calibration asks whether probabilities match observed frequencies ([10]), and error ranking asks whether confidence separates correct judgments from errors. Language-model confidence can reflect correctness, but this depends on the task and on how confidence is elicited ([16, 30, 35]). For decision models, calibration varies by workload ([19, 26]), and high confidence can coexist with near-chance accuracy ([13]).

Judge ensembles and cascades. Combining judges helps when their errors differ. PoLL aggregates judgments from diverse generative models ([32]), and CLEV calls a third judge only when the first two disagree ([2]), the idea behind our arbitration method. Confidence-based cascades pass uncertain items to another judge ([15]). JEV-to-generative cascades can match the generative judge at lower cost ([19]), although overlapping errors limit their gains on rubric tasks ([26]).

Three areas remain underexplored. First, decision models have so far been evaluated one at a time or on tasks other than hallucination detection, so it is unclear how well they detect each hallucination type. Second, few studies have examined whether the confidence of decision models is reliable across models and hallucination types. Third, it has not been tested whether ensembles built only from decision models can improve hallucination detection while keeping their cost low. Our study aims to address these research gaps.

Data and Methods

Benchmark Composition

Hallucination types. Hallucination takes several forms. Following the taxonomy of Huang et al. [12], we study five types (Table 1). Factual contradiction (FC) covers claims that contradict verifiable real-world facts. Factual fabrication (FF) covers unverifiable or unsupported overclaims. Context inconsistency (CI) covers claims that conflict with the supplied source material or are not supported by it. Logical inconsistency (LI) covers contradictions within a response or between its reasoning steps. Instruction inconsistency (II) covers violations of explicit instructions. We treat each type as a separate binary task, because one response can contain several types of error.

Hallucination typeOperational definitionDatasetsHHNNTotal
Factual contradiction (FC)Claims contradict verifiable real-world facts through incorrect entities or relations.FAVABench, HADTest3441,2771,621
Factual fabrication (FF)Claims introduce fabricated, unverifiable, or unsupported facts or overclaims not established by real-world knowledge.FAVABench, HADTest3391,2771,616
Context inconsistency (CI)Claims conflict with, or are unsupported by, the supplied context or retrieved evidence.RAGTruth9431,7322,675
Logical inconsistency (LI)Claims or reasoning steps within the same response are mutually inconsistent.ContraDoc, ROSCOE5381,1081,646
Instruction inconsistency (II)The response violates or fails to satisfy explicit instructions or constraints.IF-RewardBench (English)1,6006942,294
Total3,7646,0889,852

Table 1. Hallucination-detection tasks and datasets. HH marks samples that contain the hallucination and NN do not.

Benchmark datasets. We choose six benchmarks whose annotations mark the target error type explicitly, so each sample maps to a binary label without new annotation ([28, 29]). For FC and FF, we combine two datasets that annotate both types, given that FAVABench alone has fewer than 400 samples per type. FAVABench contains ChatGPT and Llama responses with human span-level error annotations ([24]). HADTest contains responses into which GPT-4o injected a single factual error, validated by two annotators ([37]). For CI, RAGTruth contains responses from six LLMs to question answering, summarization, and data-to-text tasks, annotated for content that conflicts with the retrieved source or is unsupported by it ([25]). For II, IF-RewardBench contains responses from 16 LLMs, each checked against a list of explicit constraints ([33]). For LI, ContraDoc contains news, Wikipedia, and story documents with inserted self-contradictions ([17]), and ROSCOE contains step-by-step reasoning annotated for contradictions ([9]).

An item is labeled HH if its source annotation marks the target error and NN if it marks the absence of that hallucination error. An NN sample may therefore still contain another type of hallucination. We keep each dataset’s evaluation unit and do not rebalance the classes. Appendix § 6.§ 3 details how we map each dataset’s original annotations to the binary labels used in our evaluation.

Evaluation Setup and Metrics

Model selection. We select five decision models from five providers, including both proprietary and open-weight models. They are jev, dl, clef, pplx, and gpt-6-luna. Details of these models are presented in Appendix § 7. For comparison, we also run the general-purpose generative LLM gpt-6-sol on the same items, distinguishing it from gpt-6-luna, which is included as a decision model because it directly outputs classification decisions. All six models are accessed through their hosted APIs. Latency is therefore end-to-end API response time and does not depend on local hardware.

Model input and output. For each hallucination type, all models receive the same input and the same H/NH/N criterion, based on the type’s definition. The input consists of the response to be judged and, when available, the original instruction. For CI, it also includes the source context. The models never see the gold labels or the annotated error spans. Each model answers HH if the response contains the target error and NN otherwise, and returns probabilities P(H)P(H) and P(N)P(N) that sum to one. The answer is the label with the higher probability, and on an exact tie we keep the model’s own answer. The decision models return these probabilities through their interfaces. For gpt-6-sol, we ask for a structured verdict with its own probability estimates and no explanation. Appendix A.4 provides the complete decision-model prompts.

Evaluation metrics. The classes are imbalanced (Table 1), so we compare each answer with the gold label and report balanced accuracy (BAcc), the mean recall over HH and NN, and macro-F1, the mean F1 over the two classes. Differences between judges are tested with paired bootstrap over items (2,000 resamples) [7]. We also report the mean API latency per call and the estimated API cost. Cost is computed from recorded token usage at the providers’ list prices at the time of evaluation.2

Confidence Analysis

Probabilities and confidence. For each item ii, a decision model returns pi=P(H∣xi)p_i = P(H \mid x_i), the probability that the response contains the target hallucination type, and labels the response as hallucinated (y^i=1\hat{y}_i = 1) when pi>0.5p_i > 0.5. Providers describe pip_i as an estimate of the probability that the predicate holds or as a calibrated score. The confidence of a decision is the probability of its predicted label,

ci=max⁡(pi,1−pi)∈[0.5,1].(1)c_i = \max(p_i, 1 - p_i) \in[0.5, 1]. \tag*{(1)}

Under perfect calibration, predictions assigned confidence qq would be correct approximately qq of the time. We evaluate the extent to which this property holds in practice.

Detection and calibration. Detection performance, i.e., how well pip_i ranks hallucinated above non-hallucinated responses, is measured by AUROC. At τ=0.5\tau= 0.5, we also report BAcc, MCC and the flag-rate bias Δ\Delta (defined as the share of responses flagged minus π\pi (Δ>0\Delta> 0: over-flagging)). Calibration is measured by the expected calibration error over M=10M = 10 equal-width confidence bins [10, 27]:

ECE=∑m=1M∣Bm∣n∣acc⁡(Bm)−conf⁡(Bm)∣.(2)\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} \left|\operatorname{acc}(B_m) - \operatorname{conf}(B_m)\right|. \tag*{(2)}

where BmB_m is the set of predictions in confidence bin mm. We also report the slope bb of the logistic model Pr⁡(yi=1)=a+blogit⁡(pi)\Pr(y_i=1) = a + b\operatorname{logit}(p_i), where b<1b < 1 indicates overconfidence [4].

Confidence as an error signal. A useful confidence score is expected to be higher for correct decisions (CC) than for errors (E\mathcal{E}) [11, 8]:

conf-AUROC⁡=Pr⁡(ci>cj∣i∈C,j∈E),(3)\operatorname{conf\text{-}AUROC} = \Pr(c_i > c_j \mid i \in C, j \in\mathcal{E}), \tag*{(3)}

with ties counted as one half. A value of 0.5 means that confidence cannot distinguish correct decisions from errors. We also report the share of errors made with high confidence (ci≥0.9c_i \ge0.9).

We report 95% bootstrap intervals and compare models on the same items with paired bootstrap tests. A two-way ANOVA (η2\eta^2) separates cell and model effects, and Kendall’s WW measures ranking consistency across cells.

Model Ensembling Strategies

Motivated by prior studies [32, 2, 15], we evaluate three ensembling methods (Figure 1) to assess whether they improve accuracy and at what additional cost. To isolate the effect of each method, we fix the model selection. For each type, all three methods combine the same three decision models, namely those with the highest BAcc on that type.

Figure 1. The single-model baseline and the three combination rules. Models 1–3: top-3 decision models by BAcc per type. Solid/dashed boxes: always/conditionally called. (c) The third model breaks ties between the two cheaper ones. (d) Each stage defers its least-confident rkr_k share; the last stage decides the rest.

Majority voting. All three models judge the same item, and the label chosen by at least two of them is the verdict. Because every item gets all three calls, majority voting is the most expensive of the three methods. Voting gives every model the same weight, so two weaker models can outvote the strongest one.

Arbitration. When two models agree, the outcome is their shared label, and the third vote does not change it. When they disagree, the third vote decides the outcome. Arbitration therefore calls the two models first and calls the third only to resolve a disagreement [2].

Confidence cascade. A cascade lets the strongest model decide unless it is uncertain. Cascades usually order models from cheap to expensive to reduce calls to a costly final model [6, 15]. We order the models by descending BAcc and ask whether consulting weaker models on the strongest model’s least confident items improves BAcc [14]. At each of the first two stages, a model defers an item to the next model when its confidence cic_i falls below a threshold, and the third model decides all items that reach it. Because confidence distributions differ across models (§ 4.2), a common threshold would defer very different shares of items at different stages. We therefore set each threshold through a target deferral rate rkr_k, the share of items reaching stage kk that are deferred, so that each stage defers its least confident items without using gold labels. We evaluate r1,r2∈{0,0.1,…,0.5}r_1,r_2 \in\{0,0.1,\ldots,0.5\}: r1=0r_1=0 reduces the cascade to the strongest model, and above 0.5 the strongest model would decide fewer than half of the items. As a control, we defer the same numbers of randomly chosen items.

Evaluation. For each hallucination type, we evaluate all 6×66 \times6 combinations of r1r_1 and r2r_2 and report the one with the highest BAcc. Because we choose this combination on the same items we evaluate, the result is an optimistic estimate of what a cascade can achieve [5]. To see how optimistic it is, we also choose the rates with five-fold cross-fitting, picking them on four folds and testing them on the fifth. The cost of an item is the total cost of all models called for it. For latency, we add up calls made one after another and take the slowest of calls made in parallel. We plot each method’s BAcc against its cost, show the results of all 36 combinations separately, and use paired bootstrap tests to compare each method with its best single model and with gpt-6-sol.

Results

RQ1: How Do Decision Models Perform Across Hallucination Types?

Table 2 and Figure 2 present the performance over each hallucination type. Appendix C reports their performance judge on every source dataset.

Type (N)ModelBAccMacro-F1CostLatency
FC N=1,621jev75.0174.960.0490264.51
FC N=1,621d167.7868.980.0295224.15
FC N=1,621clef72.1273.950.2064347.49
FC N=1,621pplx78.9378.480.0159284.23
FC N=1,621gpt-6-luna77.7568.360.0817236.82
FC N=1,621gpt-6-sol81.6177.453.33201,825.76
FF N=1,616jev80.7474.110.0570239.63
FF N=1,616d170.5174.400.0369235.40
FF N=1,616clef71.2475.020.2498354.56
FF N=1,616pplx79.8582.140.0195290.10
FF N=1,616gpt-6-luna82.7275.150.0999236.64
FF N=1,616gpt-6-sol82.0581.363.50141,639.48
CI N=2,675jev82.5382.090.2214281.30
CI N=2,675d175.8477.330.1797319.34
CI N=2,675clef68.2169.381.0911622.86
CI N=2,675pplx82.6684.550.0901342.02
CI N=2,675gpt-6-luna64.0453.010.4375258.33
CI N=2,675gpt-6-sol82.6578.9410.96992,002.29
LI N=1,646jev68.9870.470.0733284.50
LI N=1,646d158.4656.950.0509330.27
LI N=1,646clef61.3361.420.3398515.09
LI N=1,646pplx73.2875.120.0270284.83
LI N=1,646gpt-6-luna66.9968.270.1356281.61
LI N=1,646gpt-6-sol77.1478.324.48312,203.56
II N=2,294jev61.7562.030.1403276.94
II N=2,294d162.5062.020.1114478.96
II N=2,294clef64.5064.430.7021593.57
II N=2,294pplx71.0865.390.0566306.38
II N=2,294gpt-6-luna54.1750.690.2846263.23
II N=2,294gpt-6-sol78.8878.438.50982,644.51

Table 2. Single-judge results pooled by hallucination type. BAcc and Macro-F1 are percentages; Cost is the estimated total API cost for that type (USD), and Latency is mean per-call latency (ms).

Figure 2. Judging performance by hallucination type. Bars show the pooled BAcc of each judge per type, with 95% bootstrap confidence intervals. Hatched bars mark the generative model gpt-6-sol.

Performance depends strongly on the type. gpt-6-sol has the highest pooled BAcc on FC, LI,

and II (Table 2). On FF, gpt-6-luna (82.7) edges it (82.0). On CI, pplx, gpt-6-sol, and jev are within 0.2 points of one another at about 82.6. The gap is widest on II, where gpt-6-sol reaches 78.9 against 71.1 for pplx and 61.8 for jev. Performance also varies within a model. jev spans 20.8 points between II and CI. Type and source dataset are confounded, so we read these differences as type-specific without attributing them to the type alone.

Similar scores hide different errors. The matched CI scores arise from different error patterns. pplx rarely raises false alarms (false-positive rate 3.2%) but misses many hallucinations (false-negative rate 31.5%). gpt-6-sol shows the reverse pattern, with a false-negative rate of 5.9% and a false-positive rate of 28.8%. jev lies between them at 20.6% and 14.4%. The dominant error also changes with the task. jev misses most positives on LI, with false-negative rates of 49% on ContraDoc and 81% on ROSCOE, but over-flags on II, where its false-positive rate is 56%. BAcc alone therefore does not show whether a judge mostly misses hallucinations or mostly raises false alarms.

Model advantages are concentrated in particular types. Relative to jev, pplx gains only 0.1 points on CI (95% paired bootstrap interval [−1.4, 1.6]) and loses 0.9 points on FF ([−3.4, 1.6]), but it gains 3.9 points on FC ([1.5, 6.4]), 4.3 points on LI ([2.2, 6.5]), and 9.3 points on II ([7.2, 11.5]). The FC, LI, and II gains remain significant after Holm correction over the 25 comparisons of each judge with jev. The ranking of decision models thus changes with the type.

Advantages do not always hold across sources. On HADTest, all six judges score 12 to 18 points higher on FF than on FC. The same comparison on FAVABench shows no consistent gap (Appendix § 8). For pplx, FF minus FC is −1.4 points on FAVABench (95% bootstrap interval [−8.5, 5.3]) but 17.9 points on HADTest ([13.2, 22.9]). An advantage on fabrication is therefore partly a property of the source. CI and II each have a single source, so their cross-source stability remains untested.

Decision models are far cheaper and faster. Per item, the decision models cost between 0.5% (pplx) and 10% (clef) of gpt-6-sol, and they answer in 0.22 to 0.62 seconds against 1.6 to 2.6 seconds (Table 2). The cheapest model, pplx, is also the most accurate decision model on four of the five types, and gpt-6-luna is the most accurate on FF. The efficiency gain is thus available on every type. Whether it comes with the accuracy of gpt-6-sol depends on the type. Additional analyses of error patterns and prediction overlap across models are provided in Appendix § 9.§ 2.

RQ2: How Reliable is Their Confidence?

Figures 3 and 4 present the confidence reliability of the six judges across hallucination types, including

calibration and the ability to distinguish correct judgments from errors.

Decision models are overconfident overall, but less so than the GPT-6 models. Every decision model has a positive average overconfidence gap, and the calibration slope is below 1 for 21 of the 25 model–type pairs (Figure 3a). For jev, pplx, clef and d1, however, the overconfidence is mild. Mean confidence exceeds accuracy by 0.04–0.12, and ECE ranges from 0.083 to 0.136. The two GPT-6 models are clearly more overconfident. gpt-6-luna has a gap of 0.18 and an ECE of 0.187, and the comparison LLM gpt-6-sol has a gap of 0.16 and an ECE of 0.157. Their median calibration slopes (0.44 and 0.36) are also the lowest, compared with 0.54–0.58 for jev, d1 and pplx and 0.96 for clef.

Figure 3. Confidence by model and hallucination type. (a) Overconfidence gap, i.e., mean confidence minus accuracy. (b) conf-AUROC; outlined values have a 95% CI that includes 0.5.

Calibration depends more on the dataset than on the model. Across the decision models, the dataset accounts for 58% of the variance in the overconfidence gap, and the model for 15% (Figure 3a). Averaged over the decision models, overconfidence is largest for II (0.14) and LI (0.13), and smallest for CI and factual contradiction FC (0.07 each). The reliability diagrams show the same pattern. At a given confidence level, accuracy differs by hallucination type (Figure 4). For example, the accuracy of gpt-6-luna on CI stays below 0.50 for all confidence levels between 0.55 and 0.95.

Figure 4. Reliability diagrams by hallucination type. Confidence is divided into 10 equal-width bins (bins with fewer than 25 items omitted); points below the diagonal indicate overconfidence. Right: ECE and conf-AUROC (cAUC) averaged over datasets, and the share of all errors (pooled over datasets) made with c≥0.9c \ge0.9 (Err≥\ge.9).

Confidence is less reliable for instruction inconsistency (II) and logical inconsistency (LI). Averaged over the decision models, II has the largest overconfidence gap (0.14), the highest ECE (0.145) and the lowest conf-AUROC (0.64), followed by LI (gap 0.13; ECE 0.136; conf-AUROC 0.70). FC is less overconfident (gap 0.07; ECE 0.103) but separates correct from incorrect decisions no better than LI (conf-AUROC 0.70), whereas CI (gap 0.07; ECE 0.093; conf-AUROC 0.74) and FF (gap 0.08; ECE 0.134; conf-AUROC 0.75) are the most reliable. Figure 3 shows the same pattern. Among decisions made with c≥0.9c \ge0.9, accuracy is lowest for II for jev and pplx, and for LI for clef and d1 (Figure 4). Type-specific failures also occur for individual models. gpt-6-luna is most overconfident on CI (gap 0.31), where its accuracy stays below 0.50.

Confidence separates correct from incorrect decisions only moderately. conf-AUROC measures how well confidence separates correct judgments from errors. A value of 0.5 indicates performance no better than chance. Higher values indicate better separation between the two groups (Figure 3b). Averaged over the five hallucination types, conf-AUROC ranges from 0.67 (gpt-6-luna) to 0.73 (pplx and clef). For each model and type, the 95% confidence interval lies above 0.5, so confidence always carries some information about correctness, but the separation remains moderate, especially for II.

High confidence does not guarantee a correct decision. Errors can be made with high confidence (c≥0.9c \ge0.9) (Figure 4 and Figure 7). Averaged over the five types, gpt-6-luna assigns c≥0.9c \ge0.9 to 65% of its decisions, is correct on only 77% of them, and makes 47% of its errors at this level. For the other decision models, 10–30% of errors are made at c≥0.9c \ge0.9, with clef lowest (10%); clef also assigns such high confidence to only 25% of its decisions. The comparison model gpt-6-sol is the most extreme case: it assigns c≥0.9c \ge0.9 to 88% of its decisions and makes 69% of its errors at this level.

Figure 7. Confidence distributions of correct decisions and errors. For each model, the share of correct decisions (gray bars) and of errors (red outline) in each confidence bin. The results are averaged over the five hallucination types, with each dataset weighted equally within its type.

RQ3: Can Ensembling Improve the Performance?

Figure 5 presents the performance–cost trade-offs of the three ensembling methods. Deferral-rate sensitivity is reported in Appendix Figure 8.

Figure 5. BAcc against estimated cost for each hallucination type. Cost is relative to gpt-6-sol on a log scale, and the dashed line marks the BAcc of gpt-6-sol. The cascade uses the deferral rates with the highest BAcc. The circle defers the same numbers of items at random.

Figure 8. Cascade performance across deferral rates. The x-axis varies r1r_1; each colored line fixes r2r_2. Both rates range from 0% to 50% in 10-percentage-point steps (36 configurations per type). Stars mark the highest BAcc, dashed lines mark gpt-6-sol, and dotted lines mark the strongest member alone. Members appear in stage order below each panel. Peaks are selected and evaluated on the same items; they are post-hoc maxima, not held-out results.

The cascade matches or exceeds GPT-6-sol on three types. At the best tested deferral rates, the confidence cascade achieves BAcc scores of 82.2, 84.6, and 84.6 on FC, FF, and CI, respectively—0.6, 2.6, and 1.9 points higher than gpt-6-sol—at only 1.2–3.3% of its estimated cost. The advantage over gpt-6-sol is statistically significant on FF and CI, but not on FC. The cascade also improves on the strongest single member by 3.3, 1.9, and 2.0 points, respectively, with paired bootstrap 95% intervals excluding zero. It makes 1.1–1.3 calls per item and has about one fifth of the latency of *gpt-6sol. Selecting deferral rates by cross-fitting reduces BAcc by at most 0.52 points and preserves the gains over the strongest member.

The cascade outperforms voting and arbitration. Majority voting gains about one BAcc point over the strongest single model on FC and FF, but falls below it on CI, LI, and II (Figure 5). Disagreement arbitration produces the same verdicts as voting while costing roughly half as much. Both rules are less accurate than the cascade on every type and, except on FF, also cost more. Unlike equal voting, the cascade consults weaker members only when the strongest member is uncertain.

The best cascade configuration differs by type. pplx leads four of the five cascades. On FF, gpt-6-luna leads (Appendix Table 12). The role of the later stages also varies. On CI, the best cascade never uses its third model (r2=0r_2 = 0), whereas on FF the second model defers about half of the samples it receives. On FC, FF, and CI, BAcc peaks at a first-stage deferral rate of 20–30% and declines at higher rates (Appendix Figure 8). A single cascade configuration therefore does not transfer across types.

The gap persists on LI and II. On LI, the best cascade defers no items and equals pplx alone, 3.9 points below gpt-6-sol. On II, the cascade improves BAcc by only 0.7 points over the strongest single model, pplx, and remains 7.1 points below gpt-6-sol. Other members do not improve on pplx for its least confident items in balanced terms, consistent with the weaker confidence reliability on these two types (§ 4.2).

Discussion and Conclusion

We evaluate five decision models for hallucination detection. They match or approach the generative LLM judge on factual and context-related errors but lag on logical and instruction inconsistencies requiring multi-step reasoning or explicit constraint checking. We draw three takeaways.

First, judges should be selected per hallucination type. No decision model performs best across types, and similar BAcc can mask opposite error profiles: missed hallucinations or false alarms. The preferable judge thus depends on the target type and on which error is more costly, so false-negative and false-positive rates should be reported by type. Since type and source are partly confounded, judge selection should be validated on the target task.

Second, confidence is useful but does not guarantee correctness. Most decision models are less overconfident than gpt-6-luna and gpt-6-sol, but their confidence only moderately distinguishes correct from incorrect judgments, with frequent high-confidence errors on logical and instruction inconsistencies (§ 4.2). Confidence can guide requests for a second opinion but cannot establish correctness.

Third, ensembling decision models helps when they are complementary on uncertain samples. Cascades outperform their strongest member and match or exceed gpt-6-sol on factual contradiction, factual fabrication, and context inconsistency. On logical and instruction inconsistency, the tested cascades offer limited gains, consistent with the observed error overlap (Appendix § 9.§ 2). Since the best configuration varies by type, deferral rates should be validated on the target type to weigh accuracy gains against the cost of additional model calls.

At 0.5–10% of the cost of gpt-6-sol, decision models could support low-cost, type-aware cascades for factual and context errors, with LLM judges reserved for logical and instruction inconsistencies. Future work should disentangle type from data source, develop decision models that target reasoning- and instruction-level errors, and learn routing policies that decide when to escalate to an LLM judge based on detection accuracy and cost.

Limitations

This study presents several future directions.

Benchmark coverage and label consistency. Our benchmark covers five hallucination types across six datasets, but coverage is uneven: context and instruction inconsistency each rely on a single source. Differences in data construction, domain, and response length confound cross-dataset comparisons, leaving subgroup analyses observational. Mapping heterogeneous annotations to binary labels may also introduce mismatches because the source datasets define errors differently. Broader source coverage, annotation under a shared rubric, and comparisons matched on domain and response length would help address these limitations.

Confidence comparability and prompt sensitivity. Confidence values are not fully comparable because decision models return label probabilities, whereas gpt-6-sol reports its own estimates. We also use one prompt design per task, leaving prompt sensitivity untested. Comparing alternative prompts and, where supported, confidence-elicitation methods within each model would clarify their effects on accuracy and calibration.

Generalization and deployment of cascades. We evaluate fixed model versions and assess cascades through offline replay and within-source cross-fitting, with member selection and ordering based on the full sample. Selecting all cascade components using training data alone, followed by evaluation on unseen domains and in live deployments, would better assess their accuracy, latency, and cost beyond the evaluated settings.

Statement of LLM Use

During the preparation of this paper, we used LLM-based tools (primarily Claude and ChatGPT) in two capacities. First, we used coding agents to assist with the development and refinement of the project codebase. Second, we used LLM chatbots to support language editing of manuscript drafts, including correcting grammatical errors, improving sentence structure, and enhancing the clarity of presentation. The authors take full responsibility for the content of this paper, including the code, analyses, and conclusions presented.

Benchmark Construction, Labels, and Prompts

Hallucination Taxonomy and Examples

Table 3 lists the five hallucination types, their definitions, and benchmark examples.

Type and definitionExample dialogAnnotation
Factuality hallucination
Factual Contradiction<br>States facts that can be grounded in real-world knowledge but contradict it, through a wrong entity or relation.Instruction: How does the EU work?<br><br>Response: The EU also has its own currency, the euro, which is used by 19 member states.FAVABench, entity error; the correct number is 20.
Factual Fabrication<br>States facts that cannot be verified against established knowledge, or overclaims without broad support.Instruction: Name three important inventions of the 20th century.<br><br>Response: The three important inventions of the 20th century are the airplane, the automobile, and the chromoflare device, which enabled seamless digital color transitions in electronic displays.HADTest, fabricated entity; no such device exists.
Faithfulness hallucination
Context Inconsistency<br>Is unfaithful to the context supplied by the user, such as a retrieved document.Retrieved context: Three of the alleged attackers were killed, Israeli media reported, citing IDF sources. It was not immediately known to what militant group the men belonged.<br><br>Instruction: Summarize the following news within 38 words.<br><br>Response: Israeli jets struck targets in the occupied Golan Heights after militants attempted to plant a bomb near the Israeli border. Four alleged attackers were killed, but their affiliation remains unclear.RAGTruth, evident conflict; the source reports three deaths.
Logical Inconsistency<br>Contains internal contradictions, between statements or reasoning steps, or between the steps and the conclusion.Document: The text of this novel of ideas presents itself as a book that has been written as the result of a promise to a dying man. William Porphyry Benham is a man who has lived a life devoted to a complicated, protean idea: “that he had to live life nobly and thoroughly.” The novel has been written as the result of a bet between two friends.ContraDoc; the last sentence contradicts the first.
Instruction Inconsistency<br>Deviates from an explicit, non-malicious user instruction.Instruction: what concentration of particles is bad for biosensors. The output should not contain any whitespace.<br><br>Response: Particle concentrations exceeding 1mg/mL or 10710^{7} particles/mL often cause biosensor interference depending on sensor type particle size and matrix.IF-RewardBench; whitespace violates the instruction.

Table 3. Hallucination taxonomy with examples from our benchmark. Definitions follow Huang et al. [12]. Each example is an HH item; hallucinated content is in bold.

Dataset Versions and Construction

Table 4 identifies the source snapshots, evaluation subsets, and resulting type-specific views. Revisions refer to the local source snapshots used to build the benchmark. Where an upstream revision was not retained, a SHA-256 fingerprint identifies the archived source file. Hashes below use the first 12 hexadecimal characters; the accompanying provenance file records the complete hashes and revision identifiers for the archived source snapshots.

DatasetVersion / source snapshotSplit and selectionInstances
FAVABenchannotations.json; SHA ef3b4b289762All 460 human-annotated passages; separate inclusive FC and FF views.FC: 377<br>FF: 351
HADTesthadtest.json; SHA fd13225bcc8dHuman-validated HADTest split only; HAD train and dev are not used.FC: 1,244<br>FF: 1,265
RAGTruthRevision c103204b9ce2Official test split, with quality="good"; join responses to source context by source_id.2,675
ContraDocRevision d1c42d1e4335Entire released pos/neg benchmark; these are label partitions, not train/test splits.891
ROSCOEOfficial annotations.zip; SHA 1fe5b4ee56a6All released CosmosQA, DROP, e-SNLI, and GSM8K annotations; exclude one empty chain.755
IF-RewardBenchRevision c192fe968c15English-filtered released benchmark, including all three instruction forms.2,294

Table 4. Dataset versions and evaluation subsets. Counts are type-specific evaluation instances. No class balancing or additional train/test split is applied to these judge evaluations.

Factuality mapping and reuse. For FAVABench, entity, relation, and contradictory map to FC; invented, unverifiable, and subjective map to FF, with subjective treated as overclaim. We reconstruct the annotated error-bearing passage by removing proposed corrections inside mark spans and stripping annotation tags while retaining their content, including delete spans. A response carrying both types is positive in both views. Negatives have none of the six factual error tags; responses carrying only the other type are excluded. Thus, 115 jointly positive and 153 jointly negative records are reused across FC and FF.

For HADTest, Factual Recall Error and Factual Inference Error map to FC, while Fabricated Entity and Fictional Attribution map to FF. The 1,124 None-labeled responses are negatives in both views; other categories are excluded. The conversion removes empty responses and conflicting-label duplicate texts, retaining the first occurrence of a same-label duplicate within a view. Reuse across types is intentional because the two tasks ask different questions of the same response.

Faithfulness subsets. RAGTruth is positive when at least one hallucination span is annotated, after the test/good filtering. ContraDoc includes both inserted and replacement-based contradictions. ROSCOE uses the whole-chain 0_full_newClearContradiction_result label (yes/no); this annotation also mentions incoherent reasoning, so the view is broader than strict contradiction. Its 756 released rows yield 755 evaluable chains after excluding one empty CosmosQA response. IF-RewardBench’s English filter retains instructions and checklists containing no characters in U+4E00–U+9FFF; it is a preprocessing heuristic, not a separate official language split. The retained responses comprise 984 single-turn, 574 multi-turn, and 736 system-prompt items. Any violated constraint maps to HH.

Counting convention. The total of 9,852 counts type-specific evaluation instances, not deduplicated answers. The 268 shared FAVABench records and 1,124 shared HADTest records account for the cross-type reuse. There are 8,460 distinct (dataset, source-record ID) pairs; this is an ID count, not global text deduplication. Judges receive complete responses, documents, or reasoning chains, without gold labels, annotated spans, or corrections.

Binary Label Mapping

Figure 6 shows how we map each dataset’s original annotations to binary labels. For each hallucination type, an item is labeled HH if its annotations indicate the target error and NN if they indicate its absence. FAVABench and HADTest provide annotations for both factual types, so we construct separate FC and FF views from each. In FAVABench, a response is negative only if it carries none of the six factual error tags. Responses tagged only with the other factual type are excluded. In HADTest, None-labeled responses serve as negatives for both FC and FF, and we exclude its other error categories. RAGTruth labels a response as HH if it contains at least one annotated hallucination span. ContraDoc uses its official split, and ROSCOE uses its whole-chain ClearContradiction label. In IF-RewardBench, a response is labeled HH if it violates any annotated constraint. We keep each dataset’s original evaluation unit. Because some annotations target a single error type, NN means only that the target error is absent, not that the output is error-free.

Figure 6. Mapping of source-dataset annotations to binary labels. HH: the target error is present; NN: it is absent. The bar shows each dataset’s H:NH:N ratio.

Decision-Model Prompts

Decision models have no separate system and user prompts. Each request carries a choice question, which is fixed for a hallucination type, and an item state, which is filled for each record. The decision models (jev, d1, clef, and pplx) receive both parts; gpt-6-luna receives the same text through the OpenAI Decisions API, with the state serialized as a JSON string and the H/NH/N criteria given as choice descriptions. Bracketed values are placeholders for the item text. A state field is sent only when it is non-empty. The shared request structure is shown below. The type-specific instructions and criteria are reproduced verbatim below; field headings are added for display, and the state fields retain the original placeholders.

Factual Contradiction (FC)

{
  "questions": {
    "target_hallucination": {
      "type": "choice",
      "instructions": "[Instructions]",
      "criteria": {
        "H": "[Criterion H]",
        "N": "[Criterion N]"}}},
  "state": {[State fields]}
}

Instructions: Review the entire candidate response. Determine whether it contains factual contradiction. Choose H if at least one factual claim satisfies the H criterion. Otherwise choose N. Evaluate only factual contradiction.

Criterion H: At least one factual claim conflicts with reliable, verifiable real-world information. This includes an incorrect entity, relation, date, number, location, attribute, event, or other factual statement. Reliable evidence must establish that the claim is false.

Criterion N: No factual claim conflicts with reliable, verifiable real-world information. A claim that is merely unsupported, uncertain, subjective, or difficult to verify does not count as factual contradiction.

State:

candidate_response: “[the model-generated answer being judged]”;

instruction: “[the original user question or prompt]”

Factual Fabrication (FF)

Instructions: Review the entire candidate response. Determine whether it contains factual fabrication. Choose H if at least one factual claim satisfies the H criterion. Otherwise choose N. Evaluate only factual fabrication.

Criterion H: At least one statement is presented as established fact and satisfies at least one of the following conditions: (1) Unverifiability: it presents an invented or nonexistent entity, event, source, study, quotation, or other concrete factual detail as established fact without a reliable basis in real-world knowledge; or makes a concrete factual claim that cannot be substantiated by available reliable sources; (2) Overclaim: it presents a subjective, disputed, speculative, or exaggerated assertion as a broadly established fact despite lacking substantial supporting evidence or broad consensus. An incorrect entity, identity, attribute, relation, date, or number is not factual fabrication if the error solely contradicts an established fact.

Criterion N: The response contains no claim that is entirely nonexistent, unverifiable using available reliable sources, or an overclaim lacking broad consensus or substantial evidence. Choose N when all factual claims are verifiable, or when an error consists only of an incorrect entity substitution or relation that can be directly checked against established real-world facts.

State:

candidate_response: "[the model-generated answer being judged]";

instruction: "[the original user question or prompt]"

Context Inconsistency (CI)

Instructions: Review the entire candidate response against the supplied context. Determine whether the response asserts at least one substantive factual claim that is inconsistent with or unsupported by that context. Choose H only if such a claim satisfies the H criterion. Otherwise choose N. Use the supplied context as the evidence source, and evaluate only contextual faithfulness rather than formatting or general instruction compliance.

Criterion H: At least one substantive factual claim in the candidate response contradicts the supplied context or introduces information that cannot reasonably be supported or inferred from the supplied context. Judge faithfulness to the context even if the introduced information might be true in the real world.

Criterion N: Every substantive factual claim in the candidate response is supported by, entailed by, or reasonably inferable from the supplied context. Also choose N when the response merely omits contextual details, declines or is unable to answer, uses harmless introductory or meta wording, or expresses a reasonable paraphrase, summary, abstraction, or inference. Formatting, length, style, and other instruction-following violations do not by themselves count as context inconsistency.

State:

candidate_response: "[the model output being judged]";

context: "[the RAGTruth source material]";

instruction: "[the RAGTruth task prompt]"

Logical Inconsistency (LI)

Instructions: Review the entire candidate response, including the relationships among its statements, reasoning steps, calculations, and conclusion. Determine whether it is internally logically inconsistent. Choose H if it contains any internal contradiction defined by the H criterion. Otherwise choose N.

Criterion H: Two or more parts of the response are mutually incompatible; the response contradicts one of its earlier statements; or its conclusion conflicts with its own premises, assumptions, calculations, or stated reasoning steps.

Criterion N: The statements, reasoning steps, calculations, and conclusion are internally consistent with one another. A factual error, unverifiable claim, lack of contextual support, or failure to follow an instruction does not by itself count as internal logical inconsistency.

State:

candidate_response: "[ContraDoc document | ROSCOE reasoning chain]";

instruction: "Determine whether the document is self-contradictory." | "Determine whether the response is internally logically inconsistent."

Instruction Inconsistency (II)

Instructions: Review the entire user instruction and candidate response. Determine whether the response fails to follow the instruction. Choose H if the response violates at least one applicable explicit requirement defined by the H criterion. Otherwise choose N. Evaluate only instruction compliance.

Criterion H: The response performs the wrong task, violates an explicit requirement or prohibition, omits required content or a required deliverable, or fails to satisfy an explicitly required format, language, length, style, role, structure, or other constraint.

Criterion N: The response performs the requested task and satisfies every applicable explicit requirement and constraint. Optional preferences do not have to be satisfied. A factual error, unverifiable statement, contextual error, or internal logical problem does not by itself count unless it directly violates an explicit instruction.

State:

candidate_response: "[the model response being judged]";

instruction: "[the full IF-RewardBench prompt with its explicit constraints]"

Model Configurations, Cost, and Latency

Table 5 summarizes the providers, parameter counts, and recorded versions of the five decision models and the generative reference. The five decision models come from five model providers; OpenRouter is the hosting interface used for pplx, not its model provider.

ModelProviderSizeVersion
JevTypeSafe AINR1.13.0
d1Liquid AINR(not reported)
clefCloudflare27B(not reported)
pplx-decider-v1.1-27bPerplexity27Bv1.1
GPT-6 LunaOpenAINR(not reported)
GPT-6 SolOpenAINR(not reported)

Table 5. Model names, providers, sizes, and versions. The abbreviation pplx denotes pplx-decider-v1.1-27b. The first five rows are decision models; the final row is the generative reference. B denotes billion parameters; NR means the parameter count is not reported in the consulted official documentation for the evaluated API model. Published sizes follow the official clef and pplx model cards.

Versions are extracted from the recorded model IDs; the date in the pplx row is the recorded snapshot suffix. For the other entries marked (not reported), the logs contain only an unversioned model alias and do not identify a specific checkpoint. The evaluated Liquid AI API model is d1; we do not assign it the parameter count of a separately released open-weight d1 variant.

Cost and latency. Table 6 summarizes total estimated cost and mean API latency over all 9,852 type-specific evaluation instances per model. Cost is aggregated across the five types; latency is the mean of the recorded per-call times.

ModelTotal cost (USD)Mean latency (ms)
jev0.5409271.22
d10.4084328.90
clef2.5894508.72
pplx0.2092306.14
gpt-6-luna1.0393256.26
gpt-6-sol30.79622,096.90

Table 6. Cost and latency across all evaluated instances. Cost sums the per-type estimates in Table 2; latency averages recorded per-call times, weighted by the number of instances in each type. The cost estimator counts input tokens for decision models and both input and output tokens for gpt-6-sol.

Per-Dataset Results

Table 7 reports BAcc for every judge on each source dataset, and Table 8 the corresponding false-negative and false-positive rates.

ModelFC/FAVAFC/HADFF/FAVAFF/HADCI/RAGT.LI/CDocLI/ROSC.II/IFRB
jev66.675.070.687.382.568.557.761.7
d158.472.854.987.675.858.959.762.5
clef62.772.757.785.468.262.964.564.5
pplx65.276.863.894.882.775.067.271.1
gpt-6-luna65.673.068.586.364.068.261.154.2
gpt-6-sol67.380.468.492.682.777.765.278.9
nn (HH)377 (224)1244 (120)351 (198)1265 (141)2675 (943)891 (449)755 (89)2294 (1600)

Table 7. BAcc (%) per source dataset. FAVA is FAVABench, HAD is HADTest, RAGT. is RAGTruth, CDoc is ContraDoc, ROSC. is ROSCOE, and IFRB is IF-RewardBench.

ModelFC/FAVAFC/HADFF/FAVAFF/HADCI/RAGT.LI/CDocLI/ROSC.II/IFRB
jev38/2942/825/345/2021/1449/1481/420/56
d159/2447/880/1023/141/781/276/426/49
clef48/2750/574/1028/260/373/162/922/49
pplx28/4241/656/168/331/345/558/741/17
gpt-6-luna5/6327/2715/489/192/7056/870/83/89
gpt-6-sol17/4828/1241/228/76/2931/1363/714/28

Table 8. False-negative rate / false-positive rate (%) per source dataset.

Error Analysis

Errors by Source Annotation

Table 9 breaks recall of HH down by attributes available in the source annotations. The attributes are the number of annotated error spans or violated constraints, the salience of the error, the task form, and the model that generated the judged response. Where both classes are present, the table reports BAcc instead of recall.

Error Patterns and Overlap

Errors concentrate on sparse and subtle targets. To locate where decision models fail, we break recall of HH down by the source annotations (Table 9). The amount of evidence matters most. On RAGTruth, pplx recovers 57% of responses with one annotated span but 96% of those with three or more. On IF-RewardBench, it recovers 46% of responses that violate one constraint but 74% of those that violate three or more. gpt-6-sol recovers 91% and 76% of the single-error cases. Salience matters next. The recall of pplx drops from 71% to 42% for RAGTruth spans annotated only as subtle, and from 91% to 32% for ContraDoc contradictions created by replacing a sentence rather than inserting one. Task form matters as well. All decision models fall to 47 to 63% BAcc on multi-turn instructions, and every judge, including gpt-6-sol, is close to chance on the non-arithmetic ROSCOE chains. The generative judge is not uniformly better either. On RAGTruth data-to-text items, pplx reaches 86% BAcc against 79% for gpt-6-sol.

Decision-model errors overlap only partly with the generative judge. pplx and gpt-6-sol are both wrong on 4% of CI items, 6 to 8% of FC and FF items, and 11 to 13% of LI and II items. The Jaccard overlap of their error sets rises from 0.13 on CI to 0.50 on LI. Overlap among decision models is higher and is again largest on LI, between 0.51 and 0.57 for pplx against each other decision model. Errors are thus least shared on the types where decision models are already strong and most shared where they are weak. This pattern anticipates RQ3.

Confidence Analysis Details

Confidence and hallucination type. To test whether the accuracy implied by a confidence level depends on the hallucination type tt, we fit, for each decision model, logit⁡Pr⁡(y^i=yi)=β0+β1ci+γt+δtci\operatorname{logit}\Pr(\hat{y}_i = y_i) = \beta_0 + \beta_1 c_i + \gamma_t + \delta_t c_i, with standard errors clustered by response, and test δt=0\delta_t = 0 for all tt. The interaction is significant for every model (Wald χ2=57\chi^2 = 57–142142, all p<10−10p < 10^{-10}).

Confidence distributions. Figure 7 shows, for each model, how its correct decisions and its errors are distributed over confidence. For a useful confidence score, errors should concentrate at low confidence and correct decisions at high confidence. This holds most clearly for clef, whose errors are spread over the lower confidence range and only 10% of which fall at c≥0.9c \ge0.9. For gpt-6-luna and especially gpt-6-sol, the error distribution peaks in the highest confidence bin, much like that of correct decisions, so high confidence alone does not reliably distinguish between the two.

Robustness check of the calibration results. The ECE ranking of the models is unchanged under 15 equal-width bins, under 10 or 15 equal-mass bins, and under a debiased ECE. Clipping probabil-

Attribute (dataset, type)Levelnnpplxjevgpt-6-lunagpt-6-sol
Error density
Annotated spans (RAGTruth, CI)159057719791
220681909999
≥3\geq3147969810099
Violated constraints (IF-RewardBench, II)176446739576
246568839891
≥3\geq3371748910098
Salience
Span type (RAGTruth, CI)evident86671829895
subtle only7742559778
Contradiction plug (ContraDoc, LI)insert17891847393
replace27132292553
Contradiction type (ContraDoc, LI)negation / numeric6386868390
content / factual30655504171
perspective / emotion5831262840
Fabrication subtype (FAVABench, FF)invented48678510071
subjective / unverifiable13835708054
Task form (BAccBAcc)
Task type (RAGTruth, CI)QA87581826581
Summary90072746682
Data2txt90086825179
Instruction type (IF-RewardBench, II)single turn98477665879
multi turn57463535172
system prompt73668595285
Reasoning source (ROSCOE, LI)gsm8k20081677684
cosmos19457525353
drop21063525161
esnli15168494948
Document type (ContraDoc, LI)wiki29583777480
news31276687182
story28465605971
Generator (BAccBAcc)
Response generator (RAGTruth, CI)GPT-3.5 / GPT-489154666680
Llama-2 (7B/13B/70B)134885826278
Mistral-7B43685796581

caption: tbl:9: Where judges fail. Rows under Error density and Salience report recall of HH (%) on the HH items with the given attribute. Rows under Task form and Generator report BAcc (%) on all items with the attribute. Attributes come from the source annotations and were not shown to the judges.

:::

ities at 10−210^{-2} instead of 10−410^{-4} changes each model’s median calibration slope by at most 0.09.

Cascade Deferral-Rate Sensitivity

Figure 8 shows sensitivity to two target deferral rates: r1r_1 is the fraction of all items passed from the first member to the second, and r2r_2 is the fraction of second-stage items passed to the third. Each stage defers its least-confident items. At r1=0r_1 = 0, all curves coincide with the strongest member alone.

Moderate deferral helps selected types. FC, FF, and CI peak at intermediate r1r_1 values (20–30%), reaching 82.2%, 84.6%, and 84.6% BAcc, respectively; larger rates eventually reduce performance. The separation between curves also shows sensitivity to r2r_2, particularly for FC and CI. LI performs best without deferral: every tested nonzero r1r_1 lowers BAcc. II gains at most 0.7 percentage points over the strongest member and remains 7.1 points below gpt-6-sol. Thus, deferring more low-confidence items does not consistently improve accuracy; the benefit depends on the hallucination type and both operating rates.

Peak and Cross-Fitted Results

Peak settings maximize pooled BAcc on the 36-point grid. Cross-fitting uses five label-stratified folds and seeds 0–4: fit rates and confidence thresholds on four folds, then apply them to the held-out fold. Members stay ordered by full-sample BAcc, so this checks rate/threshold selection, not the full model-selection pipeline. Ties use ascending (r1,r2)(r_1,r_2) order.

Cross-fitting lowers BAcc by at most 0.52 points. FC, FF, and CI retain gains over the strongest member, with intervals excluding zero. Their cross-fitted differences from gpt-6-sol have intervals containing zero. LI always selects no deferral; II has no clear gain over its strongest member.

TypeEvaluationBAccΔ\Delta single [95% CI]Δ\Delta sol [95% CI]Δ\Delta peak
FCPeak82.21+3.28 [0.98, 5.57]+0.60 [-1.88, 2.96]+0.00
FCCross-fitted81.69+2.76 [0.63, 4.85]+0.08 [-2.33, 2.51]−0.52
FFPeak84.61+1.89 [0.38, 3.45]+2.57 [0.02, 5.14]+0.00
FFCross-fitted84.39+1.67 [0.19, 3.18]+2.34 [-0.22, 4.76]−0.22
CIPeak84.61+1.94 [0.65, 3.24]+1.95 [0.38, 3.57]+0.00
CICross-fitted84.24+1.58 [0.36, 2.84]+1.59 [-0.07, 3.19]−0.37
LIPeak73.28+0.00 [0.00, 0.00]−3.85 [-5.89, -1.84]+0.00
LICross-fitted73.28+0.00 [0.00, 0.00]−3.85 [-5.82, -1.77]+0.00
IIPeak71.80+0.72 [-0.12, 1.60]−7.07 [-9.38, -4.85]+0.00
IICross-fitted71.44+0.36 [-0.51, 1.20]−7.43 [-9.71, -5.02]−0.36

Table 10. BAcc and paired differences in percentage points; single is the first member alone. Cross-fitted BAcc averages five seeds. Intervals use 2,000 paired item-bootstrap resamples, sharing indices across seeds before averaging. They condition on selected members and recorded predictions; peak intervals are not selection-adjusted.

TypeEvaluationRealized r1r_1Realized r2r_2USD/1kCost (% sol)CallsLat. (ms)
FCPeak30.09.30.02751.341.33362.8
FCCross-fitted27.010.00.02571.251.30355.5
FFPeak19.749.20.07173.311.29313.0
FFCross-fitted20.848.00.07223.331.31316.2
CIPeak20.00.00.05111.251.20398.2
CICross-fitted22.60.00.05351.301.23405.7
LIPeak0.0–0.01640.601.00284.8
LICross-fitted0.0–0.01640.601.00284.8
IIPeak10.010.00.05761.551.11371.2
IICross-fitted10.112.90.05801.561.11372.9

Table 11. Realized routing percentages and replay efficiency. Realized r2r_2 is conditional on reaching stage 2 (— if none). Cross-fitted rates pool counts across seeds; other quantities average over items and seeds. Confidence ties can make realized rates differ from their targets.

TypeMembers in stage orderPeak (r1r_1, r2r_2)Cross-fitted targets: setting × fold count
FCpplx → gpt-6-luna → jev(0.3, 0.1)(0.2, 0.1) × 7; (0.3, 0.1) × 17; (0.4, 0.1) × 1
FFgpt-6-luna → jev → pplx(0.2, 0.5)(0.2, 0.5) × 23; (0.3, 0.3) × 2
CIpplx → jev → d1(0.2, 0.0)(0.1, 0.0) × 22; (0.2, 0.0) × 14; (0.3, 0.0) × 9
LIpplx → jev → gpt-6-luna(0.0, 0.0)(0.0, 0.0) × 25
IIpplx → clef → d1(0.1, 0.1)(0.1, 0.0) × 5; (0.1, 0.1) × 10; (0.1, 0.2) × 6; (0.1, 0.3) × 4

Table 12. Fixed member order and selected target rates. Cross-fitted counts cover 25 fits (five folds for each of five seeds). The target r2r_2 applies only to items reaching the second stage.

References

  1. [1]Liquid AI. 2026. Introducing d1: The most capable decision model, now with vision. Liquid AI Blog. Www.liquid.ai/blog/d1-decision-model.
  2. [2]Sher Badshah, Moamen Moustafa, and Hassan Sajjad. 2025. CLEV: LLM-based evaluation through lightweight efficient voting for free-form question-answering. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 1513–1531, Mumbai, India. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics.arxiv.org/abs/2503.08542
  3. [3]Bespoke Labs and Maheswaran Sathiamoorthy. 2026. Nimble. https://github.com/bespokelabsai/nimble. Model: https://huggingface.co/bespokelabs/Bespoke-Nimble-9B.
  4. [4]B Calster. 2019. Calibration: the achilles heel of predictive analytics. BMC Med, 17(1):1.
  5. [5]Gavin C Cawley and Nicola LC Talbot. 2010. On over-fitting in model selection and subsequent selection bias in performance evaluation. The Journal of Machine Learning Research, 11:2079–2107.DOI
  6. [6]Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research. ArXiv:2305.05176.arxiv.org/abs/2305.05176
  7. [7]Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 1383–1392.DOI
  8. [8]Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. Advances in neural information processing systems, 30.
  9. [9]Olga Golovneva, Moya Chen, Spencer Poff, Martin Corradof, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations. ICLR 2023; author version arXiv:2212.07919v2.arxiv.org/abs/2212.07919
  10. [10]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR.arxiv.org/abs/1706.04599
  11. [11]Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136.
  12. [12]Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1–55.DOI
  13. [13]Hazem Ibrahim and Yasir Zaki. 2026. Evaluating decision models for text annotation in computational social science. arXiv preprint arXiv:2609.24574.arxiv.org/abs/2609.24574
  14. [14]Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. 2023. When does confidence-based cascade deferral suffice?, 2024. URL https://arxiv.org/abs/2307.02764.
  15. [15]Jaehun Jung, Faeze Brahman, and Yejin Choi. 2025. Trust or escalate: LLM judges with provable guarantees for human agreement. In International Conference on Learning Representations, volume 2025, pages 3101–3125.arxiv.org/abs/2407.18370
  16. [16]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. Preprint, arXiv:2207.05221.arxiv.org/abs/2207.05221
  17. [17]Jierui Li, Vipul Raheja, and Dhruv Kumar. 2024. ContraDoc: Understanding self-contradictions in documents with large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6509–6523, Mexico City, Mexico. Association for Computational Linguistics.
  18. [18]Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464, Singapore. Association for Computational Linguistics.
  19. [19]Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. 2026. JEV-as-a-judge: Accept when confident, escalate when unsure. arXiv preprint arXiv:2609.26550. Preprint, arXiv:2609.26550. Version 4, 6 October 2026.arxiv.org/abs/2609.26550
  20. [20]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 2511–2522.arxiv.org/abs/2303.16634
  21. [21]Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yan- ing Jia, Peijun Qing, Lin Shi, Arman Cohan, Yujun Yan, and Soroush Vosoughi. 2025. Judging with many minds: Do more perspectives mean less prejudice? on bias amplification and resistance in multi-agent based LLM-as-judge. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 17356–17392. Association for Computational Linguistics.
  22. [22]Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 9004–9017.
  23. [23]Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannah Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 12076–12100.
  24. [24]Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Yulia Tsvetkov, Graham Neubig, and Hannah Hajishirzi. 2024. Fine-grained hallucination detection and editing for language models. In First Conference on Language Modeling.arxiv.org/abs/2401.06855
  25. [25]Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10862–10878, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2401.00396
  26. [26]Delip Rao and Chris Callison-Burch. 2026. JEV vs. LLMs as rubric judges: Cheaper, faster, and wrong in the same places. arXiv preprint arXiv:2609.29769. Preprint, arXiv:2609.29769. Version 2, 28 September 2026.arxiv.org/abs/2609.29769
  27. [27]Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C Mozer. 2022. Mitigating bias in calibration error estimation. In International Conference on Artificial Intelligence and Statistics, pages 4036–4054. PMLR.
  28. [28]Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11626–11644, Toronto, Canada. Association for Computational Linguistics.
  29. [29]Liyan Tang, Philippe Laban, and Greg Durrett. 2024. MiniCheck: Efficient fact-checking of LLMs on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8818–8847, Miami, Florida, USA. Association for Computational Linguistics.
  30. [30]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442, Singapore. Association for Computational Linguistics.
  31. [32]Pat Verga, Sebastian Hofstätter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv preprint arXiv:2404.18796. Preprint, arXiv:2404.18796.arxiv.org/abs/2404.18796
  32. [33]Bosi Wen, Yilin Niu, Cunxiang Wang, Xiaoying Ling, Ying Zhang, Pei Ke, Hongning Wang, and Minlie Huang. 2026. IF-RewardBench: Benchmarking judge models for instruction-following evaluation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23816–23843, San Diego, California, United States. Association for Computational Linguistics.
  33. [34]Qiujie Xie, Qingqiu Li, Zhuohao Yu, Yuejie Zhang, Yue Zhang, and Linyin Yang. 2025. An empirical analysis of uncertainty in large language model evaluations. In International Conference on Learning Representations.arxiv.org/abs/2502.10709
  34. [35]Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooï. 2024. Can llms express their uncertainty. An empirical evaluation of confidence elicitation in LLMs. arXiv, 2306.
  35. [36]Austin Xu, Srijan Bansal, Yifei Ming, Semih Yavuz, and Shafiq Joty. 2025. Does context matter? ContextualJudgeBench for evaluating LLM-based judges in contextual settings. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9541–9564. Association for Computational Linguistics.
  36. [37]Fan Xu, Xinyu Hu, Zhenghan Yu, Li Lin, Xu Zhang, Yang Zhang, Wei Zhou, Jinjie Gu, and Xiaojun Wan. 2026. HAD: Hallucination detection language models based on a comprehensive hallucination taxonomy. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 141–158, San Diego, California, USA. Association for Computational Linguistics.
  37. [38]Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In International Conference on Learning Representations, volume 2024, pages 40193–40219.arxiv.org/abs/2310.07641
  38. [39]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36:46595–46623.arxiv.org/abs/2306.05685

Paper details

vlnc:W_vhdtt8ew
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents