betaFeedback

WorksComputer Science

Feeling of Knowing in Large Language Models

Abstract

Post-training updates change what large language models (LLMs) can answer, but may leave their answerability judgments misaligned. We study this knowledge-judgment desynchronization through functional feeling of knowing (FoK), which supports selective generation, retrieval triggering, and model routing without implying subjective experience. We propose two trainable channels: a knowledge channel that changes answerability and a FoK channel that judges the current knowledge state. Three stages train them: on-policy supervised fine-tuning (OP-SFT) updates knowledge from corrected on-policy answers; knowledge-delta FoK training (KΔΔ-FT) adapts judgments across intermediate knowledge states; and FoK-guided policy optimization (FGPO) trains the subsequent answer or explanation. Factual-QA experiments, cross-dataset evaluation, and three sequential updates show that explicit FoK alignment improves judgment quality on updated knowledge. MMLU and ARC-Easy separately assess capability preservation.

Introduction

Large language models (LLMs) serve as assistants and tool-using agents [18, 52]. Their reliability depends on answer accuracy and on judging whether to answer or seek external support. Cognitive science calls this metacognitive judgment feeling of knowing (FoK) [13, 25]. We use functional FoK for LLM answerability judgments, without implying subjective experience. This scalar judgment supports selective prediction, model routing, and retrieval triggering [21]. If it fails to reflect updated knowledge, a system may confidently answer unknown questions or refuse questions the updated model could solve.

Existing LLM FoK methods mainly take three forms. Output-based methods read uncertainty from the model’s own generations, such as the probability of a “true” token or the entropy of sampled answers [9, 21, 27]. Probe-based methods train a lightweight classifier on hidden states [1, 3]. Elicitation-based methods supervise the model to verbalize its confidence directly [41, 50, 54]. However, these methods are usually calibrated and evaluated under static model parameters. This leaves open what happens when the model is later updated by supervised fine-tuning (SFT), low-rank adaptation (LoRA), knowledge editing, or continual learning [17, 37, 38]. After an update, the LLM’s knowledge may change, while a FoK estimator trained or calibrated for the base model may still judge answerability based on the base model’s knowledge boundary. The problem of keeping FoK aligned under changing model knowledge remains underexplored.

Koriat’s account motivates this study: human FoK is inferred from accessible cues rather than directly read from memory [25]. Our correctness-supervised objective does not instantiate or validate that cognitive process. In LLMs, prior causal tracing locates factual associations mainly in middle layers [38], whereas our FoK recovery peaks occur later (Section 2.2). This is evidence about the tested prompts and interventions, rather than proof of a separate human-like mechanism. Two challenges follow: 1) Knowledge–FoK desynchronization: knowledge training changes answerability without keeping the FoK judgment aligned; and 2) Context-dependent FoK: the judgment varies with the available context and generated reasoning. We therefore train a FoK channel to judge the current knowledge state and to guide the subsequent response.

We address these challenges with two trainable channels. On-policy supervised fine-tuning (OP-SFT) updates the knowledge channel from the model’s own answers, supervising only corrected answer-relevant suffixes to limit disruption to FoK. Knowledge-delta FoK training (KΔ-FT) trains the FoK channel across knowledge deltas, allowing judgment labels to reflect changing answerability. For context-dependent FoK, FoK-guided policy optimization (FGPO) trains the model to generate context before judging answerability and to produce the corresponding answer or explanation. The paper makes three contributions:

  • We study functional FoK after knowledge updates, identifying later-layer FoK recovery, sensitivity to accessible context, and a failure of knowledge updating alone to synchronize answerability judgments.

  • We introduce a two-channel framework with OP-SFT, KΔ-FT, and FoK-guided policy optimization to update knowledge, synchronize FoK, and train the model to produce an answer or explanation according to its FoK judgment.

  • We evaluate FoK on factual QA, including cross-dataset transfer and three sequential updates, with seed variation for the main method. MMLU and ARC-Easy separately measure capability preservation.

Mechanism of LLM FoK

Problem formulation

Let MθM_\theta be an LLM at parameter state θ\theta, and let qq be a question. We use Yθ(q)∈{0,1}Y_\theta(q) \in\{0,1\} for whether the model’s greedy answer is correct, and Jθ(q)∈[0,1]J_\theta(q) \in[0,1] for the model’s functional FoK score, where 0 means the model judges that it cannot answer and 1 means it judges that it can answer. The quality of FoK is reflected in the alignment between Jθ(q)J_\theta(q) and Yθ(q)Y_\theta(q) under the same decoding rule. For a generative yes/no judgment prompt, we instantiate this score as Jθ(q)=Pθ(yes∣q)Pθ(yes∣q)+Pθ(no∣q)J_\theta(q) = \frac{P_\theta(\mathrm{yes}\mid q)}{P_\theta(\mathrm{yes}\mid q)+P_\theta(\mathrm{no}\mid q)}.

When an LLM learns new knowledge through post-training, its parameters change from the base state θ\theta to an updated state θ~\tilde{\theta}. We call the induced difference κ=θ~−θ\kappa= \tilde{\theta} - \theta the knowledge delta. The target is to keep Jθ~J_{\tilde{\theta}} aligned with Yθ~Y_{\tilde{\theta}} as knowledge changes. Our method explicitly trains FoK over the current update trajectory; it does not assume that one calibration remains valid for all later updates.

In evaluation, we focus on two directional failures of the FoK judgment. An illusion occurs when the model judges that it can answer but is wrong; an omission occurs when it judges that it cannot answer although its answer would be correct. Metrics and splits are detailed in § 4.1.

FoK localization in model depth

To localize FoK ability, we first adopt ROME’s causal-tracing method [38]. For each factual question, we corrupt the subject token, restore hidden states one layer at a time, and measure whether the yes/no FoK judgment is recovered. We repeat the procedure across Qwen2.5 model sizes, using normalized depth to compare corresponding layers across architectures rather than raw layer indices.

As shown in Figure 1a, Qwen2.5-7B places the strongest FoK signal in late layers, around two-thirds of model depth. Figure 1b shows that the same late-layer pattern remains after normalizing depth across Qwen2.5 scales from 0.5B to 32B. This is later than the factual association signal reported by ROME, which is localized in middle MLP layers, around one-third of model depth [38]. Appendix § 6 gives a parameter-side check: knowledge SFT changes shallow-to-middle layers more, while judgment SFT changes concentrate in the late layers where FoK signal strength is high. Appendix § 10 extends the causal tracing across scales and outcome groups.

Causal tracing of FoK signal in a 7B LLM and FoK signal across normalized depth

Figure 1. FoK signal localizes late under causal restoration. Panel (a) shows a 7B example across tokens and layers; panel (b) aggregates FoK recovery across Qwen2.5 sizes over normalized depth.

Takeaway 1: Under causal tracing, the FoK signal peaks later in model depth than factual associations, suggesting that FoK acts as a judgment over the model’s knowledge state rather than as factual recall itself.

How FoK depends on updated knowledge

Depth separation does not mean independence. To analyze the relationship between factual knowledge and FoK, we intervene on one side and evaluate the other under matched data and optimization settings. First, we update factual knowledge with vanilla factual SFT and measure FoK on the updated model. Second, we train FoK alone and measure whether factual-answer accuracy changes. Table 1 summarizes cross-scale results. Appendix § 13 reports supplementary scale and family results from a separate setup that uses exact match (EM), so its absolute values differ from this table.

ModelBaseBaseK-SFTK-SFTJ-SFTJ-SFTK→JK \to J SFTK→JK \to J SFT
Acc.AUROCΔ\DeltaAcc.Δ\DeltaAUCΔ\DeltaAcc.Δ\DeltaAUCΔ\DeltaAcc.Δ\DeltaAUC
Qwen2.5-0.5B20%0.519+2-0.016-14+0.213+2+0.211
Qwen2.5-1.5B35%0.529+6-0.035+2+0.255+5+0.248
Qwen2.5-3B39%0.739+12-0.220+3+0.038+11+0.051
Qwen2.5-7B53%0.724+7-0.042+2+0.088+7+0.100
Qwen2.5-14B66%0.762+6+0.004+1+0.030+5+0.068
Qwen2.5-32B68%0.819+4-0.0260+0.010+4+0.020
LLaMA-3.1-8B74%0.672+2-0.038+1+0.045+2+0.063

Table 1. Knowledge and FoK under separate or sequential SFT updates. Base gives absolute values; the three SFT blocks give changes from Base. Accuracy changes are percentage points (pp); AUROC changes are unitless. Δ\DeltaAUC denotes the change in AUROC. Higher accuracy and AUROC are better. Full illusion and omission rates are reported in Appendix § 15.§ 1.

Table 1 shows that knowledge SFT (K-SFT) increases answer accuracy for every model, but FoK AUROC decreases for six of the seven models, with the largest drop on Qwen2.5-3B. Thus, the knowledge update changes the model’s answerability, but the judgment boundary does not adjust correctly. The mismatch is not a single error direction: after K-SFT, some models make more omissions, judging that they cannot answer questions they would answer correctly, whereas larger Qwen models better preserve AUROC but shift toward illusion, judging that they can answer even when their answers are wrong. The issue is therefore not simply increased conservatism or confidence, but a weakened alignment between the current knowledge state and FoK. Judgment SFT (J-SFT) provides a reference intervention: directly training judgment improves FoK without compromising answer performance in larger models. K→JK \to J SFT further shows that judgment training after a knowledge update can recalibrate FoK while preserving most of the answer-accuracy gain from K-SFT, indicating that FoK depends on the current knowledge state and should be updated after knowledge changes. Appendix E provides the FoK-recovery and reverse-direction supplementary results.

Takeaway 2: In current LLMs, knowledge updates change answerability but do not automatically update FoK. FoK depends on features of the current knowledge state, so changing knowledge can break the mapping from knowledge to self-judgment unless FoK is explicitly updated.

Reasoning before judgment increases FoK certainty

Koriat’s FoK theory suggests that the amount and intensity of accessible information increase FoK judgments [25]. We test the LLM analogue by asking whether internally generated reasoning affects the model’s FoK.

We evaluate Qwen2.5-7B on MATH [15] problems. For each problem, the model first generates a solution trace. We then re-query the same model at increasing prefixes of that trace and compute J=P(“Yes”)/[P(“Yes”)+P(“No”)]J = P(\text{“Yes”})/[P(\text{“Yes”}) + P(\text{“No”})] for the question “Can you solve this problem correctly?” Figure 2 shows that the FoK signal becomes more separated over the generated trace. As more reasoning tokens are available, the mean FoK scores for final “Yes” and “No” judgments move toward J=1J = 1 and J=0J = 0, respectively; the AUROC of JJ for the final Yes/No FoK token rises as more reasoning tokens are available. This indicates that FoK is sensitive to the model’s current context, not only to the original question.

Figure 2. FoK score and AUROC over pre-judgment reasoning tokens. Green/red curves show mean FoK score for examples whose final FoK token is Yes/No. J=0.5J = 0.5 denotes FoK uncertainty, while J≈1J \approx1 or J≈0J \approx0 denotes high certainty. The black curve reports AUROC of prefix-level JJ for predicting whether the final FoK token is Yes or No.

Takeaway 3: FoK should be judged from the model’s current context, not from the original question alone. This motivates a response-generation procedure in which the model first accumulates task-relevant context, then evaluates answerability before producing the final answer.

Methodology

Our methodology aligns FoK with updated knowledge through a two-channel model structure and a three-stage procedure that updates knowledge, synchronizes FoK, and trains FoK-guided responses.

Framework overview

Starting from a base LLM MM, we represent post-training changes as parameter deltas. The knowledge delta κ\kappa forms the knowledge-retrieval channel, which changes what the model knows and retrieves. The FoK delta φ\varphi forms the FoK channel, which judges whether the current knowledge state supports a correct answer. The training procedure has three parts:

  • On-policy supervised fine-tuning (OP-SFT). OP-SFT updates κ\kappa on the model’s own sampled responses and supervises only the corrected answer-relevant suffix. This part learns the target facts while saving a trajectory of knowledge states {κ0,κ1,…,κT}\{\kappa_0, \kappa_1, \ldots, \kappa_T\} that later serves as supervision for FoK adaptation under the same base model.

  • Knowledge-delta FoK training (KΔ\Delta-FT). KΔ\Delta-FT trains φ\varphi over the OP-SFT checkpoint trajectory κt\kappa_t. This lets FoK judgments follow the model’s knowledge as it evolves, rather than remaining tied to a fixed knowledge boundary.

  • FoK-guided policy optimization (FGPO). Finally, FGPO trains the model to generate a reasoning trace, make a yes/no FoK judgment, and produce either an answer or an explanation of why it cannot answer. Its reward evaluates the judgment and response together, so FoK is optimized as part of the generated decision rather than as a separate post-processing score.

With this two-channel update, FoK guides the LLM before answering. The model can first judge whether its current knowledge is sufficient, then either generate an answer or explain what knowledge is missing, which can support routing to a stronger model or retrieval-augmented generation.

On-policy supervised fine-tuning (OP-SFT)

Section 2.3 shows that standard knowledge training does not synchronize FoK with the updated knowledge state, and can severely interfere with the model’s original FoK ability. To reduce this interference, we introduce on-policy supervised fine-tuning (OP-SFT). Specifically, OP-SFT keeps the model’s own generated context and applies gradients only to the corrected answer-relevant span.

For each question qq with gold answer set AqA_q, we collect kk complete sampled responses from the current LLM. For a sampled response r=(r<s,r[s:e],r>e)r = (r_{<s}, r_{[s:e]}, r_{>e}), r[s:e]r_{[s:e]} denotes the generated answer span. We then handle two cases. If the sampled response already answers correctly, we keep this answer span. If it answers incorrectly, we replace only this answer span with a gold answer a∗(q)∈Aqa^*(q) \in A_q and leave the rest of the generated response unchanged. The repaired training response is

rˉ(q,r)=(r<s,aˉ(q,r),r>e),aˉ(q,r)={r[s:e]r answers correctly,a∗(q)r answers incorrectly.(1)\bar{r}(q,r) = (r_{<s}, \bar{a}(q,r), r_{>e}), \qquad\bar{a}(q,r) = \begin{cases} r_{[s:e]} & r\text{ answers correctly},\\ a^*(q) & r\text{ answers incorrectly}. \end{cases} \tag*{(1)}

The training sequence is therefore not a static QA pair: it is sampled from the current LLM, changes only the wrong answer span, and uses loss only on aˉ(q,r)\bar{a}(q,r). For example, for q=q = “Who wrote Pride and Prejudice?” and Aq={Jane Austen}A_q = \{\text{Jane Austen}\}, OP-SFT repairs one sampled response as

q: Who wrote Pride and Prejudice?,Aq={Jane Austen},q:\ \text{Who wrote \textit{Pride and Prejudice}?}, \qquad A_q = \{\text{Jane Austen}\},
rˉ(q,r): The book is a nineteenth-century novel. I think the author was Charlotte Bronte¨ Jane Austen, because she wrote English classics.(2)\bar{r}(q,r):\ \text{The book is a nineteenth-century novel. I think the author was Charlotte Brontë Jane Austen, because she wrote English classics.} \tag*{(2)}

The red span is the first answer-relevant error in the sampled response; the green span is the supervised correction. The remaining generated text stays in the response, but receives no loss. The OP-SFT objective is therefore the local corrected-span loss:

LOP-SFT=−Eq,r∑u=1∣aˉ(q,r)∣log⁡PMκ(aˉ(q,r)u∣q,r<s,aˉ(q,r)<u).(3)\mathcal{L}_{\mathrm{OP\text{-}SFT}} = -\mathbb{E}_{q,r}\sum_{u=1}^{|\bar{a}(q,r)|}\log P_{M_{\kappa}}\left(\bar{a}(q,r)_u \mid q,r_{<s},\bar{a}(q,r)_{<u}\right). \tag*{(3)}

We save checkpoints {κ0,κ1,…,κT}\{\kappa_0,\kappa_1,\ldots,\kappa_T\} during OP-SFT, keeping a sequence of knowledge states.

Knowledge-delta FoK training (KDelta-FT)

After a knowledge update, the LLM’s answerability changes, but its FoK does not automatically synchronize with that change. Training FoK only on the base or final LLM gives a state-specific answerability boundary that generalizes poorly across knowledge deltas. Final-state training also creates label imbalance because many updated questions become answerable.

To make FoK generalize across knowledge deltas, we introduce knowledge-delta FoK training (KDelta-FT), which uses the OP-SFT checkpoint trajectory {κ0,κ1,…,κT}\{\kappa_0,\kappa_1,\ldots,\kappa_T\} as supervision states rather than supervising only one endpoint. For each checkpoint κt\kappa_t, we combine it with the same FoK delta φ\varphi and train φ\varphi to predict whether MκtM_{\kappa_t} can answer question qq with the following loss:

ℓφ(q,κt)=−log⁡PMκt,φ(1[ans⁡(Mκt,q)∈Aq]∣Jφ(q)).(4)\ell_{\varphi}(q,\kappa_t) = -\log P_{M_{\kappa_t,\varphi}}\left(\mathbf{1}[\operatorname{ans}(M_{\kappa_t},q)\in A_q] \mid J_{\varphi}(q)\right). \tag*{(4)}

Here 1[⋅]1[\cdot] denotes the indicator function: it equals 1 when the answer generated by MκtM_{\kappa_t} belongs to Aq\mathcal{A}_q, and equals 0 otherwise. TJ(q)T_J(q) wraps qq in the fixed Yes/No prompt “Can you answer this question correctly?” Mκt,φM_{\kappa_t,\varphi} is the base model with the frozen knowledge delta and trainable FoK delta. For y∈{0,1}y \in\{0,1\}, PMκt,φ(y∣TJ(q))P_{M_{\kappa_t,\varphi}}(y \mid T_J(q)) is the normalized Yes/No probability: J(q)J(q) for y=1y=1 and 1−J(q)1-J(q) for y=0y=0. The probability is evaluated under Mκt,φM_{\kappa_t,\varphi}, so FoK is trained with the same combined knowledge-and-FoK state used for judgment. The overall objective averages this checkpoint-specific loss over all questions and all OP-SFT checkpoints:

LKΔ-FT(φ)=1∣Q∣(T+1)∑q∈Q∑t=0Tℓφ(q,κt).(5)\mathcal{L}_{\mathrm{K}\Delta\text{-}\mathrm{FT}}(\varphi)=\frac{1}{|\mathcal{Q}|(T+1)}\sum_{q\in\mathcal{Q}}\sum_{t=0}^{T}\ell_{\varphi}(q,\kappa_t). \tag*{(5)}

Only φ\varphi is optimized in this stage; the knowledge checkpoints are fixed supervision states. Because the same question is evaluated under multiple κt\kappa_t, KΔ\Delta-FT trains FoK to follow answerability as it changes from pre-update to post-update knowledge, rather than to memorize a static question-level label detached from the current model state during updating. This is correctness supervision across changing states, not an explicit model of human cue-based inference.

FoK-guided policy optimization (FGPO)

After OP-SFT and KΔ\Delta-FT, a continuous FoK score still requires a decision threshold. FoK-guided policy optimization (FGPO) trains the judgment and subsequent response together, allowing the model to commit to answering or explaining within one generation. It generates a short context trace, makes a FoK judgment, and produces an answer or explanation:

<think>...reasoning...</think> <answer>yes|no</answer> <response>answer or explanation</response>

For a sampled trajectory z=(h,j,u)z=(h,j,u), where hh is the reasoning trace, j∈{yes,no}j\in\{\mathrm{yes},\mathrm{no}\} is the FoK judgment, and uu is the response, we decompose the reward into FoK, answer, and format terms:

R(q,z)=RFoK(Y(q),j)+Rans(q,Y(q),j,u)+Rfmt(z)−λ∣h∣.(6)R(q,z)=R_{\mathrm{FoK}}(Y(q),j)+R_{\mathrm{ans}}(q,Y(q),j,u)+R_{\mathrm{fmt}}(z)-\lambda|h|. \tag*{(6)}

Here Y(q)Y(q) is the answerability target for the current training example. Let C(q,u)=1[u∈Aq]C(q,u)=1[u\in\mathcal{A}_q] denote answer correctness, E(q,u)E(q,u) denote whether an LLM judge accepts uu as a valid explanation for not answering, and F(z),S(z)∈{0,1}F(z),S(z)\in\{0,1\} denote valid format and proper sequence termination. The FoK term directly supervises the judgment token:

RFoK(Y,j)=α+1[ι(j)=Y]−α−1[ι(j)≠Y],(7)R_{\mathrm{FoK}}(Y,j)=\alpha_{+}1[\iota(j)=Y]-\alpha_{-}1[\iota(j)\ne Y], \tag*{(7)}

where ι(yes)=1\iota(\mathrm{yes})=1 and ι(no)=0\iota(\mathrm{no})=0. The answer term scores the response under the chosen judgment:

Rans(q,Y,j,u)=1[j=yes,Y=1](β+C(q,u)−β−(1−C(q,u)))−1[j=yes,Y=0](βˉ+C(q,u)+βˉ−(1−C(q,u)))+1[j=no,Y=0](γ+E(q,u)−γ−(1−E(q,u)))−1[j=no,Y=1]γskip.(8)\begin{aligned} R_{\mathrm{ans}}(q,Y,j,u)={}&1[j=\mathrm{yes},Y=1]\bigl(\beta_{+}C(q,u)-\beta_{-}(1-C(q,u))\bigr)\\ &-1[j=\mathrm{yes},Y=0]\bigl(\bar{\beta}_{+}C(q,u)+\bar{\beta}_{-}(1-C(q,u))\bigr)\\ &+1[j=\mathrm{no},Y=0]\bigl(\gamma_{+}E(q,u)-\gamma_{-}(1-E(q,u))\bigr)\\ &-1[j=\mathrm{no},Y=1]\gamma_{\mathrm{skip}}. \tag*{(8)} \end{aligned}

Thus, a “yes” judgment must be followed by a correct answer, while a “no” judgment must be justified by a valid explanation and is penalized on answerable examples. The format term is

Rfmt(z)=ηFF(z)−ηˉF(1−F(z))+ηSS(z)−ηˉS(1−S(z)).(9)R_{\mathrm{fmt}}(z)=\eta_FF(z)-\bar{\eta}_F(1-F(z))+\eta_SS(z)-\bar{\eta}_S(1-S(z)). \tag*{(9)}

Table 6 in Appendix § 9 lists the implemented values. In implementation, wrong answers and malformed outputs receive larger penalties than valid “no” explanations receive positive reward, so the model is not rewarded for avoiding answerable questions by choosing the negative judgment.

Experiments

We ask whether FoK can be synchronized after an update (RQ1), which training stages contribute (RQ2), and whether the result persists across datasets, sequential updates, and seeds (RQ3). Sections 4.2, 4.3, and 4.5 address RQ1–RQ2; § 4.4 and Appendix § 14 address RQ3.

Evaluation setup

Datasets. We use TriviaQA [20] as the main QA dataset, with 5,000 update questions and 1,000 held-out questions. Additional FoK evaluations use SelfAware [55], Natural Questions [30], and WebQuestions [2]. MMLU [14] and ARC-Easy [6] separately evaluate general capability; they are not additional FoK testbeds.

Baselines. We compare direct Yes/No FoK judgment, Vanilla K-SFT, verbalized confidence and 10-bin verbalized confidence [50], sequence probability and length-normalized sequence probability, lexical and semantic self-consistency, semantic entropy [9, 27], and EliCal [41].

Metrics. We report accuracy on TriviaQA, MMLU, and ARC-Easy. FoK quality is measured by AUROC [10] using the normalized Yes probability, plus two errors thresholded at J>0.5J > 0.5 (predict Yes): illusion rate (illu., Pr⁡(J^=1∣Y=0)\Pr(\hat{J}=1 \mid Y=0)) and omission rate (om., Pr⁡(J^=0∣Y=1)\Pr(\hat{J}=0 \mid Y=1)). We combine them as:

Bias⁡={illu.+om.2,if illu.≥om.,−illu.+om.2,if illu.<om.Bias⁡∈[−1,1].\operatorname{Bias} = \begin{cases} \frac{\mathrm{illu.}+\mathrm{om.}}{2}, & \text{if illu.} \ge\text{om.}, \\ -\frac{\mathrm{illu.}+\mathrm{om.}}{2}, & \text{if illu.} < \text{om.} \end{cases} \qquad\operatorname{Bias} \in[-1,1].

Implementation details. We evaluate on Qwen2.5-7B-Instruct [48] for the main experiments and on Qwen3-8B [49] as a thinking-mode backbone. We refer to them as Qwen2.5-7B and Qwen3-8B. We instantiate the three stages of § 3 as LoRA adapters on top of the base model. OP-SFT updates the knowledge channel from corrected sampled responses, KΔ-FT trains the FoK channel over the OP-SFT checkpoint trajectory, and FGPO optimizes response generation using both channels. The full data construction, baseline definitions, evaluation protocol, optimizer settings, checkpoint schedule, and inference configuration are listed in Appendix § 9.

Overall performance

We compare our two-channel update with vanilla K-SFT and a suite of existing FoK estimators after a single TriviaQA knowledge update on Qwen2.5-7B. Table 2 gives the main numbers; Figure 3 decomposes the thresholded errors into illusion and omission. After vanilla K-SFT, in-domain accuracy rises sharply but held-out accuracy falls below the base model, so the update does not simply add knowledge. All existing estimators fall well below their base-model level, and most collapse into one-sided behavior: vanilla K-SFT and confidence-based estimators are dominated by illusion, judging almost every question answerable, while consistency-based estimators trade some illusion for a correspondingly large omission, leaving both error sources substantial. A likely cause is that the knowledge update shifts the output distributions and sampling behavior that these estimators rely on, so a decision boundary calibrated before the update no longer carries the same meaning. This is what Takeaway 2 predicts: under a conventional knowledge update, existing FoK estimators are not a robust way to keep answerability judgments aligned with the updated model.

MethodTrain Acc.↑Train AUROC↑Train Bias→ 0Test Acc.↑Test AUROC↑Test Bias→ 0General MMLU↑General ARC↑
Base0.6800.7020.3810.6720.6880.4020.6870.811
+ Prob0.6800.7230.4090.6720.7300.4070.6870.811
+ N-Prob0.6800.7230.4090.6720.7300.4070.6870.811
+ EliCal0.6370.8080.2630.6220.8100.2540.7090.504
Vanilla K-SFT0.8740.4580.5000.5460.5190.5000.7130.840
+ Verbal-00.8740.4910.5430.5460.4780.5510.7130.840
+ Verbal-100.8740.5670.4560.5460.5290.4670.7130.840
+ Prob0.8740.5900.4950.5460.5410.4980.7130.840
+ N-Prob0.8740.5900.4950.5460.5420.4980.7130.840
+ Consis-Lex0.8740.5390.4820.5460.5310.4860.7130.840
+ Consis-Sem0.8740.5530.4720.5460.5670.4670.7130.840
+ SemEnt0.8740.5380.4810.5460.5260.5040.7130.840
+ EliCal0.6480.6460.4280.5870.6440.4470.7230.688
Ours0.9610.8170.2450.6920.8090.3080.7110.830

Table 2. Overall performance after one TriviaQA knowledge update on Qwen2.5-7B. Train columns use questions seen during the update; Test columns use held-out TriviaQA questions; General reports MMLU and ARC-Easy. ↑: higher is better; ↓: lower is better; → 0: closer to zero is better. Bold and underline indicate the best and second-best results.

Figure 3. FoK error decomposition for Table 2. Red circles show illusion, the conditional rate of false “can answer” judgments; blue squares show omission, the conditional rate of false “cannot answer” judgments. The horizontal gap shows which error source determines the signed Bias value.

Our two-channel update shows a different profile. Held-out accuracy now rises above the base model, so the knowledge update generalizes instead of overfitting the update shard. FoK AUROC on the updated model reaches the level of the strongest existing estimator applied to the base model, and Bias is the lowest on the update set and the second lowest on held-out questions (Figure 3), although illusion still exceeds omission on held-out questions. We attribute this to two design choices. OP-SFT supervises only the corrected answer-relevant span on the model’s own sampled responses, which only mildly disturbs FoK: held-out AUROC is 0.603 after OP-SFT alone, compared with 0.519 after vanilla K-SFT (Table 2 and Table 4); KΔ-FT then trains the FoK channel directly on the changed answerability, instead of reusing a judgment calibrated before the update. Appendix § 12 confirms this along the OP-SFT trajectory.

Table 4: Held-out stage-wise ablation on Qwen2.5-7B. OP-SFT updates knowledge, KΔ\Delta-FT adapts FoK, and FGPO optimizes judgment-guided generation. Higher accuracy and AUROC and lower error rates are better; Bias closer to zero is better. Update-set results are in Table 26.

StageAcc.↑AUROC↑Illu.↓Om.↓Bias→ 0
Base0.6720.6880.6830.1220.402
Stage 1: + OP-SFT0.6790.6030.6010.2830.442
Stage 2: + KΔ\Delta-FT0.6990.7700.4780.1870.333
Stage 3: + FGPO0.6920.8090.4320.1840.308

Table 4. Held-out stage-wise ablation on Qwen2.5-7B.

Coverage-accuracy trade-off under FoK judgment

Table 2 shows that FoK quality is recoverable; it remains to test whether the recovered FoK actually drives the decision of when to answer. We therefore vary the FoK threshold on Qwen3-8B and plot the coverage-accuracy curves in Figure 4. For Base and Vanilla K-SFT the curves are nearly flat on both splits: lowering coverage does not raise accuracy, so their FoK scores do not rank answerability. OP-SFT + KΔ\Delta-FT restores a monotonic trade-off, confirming that the learned FoK channel is consistent with answerability after the update. FGPO flattens the curves again, but for a different reason: the scores collapse to 0 or 1, and the threshold becomes irrelevant once the policy has committed to answer or abstain. KΔ\Delta-FT thus provides a tunable coverage–accuracy trade-off, while FGPO answers more selectively, achieving higher accuracy on the answered subset at the cost of lower coverage.

Figure 4. Coverage–accuracy trade-off on Qwen3-8B (thinking mode). Each panel varies the FoK threshold and plots coverage (solid, the fraction of questions the model judges itself able to answer), conditional accuracy on that answered subset (dashed), and overall accuracy with unanswered examples counted as incorrect.

Controlled comparison and sequential updates

Accuracy-preserving update. Table 3a tests whether avoiding destructive K-SFT alone suffices. After OP-SFT, direct judgment, Prob, and EliCal reach AUROC 0.603, 0.571, and 0.618, respectively; KΔ\Delta-FT reaches 0.770. Thus, FoK adaptation is useful even when held-out answer accuracy is preserved. Without an update, however, strong estimators remain competitive: Base + EliCal reaches 0.810 AUROC in Table 2, compared with 0.809 for the full method, whose held-out accuracy exceeds Base by 0.020 in that run.

Table 3: Additional evaluation on Qwen2.5-7B. Left: all methods start from the same OP-SFT knowledge checkpoint; KΔ\Delta-FT adds the FoK adapter, so its combined answering state differs. Right: φ1\varphi_1 is frozen after round 1; φi\varphi_i is retrained over each round’s knowledge trajectory. Each round adds a disjoint shard from 5,000 update questions; all test columns use the same 1,000 held-out questions.

EstimatorTest Acc.AUROC
Direct judgment0.6790.603
Prob0.6790.571
EliCal0.6790.618
KΔ\Delta-FT0.6990.770
RoundTest Acc.φ1\varphi_1 AUCφi\varphi_i AUC
10.6590.7980.798
20.6440.7760.800
30.6610.7540.797

Table 3. Additional evaluation on Qwen2.5-7B.

Sequential updates and broader evaluation. Across three updates, the per-round FoK channel stays near 0.80 AUROC while the frozen channel falls to 0.754 (Table 3b); held-out accuracy remains below the 0.672 Base value, so this does not demonstrate cumulative answer-accuracy improvement. Cross-dataset tests on SelfAware, Natural Questions, and WebQuestions, plus a separate Natural Questions replication, extend the factual-QA evaluation. Independent Stage-1+2 runs yield test AUROC 0.777 ± 0.009 (four seeds); FGPO yields 0.811 ± 0.003 (three seeds from one shared initialization). These are means and sample standard deviations. Appendix § 14 reports all runs, cross-dataset results, and a MEMIT desynchronization test; the latter tests the phenomenon rather than adaptation of our full framework to that editor.

Ablation study

To locate where the FoK improvement in Table 2 comes from, we ablate the three stages cumulatively: OP-SFT alone, then OP-SFT + KΔ\Delta-FT, then the full pipeline with FGPO (Table 4). OP-SFT alone raises update-set accuracy from 0.680 to 0.966 (Table 26) but leaves the FoK judgment misaligned with the updated knowledge state. Adding KΔ\Delta-FT is the main source of the FoK gain: AUROC rises and Bias drops while held-out accuracy changes little (0.679 to 0.699), supporting the design choice of training a dedicated FoK channel over knowledge-delta states rather than relying on knowledge learning alone. FGPO raises held-out AUROC from 0.770 to 0.809 in this run and incorporates the judgment into generation. On the separate Qwen3-8B evaluation, its nearly binary judgments explain the flattened threshold curves in Figure 4.

StageAcc.↑\uparrowAUROC↑\uparrowIllu.↓\downarrowOm.↓\downarrowBias→0\to0
Base0.6800.7020.6530.1090.381
Stage 1: + OP-SFT0.9660.5750.5890.3190.454
Stage 2: + KΔ\Delta-FT0.9670.8030.3500.1900.270
Stage 3: + FGPO0.9610.8170.2500.2400.245

Table 26. Update-set results for Base and the three cumulative training stages.

Related work

Functional metacognition. Metacognitive sensitivity and calibration are distinct quantities [11]. Human–LLM comparisons therefore require explicit behavioral definitions rather than equating similar reports with identical cognitive processes [46]. Recent work studies competing confidence biases under external advice [29] and how verbal confidence is computed from answer-related representations [28]. Our focus is correctness-supervised answerability judgment under changing model parameters. Koriat’s cue-accessibility account motivates the context analyses, but our objective does not directly model that account.

LLM uncertainty and hallucination. LLMs often generate fluent but factually false outputs, a failure studied as hallucination and factuality [16, 19]. Theoretical analyses show that this failure cannot be removed by pretraining alone: rare facts impose a Good–Turing-style lower bound on calibrated error, and common training and evaluation procedures can reward guessing rather than explicit uncertainty [22, 23]. This motivates uncertainty estimation and answerability judgment in LLMs [12]. Existing methods estimate uncertainty from generated outputs, including P(True)P(\mathrm{True}) and P(IK)P(\mathrm{IK}) (probability that the model “knows” the answer), verbalized confidence, explicit “I don’t know” (IDK) training, semantic entropy, and self-consistency [8, 9, 21, 26, 27, 36, 50, 51, 54]. Other methods read uncertainty from internal representations, including SAPLMA, CCS, INSIDE, MIND, C3C^{3}, and internal-state probes [1, 3, 4, 42, 45, 47]. A related line trains answerability behavior directly: R-Tuning teaches models to state that they do not know [56]; US-Tuning, CRaFT, and GRAIT analyze conflicts between answer learning and negative answerability supervision [32, 58, 59]; EliCal uses annotation-efficient elicitation and calibration for honesty [41]. Inference-time factuality methods such as ITI and DoLa intervene without additional training [5, 34], while TruthfulQA, HaluEval, FActScore, and SimpleQA evaluate factual support in generated answers [33, 35, 40, 53]. These works primarily ask whether a given model can express or reveal uncertainty about its current answers. We instead ask whether the FoK judgment tracks post-training updates to the model’s knowledge.

Knowledge editing and continual learning. Post-training updates, knowledge editing, and continual learning change what an LLM knows after pretraining. ROME and MEMIT edit factual associations and evaluate edit success, locality, portability, and retention [38, 39]. Continual-learning methods, including regularization and replay-based approaches, aim to preserve answer accuracy across sequential updates [17, 24, 37]. These methods focus on the update operator and on answer correctness, and usually do not test whether answerability judgment changes with the updated knowledge. We instead treat FoK as a training target co-learned with each update, so the answerability judgment moves with the model rather than lagging behind on a fixed checkpoint.

Conclusion

We studied how to keep functional FoK aligned with an LLM’s answerability after knowledge updates. In the tested factual-QA settings, knowledge updating alone does not preserve this alignment. Our two-channel framework trains knowledge and FoK separately and then optimizes the judgment together with the subsequent response. Controlled OP-SFT comparisons, cross-dataset evaluation, and three sequential updates support the usefulness of explicit FoK adaptation, with seed variation quantifying its stability in the main setting. These results concern behavioral answerability judgments; they do not establish human-like subjective experience, superiority over strong estimators on a non-updated model, or generalization to arbitrary update operators.

Limitations and future work

We evaluate the framework on short-answer factual QA with gradient-based updates, including three sequential update rounds. The MEMIT experiment in Appendix § 14 tests desynchronization under one non-gradient editor, not whether our full framework repairs that editor. Longer update sequences, procedural tasks such as code generation and mathematical proof, and backbones whose baseline FoK signal is near chance remain outside the tested scope. Strong unadapted estimators remain competitive without an update. Our main evidence concerns FoK synchronization after knowledge changes, not universal superiority of the trained channel.

We measure functional FoK as a behavioral answerability judgment, without claiming subjective experience. KΔK_{\Delta}-FT supervises answer correctness across knowledge states rather than explicitly modeling human cue-based inference. The cue analyses do not include a matched pretrained-model control, so they cannot establish whether training induces cue sensitivity or preserves sensitivity already in the backbone. Causal recovery locations depend on the prompt, token positions, and intervention; they do not establish a uniquely localized or human-equivalent metacognitive mechanism. Seed variation covers Stages 1–2 and FGPO on the main TriviaQA setting, not every preliminary or cross-dataset experiment.

Broader impact

This work targets reliability after LLM knowledge updates by making answerability judgments track the updated model state. The intended positive impact is fewer unsupported answers after fine-tuning or continual updates. A possible negative impact is that better answerability judgments could increase trust in systems whose deployment still depends on data quality, monitoring, and domain-specific safety constraints. The method is not a content-safety system on its own and should be combined with deployment-specific safeguards before release.

Additional experimental settings

Data construction. The main experiments use TriviaQA as the knowledge-update corpus and the in-domain QA benchmark. The update set contains 5,000 questions, and the held-out TriviaQA test split contains 1,000 questions. For OP-SFT, we sample multiple responses from the current answer model for each update question, repair the answer-relevant span when the sampled response is wrong, and train only the corrected answer span. The reported main setting uses 32 sampled responses per question. General capability is evaluated separately on MMLU and ARC-Easy, using the same generator whose TriviaQA result is reported in the main table.

Evaluation protocol. Answer accuracy on TriviaQA is the fraction of generated responses that match the gold answer, with surface-form equivalence checked by Qwen2.5-32B-Instruct [48] used as an LLM judge [57]. MMLU and ARC-Easy use their standard multiple-choice accuracy. Generation and judging are run with vLLM in bfloat16 precision; the 32B judge uses tensor parallelism across the available GPUs. FoK metrics use the model’s pre-answer Yes/No judgment score: AUROC is computed from the normalized probability of the Yes token, and illusion, omission, and Bias are computed from the thresholded Yes/No prediction. Since illusion and omission are conditional error rates in [0,1][0, 1], the signed Bias score lies in [−1,1][-1, 1]. Positive values mean illusion errors dominate, and negative values mean omission errors dominate.

Baseline protocol. All FoK estimators are evaluated under the answer model used by that row. Base rows use the unadapted model; Vanilla K-SFT rows use the knowledge-updated model; Ours uses the combined knowledge and FoK adapters. Verbal-0 and Verbal-10 ask the model for a direct confidence judgment, Prob and N-Prob use sequence likelihood, Consis-Lex and Consis-Sem use multiple sampled responses, semantic entropy clusters sampled responses by meaning, and EliCal uses the released EliCal adapter whenever its public checkpoint is available for the backbone. Unlike estimators that only score fixed answers, loading the EliCal adapter can change answering behavior and answer accuracy. The fixed-answer EliCal transfer protocol in Appendix § 14 is reported separately.

Training protocol. All trainable components are implemented as LoRA adapters on top of Qwen2.5-7B-Instruct. Vanilla K-SFT is trained with standard supervised fine-tuning on the same update questions. OP-SFT trains the knowledge adapter from repaired sampled responses. KΔ\Delta-FT freezes the OP-SFT knowledge checkpoints and trains a FoK adapter against labels collected from those knowledge states. FGPO then uses the knowledge and FoK adapters jointly for response optimization. Table 5 lists the full training hyperparameters.

ParameterValue
BackboneQwen2.5-7B-Instruct
Training frameworkLLaMA Factory
Adapter typeLoRA, rank 16, alpha 32, dropout 0
ParallelismMulti-GPU DDP with DeepSpeed ZeRO-2
Precisionbfloat16
Vanilla K-SFT LR / epochs10−410^{-4} / 5
OP-SFT sampled responses / epochs32 / 1
OP-SFT LR2×10−52 \times10^{-5}
KΔ\Delta-FT LR / epochs10−410^{-4} / 3
FGPO LR / epochs10−610^{-6} / 1
Per-device batch size1
Gradient accumulation1
ScheduleCosine with 0.03 warmup ratio
FoK scorenormalized Yes/No token probability

Table 5. Training hyperparameters for the main experiments.

Compute resources. The main experiments were run on a single multi-GPU server. Training used multi-GPU DDP with DeepSpeed ZeRO-2. Generation and judging used vLLM in bfloat16 precision, and the Qwen2.5-32B judge used tensor parallelism across the available GPUs. We report the parallelism and training configuration needed for reproduction, but did not log exact wall-clock time for every preliminary and failed run.

FGPO reward values. Table 6 lists the implemented FGPO rewards. The base reward gives RFoK+RansR_{\mathrm{FoK}} + R_{\mathrm{ans}} for each combination of answerability target, judgment, and response outcome. An output from which no yes/no judgment can be parsed receives −1.0-1.0 in place of the base reward. A further +0.1+0.1 is added when the output follows the full <think></think><response> format, +0.1+0.1 when generation ends with the EOS token and −0.2-0.2 otherwise, and the length term uses λ=0.001\lambda= 0.001 with ∣h∣|h| measured in characters of the reasoning trace.

YYjjResponse outcomeBase reward
1yescorrect answer+2.0
1yeswrong answer−1.5
1no—−1.0
0novalid explanation+0.5
0noinvalid explanation−0.5
0yescorrect answer−0.5
0yeswrong answer−2.0

Table 6. Base FGPO reward RFoK+RansR_{\mathrm{FoK}} + R_{\mathrm{ans}} by answerability target YY, judgment jj, and response outcome.

Existing assets and terms of use. Table 7 lists the external datasets, models, and code packages used in the experiments. We use each asset only for research evaluation or fine-tuning under the terms specified by the original publisher.

AssetUse in this paperLicense or terms
TriviaQAKnowledge-update corpus and in-domain QA benchmarkApache-2.0 (source)
MMLUGeneral capability benchmarkMIT License (source)
ARC-EasyGeneral capability benchmarkCC BY-SA 4.0 (source)
Qwen2.5-7B-Instruct and Qwen2.5-32B-InstructMain answer model and LLM judgeApache-2.0 (Qwen 7B and 32B model cards)
Qwen3-8BThinking-mode backbone for the selective-prediction evaluationApache-2.0 (Qwen model card)
LLaMA-3.1 and LLaMA-2 checkpointsSupplementary FoK synchronization results in Appendix GMeta Llama 3.1 Community License and Llama 2 Community License (3.1, 2)
LLaMA FactoryLoRA fine-tuning frameworkApache-2.0 (source)
vLLMBatched generation and LLM judgingApache-2.0 (source)
EleutherAI lm-evaluation-harnessMMLU and ARC-Easy evaluationMIT License (source)
EliCal official adapterFoK baselinePublisher model-card terms (source)

Table 7. External assets used in the experiments and their publisher-specified licenses or terms of use.

Causal tracing and weight-delta supplementary figures

Per-layer indirect effect on true-positive samples for instruction-first prompts across six Qwen2.5 sizes

Figure 5. Per-layer indirect effect on true-positive (TP) samples for the instruction-first (IF) prompt across six Qwen2.5 sizes. The solid orange curve shows the question-region effect, the dashed purple curve shows the suffix-region effect, and the red dashed line marks two-thirds of model depth. The question-region effect concentrates in early and middle layers, whereas the suffix-region effect rises mainly after this boundary at every tested scale.

Figure 6. Layer-wise weight delta ∥ΔWℓ∥F\|\Delta W_\ell\|_F after one epoch of K-SFT (red) vs. J-SFT (blue) on Qwen2.5-7B (full SFT from base, same data budget). K-SFT changes concentrate in shallow/mid layers, while J-SFT changes remain small until the deep layers. This weight-side pattern is consistent with the depth separation observed in causal tracing.

Figure 7. ROME-style causal tracing on Qwen2.5-14B, broken down by outcome group. Rows: TP (answers correctly and judges yes), FP (answers incorrectly but judges yes), TN (answers incorrectly and judges no), and FN (answers correctly but judges no). Each row shows three samples, except FN, which has only one sample with p>0.4p > 0.4. Each cell is the layer × token restoration heatmap; tokens are grouped into Sys (system), K-Q (knowledge question), and Hdr (response header). The red dashed line at layer 29 marks the depth boundary used in our analysis. The suffix-region (Hdr) signal appears mainly in deep layers across all four outcome groups, suggesting that the late FoK signal does not simply track answer correctness.

FoK dependency on knowledge

Figure 8a. FoK recovery after knowledge SFT. In this evaluation, sequential K→J training improves FoK but remains below direct J-SFT across all six Qwen2.5 scales.

Figure 8b. Reverse direction. Knowledge exact-match before and after J-SFT across Qwen2.5 scales. J-SFT has a limited effect on answer performance and can even yield modest improvements.

fig:8: FoK dependence on updated knowledge. Left: In this evaluation, K→J training improves FoK but does not reach direct J-SFT performance. Right: J-SFT alone barely shifts answer accuracy.

FoK judgment along OP-SFT training

Figure 9 tests whether the trained FoK channel only works at the final OP-SFT checkpoint. Answer accuracy changes substantially along the OP-SFT trajectory, while FoK AUROC remains above 0.80 on both splits for every checkpoint. The thresholded errors show a calibration shift: omission is larger near the base model, whereas illusion grows on later held-out checkpoints. Thus KΔ-FT provides a stable ranking signal across the knowledge trajectory, but threshold calibration still depends on the underlying knowledge state of the model.

Figure 9. In a separate small-scale study, we evaluate a fixed, trained KΔ-FT FoK channel across successive OP-SFT knowledge checkpoints. We report answer accuracy, FoK AUROC, and thresholded illusion and omission errors.

FoK synchronization results across model scale and family

Knowledge training and FoK degradation. Consistent with the main-text findings, the K-SFT results in Table 8 show that knowledge training can improve answer accuracy while impairing FoK. An update optimized for answering alone need not preserve the model’s ability to distinguish its correct answers from errors, motivating explicit FoK synchronization rather than assuming that knowledge gains automatically improve self-judgment.

ModelBaseK-SFTJ-SFTK→J SFT
EMAUROCEMAUROCIllu.Om.AUROCEMAUROCIllu.
0.5B4.2%0.52316.9%0.5020%99%0.80113.1%0.7633%
1.5B30.9%0.52735.5%0.50066%48%0.82434.8%0.81528%
3B35.4%0.72445.7%0.51924%80%0.83045.4%0.82240%
7B47.2%0.73555.8%0.59228%51%0.85555.1%0.83382%
14B59.8%0.75665.2%0.76773%9%0.85066.1%0.81485%
32B61.2%0.81266.8%0.79280%5%0.84466.8%0.83191%

Table 8. Supplementary Qwen2.5 size-scaling experiments using a dedicated setup with 5,000 update questions and 1,000 held-out questions.

LLaMA-2 negative result. The LLaMA-2 results reveal a distinction between acquiring knowledge and judging answerability. Although answer accuracy improves, FoK remains at chance after both knowledge and judgment training (Table 9). This suggests that correctness supervision alone is insufficient to elicit a discriminative FoK signal in this setting: the model can learn answers without learning to distinguish when its own answers are correct.

ModelBase AUROCK-SFT AUROCJ-SFT AUROCK→\toJ SFT AUROCK→\toJ SFT EMNote
Qwen2.5-7B0.7350.5920.8550.83355.1%Reference
LLaMA-3.1-8B (LR = 2×10−62 \times10^{-6})0.6460.5030.7860.8368.6%K→\toJ forgetting
LLaMA-3.1-8B (LR = 10−610^{-6})0.6460.5260.7820.74973.5%LR corrected
LLaMA-2-7B0.5200.5000.5000.50063.5%No FoK improvement
LLaMA-2-13B0.5250.5000.5000.50070.1%No FoK improvement

Table 9. Cross-family FoK synchronization results with 5,000 update questions and 1,000 held-out questions.

Additional validation

We conduct additional small-scale experiments, each with its own training and evaluation setup, to examine different aspects of FoK behavior.

Accuracy-preserving updates and sequential knowledge changes

OP-SFT comparison. Table 10 compares FoK estimators starting from the same OP-SFT knowledge checkpoint. Adding a FoK adapter changes the combined model state, and the KΔK\Delta-FT row therefore has a different answer accuracy. Base rows are a reference for the no-update regime, not evidence that our method uniformly improves on unadapted estimators. For Base, EliCal uses its released adapter, which is trained extensively and therefore changes answering behavior. When combined with OP-SFT, EliCal uses an adapter with only a small amount of training, so answer accuracy is unchanged.

Answer modelFoK estimatorTest Acc.Test AUROC
BaseDirect judgment0.6720.688
BaseProb0.6720.730
BaseEliCal0.6220.810
OP-SFTDirect judgment0.6790.603
OP-SFTProb0.6790.571
OP-SFTEliCal0.6790.618
OP-SFT + FoK adapterKΔK\Delta-FT0.6990.770

Table 10. FoK evaluation before and after OP-SFT on the 1,000-question held-out TriviaQA split. All OP-SFT rows start from the same knowledge checkpoint.

Three sequential updates. We divide 5,000 TriviaQA update questions into three disjoint shards. Round ii applies OP-SFT with on-policy rollouts to the previous round’s model using shard ii. The frozen FoK channel φ1\varphi_1 is trained after round 1 and kept fixed thereafter. The adaptive channel φi\varphi_i is trained with KΔK\Delta-FT on each round’s knowledge trajectory. All held-out columns use the same 1,000 questions. Table 11 reports before/after answer accuracy on the new shard for that round; it is not cumulative accuracy over all previously updated facts and does not measure their retention. The adaptive channel maintains discrimination and reduces illusion in later rounds, but held-out answer accuracy remains below Base.

StateAcc. before → afterTest Acc.φ1\varphi_1 (frozen) AUROCφ1\varphi_1 (frozen) Illusionφi\varphi_i (adaptive) AUROCφi\varphi_i (adaptive) Illusion
Base—0.672————
Round 10.575 → 0.9720.6590.7980.4550.7980.455
Round 20.676 → 0.9380.6440.7760.6880.8000.202
Round 30.631 → 0.9380.6610.7540.7520.7970.195

Table 11. Three sequential TriviaQA updates. Illusion is the conditional rate of predicting Yes on questions the current answering model gets wrong.

Cross-dataset evaluation and a non-gradient update

Zero-shot transfer. We evaluate the TriviaQA-trained systems on SelfAware (3,369 questions), Natural Questions (3,610), and WebQuestions (2,032), without additional training (Table 12). FoK targets each evaluated model’s own answer correctness. In SelfAware, unanswerable questions count as incorrect answers; this adapts the original testbed to model-specific answerability rather than using its answerability annotation as a fixed target. In this transfer experiment, EliCal scores the fixed answers of the corresponding Base or OP-SFT generator, so each paired row has the same answer accuracy. This differs from the adapter-based answering evaluation of Table 10.

SystemSA Acc.SA AUROCNQ Acc.NQ AUROCWebQ Acc.WebQ AUROC
Base0.2820.7680.3760.6650.5270.597
Base + EliCal0.2820.7020.3760.7350.5270.636
Vanilla K-SFT0.1870.5690.1910.5710.2240.519
OP-SFT + EliCal0.3050.5900.3810.5790.5390.515
Ours0.3050.8100.3810.7330.5390.644

Table 12. Zero-shot transfer from TriviaQA. SA: SelfAware; NQ: Natural Questions; WebQ: WebQuestions. Acc. measures answer correctness, whereas AUROC measures functional FoK discrimination.

SystemAUROCTPRTNRAnswer Acc.
Base0.7680.3880.8980.282
Vanilla K-SFT0.5690.3020.7600.187
Ours0.8100.7650.7010.305

Table 13. SelfAware detail for the same zero-shot systems. TPR is the rate of predicting Yes on correctly answered questions; TNR is the rate of predicting No on incorrectly answered questions.

Natural Questions replication. Separately from zero-shot transfer, we run the full pipeline on a 5,000-update / 1,000-held-out Natural Questions split (Table 14). The resulting update-shard accuracy is 0.925. This replicates the knowledge-update comparison on a second factual-QA dataset, without implying improvement across all QA settings or update operators.

SystemHeld-out Acc.FoK AUROC
Base0.3190.629
Vanilla K-SFT0.1990.585
Ours0.3660.647

Table 14. Full-pipeline replication on Natural Questions. This held-out split differs from the zero-shot population in Table 12.

Table 14: Full-pipeline replication on Natural Questions. This held-out split differs from the zero-shot population in Table 12.

MEMIT. We apply one MEMIT batch edit [39] to 200 facts in the zsRE setting [31]. Table 15 measures the edited facts and the same 1,000 held-out TriviaQA questions before and after editing. Edited-fact accuracy rises while FoK AUROC decreases. This is a test of desynchronization under a non-gradient operator; we do not evaluate KΔ\Delta-FT or FGPO as a repair for MEMIT.

Train Acc.Train AUROCTest Acc.Test AUROC
Before edit0.0350.5940.6330.726
After edit0.6400.5530.6270.709

Table 15. Knowledge and functional FoK before and after a MEMIT edit. Train denotes 200 edited zsRE facts; Test denotes 1,000 held-out TriviaQA questions. These states are compared only within this experiment.

Table 15: Knowledge and functional FoK before and after a MEMIT edit. Train denotes 200 edited zsRE facts; Test denotes 1,000 held-out TriviaQA questions. These states are compared only within this experiment.

Training-seed variation

Table 16 reports independent training runs for Vanilla K-SFT (three seeds) and OP-SFT + KΔ\Delta-FT (four seeds). The Vanilla K-SFT seed runs train a LoRA adapter (rank 16, alpha 32) on standard SFT data built from the update questions, with learning rate 10−410^{-4}, 5 epochs, per-device batch size 8, and gradient accumulation 1. Table 17 varies the FGPO seed from one shared Stage-1+2 initialization. We report the arithmetic mean and sample standard deviation, s=∑i(xi−xˉ)2/(n−1)s = \sqrt{\sum_i (x_i-\bar{x})^2/(n-1)}, over available runs; the unrun Vanilla seed-42 cells are excluded. These statistics quantify training-seed variation under the fixed evaluation split and are neither confidence intervals nor significance tests. The FGPO statistics are conditional on the shared initialization and do not quantify full-pipeline seed variation. Other supplementary experiments are reported as individual evaluations.

SeedVanilla K-SFT Acc.Vanilla K-SFT AUROCOP-SFT + KΔ\Delta-FT Acc.OP-SFT + KΔ\Delta-FT AUROC
42——0.6990.770
430.5220.6350.6650.770
440.5220.6270.6700.790
00.5200.6190.6650.778
Mean ± SD0.521 ± 0.0010.627 ± 0.0080.675 ± 0.0160.777 ± 0.009

Table 16. Training-seed variation of Vanilla K-SFT and OP-SFT + KΔ\Delta-FT on held-out TriviaQA.

StateTest Acc.Test AUROC
Shared Stage-1+2 initialization0.6990.770
FGPO, seed 10.6920.809
FGPO, seed 430.6770.810
FGPO, seed 440.6720.815
FGPO mean ± SD0.680 ± 0.0100.811 ± 0.003

Table 17. FGPO seed variation from one shared Stage-1+2 initialization, evaluated end to end on held-out TriviaQA. Seed 1 is the original full-method run.

Table 16: Training-seed variation of Vanilla K-SFT and OP-SFT + KΔ\Delta-FT on held-out TriviaQA.

Table 17: FGPO seed variation from one shared Stage-1+2 initialization, evaluated end to end on held-out TriviaQA. Seed 1 is the original full-method run.

Functional FoK and accessible cues

These analyses measure behavior of the trained Qwen2.5-7B FoK channel. They are consistent with cue-sensitive judgments but do not establish that KΔK_{\Delta}-FT induces that sensitivity, because a matched pretrained control was not included. They also do not test subjective experience.

Consistency at fixed incorrectness. From a 5,000-question TriviaQA pool, we sample 16 answers per question and retain the 1,149 questions where all 16 answers are wrong. Consistency is the fraction giving the most frequent incorrect answer. FoK rises with this consistency even though sampled correctness is zero (Table 18). The reported Spearman correlation is ρ=0.24\rho= 0.24 (p=2×10−16p = 2 \times10^{-16}).

Answer consistencyQuestionsMean JJ
0–0.28400.413
0.2–0.41780.486
0.4–0.6710.571
0.6–0.8290.591
0.8–1.0310.808

Table 18.

Table 18: Functional FoK among 1,149 questions with 16/16 incorrect sampled answers. Bin labels follow the evaluation grouping.

Accessible reasoning. We progressively reveal the generated reasoning trace on 440 MATH problems using the reasoning-prefix prompts of the preliminary study. Table 19 shows better discrimination as more of the trace becomes available. The mean-JJ gap is the difference between correct and incorrect solution groups. This does not establish a particular human cognitive mechanism.

Reasoning shownAUROCMean-JJ gap
Question only0.600.05
25%0.660.09
50%0.670.10
75%0.740.15
Full reasoning0.910.32

Table 19.

Table 19: FoK discrimination with progressively revealed reasoning on 440 MATH problems.

Context intervention. We prepend short contexts to a filtered 1,460-question subset of the same pool, removing contexts containing a gold answer. Table 20 reports FoK and answer accuracy; deltas are relative to the bare question. Related versus unrelated words raise mean JJ by 0.490 (paired Wilcoxon test on per-question JJ, reported p=5×10−239p = 5 \times10^{-239}). Incorrect related sentences increase JJ while slightly decreasing accuracy, showing that the judgment is not determined by correctness alone.

FormatConditionJJΔJ\Delta JAcc.Δ\DeltaAcc.
—None0.599—0.408—
WordsUnrelated0.184−0.4150.424+0.016
WordsRelated0.674+0.0750.478+0.070
SentencesUnrelated0.298−0.3010.436+0.028
SentencesCorrect related0.677+0.0780.510+0.102
SentencesIncorrect related0.772+0.1730.397−0.011

Table 20.

Table 20: Context interventions on 1,460 questions after filtering gold-answer-containing contexts.

Prompt sensitivity and alternative knowledge updates

Judgment templates. We repeat causal tracing on the same 40 Qwen2.5-7B samples under six judgment templates (Table 21). Recovery peaks stay at layers 20–21 of 28, with sensitive ranges within layers 16–23. This supports robustness within the tested templates, not prompt invariance in general. The templates cover both rephrasings and label changes; the sensitive range denotes the layers with strong recovery.

TemplatePeak layerSensitive range
Original2016–22
A: rephrased2117–22
B: third-person elicitation2116–21
C: numeric labels 1/22118–23
D: letter labels A/B2118–21
E: labels can/cannot2018–20

Table 21. Causal-tracing sensitivity to judgment templates on 40 shared samples. Layers are reported out of 28.

Correction strategies. Table 22 compares knowledge-update strategies on Qwen2.5-7B-Instruct using 5,000 TriviaQA update questions and 1,000 held-out questions from a split different from the main tables. General capability is measured on the 1,319-question GSM8K test set [7]. Reptile uses first-order meta-updates [43]; SDFT uses the released self-distillation setup [44]. The lookahead variant penalizes deviation from Base after a one-step SFT update; the proximal variant adds an L2 constraint to Base. Average is the unweighted mean of the three reported accuracies and is given only when all three are available; it mixes seen-data and held-out metrics and is not a generalization-only score. Span replacement has the highest such average, but does not dominate Base on GSM8K.

StrategyUpdate Acc.Held-out Acc.GSM8KAverage
Base0.6850.6030.7980.695
Vanilla SFT0.9920.4520.3680.604
Reptile meta-update0.5580.5840.6400.594
SDFT0.5290.5650.3670.487
Unlikelihood on wrong span0.5000.528——
One-step lookahead KL0.7380.6190.6460.668
L2 proximal constraint0.5600.5870.6390.595
Replace wrong span0.8320.6260.6530.704

Table 22. Alternative knowledge-update strategies on a separate TriviaQA split. Unlikelihood GSM8K was not reported, so its average is omitted.

Supplementary assets. Table 23 records publisher-specified terms for additional datasets and the editing implementation. Dataset terms are distinguished from repository software licenses, particularly for Natural Questions. The exact zsRE distribution, SDFT implementation revision, and all auxiliary checkpoint licenses are not identified in the available experiment record; this version therefore does not claim a complete asset-license audit.

AssetRoleLicense or terms
SelfAwareZero-shot QACC BY-SA 4.0 (source)
Natural QuestionsTransfer and replicationCC BY-SA 3.0 (data source)
WebQuestionsZero-shot QACC BY 4.0 (source)
MATHReasoning evaluationMIT (repository)
GSM8KCapability evaluationMIT (repository)
MEMITEditing implementationMIT (repository)

Table 23. Additional assets and publisher-specified terms.

Detailed results for the main-text tables

Knowledge and judgment SFT

These tables retain all the error-rate columns of Table 1. Accuracy, illusion, and omission are percentages; their changes are percentage points. AUROC and its changes are unitless. Illusion and omission are conditional error rates, rather than fractions of all evaluated questions.

ModelAcc. (%)AUROCIllusion (%)Omission (%)
Qwen2.5-0.5B200.5192885
Qwen2.5-1.5B350.5295638
Qwen2.5-3B390.739868
Qwen2.5-7B530.7245120
Qwen2.5-14B660.7625019
Qwen2.5-32B680.819628
LLaMA-3.1-8B740.6728116

Table 24. Base values for the preliminary SFT comparison.

Table 25: Changes from Base under K-SFT, J-SFT, and sequential K→JK \to J SFT. Positive accuracy and AUROC changes indicate improvement; negative illusion and omission changes indicate improvement. All source values are retained.

ModelK-SFT Acc.K-SFT AUCK-SFT Illu.K-SFT Om.J-SFT Acc.J-SFT AUCJ-SFT Illu.J-SFT Om.K→JK \to J SFT Acc.K→JK \to J SFT AUCK→JK \to J SFT Illu.K→JK \to J SFT Om.
Qwen2.5-0.5B+2-0.016-27+13-14+0.213-28+15+2+0.211-21-10
Qwen2.5-1.5B+6-0.035+6-2+2+0.255-28-6+5+0.248-21-12
Qwen2.5-3B+12-0.220+10+18+3+0.038+11-29+11+0.051+20-41
Qwen2.5-7B+7-0.042-15+19+2+0.088-9-4+7+0.100+6-12
Qwen2.5-14B+6+0.004+9-4+1+0.030+9-9+5+0.068+29-17
Qwen2.5-32B+4-0.026+1100+0.010-10+2+4+0.020+3-3
LLaMA-3.1-8B+2-0.038-11+9+1+0.045+7-11+2+0.063+14-14

Table 25. Changes from Base under K-SFT, J-SFT, and sequential K→JK \to J SFT.

Update-set stage-wise ablation

These are the update-set columns of the same runs used in Table 4; they do not measure held-out generalization.

References

  1. [1]Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. arXiv preprint arXiv:2304.13734, 2023.arxiv.org/abs/2304.13734
  2. [2]Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, 2013. URL https://aclanthology.org/D13-1160/.DOI
  3. [3]Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2212.03827
  4. [4]Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs’ internal states retain the power of hallucination detection. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Zj12nzQ1Qbz.arxiv.org/abs/2402.03744
  5. [5]Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. DoLa: Decoding by contrasting layers improves factuality in large language models. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Th6NyL07na.
  6. [6]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Ovyind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge, 2018.
  7. [7]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
  8. [8]Roi Cohen, Konstantin Dobler, Eden Biran, and Gerard de Melo. I don’t know: Explicit modeling of uncertainty with an [IDK] token. In Advances in Neural Information Processing Systems, 2024. doi: 10.52202/079017-0349.
  9. [9]Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630:625–630, 2024. doi: 10.1038/s41586-024-07421-0.
  10. [10]Tom Fawcett. An introduction to ROC analysis. Pattern Recognition Letters, 27(8):861–874, 2006. doi: 10.1016/j.patrec.2005.10.010.DOI
  11. [11]Stephen M. Fleming and Hawan C. Lau. How to measure metacognition. Frontiers in Human Neuroscience, 8:443, 2014. doi: 10.3389/fnhum.2014.00443.DOI
  12. [12]Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6577–6595. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.naacl-long.366. URL https://aclanthology.org/2024.naacl-long.366/.
  13. [13]Joseph T. Hart. Memory and the feeling-of-knowing experience. Journal of Educational Psychology, 56(4):208–216, 1965.pubmed.ncbi.nlm.nih.gov/5825050
  14. [14]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
  15. [15]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
  16. [16]Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2024. doi: 10.1145/3703155.
  17. [17]Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. In International Conference on Learning Representations (ICLR), 2022.
  18. [18]Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C. Park. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7036–7050. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.naacl-long.389. URL https://aclanthology.org/2024.naacl-long.389/.DOI
  19. [19]Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezhen Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023. doi: 10.1145/3571730.DOI
  20. [20]Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147/.
  21. [21]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. doi: 10.48550/arXiv.2207.05221.DOI
  22. [22]Adam Tauman Kalai and Santosh S. Vempala. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, 2024. doi: 10.1145/3618260.3649777.DOI
  23. [23]Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. Why language models hallucinate. arXiv preprint arXiv:2509.04664, 2025. doi: 10.48550/arXiv.2509.04664.
  24. [24]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabsk-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13): 3521–3526, 2017. doi: 10.1073/pnas.1611835114.DOI
  25. [25]Asher Koriat. How do we know that we know? The accessibility model of the feeling of knowing. Psychological Review, 100(4):609–639, 1993.pubmed.ncbi.nlm.nih.gov/8255951
  26. [26]Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shres hth A. Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in LLMs. arXiv preprint arXiv:2406.15927, 2024. doi: 10.48550/arXiv.2406.15927.DOI
  27. [27]Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=VD_AYtP0dve.arxiv.org/abs/2302.09664
  28. [28]Dharshan Kumaran, Arthur Conmy, Federico Barbero, Simon Osindero, Viorica Patraucean, and Petar Veličković. How do LLMs compute verbal confidence, 2026. URL https://arxiv.org/abs/2603.17839.
  29. [29]Dharshan Kumaran, Stephen M. Fleming, Larisa Markeeva, Joe Heyward, Andrea Banino, Mrinal Mathur, Razvan Pascanu, Simon Osindero, Benedetto De Martino, Petar Veličković, and Viorica Patraucean. Competing biases underlie overconfidence and underconfidence in LLMs. Nature Machine Intelligence, 8:614–627, 2026. doi: 10.1038/s42256-026-01217-9.DOI
  30. [30]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026/.DOI
  31. [31]Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, 2017.arxiv.org/abs/1706.04115
  32. [32]Jiaqi Li, Yixuan Tang, and Yi Yang. Know the unknown: An uncertainty-sensitive method for LLM instruction tuning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 2972–2989. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-acl.153. URL https://aclanthology.org/2025.findings-acl.153/.DOI
  33. [33]Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.397. URL https://aclanthology.org/2023.emnlp-main.397/.DOI
  34. [34]Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2306.03341
  35. [35]Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.acl-long.229. URL https://aclanthology.org/2022.acl-long.229/.DOI
  36. [36]Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022.arxiv.org/abs/2205.14334
  37. [37]Yun Luo, Zhicheng Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023.arxiv.org/abs/2308.08747
  38. [38]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2202.05262
  39. [39]Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.07229
  40. [40]Sewon Min, Kalpesh Krishna, X inxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannah Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.741. URL https://aclanthology.org/2023.emnlp-main.741/.DOI
  41. [41]Shiyu Ni, Keping Bi, Jiafeng Guo, Minghao Tang, Jingtong Wu, Zengxin Han, and Xueqi Cheng. Annotation-efficient universal honesty alignment. arXiv preprint arXiv:2510.17509, 2025. doi: 10.48550/arXiv.2510.17509.DOI
  42. [42]Shiyu Ni, Keping Bi, Jiafeng Guo, Lulu Yu, Baolong Bi, and Xueqi Cheng. Towards fully exploiting LLM internal states to enhance knowledge boundary perception. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24315–24329. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.1184. URL https://aclanthology.org/2025.acl-long.1184/.DOI
  43. [43]Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018. URL https://arxiv.org/abs/1803.02999.
  44. [44]Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning, 2026. URL https://arxiv.org/abs/2601.19897.
  45. [45]Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3607–3625. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.220. URL https://aclanthology.org/2023.emnlp-main.220/.DOI
  46. [46]Mark Steyvers and Megan A. K. Peters. Metacognition and uncertainty communication in humans and large language models. Current Directions in Psychological Science, 35(3): 131–139, 2026. doi: 10.1177/09637214251391158.DOI
  47. [47]Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. Unsupervised real-time hallucination detection based on the internal states of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14379–14391. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.findings-acl.854. URL https://aclanthology.org/2024.findings-acl.854/.DOI
  48. [48]Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. doi: 10.48550/arXiv.2412.15115.DOI
  49. [49]Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. doi: 10.48550/arXiv.2505.09388.
  50. [50]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.330. URL https://aclanthology.org/2023.emnlp-main.330/.DOI
  51. [51]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2203.11171
  52. [52]Yile Wang, Peng Li, Maosong Sun, and Yang Liu. Self-knowledge guided retrieval augmentation for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10303–10315. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.691. URL https://aclanthology.org/2023.findings-emnlp.691/.DOI
  53. [53]Jason Wei, Karina Nguyen, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models (SimpleQA). arXiv preprint arXiv:2411.04368, 2024. doi: 10.48550/arXiv.2411.04368.
  54. [54]Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=gjEqKFxFpZ.arxiv.org/abs/2306.13063
  55. [55]Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don’t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653–8665, 2023. doi: 10.18653/v1/2023.findings-acl.551. URL https://aclanthology.org/2023.findings-acl.551/.DOI
  56. [56]Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-Tuning: Instructing large language models to say ‘I don’t know’. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7113–7139. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.naacl-long.394. URL https://aclanthology.org/2024.naacl-long.394.
  57. [57]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhu, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuangzhuang Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2306.05685
  58. [58]Runchuan Zhu, Zinco Jiang, Jiang Wu, Zhipeng Ma, Jiahe Song, Fengshuo Bai, Dahua Lin, Lijun Wu, and Conghui He. GRAIT: Gradient-driven refusal-aware instruction tuning for effective hallucination mitigation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4006–4021. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-naacl.223. URL https://aclanthology.org/2025.findings-naacl.223/.DOI
  59. [59]Runchuan Zhu, Zhipeng Ma, Jiang Wu, Junyuan Gao, Jiaqi Wang, Dahua Lin, and Conghui He. Utilize the flow before stepping into the same river twice: Certainty represented knowledge flow for refusal-aware instruction tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26157–26165, 2025. doi: 10.1609/aaai.v39i24.34812.

Paper details

vlnc:W_c5c85gw7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents