WorksComputer Science
Feeling of Knowing in Large Language Models
Abstract
Post-training updates change what large language models (LLMs) can answer, but may leave their answerability judgments misaligned. We study this knowledge-judgment desynchronization through functional feeling of knowing (FoK), which supports selective generation, retrieval triggering, and model routing without implying subjective experience. We propose two trainable channels: a knowledge channel that changes answerability and a FoK channel that judges the current knowledge state. Three stages train them: on-policy supervised fine-tuning (OP-SFT) updates knowledge from corrected on-policy answers; knowledge-delta FoK training (K-FT) adapts judgments across intermediate knowledge states; and FoK-guided policy optimization (FGPO) trains the subsequent answer or explanation. Factual-QA experiments, cross-dataset evaluation, and three sequential updates show that explicit FoK alignment improves judgment quality on updated knowledge. MMLU and ARC-Easy separately assess capability preservation.
Introduction
Large language models (LLMs) serve as assistants and tool-using agents [18, 52]. Their reliability depends on answer accuracy and on judging whether to answer or seek external support. Cognitive science calls this metacognitive judgment feeling of knowing (FoK) [13, 25]. We use functional FoK for LLM answerability judgments, without implying subjective experience. This scalar judgment supports selective prediction, model routing, and retrieval triggering [21]. If it fails to reflect updated knowledge, a system may confidently answer unknown questions or refuse questions the updated model could solve.
Existing LLM FoK methods mainly take three forms. Output-based methods read uncertainty from the model’s own generations, such as the probability of a “true” token or the entropy of sampled answers [9, 21, 27]. Probe-based methods train a lightweight classifier on hidden states [1, 3]. Elicitation-based methods supervise the model to verbalize its confidence directly [41, 50, 54]. However, these methods are usually calibrated and evaluated under static model parameters. This leaves open what happens when the model is later updated by supervised fine-tuning (SFT), low-rank adaptation (LoRA), knowledge editing, or continual learning [17, 37, 38]. After an update, the LLM’s knowledge may change, while a FoK estimator trained or calibrated for the base model may still judge answerability based on the base model’s knowledge boundary. The problem of keeping FoK aligned under changing model knowledge remains underexplored.
Koriat’s account motivates this study: human FoK is inferred from accessible cues rather than directly read from memory [25]. Our correctness-supervised objective does not instantiate or validate that cognitive process. In LLMs, prior causal tracing locates factual associations mainly in middle layers [38], whereas our FoK recovery peaks occur later (Section 2.2). This is evidence about the tested prompts and interventions, rather than proof of a separate human-like mechanism. Two challenges follow: 1) Knowledge–FoK desynchronization: knowledge training changes answerability without keeping the FoK judgment aligned; and 2) Context-dependent FoK: the judgment varies with the available context and generated reasoning. We therefore train a FoK channel to judge the current knowledge state and to guide the subsequent response.
We address these challenges with two trainable channels. On-policy supervised fine-tuning (OP-SFT) updates the knowledge channel from the model’s own answers, supervising only corrected answer-relevant suffixes to limit disruption to FoK. Knowledge-delta FoK training (KΔ-FT) trains the FoK channel across knowledge deltas, allowing judgment labels to reflect changing answerability. For context-dependent FoK, FoK-guided policy optimization (FGPO) trains the model to generate context before judging answerability and to produce the corresponding answer or explanation. The paper makes three contributions:
We study functional FoK after knowledge updates, identifying later-layer FoK recovery, sensitivity to accessible context, and a failure of knowledge updating alone to synchronize answerability judgments.
We introduce a two-channel framework with OP-SFT, KΔ-FT, and FoK-guided policy optimization to update knowledge, synchronize FoK, and train the model to produce an answer or explanation according to its FoK judgment.
We evaluate FoK on factual QA, including cross-dataset transfer and three sequential updates, with seed variation for the main method. MMLU and ARC-Easy separately measure capability preservation.
Mechanism of LLM FoK
Problem formulation
Let be an LLM at parameter state , and let be a question. We use for whether the model’s greedy answer is correct, and for the model’s functional FoK score, where 0 means the model judges that it cannot answer and 1 means it judges that it can answer. The quality of FoK is reflected in the alignment between and under the same decoding rule. For a generative yes/no judgment prompt, we instantiate this score as .
When an LLM learns new knowledge through post-training, its parameters change from the base state to an updated state . We call the induced difference the knowledge delta. The target is to keep aligned with as knowledge changes. Our method explicitly trains FoK over the current update trajectory; it does not assume that one calibration remains valid for all later updates.
In evaluation, we focus on two directional failures of the FoK judgment. An illusion occurs when the model judges that it can answer but is wrong; an omission occurs when it judges that it cannot answer although its answer would be correct. Metrics and splits are detailed in § 4.1.
FoK localization in model depth
To localize FoK ability, we first adopt ROME’s causal-tracing method [38]. For each factual question, we corrupt the subject token, restore hidden states one layer at a time, and measure whether the yes/no FoK judgment is recovered. We repeat the procedure across Qwen2.5 model sizes, using normalized depth to compare corresponding layers across architectures rather than raw layer indices.
As shown in Figure 1a, Qwen2.5-7B places the strongest FoK signal in late layers, around two-thirds of model depth. Figure 1b shows that the same late-layer pattern remains after normalizing depth across Qwen2.5 scales from 0.5B to 32B. This is later than the factual association signal reported by ROME, which is localized in middle MLP layers, around one-third of model depth [38]. Appendix § 6 gives a parameter-side check: knowledge SFT changes shallow-to-middle layers more, while judgment SFT changes concentrate in the late layers where FoK signal strength is high. Appendix § 10 extends the causal tracing across scales and outcome groups.

Figure 1. FoK signal localizes late under causal restoration. Panel (a) shows a 7B example across tokens and layers; panel (b) aggregates FoK recovery across Qwen2.5 sizes over normalized depth.
Takeaway 1: Under causal tracing, the FoK signal peaks later in model depth than factual associations, suggesting that FoK acts as a judgment over the model’s knowledge state rather than as factual recall itself.
How FoK depends on updated knowledge
Depth separation does not mean independence. To analyze the relationship between factual knowledge and FoK, we intervene on one side and evaluate the other under matched data and optimization settings. First, we update factual knowledge with vanilla factual SFT and measure FoK on the updated model. Second, we train FoK alone and measure whether factual-answer accuracy changes. Table 1 summarizes cross-scale results. Appendix § 13 reports supplementary scale and family results from a separate setup that uses exact match (EM), so its absolute values differ from this table.
| Model | Base | Base | K-SFT | K-SFT | J-SFT | J-SFT | SFT | SFT |
| Acc. | AUROC | Acc. | AUC | Acc. | AUC | Acc. | AUC | |
| Qwen2.5-0.5B | 20% | 0.519 | +2 | -0.016 | -14 | +0.213 | +2 | +0.211 |
| Qwen2.5-1.5B | 35% | 0.529 | +6 | -0.035 | +2 | +0.255 | +5 | +0.248 |
| Qwen2.5-3B | 39% | 0.739 | +12 | -0.220 | +3 | +0.038 | +11 | +0.051 |
| Qwen2.5-7B | 53% | 0.724 | +7 | -0.042 | +2 | +0.088 | +7 | +0.100 |
| Qwen2.5-14B | 66% | 0.762 | +6 | +0.004 | +1 | +0.030 | +5 | +0.068 |
| Qwen2.5-32B | 68% | 0.819 | +4 | -0.026 | 0 | +0.010 | +4 | +0.020 |
| LLaMA-3.1-8B | 74% | 0.672 | +2 | -0.038 | +1 | +0.045 | +2 | +0.063 |
Table 1. Knowledge and FoK under separate or sequential SFT updates. Base gives absolute values; the three SFT blocks give changes from Base. Accuracy changes are percentage points (pp); AUROC changes are unitless. AUC denotes the change in AUROC. Higher accuracy and AUROC are better. Full illusion and omission rates are reported in Appendix § 15.§ 1.
Table 1 shows that knowledge SFT (K-SFT) increases answer accuracy for every model, but FoK AUROC decreases for six of the seven models, with the largest drop on Qwen2.5-3B. Thus, the knowledge update changes the model’s answerability, but the judgment boundary does not adjust correctly. The mismatch is not a single error direction: after K-SFT, some models make more omissions, judging that they cannot answer questions they would answer correctly, whereas larger Qwen models better preserve AUROC but shift toward illusion, judging that they can answer even when their answers are wrong. The issue is therefore not simply increased conservatism or confidence, but a weakened alignment between the current knowledge state and FoK. Judgment SFT (J-SFT) provides a reference intervention: directly training judgment improves FoK without compromising answer performance in larger models. SFT further shows that judgment training after a knowledge update can recalibrate FoK while preserving most of the answer-accuracy gain from K-SFT, indicating that FoK depends on the current knowledge state and should be updated after knowledge changes. Appendix E provides the FoK-recovery and reverse-direction supplementary results.
Takeaway 2: In current LLMs, knowledge updates change answerability but do not automatically update FoK. FoK depends on features of the current knowledge state, so changing knowledge can break the mapping from knowledge to self-judgment unless FoK is explicitly updated.
Reasoning before judgment increases FoK certainty
Koriat’s FoK theory suggests that the amount and intensity of accessible information increase FoK judgments [25]. We test the LLM analogue by asking whether internally generated reasoning affects the model’s FoK.
We evaluate Qwen2.5-7B on MATH [15] problems. For each problem, the model first generates a solution trace. We then re-query the same model at increasing prefixes of that trace and compute for the question “Can you solve this problem correctly?” Figure 2 shows that the FoK signal becomes more separated over the generated trace. As more reasoning tokens are available, the mean FoK scores for final “Yes” and “No” judgments move toward and , respectively; the AUROC of for the final Yes/No FoK token rises as more reasoning tokens are available. This indicates that FoK is sensitive to the model’s current context, not only to the original question.
Figure 2. FoK score and AUROC over pre-judgment reasoning tokens. Green/red curves show mean FoK score for examples whose final FoK token is Yes/No. denotes FoK uncertainty, while or denotes high certainty. The black curve reports AUROC of prefix-level for predicting whether the final FoK token is Yes or No.
Takeaway 3: FoK should be judged from the model’s current context, not from the original question alone. This motivates a response-generation procedure in which the model first accumulates task-relevant context, then evaluates answerability before producing the final answer.
Methodology
Our methodology aligns FoK with updated knowledge through a two-channel model structure and a three-stage procedure that updates knowledge, synchronizes FoK, and trains FoK-guided responses.
Framework overview
Starting from a base LLM , we represent post-training changes as parameter deltas. The knowledge delta forms the knowledge-retrieval channel, which changes what the model knows and retrieves. The FoK delta forms the FoK channel, which judges whether the current knowledge state supports a correct answer. The training procedure has three parts:
On-policy supervised fine-tuning (OP-SFT). OP-SFT updates on the model’s own sampled responses and supervises only the corrected answer-relevant suffix. This part learns the target facts while saving a trajectory of knowledge states that later serves as supervision for FoK adaptation under the same base model.
Knowledge-delta FoK training (K-FT). K-FT trains over the OP-SFT checkpoint trajectory . This lets FoK judgments follow the model’s knowledge as it evolves, rather than remaining tied to a fixed knowledge boundary.
FoK-guided policy optimization (FGPO). Finally, FGPO trains the model to generate a reasoning trace, make a yes/no FoK judgment, and produce either an answer or an explanation of why it cannot answer. Its reward evaluates the judgment and response together, so FoK is optimized as part of the generated decision rather than as a separate post-processing score.
With this two-channel update, FoK guides the LLM before answering. The model can first judge whether its current knowledge is sufficient, then either generate an answer or explain what knowledge is missing, which can support routing to a stronger model or retrieval-augmented generation.
On-policy supervised fine-tuning (OP-SFT)
Section 2.3 shows that standard knowledge training does not synchronize FoK with the updated knowledge state, and can severely interfere with the model’s original FoK ability. To reduce this interference, we introduce on-policy supervised fine-tuning (OP-SFT). Specifically, OP-SFT keeps the model’s own generated context and applies gradients only to the corrected answer-relevant span.
For each question with gold answer set , we collect complete sampled responses from the current LLM. For a sampled response , denotes the generated answer span. We then handle two cases. If the sampled response already answers correctly, we keep this answer span. If it answers incorrectly, we replace only this answer span with a gold answer and leave the rest of the generated response unchanged. The repaired training response is
The training sequence is therefore not a static QA pair: it is sampled from the current LLM, changes only the wrong answer span, and uses loss only on . For example, for “Who wrote Pride and Prejudice?” and , OP-SFT repairs one sampled response as
The red span is the first answer-relevant error in the sampled response; the green span is the supervised correction. The remaining generated text stays in the response, but receives no loss. The OP-SFT objective is therefore the local corrected-span loss:
We save checkpoints during OP-SFT, keeping a sequence of knowledge states.
Knowledge-delta FoK training (KDelta-FT)
After a knowledge update, the LLM’s answerability changes, but its FoK does not automatically synchronize with that change. Training FoK only on the base or final LLM gives a state-specific answerability boundary that generalizes poorly across knowledge deltas. Final-state training also creates label imbalance because many updated questions become answerable.
To make FoK generalize across knowledge deltas, we introduce knowledge-delta FoK training (KDelta-FT), which uses the OP-SFT checkpoint trajectory as supervision states rather than supervising only one endpoint. For each checkpoint , we combine it with the same FoK delta and train to predict whether can answer question with the following loss:
Here denotes the indicator function: it equals 1 when the answer generated by belongs to , and equals 0 otherwise. wraps in the fixed Yes/No prompt “Can you answer this question correctly?” is the base model with the frozen knowledge delta and trainable FoK delta. For , is the normalized Yes/No probability: for and for . The probability is evaluated under , so FoK is trained with the same combined knowledge-and-FoK state used for judgment. The overall objective averages this checkpoint-specific loss over all questions and all OP-SFT checkpoints:
Only is optimized in this stage; the knowledge checkpoints are fixed supervision states. Because the same question is evaluated under multiple , K-FT trains FoK to follow answerability as it changes from pre-update to post-update knowledge, rather than to memorize a static question-level label detached from the current model state during updating. This is correctness supervision across changing states, not an explicit model of human cue-based inference.
FoK-guided policy optimization (FGPO)
After OP-SFT and K-FT, a continuous FoK score still requires a decision threshold. FoK-guided policy optimization (FGPO) trains the judgment and subsequent response together, allowing the model to commit to answering or explaining within one generation. It generates a short context trace, makes a FoK judgment, and produces an answer or explanation:
<think>...reasoning...</think> <answer>yes|no</answer> <response>answer or explanation</response>
For a sampled trajectory , where is the reasoning trace, is the FoK judgment, and is the response, we decompose the reward into FoK, answer, and format terms:
Here is the answerability target for the current training example. Let denote answer correctness, denote whether an LLM judge accepts as a valid explanation for not answering, and denote valid format and proper sequence termination. The FoK term directly supervises the judgment token:
where and . The answer term scores the response under the chosen judgment:
Thus, a “yes” judgment must be followed by a correct answer, while a “no” judgment must be justified by a valid explanation and is penalized on answerable examples. The format term is
Table 6 in Appendix § 9 lists the implemented values. In implementation, wrong answers and malformed outputs receive larger penalties than valid “no” explanations receive positive reward, so the model is not rewarded for avoiding answerable questions by choosing the negative judgment.
Experiments
We ask whether FoK can be synchronized after an update (RQ1), which training stages contribute (RQ2), and whether the result persists across datasets, sequential updates, and seeds (RQ3). Sections 4.2, 4.3, and 4.5 address RQ1–RQ2; § 4.4 and Appendix § 14 address RQ3.
Evaluation setup
Datasets. We use TriviaQA [20] as the main QA dataset, with 5,000 update questions and 1,000 held-out questions. Additional FoK evaluations use SelfAware [55], Natural Questions [30], and WebQuestions [2]. MMLU [14] and ARC-Easy [6] separately evaluate general capability; they are not additional FoK testbeds.
Baselines. We compare direct Yes/No FoK judgment, Vanilla K-SFT, verbalized confidence and 10-bin verbalized confidence [50], sequence probability and length-normalized sequence probability, lexical and semantic self-consistency, semantic entropy [9, 27], and EliCal [41].
Metrics. We report accuracy on TriviaQA, MMLU, and ARC-Easy. FoK quality is measured by AUROC [10] using the normalized Yes probability, plus two errors thresholded at (predict Yes): illusion rate (illu., ) and omission rate (om., ). We combine them as:
Implementation details. We evaluate on Qwen2.5-7B-Instruct [48] for the main experiments and on Qwen3-8B [49] as a thinking-mode backbone. We refer to them as Qwen2.5-7B and Qwen3-8B. We instantiate the three stages of § 3 as LoRA adapters on top of the base model. OP-SFT updates the knowledge channel from corrected sampled responses, KΔ-FT trains the FoK channel over the OP-SFT checkpoint trajectory, and FGPO optimizes response generation using both channels. The full data construction, baseline definitions, evaluation protocol, optimizer settings, checkpoint schedule, and inference configuration are listed in Appendix § 9.
Overall performance
We compare our two-channel update with vanilla K-SFT and a suite of existing FoK estimators after a single TriviaQA knowledge update on Qwen2.5-7B. Table 2 gives the main numbers; Figure 3 decomposes the thresholded errors into illusion and omission. After vanilla K-SFT, in-domain accuracy rises sharply but held-out accuracy falls below the base model, so the update does not simply add knowledge. All existing estimators fall well below their base-model level, and most collapse into one-sided behavior: vanilla K-SFT and confidence-based estimators are dominated by illusion, judging almost every question answerable, while consistency-based estimators trade some illusion for a correspondingly large omission, leaving both error sources substantial. A likely cause is that the knowledge update shifts the output distributions and sampling behavior that these estimators rely on, so a decision boundary calibrated before the update no longer carries the same meaning. This is what Takeaway 2 predicts: under a conventional knowledge update, existing FoK estimators are not a robust way to keep answerability judgments aligned with the updated model.
| Method | Train Acc.↑ | Train AUROC↑ | Train Bias→ 0 | Test Acc.↑ | Test AUROC↑ | Test Bias→ 0 | General MMLU↑ | General ARC↑ |
| Base | 0.680 | 0.702 | 0.381 | 0.672 | 0.688 | 0.402 | 0.687 | 0.811 |
| + Prob | 0.680 | 0.723 | 0.409 | 0.672 | 0.730 | 0.407 | 0.687 | 0.811 |
| + N-Prob | 0.680 | 0.723 | 0.409 | 0.672 | 0.730 | 0.407 | 0.687 | 0.811 |
| + EliCal | 0.637 | 0.808 | 0.263 | 0.622 | 0.810 | 0.254 | 0.709 | 0.504 |
| Vanilla K-SFT | 0.874 | 0.458 | 0.500 | 0.546 | 0.519 | 0.500 | 0.713 | 0.840 |
| + Verbal-0 | 0.874 | 0.491 | 0.543 | 0.546 | 0.478 | 0.551 | 0.713 | 0.840 |
| + Verbal-10 | 0.874 | 0.567 | 0.456 | 0.546 | 0.529 | 0.467 | 0.713 | 0.840 |
| + Prob | 0.874 | 0.590 | 0.495 | 0.546 | 0.541 | 0.498 | 0.713 | 0.840 |
| + N-Prob | 0.874 | 0.590 | 0.495 | 0.546 | 0.542 | 0.498 | 0.713 | 0.840 |
| + Consis-Lex | 0.874 | 0.539 | 0.482 | 0.546 | 0.531 | 0.486 | 0.713 | 0.840 |
| + Consis-Sem | 0.874 | 0.553 | 0.472 | 0.546 | 0.567 | 0.467 | 0.713 | 0.840 |
| + SemEnt | 0.874 | 0.538 | 0.481 | 0.546 | 0.526 | 0.504 | 0.713 | 0.840 |
| + EliCal | 0.648 | 0.646 | 0.428 | 0.587 | 0.644 | 0.447 | 0.723 | 0.688 |
| Ours | 0.961 | 0.817 | 0.245 | 0.692 | 0.809 | 0.308 | 0.711 | 0.830 |
Table 2. Overall performance after one TriviaQA knowledge update on Qwen2.5-7B. Train columns use questions seen during the update; Test columns use held-out TriviaQA questions; General reports MMLU and ARC-Easy. ↑: higher is better; ↓: lower is better; → 0: closer to zero is better. Bold and underline indicate the best and second-best results.
Figure 3. FoK error decomposition for Table 2. Red circles show illusion, the conditional rate of false “can answer” judgments; blue squares show omission, the conditional rate of false “cannot answer” judgments. The horizontal gap shows which error source determines the signed Bias value.
Our two-channel update shows a different profile. Held-out accuracy now rises above the base model, so the knowledge update generalizes instead of overfitting the update shard. FoK AUROC on the updated model reaches the level of the strongest existing estimator applied to the base model, and Bias is the lowest on the update set and the second lowest on held-out questions (Figure 3), although illusion still exceeds omission on held-out questions. We attribute this to two design choices. OP-SFT supervises only the corrected answer-relevant span on the model’s own sampled responses, which only mildly disturbs FoK: held-out AUROC is 0.603 after OP-SFT alone, compared with 0.519 after vanilla K-SFT (Table 2 and Table 4); KΔ-FT then trains the FoK channel directly on the changed answerability, instead of reusing a judgment calibrated before the update. Appendix § 12 confirms this along the OP-SFT trajectory.
Table 4: Held-out stage-wise ablation on Qwen2.5-7B. OP-SFT updates knowledge, K-FT adapts FoK, and FGPO optimizes judgment-guided generation. Higher accuracy and AUROC and lower error rates are better; Bias closer to zero is better. Update-set results are in Table 26.
| Stage | Acc.↑ | AUROC↑ | Illu.↓ | Om.↓ | Bias→ 0 |
| Base | 0.672 | 0.688 | 0.683 | 0.122 | 0.402 |
| Stage 1: + OP-SFT | 0.679 | 0.603 | 0.601 | 0.283 | 0.442 |
| Stage 2: + K-FT | 0.699 | 0.770 | 0.478 | 0.187 | 0.333 |
| Stage 3: + FGPO | 0.692 | 0.809 | 0.432 | 0.184 | 0.308 |
Table 4. Held-out stage-wise ablation on Qwen2.5-7B.
Coverage-accuracy trade-off under FoK judgment
Table 2 shows that FoK quality is recoverable; it remains to test whether the recovered FoK actually drives the decision of when to answer. We therefore vary the FoK threshold on Qwen3-8B and plot the coverage-accuracy curves in Figure 4. For Base and Vanilla K-SFT the curves are nearly flat on both splits: lowering coverage does not raise accuracy, so their FoK scores do not rank answerability. OP-SFT + K-FT restores a monotonic trade-off, confirming that the learned FoK channel is consistent with answerability after the update. FGPO flattens the curves again, but for a different reason: the scores collapse to 0 or 1, and the threshold becomes irrelevant once the policy has committed to answer or abstain. K-FT thus provides a tunable coverage–accuracy trade-off, while FGPO answers more selectively, achieving higher accuracy on the answered subset at the cost of lower coverage.
Figure 4. Coverage–accuracy trade-off on Qwen3-8B (thinking mode). Each panel varies the FoK threshold and plots coverage (solid, the fraction of questions the model judges itself able to answer), conditional accuracy on that answered subset (dashed), and overall accuracy with unanswered examples counted as incorrect.
Controlled comparison and sequential updates
Accuracy-preserving update. Table 3a tests whether avoiding destructive K-SFT alone suffices. After OP-SFT, direct judgment, Prob, and EliCal reach AUROC 0.603, 0.571, and 0.618, respectively; K-FT reaches 0.770. Thus, FoK adaptation is useful even when held-out answer accuracy is preserved. Without an update, however, strong estimators remain competitive: Base + EliCal reaches 0.810 AUROC in Table 2, compared with 0.809 for the full method, whose held-out accuracy exceeds Base by 0.020 in that run.
Table 3: Additional evaluation on Qwen2.5-7B. Left: all methods start from the same OP-SFT knowledge checkpoint; K-FT adds the FoK adapter, so its combined answering state differs. Right: is frozen after round 1; is retrained over each round’s knowledge trajectory. Each round adds a disjoint shard from 5,000 update questions; all test columns use the same 1,000 held-out questions.
| Estimator | Test Acc. | AUROC |
| Direct judgment | 0.679 | 0.603 |
| Prob | 0.679 | 0.571 |
| EliCal | 0.679 | 0.618 |
| K-FT | 0.699 | 0.770 |
| Round | Test Acc. | AUC | AUC |
| 1 | 0.659 | 0.798 | 0.798 |
| 2 | 0.644 | 0.776 | 0.800 |
| 3 | 0.661 | 0.754 | 0.797 |
Table 3. Additional evaluation on Qwen2.5-7B.
Sequential updates and broader evaluation. Across three updates, the per-round FoK channel stays near 0.80 AUROC while the frozen channel falls to 0.754 (Table 3b); held-out accuracy remains below the 0.672 Base value, so this does not demonstrate cumulative answer-accuracy improvement. Cross-dataset tests on SelfAware, Natural Questions, and WebQuestions, plus a separate Natural Questions replication, extend the factual-QA evaluation. Independent Stage-1+2 runs yield test AUROC 0.777 ± 0.009 (four seeds); FGPO yields 0.811 ± 0.003 (three seeds from one shared initialization). These are means and sample standard deviations. Appendix § 14 reports all runs, cross-dataset results, and a MEMIT desynchronization test; the latter tests the phenomenon rather than adaptation of our full framework to that editor.
Ablation study
To locate where the FoK improvement in Table 2 comes from, we ablate the three stages cumulatively: OP-SFT alone, then OP-SFT + K-FT, then the full pipeline with FGPO (Table 4). OP-SFT alone raises update-set accuracy from 0.680 to 0.966 (Table 26) but leaves the FoK judgment misaligned with the updated knowledge state. Adding K-FT is the main source of the FoK gain: AUROC rises and Bias drops while held-out accuracy changes little (0.679 to 0.699), supporting the design choice of training a dedicated FoK channel over knowledge-delta states rather than relying on knowledge learning alone. FGPO raises held-out AUROC from 0.770 to 0.809 in this run and incorporates the judgment into generation. On the separate Qwen3-8B evaluation, its nearly binary judgments explain the flattened threshold curves in Figure 4.
| Stage | Acc. | AUROC | Illu. | Om. | Bias |
| Base | 0.680 | 0.702 | 0.653 | 0.109 | 0.381 |
| Stage 1: + OP-SFT | 0.966 | 0.575 | 0.589 | 0.319 | 0.454 |
| Stage 2: + K-FT | 0.967 | 0.803 | 0.350 | 0.190 | 0.270 |
| Stage 3: + FGPO | 0.961 | 0.817 | 0.250 | 0.240 | 0.245 |
Table 26. Update-set results for Base and the three cumulative training stages.
Related work
Functional metacognition. Metacognitive sensitivity and calibration are distinct quantities [11]. Human–LLM comparisons therefore require explicit behavioral definitions rather than equating similar reports with identical cognitive processes [46]. Recent work studies competing confidence biases under external advice [29] and how verbal confidence is computed from answer-related representations [28]. Our focus is correctness-supervised answerability judgment under changing model parameters. Koriat’s cue-accessibility account motivates the context analyses, but our objective does not directly model that account.
LLM uncertainty and hallucination. LLMs often generate fluent but factually false outputs, a failure studied as hallucination and factuality [16, 19]. Theoretical analyses show that this failure cannot be removed by pretraining alone: rare facts impose a Good–Turing-style lower bound on calibrated error, and common training and evaluation procedures can reward guessing rather than explicit uncertainty [22, 23]. This motivates uncertainty estimation and answerability judgment in LLMs [12]. Existing methods estimate uncertainty from generated outputs, including and (probability that the model “knows” the answer), verbalized confidence, explicit “I don’t know” (IDK) training, semantic entropy, and self-consistency [8, 9, 21, 26, 27, 36, 50, 51, 54]. Other methods read uncertainty from internal representations, including SAPLMA, CCS, INSIDE, MIND, , and internal-state probes [1, 3, 4, 42, 45, 47]. A related line trains answerability behavior directly: R-Tuning teaches models to state that they do not know [56]; US-Tuning, CRaFT, and GRAIT analyze conflicts between answer learning and negative answerability supervision [32, 58, 59]; EliCal uses annotation-efficient elicitation and calibration for honesty [41]. Inference-time factuality methods such as ITI and DoLa intervene without additional training [5, 34], while TruthfulQA, HaluEval, FActScore, and SimpleQA evaluate factual support in generated answers [33, 35, 40, 53]. These works primarily ask whether a given model can express or reveal uncertainty about its current answers. We instead ask whether the FoK judgment tracks post-training updates to the model’s knowledge.
Knowledge editing and continual learning. Post-training updates, knowledge editing, and continual learning change what an LLM knows after pretraining. ROME and MEMIT edit factual associations and evaluate edit success, locality, portability, and retention [38, 39]. Continual-learning methods, including regularization and replay-based approaches, aim to preserve answer accuracy across sequential updates [17, 24, 37]. These methods focus on the update operator and on answer correctness, and usually do not test whether answerability judgment changes with the updated knowledge. We instead treat FoK as a training target co-learned with each update, so the answerability judgment moves with the model rather than lagging behind on a fixed checkpoint.
Conclusion
We studied how to keep functional FoK aligned with an LLM’s answerability after knowledge updates. In the tested factual-QA settings, knowledge updating alone does not preserve this alignment. Our two-channel framework trains knowledge and FoK separately and then optimizes the judgment together with the subsequent response. Controlled OP-SFT comparisons, cross-dataset evaluation, and three sequential updates support the usefulness of explicit FoK adaptation, with seed variation quantifying its stability in the main setting. These results concern behavioral answerability judgments; they do not establish human-like subjective experience, superiority over strong estimators on a non-updated model, or generalization to arbitrary update operators.
Limitations and future work
We evaluate the framework on short-answer factual QA with gradient-based updates, including three sequential update rounds. The MEMIT experiment in Appendix § 14 tests desynchronization under one non-gradient editor, not whether our full framework repairs that editor. Longer update sequences, procedural tasks such as code generation and mathematical proof, and backbones whose baseline FoK signal is near chance remain outside the tested scope. Strong unadapted estimators remain competitive without an update. Our main evidence concerns FoK synchronization after knowledge changes, not universal superiority of the trained channel.
We measure functional FoK as a behavioral answerability judgment, without claiming subjective experience. -FT supervises answer correctness across knowledge states rather than explicitly modeling human cue-based inference. The cue analyses do not include a matched pretrained-model control, so they cannot establish whether training induces cue sensitivity or preserves sensitivity already in the backbone. Causal recovery locations depend on the prompt, token positions, and intervention; they do not establish a uniquely localized or human-equivalent metacognitive mechanism. Seed variation covers Stages 1–2 and FGPO on the main TriviaQA setting, not every preliminary or cross-dataset experiment.
Broader impact
This work targets reliability after LLM knowledge updates by making answerability judgments track the updated model state. The intended positive impact is fewer unsupported answers after fine-tuning or continual updates. A possible negative impact is that better answerability judgments could increase trust in systems whose deployment still depends on data quality, monitoring, and domain-specific safety constraints. The method is not a content-safety system on its own and should be combined with deployment-specific safeguards before release.
Additional experimental settings
Data construction. The main experiments use TriviaQA as the knowledge-update corpus and the in-domain QA benchmark. The update set contains 5,000 questions, and the held-out TriviaQA test split contains 1,000 questions. For OP-SFT, we sample multiple responses from the current answer model for each update question, repair the answer-relevant span when the sampled response is wrong, and train only the corrected answer span. The reported main setting uses 32 sampled responses per question. General capability is evaluated separately on MMLU and ARC-Easy, using the same generator whose TriviaQA result is reported in the main table.
Evaluation protocol. Answer accuracy on TriviaQA is the fraction of generated responses that match the gold answer, with surface-form equivalence checked by Qwen2.5-32B-Instruct [48] used as an LLM judge [57]. MMLU and ARC-Easy use their standard multiple-choice accuracy. Generation and judging are run with vLLM in bfloat16 precision; the 32B judge uses tensor parallelism across the available GPUs. FoK metrics use the model’s pre-answer Yes/No judgment score: AUROC is computed from the normalized probability of the Yes token, and illusion, omission, and Bias are computed from the thresholded Yes/No prediction. Since illusion and omission are conditional error rates in , the signed Bias score lies in . Positive values mean illusion errors dominate, and negative values mean omission errors dominate.
Baseline protocol. All FoK estimators are evaluated under the answer model used by that row. Base rows use the unadapted model; Vanilla K-SFT rows use the knowledge-updated model; Ours uses the combined knowledge and FoK adapters. Verbal-0 and Verbal-10 ask the model for a direct confidence judgment, Prob and N-Prob use sequence likelihood, Consis-Lex and Consis-Sem use multiple sampled responses, semantic entropy clusters sampled responses by meaning, and EliCal uses the released EliCal adapter whenever its public checkpoint is available for the backbone. Unlike estimators that only score fixed answers, loading the EliCal adapter can change answering behavior and answer accuracy. The fixed-answer EliCal transfer protocol in Appendix § 14 is reported separately.
Training protocol. All trainable components are implemented as LoRA adapters on top of Qwen2.5-7B-Instruct. Vanilla K-SFT is trained with standard supervised fine-tuning on the same update questions. OP-SFT trains the knowledge adapter from repaired sampled responses. K-FT freezes the OP-SFT knowledge checkpoints and trains a FoK adapter against labels collected from those knowledge states. FGPO then uses the knowledge and FoK adapters jointly for response optimization. Table 5 lists the full training hyperparameters.
| Parameter | Value |
| Backbone | Qwen2.5-7B-Instruct |
| Training framework | LLaMA Factory |
| Adapter type | LoRA, rank 16, alpha 32, dropout 0 |
| Parallelism | Multi-GPU DDP with DeepSpeed ZeRO-2 |
| Precision | bfloat16 |
| Vanilla K-SFT LR / epochs | / 5 |
| OP-SFT sampled responses / epochs | 32 / 1 |
| OP-SFT LR | |
| K-FT LR / epochs | / 3 |
| FGPO LR / epochs | / 1 |
| Per-device batch size | 1 |
| Gradient accumulation | 1 |
| Schedule | Cosine with 0.03 warmup ratio |
| FoK score | normalized Yes/No token probability |
Table 5. Training hyperparameters for the main experiments.
Compute resources. The main experiments were run on a single multi-GPU server. Training used multi-GPU DDP with DeepSpeed ZeRO-2. Generation and judging used vLLM in bfloat16 precision, and the Qwen2.5-32B judge used tensor parallelism across the available GPUs. We report the parallelism and training configuration needed for reproduction, but did not log exact wall-clock time for every preliminary and failed run.
FGPO reward values. Table 6 lists the implemented FGPO rewards. The base reward gives for each combination of answerability target, judgment, and response outcome. An output from which no yes/no judgment can be parsed receives in place of the base reward. A further is added when the output follows the full <think></think><response> format, when generation ends with the EOS token and otherwise, and the length term uses with measured in characters of the reasoning trace.
| Response outcome | Base reward | ||
| 1 | yes | correct answer | +2.0 |
| 1 | yes | wrong answer | −1.5 |
| 1 | no | — | −1.0 |
| 0 | no | valid explanation | +0.5 |
| 0 | no | invalid explanation | −0.5 |
| 0 | yes | correct answer | −0.5 |
| 0 | yes | wrong answer | −2.0 |
Table 6. Base FGPO reward by answerability target , judgment , and response outcome.
Existing assets and terms of use. Table 7 lists the external datasets, models, and code packages used in the experiments. We use each asset only for research evaluation or fine-tuning under the terms specified by the original publisher.
| Asset | Use in this paper | License or terms |
| TriviaQA | Knowledge-update corpus and in-domain QA benchmark | Apache-2.0 (source) |
| MMLU | General capability benchmark | MIT License (source) |
| ARC-Easy | General capability benchmark | CC BY-SA 4.0 (source) |
| Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct | Main answer model and LLM judge | Apache-2.0 (Qwen 7B and 32B model cards) |
| Qwen3-8B | Thinking-mode backbone for the selective-prediction evaluation | Apache-2.0 (Qwen model card) |
| LLaMA-3.1 and LLaMA-2 checkpoints | Supplementary FoK synchronization results in Appendix G | Meta Llama 3.1 Community License and Llama 2 Community License (3.1, 2) |
| LLaMA Factory | LoRA fine-tuning framework | Apache-2.0 (source) |
| vLLM | Batched generation and LLM judging | Apache-2.0 (source) |
| EleutherAI lm-evaluation-harness | MMLU and ARC-Easy evaluation | MIT License (source) |
| EliCal official adapter | FoK baseline | Publisher model-card terms (source) |
Table 7. External assets used in the experiments and their publisher-specified licenses or terms of use.
Causal tracing and weight-delta supplementary figures

Figure 5. Per-layer indirect effect on true-positive (TP) samples for the instruction-first (IF) prompt across six Qwen2.5 sizes. The solid orange curve shows the question-region effect, the dashed purple curve shows the suffix-region effect, and the red dashed line marks two-thirds of model depth. The question-region effect concentrates in early and middle layers, whereas the suffix-region effect rises mainly after this boundary at every tested scale.
Figure 6. Layer-wise weight delta after one epoch of K-SFT (red) vs. J-SFT (blue) on Qwen2.5-7B (full SFT from base, same data budget). K-SFT changes concentrate in shallow/mid layers, while J-SFT changes remain small until the deep layers. This weight-side pattern is consistent with the depth separation observed in causal tracing.
Figure 7. ROME-style causal tracing on Qwen2.5-14B, broken down by outcome group. Rows: TP (answers correctly and judges yes), FP (answers incorrectly but judges yes), TN (answers incorrectly and judges no), and FN (answers correctly but judges no). Each row shows three samples, except FN, which has only one sample with . Each cell is the layer × token restoration heatmap; tokens are grouped into Sys (system), K-Q (knowledge question), and Hdr (response header). The red dashed line at layer 29 marks the depth boundary used in our analysis. The suffix-region (Hdr) signal appears mainly in deep layers across all four outcome groups, suggesting that the late FoK signal does not simply track answer correctness.
FoK dependency on knowledge
Figure 8a. FoK recovery after knowledge SFT. In this evaluation, sequential K→J training improves FoK but remains below direct J-SFT across all six Qwen2.5 scales.
Figure 8b. Reverse direction. Knowledge exact-match before and after J-SFT across Qwen2.5 scales. J-SFT has a limited effect on answer performance and can even yield modest improvements.
fig:8: FoK dependence on updated knowledge. Left: In this evaluation, K→J training improves FoK but does not reach direct J-SFT performance. Right: J-SFT alone barely shifts answer accuracy.
FoK judgment along OP-SFT training
Figure 9 tests whether the trained FoK channel only works at the final OP-SFT checkpoint. Answer accuracy changes substantially along the OP-SFT trajectory, while FoK AUROC remains above 0.80 on both splits for every checkpoint. The thresholded errors show a calibration shift: omission is larger near the base model, whereas illusion grows on later held-out checkpoints. Thus KΔ-FT provides a stable ranking signal across the knowledge trajectory, but threshold calibration still depends on the underlying knowledge state of the model.
Figure 9. In a separate small-scale study, we evaluate a fixed, trained KΔ-FT FoK channel across successive OP-SFT knowledge checkpoints. We report answer accuracy, FoK AUROC, and thresholded illusion and omission errors.
FoK synchronization results across model scale and family
Knowledge training and FoK degradation. Consistent with the main-text findings, the K-SFT results in Table 8 show that knowledge training can improve answer accuracy while impairing FoK. An update optimized for answering alone need not preserve the model’s ability to distinguish its correct answers from errors, motivating explicit FoK synchronization rather than assuming that knowledge gains automatically improve self-judgment.
| Model | Base | K-SFT | J-SFT | K→J SFT | ||||||
| EM | AUROC | EM | AUROC | Illu. | Om. | AUROC | EM | AUROC | Illu. | |
| 0.5B | 4.2% | 0.523 | 16.9% | 0.502 | 0% | 99% | 0.801 | 13.1% | 0.763 | 3% |
| 1.5B | 30.9% | 0.527 | 35.5% | 0.500 | 66% | 48% | 0.824 | 34.8% | 0.815 | 28% |
| 3B | 35.4% | 0.724 | 45.7% | 0.519 | 24% | 80% | 0.830 | 45.4% | 0.822 | 40% |
| 7B | 47.2% | 0.735 | 55.8% | 0.592 | 28% | 51% | 0.855 | 55.1% | 0.833 | 82% |
| 14B | 59.8% | 0.756 | 65.2% | 0.767 | 73% | 9% | 0.850 | 66.1% | 0.814 | 85% |
| 32B | 61.2% | 0.812 | 66.8% | 0.792 | 80% | 5% | 0.844 | 66.8% | 0.831 | 91% |
Table 8. Supplementary Qwen2.5 size-scaling experiments using a dedicated setup with 5,000 update questions and 1,000 held-out questions.
LLaMA-2 negative result. The LLaMA-2 results reveal a distinction between acquiring knowledge and judging answerability. Although answer accuracy improves, FoK remains at chance after both knowledge and judgment training (Table 9). This suggests that correctness supervision alone is insufficient to elicit a discriminative FoK signal in this setting: the model can learn answers without learning to distinguish when its own answers are correct.
| Model | Base AUROC | K-SFT AUROC | J-SFT AUROC | KJ SFT AUROC | KJ SFT EM | Note |
| Qwen2.5-7B | 0.735 | 0.592 | 0.855 | 0.833 | 55.1% | Reference |
| LLaMA-3.1-8B (LR = ) | 0.646 | 0.503 | 0.786 | 0.836 | 8.6% | KJ forgetting |
| LLaMA-3.1-8B (LR = ) | 0.646 | 0.526 | 0.782 | 0.749 | 73.5% | LR corrected |
| LLaMA-2-7B | 0.520 | 0.500 | 0.500 | 0.500 | 63.5% | No FoK improvement |
| LLaMA-2-13B | 0.525 | 0.500 | 0.500 | 0.500 | 70.1% | No FoK improvement |
Table 9. Cross-family FoK synchronization results with 5,000 update questions and 1,000 held-out questions.
Additional validation
We conduct additional small-scale experiments, each with its own training and evaluation setup, to examine different aspects of FoK behavior.
Accuracy-preserving updates and sequential knowledge changes
OP-SFT comparison. Table 10 compares FoK estimators starting from the same OP-SFT knowledge checkpoint. Adding a FoK adapter changes the combined model state, and the -FT row therefore has a different answer accuracy. Base rows are a reference for the no-update regime, not evidence that our method uniformly improves on unadapted estimators. For Base, EliCal uses its released adapter, which is trained extensively and therefore changes answering behavior. When combined with OP-SFT, EliCal uses an adapter with only a small amount of training, so answer accuracy is unchanged.
| Answer model | FoK estimator | Test Acc. | Test AUROC |
| Base | Direct judgment | 0.672 | 0.688 |
| Base | Prob | 0.672 | 0.730 |
| Base | EliCal | 0.622 | 0.810 |
| OP-SFT | Direct judgment | 0.679 | 0.603 |
| OP-SFT | Prob | 0.679 | 0.571 |
| OP-SFT | EliCal | 0.679 | 0.618 |
| OP-SFT + FoK adapter | -FT | 0.699 | 0.770 |
Table 10. FoK evaluation before and after OP-SFT on the 1,000-question held-out TriviaQA split. All OP-SFT rows start from the same knowledge checkpoint.
Three sequential updates. We divide 5,000 TriviaQA update questions into three disjoint shards. Round applies OP-SFT with on-policy rollouts to the previous round’s model using shard . The frozen FoK channel is trained after round 1 and kept fixed thereafter. The adaptive channel is trained with -FT on each round’s knowledge trajectory. All held-out columns use the same 1,000 questions. Table 11 reports before/after answer accuracy on the new shard for that round; it is not cumulative accuracy over all previously updated facts and does not measure their retention. The adaptive channel maintains discrimination and reduces illusion in later rounds, but held-out answer accuracy remains below Base.
| State | Acc. before → after | Test Acc. | (frozen) AUROC | (frozen) Illusion | (adaptive) AUROC | (adaptive) Illusion |
| Base | — | 0.672 | — | — | — | — |
| Round 1 | 0.575 → 0.972 | 0.659 | 0.798 | 0.455 | 0.798 | 0.455 |
| Round 2 | 0.676 → 0.938 | 0.644 | 0.776 | 0.688 | 0.800 | 0.202 |
| Round 3 | 0.631 → 0.938 | 0.661 | 0.754 | 0.752 | 0.797 | 0.195 |
Table 11. Three sequential TriviaQA updates. Illusion is the conditional rate of predicting Yes on questions the current answering model gets wrong.
Cross-dataset evaluation and a non-gradient update
Zero-shot transfer. We evaluate the TriviaQA-trained systems on SelfAware (3,369 questions), Natural Questions (3,610), and WebQuestions (2,032), without additional training (Table 12). FoK targets each evaluated model’s own answer correctness. In SelfAware, unanswerable questions count as incorrect answers; this adapts the original testbed to model-specific answerability rather than using its answerability annotation as a fixed target. In this transfer experiment, EliCal scores the fixed answers of the corresponding Base or OP-SFT generator, so each paired row has the same answer accuracy. This differs from the adapter-based answering evaluation of Table 10.
| System | SA Acc. | SA AUROC | NQ Acc. | NQ AUROC | WebQ Acc. | WebQ AUROC |
| Base | 0.282 | 0.768 | 0.376 | 0.665 | 0.527 | 0.597 |
| Base + EliCal | 0.282 | 0.702 | 0.376 | 0.735 | 0.527 | 0.636 |
| Vanilla K-SFT | 0.187 | 0.569 | 0.191 | 0.571 | 0.224 | 0.519 |
| OP-SFT + EliCal | 0.305 | 0.590 | 0.381 | 0.579 | 0.539 | 0.515 |
| Ours | 0.305 | 0.810 | 0.381 | 0.733 | 0.539 | 0.644 |
Table 12. Zero-shot transfer from TriviaQA. SA: SelfAware; NQ: Natural Questions; WebQ: WebQuestions. Acc. measures answer correctness, whereas AUROC measures functional FoK discrimination.
| System | AUROC | TPR | TNR | Answer Acc. |
| Base | 0.768 | 0.388 | 0.898 | 0.282 |
| Vanilla K-SFT | 0.569 | 0.302 | 0.760 | 0.187 |
| Ours | 0.810 | 0.765 | 0.701 | 0.305 |
Table 13. SelfAware detail for the same zero-shot systems. TPR is the rate of predicting Yes on correctly answered questions; TNR is the rate of predicting No on incorrectly answered questions.
Natural Questions replication. Separately from zero-shot transfer, we run the full pipeline on a 5,000-update / 1,000-held-out Natural Questions split (Table 14). The resulting update-shard accuracy is 0.925. This replicates the knowledge-update comparison on a second factual-QA dataset, without implying improvement across all QA settings or update operators.
| System | Held-out Acc. | FoK AUROC |
| Base | 0.319 | 0.629 |
| Vanilla K-SFT | 0.199 | 0.585 |
| Ours | 0.366 | 0.647 |
Table 14. Full-pipeline replication on Natural Questions. This held-out split differs from the zero-shot population in Table 12.
Table 14: Full-pipeline replication on Natural Questions. This held-out split differs from the zero-shot population in Table 12.
MEMIT. We apply one MEMIT batch edit [39] to 200 facts in the zsRE setting [31]. Table 15 measures the edited facts and the same 1,000 held-out TriviaQA questions before and after editing. Edited-fact accuracy rises while FoK AUROC decreases. This is a test of desynchronization under a non-gradient operator; we do not evaluate K-FT or FGPO as a repair for MEMIT.
| Train Acc. | Train AUROC | Test Acc. | Test AUROC | |
| Before edit | 0.035 | 0.594 | 0.633 | 0.726 |
| After edit | 0.640 | 0.553 | 0.627 | 0.709 |
Table 15. Knowledge and functional FoK before and after a MEMIT edit. Train denotes 200 edited zsRE facts; Test denotes 1,000 held-out TriviaQA questions. These states are compared only within this experiment.
Table 15: Knowledge and functional FoK before and after a MEMIT edit. Train denotes 200 edited zsRE facts; Test denotes 1,000 held-out TriviaQA questions. These states are compared only within this experiment.
Training-seed variation
Table 16 reports independent training runs for Vanilla K-SFT (three seeds) and OP-SFT + K-FT (four seeds). The Vanilla K-SFT seed runs train a LoRA adapter (rank 16, alpha 32) on standard SFT data built from the update questions, with learning rate , 5 epochs, per-device batch size 8, and gradient accumulation 1. Table 17 varies the FGPO seed from one shared Stage-1+2 initialization. We report the arithmetic mean and sample standard deviation, , over available runs; the unrun Vanilla seed-42 cells are excluded. These statistics quantify training-seed variation under the fixed evaluation split and are neither confidence intervals nor significance tests. The FGPO statistics are conditional on the shared initialization and do not quantify full-pipeline seed variation. Other supplementary experiments are reported as individual evaluations.
| Seed | Vanilla K-SFT Acc. | Vanilla K-SFT AUROC | OP-SFT + K-FT Acc. | OP-SFT + K-FT AUROC |
| 42 | — | — | 0.699 | 0.770 |
| 43 | 0.522 | 0.635 | 0.665 | 0.770 |
| 44 | 0.522 | 0.627 | 0.670 | 0.790 |
| 0 | 0.520 | 0.619 | 0.665 | 0.778 |
| Mean ± SD | 0.521 ± 0.001 | 0.627 ± 0.008 | 0.675 ± 0.016 | 0.777 ± 0.009 |
Table 16. Training-seed variation of Vanilla K-SFT and OP-SFT + K-FT on held-out TriviaQA.
| State | Test Acc. | Test AUROC |
| Shared Stage-1+2 initialization | 0.699 | 0.770 |
| FGPO, seed 1 | 0.692 | 0.809 |
| FGPO, seed 43 | 0.677 | 0.810 |
| FGPO, seed 44 | 0.672 | 0.815 |
| FGPO mean ± SD | 0.680 ± 0.010 | 0.811 ± 0.003 |
Table 17. FGPO seed variation from one shared Stage-1+2 initialization, evaluated end to end on held-out TriviaQA. Seed 1 is the original full-method run.
Table 16: Training-seed variation of Vanilla K-SFT and OP-SFT + K-FT on held-out TriviaQA.
Table 17: FGPO seed variation from one shared Stage-1+2 initialization, evaluated end to end on held-out TriviaQA. Seed 1 is the original full-method run.
Functional FoK and accessible cues
These analyses measure behavior of the trained Qwen2.5-7B FoK channel. They are consistent with cue-sensitive judgments but do not establish that -FT induces that sensitivity, because a matched pretrained control was not included. They also do not test subjective experience.
Consistency at fixed incorrectness. From a 5,000-question TriviaQA pool, we sample 16 answers per question and retain the 1,149 questions where all 16 answers are wrong. Consistency is the fraction giving the most frequent incorrect answer. FoK rises with this consistency even though sampled correctness is zero (Table 18). The reported Spearman correlation is ().
| Answer consistency | Questions | Mean |
| 0–0.2 | 840 | 0.413 |
| 0.2–0.4 | 178 | 0.486 |
| 0.4–0.6 | 71 | 0.571 |
| 0.6–0.8 | 29 | 0.591 |
| 0.8–1.0 | 31 | 0.808 |
Table 18.
Table 18: Functional FoK among 1,149 questions with 16/16 incorrect sampled answers. Bin labels follow the evaluation grouping.
Accessible reasoning. We progressively reveal the generated reasoning trace on 440 MATH problems using the reasoning-prefix prompts of the preliminary study. Table 19 shows better discrimination as more of the trace becomes available. The mean- gap is the difference between correct and incorrect solution groups. This does not establish a particular human cognitive mechanism.
| Reasoning shown | AUROC | Mean- gap |
| Question only | 0.60 | 0.05 |
| 25% | 0.66 | 0.09 |
| 50% | 0.67 | 0.10 |
| 75% | 0.74 | 0.15 |
| Full reasoning | 0.91 | 0.32 |
Table 19.
Table 19: FoK discrimination with progressively revealed reasoning on 440 MATH problems.
Context intervention. We prepend short contexts to a filtered 1,460-question subset of the same pool, removing contexts containing a gold answer. Table 20 reports FoK and answer accuracy; deltas are relative to the bare question. Related versus unrelated words raise mean by 0.490 (paired Wilcoxon test on per-question , reported ). Incorrect related sentences increase while slightly decreasing accuracy, showing that the judgment is not determined by correctness alone.
| Format | Condition | Acc. | Acc. | ||
| — | None | 0.599 | — | 0.408 | — |
| Words | Unrelated | 0.184 | −0.415 | 0.424 | +0.016 |
| Words | Related | 0.674 | +0.075 | 0.478 | +0.070 |
| Sentences | Unrelated | 0.298 | −0.301 | 0.436 | +0.028 |
| Sentences | Correct related | 0.677 | +0.078 | 0.510 | +0.102 |
| Sentences | Incorrect related | 0.772 | +0.173 | 0.397 | −0.011 |
Table 20.
Table 20: Context interventions on 1,460 questions after filtering gold-answer-containing contexts.
Prompt sensitivity and alternative knowledge updates
Judgment templates. We repeat causal tracing on the same 40 Qwen2.5-7B samples under six judgment templates (Table 21). Recovery peaks stay at layers 20–21 of 28, with sensitive ranges within layers 16–23. This supports robustness within the tested templates, not prompt invariance in general. The templates cover both rephrasings and label changes; the sensitive range denotes the layers with strong recovery.
| Template | Peak layer | Sensitive range |
| Original | 20 | 16–22 |
| A: rephrased | 21 | 17–22 |
| B: third-person elicitation | 21 | 16–21 |
| C: numeric labels 1/2 | 21 | 18–23 |
| D: letter labels A/B | 21 | 18–21 |
| E: labels can/cannot | 20 | 18–20 |
Table 21. Causal-tracing sensitivity to judgment templates on 40 shared samples. Layers are reported out of 28.
Correction strategies. Table 22 compares knowledge-update strategies on Qwen2.5-7B-Instruct using 5,000 TriviaQA update questions and 1,000 held-out questions from a split different from the main tables. General capability is measured on the 1,319-question GSM8K test set [7]. Reptile uses first-order meta-updates [43]; SDFT uses the released self-distillation setup [44]. The lookahead variant penalizes deviation from Base after a one-step SFT update; the proximal variant adds an L2 constraint to Base. Average is the unweighted mean of the three reported accuracies and is given only when all three are available; it mixes seen-data and held-out metrics and is not a generalization-only score. Span replacement has the highest such average, but does not dominate Base on GSM8K.
| Strategy | Update Acc. | Held-out Acc. | GSM8K | Average |
| Base | 0.685 | 0.603 | 0.798 | 0.695 |
| Vanilla SFT | 0.992 | 0.452 | 0.368 | 0.604 |
| Reptile meta-update | 0.558 | 0.584 | 0.640 | 0.594 |
| SDFT | 0.529 | 0.565 | 0.367 | 0.487 |
| Unlikelihood on wrong span | 0.500 | 0.528 | — | — |
| One-step lookahead KL | 0.738 | 0.619 | 0.646 | 0.668 |
| L2 proximal constraint | 0.560 | 0.587 | 0.639 | 0.595 |
| Replace wrong span | 0.832 | 0.626 | 0.653 | 0.704 |
Table 22. Alternative knowledge-update strategies on a separate TriviaQA split. Unlikelihood GSM8K was not reported, so its average is omitted.
Supplementary assets. Table 23 records publisher-specified terms for additional datasets and the editing implementation. Dataset terms are distinguished from repository software licenses, particularly for Natural Questions. The exact zsRE distribution, SDFT implementation revision, and all auxiliary checkpoint licenses are not identified in the available experiment record; this version therefore does not claim a complete asset-license audit.
| Asset | Role | License or terms |
| SelfAware | Zero-shot QA | CC BY-SA 4.0 (source) |
| Natural Questions | Transfer and replication | CC BY-SA 3.0 (data source) |
| WebQuestions | Zero-shot QA | CC BY 4.0 (source) |
| MATH | Reasoning evaluation | MIT (repository) |
| GSM8K | Capability evaluation | MIT (repository) |
| MEMIT | Editing implementation | MIT (repository) |
Table 23. Additional assets and publisher-specified terms.
Detailed results for the main-text tables
Knowledge and judgment SFT
These tables retain all the error-rate columns of Table 1. Accuracy, illusion, and omission are percentages; their changes are percentage points. AUROC and its changes are unitless. Illusion and omission are conditional error rates, rather than fractions of all evaluated questions.
| Model | Acc. (%) | AUROC | Illusion (%) | Omission (%) |
| Qwen2.5-0.5B | 20 | 0.519 | 28 | 85 |
| Qwen2.5-1.5B | 35 | 0.529 | 56 | 38 |
| Qwen2.5-3B | 39 | 0.739 | 8 | 68 |
| Qwen2.5-7B | 53 | 0.724 | 51 | 20 |
| Qwen2.5-14B | 66 | 0.762 | 50 | 19 |
| Qwen2.5-32B | 68 | 0.819 | 62 | 8 |
| LLaMA-3.1-8B | 74 | 0.672 | 81 | 16 |
Table 24. Base values for the preliminary SFT comparison.
Table 25: Changes from Base under K-SFT, J-SFT, and sequential SFT. Positive accuracy and AUROC changes indicate improvement; negative illusion and omission changes indicate improvement. All source values are retained.
| Model | K-SFT Acc. | K-SFT AUC | K-SFT Illu. | K-SFT Om. | J-SFT Acc. | J-SFT AUC | J-SFT Illu. | J-SFT Om. | SFT Acc. | SFT AUC | SFT Illu. | SFT Om. |
| Qwen2.5-0.5B | +2 | -0.016 | -27 | +13 | -14 | +0.213 | -28 | +15 | +2 | +0.211 | -21 | -10 |
| Qwen2.5-1.5B | +6 | -0.035 | +6 | -2 | +2 | +0.255 | -28 | -6 | +5 | +0.248 | -21 | -12 |
| Qwen2.5-3B | +12 | -0.220 | +10 | +18 | +3 | +0.038 | +11 | -29 | +11 | +0.051 | +20 | -41 |
| Qwen2.5-7B | +7 | -0.042 | -15 | +19 | +2 | +0.088 | -9 | -4 | +7 | +0.100 | +6 | -12 |
| Qwen2.5-14B | +6 | +0.004 | +9 | -4 | +1 | +0.030 | +9 | -9 | +5 | +0.068 | +29 | -17 |
| Qwen2.5-32B | +4 | -0.026 | +11 | 0 | 0 | +0.010 | -10 | +2 | +4 | +0.020 | +3 | -3 |
| LLaMA-3.1-8B | +2 | -0.038 | -11 | +9 | +1 | +0.045 | +7 | -11 | +2 | +0.063 | +14 | -14 |
Table 25. Changes from Base under K-SFT, J-SFT, and sequential SFT.
Update-set stage-wise ablation
These are the update-set columns of the same runs used in Table 4; they do not measure held-out generalization.
References
- [1]Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. arXiv preprint arXiv:2304.13734, 2023.arxiv.org/abs/2304.13734
- [2]Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, 2013. URL https://aclanthology.org/D13-1160/.DOI
- [3]Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2212.03827
- [4]Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs’ internal states retain the power of hallucination detection. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Zj12nzQ1Qbz.arxiv.org/abs/2402.03744
- [5]Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. DoLa: Decoding by contrasting layers improves factuality in large language models. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Th6NyL07na.
- [6]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Ovyind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge, 2018.
- [7]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
- [8]Roi Cohen, Konstantin Dobler, Eden Biran, and Gerard de Melo. I don’t know: Explicit modeling of uncertainty with an [IDK] token. In Advances in Neural Information Processing Systems, 2024. doi: 10.52202/079017-0349.
- [9]Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630:625–630, 2024. doi: 10.1038/s41586-024-07421-0.
- [10]Tom Fawcett. An introduction to ROC analysis. Pattern Recognition Letters, 27(8):861–874, 2006. doi: 10.1016/j.patrec.2005.10.010.DOI
- [11]Stephen M. Fleming and Hawan C. Lau. How to measure metacognition. Frontiers in Human Neuroscience, 8:443, 2014. doi: 10.3389/fnhum.2014.00443.DOI
- [12]Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6577–6595. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.naacl-long.366. URL https://aclanthology.org/2024.naacl-long.366/.
- [13]Joseph T. Hart. Memory and the feeling-of-knowing experience. Journal of Educational Psychology, 56(4):208–216, 1965.pubmed.ncbi.nlm.nih.gov/5825050
- [14]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
- [15]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
- [16]Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2024. doi: 10.1145/3703155.
- [17]Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. In International Conference on Learning Representations (ICLR), 2022.
- [18]Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C. Park. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7036–7050. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.naacl-long.389. URL https://aclanthology.org/2024.naacl-long.389/.DOI
- [19]Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezhen Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023. doi: 10.1145/3571730.DOI
- [20]Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147/.
- [21]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. doi: 10.48550/arXiv.2207.05221.DOI
- [22]Adam Tauman Kalai and Santosh S. Vempala. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, 2024. doi: 10.1145/3618260.3649777.DOI
- [23]Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. Why language models hallucinate. arXiv preprint arXiv:2509.04664, 2025. doi: 10.48550/arXiv.2509.04664.
- [24]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabsk-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13): 3521–3526, 2017. doi: 10.1073/pnas.1611835114.DOI
- [25]Asher Koriat. How do we know that we know? The accessibility model of the feeling of knowing. Psychological Review, 100(4):609–639, 1993.pubmed.ncbi.nlm.nih.gov/8255951
- [26]Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shres hth A. Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in LLMs. arXiv preprint arXiv:2406.15927, 2024. doi: 10.48550/arXiv.2406.15927.DOI
- [27]Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=VD_AYtP0dve.arxiv.org/abs/2302.09664
- [28]Dharshan Kumaran, Arthur Conmy, Federico Barbero, Simon Osindero, Viorica Patraucean, and Petar Veličković. How do LLMs compute verbal confidence, 2026. URL https://arxiv.org/abs/2603.17839.
- [29]Dharshan Kumaran, Stephen M. Fleming, Larisa Markeeva, Joe Heyward, Andrea Banino, Mrinal Mathur, Razvan Pascanu, Simon Osindero, Benedetto De Martino, Petar Veličković, and Viorica Patraucean. Competing biases underlie overconfidence and underconfidence in LLMs. Nature Machine Intelligence, 8:614–627, 2026. doi: 10.1038/s42256-026-01217-9.DOI
- [30]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026/.DOI
- [31]Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, 2017.arxiv.org/abs/1706.04115
- [32]Jiaqi Li, Yixuan Tang, and Yi Yang. Know the unknown: An uncertainty-sensitive method for LLM instruction tuning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 2972–2989. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-acl.153. URL https://aclanthology.org/2025.findings-acl.153/.DOI
- [33]Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.397. URL https://aclanthology.org/2023.emnlp-main.397/.DOI
- [34]Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2306.03341
- [35]Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.acl-long.229. URL https://aclanthology.org/2022.acl-long.229/.DOI
- [36]Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022.arxiv.org/abs/2205.14334
- [37]Yun Luo, Zhicheng Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023.arxiv.org/abs/2308.08747
- [38]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2202.05262
- [39]Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.07229
- [40]Sewon Min, Kalpesh Krishna, X inxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannah Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.741. URL https://aclanthology.org/2023.emnlp-main.741/.DOI
- [41]Shiyu Ni, Keping Bi, Jiafeng Guo, Minghao Tang, Jingtong Wu, Zengxin Han, and Xueqi Cheng. Annotation-efficient universal honesty alignment. arXiv preprint arXiv:2510.17509, 2025. doi: 10.48550/arXiv.2510.17509.DOI
- [42]Shiyu Ni, Keping Bi, Jiafeng Guo, Lulu Yu, Baolong Bi, and Xueqi Cheng. Towards fully exploiting LLM internal states to enhance knowledge boundary perception. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24315–24329. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.1184. URL https://aclanthology.org/2025.acl-long.1184/.DOI
- [43]Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018. URL https://arxiv.org/abs/1803.02999.
- [44]Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning, 2026. URL https://arxiv.org/abs/2601.19897.
- [45]Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3607–3625. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.220. URL https://aclanthology.org/2023.emnlp-main.220/.DOI
- [46]Mark Steyvers and Megan A. K. Peters. Metacognition and uncertainty communication in humans and large language models. Current Directions in Psychological Science, 35(3): 131–139, 2026. doi: 10.1177/09637214251391158.DOI
- [47]Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. Unsupervised real-time hallucination detection based on the internal states of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14379–14391. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.findings-acl.854. URL https://aclanthology.org/2024.findings-acl.854/.DOI
- [48]Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. doi: 10.48550/arXiv.2412.15115.DOI
- [49]Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. doi: 10.48550/arXiv.2505.09388.
- [50]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.330. URL https://aclanthology.org/2023.emnlp-main.330/.DOI
- [51]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2203.11171
- [52]Yile Wang, Peng Li, Maosong Sun, and Yang Liu. Self-knowledge guided retrieval augmentation for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10303–10315. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.691. URL https://aclanthology.org/2023.findings-emnlp.691/.DOI
- [53]Jason Wei, Karina Nguyen, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models (SimpleQA). arXiv preprint arXiv:2411.04368, 2024. doi: 10.48550/arXiv.2411.04368.
- [54]Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=gjEqKFxFpZ.arxiv.org/abs/2306.13063
- [55]Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don’t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653–8665, 2023. doi: 10.18653/v1/2023.findings-acl.551. URL https://aclanthology.org/2023.findings-acl.551/.DOI
- [56]Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-Tuning: Instructing large language models to say ‘I don’t know’. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7113–7139. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.naacl-long.394. URL https://aclanthology.org/2024.naacl-long.394.
- [57]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhu, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuangzhuang Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2306.05685
- [58]Runchuan Zhu, Zinco Jiang, Jiang Wu, Zhipeng Ma, Jiahe Song, Fengshuo Bai, Dahua Lin, Lijun Wu, and Conghui He. GRAIT: Gradient-driven refusal-aware instruction tuning for effective hallucination mitigation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4006–4021. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-naacl.223. URL https://aclanthology.org/2025.findings-naacl.223/.DOI
- [59]Runchuan Zhu, Zhipeng Ma, Jiang Wu, Junyuan Gao, Jiaqi Wang, Dahua Lin, and Conghui He. Utilize the flow before stepping into the same river twice: Certainty represented knowledge flow for refusal-aware instruction tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26157–26165, 2025. doi: 10.1609/aaai.v39i24.34812.







