betaFeedback

WorksComputer Science

GT-FENG: General Theory of Factual Evaluation and Normative Grounding in Language Models *Author(s)

Abstract

Whether a language model's answer is correct often depends on context: the same value can be right under one jurisdiction, date, or legal definition and wrong under another. We introduce GT-FENG (General Theory of Factual Evaluation and Normative Grounding), the first general framework to unify contextual/temporal correctness and rule-based error attribution across law, policy, and other rule-governed domains. GT-FENG checks answers against the applicable rules and identifies documented rules that match incorrect answers. It shows that when multiple rules give the same value, the answer alone cannot tell which rule the model used. In Medicaid income-limit answers from two models, documenting the asked state's own records reduces the number of wrong answers that match only other states' values by 68-78%, and more than half of stable wrong answers then match two or more documented sources. GT-FENG provides a foundation for evaluating language models in rule-governed domains and for future benchmarks and evaluation toolkits. We release GT-FENG as an open-source Python package, gtfeng.

Introduction

People ask language models about rules: the income limit for a public benefit, the deadline for a tax filing, or the duties of an employer. The correct answer depends on the jurisdiction, date, and definition specified in the question.

Standard factuality evaluations check answers against reference answers or sources [8] (Wei et al. 2024), but do not show which documented rules match a wrong answer. When asked for Oklahoma’s parent income standard in July 2025, Claude Sonnet 5.5 answered 41%, a wrong value that equals Iowa’s current standard and two earlier Oklahoma standards. A count of cross-state errors would attribute this answer to Iowa, and a count of outdated answers would attribute it to Oklahoma’s past.

We introduce GT-FENG (General Theory of Factual Evaluation and Normative Grounding), the first general framework to unify contextual/temporal correctness and rule-based error attribution across law, policy, and other rule-governed domains. GT-FENG defines contextual correctness (§ 3), finds the documented rules that match a wrong answer (§ 4), and shows the limits of this attribution (§ 5). We apply it to Medicaid eligibility answers from two models

Related Work

Many factual questions have no answer that is true everywhere and at all times ([4]). Zhang and Choi (2021) found that roughly 16.5% of information-seeking questions in NQ-Open have answers that depend on when or where the question is asked, and Jiang et al. [7] found that a considerable share of samples in widely used factuality benchmarks are now outdated. In law and policy, correctness often depends on which rules apply at a given place and time (Zheng et al., 2025). Dahl et al. [3] documented legal hallucinations across jurisdictions, courts, and time periods, and Yang et al. (2026) built CrossLex to test whether models recognize that the same facts call for different rules in China, California, and Germany. Several benchmarks focus on time. Prior et al. [11] studied stale and recency-biased answers to German statutory questions, Cymbler et al. [2] showed that legal retrieval systems cite the version currently in force when an earlier or later version applies, Sobhani et al. [13] tested version resolution on Bangladeshi customs instruments, and Nicolaysen [9] classified answers about numeric values in Norwegian law as correct, outdated, wrong, or uninterpretable against each provision’s dated history. These studies treat jurisdictional and temporal validity as part of correctness.

General factuality frameworks check answers against evidence [12]. Iqbal et al. [6] built OpenFactCheck, a unified framework that checks the claims in a response and compares fact-checking systems. Xu et al. (2025) added error attribution to evaluation and assigned each failed answer to one of 15 secondary error categories with a fine-tuned judge model. Alvarez et al. [1] identified cross-source conflation in tool-using agents, where a claim is supported by the evidence but attributed to the wrong source. These frameworks ask whether an answer is supported or why it failed. GT-FENG asks a narrower, rule-based attribution question: which documented rules set the value that a wrong answer gives.

Work on normative documents treats version and scope as part of the answer. The LegalRuleML standard represents legal rules together with their jurisdiction, authority, and temporal validity [10]. Wang et al. (2026) found that a governed system resolving version and scope through explicit rules before generation outperformed a hosted retrieval service on questions over normative documents, and Ulizko et al. (2026) introduced a benchmark for normative reasoning when the supplied evidence is inconsistent or defeasible. Feng [5] linked wrong Medicaid answers to other states’ values and to the asked state’s own earlier values and conventions, and found that the number of cross-state matches depends on how completely the asked state’s own records are documented. Although prior work addresses contextual correctness, rule applicability, and error attribution, it does not provide a unified account of how wrong answers correspond to documented rules, how such correspondences become ambiguous, or how attribution changes with the reference set. GT-FENG brings these elements together by checking answers against the applicable rule, identifying all matching rules within a specified reference set, and formalizing the resulting attribution ambiguity and reference-set dependence.

GT-FENG: General Theory

Contextual Correctness

We call a fact whose value depends on the jurisdiction, the date, and the legal definition in use a normative fact: its value is set by a rule, and the rule applies only within its context.

Definition 1 (Contextual correctness). An answer A to a question about quantity Q in jurisdiction J at time T under definition D is contextually correct when A equals the value of the rule that applies at (Q, J, T, D). Correctness is therefore a relation between an answer and a context. The same A can be correct in one context and wrong in another.

Indexed Normative Facts

GT-FENG indexes each rule by four coordinates: the quantity q, the jurisdiction j, the time t, and the definition d (the reporting convention or legal definition under which the value is stated). The gold value G(q, j, t, d) is the value that the rule sets at that index.

Value types. G can return a number (an income limit), a Boolean (whether a duty applies), a category (which program covers a person), or a set (the conditions that qualify a person). Equality is decided by a comparison rule fixed for each domain before scoring: numbers after normalization to the unit and precision of the source, Booleans and categories exactly, and sets as sets.

Further coordinates. The four coordinates cover the cases studied here. A domain can add coordinates, such as the issuing authority or the population a rule covers. The definitions and propositions below apply in the same way, and a source is then defined by every coordinate except time.

Normative Grounding

Definition 2 (Grounding). A wrong answer A is grounded in the rule at index (q, j, t, d) when that index is documented in R and G(q, j, t, d) = A.

Grounding is a documented correspondence between an answer and a rule. It states which documented rules agree with the answer. It makes no claim about the process inside the model that produced the answer, and Proposition 3 shows that no procedure using only the question, the answer, and R can make such a claim when two sources share a value.

Evaluation and Attribution

A question asks for one index, written (Q, J, T, D), and the model returns an answer A. An item is included in the analysis when every independent repeat returns the same answer.

SymbolNameMeaning
QQ, JJ, TT, DDQuantity, Jurisdiction, Time, Definitioncoordinates the question asks about
qq, jj, tt, dd(same)any coordinate value
GGGoldG(q,j,t,d)G(q, j, t, d), the value set by the rule
AAAnswerthe model’s answer
RRReferencedocumented indices
MMMatchesindices in RR whose gold value equals AA
SSShiftcoordinates in which a match differs
FFFamilya set of shifts
UUUnmatchedwrong answer with no match
OOObservedobserved count
NNNullcount expected under the local perturbation baseline
EEExcessE=O−NE = O - N
IIIncrementsoffsets used to build nearby values
PPPerturbationsnumber of random perturbations for the pp-value

Table 1. GT-FENG notation. Each symbol is the initial of its name.

Factual Evaluation

Rule 1 (Correct). If A=G(Q,J,T,D)A = G(Q, J, T, D), the answer is correct.

Rule 2 (Matches). Otherwise, the match set MM contains every index (q,j,t,d)(q, j, t, d) in RR that satisfies both conditions:

  • (q,j,t,d)≠(Q,J,T,D)(q, j, t, d) \ne(Q, J, T, D) (this excludes the correct rule itself)

  • G(q,j,t,d)=AG(q, j, t, d) = A

Rule 3 (Unmatched). If MM is empty, the answer is labeled U.

Normative Grounding

Rule 4 (Shift). The shift SS of a match is the set of coordinates in which it differs from (Q,J,T,D)(Q, J, T, D), written as a string of coordinate letters. A match (Q,j,T,D)(Q, j, T, D) with j≠Jj \ne J has shift J. A match (Q,J,t,D)(Q, J, t, D) with t≠Tt \ne T has shift T. A match (Q,j,t,D)(Q, j, t, D) with j≠Jj \ne J and t≠Tt \ne T has shift JT.

Rule 5 (Label and ambiguity). If every match in MM has the same shift SS, the answer is labeled S. Otherwise the answer is labeled ambiguous, and the label lists every shift present in $M. GT-FENG distinguishes two kinds of ambiguity:

  • Shift ambiguity: MM contains matches with two or more different shifts. Oklahoma’s parent standard for July 2025 is not 41%. At reference level R3R3 (§ 6), 41% is Iowa’s current standard (shift J), Oklahoma’s 2018 standard (shift T), and Oklahoma’s 2022 standard under the other disregard convention (shift TD), so the label is ambiguous {J, T, TD}.

  • Source ambiguity: MM contains two or more distinct sources. Indices with the same quantity, jurisdiction, and definition count as one source, because they describe one rule over several periods. GPT-5.6 Sol answers 305% for Massachusetts’s pregnancy limit (gold 205%). At R3, 305% is the current limit of both Missouri and West Virginia. Both matches have shift J, so the label is J and the answer is source-ambiguous.

Shifts are computed over every matching index, and sources are counted after indices of the same quantity, jurisdiction, and definition are merged. The two kinds of ambiguity are counted separately. A single source can have two shifts: if another state set the answer’s value in both 2018 and 2025, its two indices form one source with shifts J and JT.

An item is grounded in a family of shifts FF (an error category) when MM is nonempty and every shift present in MM belongs to FF. FF can be any set of shifts, such as {J, JD} for cross-jurisdiction errors or {T, TD} for outdated values.

Rule 6 (Reference levels). The evaluator reports labels at nested reference sets R1 ⊆\subseteq R2 ⊆\subseteq R3 ⊆⋯\subseteq\cdots. R1 contains the indices that a naive check would use, and the last level contains the most complete documentation available.

Rule 7 (Local perturbation baseline). For numeric values and a family FF, OO is the number of wrong items grounded in FF. For each wrong item, the evaluator replaces AA by every nearby value A+iA + i, where ii runs over a fixed set of increments I. II excludes increments equal to a known definition shift, and increments that reach the correct value are dropped for that item. NN is the sum over wrong items of the share of nearby values that are grounded in FF. The excess E=O−NE = O - N, measured relative to the specified local perturbation baseline, is the number of grounded items beyond those that nearby values produce. The p-value is the share of PP random perturbations (each taking one increment per item) for which the number of grounded items reaches OO.

Standard Report

At each reference level, a GT-FENG evaluation reports seven quantities: (1) correct items; (2) items labeled U; (3) single-shift labels, by shift; (4) shift-ambiguous items; (5) source-ambiguous items; (6) the mean number of distinct sources in MM among matched items; and (7) OO for each family of interest. Reporting the same quantities makes the results comparable.

Theoretical Properties

Write M(R)M(R) for the match set computed with reference set RR.

Proposition 1 (Monotonicity). Let R1⊆R2R1 \subseteq R2. For every item, M(R1)⊆M(R2)M(R1) \subseteq M(R2). Hence (a) an item labeled U under R2R2 is labeled U under R1R1, and (b) an item that is shift-ambiguous or source-ambiguous under R1R1 stays so under R2R2.

Proof. Rule 2 selects indices from RR by a condition that does not depend on RR, so a larger RR yields a larger or equal MM. For (a), M(R2)M(R2) empty implies M(R1)M(R1) empty. For (b), two matches in M(R1)M(R1) also belong to M(R2)M(R2), with their shifts unchanged. □

Proposition 2 (Upper bound under full coverage). Let FF be a family of shifts and R1⊆R2R1 \subseteq R2. Assume coverage: R1R1 contains every index of R2R2 whose shift belongs to FF. Then every item grounded in FF under R2R2 is grounded in FF under R1R1, and the number of items grounded in FF cannot increase from R1R1 to $R2.

Proof. Take an item grounded in FF under R2R2. Every match in M(R2)M(R2) has a shift in FF, so by coverage every match lies in R1R1, and M(R2)⊆M(R1)M(R2) \subseteq M(R1). Proposition 1 gives M(R1)⊆M(R2)M(R1) \subseteq M(R2). The two sets are equal, so the item is grounded in FF under R1R1. □

The coverage assumption is the only condition. It holds for F={J,JD}F = \{J, JD\} whenever R1R1 lists every other jurisdiction’s value at time TT under every definition, and it holds for joint families such as {JT,JTD}\{JT, JTD\} whenever R1R1 lists every other jurisdiction’s full history under every definition.

Proposition 3 (Attribution non-identifiability). Suppose MM contains matches from two distinct sources, with indices (q1,j1,t1,d1)(q1, j1, t1, d1) and (q2,j2,t2,d2)(q2, j2, t2, d2). Consider two accounts of the answer: the model drew on the first source, or the model drew on the second. Any attribution procedure that uses only the question, the answer, and RR returns the same output under both accounts, so a procedure that names a single source is wrong under at least one of them.

Proof. Both sources set the value A, so both accounts produce the same observable answer. The question and RR are fixed. An attribution procedure is a function of these observations, so it returns the same output under both accounts. If that output names one source, it names the wrong source under the other account. □

Proposition 3 defines the limit of attribution from answers. Under source ambiguity, MM is the most that a procedure using only the question, the answer, and RR can return. A precedence order, such as checking the asked jurisdiction’s own values first, makes a reporting choice among the rules in MM, and GT-FENG records MM in full alongside any such choice.

Instantiation: Medicaid Income Eligibility

We instantiate GT-FENG on the Medicaid eligibility data of Feng [5]. QQ is one of three quantities (pregnancy, infant, parent). JJ is one of 51 jurisdictions (the 50 states and the District of Columbia). TT is July 2025, and earlier times are the MACStats editions 2018–2024. DD is the treatment of the 5-point income disregard (included or excluded). Federal anchors (100%, 133%, 138%) are matched as a separate fixed set. A match to an anchor carries the label federal, a match to another number in the asked state’s data book carries the label Q (Table 2), and both labels lie outside FF.

R1R1 lists every index with shift J or JD, so the coverage assumption of Proposition 2 holds for F={J,JD}F = \{J, JD\} at every later level.

The counts reproduce the four protocol levels of the original study exactly (37, 16, 10, 8 and 47, 32, 25, 15). Each column moves in the direction Propositions 1 and 2 require: U and O never rise, and both kinds of ambiguity never fall. At R3, about half of all wrong answers are shift-ambiguous (33

LevelIndices addedShifts added
R1other states, July 2025, both definitionsJ, JD
R2asked state: other definition; editions 2021, 2023, 2024; federal anchorsD, T, TD, federal
R3asked state: every edition 2018–2024T, TD
R4asked state: every number in its data-book rows 2018–2025, with and without the disregard, other than values of the asked quantityshifts containing Q, recorded as Q
LevelIndices addedShifts added
R1other states, July 2025, both definitionsJ, JD
R2asked state: other definition; editions 2021, 2023, 2024; federal anchorsD, T, TD, federal
R3asked state: every edition 2018–2024T, TD
R4asked state: every number in its data-book rows 2018–2025, with and without the disregard, other than values of the asked quantityshifts containing Q, recorded as Q

Table 5.

ModelLevelUUShift-ambiguousSource-ambiguousSources per matched itemOO
ClaudeR12513303.137
ClaudeR21528303.216
ClaudeR31133353.210
ClaudeR4635363.28
GPTR12121332.647
GPTR21729362.932
GPTR31532383.025
GPTR4940413.015

Table 2. Reference levels for Medicaid. Each level contains the one above it.

Table 3. Quantities (2) and (4)–(7) of the GT-FENG standard report for stable wrong items (Claude Sonnet 5.5: 62 items; GPT-5.6 Sol: 68 items). Wrong items include answers under the other disregard convention (11 and 6), which have shift D. O counts items grounded in F={J,JD}F = \{J, JD\}.

of 62 for Claude, 32 of 68 for GPT), and more than half are source-ambiguous (35 and 38). Under Proposition 3, these items have a documented set of supporting rules and no single identifiable source.

Generality

GT-FENG applies to facts whose value is set by a rule in context. Table 4 shows how the same coordinates and rules apply to four types of question. The rows illustrate the representation and are not empirical results.

Question typeQJTDValue typeExample of a grounded error
Benefit eligibilityincome limit for a groupstateeffective dateincome counting conventionnumberthe limit of another state (J)
Statutory deadlinedays allowed for a filingcountry or statestatute versioncalendar or business daysnumberthe period of the repealed version (T)
Legal dutywhether an employer must provide paid leavestate or cityeffective datedefinition of a covered employerBooleanthe rule for a larger employer class (D)
Program conditionsconditions that qualify a personstateprogram yearstatutory or administrative listsetlast year’s list (T)

Table 4. GT-FENG coordinates across value types (illustrative).

Value types differ in how often grounding is informative. A Boolean question has two possible answers, so whenever RR documents many rules, almost every wrong answer is grounded in many of them and is source-ambiguous. A numeric question with many distinct values allows finer attribution.

Conclusion

We introduced GT-FENG, a general framework for evaluating answers whose correctness depends on the rule that applies. GT-FENG checks each answer against the rule that applies and finds the documented rules that match a wrong answer. We show that when two rules share a value, the answer alone

cannot tell which rule a model used. In Medicaid eligibility answers from two models, documenting the asked state’s own records reduces the number of wrong answers that match only other states’ values from 37 to 8 and from 47 to 15, and at the most complete level more than half of stable wrong answers match two or more documented sources. Evaluations in law, policy, and other rule-governed domains can report these correspondences, their ambiguity, and their dependence on the reference set.

Limitations

The empirical instance covers one policy domain, three quantities, two models, and 62 and 68 stable wrong answers; the other value types in Section 7 are illustrations. GT-FENG matches answers only to documented rules, so an answer that equals an undocumented rule is labeled U. It reports which rules agree with an answer and makes no claim about the process inside the model. Reference levels and families of shifts are chosen by the evaluator, and counts depend on these choices, which is why the standard report includes every level. Numbers are compared exactly after normalization, so near misses are unmatched. Items whose repeats disagree are left out.

References

  1. [1]Ander Alvarez, Santhiya Rajan, Alessandro Genuardi, Oliver Wirjadi, Samuel Mugel, and Román Orús. 2026. ProvenanceGuard: Source-aware factuality verification for MCP-based LLM agents. arXiv preprint arXiv:2606.18037.arxiv.org/abs/2606.18037
  2. [2]Rose Cymbler, Daniel Guez, and Laurent Fabre. 2026. Temporal misgrounding in legal RAG: A versioned-corpus benchmark for French tax law. arXiv preprint arXiv:2608.09393.arxiv.org/abs/2608.09393
  3. [3]Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1):64–93.DOI
  4. [4]Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eischlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257–273.
  5. [5]Jiayu Feng. 2026. Right number, wrong state? measuring cross-jurisdiction substitution in LLM recall of state policy. arXiv preprint arXiv:2610.09458.arxiv.org/abs/2610.09458
  6. [6]Hasan Iqbal, Yuxia Wang, Minghan Wang, Georgi Georgiev, Jiahui Geng, Iryna Gurevych, and Preslav Nakov. 2024. OpenFactCheck: A unified framework for factuality evaluation of LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 219–229, Miami, Florida, USA. Association for Computational Linguistics.
  7. [7]Xunyi Jiang, Dingyi Chang, Julian McAuley, and Xin Xu. 2026. When benchmarks age: Temporal misalignment through large language model factuality evaluation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 500–512, Rabat, Morocco. Association for Computational Linguistics.
  8. [8]Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannah Hajishirzi. 2023. FACtScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics.
  9. [9]Eirik Botten Nicolaysen. 2026. norregel: a preregistered benchmark of dated numeric values in Norwegian law. arXiv preprint arXiv:2610.29207.
  10. [10]OASIS. 2021. LegalRuleML core specification version 1.0. https://docs.oasis-open.org/legalruleml/legalruleml-core-spec/v1.0/os/legalruleml-core-spec-v1.0-os.html. OASIS Standard, 30 August 2021.
  11. [11]Max Prior, Andreas Schultz, and Matthias Grabmair. 2026. Asking for an old friend: Diagnosing and mitigating temporal failure modes in LLM-based statutory question answering. arXiv preprint arXiv:2605.23497.arxiv.org/abs/2605.23497
  12. [12]Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring attribution in natural language generation models. Computational Linguistics, 49(4):777–840.
  13. [13]Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Fahmid Hasan Chowdhury, Md Adnan Arefeen, Farig Sadeque, Md. Faizul Bari, and Swakkhar Shatabda. 2026. Time present and time past: Benchmarking large language models on temporally evolving document understanding. arXiv preprint arXiv:2608.08512.arxiv.org/abs/2608.08512

Paper details

vlnc:W_8xst8j4u
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents