NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains

Introduction

Language models are already asked questions of Thai law. Lawyers, students and the public ask which section applies to a set of facts, what the consequence is, and how a court would decide. A benchmark that measures how well a model does this needs four things. (1) It needs questions from real disputes, with the verdict and the reasoning a court gave. (2) It needs the grading a Thai lawyer would apply, which scores the reasoning and not only the outcome. (3) It needs the full statute book, so that a model can be tested as a retrieval agent and not only from memory. And (4) it needs the version of each section that was in force when the case was decided, because the law changes and a court applies the law of its day.

No Thai legal benchmark has had the four together. Our earlier benchmark, NitiBench [1], asks questions written for the Civil and Commercial Code and the Revenue Code against a 2024 snapshot of the text, and two years later 2.3% of its citations are already outdated (§ 2). Benchmarks for English and Chinese law treat the law as a fixed text as well as [27, 28, 33].

NitiBench+, its sequel, has all of them. The benchmark is temporal in one sense only. Every question carries the date of its case, and the answer is graded against the law in force on that date. It holds 1,670 items and 4,071 sub-questions from decisions of three Thai courts, in 17 domains of law from family and succession to administrative contracts (§ 4). Each item gives the fact pattern and asks for the verdict, the reasoning and the sections applied, and the gold answer is copied from the decision. Answers are graded on the Thai Bar Council rubric, from 0 to 10, which we call the Bar-standard score, and on the overlap between the sections the model cites and the sections the court relied on (§ 5). Every gold citation is resolved to the version in force on the decision date and paired with the current text (Figure 1). The retrieval corpus holds every version of every Thai statute section, 394,943 dated texts of 106,041 sections.

We built the benchmark for two purposes.

The first is to show that the version of the text matters, and to set up the task so that it does not distort the score. Law is not a fixed text. Statutes are amended and repealed. A question taken from a court decision has its answer in the text in force on that day, not in the current text. Across 156,650 decisions of four Thai courts, the cited law has changed since the decision in up to 67% of the cases by court, and of the 1,031 Acts Parliament has passed since 2002, 55% amend or repeal an earlier one (§ 3). Of 53 legal benchmarks from 19 jurisdictions, none grades the version of the law. A benchmark that checks the section number but not the version rewards an answer drawn from the wrong text. We ask what happens when a model reads the current text of a correct section, and whether searching the law in force on the decision date fixes the problem. We ran two experiments on 13 open-weight models over the full benchmark. First, we gave the model the correct citation and swapped the version, the text in force or the current text. Second, we placed the model in a retrieval agent and changed its index from the current text to the text in force on the decision date. Reading the current text of an amended section lowers the Bar-standard score by 0.25 points. For questions on repealed sections, the current text contains no provision for the model to cite, and the score falls by 0.79 points. Searching the law in force recovers part of these points for the models that search, and none for the models that do not (Part I, § 6). The text in force is the fair setting for a question taken from a dated decision, and every result in Part II uses it.

The second is to measure today’s models on that setting. Lawyers, students and the public already put legal questions to language models, and no measurement says how far the answers can be trusted. We run a public leaderboard on a held-out test split of 500 items stratified by domain, 29 or 30 per domain, so that a new model can be added at a bounded cost, and we release a validation split of the same shape. It ranks 28 models in three settings, from memory alone, with the court’s sections provided, and as a retrieval agent over the law in force, and it reports the cost per item, the hallucinated citations and the score by domain. The leaderboard then asks what limits the models, the knowledge of the statutes, the use of a search tool, or the application of the law to the facts, and what Thai and legal post-training adds to a base. The leading model scores 7.78 out of 10 as a retrieval agent and the best open-weight model 6.39 (Part II, § 7).

This report makes three contributions.

1. A temporal benchmark. NitiBench+, 1,670 items from three courts in 17 domains, with every gold citation resolved to the version in force on the decision date and paired with the current text, and the full dated statute corpus to retrieve from.

2. The cost of the version. An audit of 156,650 Thai decisions, and the score 13 open-weight models lose when they read the current text instead of the text in force, by drift band and by the kind of change. The text in force is the fair setting.

3. A leaderboard and an analysis of today’s models. A public ranking of 28 models on a held-out 500–item test split in three settings, with citation precision and recall, hallucinated citations and the cost per item, an analysis of what limits the models, in statute knowledge, tool use and the application of the law, and a comparison with a general intelligence index, which sorts the tiers and not the frontier.

Related Work

Legal QA benchmarks. Legal QA benchmarks exist for English (LegalBench [33], LexGLUE [17], CaseHOLD [89]), Chinese (LawBench [28], LexEval [51]), law-school examinations (LEXam [27], KCL [64]), retrieval (LegalBench-RAG [69]), tool-using agents (LegalAgentBench [53]), and Thai law (NitiBench [1]). Most of them score a multiple-choice answer or a retrieved passage, and none of them grades the version of the law ([83]). NitiBench+ scores a free-form verdict and its reasoning on the rubric Thai bar examiners use, and the cited sections against the court’s own.

Thai legal NLP. NitiBench+ is the sequel of NitiBench [1], our earlier Thai legal QA benchmark, and it is built to measure a language model on Thai law in every aspect a Thai lawyer would ask of it. NitiBench has a CCL split of 3,729 questions on the Civil and Commercial Code and a Tax split of 50 questions from Revenue Department rulings, each written against a 2024 snapshot of the statute text. Two years later, 2.3% of its CCL citations no longer match the current text, and 25% of its cited sections have been amended at some point (§ 3). NitiBench+ widens every part of it. The questions come from decided cases of three courts across 17 domains instead of two codes. The answer is a free-form verdict with its reasoning, graded on the rubric Thai bar examiners use, with the cited sections checked against the court’s own. Every citation is resolved to the version in force on the decision date and paired with the current text, and the retrieval corpus holds every dated version of every section, so a model can be tested from memory, with the court’s sections, and as a retrieval agent. A public leaderboard keeps the measurement current as models are released. On the model side, several Thai post-trainings of open bases have been released, among them Typhoon [68], Pathumma [60], ThaLLE [46], OpenThaiGPT [67], SEA-LION [10] and OpenThai 2.0 Legal [40]. § 7.6 tests what they add to their bases on Thai law.

Versioned legal text. Several jurisdictions publish point-in-time statutes, including the UK, Australia, and Canada, and citation graphs exist across multiple legal systems [44, 49]. No single resource combines dated statute versions, dated court rulings with resolved citations, and an evaluation benchmark. FourCorners [3] unifies all three for Thai law and supplies our corpus of dated statutes and rulings. Recent studies of time in legal QA add dates to German statutory questions [71], track versions of the French tax code [19], and compare Chinese judgments before and after the 2021 Civil Code [38]. Each relies on a task-specific dataset. NitiBench+ covers a national case-law corpus with the complete statute book in every version.

LLM judges for legal QA. LLM evaluators suffer from length and position bias and from variance between models [52]. LeMAJ [25] breaks legal answers into atomic components. NitiBench+ uses the official Thai Bar Council grading rubric, applied by models calibrated against human bar examination graders [2].

How Much Law Drifts

Using FourCorners [3], we examine drift from three angles, how often court citations become outdated, what existing benchmarks record about time, and how often Parliament amends existing statutes rather than passing new ones. To measure court drift, we compare the statute text in force on the decision date with the text current on 1 September 2026. A cited section has drifted if its text was amended or repealed between those two dates (Figure 1). We evaluated every decision with resolved citations across four courts: the Supreme Court (51,566 decisions since 1935), the Constitutional Court (2,025 rulings since 1998), the Administrative Courts (101,315 judgments since 2001), and the Court of Appeal for Specialised Cases (1,744 decisions since 2016).

Most decisions cite law that has since changed. Table 1 categorizes each decision by its most severely drifted citation. Across all four courts, 79,394 of 156,650 decisions (50.7%) cite law that has changed. Specifically, cited law changed in 67.1% of Supreme Court decisions, 62.4% of Constitutional Court rulings, 42.6% of Administrative Court judgments, and 23.3% of specialised appeal court decisions. Each court drifts for different reasons The Supreme Court relies on legal codes amended incrementally over decades. The Constitutional Court cites constitutions replaced in 2007 and 2017 along with their organic acts. The Administrative Courts cite general codes alongside repealed administrative laws (one in six judgments cites a repealed act, such as the 1992 Civil Service Act or the 1987 Expropriation Act). The specialised appeal court cites commercial, labor, and family codes amended over the past decade. This drift is not limited to older cases. As shown in Figure 2, only Supreme Court decisions from the 2020s maintain over 80% alignment with current text. For cases from the 2010s, half had a cited section rewritten within five years.

Court	Decisions	Current	Amended	Repealed	Drifted
Supreme Court	51,566	32.9	56.1	11.0	67.1
Constitutional Court	2,025	37.6	2.7	59.8	62.4
Administrative Courts	101,315	57.4	26.2	16.4	42.6
Court of Appeal for Specialised Cases	1,744	76.7	19.3	3.9	23.3

Existing benchmarks cannot see it. We checked 53 legal benchmarks across 19 jurisdictions for what they record about time (Table 2). Of the 53, 44 cite statutes in their gold answers, 10 give the date of every item, 9 carry a statute version, and none grades a model on the version. Every reported score therefore reflects the current law, even for a benchmark built from past court decisions. Table 3 lists the 53 benchmarks. Each cell was read from the dataset’s schema, its data files or its paper on 5 October 2026. The audit file released with the code holds, for every cell, the field names or the quoted sentence it rests on and a confidence grade. Two rows are graded low confidence because only the paper was available. In total, 44 benchmarks cite statutes in their gold answers (20 fully), 10 give full item dates and 15 partial ones, 9 carry a statute version, 5 of them the concurrent temporal studies, and 10 state a snapshot date.

Released	Bench.	Cites statute	Item date	Version	Snapshot
before 2024	20	5 (+11)	2 (+6)	1	2 (+2)
2024–2025	19	9 (+5)	1 (+5)	3	3 (+2)
2026	14	6 (+8)	7 (+4)	5 (+1)	5 (+1)
All	53	20 (+24)	10 (+15)	9 (+1)	10 (+5)

Benchmark	Year	Jurisdiction	Cites	Date	Ver.	Snap.
before 2024
CAIL2018 [82]	2018	China	✓	–	–	–
JEC-QA [91]	2020	China	–	–	–	–
SARA [37]	2020	US	✓	✓	–	–
CaseHOLD [89]	2021	US	(✓)	–	–	–
COLIEE statute law tasks [74]	2021	Japan	✓	(✓)	–	✓
CUAD [35]	2021	US	–	(✓)	–	–
ECtHR datasets [16]	2021	Council of Europe	(✓)	–	–	–
ILDC [56]	2021	India	–	(✓)	–	–
Zalo AI Challenge 2021 Legal Text [32]	2021	Vietnam	✓	–	–	–
BSARD [55]	2022	Belgium	✓	–	–	✓
LBOX OPEN [39]	2022	Korea	(✓)	–	–	–
LexGLUE [17]	2022	Council of Europe, EU,	(✓)	–	–	–
NCBE MBE practice exam set used in [45]	2022	US	–	–	–	(✓)
DISC-Law-Eval [84]	2023	China	(✓)	(✓)	–	–
LAiW [20]	2023	China	(✓)	–	–	–
LawBench [28]	2023	China	(✓)	–	–	–
LegalBench [33]	2023	US	(✓)	(✓)	–	–
LeXFiles / LegalLAMA [18]	2023	Canada, Council of Eur	(✓)	–	–	–
Open Australian Legal QA [13]	2023	Australia	(✓)	(✓)	✓	(✓)
SCALE / Swiss legal benchmark suit [77]	2023	Switzerland	(✓)	✓	–	–
2024–2025
ArabLegalEval [36]	2024	Saudi Arabia	(✓)	–	–	–
IL-TUR [42]	2024	India	✓	(✓)	–	–
KBL: Korean Benchmark for Legal Ia [47]	2024	Korea	(✓)	(✓)	–	✓
LegalBench-RAG [69]	2024	US	–	–	–	–
LeKUBE [81]	2024	China	✓	–	✓	–
LexEval [51]	2024	China	(✓)	–	✓	–
NyayaAnumana [62]	2024	India	–	(✓)	–	–
PredEx [61]	2024	India	–	–	–	–
STARD [78]	2024	China	✓	–	–	–
ALQAC [4]	2025	Vietnam	✓	–	–	–
Bar Exam QA [90]	2025	US	(✓)	–	–	(✓)
CLaw [83]	2025	China	✓	(✓)	✓	(✓)
Housing Statute QA [90]	2025	US	✓	(✓)	–	✓
JBE-QA [14]	2025	Japan	–	✓	–	–
JuDGE [79]	2025	China	✓	–	–	–
LaborBench [34]	2025	US	–	–	–	✓
LegalAgentBench [53]	2025	China	(✓)	–	–	–
LexRAG [54]	2025	China	✓	–	–	–
NitiBench [1]	2025	Thailand	✓	–	–	–
2026
BenGER [59]	2026	Germany	(✓)	(✓)	–	–
CALRK-Bench [43]	2026	Korea	(✓)	(✓)	–	–
ClaimRAG-LAW [21]	2026	EU, Luxembourg	(✓)	–	–	(✓)
FiscalQA Pro [19]	2026	France	✓	✓	✓	✓
GreekBarRetrieval [11]	2026	Greece	✓	✓	–	✓
Legal RAG Bench [12]	2026	Australia	(✓)	–	–	–
Legal Research Bench [24]	2026	US	(✓)	(✓)	–	–
LegalSearch-R1 benchmark [26]	2026	China	✓	✓	✓	✓
LEXam [27]	2026	Switzerland	(✓)	✓	–	–
LexKairos [50]	2026	China	(✓)	✓	✓	–
Prior et al. (German statutory QA) [71]	2026	Germany	✓	(✓)	(✓)	–
SearchFireSafety [15]	2026	Korea	✓	–	–	✓
TALD (applicable-law determination [38])	2026	China	✓	✓	✓	–
TIDE [76]	2026	Bangladesh	(✓)	✓	✓	✓

The law itself keeps moving. Legislative records confirm this pattern. Since 2002, Parliament published 1,031 Acts in the Royal Gazette. Of these, 567 (55%) amended or repealed existing statutes rather than introducing new ones. This proportion rose from 53% in the 2000s to 65% in the 2020s. Over the same period, the executive issued 3,299 royal decrees and 4,440 ministerial regulations. Because most legislative activity updates existing law, static benchmarks based on snapshot statutes quickly fall out of sync.

NitiBench has already drifted. Without version tracking, benchmarks cannot detect when their gold answers expire (§ 2). We evaluated our own earlier benchmark, NitiBench [1], whose questions include the statutory text used during annotation (Table 4): NitiBench-CCL was built on 2024 statutes. Two years later, 2.3% of its citations are outdated, including Section 1435 of the Civil and Commercial Code (marriage age, amended by the 2025 marriage equality act). Over 25% of its cited sections have been

Benchmark	Questions	Amended ever	Stale
NitiBench-CCL	3,729	25%	2.3%
NitiBench-Tax	50	72%	0.0%

amended previously, and errors will accumulate undetected. To avoid drift, NitiBench-Tax sampled only post-2021 Revenue Department rulings that match the latest Revenue Code. While this preserved temporal accuracy, it shrank the benchmark to just 50 questions. NitiBench+ avoids this trade-off by preserving decision dates and indexing both historical and current texts.

The NitiBench+ Benchmark

NitiBench+ consists of 1,670 items and 4,071 sub-questions spanning 17 Thai legal domains, sourced from three courts (Table 5). The Supreme Court provides private, criminal, and regulatory domains (1,303 items). The Constitutional Court provides public constitutional law (50 items). The Supreme Administrative Court provides administrative domains covering official acts, state contracts, public liability, and personnel (317 items).

Domain	Items	Sub-q.	No drift	Amended	Repealed	Court’s own	Governs	Conseq.	Combine	Lower ct.
Family & Succession	146	326	60	85	1	153	15	61	57	40
Criminal	146	352	13	106	27	157	39	52	43	61
Civil & Commercial	144	351	63	60	21	153	31	63	56	48
Land & Property	117	284	43	42	32	123	22	56	50	33
Labor	116	239	41	63	12	117	12	37	35	38
Tax & Revenue	113	290	18	47	48	128	25	53	52	32
Intellectual Property	109	255	36	48	25	116	23	44	41	31
Public Law	98	263	27	26	45	119	26	47	55	16
Civil Procedure	87	180	13	61	13	91	11	26	17	35
Natural Resources & Licensing	84	173	15	36	33	87	11	31	28	16
Administrative Acts	82	233	36	34	12	99	32	29	63	10
Public Personnel	81	244	32	21	28	98	31	36	55	24
State Liability	80	241	36	21	23	96	20	43	53	29
Narcotics	74	159	3	2	69	78	18	20	13	30
Administrative Contracts	74	227	28	31	15	82	20	49	41	35
Traffic & Transport	63	117	8	48	7	64	8	19	11	15
Securities & Finance	56	137	15	28	13	59	10	25	25	18
Total	1,670	4,071	487	759	424	1,820	354	691	695	511

Domains reflect the core areas of each court’s reasoning. We preserved major fields, split standalone statutory areas (such as narcotics and traffic), and merged smaller categories. Each domain contains 56 to 146 items. Benchmark summary scores are equal-weighted macro-averages across domains. Because judicial decisions do not follow question-answer formats, an extraction pipeline converts each decision into a fact pattern, sub-questions, and statutory provisions (§ 4.1). Gold verdicts and reasoning are preserved as verbatim excerpts from the decisions.

Citations and sub-questions. Each item contains one or more sub-questions with its own verdict, reasoning, and citations. Dependent sub-questions are flagged and answered sequentially. Gold citations are resolved via FourCorners [3] to the version active on the ruling date. Citations mentioned only for background context are kept in the record but ignored during scoring. Figure 1 illustrates an item with both texts for a drifted citation. Appendix § 12.§ 1 contains the full schema and a worked item.

Construction pipeline

The pipeline extracts facts, reasoning, and citations directly from the court’s decision.

Selection. Candidate decisions are stratified by decade and drift status. Claude Opus 5 screens candidates, keeping challenging cases where the court applies at least three substantive statutory provisions. Fact-only disputes and procedural rulings are excluded. Duplicate holdings are grouped so no question appears more than twice, maximizing question diversity.

Extraction. Claude Fable 5 generates structured records from selected decisions. Each record contains the fact pattern, decision date, sub-questions with corresponding verdicts and reasoning, and exact source spans. Citations are classified as either primary or background. In addition to direct judicial rulings, we create four targeted question types: (1) Identifying governing provisions. (2) Determining legal consequences. (3) Interpreting interactions between multiple provisions. (4) Identifying departures from lower court decisions. All gold answers remain verbatim text spans. Table 5 provides full item counts.

Verification. Deterministic checks confirm that text spans match the source records and citations are well-formed. Citations are resolved to the version active on the ruling date; items with unresolvable primary citations are removed. Citations to external materials like ministerial notifications remain unscored, and joined cases are consolidated. Finally, human annotators review every item against the original ruling to verify legal accuracy.

Drift bands

Decisions span from 1949 to 2026. Items are assigned to one of three drift bands based on their most heavily altered citation: No-drift (Control): 487 items (29%) where cited statutes remain unchanged. Amended: 759 items (46%) where statutory text has been modified. Repealed: 424 items (25%) where the cited statute was subsequently repealed, or was already repealed at the time of the ruling (Appendix § 12.§ 2). Because a single changed citation shifts an entire item out of the control band, 71% of items exhibit drift even though 65% of individual citations remain current (Table 5).

Current and In-force corpora

The retrieval corpus indexes 394,943 texts across 106,041 sections in 5,912 laws. Each section includes effective dates and operational status as of 1 September 2026. Across all benchmark items, 9,223 citations reference 3,006 sections across 298 laws. Each citation includes both its in-force and modern text: 5,869 citations are unchanged between versions. 1,934 citations are amended text. 1,263 citations are repealed laws with no modern counterpart.

We evaluate models across two statutory corpora. The Current corpus reflects statutory text on 1 September 2026, matching traditional benchmark setups. The In-force corpus reflects the statutory text active on each item’s decision date. These corpora differ only for amended and repealed sections. Experiments in § 5.2 supply models with cited text from one corpus or grant access to a retrieval index built over it.

Release

The benchmark is released in three parts, defined in § 7.1. The validation split, 498 cases, is public. For each case it gives the facts, the questions, the court’s verdict and reasoning, and the gold citations with the version in force on the decision date and the current version, together with the text of those sections in both versions and the grading instruments, the rubric, the judge prompt and the citation-matching rules. The test split, 500 cases, is held out and is never published, neither its ids nor its text. The leaderboard runs on it, and a model joins the leaderboard when we run it. The remaining 672 cases are kept in reserve. The statute corpus, its index, the retrieval service and the harness are not released. A user of the validation split can run the Parametric and Gold-cited In-force settings and not the retrieval settings. Every released file carries a canary string so that it can be filtered from training corpora.

Experimental Setup

Metrics

We report one score and one diagnostic. The Bar-standard score is the main metric. It grades the whole answer as a Thai bar examiner would, the verdict, the reasoning and the sections applied. Citation F1 is a diagnostic of one part of that answer, the statute references. It takes the sections the answer cites and compares them with the sections the court relied on. The judge reads the answer with its citations, and the citations are therefore already graded inside the Bar-standard score. Citation F1 measures that one part on its own. We do not add the two into one number, because that would count the citations twice. We show Citation F1 next to the score because the two can move apart, and the gap is informative. The current text of a changed law lowers the score and leaves Citation F1 almost unchanged (§ 6.4). A model can cite more sections than the court did and still score in the good tier (§ 7.2). Within a model, the citations account for 13 percent of the differences between answers, and the rest is the reasoning (§ 7.6).

Bar-standard score. The Bar-standard score is the grade of a free-form answer on the official Thai Bar Council grading rubric (Table 6, Appendix B.1). The rubric is the instrument and the score is its result. The rubric combines verdict accuracy with five tiers of reasoning quality. A matching verdict (T4 to T8) scores from 0 to 10 points. An incorrect verdict (T1 to T3) is capped at 4 points. We call the resulting 0 to 10 grade the Bar-standard score. Following Akarajaradwong et al. [2], we use a panel of three LLM judges (Claude 4.6 Opus, Gemini 3.1 Pro Preview, and GPT-5.4 Pro) evaluated against human bar examination standards. Each judge receives the verbatim rubric instructions (Appendix B.2), and we report the median score across the panel.

Citation F1. Gold citations include all statutory sections required for a sub-question’s verdict. Model citations are resolved to the statute corpus, and background citations mentioned in the ruling are excluded from the predicted set before scoring. We compute Citation F1 per sub-question as the harmonic mean of precision and recall (0 if no sections are cited). Precision and recall breakdowns appear in Table E.7.

Aggregation. Scores are computed hierarchically. Sub-questions are averaged within each item, items are averaged within each domain, and the overall benchmark score is the macro-average across all 17 domains.

Settings

We test five experimental configurations across all items:

1. Parametric. The model receives only the question and decision date, testing pretraining recall.

2. Gold-cited Current. The prompt includes the question, decision date, and verbatim text of gold-cited sections on current version.

3. Gold-cited In-force. Identical to Gold-cited Current, but instead uses statutory text active on the decision date, including repealed provisions (serving as the upper bound for perfect retrieval).

4. RAG Current. An agent retrieves sections using a dense index of current statutes via Qwen3-Embedding-8B [88], chosen for Thai following SEA-BED [70]. The tool returns the top 10 matches, and the model controls the tool freely.

5. RAG In-force. Identical to RAG Current, but the evaluation harness filters retrieval candidates by the decision date. The model never selects dates or versions directly, isolating the impact of the retrieved text.

The public leaderboard labels Parametric, Gold-cited In-force and RAG In-force as Closed book, Handout and Open book. This report keeps the names above.

Models

Open-weight models. We evaluate 19 open-weight models. Thirteen are served locally with vLLM [48] on four A100 80GB GPUs (Table C.1 lists checkpoints and serving configuration, Table C.2 the compute), in three groups. General instruction models are Gemma 4 31B and Gemma 4 26B-A4B [29], Nemotron 3.5 Lightning 30B-A3B [63], and Muse Glimmer 30B [58]. The Nemotron 3 post-training family holds Nano 30B-A3B and Super 120B-A12B [63], their SEA-LION v4.8 derivatives [10], and OpenThai 2.0 Legal [40], which is based on Nano 30B-A3B. The Thai specialized models are the four ThaiLLM 8B models, Typhoon-S [68], Pathumma [60], THaLLE [46] and OpenThaiGPT [67]. Each is a post-training of ThaiLLM-8B [80], a Thai continued pretraining of Qwen3-8B-Base [72] that has no chat template and cannot answer the benchmark itself. The other six are Qwen3.8 27B [73], OpenThai 2.0 27B [41], DeepSeek V4.1 Flash [23], DeepSeek V4 Flash [22], GLM 5.3 Flash [30, 85] and Qwen3-8B [72], Alibaba’s own post-training of Qwen3-8B-Base, the base the ThaiLLM 8B models descend from, which stands in for their unservable base in the comparisons of § 7.6.

Proprietary models. Nine models are available only through their providers’ APIs. Claude Haiku 5.5 [6], Claude Sonnet 5.5 [8], Claude Opus 5.5 [7] and Claude Fable 5.1 [5], Gemini 4 Argon [31], GPT-6.1 Sol [65] and GPT-6.1 Terra [66], Muse Spark 1.3 [57], and GLM 5.3 Flash.

Serving and sampling. Every model receives the same prompts and the same retrieval tool. A locally served model runs with the sampling parameters of its model card. A model behind an API runs with its provider’s defaults. No prompt is tuned per model.

Two protocols

Every table caption names the protocol it follows. The benchmark protocol runs the 13 locally served open-weight models on all 1,670 items, in the five settings, with $k = 3$ samples per item, and reports avg@3 (additional results in Appendix E). Part I follows the leaderboard protocol: it runs all 28 models on the held-out 500-item test split (§ 7.1), in Parametric, Gold-cited In-force and RAG In-force. The 13 models of the benchmark protocol keep their three samples there, and Qwen3-8B has three as well. Every other row has one sample per item.

Part I. The Cost of the Current Text

Part I asks whether the version of the text matters. It follows the benchmark protocol, 13 open-weight models, 1,670 items, three samples per item. It first reports the 13 models in the five settings, the base of every comparison that follows. It then swaps the version of the cited sections, the text in force against the current text, and measures what a model loses when it reads the current text of a law that changed after the decision (§3). It asks whether a search tool over the law in force gives the loss back, and whether the loss comes from the rule or from the wording. The answer sets the setting that Part II uses.

Open-weight models on the full benchmark

Table 7 reports the Bar-standard score and Citation F1 for the 13 open-weight models in the five settings. Gold-cited In-force provides the historically accurate text and is the best setting for every model. Citation F1 ranges from 0.47 to 0.88, and Bar-standard scores range from 2.0 to 5.8 out of 10. Without provided text (Parametric), only Gemma 4 and Muse Glimmer can recall some laws from memory, reaching a maximum Citation F1 of 0.30. Even with the correct sections provided, no model exceeds 5.8 in Bar-standard score. The sections supply the correct law. Applying that law to the facts remains the major challenge, and leaves substantial headroom on NitiBench+.

Model	Citation F1: Param.	Citation F1: GC Cur.	Citation F1: GC InF.	Citation F1: RAG Cur.	Citation F1: RAG InF.	Bar-standard (0–10): Param.	Bar-standard (0–10): GC Cur.	Bar-standard (0–10): GC InF.	Bar-standard (0–10): RAG Cur.	Bar-standard (0–10): RAG InF.
General
Gemma 4 31B	0.304	0.645	0.778	<u>0.432</u>	<u>0.512</u>	3.68	<u>4.60</u>	<u>5.11</u>	<u>4.20</u>	<u>4.55</u>
Gemma 4 26B-A4B	<u>0.272</u>	0.667	0.791	0.228	0.262	<u>3.42</u>	4.37	4.79	3.61	3.71
Nemotron 3.5 Lightning 30B-A3B	0.028	0.655	0.770	0.287	0.354	1.76	3.08	3.44	2.26	2.47
Muse Glimmer 30B	0.198	0.742	0.878	0.456	0.551	3.34	5.18	5.75	4.39	4.94
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B-A3B	0.020	0.650	0.762	0.205	0.260	1.22	2.44	2.71	1.52	1.66
↪ SEA-LION v4.8 30B-A3B	0.038	0.644	0.753	0.283	0.347	1.69	3.16	3.49	2.25	2.50
↪ OpenThai 2.0 Legal 30B-A3B	0.073	0.427	0.474	0.004	0.005	1.16	1.84	1.95	0.88	0.88
Nemotron 3 Super 120B-A12B	0.099	<u>0.691</u>	<u>0.814</u>	0.358	0.444	2.08	3.79	4.27	2.92	3.31
↪ SEA-LION v4.8 120B-A12B	0.092	0.677	0.801	0.305	0.362	2.05	3.78	4.26	2.71	2.94
ThaiLLM 8B derivatives
Typhoon-S 8B	0.096	0.604	0.687	0.013	0.036	1.73	2.62	2.79	1.61	1.56
Pathumma 8B Think	0.140	0.592	0.669	0.124	0.161	1.71	2.61	2.70	1.40	1.47
THaLLE 0.2 8B	0.022	0.611	0.713	0.001	0.002	1.21	2.34	2.61	1.11	1.11
OpenThaiGPT 8B	0.020	0.567	0.667	0.000	0.000	0.94	1.79	1.95	0.88	0.88

RAG In-force is the setting closest to a deployed assistant, a model with a search tool over the law in force. Scores there run from 0.88 to 4.94, with a mean of 2.45 over the 13 models. Muse Glimmer 30B leads with 4.94, Gemma 4 31B follows with 4.55, then Gemma 4 26B-A4B with 3.71 and Nemotron 3 Super with 3.31. The four ThaiLLM 8B models and OpenThai 2.0 Legal score below 1.6. Citation F1 follows the same order, 0.551 for Muse Glimmer and 0.512 for Gemma 4 31B, and no 8B model reaches 0.17. The spread across the three samples is 0.00 to 0.09 points on this setting (Table E.5), so the ranking is stable. The best of the three samples per item is in Table E.4 (Appendix E.2).

The cost of the current text

Table 8 compares the Bar-standard scores of historically accurate text (In-force) against modern text (Current). The No-drift control group shifts by only 0.01. In contrast, using the correct historical text improves scores by 0.25 on Amended items and 0.79 on Repealed items. Every model improves on Repealed items, and every model but Pathumma on Amended items. Stronger models see the largest gains, improving by up to 0.45 on Amended items and 1.35 on Repealed items. Because the rubric evaluates verdicts and reasoning together, Table 10 reports court agreement alongside scores for agreeing answers. For drifted items, in-force text yields modest agreement gains (+1.6 pp on amended, +4.5 pp on repealed) but larger reasoning gains among agreeing answers (+0.29 and +0.95 points). The right text occasionally corrects a verdict. Its main benefit is a stronger reasoning behind it.

Model	Gold-cited		RAG
	Amend.	Repeal.	Amend.	Repeal.
General
Gemma 4 31B	+0.45*	+0.98*	+0.29*	+0.87*
Gemma 4 26B-A4B	+0.31*	+0.97*	+0.03	+0.35*
Nemotron 3.5 Lightning 30B-A3B	+0.31*	+0.86*	+0.14*	+0.60*
Muse Glimmer 30B	+0.45*	+1.35*	+0.32*	+1.51*
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B-A3B	+0.22*	+0.58*	+0.10*	+0.38*
↪ SEA-LION v4.8 30B-A3B	+0.29*	+0.88*	+0.18*	+0.63*
↪ OpenThai 2.0 Legal 30B-A3B	+0.12*	+0.27*	+0.03	−0.00
Nemotron 3 Super 120B-A12B	+0.39*	+1.10*	+0.29*	+0.92*
↪ SEA-LION v4.8 120B-A12B	+0.42*	+1.13*	+0.14*	+0.70*
ThaiLLM 8B derivatives
Typhoon-S 8B	+0.09*	+0.48*	−0.05	−0.05
Pathumma 8B Think	−0.01	+0.45*	+0.01	+0.22*
THaLLE 0.2 8B	+0.19*	+0.70*	+0.01	−0.00
OpenThaiGPT 8B	+0.09*	+0.48*	+0.00	−0.01
All models	+0.25*	+0.79*	+0.11*	+0.47*
No-drift control	+0.01		−0.02

Band	Agrees with court (%) Current	Agrees with court (%) In force	Score if agreeing Current	Score if agreeing In force
No drift	60.9	61.2	5.07	5.06
Amended	57.6	59.2	4.83	5.12
Repealed	53.9	58.4	3.88	4.83

Recovery through historical retrieval

The RAG columns of Table 8 test these effects inside a retrieval agent. Searching on a historical index recovers a Bar-standard score of 0.11 on Amended items and 0.47 on Repealed items. Recovery is limited by the search tool. The modern index finds the gold sections 40% of the time, the historical index 48% of the time. Repealed laws do not appear in a modern index. Models that search often, such as Muse Glimmer, recover more. Weaker models that rarely search recover nothing.

Changes of the rule and changes of the wording

The cost comes from changes of the rule, not of the wording. The Amended band holds 759 items whose cited sections changed but still exist (§4.2). We manually labeled each of them by whether the amendment changed the rule the question turns on or only the phrasing. Figure 3 shows one of each. For the 428 rule-changed items, the current text lowers the score by 0.45. For the 331 wording-only items, it costs 0.00, the same as the No-drift control (Table 9). The overall cost of 0.25 on Amended items is the average of the two groups.

Amendment	Median $d$	$\Delta$ Bar-standard	$\Delta$ Citation F1
Rule changed (428 items)	0.34	+0.45 [+0.38, +0.53]	+0.04 [+0.03, +0.06]
Wording only (331 items)	0.10	−0.00 [−0.03, +0.03]	−0.00 [−0.01, +0.00]

Figure 4 sets this label against the size of the text change, measured with character trigrams. If $T(x)$ is the set of three-character sequences of a text $x$, the distance between the text in force $a$ and the current text $b$ is

The distance is 0 when the texts are identical and 1 when they share no trigram. Wording-only edits cost nothing at any size, −0.02 below the median distance ($d < 0.24$) and 0.05 above it. Rule changes cost at every size, 0.21 points below the median and 0.54 above it. The size of an edit does not predict its cost. Whether the amendment changes the rule does. A second measure of the size, the normalized Levenshtein distance between the two texts, agrees. Table 11 correlates both measures, averaged over the amended gold sections of an item, with the per-item change from Gold-cited Current to Gold-cited In-force. Larger amendments cost more, on the Bar-standard score and on Citation F1, and the correlations are small. The size of an amendment is one of several factors behind the per-item cost.

Amendment size	$\Delta$ Bar-standard	$\Delta$ Citation F1
1 − character-trigram Jaccard	+0.35 [+0.28, +0.41]	+0.29 [+0.22, +0.37]
Normalized Levenshtein distance	+0.34 [+0.27, +0.41]	+0.28 [+0.20, +0.35]

The models still cite the right sections under both texts. On rule-changed items Citation F1 drops by 0.04 while the Bar-standard score drops by 0.45.

Part II. The Leaderboard and the Limits of the Open-Weight Models

Part II measures today’s models on the setting Part I established, the text in force. It follows the leaderboard protocol (§ 5.4). Every model runs on the held-out test split of 500 items in three settings, and the public leaderboard page reports the same runs. The page is live and takes new models as we run them. The numbers in this report are the leaderboard as of 3 November 2026. The page names the settings for a general reader, Closed book for Parametric, Open book for RAG In-force and Handout for Gold-cited In-force. This report keeps the names of § 5.2. In short, Parametric is the model alone, RAG In-force is the model with a tool that searches the law in force, and Gold-cited In-force is the model with the court’s sections handed to it. Part II ranks the models (§ 7.2), sets the score against its cost (§ 7.3), reads the score by domain (§ 7.4) and counts invented citations (§ 7.5). It ends with what limits the models (§ 7.6) and sets a general intelligence index against the leaderboard (§ 7.8).

The test split and the validation split

A leaderboard has to take in new models as they are released, and a run over all 1,670 items at three samples is too costly to repeat for each one (§ 5.4). A leaderboard also has to stay honest once its questions are public. A model can be trained on a published test set, and its public score then stops meaning anything. GSM1k rebuilt a public arithmetic benchmark in private and found several model families up to 13 percent weaker on the private twin [87], and an audit of Chatbot Arena found providers testing dozens of private variants and publishing the best [75]. We therefore cut the benchmark into three parts (Figure 5). The test split holds 500 items, 29 or 30 from each of the 17 domains, with the drift bands of each domain in the domain’s own proportion and one item per shared-question cluster. The leaderboard runs on it, and it is never published. The validation split is drawn by the same recipe from the remaining items, 498 in all, with no item that shares a question with a test item, and it is released with the text of its gold sections (§ 4.4). The 672 items left over are the reserve, which replaces the test split if it ever leaks. The two splits are drawn to be alike, and the full-benchmark runs of Part I show that they are.

Table 12 gives the 13 models of the benchmark protocol on the full benchmark, the test split and the validation split, from the same judged rows. A model’s test score and validation score differ by 0.05 points on average and by 0.16 at most, and the two rank the models the same way, with a Spearman correlation of 0.97 to 0.99 across the three settings. The test split also tracks the full benchmark within 0.07 points for every model. Scores on a split are macro-averaged over the 17 domains, as on the full benchmark. The gap between a model’s validation score and its test score is also the leaderboard’s check against training on the released data. For the 13 models it stays within 0.16 points. A new model whose validation score runs far ahead of its test score has seen the validation split.

Model	Parametric Full	Parametric Test	Parametric Val.	Gold-cited In-force Full	Gold-cited In-force Test	Gold-cited In-force Val.	RAG In-force Full	RAG In-force Test	RAG In-force Val.
General
Gemma 4 31B	3.68	3.66	3.74	5.11	5.16	5.07	4.55	4.57	4.52
Gemma 4 26B-A4B	3.42	3.44	3.43	4.79	4.84	4.76	3.71	3.68	3.74
Nemotron 3.5 Lightning 30B-A3B	1.76	1.73	1.82	3.44	3.49	3.46	2.47	2.47	2.47
Muse Glimmer 30B	3.34	3.35	3.40	5.75	5.83	5.72	4.94	4.93	4.95
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B-A3B	1.22	1.15	1.27	2.71	2.71	2.75	1.66	1.62	1.67
↪ SEA-LION v4.8 30B-A3B	1.69	1.66	1.71	3.49	3.53	3.54	2.50	2.45	2.54
↪ OpenThai 2.0 Legal 30B-A3B	1.16	1.18	1.18	1.95	1.92	1.95	0.88	0.87	0.89
Nemotron 3 Super 120B-A12B	2.08	2.05	2.15	4.27	4.29	4.37	3.31	3.23	3.36
↪ SEA-LION v4.8 120B-A12B	2.05	2.04	2.07	4.26	4.29	4.28	2.94	2.88	3.04
ThaiLLM 8B derivatives
Typhoon-S 8B	1.73	1.74	1.77	2.79	2.77	2.82	1.56	1.54	1.56
Pathumma 8B Think	1.71	1.71	1.70	2.70	2.70	2.71	1.47	1.46	1.50
THaLLE 0.28B	1.21	1.18	1.25	2.61	2.62	2.64	1.11	1.10	1.11
OpenThaiGPT 8B	0.94	0.95	0.97	1.95	1.90	2.00	0.88	0.87	0.89
Agreement over the 13 models
Spearman, test against validation	0.967			0.995			0.989
Spearman, test against full	0.973			1.000			0.995
Mean absolute difference, test against validation	0.05			0.05			0.05
Largest absolute difference, test against validation	0.12			0.11			0.16

The leaderboard

Figure 6 ranks the 28 models by their Bar-standard score in RAG In-force, the setting closest to a deployed assistant, where the model searches the law in force itself. Table D.1 in Appendix D gives every number. Claude Fable 5.1 is the best model at 7.78. Gemini 4 Argon follows with 7.54, GPT-6.1 Sol with 7.28 and Claude Opus 5.5 with 7.03. The best open-weight model is DeepSeek V4.1 Flash at 6.39, and its predecessor DeepSeek V4 Flash scores 5.71. Qwen3.8 27B scores 5.48 and its Thai post-training OpenThai 2.0 27B 5.10. Muse Glimmer 30B, the best model of the benchmark protocol, is twelfth with 4.93. Per-domain scores of every row are in Figure 8 and Table E.3.

The Parametric column shows what a model knows with no text in front of it. Claude Fable 5.1 scores 7.40 from memory alone, more than any open-weight model reaches with the search tool, and GPT-6.1 Sol 6.90. The proprietary models score 3.49 to 7.40 in this column, the open-weight models 0.95 to 3.92.

The Gold-cited In-force column is each row’s ceiling, the score it reaches when the court’s sections are handed to it. The five best models reach 7.56 to 8.04 there, and their RAG In-force runs stay within 0.6 points of it. Finding the sections costs them little. Lower in the table the distance grows, to 0.74 for Claude Haiku 5.5, 1.06 for Nemotron 3 Super and 1.41 for SEA-LION v4.8 120B.

The cited sections show two kinds of answer. Every proprietary model has a recall above its precision, 0.69 to 0.88 against 0.39 to 0.61, and so do the two DeepSeek models, 0.80 against 0.37 and 0.74 against 0.48. They cite more sections than the court did and catch most of the court’s sections among them. GPT-6.1 Sol has the best F1, 0.657. Most open-weight models of the benchmark protocol do the reverse. They cite few sections and miss many of the court’s. Gemma 4 31B has a precision of 0.642 and a recall of 0.460.

What the score costs

An item’s cost is what the answer would have cost to buy. For a model behind an API it is the provider’s list price on 26 October 2026 applied to the measured input, cached and output tokens. For an open-weight model served locally it is the public API price of the same weights applied to the measured tokens, or the price of the base model when the post-training has no public price. Figure 7 sets the score of each row against that cost in RAG In-force. Table D.2 in Appendix § 15 gives the cost, the output length, the searches and the hallucinated citations of each row. An item costs from a tenth of a cent for the 8B models to 1.155 dollars for Claude Fable 5.1. GPT-6.1 Sol at 0.134 dollars scores 7.28, at about one ninth of that price. DeepSeek V4.1 Flash scores 6.39 at 0.024 dollars, the price of Claude Haiku 5.5, which scores 4.62. Writing more does not buy score. DeepSeek V4.1 Flash writes 25.6 thousand tokens per item, DeepSeek V4 Flash 45.4 thousand and Qwen3.8 27B 52.3 thousand. GPT-6.1 Sol writes 4.1 thousand and scores above both. Searching more does not buy score either. Among the models that search, the calls per item run from 5.3 for Claude Opus 5.5 to 19.0 for DeepSeek V4 Flash, with no relation to rank.

Performance by domain

Figure 8 gives the RAG In-force score of every row in every domain. Averaged over the 28 models, the easiest domain is Natural Resources & Licensing at 4.57 and the hardest is Public Personnel at 3.83. The

three lowest domains are Narcotics, Tax & Revenue and Public Personnel, and the three highest Natural Resources & Licensing, Public Law and Traffic & Transport. The order of the rows holds from one domain to the next. Claude Fable 5.1 is the best model in 12 of the 17 domains and Gemini 4 Argon in the other five, and the scores of Claude Fable 5.1 run from 7.16 to 8.17. A reader who needs one domain should read that column rather than the overall score. The 13 open-weight models of the benchmark protocol give the same order on all 1,670 items at three samples per item, with Public Personnel again the lowest domain (Appendix E.1). Narcotics is low for a reason of its own. In 69 of its 74 cases the court applied law that has since been repealed, and a model must answer from the text in force and not from what it learned about the current code.

Hallucinated citations

A model can cite a section that does not exist. We check every cited section against the statute corpus. A citation is hallucinated when the cited law is not in the corpus, or when the law exists and has no such section. Citations the corpus cannot decide are left out of the count, as are references without a section number. The corpus cannot decide an amending act or a ministerial regulation, a royal decree it does not hold, or a section of a repealed law whose text it does not hold. A real instrument the corpus lacks never counts as hallucinated. The rate is the share of hallucinated citations among the checked citations of a setting. Figure 9 gives the share of such citations in the Parametric setting, where every citation comes from the model’s memory. The best models almost never do it. Claude Fable 5.1 invents none of the 6,159 sections it cites and Gemini 4 Argon one of 3,692. Claude Opus 5.5 invents 0.2 percent and GPT-6.1 Terra 0.4 percent. GLM 5.3 FlashX Legal, a legal post-training, invents 14.0 percent, and its base GLM 5.3 Flash 18.1 percent. At the other end, Nemotron 3 Nano 30B-A3B invents 45.9 percent of the sections it cites, Nemotron 3.5 Lightning 30B-A3B 42.4 percent, Qwen3−8B 29.0 percent and SEA-LION v4.8 30B-A3B 27.1 percent. Most invented citations are invented laws. Over the 28 models, 16,073 of the 19,428 hallucinated citations name a law that is not in the statute book, most of them from Nemotron 3 Nano 30B-A3B, and 3,355 name a real law with a section it does not have. With the search tool (Table D.2) one thing decides the rate, whether the model searches. The nine proprietary models are at or near zero, and so are the two open-weight DeepSeek models. The ThaiLLM 8B models invent 3.0 to 27.6 percent of the sections they cite, Qwen3−8B 25.5 percent and OpenThai 2.0 Legal 24.7 percent. These are the models that never or rarely call the tool (§ 7.6).

What limits the models

Three things can hold a model back on this benchmark, what it knows of the statutes, how it uses the search tool and how it applies the law to the facts. The three settings separate them. Parametric shows what the model knows from memory, RAG In-force what it does with the search tool, and Gold-cited In-force how it applies the court’s sections once it has them. We take them in turn.

Knowledge of the statutes. The Parametric setting asks how much Thai law a model knows from training alone. The model gets the question and nothing else, and every section it cites comes from memory. The proprietary models of the large providers know a good deal. They score 3.49 to 7.40 and cite the court’s sections with an F1 of 0.28 to 0.58 (Table D.1). GLM 5.3 FlashX Legal is the exception among them, 2.91 and an F1 of 0.11, the level of its open-weight base. The open-weight models know little. They score 0.95 to 3.92, and only five of them recall some Thai law, the two DeepSeek models with an F1 of 0.31, Gemma 4 31B 0.30, Gemma 4 26B-A4B 0.27 and MuseGlimmer 0.20. No other open-weight model reaches 0.15. The ThaiLLM 8B models cite the court’s sections for 2 to 15 percent of sub-questions, and the Thai post-trainings know little more law than their reference models (Table 14). An open-weight model that answers from memory is guessing at the law for most questions.

Model	Param.	GC InF.	RAG InF.	Citation F1	Searches
Nemotron 3 Nano 30B-A3B	1.15	2.71	1.62	0.251	2.9
↪ SEA-LION v4.8 30B-A3B	1.66 (+0.51)	3.53 (+0.82)	2.45 (+0.83)	0.335 (+0.084)	4.7 (+1.8)
↪ OpenThai 2.0 Legal 30B-A3B	1.18 (+0.03)	1.92 (−0.79)	0.87 (−0.75)	0.005 (−0.246)	0.0 (−2.9)
Nemotron 3 Super 120B-A12B	2.05	4.29	3.23	0.424	6.5
↪ SEA-LION v4.8 120B-A12B	2.04 (−0.01)	4.29 (+0.00)	2.88 (−0.35)	0.351 (−0.073)	6.2 (−0.3)
Qwen3.8 27B	3.18	6.04	5.48	0.480	17.6
↪ OpenThai 2.0 27B	2.96 (−0.22)	5.85 (−0.19)	5.10 (−0.38)	0.509 (+0.029)	16.6 (−1.0)
Qwen3−8B	1.17	2.57	0.96	0.000	0.0
↪ Typhoon-S 8B	1.74 (+0.57)	2.77 (+0.20)	1.54 (+0.58)	0.034 (+0.034)	0.4 (+0.4)
↪ Pathumma 8B Think	1.71 (+0.54)	2.70 (+0.13)	1.46 (+0.50)	0.155 (+0.155)	2.8 (+2.8)
↪ THaLLE 0.2 8B	1.18 (+0.01)	2.62 (+0.05)	1.10 (+0.14)	0.003 (+0.003)	0.0 (+0.0)
↪ OpenThaiGPT 8B	0.95 (−0.22)	1.90 (−0.67)	0.87 (−0.09)	0.000 (+0.000)	0.0 (+0.0)
GLM 5.3 Flash	3.08	5.89	4.80	0.414	10.3
↪ GLM 5.3 FlashX Legal	2.91 (−0.17)	6.48 (+0.59)	5.43 (+0.63)	0.459 (+0.045)	16.1 (+5.8)

Use of the search tool. The RAG In-force setting gives the model a tool that searches the law in force. Table 13 shows two numbers for each of the 28 models. The gain is what the tool adds, the RAG In-force score minus the Parametric score. The gap is how far the model still stays below its Gold-cited In-force score, where the court’s sections are given, and it is the most a better search could still add. For most models the tool helps. Twenty-two gain, by 0.24 to 2.60 points. Six lose, by 0.08 to 0.31. The models that search most gain most. DeepSeek V4 Flash makes 19.0 calls per item and gains 1.79 points, DeepSeek V4.1 Flash 17.9 calls and 2.60 points, Qwen3.8 27B 17.6 and 2.31, OpenThai 2.0 27B 16.6 and 2.14, and Muse Glimmer 15.8 and 1.58. Eight of the nine proprietary models gain less, 0.38 to 1.37 points with 5.3 to 9.2 calls per item, because they knew more before they searched. GLM 5.3 FlashX Legal knows little from memory, 2.91, and gains 2.52 points with 16.1 calls. The six that lose are the four ThaiLLM 8B models, Qwen3−8B and OpenThai 2.0 Legal, and they lose because they do not use the tool. Qwen3−8B, ThaiLLE, OpenThaiGPT and OpenThai 2.0 Legal never call it. Typhoon-S calls it on one item in three. Pathumma calls it 2.8 times per item and still loses 0.25 points. It retrieves text it cannot use. For these six models the tool only fills the context. It brings them no law. The gap is small at the top. For the proprietary models it is 0.07 to 1.05 points, and DeepSeek V4.1 Flash has closed it, 6.39 as an agent against 6.38 with the sections provided. Every other open-weight model stays 0.48 to 1.61 points below its Gold-cited In-force score. Even the best agents find the right sections for only part of the questions. With the tool the proprietary models cite the court’s sections with an F1 of 0.46 to 0.66. With the sections provided they reach 0.86 to 0.92.

Model	Param.	RAG InF.	GC InF.	Gain↑	Gap↓	Searches
Proprietary
Claude Fable 5.1	7.40	7.78	8.04	+0.38	+0.26	8.6
Gemini 4 Argon	6.17	<u>7.54</u>	7.63	+1.37	+0.09	7.6
GPT-6.1 Sol	<u>6.90</u>	7.28	<u>7.69</u>	+0.38	+0.41	6.3
Claude Opus 5.5	6.25	7.03	7.56	+0.78	+0.53	5.3
Muse Spark 1.3	5.28	6.37	7.09	+1.09	+0.72	9.0
GPT-6.1 Terra	5.36	6.14	6.21	+0.78	<u>+0.07</u>	9.2
Claude Sonnet 5.5	4.79	5.97	6.67	+1.18	+0.70	5.7
GLM 5.3 FlashX Legal	2.91	5.43	6.48	<u>+2.52</u>	+1.05	16.1
Claude Haiku 5.5	3.49	4.62	5.36	+1.14	+0.74	7.4
General
DeepSeek V4.1 Flash	3.79	6.39	6.38	+2.60	−0.01	17.9
DeepSeek V4 Flash	3.92	5.71	6.19	+1.79	+0.48	19.0
Muse Glimmer 30B	3.35	4.93	5.83	+1.58	+0.89	15.8
GLM 5.3 Flash	3.08	4.80	5.89	+1.72	+1.09	10.3
Gemma 4 31B	3.66	4.57	5.16	+0.91	+0.59	3.4
Gemma 4 26B-A4B	3.44	3.68	4.84	+0.24	+1.17	1.2
Nemotron 3.5 Lightning 30B-A3B	1.73	2.47	3.49	+0.74	+1.01	5.3
Qwen3.8 27B and its Thai adaptation
Qwen3.8 27B	3.18	5.48	6.04	+2.31	+0.55	17.6
↳ OpenThai 2.0 27B	2.96	5.10	5.85	+2.14	+0.75	16.6
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B-A3B	1.15	1.62	2.71	+0.47	+1.09	2.9
↳ SEA-LION v4.8 30B-A3B	1.66	2.45	3.53	+0.78	+1.08	4.7
↳ OpenThai 2.0 Legal 30B-A3B	1.18	0.87	1.92	−0.31	+1.05	0.0
Nemotron 3 Super 120B-A12B	2.05	3.23	4.29	+1.18	+1.06	6.5
↳ SEA-LION v4.8 120B-A12B	2.04	2.88	4.29	+0.84	+1.41	6.2
Qwen3−8B-Base descendants
Qwen3−8B	1.17	0.96	2.57	−0.21	+1.61	0.0
↳ Typhoon-S 8B	1.74	1.54	2.77	−0.20	+1.24	0.4
↳ Pathumma 8B Think	1.71	1.46	2.70	−0.25	+1.24	2.8
↳ THaLLE 0.2 8B	1.18	1.10	2.62	−0.08	+1.52	0.0
↳ OpenThaiGPT 8B	0.95	0.87	1.90	−0.08	+1.03	0.0

Application of the law. The Gold-cited In-force setting gives the model the court’s sections in the prompt. Nothing is left to find, and the score measures one thing, how well the model applies the law it was given to the facts of the case. With the sections provided the proprietary models score 5.36 to 8.04 and the open-weight models 1.90 to 6.38. Three open-weight models pass 6, DeepSeek V4.1 Flash at 6.38, DeepSeek V4 Flash at 6.19 and Qwen3.8 27B at 6.04. The share of answers that reach the court’s verdict shows the same distance. The proprietary models reach it in 72 to 91 percent of their answers, the open-weight models in 39 to 79 percent. Retrieval does not set this distance. The sections were in the prompt. What remains between the model and a good answer is the reasoning from the statute to the case, and no open-weight model closes it.

Does citing the right sections increase the Bar-standard score? Figure 10 sorts each model’s answers by their Citation F1 and plots the mean score of each group. Every line rises. On the average model, answers with an F1 of 0.75 or more score 2.2 points above answers that cited none of the court’s sections. F1 is the one citation number to track. Within a model it follows the score as closely as recall does, a median correlation of 0.34 against 0.35, and more closely than precision, 0.30. Of its two halves, recall carries more weight, 0.28 against 0.11 in a regression with both. Citing related law beyond the court’s sections costs little. Answers that cite all of the court’s sections and more than as many again still score 0.45 above the model’s mean, and 78 percent of their extra sections are in the same act as one of the court’s. The gray lines sit about six points apart. The model matters more than its citations. Citations explain 13 percent of the differences between answers. The rest is the reasoning the rubric grades. With the court’s sections handed to the model, in Gold-cited In-force, the share falls to 5 percent. The judge sees the citations and the rubric rewards a right one, and part of the link is built in. Even so, the score is not citation matching in disguise. Better citations are worth about two points of ten. The reasoning is worth the rest.

Thai and legal post-training. Several groups take an open-weight model and train it further on Thai text or on legal text. The aim is a model that handles Thai law. Nine such models are on the split, each with a reference model (Table 14). For five of them the reference is the base they were trained from. The four ThaiLLM 8B models are post-trainings of ThaiLLM-8B, a Thai continued pretraining of Qwen3-8B-Base that has no chat template and cannot answer the benchmark. Their reference is Qwen3-8B, Alibaba’s own post-training of the same Qwen3-8B-Base, a sibling rather than a parent. That comparison measures the Thai route, continued pretraining on Thai text and then post-training, against Alibaba’s post-training of the same base.

The extra training taught the models little law. In the Parametric setting, where a model answers from memory, each post-training scores close to its base. SEA-LION v4.8 30B-A3B gains the most, 0.51

A general index does not order the Thai legal frontier

points over Nemotron 3 Nano, and still cites almost none of the court’s sections from memory, a Citation F1 of 0.04 in that setting (Table D.1). OpenThai 2.0 Legal, trained on Thai legal text, scores within 0.03 of its base. GLM 5.3 FlashX Legal, Z.ai’s legal post-training of GLM 5.3 Flash, scores 0.17 below its base. OpenThai 2.0 27B scores 0.22 below Qwen3.8 27B. At 8B the Thai route is worth about half a point for two of the four models. Typhoon-S and Pathumma score 0.57 and 0.54 above Qwen3−8B, THaLLE is level with it and OpenThaiGPT is 0.22 below.

What the extra training changed in the larger pairs is how the model behaves with the search tool, and in three of the five it changed it for the worse. SEA-LION v4.8 30B-A3B searches more than its base, 4.7 calls per item against 2.9. It gains 0.83 points in RAG In-force and its Citation F1 rises from 0.25 to 0.34. SEA-LION v4.8 120B-A12B searches as often as Nemotron 3 Super and scores 0.35 below it. OpenThai 2.0 27B scores 0.38 below Qwen3.8 27B. OpenThai 2.0 Legal stopped calling the tool. Its base makes 2.9 calls per item and it makes none, and it loses 0.75 points. GLM 5.3 FlashX Legal is the one post-training that gains with the tool. It searches more than its base, 16.1 calls per item against 10.3, scores 0.63 above it in RAG In-force and 0.59 above it with the court’s sections provided, and its Citation F1 rises from 0.41 to 0.46. From memory it scores 0.17 below its base. The legal training taught it to search and to apply the law, not the law itself. Searching the law in force is the one thing that lifts an open-weight model on this benchmark (Table 13). Losing it costs more than the training added.

At 8B the base is the limit. Qwen3−8B and the four ThaiLLM 8B models land within 0.9 points of each other in every setting. None of the five calls the search tool, and the half point that Typhoon-S and Pathumma gain in Parametric is the same half point they gain in RAG In-force.

Two things raise an open-weight model on this benchmark, a stronger base and a search tool the model uses. Across the nine comparisons, training on Thai or legal text added tool use once, in GLM 5.3 FlashX Legal, and removed it once. It added statute knowledge in two of the four ThaiLLM 8B models, about half a point over Qwen3−8B, and in none of the larger pairs. Every model ran with its model-card sampling parameters and the same prompt (§5.3). A post-training tuned for a different prompt style may do better than it does here.

A general index does not order the Thai legal frontier

Model releases are judged by general indices built from coding, agentic and knowledge tests. We asked whether such an index predicts the standing on Thai legal work. Thirteen of the 28 leaderboard models have an Artificial Analysis Intelligence Index v5.0 [9], read through its API on 3 November 2026 at the headline configuration of each model. Figure 11 ranks the 13 by the index and by their RAG In-force score. Over the 13 models the two rankings agree on the tiers, a Spearman correlation of 0.67. Among the eight best models by the index they do not, 0.19. Claude Fable 5.1 is sixth by the index and first on NitiBench+. Claude Sonnet 5.5 is third and eighth. DeepSeek V4.1 Flash is ninth and fifth. Of the tests the index reports, Humanity’s Last Exam (HLE) orders the eight closest to NitiBench+, 0.69, and Terminal-Bench 4.0 farthest, 0.14. Two caveats apply. The index scores the Claude models with extended thinking and the leaderboard runs them without it. Eight models cannot establish a correlation. The finding is the narrow one. A general index tells the tier of a model. The choice among the frontier models for Thai legal work needs an in-domain measurement.

Discussion

What the setup must hold. A benchmark built from dated court decisions and scored against the current law measures two things at once, how well a model reasons and how far the law has moved since the decision. The second part is large. Half of the 156,650 decisions we audited cite law that has since changed, and in the Supreme Court two in three do (§ 3). It is also easy to miss. The current text rarely flips the verdict. It weakens the reasoning behind a verdict that was right either way (§ 6.2). On the items where the Bar-standard score drops by 0.45, Citation F1 moves by 0.04 (§ 6.4). A benchmark that checks the section number or the outcome alone does not see this loss. Its score mixes legal reasoning with statutory drift, in a share that changes every time the legislature acts. A fair setup records the date of each decision, names the version of each gold citation, stores the text in force next to the current text, and releases the full versioned corpus. Without the corpus nobody can repeat the experiments or audit the gold answers. Without the dates a benchmark goes stale and nobody notices, as NitiBench shows two years after release (§ 3). Every result of Part II is measured on this setup. Of the 53 legal benchmarks we audited, none meets it (§ 3).

For system builders. Searching the current law is the wrong default for a dispute rooted in a past event. Missing repealed law costs three times more than reading amended law, 0.79 against 0.25 points. Repealed provisions must stay in the index. The date filter belongs in the retrieval layer rather than in the prompt. A search over the law in force recovers 0.11 points on amended law and 0.47 on repealed law for the models that search, and nothing for the models that do not (§ 6.3). The strongest models gain the most from the text in force, up to 0.45 points on amended law and 1.35 on repealed law, because they build their answer on the text they are given. As models improve, the penalty for the wrong version grows.

Where the models stand. The best proprietary models score 7.0 to 7.8 on the leaderboard with the search tool, and Claude Fable 5.1 scores 7.4 from memory alone. On the Bar Council scale this is the good tier. These models know the Thai statutes, they search, and they stay within half a point of their own ceiling. They cite more sections than the court did, a precision of 0.39 to 0.61 against a recall of 0.69 to 0.88 (§ 7.2). The best open-weight model, DeepSeek V4.1 Flash, scores 6.39, 1.4 points below Claude Fable 5.1, at 0.02 dollars per answer (§ 7.3). The 13 open-weight models of the benchmark protocol stop at 4.94 with the search tool and 5.75 with the court’s sections in the prompt, the acceptable tier. Two things stand between them and a good answer. The first is finding the law. The in-force index returns the court’s sections for 48 percent of the searches (§ 6.3). The second is applying it. With the right sections in hand no model passes 6, and about six answers in ten reach the court’s verdict (Table 10).

The tool is the dividing line. The 28 models split into those that use the search tool and those that do not. The 22 that use it gain 0.24 to 2.60 points. The six that do not lose points. Four of the six never call the tool. Two call it and cannot use what it returns. A Thai legal system built on a model that does not search can only know what the model remembers. The Parametric column of Table 13 shows how little that is for every open-weight model but the two DeepSeek models, Gemma 4 and Muse Glimmer. Tool use comes first for Thai legal QA, before Thai data and before legal data.

A general index does not pick the model. Launch posts quote general indices. On the 13 leaderboard models that have one, the Artificial Analysis Intelligence Index sorts the tiers and not the frontier (§ 7.8). A buyer who picks a model for Thai legal work from a general index gets the tier right and the order within it no better than chance. The in-domain measurement is the one to use.

What post-training on Thai data adds. Training an open-weight model further on Thai or legal text is the main route Thai groups take to a Thai legal model. On NitiBench{sup}+ it added little law, about half a point in two of the four ThaiLLM 8B models over Qwen3–8B, which shores their base, and nothing in the larger pairs. In RAG In-force it raised one of the larger bases and lowered three, and in one pair the model stopped using the tool ({xref}§7.6). Qwen3–8B and the four ThaiLLM 8B models land within 0.9 points of each other, and none of the five searches. The one legal post-training that gains, GLM 5.3 FlashX Legal, gains in the two grounded settings and not from memory ({xref}§7.6`). The gains on this benchmark come from a stronger base and from searching the law in force. The open problem is a post-training that keeps the tool use of the base and teaches the model to apply the statutes to the facts. The benchmark measures both.

Domains. Over the 27 leaderboard rows the mean score runs from 4.49 in Natural Resources & Licensing to 3.76 in Public Personnel. The five lowest domains are Public Personnel, Administrative Contracts, Tax & Revenue, Narcotics and Criminal (§ 7.4). Two of the five are domains of the Supreme Administrative Court. Its cases cite acts, decrees and regulations outside the codes, and each of these has its own history of amendment and repeal. A model trained on the codes alone is not a model of Thai law. The per-domain scores are the numbers a user should read, and Appendix § 16.§ 1 gives them for every model.

Conclusion

NitiBench{sup}+` measures Thai legal question answering the way a Thai court and a Thai bar examiner would. The questions come from decided cases, the grade covers the verdict and the reasoning, and the sections the court relied on are read in the version in force on the day of the decision. Part I shows that the version of the text is part of the task. Half of 156,650 Thai decisions cite law that has since changed. When a model reads the current text of a changed law, its Bar-standard score falls. This holds for every model, and verdict agreement and Citation F1 barely move. A legal benchmark must therefore record the date of each question, name the version of the law in force on that date, and grade against that text. A deployed system must index the law by date and keep repealed provisions. The text in force is the fair setting, and it is the setting of the leaderboard.

Part II measures today’s models on that setting. The best proprietary model is graded good. The best open-weight model of the benchmark protocol is graded acceptable, and no open-weight model reaches the good tier. With no text in front of them, the open-weight models know almost no Thai law. The two DeepSeek Flash models, Gemma 4 and Muse Glimmer are the exceptions. A search tool over the law in force closes part of the distance for the models that use it, and none for the models that do not. The court’s sections in the prompt close more. What remains is the application of the law to the facts. The administrative domains are the hardest. Training on Thai or legal data added about half a point of law knowledge in two of the four ThaiLLM 8B models, against Alibaba’s post-training of the same base, and none in the three larger pairs we can test. The leaderboard takes new models as they are released, on the same setup. Its test split is held out, and a validation split of the same shape is public.

Limitations

Scope. NitiBench⁺ covers Thai law and follows Thai legal practice. To repeat the method in another jurisdiction, one needs the statute text as it stood on each date and a corpus of dated decisions. The benchmark samples from three high courts. It leaves out military courts, lower courts, and advisory rulings from government agencies. The 17 domains hold between 56 and 146 items each. The macro-average weights them equally.

Judges. An LLM judge scores the reasoning. Its agreement with human bar examination graders was checked in earlier work [2]. A judge model cannot fully replace a human expert, least of all on a fine or borderline point of law.

Models. The benchmark protocol holds 13 open-weight models of 8B to 120B parameters, served locally. The leaderboard adds nine proprietary models and six further open-weight models on the 500-item split, with one sample per item for every row outside the 13 and Qwen3–8B. A single sample carries the sampling noise shown in Table E.5. Every model runs with the sampling parameters of its model card and the same prompt. A prompt or an agent loop tuned to one model can move its score. The test split is held out, so a model appears on the leaderboard when we run it and not when its maker does.

Magnitudes. The version effects are modest. Reading the current text of amended law costs about 0.25 points of the Bar-standard score on its 10-point scale, and missing repealed law costs about 0.8 points. Seeing any legal text at all matters more. It adds 0.8 to 2.4 points across models.

Ethics Statement

All rulings and decisions used are publicly available government documents. Under Section 7(2) to (5) of Thailand’s Copyright Act B.E. 2537 (1994), government documents are explicitly excluded from copyright protection and reside in the public domain. NitiBench⁺ is intended for research evaluation and is not a substitute for professional legal advice.

Dataset

Released files and dataset example

The validation split is released as two JSON Lines files (§ 4.4). Every record of both files carries the canary string in a canary field. In the text fields of the validation split the names of private individuals and of officials named in person are replaced by their initials in the style of Thai law reports, with the same initial for the same person within a case, and citizen ID numbers, personal phone numbers, bank account numbers, vehicle plates and house numbers are masked. The example that follows is a case of the validation split, the item of Figure 1.

validation_split.jsonl, one record per case. id, domain (one of the 17 domains), court and source (the deciding court, as a name and as a key), source_id (the decision number), decision_date (ISO-8601), year_be, decade, title, fact_pattern (the facts of the case, Thai), drift_band, n_sub_questions, n_cited_laws, asks_shared_question (the cases that ask the same question, used to cluster confidence intervals and to keep the two splits apart), corpus_date (the date of the current text) and sub_questions.

Sub-question. question, question_type (ruling for a question the court itself answered, or governing_provisions, legal_consequences, provision_interaction and lower_court_departure for the four added kinds of §4.1), depends_on_previous (true when this sub-question’s verdict is conditioned on the prior one), verdict and reasoning (the gold answer, copied from the decision), and citations.

Citation. law_code and law_name_th (the law), label (the section as cited), is_loadbearing (true when the court relies on the section for the verdict), version_uri with valid_from and valid_to (the version in force on the decision date), current_uri (the version in force on the corpus date) and drift (unchanged, amended or repealed). Only citations the court relies on are scored. Background mentions are kept with is_loadbearing set to false. A citation of an instrument the statute corpus does not hold, a notification or an order for example, carries the instrument’s name in law and no identifier, and is left out of Citation F1.

gold_sections.jsonl, one record per section version. version_uri (of the form /law/{law_code}/section-{label}/v/{valid_from}), law_code, law_name_th, label, component_type, valid_from, valid_to, is_law_active and text. The file holds both versions of every gold section of the split, 3,253 section versions, so that the Gold-cited In-force setting can be run from the release by joining on version_uri.

The rest of the release. The rubric, the judge prompt and the citation-matching rules as documents, the composition of the split, a license file and checksums.

Item labor::deka::18883-2557. Domain: labor. Source: Supreme Court Decision 18883/2557. Decision date: 2014. Drift band: Amended.

Fact pattern (Thai, translated): A voluntarily insured person under the Social Security Act B.E. 2533 died of causes unrelated to work. She had no spouse and no children, her parents had died before her, and she had named no beneficiary. Her full sister claimed the death grant and the old-age lump sum. The Social Security Office refused both, and the appeal committee upheld the refusals. The sister sued, arguing that the contributions were savings that passed to her as a statutory heir. The Central Labour Court upheld the refusal of the death grant. It held that the old-age lump sum was part of the estate and passed to the statutory heirs. Both parties appealed.

Sub-question 2 (contrast with the lower court): The Central Labour Court held that the right of a deceased insured person to the old-age lump sum passes to the statutory heirs under the Civil and Commercial Code, and that the full sister may therefore claim it. Was that correct, and why?

Gold verdict: No. The sister is not among the heirs that section 77 quater, paragraph two, entitles to the lump sum. Even when none of those heirs exists, money of the Social Security Fund cannot be paid to other heirs against sections 24 and 77 quater.

Gold citations: Social Security Act B.E. 2533, sections 77 quater, 77(2), and 24 paragraph one, all relied on. Section 77 bis is kept as background.

Section 77 quater. In force on the decision date: version_uri .../v/2002-10-09, valid 2002-10-09 to 2015-10-19. Current: current_uri .../v/2023-10-25, drift amended. Both texts are in gold_sections.jsonl. The current text adds a paragraph that gives the lump sum to full siblings first when no heir of paragraph two exists.

Drift bands

Each gold citation is resolved to two versions of its section in the dated section corpus. One is the version in force on the decision date. The other is the current version, the one in force on 1 September 2026 in a law that is still active. A citation is repealed when its law is no longer active or the section no longer exists. It is amended when the two versions have different content hashes, and unchanged otherwise. An item takes the most severe state among the citations its verdicts rely on, with repealed above amended above unchanged. Background citations do not set the band. The Repealed band also holds the few citations to a law that was already repealed on the decision date, because a court applies the law in force when the facts occurred (Penal Code §2–3 set this rule for criminal matters). Such citations are rare. They are not drift, since the text did not change after the decision, but for a retriever over current law the text is missing all the same.

Judging

Bar Council eight-tier rubric

Table 6 in the body summarizes the grading regulation reproduced inline in the judge prompt. The regulation was issued by the Office of Legal Education Training of the Thai Bar Council in 2014 and remains the standardized grading instrument for Thai legal essay answers. We reproduce here an English translation of the scoring clause.

	Reasoning quality
Final answer	Unusable	Some	Acceptable	Good	Very good
matches gold	T4: 0–1	T5: 2–5	T6: 5–6	T7: 7–8	T8: 9–10
differs	T1: 0	T2: 1–2		T3: 2–4

The grading considers the final answer and supporting reasons together, on a per-question basis:

1. Final answer differs from gold, reasoning unusable: 0.

2. Final answer differs, reasoning somewhat acceptable: 1 to 2.

3. Final answer differs, reasoning good: 2 to 4.

4. Final answer matches gold, reasoning unusable: 0 to 1.

5. Final answer matches gold, reasoning somewhat acceptable: 2 to 5.

6. Final answer matches gold, reasoning acceptable: 5 to 6.

7. Final answer matches gold, reasoning good: 7 to 8.

8. Final answer matches gold, reasoning very good: 9 to 10.

If a question contains multiple issues, the grader fixes the score for each issue first and then applies the criteria to each. For items with several sub-questions, we apply the rubric per sub-question and average within an item.

Judge prompt

The prompt is identical across judge models and across all items. Only the question text, decision date, gold answer, and model answer change per item. The judge is called once per answer at temperature 0, with no tools, no retrieval, and no few-shot examples. It receives four inputs, the question, the decision date, the gold answer and the model answer, and returns for each sub-question a tier of the regulation, a score within that tier’s range and a one-paragraph justification, as strict JSON. The system prompt and the message builder are published verbatim in the release, file grading/judge_prompt.md (§ 4.4).

Inputs the judge does not receive. The judge receives no information about the model that produced the answer, no information about the setting under which the answer was produced, no scores from any other judge, and no access to the FourCorners graph. The citation scores are computed independently of the judge.

Metric implementation

Citation scores are computed in two stages. Resolution maps each predicted citation to a section identifier in the corpus. Matching compares the resolved set with the gold set of the sub-question (§ 5.1).

1. Resolution. In every setting except Parametric the model copies a version identifier from the text it was given or retrieved. The scorer looks the identifier up in the dated section corpus and takes that version’s section. In Parametric the model names a law and a section, or gives the section identifier. A law name is resolved against the law catalog of the corpus with rapidfuzz.WRatio (threshold 85). The section label is normalized (Thai numerals to Arabic digits, sub-paragraph markers removed). The section is then resolved to its version in force on the decision date.

2. Matching. Citation F1 counts a match when the section identifiers agree. A predicted section that the sub-question’s source cites only as background is removed from the prediction set first, so it is neither a hit nor a false positive.

3. Aggregation. Precision, recall, and F1 are computed per sub-question, averaged within an item, averaged within a domain, and then macro-averaged over the seventeen domains.

Edge cases. A version identifier that the corpus does not hold still resolves to its section. The 44 sub-questions whose gold has no resolvable citation are scored on the rubric only. An answer that does not parse scores zero on every citation metric.

Serving and Compute

Table C.1 lists every evaluated model with its checkpoint, size, context window, reasoning mode, and tool-call parser.

Model	Checkpoint (Hugging Face)	Params	TP	Window	Reasoning	Tool parser
General
Gemma 4 31B	google/gemma-4-31B-it	31B	2	262k	on (gemma4)	gemma4
Gemma 4 26B-A4B	google/gemma-4-26B-A4B-it	26B/4B	2	262k	on (gemma4)	gemma4
Nemotron 3.5 Lightning 30B-A3B	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16	32B/3B	2	262k	on (nemotron_v3)	qwen3_xml
Muse Glimmer 30B	meta-models/Muse-Glimmer-30B	30B	2	131k	on (muse_glimmer)	muse_glimmer
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B-A3B	nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16	32B/3B	2	262k	on (nemotron_v3)	qwen3_coder
↪ SEA-LION v4.8 30B-A3B	aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B	33B/3B	2	262k	on (nemotron_v3)	qwen3_coder
↪ OpenThai 2.0 Legal 30B-A3B	iapp/openthai2.0-legal-thaillm-nemotron-3-nano-30b-a3b	32B/3B	2	262k	on (nemotron_v3)	qwen3_coder
Nemotron 3 Super 120B-A12B	nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16	124B/12B	4	262k	on (nemotron_v3)	qwen3_coder
↪ SEA-LION v4.8 120B-A12B	aisingapore/Nemotron-SEA-LION-v4.8-120B-A12B	124B/12B	4	262k	on (nemotron_v3)	qwen3_coder
ThaiLLM 8B derivatives
Typhoon-S 8B	typhoon-ai/typhoon-s-thaillm-8b-instruct-research-preview	8B	1	32k	none	hermes
Pathumma 8B Think	nectec/pathumma-thaillm-8b-think-3.0.0	8B	1	40k	on (qwen3)	hermes
THaLLE 0.2 8B	KBTG-Labs/THaLLE-0.2-ThaiLLM-8B-fa	8B	1	40k	on (qwen3)	hermes
OpenThaiGPT 8B	openthaigpt/openthaigpt-thaillm-8b-instruct-v0.7.2-research-preview	8B	1	40k	on (qwen3)	hermes

Every model runs locally with vLLM on one node of four A100 80GB GPUs. Table C.2 reports, over the five settings and three samples, the tokens each model reads and writes, the share of what it writes that is reasoning, and the number of searches per item in the two RAG settings. The GPU hours column gives the wall-clock time of each model’s runs on the node times the GPUs it occupied.

The two best models are also economical in the RAG In-force setting. Gemma 4 31B reaches 4.55 with 7.8 thousand output tokens and 3.3 searches per item. Muse Glimmer reaches 4.94 with 9.8 thousand output tokens, but its 15.6 searches per item bring 341 thousand input tokens of tool results, more than any other model. Nemotron 3 Nano writes 30.7 thousand output tokens per item for a score of 1.66. Output length does not predict the score.

Model	Tokens (M) In	Tokens (M) Out	Reasoning share	Searches per item RAG Cur.	Searches per item RAG Inf.	GPU hours
General
Gemma 4 31B	416	126.4	84%	3.2	3.3	315
Gemma 4 26B-A4B	193	202.8	91%	1.3	1.2	200
Nemotron 3.5 Lightning 30B-A3B	1563	424.2	92%	5.7	5.1	380
Muse Glimmer 30B	3599	165.7	84%	16.4	15.6	455
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B-A3B	729	389.1	93%	3.2	2.9	235
SEA-LION v4.8 30B-A3B	1251	330.5	92%	5.2	4.6	175
OpenThai 2.0 Legal 30B-A3B	101	195.0	93%	0.0	0.0	42
Nemotron 3 Super 120B-A12B	1853	183.7	83%	5.8	5.9	440
SEA-LION v4.8 120B-A12B	2117	156.5	86%	6.7	6.1	375
ThaiLLM 8B derivatives
Typhoon-S 8B	106	69.0	0%	0.2	0.3	60
Pathumma 8B Think	264	69.6	74%	2.9	2.8	40
THaLLE 0.2 8B	92	53.2	57%	0.0	0.0	25
OpenThaiGPT 8B	100	62.4	34%	0.0	0.0	36

Leaderboard Tables

Tables D.1 and D.2 give the numbers behind Figure 6 and Figure 7.

#	Model	Access	RAG InF.	Parametric	GC InF.	F1	Precision	Recall
1	Claude Fable 5.1	Proprietary	7.78	7.40	8.04	0.515	0.394	0.880
2	Gemini 4 Argon	Proprietary	7.54	6.17	7.63	0.629	0.549	0.857
3	GPT-6.1 Sol	Proprietary	7.28	6.90	7.69	0.657	0.614	0.799
4	Claude Opus 5.5	Proprietary	7.03	6.25	7.56	0.585	0.500	0.809
5	DeepSeek V4.1 Flash	Open	6.39	3.79	6.38	0.477	0.372	0.800
6	Muse Spark 1.3	Proprietary	6.37	5.28	7.09	0.576	0.530	0.720
7	GPT-6.1 Terra	Proprietary	6.14	5.36	6.21	0.624	0.576	0.769
8	Claude Sonnet 5.5	Proprietary	5.97	4.79	6.67	0.583	0.531	0.747
9	DeepSeek V4 Flash	Open	5.71	3.92	6.19	0.543	0.476	0.742
10	Qwen3 27B	Open	5.48	3.18	6.04	0.480	0.402	0.704
11	GLM 5.3 FlashX Legal	Proprietary	5.43	2.91	6.48	0.459	0.366	0.734
12	OpenThai 2.0 27B	Open	5.10	2.96	5.85	0.509	0.459	0.665
13	Muse Glimmer 30B	Open	4.93	3.35	5.83	0.541	0.536	0.620
14	GLM 5.3 Flash	Open	4.80	3.08	5.89	0.414	0.355	0.577
15	Claude Haiku 5.5	Proprietary	4.62	3.49	5.36	0.513	0.454	0.692
16	Gemma 4 31B	Open	4.57	3.66	5.16	0.503	0.642	0.460
17	Gemma 4 26B-A4B	Open	3.68	3.44	4.84	0.254	0.358	0.217
18	Nemotron 3 Super 120B-A12B	Open	3.23	2.05	4.29	0.424	0.498	0.421
19	SEA-LION v4.8 120B-A12B	Open	2.88	2.04	4.29	0.351	0.394	0.367
20	Nemotron 3.5 Lightning 30B-A3B	Open	2.47	1.73	3.49	0.342	0.420	0.330
21	SEA-LION v4.8 30B-A3B	Open	2.45	1.66	3.53	0.335	0.401	0.330
22	Nemotron 3 Nano 30B-A3B	Open	1.62	1.15	2.71	0.251	0.292	0.256
23	Typhoon-S 8B	Open	1.54	1.74	2.77	0.034	0.045	0.030
24	Pathumma 8B Think	Open	1.46	1.71	2.70	0.155	0.210	0.137
25	THaLLE 0.2 8B	Open	1.10	1.18	2.62	0.003	0.004	0.003
26	Qwen3-8B	Proprietary	0.96	1.17	2.57	0.000	0.000	0.000
27	OpenThaiGPT 8B	Open	0.87	0.95	1.90	0.000	0.000	0.000
28	OpenThai 2.0 Legal 30B-A3B	Open	0.87	1.18	1.92	0.005	0.006	0.004

#	Model	Bar-standard	Cost (USD)	Output tokens (K)	Searches	Hallucinated citations
1	Claude Fable 5.1	7.78	1.155	8.3	8.6	0.0%
2	Gemini 4 Argon	7.54	0.338	5.2	7.6	0.0%
3	GPT-6.1 Sol	7.28	0.134	4.1	6.3	0.0%
4	Claude Opus 5.5	7.03	0.411	5.3	5.3	0.0%
5	DeepSeek V4.1 Flash	6.39	0.024	25.6	17.9	0.0%
6	Muse Spark 1.3	6.37	0.149	9.2	9.0	0.0%
7	GPT-6.1 Terra	6.14	0.086	3.4	9.2	0.0%
8	Claude Sonnet 5.5	5.97	0.204	5.1	5.7	0.0%
9	DeepSeek V4 Flash	5.71	0.039	45.4	19.0	0.0%
10	Qwen3.8 27B	5.48	0.235	52.3	17.6	0.1%
11	GLM 5.3 FlashX Legal	5.43	0.171	44.3	16.1	0.1%
12	OpenThai 2.0 27B	5.10	0.252	43.0	16.6	0.0%
13	Muse Glimmer 30B	4.93	0.120	10.1	15.8	0.0%
14	GLM 5.3 Flash	4.80	0.044	35.3	10.3	1.5%
15	Claude Haiku 5.5	4.62	0.024	12.3	7.4	0.0%
16	Gemma 4 31B	4.57	0.011	7.9	3.4	0.0%
17	Gemma 4 26B-A4B	3.68	0.005	11.8	1.2	0.3%
18	Nemotron 3 Super 120B-A12B	3.23	0.096	12.9	6.5	0.0%
19	SEA-LION v4.8 120B-A12B	2.88	0.084	8.3	6.2	1.0%
20	Nemotron 3.5 Lightning 30B-A3B	2.47	0.015	22.6	5.3	0.7%
21	SEA-LION v4.8 30B-A3B	2.45	0.010	15.3	4.7	0.1%
22	Nemotron 3 Nano 30B-A3B	1.62	0.011	30.1	2.9	0.5%
23	Typhoon-S 8B	1.54	0.001	2.6	0.4	8.6%
24	Pathumma 8B Think	1.46	0.001	2.7	2.8	3.0%
25	THaLLE 0.2 8B	1.10	0.000	2.8	0.0	13.4%
26	Qwen3−8B	0.96	0.000	2.9	0.0	25.5%
27	OpenThaiGPT 8B	0.87	0.001	3.5	–	27.6%
28	OpenThai 2.0 Legal 30B-A3B	0.87	0.010	11.5	0.0	24.7%

Additional Results

Score by model and domain

Table E.1 gives the mean Bar-standard score per domain over the 13 open-weight models of the benchmark protocol in three settings, and Table E.2 the score of every model in every domain in RAG In-force.

Domain	Items	Param.	GC InF.	RAG InF.	Model	Score	Lowest
Family & Succession	146	2.02	3.99	2.72	Muse Glimmer 30B	5.09	0.90
Intellectual Property	109	2.31	3.66	2.70	Muse Glimmer 30B	5.30	1.04
Public Law	98	2.26	3.73	2.69	Muse Glimmer 30B	5.49	0.85
Natural Resources & Licensing	84	2.07	3.82	2.66	Muse Glimmer 30B	5.39	0.93
Traffic & Transport	63	1.88	3.57	2.62	Muse Glimmer 30B	5.45	0.88
Labor	116	1.97	3.56	2.56	Muse Glimmer 30B	5.10	0.89
Civil & Commercial	144	1.99	3.76	2.49	Gemma 4 31B	4.92	0.82
Criminal	146	2.02	3.51	2.47	Muse Glimmer 30B	4.98	0.81
Tax & Revenue	113	1.94	3.55	2.45	Muse Glimmer 30B	4.98	0.79
Securities & Finance	56	2.01	3.56	2.44	Muse Glimmer 30B	4.88	0.73
Civil Procedure	87	1.95	3.55	2.43	Muse Glimmer 30B	5.00	0.81
Administrative Acts	82	2.03	3.50	2.40	Muse Glimmer 30B	4.94	0.82
Land & Property	117	1.84	3.38	2.36	Gemma 4 31B	4.46	0.74
State Liability	80	2.08	3.39	2.31	Muse Glimmer 30B	4.73	0.76
Narcotics	74	1.80	3.13	2.30	Muse Glimmer 30B	4.70	0.83
Administrative Contracts	74	2.00	3.30	2.21	Muse Glimmer 30B	4.46	0.78
Public Personnel	81	1.81	2.97	1.99	Muse Glimmer 30B	4.22	0.53
All domains	1,670	2.00	3.52	2.46

Model	All	Family & Succession	Intellectual Property	Public Law	Natural Resources & Licensing	Traffic & Transport	Labor	Civil & Commercial	Criminal	Tax & Revenue	Securities & Finance	Civil Procedure	Administrative Acts	Land & Property	State Liability	Narcotics	Administrative Contracts	Public Personnel
Muse Glimmer 30B	4.94	5.09	5.30	5.49	5.39	5.45	5.10	4.87	4.98	4.98	4.88	5.00	4.94	4.45	4.73	4.70	4.46	4.22
Gemma 4 31B	4.55	5.05	4.83	4.85	4.75	4.59	4.76	4.92	4.73	4.59	4.64	4.50	4.37	4.46	4.20	4.16	4.23	3.79
Gemma 4 26B-A4B	3.71	3.98	4.04	4.08	3.78	3.77	3.85	3.94	3.92	3.76	3.75	3.74	3.68	3.58	3.47	3.33	3.51	2.97
Nemotron 3 Super	3.31	3.70	3.45	3.52	3.52	3.79	3.48	3.37	3.39	3.25	3.32	3.31	3.13	3.20	3.00	3.21	2.93	2.63
SEA-LION v4.8 120B	2.94	3.19	3.10	3.43	3.47	3.33	3.01	2.93	2.84	2.98	2.99	2.86	2.80	2.79	2.71	2.73	2.56	2.21
SEA-LION v4.8 30B	2.50	2.84	2.82	2.64	2.81	2.66	2.62	2.52	2.35	2.46	2.51	2.31	2.54	2.55	2.53	2.28	2.08	1.96
Nemotron 3.5 Lightning	2.47	2.81	2.70	2.57	2.74	2.70	2.61	2.46	2.51	2.56	2.46	2.36	2.45	2.45	2.13	2.34	2.09	2.02
Nemotron 3 Nano	1.66	2.02	1.97	1.69	1.81	1.74	1.74	1.65	1.64	1.68	1.58	1.51	1.63	1.70	1.54	1.64	1.28	1.34
Typhoon-S 8B	1.56	1.64	1.93	1.92	1.56	1.59	1.57	1.45	1.50	1.49	1.49	1.56	1.44	1.45	1.69	1.30	1.59	1.31
Pathumma 8B Think	1.47	1.76	1.51	1.55	1.69	1.56	1.66	1.41	1.52	1.34	1.33	1.53	1.35	1.46	1.28	1.50	1.28	1.28
ThALLE 0.2 8B	1.11	1.26	1.34	1.34	1.17	1.05	1.09	1.10	1.06	1.08	1.08	1.06	1.15	0.97	1.13	1.00	1.07	0.94
OpenThaiGPT 8B	0.88	0.90	1.06	1.02	0.93	0.88	0.89	0.82	0.81	0.86	0.96	0.81	0.94	0.74	0.90	0.83	0.83	0.72
OpenThai 2.0 Legal	0.88	1.06	1.04	0.85	0.98	0.93	0.93	0.94	0.90	0.79	0.73	1.07	0.82	0.84	0.76	0.95	0.78	0.53
Mean	2.46	2.72	2.70	2.69	2.66	2.62	2.56	2.49	2.47	2.45	2.44	2.43	2.40	2.36	2.31	2.30	2.21	1.99

Table E.3 gives the RAG In-force score of every leaderboard row in every domain.

Model	Land & Property	Intellectual Property	Family & Succession	Tax & Revenue	Civil & Commercial	Criminal	Labor	Securities & Finance	Narcotics	Civil Procedure	Natural Resources & Licensing	Traffic & Transport	Public Law	Administrative Acts	Administrative Contracts	Public Personnel	State Liability
Claude Fable 5.1	7.83	8.08	8.09	7.28	7.81	7.41	8.06	7.83	7.59	7.75	8.02	8.13	8.17	8.10	7.16	7.21	7.80
Gemini 4 Argon	7.23	7.57	7.62	6.76	7.68	7.51	8.28	7.65	6.39	7.82	8.04	7.62	8.16	7.67	7.00	7.45	7.71
GPT-6.1 Sol	7.01	7.68	6.96	6.38	7.55	7.31	7.75	7.70	6.50	7.52	7.77	7.39	7.84	7.84	6.40	6.98	7.19
Claude Opus 5.5	6.75	7.49	6.95	6.12	7.30	6.74	7.85	7.13	6.74	7.20	7.59	7.23	7.25	7.01	6.28	6.76	7.16
DeepSeek V4.1 Flash	5.97	6.37	6.18	5.93	6.64	5.67	6.67	6.49	6.58	6.62	6.84	6.85	6.87	6.76	5.58	6.23	6.36
Muse Spark 1.3	5.97	6.82	6.35	5.85	6.83	6.31	6.58	6.79	6.00	5.72	6.77	6.92	6.85	6.12	5.95	6.07	6.32
GPT-6.1 Terra	6.02	6.52	5.93	5.57	6.34	5.97	6.23	6.40	5.45	5.74	6.50	6.61	6.46	6.42	5.96	6.05	6.19
Claude Sonnet 5.5	5.36	6.25	6.05	5.49	6.37	5.38	6.35	6.54	5.67	5.87	6.47	6.09	6.39	6.23	5.58	5.44	6.03
DeepSeek V4 Flash	5.76	6.02	5.66	5.17	6.11	5.73	6.12	5.78	4.96	5.41	6.32	5.71	6.25	6.30	5.29	4.80	5.66
Qwen3.8 27B	5.68	5.06	5.78	5.20	5.88	4.95	5.75	5.26	4.28	5.18	5.89	6.02	6.15	6.11	5.42	4.99	5.65
GLM 5.3 FlashX Legal	5.58	5.90	5.20	5.28	6.25	4.42	5.70	5.92	5.09	4.98	6.07	4.74	6.05	5.71	5.01	5.24	5.14
OpenThai 2.0 27B	5.27	5.12	5.41	4.83	4.54	4.15	5.18	5.39	5.00	5.35	5.93	5.68	5.88	4.84	4.69	4.47	4.99
Muse Glimmer 30B	4.77	5.36	4.98	4.45	4.53	4.70	5.04	5.14	4.69	5.02	5.42	5.53	5.45	5.35	4.58	4.36	4.49
GLM 5.3 Flash	4.62	5.97	5.34	4.88	5.21	3.92	4.67	4.60	3.89	4.99	5.45	4.75	5.87	4.95	4.33	4.11	4.07
Claude Haiku 5.5	4.64	4.51	4.49	4.17	4.80	4.42	5.05	4.62	4.41	4.59	4.78	5.26	5.25	4.68	4.47	4.05	4.41
Gemma 4 31B	4.56	4.64	4.88	4.30	4.81	4.86	4.71	4.89	4.09	4.65	5.05	4.91	4.62	4.22	4.54	3.84	4.11
Gemma 4 26B-A4B	3.59	3.58	3.77	3.60	3.76	4.06	4.24	3.83	3.16	3.75	3.88	4.05	3.52	3.60	3.81	2.95	3.36
Nemotron 3 Super 120B-A12B	3.35	3.23	3.80	3.12	3.31	2.99	3.15	3.32	3.00	3.21	3.33	3.76	3.33	3.28	3.07	2.87	2.84
SEA-LION v4.8 120B-A12B	2.89	2.98	3.18	2.86	2.71	2.63	2.74	2.90	2.74	2.59	3.48	3.53	2.94	3.00	2.74	2.30	2.77
Nemotron 3.5 Lightning 30B-A3B	2.61	2.50	2.94	2.31	2.44	2.45	2.48	2.61	2.31	2.29	2.62	2.69	2.50	2.56	2.26	2.17	2.32
SEA-LION v4.8 30B-A3B	2.77	2.63	2.76	2.20	2.27	2.28	2.19	2.55	2.31	2.41	2.56	2.68	2.56	2.55	2.02	2.29	2.55
Nemotron 3 Nano 30B-A3B	1.64	2.06	2.02	1.39	1.62	1.63	1.43	1.78	1.59	1.46	1.82	1.60	1.71	1.63	1.33	1.38	1.49
Typhoon-S 8B	1.53	1.82	1.66	1.35	1.44	1.60	1.62	1.54	1.36	1.38	1.62	1.55	1.76	1.50	1.56	1.23	1.62
Pathumma 8B Think	1.59	1.38	1.87	1.30	1.18	1.46	1.70	1.33	1.63	1.51	1.63	1.66	1.49	1.40	1.37	1.15	1.23
ThaLLEO 2.8B	0.81	1.25	1.12	1.10	1.09	1.12	1.14	1.03	1.07	1.09	1.18	1.01	1.18	1.37	1.14	0.85	1.19
Qwen3-8B	0.77	1.14	1.00	0.92	0.97	0.95	0.87	0.94	1.03	0.64	1.16	0.80	1.19	1.03	1.02	0.71	1.17
OpenThaiGPT 8B	0.73	0.87	0.88	0.76	0.72	0.86	0.93	0.96	1.06	0.71	0.98	0.85	0.88	1.07	0.96	0.70	0.91
OpenThai 2.0 Legal 30B-A3B	0.77	0.88	0.95	0.79	0.87	0.97	0.98	0.77	1.04	1.15	0.92	0.93	0.85	0.76	0.84	0.55	0.81

Best of three samples

Table E.4 takes the best of the three samples of each item and then aggregates as for avg@3. It is a capability ceiling. No claim in the report rests on it.

Model	Param.	GC Cur.	GC InF.	RAG Cur.	RAG InF.	Param.	GC Cur.	GC InF.	RAG Cur.	RAG InF.
General
Gemma 4 31B	0.373	0.715	0.861	<u>0.506</u>	<u>0.604</u>	4.38	<u>5.46</u>	<u>6.05</u>	<u>4.93</u>	<u>5.33</u>
Gemma 4 26B–A4B	<u>0.341</u>	0.712	0.850	0.352	0.414	<u>4.11</u>	5.07	5.53	4.37	4.52
Nemotron 3.5 Lightning 30B–A3B	0.054	0.717	0.842	0.377	0.466	2.28	3.81	4.21	2.95	3.18
Muse Glimmer 30B	0.271	0.773	0.914	0.521	0.632	4.00	5.94	6.58	5.16	5.78
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B–A3B	0.036	0.719	0.842	0.306	0.379	1.70	3.09	3.39	2.09	2.26
↪ SEA-LION v4.8 30B–A3B	0.071	0.716	0.839	0.383	0.467	2.23	3.96	4.33	2.95	3.29
↪ OpenThai 2.0 Legal 30B–A3B	0.143	0.627	0.705	0.011	0.013	1.85	2.76	2.90	1.49	1.50
Nemotron 3 Super 120B–A12B	0.160	<u>0.740</u>	<u>0.874</u>	0.453	0.550	2.62	4.57	5.10	3.70	4.12
↪ SEA-LION v4.8 120B–A12B	0.153	0.733	0.873	0.439	0.530	2.65	4.61	5.16	3.61	4.00
ThaiLLM 8B derivatives
Typhoon-S 8B	0.163	0.688	0.796	0.034	0.092	2.26	3.36	3.58	2.18	2.22
Pathumma 8B Think	0.235	0.686	0.787	0.228	0.285	2.34	3.43	3.57	2.17	2.26
THaLLE 0.2 8B	0.043	0.695	0.817	0.004	0.006	1.76	3.07	3.40	1.64	1.65
OpenThaiGPT 8B	0.041	0.695	0.815	0.000	0.000	1.51	2.55	2.74	1.43	1.42

Variation across samples

Table E.5 scores each of the three samples alone and reports the standard deviation of the setting-level score.

Model	Citation F1, SD over samples					Bar-standard, SD over samples
	Param.	GC Cur.	GC InF.	RAG Cur.	RAG InF.	Param.	GC Cur.	GC InF.	RAG Cur.	RAG InF.
General
Gemma 4 31B	0.003	0.005	0.004	0.002	0.007	0.00	0.02	0.02	0.01	0.02
Gemma 4 26B-A4B	0.005	0.001	0.004	0.002	0.004	0.04	0.02	0.03	0.02	0.02
Nemotron 3.5 Lightning 30B-A3B	0.002	0.003	0.004	0.001	0.002	0.00	0.02	0.01	0.01	0.03
Muse Glimmer 30B	0.003	0.003	0.002	0.002	0.002	0.01	0.01	0.02	0.02	0.04
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B-A3B	0.001	0.004	0.002	0.006	0.005	0.01	0.02	0.03	0.02	0.00
↪ SEA-LION v4.8 30B-A3B	0.001	0.001	0.003	0.002	0.001	0.02	0.03	0.02	0.00	0.02
↪ OpenThai 2.0 Legal 30B-A3B	0.002	0.005	0.004	0.001	0.000	0.01	0.01	0.02	0.01	0.01
Nemotron 3 Super 120B-A12B	0.001	0.003	0.002	0.001	0.004	0.02	0.03	0.03	0.01	0.03
↪ SEA-LION v4.8 120B-A12B	0.003	0.006	0.002	0.010	0.008	0.01	0.03	0.05	0.07	0.02
ThaiLLM 8B derivatives
Typhoon-S 8B	0.003	0.001	0.002	0.001	0.002	0.00	0.01	0.02	0.01	0.09
Pathumma 8B Think	0.002	0.010	0.003	0.001	0.006	0.03	0.06	0.03	0.01	0.02
THaLLE 0.2 8B	0.001	0.004	0.007	0.000	0.001	0.01	0.03	0.03	0.01	0.01
OpenThaiGPT 8B	0.000	0.001	0.001	0.000	0.000	0.02	0.01	0.01	0.01	0.01

Per-model contrasts, precision and recall

Table E.6 gives the per-model contrasts of Table 8 with their 95% intervals and the No-drift control of each model. Table E.7 reports the section-level precision and recall behind the Citation F1 of Table 7.

Model	Gold-cited, In-force minus Current: No drift	Gold-cited, In-force minus Current: Amended	Gold-cited, In-force minus Current: Repealed	RAG, In-force minus Current: No drift	RAG, In-force minus Current: Amended	RAG, In-force minus Current: Repealed
General
Gemma 4 31B	+0.14 [+0.04, +0.24]	+0.45 [+0.35, +0.56]	+0.98 [+0.83, +1.12]	−0.02 [−0.09, +0.05]	+0.29 [+0.21, +0.38]	+0.87 [+0.74, +0.99]
Gemma 4 26B-A4B	+0.08 [+0.01, +0.14]	+0.31 [+0.22, +0.39]	+0.97 [+0.83, +1.11]	+0.01 [−0.07, +0.08]	+0.03 [−0.05, +0.10]	+0.35 [+0.26, +0.44]
Nemotron 3.5 Lightning 30B-A3B	+0.02 [−0.05, +0.09]	+0.31 [+0.23, +0.39]	+0.86 [+0.72, +1.00]	−0.06 [−0.13, +0.01]	+0.14 [+0.08, +0.20]	+0.60 [+0.50, +0.70]
Muse Glimmer 30B	+0.02 [−0.05, +0.09]	+0.45 [+0.35, +0.55]	+1.35 [+1.19, +1.51]	−0.01 [−0.10, +0.07]	+0.32 [+0.23, +0.40]	+1.51 [+1.35, +1.67]
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B-A3B	+0.08 [+0.01, +0.14]	+0.22 [+0.16, +0.29]	+0.58 [+0.47, +0.68]	−0.01 [−0.06, +0.04]	+0.10 [+0.05, +0.15]	+0.38 [+0.31, +0.46]
↪ SEA-LION v4.8 30B-A3B	−0.08 [−0.15, +0.00]	+0.29 [+0.21, +0.38]	+0.88 [+0.74, +1.01]	−0.00 [−0.07, +0.07]	+0.18 [+0.12, +0.24]	+0.63 [+0.52, +0.73]
↪ OpenThai 2.0 Legal 30B-A3B	−0.03 [−0.12, +0.05]	+0.12 [+0.04, +0.19]	+0.27 [+0.17, +0.37]	−0.00 [−0.06, +0.05]	+0.03 [−0.02, +0.07]	−0.00 [−0.06, +0.06]
Nemotron 3 Super 120B-A12B	+0.03 [−0.05, +0.10]	+0.39 [+0.30, +0.49]	+1.10 [+0.95, +1.25]	+0.02 [−0.05, +0.10]	+0.29 [+0.21, +0.37]	+0.92 [+0.80, +1.05]
↪ SEA-LION v4.8 120B-A12B	−0.05 [−0.14, +0.04]	+0.42 [+0.32, +0.52]	+1.13 [+0.98, +1.29]	−0.12 [−0.22, −0.01]	+0.14 [+0.05, +0.23]	+0.70 [+0.57, +0.83]
ThaiLLM 8B derivatives
Typhoon-S 8B	+0.02 [−0.06, +0.09]	+0.09 [+0.02, +0.16]	+0.48 [+0.37, +0.59]	−0.05 [−0.11, +0.01]	−0.05 [−0.10, +0.00]	−0.05 [−0.11, +0.01]
Pathumma 8B Think	−0.11 [−0.19, −0.03]	−0.01 [−0.09, +0.07]	+0.45 [+0.34, +0.56]	+0.04 [−0.04, +0.11]	+0.01 [−0.05, +0.07]	+0.22 [+0.15, +0.30]
THaLLE 0.2 8B	+0.03 [−0.04, +0.09]	+0.19 [+0.12, +0.26]	+0.70 [+0.59, +0.81]	−0.00 [−0.05, +0.04]	+0.01 [−0.03, +0.05]	−0.00 [−0.05, +0.05]
OpenThaiGPT 8B	−0.06 [−0.14, +0.01]	+0.09 [+0.03, +0.15]	+0.48 [+0.38, +0.58]	−0.00 [−0.05, +0.05]	+0.00 [−0.04, +0.04]	−0.01 [−0.06, +0.05]
All models	+0.01 [−0.02, +0.03]	+0.25 [+0.21, +0.30]	+0.79 [+0.70, +0.88]	−0.02 [−0.04, +0.00]	+0.11 [+0.09, +0.14]	+0.47 [+0.42, +0.52]
minus No drift	–	+0.25 [+0.20, +0.30]	+0.78 [+0.69, +0.87]	–	+0.13 [+0.10, +0.16]	+0.49 [+0.43, +0.54]

Model	Param.	GC Cur.	GC InF.	RAG Cur.	RAG InF.
Citation precision
General
Gemma 4 31B	0.411	0.761	0.881	0.551	0.647
Gemma 4 26B-A4B	0.375	0.791	0.904	0.319	0.363
Nemotron 3.5 Lightning 30B-A3B	0.035	0.763	0.871	0.351	0.430
Muse Glimmer 30B	0.206	0.791	0.898	0.458	0.548
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B-A3B	0.023	0.746	0.851	0.232	0.302
↪ SEA-LION v4.8 30B-A3B	0.046	0.764	0.869	0.338	0.419
↪ OpenThai 2.0 Legal 30B-A3B	0.090	0.510	0.546	0.005	0.006
Nemotron 3 Super 120B-A12B	0.108	0.786	0.895	0.421	0.522
↪ SEA-LION v4.8 120B-A12B	0.099	0.775	0.886	0.339	0.403
ThaiLLM 8B derivatives
Typhoon-S 8B	0.132	0.758	0.836	0.017	0.050
Pathumma 8B Think	0.189	0.754	0.852	0.169	0.219
THaLLE 0.2 8B	0.029	0.755	0.862	0.002	0.003
OpenThaiGPT 8B	0.026	0.677	0.779	0.000	0.000
Citation recall
General
Gemma 4 31B	0.268	0.601	0.739	0.396	0.470
Gemma 4 26B-A4B	0.239	0.621	0.750	0.196	0.226
Nemotron 3.5 Lightning 30B-A3B	0.027	0.629	0.751	0.280	0.345
Muse Glimmer 30B	0.219	0.748	0.902	0.517	0.630
Nemotron 3 bases and their adaptations
Nemotron 3 Nano 30B-A3B	0.019	0.633	0.753	0.213	0.266
↪ SEA-LION v4.8 30B-A3B	0.038	0.611	0.724	0.282	0.342
↪ OpenThai 2.0 Legal 30B-A3B	0.071	0.408	0.461	0.004	0.004
Nemotron 3 Super 120B-A12B	0.108	0.672	0.804	0.356	0.441
↪ SEA-LION v4.8 120B-A12B	0.100	0.652	0.785	0.321	0.379
ThaiLLM 8B derivatives
Typhoon-S 8B	0.085	0.555	0.643	0.011	0.032
Pathumma 8B Think	0.126	0.536	0.608	0.110	0.144
THaLLE 0.2 8B	0.020	0.565	0.664	0.001	0.002
OpenThaiGPT 8B	0.019	0.545	0.647	0.000	0.000

Rule change labels. For each changed gold section of an Amended item, we manually annotated whether the amendment changed the rule the question turns on or only the wording. Table 9 in §6.4 gives the loss by group.