WorksComputer Science
GPBam: A Retrieval-Aware German Public Law Exam Benchmark
Abstract
Legal reasoning in civil-law systems is not rule-lookup: the German state examination asks for a structured opinion (Gutachten) that names the governing norms, applies them to the facts, and argues to a conclusion. We introduce GPBam (German Public Law Bar-style Exam), 81 public-law examination cases with reference solutions, paired with a recitation task over 200 statutory provisions. Drawn from state examination preparation, the cases are broad and rarely repeat, and are not publicly available - we find no evidence of exact-solution memorisation in the models we evaluate, and keeping them private limits contamination. Because each reference solution's cited norms form a retrieval gold set over the very instances the essays are graded on, GPBam scores retrieval and long-form generation jointly, which makes it a benchmark for legal retrieval as well as for legal writing and lets us ask how much retrieval quality a legal opinion actually needs. Across 32 open-weight and proprietary models evaluated without retrieval, scores span 4.81-76.42 out of 100. Supplying the law helps modestly: across seven models and eight retrieval configurations, retrieval performance varies nearly threefold and buys at most 4.1 essay points, and nothing significant for three of the seven, while an oracle with 91% gold-norm recall improves scores by 2-6 points. Essay quality strongly tracks prior knowledge: verbatim statute recall, measured in a separate task, correlates at over 32 models.
Introduction
Many established legal benchmarks use classification, multiple-choice, or short-answer formats [1, 2, 3]. This permits standardized automatic scoring, but does not represent how case-based tasks in German legal education and the First State Examination assesses legal reasoning. A typical case task requires a complete legal opinion (Gutachten): identify the issues and governing norms, state the requirements of each norm, apply them to the facts, and reach a conclusion. The same structure recurs over the issues and sub-issues of the case. A correct final conclusion can therefore conceal missing or incorrect intermediate analysis, which label-based benchmarks do not measure directly [1, 2, 3]. Evaluating the full reasoning document is necessary for this examination setting [4].
Several recent benchmarks include open-ended legal tasks. GreekBarBench asks questions from five areas of the Greek bar examination, supplies chapter-level statutory context, and requires citations to both articles and case facts [5]. LEXam contains 2,841 open-ended questions from law examinations and accompanies them with guidance on the expected reasoning approach [6]. For German law, BenGER evaluates complete exam-style case solutions alongside short doctrinal tasks, but disables retrieval and tool-use endpoints for the evaluated systems [4]. Retrieval-focused benchmarks address a different stage: LegalBench-RAG measures whether a system locates the relevant spans, not whether those spans improve a complete legal opinion [7]. These benchmarks therefore emphasize different stages of legal problem solving. Their complementary designs motivate evaluating statutory recall, retrieval, and downstream use within the same document-level task.
We introduce GPBam (German Public Law Bar-style Exam), a benchmark that scores statute retrieval and long-form legal writing on the same 81 state-exam-style public-law cases, drawn from the University of Passau’s State Examination Preparation Course in Public Law. Each case pairs a fact pattern (Sachverhalt) with an extensively reasoned reference solution. The task is document-level: a model receives the case and writes a complete German legal opinion (Gutachten), including a table of contents and an assessment of every question raised. The reference solutions average 7,080 words, compared with 924 for the case facts, and contain substantially more statutory references. This asymmetry is part of the task: the model must determine much of the governing law rather than extract it from the question. The cases are also broad and rarely repeat: the topics their authors label are mostly unique to a single case, and one case raises a median of eight of them across a median of twelve law books. Keeping the cases and reference solutions non-public also reduces the risk that they enter model-training corpora and contaminate the evaluation [8]. A direct text-overlap analysis finds no evidence of exact-solution memorisation of the reference solutions (§ 3.2).
GPBam evaluates statutory access in two complementary ways. First, the Essay Writing task is run without retrieval and with statutory context retrieved from federal and Bavarian statutes. We compare retrieval strategies and context budgets, and include an oracle condition that supplies the statutes cited by the reference solution, separating retrieval failure from failure to use relevant provisions. Second, Article Recitation names a provision and asks the model to reproduce its text. The task covers 200 provisions: 100 frequently cited by the GPBam reference solutions and 100 frequently cited by German courts. It removes issue spotting and subsumption, providing a separate measure of closed-book statutory-text recall. Essays are scored against the reference solution by three open-weight judges; we take their per-case median and separately measure the overlap of statutory citations.
Across 32 models evaluated without retrieval, essay scores span 4.81–76.42 points on the 0–100 scale. Calibration controls anchor the scale empirically: the reference solutions score 100, whereas wrong-case solutions and fact-pattern echoes score 0. Essay score correlates with Article Recitation on the GPBam provisions at , while correlation with citation overlap, measured against the reference solution, is . Supplying statutes via retrieval system has a limited effect in the present setup. Across the tested retrieval configurations, gold-norm recall rises nearly threefold, from 3.50% to 9.61% at and from 8.52% to 23.92% at , while the largest non-oracle improvement over no retrieval is 4.1 essay score points, and for three out of seven generators no configuration improves significantly on no retrieval at all. The oracle, which supplies the statutes the reference solution cites and reaches 91.31% recall, raises essay scores by 2–6 points and does so for every one of the seven evaluated models. These results do not show that statutory knowledge causes higher essay scores. They do show that supplying statutory text under the tested conditions does not close the observed performance gap, while closed-book statutory recall covaries strongly with essay quality. Our contributions are the following:
We introduce GPBam, 81 German state-exam-style public-law cases with reference solutions, formulated as complete legal-opinion writing rather than classification or short-answer prediction, and held out rather than published: we find no evidence of exact-solution memorisation of the reference solutions in any evaluated model, and keeping the material private limits contamination.
Because each reference solution’s cited norms form a retrieval gold set over the same 81 instances the essays are scored on, GPBam measures retrieval and long-form generation jointly. This is what lets us ask how much retrieval quality a legal essay actually needs, and it makes the benchmark usable for legal retrieval independently of the essay task.
We pair the essay task with Article Recitation over 200 provisions, making statutory recall observable separately from issue spotting, subsumption, and legal writing.
We report a 32-model no-retrieval study using a three-judge ensemble, citation overlap, together with an analysis of inter-judge agreement.
We evaluate statute retrieval for complete opinions across seven models, with controlled changes to retrieval method, corpus, and context budget, and an oracle upper bound using the reference solution’s cited statutes.
Findings show that supplying statutes helps modestly: a nearly threefold spread in gold-norm recall is worth at most 4.1 points of essay score, with no significant gain for three of the seven generators, while an oracle at 91.31% recall raises every generator by 2–6 points.
Related Work
From Legal Understanding to Open-Ended Examination Tasks
Many early legal NLP benchmarks framed the domain as classification, multiple-choice, or other short/task-specific outputs. LexGLUE groups document and clause classification tasks [1], LegalBench assembles hand-built tasks around common-law reasoning [2], and LawBench organises Chinese legal tasks into memorisation, understanding and application [3]. Their evaluations are often outcome-focused, so a model can be credited for a correct answer reached by faulty reasoning. Prompting a general LLM on legal argument mining task did not by itself dissolve these tasks [9]. Nearer to the present setting, open-weight LLMs have been put to a range of German legal-education tasks, where capability falls away as the material turns doctrinal: labelling the components of Gutachtenstil, and working over whole legal opinions rather than short spans. On those two, neither a longer context window nor prompt engineering carried the models past a bag-of-words baseline [10].
Recent benchmarks increasingly include open-ended examination tasks. Outside law, SciEx grades free-text exam answers with both human experts and automatic graders [11]. Within law, OAB-BENCH draws 105 questions from the Brazilian bar examination and applies the official grading guidelines through an LLM judge [12]. GreekBarBench covers five areas of the Greek bar and requires citations to statutory articles and to case facts, scored on three dimensions [5]. LEXam derives 7,537 questions from 340 Swiss law exams, 2,841 of them open-ended. Each open question is paired with a reference answer and reasoning approach expected [6]. PLawBench and JurisBench extend the setting to professional practice in Chinese law, the former with expert-written rubric items across 13 scenarios [13], the latter through a multi-stage litigation workflow in which errors cascade [14]. Surveys of the area report the same trajectory [15].
Two properties separate this line of work from GPBam. The first is the unit of evaluation. LEXam, PLawBench and JurisBench score answers to individual questions or workflow stages, and GreekBarBench scores one answer to a case-specific question against span-based rubrics. GPBam scores one complete Gutachten per case: a table of contents followed by a full assessment that answers every question raised in the Sachverhalt, with no study-imposed output-token cap. The reference solutions are correspondingly long, at 7,080 words on average against 924 for the facts - roughly twenty-nine times the 247-word mean reference answer of LEXam [6], whose open questions are individually scorable sub-problems rather than whole opinions. GreekBarBench scores one answer per question against expert-annotated spans for facts, cited articles and analysis [5], so its unit is a passage within an answer. The biggest difference is jurisdiction. Common-law and Chinese material dominates, while civil-law systems, in which the analysis is anchored in codified statutes rather than precedent, are less well covered [6]. For German law, employment-contract clauses have been classified as valid, unfair or void [16], which isolates subsumption at clause level.
Among the benchmarks reviewed, BenGER is the closest work to ours. It evaluates 596 exam-style cases spanning civil, criminal, and public law and multiple stages of legal education, together with 531 shorter doctrinal items. Its judge scores ten weighted dimensions and is checked against three independent blind human gradings on a controlled subset [4]. Retrieval and tool-use endpoints are switched off throughout, so its systems answer from parametric knowledge alone. GPBam is narrower - 81 state-exam-style public-law cases from a single state-examination preparation course - and uses that depth for two things BenGER does not measure: what happens when the statutes are supplied via retrieval, and how much of the statutory text models retain without external knowledge analogues to closed-book performance.
Legal Context and LLM-Based Evaluation
Supplying legal context. Retrieval augmentation conditions generation on passages fetched at inference time [17], and the pipelines now in use extend that basic loop on both sides of the retriever - rewriting or diversifying the query before search [18, 19], fusing a dense and a lexical channel [20, 21], and re-ranking the pool before it reaches the generator - a family the surveys group as advanced and modular RAG [22]. Our retrieval ladder walks exactly these components over statutory law. The central question, however, is not only whether these techniques retrieve more relevant legal material, but whether those retrieval gains improve the model’s final answers. A relationship that legal RAG evaluations have only recently begun to examine directly.
Much of the existing legal literature evaluates retrieval separately from generation. LegalBench-RAG, for example, scores the retrieval step alone [7]. Pairing a statutory corpus with a labour-law benchmark puts retrieval-augmented accuracy well below the claims made for it [23]. An evaluation of three commercial legal research tools built on retrieval measures hallucination rates between 17% and 33% [24]. Varying the German legal context supplied to the model shows that full-text sources help only moderately, while distilled examination guidelines help more [16]. Context length can reduce performance even under perfect retrieval and when irrelevant tokens are masked [25]. Where retrieval and downstream answering are evaluated jointly, better access to relevant legal context does not necessarily yield better answers. In Bar Exam QA, which uses short, stylized fact patterns with four answer choices, and Housing Statute QA, which uses binary yes/no questions, structured legal-reasoning query expansion improves recall for several retrievers, but these gains do not consistently translate into higher answer accuracy [26].
These studies therefore either evaluate retrieval in isolation or measure its downstream effect on question-answering or classification tasks. None of the reviewed studies examines whether retrieved statutory text improves a complete examination essay in a civil-law system. GPBam runs the same 81 cases with and without retrieval and adds an oracle condition that supplies the statutes cited by the reference solution, which separates “retrieval cannot find the norms” from “the norms do not help”.
Probing statutory knowledge directly. A parallel line measures whether models hold statutory text at all. LawBench includes a memorisation level [3]; CLaw segments 306 Chinese national statutes to subparagraph level and reports that models struggle to reproduce provisions faithfully [27]; LeKUBE evaluates statute recitation and recall under knowledge updates [28]. GPBam adopts the same idea as a separate closed-book task over 200 provisions, and then correlates it with essay quality across models. This is what lets us ask whether the bottleneck on a Gutachten is knowing the law or applying it.
LLM judges. Judging long-form legal answers at benchmark scale is constrained by the cost of expert annotation: it is slow, requires specialised lawyers, and does not scale to the number of models and conditions a benchmark must cover [5, 29]. The expert grades themselves are also noisy. Where the same law-exam answers have been regraded by many markers, the harshest and the most generous mark on one answer differ by 6.47 points on average, on a scale that runs from 0 to 18 [30, 4]. This variability cautions against treating one marker’s grade as definitive ground truth. Lexical overlap with a reference solution is no substitute [6], because legally equivalent analyses differ in wording. Recent benchmarks therefore grade with LLM judges [31, 5, 6, 4]. Reported failure modes include self-preference, where an evaluator favours its own generations [32], and sensitivity to prompt [33]. Benchmarks respond by validating the judge: GreekBarBench builds a meta-evaluation set and finds that span-based rubrics improve alignment [5]; LEXam uses a judge ensemble with expert validation [6]; BenGER benchmarks its judge against a pool of human markers, obtaining under Pearson and [4]; OAB-BENCH reports strong correlation between a frontier judge and human examiners [12]. A parallel line changes the unit of assessment instead of validating a holistic one: LeMAJ splits a legal answer into small, individually checkable units it calls legal data points, and scores those without consulting a reference answer [29], and DeCE separates precision from recall against criteria extracted from the gold answer, reporting a correlation with expert judgement of on a legal question-answering task with citation grounding, against for pointwise scoring [34]. For German exam scripts in particular, a comparison of 27 models against examiner marks under quadratic weighted kappa finds that pooling several models tracks the examiners more closely than any single model does, by as much as 0.15 kappa [35]. Within GradeLegal’s two datasets, agreement was far higher on public-law scripts than on criminal-law ones, so the domain GPBam covers is the one where automatic marking has so far proved most tractable. GPBam scores each essay against the reference solution with three open-weight judges and takes the per-case median. We report inter-judge agreement at essay and model level and report a fitted adjustment for judge-generator self-preference.
The GPBam Benchmark
Examination Setting
German public-law examination cases present a fact pattern and one or more questions whose solution often requires procedural and substantive legal analysis. The expected solution is not a list of rules but a structured legal opinion (Gutachten). Following the legal opinion style (Gutachtenstil), the reasoning typically proceeds through four components: an Obersatz identifying the legal issue to be examined, a Definition specifying the relevant legal requirements, the Subsumtion of the facts under those requirements, and an Ergebnis stating the resulting conclusion. This structure recurs across relevant issues and sub-issues in the solution, making the appropriate unit of evaluation an entire reasoning document rather than an isolated answer span.
The source documents of GPBam belong to the Examenskurs im Öffentlichen Recht (State Examination Preparation Course in Public Law) of the University of Passau’s Institute for Legal Didactics. They are practice examination written at the level and in the format of the German state examination. We therefore use “state-exam style” rather than treating the documents as official state-examination papers. Each case fact pattern is accompanied by an extensively reasoned model solution that serves as the reference solution for the respective case.
Dataset Construction and Characteristics
GPBam comprises 81 German public-law examination cases sourced from Examenskurs im Öffentlichen Recht (State Examination Preparation Course in Public Law) of the University of Passau’s Institute for Legal Didactics. Each case pairs a Sachverhalt, a factual scenario followed by the questions the examinee must answer, with a Lösung, a reference solution that develops the expected legal analysis in full.
Length. Table 1 summarizes the corpus. The two components are strongly asymmetric: reference solutions are around longer than case facts in words (7080 vs. 924) and in tokens (13855 vs. 1657). The task is therefore long-form generation, rather than the short-answer and multiple-choice formats of many widely used legal benchmarks [2, 1], placing it alongside with recent free-text legal benchmarks [6, 5].
| Component | #Docs | Words | Tokens | §-refs | Books | Topics |
| Case facts (Sachverhalt) | 81 | – | ||||
| Reference solution (Lösung) | 81 | |||||
| Range (min–max) | ||||||
| Case facts | 272–1904 | 469–3103 | 0–13 | 0–10 | – | |
| Reference solution | 2989–10917 | 5788–22721 | 20–97 | 5–20 | 3–15 |
Table 1. Table 1. Statistics of the GPBam corpus. 81 case–solution pairs, 1.3M tokens. Values are the mean over instances with the sample standard deviation as a subscript, rounded to integers. §-refs is the number of unique (law book, § or Art.) statutory references per document, counting both §- and Art.-form citations, Books the number of distinct law books those references fall in, and Topics the number of entries in the Themen des Falles list the course authors append to each solution (present in 80 of the 81; case facts carry none).
Language: German · Domain: Public law (Öffentliches Recht)
Source: State Examination Preparation Course in Public Law, University of Passau
Aligned case–solution pairs: 81 · Corpus size: 1.3M tokens
Topics: 520 distinct; 83% occur in one case, none in more than 13
Legal density. Length alone understates the difficulty. Case facts cite almost no law ( unique statutory references), whereas the corresponding reference solutions cite . This gap characterises the core challenge: models must identify the governing law themselves rather than apply law supplied to them. It motivates both experimental conditions: supplying the relevant statutes through retrieval (§ 3.5), and probing statutory knowledge directly through article recitation (§ 3.3.2).
Topical coverage. The cases are written as preparation for the state examination, therefore they are set to span the examinable material rather than to focus on one doctrine. Each reference solution closes with a list of the topics its case examines, compiled by the course authors; 80 of the 81 solutions carry one. These lists name 520 distinct topics, of which 83% appear in exactly one case, and the most frequent of them, the Anfechtungsklage (action for annulment), in 13. A single case raises a median of eight topics and cites a median of twelve distinct law books, so breadth is a property of the individual case and not only of the collection.
Contamination analysis. A case whose solution a model had already read would be answered from memory rather than from reasoning. We probe this on the essays directly: for each, we measure the longest exact word-span it shares with its own reference solution, and the same quantity against four other control cases’ solutions, which share the genre, the Gutachtenstil conventions and much statutory wording but nothing case-specific. Memorisation of a Lösung would show as spans of dozens to hundreds of words above that control; it does not appear. The median longest match is 7 words against a case’s own solution and 5 against the controls, an excess of two; the longest raw match is 38 words and the largest excess over control 34, with only five essays at twenty or above. Per-model median excess ranges from one to three words across all 32 generators, including the two highest-scoring systems, whose longer raw matches are offset by equally longer control matches. All five are quoted source material rather than recalled analysis - the question posed in the Sachverhalt, passages of statutes, and a court passage that essay and solution both cite. The test only bounds the strongest exposure, to the exact reference solutions, but does not test for paraphrases or the Sachverhalt alone.
Benchmark Tasks
Figure 1 summarizes the GPBam benchmark tasks, which separates integrated legal writing from statutory recall and consists of two distinct but related tasks: (1) full legal essay writing and (2) article recitation. Essay writing uses 81 case/solution pairs of § 3.2 and is run in two context conditions, without and with retrieved context; besides that context, the two conditions also differ in the framing of the instruction (§ 3.3.1). Article recitation uses two fixed sets of 100 provisions, involves no fact pattern, and is scored against official statutory text. The first task measures what a model produces when the applicable law has to be identified and applied. The second measures how accurately it reproduces statutory wording when the provision is named for it. Table 2 provides further details.
Figure 1. The two GPBam tasks and their data. Essay writing gives a model one Sachverhalt and asks for a complete Gutachten, with and without retrieved context; Article Recitation names a provision and asks for its text, with no case and no retrieval. The reference solution is only used as the judges’ target, and the norms it cites are both the retrieval gold set and the GPBam laws provision set. The excerpt illustrates one case.
| Full Legal Essay Writing | Article Recitation | |
| Ability probed | Issue spotting, rule selection, subsumption, structured argument | Recall of statutory wording |
| Input | Case facts (Sachverhalt) and the questions posed | Provision identifier only (e.g. VwGO § 154) |
| Context conditions | No-RAG (facts only) and Law-RAG (facts + retrieved context: ten passages by default, fifty in the arms, headnote text in the LS-text arm, and the full text of all provisions cited by the solution in the oracle; Table 8) | One condition; no retrieval, no case facts |
| Output | Complete German Gutachten: table of contents and full legal assessment; no length limit | Text of the named provision |
| Sets × items | 1 × 81 aligned case-solution pairs | 2 × 100 provisions |
| Item selection | Examenskurs im Öffentlichen Recht, University of Passau | GPBam laws: 100 most-cited provisions (§ and Art.) of the 81 solutions; Most-cited laws: 100 most cited per the Open Legal Data citation network from [36] |
| Evaluation target | Reference solution (Lösung) | Official statutory text (federal-law snapshot 67e0567; graph records) |
| Primary metric | Median of a three-judge LLM ensemble, rescaled to 0–100 | ROUGE-L F1 against the statutory text, without stemming |
| Secondary signal | Legal Reference Similarity: Jaccard overlap of unique (law book, §/Art.) pairs | - |
| Prompts per model | 81 (81 cases, No-RAG); 810 for the seven models that also carry the retrieval ladder (81 cases × 10 settings: No-RAG, eight retrieval arms and the oracle; Table 8) | 200 (2 sets × 100 provisions) |
Table 2. Table 2. GPBam tasks and data. Essay writing measures integrated legal analysis under two context conditions; Article Recitation isolates verbatim statutory knowledge in a single closed-book condition. Word counts are means with sample standard deviation (Table 1); the reference solution is never part of the model input.
Full Legal Essay Writing
Task. The input is the case facts to the model and the expected output is the complete legal opinion in German that answers every question raised in the case facts. The reference solution is withheld during generation and is never part of the models’ input, it serves as the evaluation target used by the judge ensemble (§ 3.7, § 3.8).
Instruction. All 81 cases are inserted into a German instruction template with two slots, “context” and “case text”. The template describes the input as a state-examination-level case to be answered within four hours and asks for a table of contents followed by a complete legal assessment. It requires citations to identify the paragraph (Abs.), sentence (S.), number (Nr.), or letter (Buchst.) of a provision where applicable, and instructs the model to avoid unspecific references. The prompt specifies no output-length limit. No-RAG and Law-RAG use variants of the same template, with the differences described under Conditions below. See the complete prompts in Appendix § 10.§ 2.1 and A.2.2.
Conditions. In No-RAG the context slot is empty, so the case facts are the only case-specific input. In Law-RAG the slot holds a <context> block containing the retrieved provisions, each wrapped in a <norm> element that carries its citation, its jurisdiction and its heading as attributes, so that the model can cite the provision rather than its position in the list.
The case text is identical across the two conditions, but the instruction is not. The Law-RAG template adds guidance on how to read the retrieved material: that the selection is incomplete and partly irrelevant, that state law may be missing from it even where it governs, that a provision’s absence from the block is not evidence that it does not apply, and that the essay should cite the applicable norms whether or not their text is printed. It also drops three No-RAG-only sentences, including an instruction to cite exclusively from the considerations the model has identified. The two conditions therefore differ in access to statutory text and in the instruction that frames it, which is a confound to keep in view when reading any No-RAG–Law-RAG difference.
Article Recitation
Task. Article Recitation asks a model to reproduce a single statutory provision from its identifier alone. The input is a law book and a number, as stored in the respective set. The expected output is the text of that provision. The German prompt requests the exact wording of the current version, without warnings, an introduction, or follow-up questions, and the most detailed available summary if exact reproduction is not possible. The generated text is compared with the stored official text of the provision using ROUGE-L F1 (§ 3.8). No case facts, retrieval, or reference solution are involved in this task. See the complete prompt in Section A.2.3.
Provision sets. A provision is named the way a German lawyer cites one: by the law book it sits in – the statute, such as the Verwaltungsgerichtsordnung (VwGO) – and its number within that statute. Each of the two sets holds 100 such provisions.
The GPBam laws set comes out of the benchmark itself. We extract the statutory citations of the 81 reference solutions in both their §- and Art.-form and order the provisions they cite by how often they are cited, which brings the material the examiners lean on to the top. Scoring an answer requires the provision’s official text, so we look each candidate up in the pinned gesetze-im-internet snapshot (§ 3.3.3) and keep the 100 most-cited for which that lookup returns a text. They come from 19 law books, led by the Verwaltungsgerichtsordnung (VwGO), the Grundgesetz (GG) and the Bürgerliches Gesetzbuch (BGB).
The Most-cited laws set comes from outside the benchmark, and reflects what German courts cite rather than what these examinations ask: the 100 provisions appearing in the largest number of decisions in the Open Legal Data3 citation network [36]. These come from nine law books, most of them the Zivilprozessordnung (ZPO), the Verwaltungsgerichtsordnung (VwGO) and the Bürgerliches Gesetzbuch (BGB).
Scope. Article recitation removes issue spotting and subsumption by construction. It probes access to statutory wording; not legal analysis, and does not subLanguage: German · Domain: Public law (Öffentliches Recht)
Population: the same 32 models receive every No-RAG essay prompt and every Article Recitation prompt (§ 3.4; recitation exceptions in § 3.3.2); Law-RAG covers seven models (Table 8)
| no RAG | RRF | cite-rerank | cite-only | cite+RRF | RRF | cite+RRF | LS text | cite+LS+RRF | oracle | |
| Retrieval quality (%), against RRF | ||||||||||
| gold-norm recall@k | – | 3.50 | 5.33 | 9.04 | 9.61 | 8.52 | 16.49 | 6.33 | 23.92 | 91.31 |
| state-law recall@k | – | 4.31 | 6.15 | 3.26 | 4.31 | 11.62 | 12.98 | 5.17 | 19.07 | 94.90 |
| cases with a hit@k | – | 80.2 | 95.1 | 96.3 | 98.8 | 97.5 | 100.0 | 90.1 | 100.0 | 100.0 |
| Essay score (0–100), against no RAG | ||||||||||
| DeepSeek-V4-Flash | 43.77 | 45.06 | 43.83 | 44.44 | 42.78 | 44.81 | 43.15 | 42.96 | 47.90 | 47.90 |
| Gemma-4-31B | 32.90 | 33.95 | 33.95 | 34.94 | 36.30 | 33.64 | 35.19 | 34.94 | 32.22 | 38.27 |
| Qwen3-Next-80B-A3B | 30.62 | 28.52 | 29.69 | 32.65 | 31.60 | 29.51 | 30.62 | 30.99 | 31.23 | 32.84 |
| Qwen3.6-35B-A3B | 30.62 | 31.48 | 32.72 | 32.96 | 31.05 | 32.59 | 33.09 | 31.98 | 33.09 | 35.31 |
| gpt-oss-120B | 29.75 | 29.26 | 28.89 | 32.35 | 32.35 | 28.46 | 30.49 | 30.25 | 30.62 | 35.62 |
| Magistral-Small | 23.33 | 21.48 | 22.04 | 22.47 | 22.47 | 22.59 | 22.84 | 21.98 | 22.72 | 25.93 |
| Mistral-Small-3.2 | 21.98 | 19.51 | 21.23 | 21.98 | 21.48 | 22.59 | 20.99 | 22.90 | 22.90 | 25.19 |
| Gold norms cited by the essay (%), against no RAG | ||||||||||
| DeepSeek-V4-Flash | 25.86 | 26.36 | 27.06 | 25.93 | 25.44 | 27.48 | 28.24 | 26.82 | 29.11 | 41.36 |
| Gemma-4-31B | 11.15 | 11.85 | 12.32 | 12.16 | 12.27 | 13.16 | 13.96 | 12.65 | 14.59 | 30.57 |
| Qwen3-Next-80B-A3B | 14.64 | 16.33 | 16.06 | 16.57 | 16.72 | 17.50 | 17.76 | 16.44 | 20.86 | 40.59 |
| Qwen3.6-35B-A3B | 17.05 | 18.28 | 19.12 | 18.02 | 18.40 | 20.34 | 19.65 | 18.79 | 20.97 | 36.57 |
| gpt-oss-120B | 15.04 | 14.90 | 15.27 | 15.25 | 16.48 | 16.11 | 16.53 | 15.85 | 19.49 | 35.61 |
| Magistral-Small | 11.43 | 10.70 | 10.74 | 10.91 | 11.37 | 10.67 | 11.83 | 11.59 | 12.64 | 19.78 |
| Mistral-Small-3.2 | 10.71 | 8.96 | 9.28 | 9.89 | 9.32 | 9.83 | 10.31 | 10.56 | 11.70 | 19.26 |
Table 8. RAG approaches for GPBam Essay Writing: RRF for dense-hybrid rank-fusion; rerank reorders that pool with Qwen3-Reranker-4B; cite-only resolves norms a rewriter suggests; cite+RRF fills the remaining budget behind them; cite+LS+RRF also resolves the Normenketten of the retrieved Leitsätze, LS text provides their text. The oracle supplies the provisions the reference solution cites, a coverage ceiling, not a RAG system. The last block notes the share of the solution’s norms that the generated essay cites too. Shading is the difference from the block’s own baseline, blue is better; bold is a paired difference beyond twice its standard error, uncorrected across cells. Essay scores are 3-judge medians.
stitute for essay evaluation. ROUGE-L credits reproduction of the statutory text, therefore paraphrases and correct but abridged text receive partial credit at best.
Every model answers the same 200 prompts4, which puts the two sets on one scale (Table 4) and allows either to be related to essay quality at model level (§ 4.2).
Table 4. Model performance without retrieval, for both GPBam tasks. Median is the mean of the per-case median over three judges, see per-judge columns, the subscript is its standard error, . Deb. removes self-preference of a judge related to the model: M same model, V same vendor, blank if none. Cells are colored (red below and blue above, saturating at ) for the deviation from the median of that row; best in class, second best and best open-weights results are marked. Ref. is the Reference Similarity between references cited in the essay and in the solution. GP and Cited are ROUGE-L F1 for reciting the most frequent provisions in GPBam solutions and German court decisions respectively. ‡Scored on the 73 GPBam laws provisions only, which may differ slightly from a full-set score; see § 3.3.2. AA is General capability via Artificial Analysis index, where available. † run through an agent harness, not API, so it carries a harness system prompt where every other row has none; transcripts show no access to anything but the prompt file.
| Model | Median | Deb. | GPT-oss | Qwen3.6 | DS-V4 | Ref. | GP | Cited | AA |
| Claude-opus-5† | 76.42±2.35 | 70.12 | 90.49 | 74.32 | 39.63 | 74.54 | 89.97 | 63 | |
| Gemini-3.7-flash | 67.10±1.92 | 62.35 | 79.01 | 61.17 | 36.53 | 74.92 | 69.83 | 56 | |
| DeepSeek-V4-Flash | 43.77±1.58 | 41.05ᴹ | 41.54 | 53.83 | 37.78 | 22.21 | 50.82 | 39.06 | 52 |
| Qwen3.5-397b | 41.48±1.49 | 40.86ⱽ | 41.30 | 50.37 | 34.32 | 20.69 | 40.74 | 34.38 | 34 |
| Mistral-L | 37.84±1.37 | 38.70 | 46.42 | 31.11 | 19.86 | 56.46 | 50.54 | 16 | |
| Mistral-M-3.5-128B | 37.41±1.39 | 38.77 | 44.81 | 29.88 | 18.47 | 36.09 | 28.84 | 30 | |
| DeepSeek-v3.2 | 36.79±1.19 | 36.67ⱽ | 37.04 | 44.57 | 29.63 | 17.29 | 33.84 | 27.29 | 33 |
| Gemini-2.5-flash-lite | 35.19±1.19 | 33.83 | 41.98 | 30.00 | 14.80 | 21.09 | 20.28 | 11 | |
| GPT-5-mini | 34.81±1.17 | 34.81ⱽ | 35.80 | 40.62 | 29.01 | 12.33 | 21.74 | 17.41 | 26 |
| Claude-haiku-4.5 | 34.81±1.08 | 36.30 | 39.51 | 26.60 | 13.59 | 25.44 | 16.07 | 30 | |
| DeepSeek-r1 | 33.21±1.19 | 32.96ⱽ | 33.46 | 39.75 | 28.40 | 17.21 | 27.25 | 22.42 | 20 |
| Qwen3.5-122b | 32.96±1.18 | 32.96ⱽ | 33.58 | 40.99 | 27.12 | 15.79 | 31.44 | 25.50 | 33 |
| Gemma-4-31B-I | 32.90±1.07 | 32.62 | 41.60 | 29.63 | 10.72 | 18.53 | 14.30 | 30 | |
| Qwen3-235b-T | 31.11±1.10 | 31.11ⱽ | 32.72 | 36.54 | 24.20 | 12.09 | 20.05 | 16.15 | 20 |
| Qwen3.6-35B-A3B | 30.62±1.00 | 28.27ᴹ | 30.37 | 36.91 | 23.58 | 14.56 | 23.56 | 20.27 | 32 |
| Qwen3-next-80b | 30.62±0.94 | 30.62ⱽ | 33.95 | 34.81 | 23.83 | 12.57 | 14.27 | 13.60 | 14 |
| GPT-oss-120b | 29.75±0.99 | 28.15ᴹ | 36.30 | 30.74 | 21.48 | 12.19 | 18.36 | 13.75 | 24 |
| DeepSeek-chat-v3 | 26.36±0.82 | 26.36ⱽ | 25.86 | 31.36 | 21.73 | 13.64 | 25.00 | 24.06 | 15 |
| Soofi-S-Isar | 26.11±0.91 | 26.48 | 33.09 | 22.35 | 14.62 | 26.16 | 24.92 | – | |
| Llama-4-maverick | 24.44±0.68 | 25.56 | 27.28 | 22.22 | 10.96 | 23.11 | 36.13 | 14 | |
| Magistral-S | 23.33±0.70 | 23.09 | 26.79 | 20.49 | 10.33 | 14.06 | 14.85 | 11 | |
| Mistral-S-3.2-24B-I | 21.98±0.62 | 21.85 | 24.57 | 20.00 | 9.79 | 15.49 | 15.76 | 11 | |
| GPT-5-nano | 21.48±0.66 | 21.48ⱽ | 22.28 | 23.09 | 19.01 | 10.08 | 14.06 | 11.77 | 20 |
| GPT-4o-mini | 19.51±0.66 | 19.51ⱽ | 18.27 | 22.96 | 17.53 | 6.75 | 16.53 | 16.30 | 7 |
| Open_stuerllm | 17.78±0.63 | 16.91 | 20.37 | 16.17 | 7.27 | 14.01‡ | 17.16 | – | |
| Llama-3.3-70b-I | 16.79±0.63 | 15.56 | 17.53 | 15.56 | 6.22 | 14.52 | 17.12 | 9 | |
| Qwen3-32b | 13.21±0.82 | 13.21ⱽ | 13.70 | 12.35 | 12.84 | 7.62 | 10.26 | 9.95 | 11 |
| EuroLLM-22B-I | 9.26±0.76 | 7.90 | 10.74 | 9.38 | 4.17 | 9.86‡ | 12.11 | – | |
| Ministral-3-14B-R | 8.77±0.75 | 9.26 | 6.42 | 10.99 | 8.66 | 9.28‡ | 12.30 | 10 | |
| Granite-4.1-3b | 7.53±0.57 | 5.80 | 7.28 | 9.14 | 3.83 | 7.80 | 9.73 | 4 | |
| Phi-4-mini-I | 5.43±0.58 | 4.44 | 5.56 | 7.65 | 3.84 | 7.68 | 4.86 | 6 | |
| Llama-3.1-8b-I | 4.81±0.68 | 3.46 | 5.31 | 7.65 | 3.07 | 7.42 | 10.20 | 7 |
Table 4.
Benchmark Records and Access
Both halves of the law corpus are pinned: federal law to gesetze-im-internet revision 67e05675, a scrape dated 2026-02-09 that parses to 106,354 provisions across 5,936 law books, and Bavarian state law to a crawl of BAYERN.RECHT’s statute index taken on 2026-08-31, giving 7,008 articles from 720 statutes.
The cases and reference solutions are teaching material of the Institute for Legal Didactics. Most are based on original state-examination papers, which places them under copyright, and they remain in active use in the examination course they were written for. They are therefore not redistributed. GPBam is a held-out benchmark: we evaluate on it and release raw scores, but not the underlying documents or generations. Scores, code and the retrieval task (query rewrites, statute corpus) are released at https://github.com/wendli01/gpbam
Model Selection
Table 3 shows the models used in our experiments. We selected a heterogeneous candidate pool to cover dimensions plausibly related to long-form legal performance: capability, parameter scale, architecture, training objective, developer family, and deployment regime. This breadth reduces the risk that the benchmark findings reflects one model family or configuration. Because a complete legal opinion requires sustained, multi-step analysis, we include both standard instruction-tuned and explicitly reasoning-oriented models rather than restricting the evaluation to one inference paradigm. We likewise include proprietary hosted systems and open-weight models. The later permit on-premise deployment and processing, thus greater control over sensitive legal texts in contrast to proprietary systems [37, 38]. Across these deployment settings, the model coverage spans dense and mixture-of-experts architectures and a wide range of parameter scales. Beyond the general-purpose families, we include SteuerLLM [39], a German tax-law adapted model, to examine whether legal-domain adaptation transfers to public-law examination tasks, and two European models trained outside the major vendor families: the multilingual EuroLLM [40] and Soofi-S-Isar [41], a German sovereign-AI consortium release.
Table 3. Models used in the experiments. Identifier is the string used to query the serving endpoint; short name is the label used in the results tables. Size is the parameter count published by the model provider, given as total/active for mixture-of-experts models and as a single figure for dense ones. n/d marks systems whose provider does not disclose a count. ‡ Claude-Opus-5 was written through an agent harness rather than the API path every other row used (Table 4).
| Family | Identifier | Short name | Size | Serving |
| Anthropic | anthropic/claude-haiku-4.5 | Claude-haiku-4.5 [42] | n/d | OpenRouter |
| anthropic/claude-opus-5‡ | Claude-opus-5 [43] | n/d | agent harness | |
| DeepSeek | deepseek/deepseek-chat-v3-0324 | DeepSeek-chat-v3 [44] | 685B/37B | OpenRouter |
| deepseek/deepseek-r1-0528 | DeepSeek-r1 [45] | 685B/37B | OpenRouter | |
| deepseek/deepseek-v3.2 | DeepSeek-v3.2 [46] | 685B/37B | OpenRouter | |
| deepseek-ai/DeepSeek-V4-Flash | DeepSeek-V4-Flash [47] | 284B/13B | NHR@FAU | |
| google/gemini-2.5-flash-lite | Gemini-2.5-flash-lite [48] | n/d | OpenRouter | |
| google/gemini-3.7-flash | Gemini-3.7-flash [49] | n/d | OpenRouter | |
| RedHatAI/gemma-4-31B-it-FP8-block | Gemma-4-31B-I [50] | 31B | NHR@FAU | |
| IBM | ibm-granite/granite-4.1-3b | Granite-4.1-3b [51] | 3B | NHR@FAU |
| Meta | meta-llama/llama-3.1-8b-instruct | Llama-3.1-8b-I [52] | 8B | OpenRouter |
| meta-llama/llama-3.3-70b-instruct | Llama-3.3-70b-I [52] | 70B | OpenRouter | |
| meta-llama/llama-4-maverick | Llama-4-maverick [53] | 400B/17B | OpenRouter | |
| Microsoft | Microsoft/Phi-4-mini-instruct | Phi-4-mini-I [54] | 3.8B | NHR@FAU |
| Mistral | GaleneAI/Magistral-Small-2509-FP8-Dynamic | Magistral-S [55] | 24B | NHR@FAU |
| mistralai/Ministral-3-14B-Reasoning-2512 | Ministral-3-14B-R [56] | 14B | local vLLM | |
| RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8 | Mistral-S-3.2-24B-I [57] | 24B | NHR@FAU | |
| mistralai/Mistral-Medium-3.5-128B | Mistral-M-3.5-128B [58] | 128B | NHR@FAU | |
| mistralai/mistral-large-2512 | Mistral-L [59] | 675B/41B | OpenRouter | |
| OpenAI | openai/gpt-4o-mini-2024-07-18 | GPT-4o-mini [60] | n/d | OpenRouter |
| openai/gpt-5-nano | GPT-5-nano [61] | n/d | OpenRouter | |
| openai/gpt-5-mini | GPT-5-mini [61] | n/d | OpenRouter | |
| openai/gpt-oss-120b | GPT-oss-120b [62] | 120B/5B | NHR@FAU | |
| Qwen | qwen/qwen3-32b | Qwen3-32b [63] | 32B | OpenRouter |
| Qwen/Qwen3.6-35B-A3B-FP8 | Qwen3.6-35B-A3B [64] | 35B/3B | NHR@FAU | |
| qwen3-next-80b-a3b-instruct | Qwen3-next-80b [65] | 80B/3B | InnKube | |
| qwen/qwen3.5-122b-a10b | Qwen3.5-122b [66] | 122B/10B | OpenRouter | |
| qwen/qwen3-235b-a22b-thinking-2507 | Qwen3-235b-T [63] | 235B/22B | OpenRouter | |
| qwen/qwen3.5-397b-a17b | Qwen3.5-397b [66] | 397B/17B | OpenRouter | |
| Other | utter-project/EuroLLM-22B-Instruct-2512 | EuroLLM-22B-I [40] | 22B | local vLLM |
| soofi-s-siar-preview | Soofi-S-Isar [41] | 32B/3B | InnKube | |
| windprank/open_steuerllm | Open_steuerllm [39] | 28B | local vLLM |
Table 3.
The main evaluation set contains 32 models: 29 from nine general-purpose developer families, together with SteuerLLM, EuroLLM and Soofi-S-Isar. Twenty-five are open-weight releases, all with published parameter counts ranging from 3B to 685B; the remaining seven are proprietary hosted systems whose providers disclose none. Among the 25 open-weight models, twelve use dense architectures and thirteen use mixture-of-experts architectures with 3B–41B active parameters. The retrieval ladder covers seven generators.
Generation Without and With Statute Retrieval
We evaluate two generation conditions over the same case facts: all 32 models without retrieval, and a subset of seven of them, chosen for cost reasons, with retrieval; the prompts are reproduced in Section A.2. The conditions differ in whether statutory text is inserted into the instruction’s context slot and, as a consequence, in the instruction that frames that slot (§ 3.3.1). Except in the oracle condition described below, no material derived from the reference solution is available during generation.
Generation without retrieval. Each model receives the fixed German essay instruction and the facts of one case, and generates a complete legal opinion (Gutachten). No statutory text or other external legal material is supplied. Every model receives the same 81 cases under this condition; the few essays that came back empty, truncated or errored are reported in § 4.1.
Generation with statute retrieval. For cost reasons, this condition is run on a subset of seven of the 32 models (Table 8). They receive the same case facts with statutory text inserted into the instruction’s context slot, retrieved automatically from a corpus of German federal and Bavarian law. § 3.6 describes that corpus, the retrieval pipeline and the arms we vary; Section 4.3 reports what they achieve.
Across both conditions, we set no additional output-token limit, and we do not set temperature or top-: each request uses the serving endpoint’s own defaults, which are recorded in the run records where the endpoint reports them (§ 3.3.3). If a response contains a reasoning trace delimited by <think> tags, the trace is removed before the answer is scored.
Statute Corpus and Retrieval Pipeline
The retrieval task in GPBam has a built-in evaluation set through the norms cited in the case solution. This lets us gauge retrieval quality independently of the final essay quality, and therefore ablate on the integration and generation aspect separately from the search itself. That opportunity is why statutory text is the primary context throughout; secondary sources enter as a second knowledge source, evaluated both as context in their own right and as a route to the statutes they cite.
Corpus. German public law lies at the intersection of federal regulations and respective state law, with applicability a crucial part of the legal appraisal. We therefore construct a combined corpus of federal law from the gesetze-im-internet dump maintained by the QuantLaw project [67],6 parsed into 106,354 provisions across 5,936 law books, and of Bavarian state law collected from BAYERN.RECHT, the portal of the Bayerische Staatskanzlei [68], comprising all 7,008 available provisions from 720 statutes. The combined index holds 113,362 provisions. Both halves are built the same way: we extract the provision text, chunk at law-paragraph borders, and fold the law metadata (name, law book, paragraph) into the indexed text. The federal dump contains no Landesrecht at all, and 38.7% of the distinct norms a reference solution cites are exactly that, so a federal-only corpus caps recall at 54.2% against the combined corpus’s 91.3% (Table 6).
| Jurisdiction | Share | Federal | + Bavarian | |
| Federal (Bundesrecht) | 2,067 | 49.2% | 99.8% | 99.8% |
| State (Landesrecht) | 1,626 | 38.7% | 0.0% | 94.8% |
| EU law | 69 | 1.6% | 7.2% | 7.2% |
| Unattributed | 440 | 10.5% | 48.0% | 51.6% |
| All | 4,202 | 100.0% | 54.2% | 91.3% |
Table 6. Table 6. What the reference solutions cite, and how much of it the retrieval corpus holds. Every distinct statutory reference in the 81 reference solutions, counted once per case, (extracted with the dual §/Art. extractor rather than refex, so constitutional and Bavarian Art. citations are included), classified by jurisdiction and checked against the two knowledge bases. Federal is the gesetze-im-internet dump the deployed pipeline searched; + Bavarian adds the 7,008 articles of Bavarian state law we collected from BAYERN.RECHT. A norm absent from the index cannot be retrieved however good the search is, so these figures bound recall from above.
As an additional knowledge source we index Leitsätze, headnotes produced over legal decisions by a German legal publisher as secondary literature. Headnotes are short - 306 characters and 40 words on the mean, 285 characters at the median - but they carry explicit guidance on how the law was applied and summarise the decision at the same time. We index 38,966 of them, extracted from the German legal decision corpus of Urchs et al. [69] and covering the Bavarian public-law courts over 2015–2020. Each carries the Normenkette of its decision, which is what makes a second hop to statutes possible. Roughly 4% are amtliche headnotes written by the court itself, the remainder editorial work by a legal publisher.
Query construction. Before generation, Qwen3.6-35B-A3B rewrites each case into three to five stand-alone legal concepts or questions, an implicit query diversification scheme. This moves some of the legal knowledge boundary onto the rewriter, but both previous retrieval studies [18, 19] and our own find that it clearly outperforms other rewriting strategies as well as the raw task description (tripling gold-norm recall@50 against the raw case text). The concept rewrites are generated once and reused across every arm that searches, so that two search arms differ in the retrieval component under study and not in the query.
Search and fusion. For dense retrieval, we use LanceDB’s Sentence Transformers [70] integration with jina-embeddings-v5-text-small-retrieval, a task-targeted distillation of the Jina v5 text family [71]. We retain the adapter’s default encoding behaviour for both corpus indexing and query embedding and combine this dense channel with lexical BM25 [21] retrieval over the same corpus. For each rewritten query, reciprocal rank fusion (RRF; ) [20] first merges the dense and BM25 rankings. RRF is then applied a second time across the resulting rankings from all rewritten queries. The ten highest-ranked passages are passed to the generator unless a different passage budget is specified.
In short, the two retrieval methods produce candidate rankings for each rewritten query, and RRF is applied twice: first to merge the dense and BM25 rankings within each query, and then to merge the results across the rewritten queries before the highest-ranked passages are passed to the generator.
Retrieval arms The retrieval arms vary passage budget and components (Table 8). The cite arms resolve norms named by a separate rewriter; cite-only provides them alone, while cite+RRF fills the remaining budget with hybrid search. Other arms re-rank with Qwen3-Reranker-4B [72], resolve the Normenketten of twenty retrieved headnotes, or supply headnotes directly. Finally, an oracle supplies the full text of provisions cited by the solution. It bounds corpus retrieval and is not deployable.
LLM Judge Ensemble
Complete legal case solutions are open-ended: legally adequate answers may differ substantially in wording while expressing equivalent legal analyses. Consequently, lexical or shallow semantic similarity to a reference solution cannot by itself assess issue spotting, doctrinal assessment, completeness, structure, or subsumption. At benchmark scale, manual assessment of such responses requires costly annotation by legal experts [12, 5]. Recent legal benchmarks therefore employ rubric- or reference-guided LLM judges as scalable semantic evaluators. Comparisons with expert annotation show that carefully specified judges can provide useful assessments [6, 5, 4]. These studies also identify differences in judge severity, prompt sensitivity, and model-family preference [33, 5, 4, 32].
Each essay is therefore graded independently by three open-weight LLM judges drawn from three different developers: GPT-oss-120b, Qwen3.6-35B-A3B, and DeepSeek-V4-Flash. Panels of this kind have been reported to grade legal answers more reliably than their strongest single member: on German examination scripts, as shown by [35], pooling three models raises agreement with expert marks by up to 0.15 QWK over the best member when the full grading context is supplied, and an ensemble judge is also what LEXam validates against expert assessment [6]. The composition additionally reduces dependence on any single evaluator and limits the influence of developer or model-specific evaluation preferences.
Because judges may nevertheless differ systematically in severity, we aggregate their essay-level scores using the median. With three judges, the median retains the central assessment while limiting the influence of one atypically high or low score. We report absolute and rank agreement between the seats at essay and model level in § 5.1.
All judges receive the same evidence in the prompt: the case, its reference solution, and the generated essay. The instructions guide each judge to align the reference and generated answers by section and assess correctness, completeness, legal reasoning, structure, depth of subsumption, and statutory citations, scoring how far the essay matches the solution’s results, subsumption, and citations, and crediting deviations from the solution only if they are legally sound. Each judge is instructed to return a score from 0.0 to 1.0 in increments of 0.1. The complete judge prompt is reproduced in Section A.2.4. To anchor the judge scale, we additionally submit three controls for each case: its reference solution, another case’s reference solution, and an answer that repeats the case facts (§ 4.1).
Evaluation Metrics
Ensemble Judge Score Let be the score judge assigns to the essay that model writes for case , and let be the set of judges that returned a score for that essay. The per-essay score and a model’s reported Score are
Missing judge scores are skipped rather than imputed, so may contain fewer than three judges. The error term reported with in Table 4 is its standard error, , where is the sample standard deviation of the 81 case-level scores .
Self-preference adjustment Three judges are also generators and eleven more share vendors. Debiased in Table 4 subtracts the excess score that judges give to related models before taking the median. We estimate that excess at matched essay quality: each judge’s scores become percentiles within that judge, we fit them against the mean percentile of the other two on the same essay, and read the offset off that fit.
Legal Reference Similarity For each generated essay and its reference solution, we extract the unique pairs of law book and provision, resolving both §- and Art.-form citations.7 Legal reference similarity is the Jaccard overlap of these pairs, , where and are the pairs extracted from the generated essay and the reference solution, respectively. We scale essay-level values by 100. The widely used refex8 resolves only §-form citations and so misses constitutional and Bavarian provisions, finding about 21 norms per solution instead of 52.
Article Recitation For each provision in the GPBam laws and Most-cited laws sets defined in § 3.3.2, the generated text is compared with its stored target using ROUGE-L F1 without stemming. This metric measures longest-common-subsequence overlap with the statutory wording. For each model and provision set, we scale scores by 100 and report the mean over the set’s 100 provisions, or over the provisions actually scored for the four models noted in § 3.3.2.
Retrieval Recall Gold-norm recall@k is the share of the statutory references a reference solution cites that appear among the passages an arm returns for that case, pooled over the 81 cases rather than averaged per case, so a solution weighs in proportion to how much law it names; is the arm’s own passage budget, ten unless marked . State-law recall@k is the same share over the Landesrecht references alone, and cases with a hit@k the share of cases where at least one cited norm is returned.
In short, gold-norm recall@k asks what share of all cited norms across the 81 cases is retrieved within the passage budget, state-law recall@k asks the same question for Landesrecht only, and hit@k asks in what share of cases at least one cited norm is found.
Across Model Relationships To examine which model-level characteristics are associated with essay quality, we compute Spearman rank correlations between each model’s mean ensemble score and its Article Recitation score on either provision set, its Artificial Analysis index, its answer tokens, its generation cost, and its Legal Reference Similarity (Table Table 5). We report the No-RAG condition only. Each model is one observation, and only the model-level point estimates are correlated. Percentile 95% confidence intervals use 10,000 bootstrap resamples of the paired model observations; pairwise-complete data are used, so is smaller for the Artificial Analysis index, which does not list every model we evaluate, and for generation cost, where provider pricing is unavailable.
Inter-judge agreement We compare every judge pair using Spearman’s and quadratic-weighted kappa (QWK, ). At model level, one observation is a generation model and each judge’s value is its mean over that model’s available essays; percentile 95% confidence intervals are computed from 10,000 bootstrap resamples, each formed by sampling the 32 generation models with replacement. At essay level, one observation is an essay scored by both judges, and confidence intervals use 10,000 cluster-bootstrap resamples of the 32 generation-model clusters with replacement, keeping all essays from each selected model together to preserve dependence among its 81 essays. Spearman measures rank agreement, whereas QWK measures agreement in absolute score levels: at essay level the – scores are placed on the judges’ own fixed 11-point ordinal grid (– in steps of , rescaled to –), pinned to all eleven levels rather than to those a given pair happens to use, so that the index spacing, and hence the quadratic weights, means the same thing for every pair; the few judgements that land off that grid are parser strays and are snapped to the nearest step. At model level, where the values are means rather than grid points, we use the bin-width-to-zero limit of the same statistic. For each judge pair, we compute the coefficients using only essays or models scored by both judges. See Table 11
| Level | Judge pair | Spearman | QWK | |
| Model | GPT-oss-120b / Qwen3.6-35B | 32 | 0.95 [0.84, 0.99] | 0.92 [0.87, 0.94] |
| Model | GPT-oss-120b / DS-V4-Flash | 32 | 0.95 [0.82, 0.99] | 0.92 [0.79, 0.96] |
| Model | Qwen3.6-35B / DS-V4-Flash | 32 | 0.99 [0.98, 1.00] | 0.82 [0.63, 0.89] |
| Essay | GPT-oss-120b / Qwen3.6-35B | 2591 | 0.81 [0.71, 0.86] | 0.80 [0.71, 0.84] |
| Essay | GPT-oss-120b / DS-V4-Flash | 2591 | 0.78 [0.68, 0.84] | 0.79 [0.64, 0.86] |
| Essay | Qwen3.6-35B / DS-V4-Flash | 2592 | 0.81 [0.71, 0.87] | 0.75 [0.59, 0.82] |
Table 11. Inter-judge agreement for no-retrieval essays. At model level, each observation is a generator’s mean score; at essay level, it is one essay. Spearman’s measures rank agreement and quadratic-weighted kappa () agreement in score levels. Brackets show 95% bootstrap CIs, clustered by generation model at essay level.
Results
Performance on Complete Legal Opinions
Overall performance. Table 4 shows that GPBam is not saturated by the current model population. Without retrieval, the 32 models occupy 4.81–76.42 points of the – scale, and the distribution is concentrated in its lower half. The median model score is 30.2, 21 of the 32 fall below 33, and nine remain under 20. However, two models are clearly separated from the remainder. Claude-opus-5 (76.42) and Gemini-3.7-flash (67.10) precede a 23.3-point discontinuity to the strongest open-weight entry, DeepSeek-V4-Flash (43.77). This single interval exceeds the entire range from rank three to rank eighteen (43.77 to 26.36), which comprises sixteen models. Below the leading two, the table constitutes a dense band rather than an ordering. The distinctions the measurements support are those between Claude-opus-5, Gemini-3.7-flash, and the remaining thirty. The two recitation columns (Art. recit.) and the external index (AA) are discussed in § 4.2.
Non-response is negligible except at the bottom of the table. Over the 2,592 essays (32 models over 81 cases), twenty-one (0.8%) came back empty, truncated or errored, twelve of them Phi-4-mini-I (15% of its cases).
What the scale means. Three synthetic answers per case, judged by the same three seats under the same instruction, anchor the – axis. A case’s reference solution, submitted as an answer, scores 100.0 on all 81 cases, so the top of the scale is attainable rather than notional. A different case’s reference solution - expert-written, correctly structured, statute-dense German legal prose answering the wrong question - scores 0.0, as does echoing the Sachverhalt back. The panel median is invariant at both anchors; individual seats depart only marginally. The anchors fix the endpoints, and those are the easy case - the judge is given the reference solution, so spotting a wrong-case answer needs no legal judgement. They show the scale is not degenerate, not that it is linear between; § 5.2 locates the remaining disagreement in the upper interior, where the field’s spread lives.
Reference Similarity Legal Reference Similarity (Ref.) gives another measure of essay quality, with a similar gap: the leading two models achieve an overlap of 39.63% and 36.53% with the solution references, while the third model falls below one quarter, and the final nine rows remain under 9%. This column tracks the essay-score ranking closely at with only three models exhibiting a difference of more than five positions. Splitting the Jaccard similarity shows that mean precision is 0.56 but mean recall only 0.16: a model cites 14.5 provisions where the solution cites 52. These results indicate that the dominant citation failure is incomplete coverage, not lacking precision over a small citation set.
Scale and family. We observe that neither parameter count nor developer family accounts for the ordering of the models. Gemma-4-31B-I (32.90) ranks above DeepSeek-chat-v3 (26.36) and Llama-4-maverick (24.44), both 400B or larger in total parameters compared to 31B, while Qwen3-32b attains 13.21 at comparable scale. Variation within a single family matches variation across families with Mistral 8.77–37.84, Qwen 13.21–41.48. Building for a language or a legal domain is no substitute either, and the three systems selected outside the general-purpose developer families place 19th, 25th and 28th. Soofi-S-Isar, pretrained on roughly 27 trillion tokens with German deliberately up-weighted [41], reaches 26.11 and leads the other two, but stays below Gemma-4-31B-I (32.90) at comparable total scale and below the other two A3B models, Qwen3.6-35B-A3B and Qwen3-next-80b (30.62 each); EuroLLM-22B, built for the official EU languages [40], scores 9.26. Open_steuerllm, adapted to German tax law [39], ranks 25th at 17.78 points, below three of the four general-purpose models within a few billion parameters of its size.9 adaptation to one legal domain shows no advantage for public-law examination writing here.
Judge behaviour. The three seats differ in level more than in ordering: Qwen3.6-35B-A3B returns the highest of the three scores on 26 of the 32 rows and DeepSeek-V4-Flash the lowest on 25, but the between-judge spread reaches 20.4 points on Claude-opus-5 and at most 4.6 points on the four weakest models, where it also reverses direction, the otherwise strictest judge becoming the most lenient. The median in Equation (1) absorbs this. Self-preference is narrower: the only corrections above one point fall on the three models that are themselves judges (−2.72, −2.35, −1.60), while the eleven vendor-related rows move by at most 0.62 points. Model order is unchanged, but the gap between DeepSeek-V4-Flash and Qwen3.5-397b narrows (2.29 to 0.19).
What Predicts Essay Quality
Article Recitation measures both general and domain-specific statutory knowledge while the Artificial Analysis Index [73] measures general capability. Essay score correlates with recitation at on GPBam laws and 0.81 on Most-cited laws, and with the index at 0.85 over its 29 rated models. These external-predictor relationships are visualized in Figure 2, while Table 5 summarizes all model-level associations.
(a)
(b)
Figure 2. Article Recitation on both provision sets ( between sets). Artificial Analysis Intelligence Index.
| Predictor | [95% CI] | |
| Knowledge | ||
| Article Recitation, GPBam | 32 | 0.92 [0.79, 0.98] |
| Article Recitation, Most cited | 32 | 0.81 [0.56, 0.92] |
| Artificial Analysis index | 29 | 0.85 [0.65, 0.95] |
| Length | ||
| Answer Tokens (k) | 32 | 0.71 [0.51, 0.83] |
| Cost | ||
| Gen. Cost ($) | 27 | 0.77 [0.52, 0.90] | ||
| Same essay | ||
| Legal Ref. Sim. | 32 | 0.94 [0.84, 0.98] |
Table 5. Associations with essay quality without retrieval. Spearman correlations between mean essay score and model-level metrics across up to 32 models; brackets show 95% bootstrap CIs. Generation costs for self-hosted models use OpenRouter list prices.
The Artificial Analysis index correlates with essay score at (Figure 2b), so GPBam ranks the model field much as the general capability AA index already does. However, GPBam adds further insight where the two disagree. Four models carry the same index value of 11 and still span 13.21 to 35.19 essay points, which the index cannot resolve. Mistral-L is the clearest case: an index of 16 puts it in the lower half of the rated field, yet it scores 37.84, ahead of eleven models the index rates at 20 or above, and it holds the best recitation scores of any open-weight system in the study.
After controlling for the AA index, recitation still predicts essay score (; 0.64 on Most-cited laws). Length also correlates at 0.71. Reference Similarity is strongest at [0.84, 0.98], but is extracted from the graded essays and not independent evidence. These associations don’t answer whether providing missing norms improves essays, but our RAG experiments do.
Figure 2. External predictors of No-RAG essay quality. Each point represents one model. Essay scores averaged across 81 cases. Panels report Spearman with 95% percentile-bootstrap CIs. AA Intelligence Index values were taken from the public Artificial Analysis model pages (index v4.1.1, accessed 2026-08-17). For 11 of the 29 models, Artificial Analysis reports the value as an estimate pending independent evaluation10; we use these values as published. For Claude-Opus-5 and DeepSeek-V4-Flash, only a max-effort configuration is listed, whereas our runs use high effort. Later index versions rescale the values.
Can Retrieval Close the Knowledge Gap?
The following experiment tests whether this knowledge gap is indeed a binding constraint. We observe that the correlation between judged essay quality and reference similarity is high, which highlights the importance of correct context sourcing. As article recitation also closely tracks essay quality, we attempt to fill the legal knowledge gap via retrieval over a law corpus (§ 3.6). By providing a set of relevant norms and their text, we attempt to equalize across different levels of inherent static knowledge and ablate on the dynamic ability to subsume and apply a law.
The corpus is not the binding constraint. A federal-only index caps recall at 54.2% of the gold norms, against 91.3% for the combined corpus (Table 6).
An ablation holding the pipeline fixed and swapping only the corpus (Table 7) recovers that ceiling without converting it: state-law recall@10 rises from 0, which the federal dump forces by construction, to 4.31%, and gold-norm recall@10 from 2.28% to 3.50%. However, over the five generators that ran both corpora under the same RRF arm, the essay-score differences span −0.80 to +1.79 points and only DeepSeek-V4-Flash gains a point, while Gemma-4-31B loses ground. Therefore, adding Bavarian statutes helps retrieval coverage, but does not consistently improve downstream essay quality.
| federal only | combined | |
| Retrieval quality (%), against federal only | ||
| gold-norm recall@10 | 2.28 | 3.50 |
| state-law recall@10 | 0.00 | 4.31 |
| Essay score (0–100), against federal only | ||
| DeepSeek-V4-Flash | 43.33 | 45.12 |
| Gemma-4-31B-it | 34.57 | 33.77 |
| Qwen3-Next-80B-A3B | 29.63 | 30.00 |
| Qwen3.6-35B-A3B | 31.73 | 32.16 |
| gpt-oss-120B | 28.64 | 29.26 |
Table 7. What the Bavarian half of the corpus is worth. The retrieval rows hold the pipeline at RRF, so they read at the same setting as Table 8; the essay rows are the round-robin arms, the only ones generated over the federal corpus. The federal corpus contains no state law, so its state-law recall is zero by construction while its overall recall is barely distinguishable – overall recall is the wrong number to report here. Rows and shading follow Table 8; the baseline column is the federal corpus.
Table 8 compares the arms on retrieval quality, essay score and citation coverage. Gold-norm recall@ rises nearly threefold across the eight search arms, from 3.50% to 9.61% of a ≈19% ceiling at =10 (precision@10 of 50%) and 8.52% to 23.92% at = 50, while the largest gain of any of them buys is 4.13 points – DeepSeek-V4-Flash at cite+LS+RRF, 47.90 (tying its oracle score) against 43.77 - and no other generator gains more than 3.4. In short, gold-norm recall roughly triples across the retrieval methods, but the largest corresponding improvement in essay score is only 4.13 points. Thus, substantial gains in retrieval quality do not translate into corresponding improvements in downstream legal-writing quality.
Overall, the norm oracle significantly improves six of seven generators, but retrieval cannot always close that gap (Table 8). The oracle raises the point estimate for all seven generators and significantly improves six. However, it does not narrow the differences between generators: the two weakest generators gain 3.21 and 2.60 points, below the seven-generator mean gain of 4.01, and the across-generator standard deviation increases from 7.15 without retrieval to 7.73 with the oracle (Table 9). The gain is the statute wording, not leaked structure or norm names: for all three, the oracle’s norms without text do not help (Table 10). Specialized approaches, like norm lookup through headnotes, improve retrieval and generation, but not for all generators (only one reaches oracle level), while supplying headnote text directly does not significantly improve.
| Oracle comparison | Result |
| Generators with a positive point estimate | 7/7 |
| Generators with a significant improvement | 6/7 |
| Mistral-Small-3.2 gain (weakest) | +3.21 |
| Magistral-Small gain (second weakest) | +2.60 |
| Mean gain across seven generators | +4.01 |
| Across-generator SD without retrieval | 7.15 |
| Across-generator SD with oracle | 7.73 |
Table 9. Summary of the oracle effect on essay scores, derived from Table 8.
| no retrieval | + uncov. oracle | named | order shuffled | + junk norms | wording withheld | top-10 only | |
| DeepSeek-V4-Flash | 43.77 | +4.13 | +5.06 | +2.22 | +4.20 | −0.19 | +2.84 |
| Gemma-4-31B | 32.90 | +5.37 | +6.23 | +5.25 | +5.25 | +0.56 | +5.31 |
| gpt-oss-120B | 29.75 | +5.87 | +5.74 | +3.46 | +2.04 | +0.00 | +5.43 |
Table 10. Oracle advantage decomposition, relative to No-RAG. Bold exceeds twice the standard error. Wording withheld supplies norms without their text; padding with junk norms and keeping 10 most-cited references test dilution; + uncov. named additionally names the cited norms not in the corpus. Order shuffled drops the ranking by citation frequency, not an argument order.
Retrieval is not uniformly beneficial either (Table 8): under hybrid search Mistral-Small-3.2 falls by 2.47, and Magistral-Small by 1.85, compared to no retrieval. Neither of them, nor Qwen3-Next-80B, improves significantly with any RAG setup, though Mistral-Small-3.2 and Magistral-Small still gain significantly from the oracle (+3.21, +2.60), so the law does help them.
Reliability of the Evaluation
Inter-Judge Agreement
Table 11 reports pairwise agreement between the three judge seats, measured both at model and at essay level, and Figure 3 shows the score distributions those coefficients summarise.
Figure 3. Pairwise judge agreement without retrieval. Columns compare judge pairs. The top row shows joint essay-score distributions, with colour indicating the percentage of essays; the bottom row compares each judge’s mean score per generation model. Dashed lines mark exact agreement. Judges rank outputs similarly but differ systematically in score level.
Rank agreement. The three seats agree most strongly at the resolution the benchmark reports. Over the 32 generation models every pair ranks alike at or above; over individual essays the same pairs reach 0.78–0.81 (– 2,592). Averaging 81 essays per model cancels much of the per-essay variation, so the model ordering in Table 4 is supported by stronger agreement than any individual essay score is.
Level agreement. Quadratic-weighted kappa separates the pairs differently because it is sensitive to scoring severity whereas is not. Qwen3.6-35B and DS-V4-Flash rank models most similarly (), yet show the lowest agreement in score level ( at model level and 0.75 at essay level). Qwen3.6-35B scores above DS-V4-Flash for 27 of the 32 models. Together with their high rank correlation, this indicates that the pair differs primarily in severity rather than model ordering. The judges’ mean scores across the 32 models - 28.4 for GPT-oss-120B, 33.4 for Qwen3.6-35B, and 24.8 for DS-V4-Flash - show the same severity ordering visible in Table 4. The three judges therefore largely agree on model ranking and show substantial model-level agreement in absolute scores. Their lower essay-level agreement is consistent with the greater variability of individual essays compared with averages over 81 cases.
A Strong External Judge
Inter-judge agreement measures consistency, not validity, and all three seats are mid-sized open-weight models. As a partial check we had Claude-Opus-5 - well above every panel member on general capability - grade the 21 essays of the expert-annotation packet, seven cases crossed with three a priori capability tiers, using the byte-identical production request the panel seats received. Two further items are blind duplicates judged by independent instances; both returned the same score, so the external judge’s own measurement noise is not what limits this comparison.
Opus-5 orders the essays much as the panel does. Against the panel median it reaches , and against the individual seats 0.65–0.92, comparable to the 0.62–0.83 agreement among the panel judges on the same essays (Table 12). Within a case, where the three essays differ only in the model that wrote them, mean Kendall is +0.88. The prespecified capability tiers have the same average ordering under both Opus-5 and the panel median.
| Pair | ||
| Opus-5 against the panel | ||
| vs. panel median | 0.85 | 0.65 |
| vs. gpt-oss-120b | 0.92 | 0.64 |
| vs. Qwen3.6-35B-A3B | 0.82 | 0.58 |
| vs. DeepSeek-V4-Flash | 0.65 | 0.67 |
| Panel internal, same 21 essays | ||
| gpt-oss-120b vs. Qwen3.6-35B-A3B | 0.83 | 0.85 |
| gpt-oss-120b vs. DeepSeek-V4-Flash | 0.62 | 0.55 |
| Qwen3.6-35B-A3B vs. DeepSeek-V4-Flash | 0.78 | 0.76 |
Table 12. External-judge agreement on 21 essays. Opus-5–panel and within-panel agreement across seven cases and three prespecified capability tiers. is Spearman correlation and quadratic-weighted kappa.
Agreement on absolute level is weaker, and in one direction. Regressing Opus-5 on the panel median gives a slope of 0.56: the two agree almost exactly at the bottom of the range and diverge as the panel’s score rises. The written verdicts locate the disagreement in fabricated norms and inverted results, which the panel appears to penalise less than a stronger judge does, rather than style - and that is the region the benchmark’s spread lives in. We report this as a bound on where the panel is most likely to be wrong, not as validation. Opus-5 is one more judge with its own strictness prior, and no human grades exist for GPBam.
Discussion & Conclusion
We build GPBam to answer what short-answer legal benchmarks cannot: “When a model fails to write a Gutachten, is it because it does not know the law, or because it cannot apply it?” The retrieval ladder answers that by supplying the law and we find that the law helps, by a real but bounded amount.
Supplying the statute helps, but closes no gap. An oracle supplying the provisions the solution cites is worth 2–6 points across the seven generators (Table 8); handed that 91.3%, no generator cites even half of what it is given (41.4% at best).
Retrieval gains lag far behind recall. Gold-norm recall varies roughly threefold across our configurations while essay score moves by at most 4.1 points, and the oracle’s 91.31% buys at most 3.3 more points over RAG. How those norms are packaged matters far less consistently: padding, reordering and a ten-norm subset each cost some generators much of the gain and others almost none (Table 10). That bears directly on work that evaluates the retrieval step in isolation [7]: for these generators, a pipeline can be made much better at finding the right norms without becoming better at the task the norms are for.
The rewriter’s memory outranks the search channel. Naming the provisions and resolving them by exact key – no search at all – beats hybrid search on gold-norm recall, but not on state law (Table 8). One reading is that statute text is simply the wrong register to search with a case: a Sachverhalt raises doctrinal concepts, and a statute rarely names the doctrine it embodies. However, some models reproduce statutory text better than they write legal opinions while the mid-tier proprietary systems do the reverse. Perhaps they hold the material but cannot apply it in the Gutachtenstil the examination rewards.
The panel is internally consistent. Three diverse judges rank the 32 models near-identically (–) so the ranking is not an artefact of any single seat. For our public-law cases, the difficulty of the Gutachten lies in applying law, not obtaining statutes – which a larger corpus cannot solve alone.
Limitations & Future Work
Limitation. GPBAM contains 81 state-exam-style public-law cases from one Bavarian preparation course; results may not generalize to official examiners, other legal domains, institutions, or jurisdictions. All essay scores come from three LLM judges, without expert grades. Their agreement measures consistency rather than validity and cannot detect shared bias. Every RAG delta also carries a template change (Section 3.3.1), so none isolates retrieval from framing. The wording withheld arm bounds it at −0.19 to +0.56 across all three (Table 10), including the generator with the largest retrieval gain. Claude-opus-5 alone ran with a harness system prompt; transcripts show no access beyond the prompt file, but the prompt may account for part of its lead. The scores should not be read as German examination grades or used to assess students. At model level, score also correlates with answer length (), and our design cannot distinguish greater completeness from a preference for longer answers [74, 33]. The material is held out rather than released, so results cannot be independently reproduced from the sources, except for the released retrieval task. Finally, the reported model-level correlations are descriptive and do not establish that statutory recall or citation overlap causes better legal reasoning. The corpus is Bavarian and federal only: it excludes the 1,476 Verwaltungsvorschriften and holds just 7.2% of the EU provisions the solutions cite. It holds statutes in their current version, so a provision may have been amended since the reference solution was written. All three judges are also generators, though the per-case median limits bias effects.
Future work. The judge panel should be validated against independent grades from qualified examiners for GPBam outputs while accounting for variation in German exam grading [30] and to establish whether the judge panel sits inside the range of human disagreement or outside it. Our retrieval findings suggest that statutes are the wrong register to search with: in our corpus, doctrinal vocabulary of the kind a Sachverhalt raises (e.g. Rücksichtnahmegebot, Störerauswahl) occurs only in 0.39% of statute passages, while 38.5% of court decisions use those terms. Leitsätze supply that missing register. Our highest-recall arm already routes through them, resolving each retrieved headnote’s Normenkette rather than reading its text (Table 8), so searching the court decisions behind them and integrating full commentaries are the next steps.
Acknowledgements
The paper has been funded by (SCCH Comet) COMET K1-Competence Center for Integrated Software and AI Systems (INTEGRATE) within the Austrian COMET Program. This work was supported by the project FrAInderl, which is co-funded by the European Union. We thank the Erlangen National High Performance Computing Center (NHR@FAU) for their LLM-as-a-Service offering.
GenAI Disclosure
GenAI was used during the preparation of this manuscript to assist with code development and with drafting and revising text and tables to improve clarity and readability. All GenAI-assisted outputs were carefully reviewed, verified, edited, and incorporated by the authors as appropriate, who take full responsibility for the final manuscript.
Additional Material
Price and Performance
Table 4 reports what each model achieves on the essay task while Figure 4 reports on the cost dimension for the No-RAG condition.
Figure 4. GPBam Essay Writing price to performance. One point per model, for the 29 paired models of Table 4 with published prices. The horizontal axis is the full 81 generation cost in USD (log scale); the vertical axis is the essay writing Score. Bubble area is verbosity as the total number of tokens the model produced over the 81 essays, colour is the vendor family. The dashed step line is the price/performance frontier. Models marked * were served self-hosted and are instead priced at OpenRouter list rates (retrieved 2026-08-17).
Figure 4 plots each model’s essay score against the dollar cost of generating all 81 opinions. The two are far from proportional: cost ranges over four orders of magnitude while scores vary by a factor of sixteen, and the price–performance frontier advances in a small number of discrete steps rather than smoothly. Absolute expenditure is nonetheless modest. At the list prices used here, the complete no-retrieval study projects to $99 for generation, $32 for judging and $25 for article recitation (Table 4).
Prompts
All prompts are reproduced verbatim from the code that sent them. The German wording is the only one that was ever sent to a model. Each German block is followed by an English translation for the reader, which was used in no run. The translation keeps the layout of the German line for line – same line numbers, blank lines, indentation, list markers and slots – so that line of the English translates line of the German. Strings that a model received literally or that the score parser matches (XML tags, attribute names, the score format) are left untranslated. {} marks a slot filled per call. Line numbers count lines of the prompt; a ↪ marks where a long line is wrapped for print and is not part of the prompt.
No-RAG: Question Answer Prompt
Every legal opinion scored in this paper is produced by this prompt or by its Law-RAG variant (Section A.2.2). The request consists of the user message below and an empty system message; the one exception is Claude-opus-5, which ran through an agent harness that supplies its own system prompt. The user message has two slots: the context slot (line 17), which is empty under No-RAG, and the Sachverhalt of one case (line 21). No reference solution enters the prompt. shaded lines 11, 13 and 14 occur only under No-RAG; the Law-RAG prompt drops them (Section A.2.2).
User message (German, as sent)
1 Du bist ein Experte für deutsches öffentliches Recht und bearbeitest Rechtsfragen
↪ in strukturierter Weise im Stil eines juristischen Gutachtens.
2 Du sollst ein Rechtsgutachten zu den dargestellten Rechtsfällen erstellen, wie in
↪ der Frage angewiesen.
3 Es handelt sich um eine juristische Examensklausur auf Staatsexamensniveau, für
↪ deren Beantwortung den Studierenden 4 Stunden zur Verfügung stehen, mit einem
↪ ausführlichen Inhaltsverzeichnis, gefolgt von einer vollständigen
↪ juristischen Würdigung.
4 Gehe davon aus, dass ausschließlich deutsches Recht anwendbar ist, sofern nicht
↪ ausdrücklich etwas anderes erwähnt wird.
5 Verwende präzise juristische Sprache und antworte so gründlich wie möglich.
6 Gib KEINEN Haftungsausschluss ab und verweise nicht auf die Notwendigkeit
↪ externer Rechtsberatung.
7 Fordere den Nutzer NICHT auf, Gesetze zu konsultieren oder eigene Recherchen
↪ anzustellen.
8 Sprich direkt und autoritativ, ohne zu erwähnen, dass deine Antwort lediglich
↪ allgemeinen Informationszwecken dient.
9 Integriere deutschlandspezifische juristische Terminologie.
10 Wenn du relevante rechtliche Erwägungen entdeckt hast, antworte mit einer
↪ knappen, klaren juristischen Analyse.
11 Zitiere ausschließlich aus den von dir identifizierten Erwägungen.
12 Zitiere stets die konkrete Rechtsnorm und gib ausdrücklich Absätze (Abs.), Sätze
↪ (S.), Nummern (Nr.) oder Buchstaben (Buchst.) an, soweit vorhanden, z. B.
↪ „Art. 12 Abs. 1 GG“, „§ 35 Satz 1 VwVfG“, „§ 42 Abs. 2 VwGO“, „§ 823 Abs. 1
↪ BGB“ oder „§ 263 Abs. 1 StGB“. Vermeide allgemeine Verweise wie „Art. 3 GG“
↪ oder „§ 242 BGB“, ohne den konkreten Absatz, Satz, die Nummer oder den
↪ Buchstaben zu nennen, sofern einschlägig.
13 Wenn keine relevanten Erwägungen gefunden werden, stelle ausdrücklich fest, dass
↪ keine einschlägigen Informationen verfügbar sind.
14 Wenn dir verlässliche Quellen vorliegen, teile daraus praktische Hinweise oder
↪ Erkenntnisse mit.
15 Wenn die Eingabe verlangt, einen konkreten in der Klausur angegebenen Fall zu
↪ analysieren, der Text oder die Einzelheiten dieses Falls jedoch in der
↪ Eingabe nicht bereitgestellt wurden, weise ausdrücklich darauf hin, dass das
↪ erforderliche Fallmaterial fehlt.
16
17 {}
18
19 Frage:
20 ```
21 {}
22 ```
23
24 Antwort:
User message (English translation, not used in any run)
1 You are an expert in German public law and treat legal questions in a structured
↪ way in the style of a legal opinion.
2 You are to produce a legal opinion on the cases presented, as directed in the
↪ question.
3 This is a law examination at state-examination level, for which students have 4
↪ hours, with a detailed table of contents followed by a complete legal
↪ assessment.
4 Assume that only German law applies unless something else is expressly mentioned.
5 Use precise legal language and answer as thoroughly as possible.
6 Do NOT give a disclaimer and do not refer to the need for external legal advice.
7 Do NOT ask the user to consult statutes or to carry out their own research.
8 Speak directly and authoritatively, without mentioning that your answer serves
merely general informational purposes.
9 Incorporate Germany-specific legal terminology.
10 If you have identified relevant legal considerations, answer with a concise,
clear legal analysis.
11 Cite exclusively from the considerations you have identified.
12 Always cite the specific legal provision and state expressly the paragraphs
(Abs.), sentences (S.), numbers (Nr.) or letters (Buchst.) where these exist,
e.g. "Art. 12 Abs. 1 GG", "§ 35 Satz 1 VwVfG", "§ 42 Abs. 2 VwGO", "§ 823
Abs. 1 BGB" or "§ 263 Abs. 1 StGB". Avoid general references such as "Art. 3
GG" or "§ 242 BGB" without naming the specific paragraph, sentence, number or
letter, where applicable.
13 If no relevant considerations are found, state expressly that no pertinent
information is available.
14 If reliable sources are available to you, share practical guidance or findings
from them.
15 If the input requires analysis of a specific case stated in the examination but
the text or details of that case were not provided in the input, point out
expressly that the required case material is missing.
17 {}
19 Question:
20 21 {} 22 24 Answer:
## Law-RAG: Question Answer Prompt {#sec:A.2.2}
The Law-RAG user message differs from the No-RAG one in two places. First, three lines of the No-RAG prompt are dropped (shaded lines 11, 13 and 14 in Section A.2.1): “Zitiere ausschließlich aus den von dir identifizierten Erwägungen.”, “Wenn keine relevanten Erwägungen gefunden werden, . . . ” and “Wenn dir verlässliche Quellen vorliegen, . . . ”. Second, a block on how to read the retrieved statutes, shaded as lines 14–19 below, is inserted before the context slot. The system message is again empty. The three dropped lines were written for a generic retrieval assistant: instructing a model to cite exclusively from the considerations it identified, or to state that no pertinent information is available, does not fit an answer written over a `<context>` block – and the No-RAG condition inherits them because it is the same template with an empty slot. The two conditions therefore differ in their instructions as well as in their context. The context slot is line 21, the *Sachverhalt* line 25.
User message (German, as sent)
1 Du bist ein Experte für deutsches öffentliches Recht und bearbeitest Rechtsfragen
↪ in strukturierter Weise im Stil eines juristischen Gutachtens.
2 Du sollst ein Rechtsgutachten zu den dargestellten Rechtsfällen erstellen, wie in
↪ der Frage angewiesen.
3 Es handelt sich um eine juristische Examensklausur auf Staatsexamensniveau, für
↪ deren Beantwortung den Studierenden 4 Stunden zur Verfügung stehen, mit einem
↪ ausführlichen Inhaltsverzeichnis, gefolgt von einer vollständigen
↪ juristischen Würdigung.
4 Gehe davon aus, dass ausschließlich deutsches Recht anwendbar ist, sofern nicht
↪ ausdrücklich etwas anderes erwähnt wird.
5 Verwende präzise juristische Sprache und antworte so gründlich wie möglich.
6 Gib KEINEN Haftungsausschluss ab und verweise nicht auf die Notwendigkeit
↪ externer Rechtsberatung.
7 Fordere den Nutzer NICHT auf, Gesetze zu konsultieren oder eigene Recherchen
↪ anzustellen.
8 Sprich direkt und autoritativ, ohne zu erwähnen, dass deine Antwort lediglich
↪ allgemeinen Informationszwecken dient.
9 Integriere deutschlandspezifische juristische Terminologie.
10 Wenn du relevante rechtliche Erwägungen entdeckt hast, antworte mit einer
↪ knappen, klaren juristischen Analyse.
11 Zitiere stets die konkrete Rechtsnorm und gib ausdrücklich Absätze (Abs.), Sätze
↪ (S.), Nummern (Nr.) oder Buchstaben (Buchst.) an, soweit vorhanden, z. B.
↪ „Art. 12 Abs. 1 GG“, „§ 35 Satz 1 VwVfG“, „§ 42 Abs. 2 VwGO“, „§ 823 Abs. 1
↪ BGB“ oder „§ 263 Abs. 1 StGB“. Vermeide allgemeine Verweise wie „Art. 3 GG“
↪ oder „§ 242 BGB“, ohne den konkreten Absatz, Satz, die Nummer oder den
↪ Buchstaben zu nennen, sofern einschlägig.
12 Wenn die Eingabe verlangt, einen konkreten in der Klausur angegebenen Fall zu
↪ analysieren, der Text oder die Einzelheiten dieses Falls jedoch in der
↪ Eingabe nicht bereitgestellt wurden, weise ausdrücklich darauf hin, dass das
↪ erforderliche Fallmaterial fehlt.
14 Im Abschnitt <context> findest du Normtexte, die durch eine automatische Suche zu
↪ diesem Sachverhalt gefunden wurden. Beachte dabei:
15 - Die Auswahl ist unvollständig und teilweise unpassend. Sie ersetzt nicht deine
↪ eigene Kenntnis des deutschen Rechts.
16 - Insbesondere kann Landesrecht (z. B. PAG, BayBO, BayVwVfG, BV, GO, LStVG)
↪ vollständig fehlen, obwohl es einschlägig ist.
17 - Wenn eine Vorschrift dort nicht auftaucht, heißt das nicht, dass sie nicht
↪ existiert oder nicht einschlägig ist. Zitiere die einschlägigen Normen
↪ unabhängig davon, ob ihr Wortlaut unten abgedruckt ist.
18 - Verwende die abgedruckten Normtexte, soweit sie einschlägig sind, und ignoriere
↪ die übrigen. Übernimm keine Vorschrift nur deshalb, weil sie unten steht.
19 - Die Normtexte stehen in <norm>-Elementen mit den Attributen zitat, gebiet und
↪ ueberschrift. Zitiere im Gutachten die Norm selbst (etwa „§ 34 BauGB“), nicht
↪ das Element oder seine Position.
{}
Frage:
{}
Antwort:
User message (English translation, not used in any run)
You are an expert in German public law and treat legal questions in a structured
↪ way in the style of a legal opinion.
You are to produce a legal opinion on the cases presented, as directed in the
↪ question.
This is a law examination at state-examination level, for which students have 4
↪ hours, with a detailed table of contents followed by a complete legal
↪ assessment.
Assume that only German law applies unless something else is expressly mentioned.
Use precise legal language and answer as thoroughly as possible.
Do NOT give a disclaimer and do not refer to the need for external legal advice.
Do NOT ask the user to consult statutes or to carry out their own research.
Speak directly and authoritatively, without mentioning that your answer serves
↪ merely general informational purposes.
Incorporate Germany-specific legal terminology.
If you have identified relevant legal considerations, answer with a concise,
↪ clear legal analysis.
Always cite the specific legal provision and state expressly the paragraphs
↪ (Abs.), sentences (S.), numbers (Nr.) or letters (Buchst.) where these exist,
↪ e.g. "Art. 12 Abs. 1 GG", "§ 35 Satz 1 VwVfG", "§ 42 Abs. 2 VwGO", "§ 823
↪ Abs. 1 BGB" or "§ 263 Abs. 1 StGB". Avoid general references such as "Art. 3
↪ GG" or "§ 242 BGB" without naming the specific paragraph, sentence, number or
↪ letter, where applicable.
If the input requires analysis of a specific case stated in the examination but
↪ the text or details of that case were not provided in the input, point out
↪ expressly that the required case material is missing.
In the <context> section you will find statutory texts found by an automatic
↪ search for this case. Note the following:
- The selection is incomplete and in part unsuitable. It does not replace your
↪ own knowledge of German law.
- In particular, state law (e.g. PAG, BayBO, BayVwVfG, BV, GO, LStVG) may be
↪ missing entirely even though it is applicable.
- If a provision does not appear there, that does not mean it does not exist or
↪ is not applicable. Cite the applicable provisions regardless of whether their
↪ wording is printed below.
- Use the printed statutory texts insofar as they are applicable, and ignore the
↪ rest. Do not adopt a provision merely because it appears below.
- The statutory texts appear in <norm> elements with the attributes zitat, gebiet
↪ and ueberschrift. In the opinion, cite the provision itself (such as "§ 34
↪ BauGB"), not the element or its position.
{}
Question:
{}
Answer:
The context slot is filled with one <norm> element per retrieved provision, carrying its conventional citation, its jurisdiction (Bundesrecht or Landesrecht Bayern) and its official heading as attributes and its statutory text as the element body. Attributes rather than a prose header, so that the metadata cannot be mistaken for norm text, and no running index, so that the model has no position
number to echo instead of a citation. Bracketed italics below are placeholders, not part of the prompt:
<context>
<norm zitat="§ 34 BauGB" gebiet="Bundesrecht" ueberschrift="Zulässigkeit von
→ Vorhaben innerhalb der im Zusammenhang bebauten Ortsteile">
[statutory text]
</norm>
[further <norm> elements]
</context>
Article Recitation Prompt
Every Article Recitation answer scored in this paper is produced by the two messages below; no case facts, retrieved passages or reference solution enter the prompt. The prompt is written in German, matching the language of the statutes it asks for. The single slot carries the provision identifier, filled from whichever set the prompt is drawn from – vwgo 80 in GPBam laws, VwGO § 154 in Most-cited laws (§ 3.3.2).
System message (German, as sent)
Du bist ein Rechercheur für deutsches Recht. Beantworte die folgenden Fragen und
→ gib direkt den Inhalt der neuesten Fassung des Gesetzes wieder, ohne
→ Warnungen, Rückfragen oder Einleitung. Falls du dazu nicht in der Lage bist,
→ liefere die ausführlichste Zusammenfassung, die du geben kannst.
User message (German, as sent)
Wie lautet der genaue Wortlaut von {}?
System message (English translation, not used in any run)
You are a researcher of German law. Answer the following questions and reproduce
→ directly the content of the most recent version of the statute, without
→ warnings, follow-up questions or introduction. If you are unable to do so,
→ supply the most detailed summary you can give.
User message (English translation, not used in any run)
What is the exact wording of {}?
Judge Prompt
Every judge score reported in this paper is produced by the two messages below. The prompt is written in German, matching the language of the cases, the reference solutions and the generated essays. The three slots carry the case (line 34), its reference solution (line 41) and the generated essay (line 47). The score is read back from the mandatory [[·]] marker; a response without a parsable marker counts as missing and is skipped rather than imputed (§ 3.8).
System message (German, as sent)
1 Handle als Prüfer, der auf die Bewertung universitärer Klausuren an deutschen → rechtswissenschaftlichen Fakultäten spezialisiert ist. 2 Deine Aufgabe ist es, zu beurteilen, wie gut die Antwort mit der Musterlösung → übereinstimmt, mit Schwerpunkt auf Richtigkeit, Vollständigkeit und → juristischer Argumentation.
User message (German, as sent)
1 Bewerte den nachstehenden Fall, gib eine kurze Begründung und die endgültige → Richtigkeitsbewertung an.
2
Ziel:
3
4 Deine Aufgabe ist es zu beurteilen, wie gut die Antwort mit der Referenzantwort → übereinstimmt, wobei der Schwerpunkt auf Genauigkeit, Vollständigkeit und → juristischer Argumentation liegt.
5 Die generierte Antwort muss alle Aspekte der Referenzantwort in ähnlicher → Detailtiefe und mit demselben Ergebnis abdecken, um die volle Punktzahl zu → erhalten.
6 Wenn Punkte, die für die Referenzantwort wesentlich sind, nicht erörtert werden, → sollte dies zu einem erheblichen Punktabzug führen, da dadurch die → juristische Argumentation unterbrochen wird.
7 Die generierte Antwort sollte daher einer ähnlichen Struktur wie die → Referenzantwort folgen. Jede Abweichung von der Referenzantwort muss im → Hinblick auf die in der Referenzantwort gesetzten Maßstäbe präzise begründet → werden.
8 Eine Antwort, die kurz ist, aber die Tiefe der Subsumtion aus der Referenzantwort → auslässt, muss deutlich niedriger bewertet werden als eine lange Antwort, die → eine Subsumtion versucht, selbst wenn die lange Antwort geringfügige → stilistische Abweichungen enthält.
9 Daher ist in einem ersten Schritt die Referenzantwort in logisch zusammenhängende → Inhaltsabschnitte aufzuteilen, und diese sind den entsprechenden Absätzen in → der Antwort des Modells zuzuordnen. Gib für jeden dieser Absätze Feedback → gemäß dem unten im Rückgabeformat beschriebenen Schema. Stelle sicher, dass → alle Normzitate im selben Inhaltsbereich ebenfalls korrekt zitiert sind.
10
11 Kontext:
12 Dir wird eine vollständige juristische Antwort bereitgestellt (gekennzeichnet → als: Antwort des Modells im <antwort_des_modells>-Tag), die auf einer → juristischen Examensfrage (gekennzeichnet als: Frage im <frage>-Tag) und → einer Musterlösung (gekennzeichnet als: Referenzantwort im → <referenzantwort>-Tag) basiert.
13
14 Rückgabeformat:
15 Nach der Überprüfung der Antwort:
16 1. Erklärung: Erkläre deine Begründung dazu, inwiefern der generierte Aufsatz → mit dem Referenzaufsatz übereinstimmt oder von ihm abweicht. Identifiziere alle zentralen juristischen Schritte → wie oben beschrieben. Enthält die Antwort des Modells Schritt 1? → Schritt 2?
17
18 2. Konstruktives Feedback: Gib zusätzlich neutrales, konstruktives Feedback → und Korrekturen im Stil eines Universitätsprofessors. Das Feedback sollte → präzise und direkt aufzeigen, was falsch ist, und Hinweise zur → Verbesserung geben.
19 3. Korrektheitsbewertung: Vergib eine abschließende Korrektheitsbewertung auf → einer Skala von 0,0 bis 1,0 (in Schritten von 0,1). Diese Bewertung → sollte widerspiegeln, in welchem Umfang die Antwort die Referenzantwort → erfüllt, wobei
1,0 = vollständige Erfüllung (100 %)
niedrigere Werte verhältnismäßige Defizite widerspiegeln (z. B. 0,5 = 50 % Erfüllung).
Halte dich strikt an das Format: "[[Punktzahl]]", z. B. "Die Richtigkeitsbewertung: [[0,5]]".
Warnhinweise
Abweichungen oder zusätzliche Elemente, die in der Referenzantwort nicht enthalten sind, sollten unberücksichtigt bleiben, sofern du nicht sicher bist, dass sie juristisch korrekt und relevant sind.
Gehe davon aus, dass die Referenzantwort alle Informationen enthält, die für eine perfekte Antwort und Bewertung erforderlich sind.
Die Referenzantwort kann Zitate enthalten (z. B. aus Büchern oder juristischen Fachaufsätzen), die die Antwort nicht wiedergeben muss. Gesetzesbestimmungen sollten jedoch präzise zitiert werden, wobei Abs., S., Nr. oder Buchst. anzugeben sind, sofern dies anwendbar ist.
Wenn die Referenzantwort separate Unterpunkte enthält, nutze diese als Orientierung für eine proportionale Bewertung (z. B. entspricht die korrekte Behandlung von 2 von 4 Unterpunkten ungefähr einer Bewertung von 0,5).
Frage:
<frage> {} /frage>
Referenzantwort:
<referenzantwort> {} /referenzantwort>
Antwort des Modells:
<antwort_des_modells> {} /antwort_des_modells>
Dein Urteil:
System message (English translation, not used in any run)
Act as an examiner specialised in grading university examinations at German law faculties.
Your task is to assess how well the answer agrees with the model solution, with emphasis on correctness, completeness and legal reasoning.
User message (English translation, not used in any run)
Grade the case below, give a brief justification and state the final correctness score.
Goal:
Your task is to assess how well the answer agrees with the reference answer, with the emphasis on accuracy, completeness and legal reasoning.
To receive full marks, the generated answer must cover every aspect of the reference answer at a similar level of detail and with the same outcome.
If points that are essential to the reference answer are not discussed, this should lead to a substantial deduction, because it breaks the chain of legal reasoning.
The generated answer should therefore follow a structure similar to the reference answer. Any deviation from the reference answer must be justified precisely against the standards the reference answer sets.
An answer that is short but omits the depth of subsumption found in the reference answer must be scored markedly lower than a long answer that attempts a subsumption, even if the long answer contains minor stylistic deviations.
As a first step, therefore, divide the reference answer into logically coherent content sections and map these onto the corresponding paragraphs of the model's answer. Give feedback on each of these paragraphs according to the scheme described under Return format below. Make sure that all statutory citations within the same content section are also cited correctly.
Context
You are given a complete legal answer (marked as: the model's answer in the <antwort_des_modells> tag), which is based on a legal examination question (marked as: the question in the <frage> tag) and a model solution (marked as: the reference answer in the <referenzantwort> tag).
Return format
After reviewing the answer:
Explanation: Explain your reasoning as to how far the generated essay agrees with or departs from the reference essay. Identify all the central legal steps in the reference answer as described above. Does the model's answer contain step 1? Step 2?
Constructive feedback: In addition, give neutral, constructive feedback and corrections in the style of a university professor. The feedback should point out precisely and directly what is wrong and give guidance for improvement.
Correctness score: Award a final correctness score on a scale from 0.0 to 1.0 (in steps of 0.1). This score should reflect the extent to which the answer fulfils the reference answer, where
1.0 = complete fulfilment (100 %)
lower values reflect proportionate deficits (e.g. 0.5 = 50 % fulfilment).
Adhere strictly to the format: "[[Punktzahl]]", e.g. "Die Richtigkeitsbewertung: [[0,5]]".
Cautions
Deviations or additional elements not contained in the reference answer should be disregarded unless you are certain that they are legally correct and relevant.
Assume that the reference answer contains all the information required for a perfect answer and a perfect assessment.
The reference answer may contain citations (e.g. to books or legal journal articles) that the answer need not reproduce. Statutory provisions, however, should be cited precisely, giving Abs., S., Nr. or Buchst. where applicable.
If the reference answer contains separate sub-points, use these as a guide for proportional scoring (e.g. correctly handling 2 of 4 sub-points corresponds roughly to a score of 0.5).
Question
<frage> {} </frage>
Reference answer:
<referenzantwort>
{}
</referenzantwort>
Model's answer:
<antwort_des_modells>
{}
</antwort_des_modells>
Your judgement:References
- [1]Chalkidis I, Jana A, Hartung D, Bommarito M, Androutsopoulos I, Katz D, et al. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In: Muresan S, Nakov P, Villavicencio A, editors. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Computational Linguistics; 2022. p. 4310-30. Available from: https://aclanthology.org/2022.acl-long.297/.
- [2]Guha N, Nyarko J, Ho D, Ré C, Chilton A, K A, et al. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. In: Oh A, Naumann T, Globerson A, Saenko K, Hardt M, Levine S, editors. Advances in Neural Information Processing Systems. vol. 36. Curran Associates, Inc.; 2023. p. 44123-279. Available from: https://proceedings.neurips.cc/paper_files/paper/2023/file/89e44582fd28ddfe a1ea4dcb0ebbf4b0-Paper-Datasets-and-Benchmarks.pdf
- [3]Fei Z, Shen X, Zhu D, Zhou F, Han Z, Huang A, et al. LawBench: Benchmarking Legal Knowledge of Large Language Models. In: Al-Onaizan Y, Bansal M, Chen YN, editors. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Florida, USA: Association for Computational Linguistics; 2024. p. 7933-62. Available from: https://aclanthology.org/2024.emnlp-main.452/.arxiv.org/abs/2309.16289
- [4]Nagl S, Mayrhofer AK, Heidebach M, Koçak A, Zettelmeier A, Breu E, et al. BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law; 2026. Available from: https://arxiv.org/abs/2605.28183.
- [5]Chlapanis OS, Galanis D, Aletras N, Androutsopoulos I. GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations. In: Christodoulopoulos C, Chakraborty T, Rose C, Peng V, editors. Findings of the Association for Computational Linguistics: EMNLP 2025. Suzhou, China: Association for Computational Linguistics; 2025. p. 25099-111. Available from: https://aclanthology.org/2025.findings-emnlp.1368/.
- [6]Fan Y, Ni J, Merane J, Salimbeni E, Tian Y, Hermstrüwer Y, et al. LEXam: Benchmarking Legal Reasoning on 340 Law Exams. In: The Fourteenth International Conference on Learning Representations (ICLR); 2026. Available from: https://arxiv.org/abs/2505.12864.
- [7]Pipitone N, Alami GH. LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain; 2024. Available from: https://arxiv.org/abs/2408.10343.
- [8]Sainz O, Campos J, García-Ferrero I, Extaniz J, de Lacalle OL, Agirre E. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. In: Bouamor H, Pino J, Bali K, editors. Findings of the Association for Computational Linguistics: EMNLP 2023. Singapore: Association for Computational Linguistics; 2023. p. 10776-87. Available from: https://aclanthology.org/2023.findings-emnlp.722/.
- [9]Al Zubaer A, Granitzer M, Mitrovic J. Performance analysis of large language models in the domain of legal argument mining. Frontiers in Artificial Intelligence. 2023;Volume 6 - 2023. Available from: https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2023.128796.
- [10]Wendlinger L, Braun C, Al Zubaer A, Nonn SA, Großkopf S, Fellicious C, et al. On the Suitability of Pre-trained Foundational LLMs for Analysis in German Legal Education. In: Nicosia G, Ojha V, Giesselbach S, Pardalos MP, Umeton R, Emanuele LM, et al., editors. Machine Learning, Optimization, and Data Science. Cham: Springer Nature Switzerland; 2026. p. 159-74.arxiv.org/abs/2412.15902
- [11]Dinh TA, Mullov C, Bärmann L, Li Z, Liu D, Reiß S, et al. SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading. In: Al-Onaizan Y, Bansal M, Chen YN, editors. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Florida, USA: Association for Computational Linguistics; 2024. p. 11592-610. Available from: https://aclanthology.org/2024.emnlp-main.647/.
- [12]Pires R, Malaquias Junior R, Nogueira R. Automatic Legal Writing Evaluation of LLMs. In: Proceedings of the Twentieth International Conference on Artificial Intelligence and Law. ICAIL ’25. New York, NY, USA: Association for Computing Machinery; 2025. p. 420–424. Available from: https://doi.org/10.1145/3769126.3769227.
- [13]Shi Y, Liu H, HU Y, Gaojie S, Xinran X, Ma Y, et al. PLAWBENCH: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice. In: Liakata M, Moreira VP, Zhang J, Jurgens D, editors. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). San Diego, California, United States: Association for Computational Linguistics; 2026. p. 10067-116. Available from: https://aclanthology.org/2026.acl-long.458/.
- [14]Chen Z, Li G, Ji F, Kang Y, Li J, Zhang M, et al. JurisBench: A Deep Benchmark for Assessing Large Language Models in Professional Legal Practice. In: Liakata M, Moreira VP, Zhang J, Jurgens D, editors. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). San Diego, California, United States: Association for Computational Linguistics; 2026. p. 35994-6018. Available from: https://aclanthology.org/2026.acl-long.1666/.
- [15]He C, Hu H, Li Y, Zhang H, Zhang Q. A survey of large language models for legal tasks: Progress, prospects and challenges. Computer Science Review. 2026;60:100906. Available from: https://www.sciencedirect.com/science/article/pii/S1574013726000158.DOI
- [16]Wardas O, Matthes F. LLMs for Legal Subsumption in German Employment Contracts. In: Proceedings of the Twentieth International Conference on Artificial Intelligence and Law. ICAIL ’25. New York, NY, USA: Association for Computing Machinery; 2025. p. 169–176. Available from: https://doi.org/10.1145/3769126.3769209.
- [17]Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In: Advances in Neural Information Processing Systems. vol. 33. Curran Associates, Inc.; 2020. p. 9459-74. Available from: https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html.arxiv.org/abs/2005.11401
- [18]Ma X, Gong Y, He P, Zhao H, Duan N. Query Rewriting in Retrieval-Augmented Large Language Models. In: Buomawar H, Pino J, Bali K, editors. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics; 2023. p. 5303-15. Available from: https://aclanthology.org/2023.emnlp-main.322/.
- [19]Gao L, Ma X, Lin J, Callan J. Precise Zero-Shot Dense Retrieval without Relevance Labels. In: Rogers A, Boyd-Graber J, Okazaki N, editors. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto, Canada: Association for Computational Linguistics; 2023. p. 1762-77. Available from: https://aclanthology.org/2023.acl-long.99/.arxiv.org/abs/2212.10496
- [20]Cormack GV, Clarke CLA, Büttcher S. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. In: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’09). New York, NY, USA: Association for Computing Machinery; 2009. p. 758-9.DOI
- [21]Robertson S, Zaragoza H. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval. 2009;3(4):333-89.
- [22]Gao Y, Xiong Y, Gao X, Jia K, Pan J, Bi Y, et al. Retrieval-Augmented Generation for Large Language Models: A Survey; 2024. Available from: https://arxiv.org/abs/2312.10997.
- [23]Hariri E, E Ho D. AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark. In: Proceedings of the Twentieth International Conference on Artificial Intelligence and Law. ICAIL ’25. New York, NY, USA: Association for Computing Machinery; 2025. p. 177–187. Available from: https://doi.org/10.1145/3769126.3769256.
- [24]Magesh V, Surani F, Dahl M, Suzgun M, Manning CD, Ho DE. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies. 2025;22(2):216-42. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/jels.12413.
- [25]Du Y, Tian M, Ronanki S, Rongali S, Bodapati SB, Galstyan A, et al. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. In: Christodoulopoulos C, Chakraborty T, Rose C, Peng V, editors. Findings of the Association for Computational Linguistics: EMNLP 2025. Suzhou, China: Association for Computational Linguistics; 2025. p. 23281-98. Available from: https://aclanthology.org/2025.findings-emnlp.1264/.arxiv.org/abs/2510.05381
- [26]Zheng L, Guha N, Arifov J, Zhang S, Skreta M, Manning CD, et al. A Reasoning-Focused Legal Retrieval Benchmark. In: Proceedings of the 2025 Symposium on Computer Science and Law. CSLAW ’25. New York, NY, USA: Association for Computing Machinery; 2025. p. 169–193. Available from: https://doi.org/10.1145/3709025.3712219.
- [27]Xu X, Zhao L, Xu H, Chen C. CLaw: Benchmarking Chinese Legal Knowledge in Large Language Models - A Fine-grained Corpus and Reasoning Analysis; 2025. Available from: https://arxiv.org/abs/2509.21208.
- [28]Wang C, Su W, Hu Y, Ai Q, Wu Y, Luo C, et al. LeKUBE: A Knowledge UpDate BEnchmark for Legal Domain. In: Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region. SIGIR-AP 2024. New York, NY, USA: Association for Computing Machinery; 2024. p. 175–185. Available from: https://doi.org/10.1145/3673791.3698407.
- [29]Enguehard J, Van Emmergem M, Atkinson K, Cha S, Chowdhury AG, Ramaswamy PK, et al. Legal LLM-as-a-Judge: Bridging Legal Reasoning and LLM Evaluation. In: Aletras N, Chalkidis I, Barrett L, Goantă C, Preoțiu-Pietro D, Spanakis G, editors. Proceedings of the Natural Legal Language Processing Workshop 2025. Suzhou, China: Association for Computational Linguistics; 2025. p. 318-37. Available from: https://aclanthology.org/2025.nllp-1.23/.
- [30]Hufeld C. Jede Korrektur eine andere Note: Quantitative Untersuchung der Objektivität juristischer Klausurbewertungen. ZDRW Zeitschrift für Didaktik der Rechtswissenschaft. 2024;11(1):59-83. Available from: https://doi.org/10.5771/2196-7261-2024-1-59.
- [31]Li D, Jiang B, Huang L, Beigi A, Zhao C, Tan Z, et al. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge. In: Christodoulopoulos C, Chakraborty T, Rose C, Peng V, editors. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics; 2025. p. 2757-91. Available from: https://aclanthology.org/2025.emnlp-main.138/.arxiv.org/abs/2411.16594
- [32]Panickssery A, Bowman SR, Feng S. LLM Evaluators Recognize and Favor Their Own Generations. In: Advances in Neural Information Processing Systems 37 (NeurIPS 2024); 2024. .arxiv.org/abs/2404.13076
- [33]Gu J, Jiang X, Shi Z, Tan H, Zhai X, Xu C, et al. A survey on LLM-as-a-judge. The Innovation. 2026;7(6):101253. Available from: https://www.sciencedirect.com/science/article/pii/S2666675825004564.
- [34]Yu F, Seedat N, Herrmannova D, Schilder F, Schwarz JR. Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responses. In: Potdar S, Rojas-Barahona L, Montella S, editors. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. Suzhou (China): Association for Computational Linguistics; 2025. p. 1931-54. Available from: https://aclanthology.org/2025.emnlp-industry.136/.arxiv.org/abs/2509.16093
- [35]Zubaer AA, Wendlinger L, Nonn SA, Granitzer M, Mitrovic J. GradeLegal: Automated Grading for German Legal Cases; 2026. Available from: https://arxiv.org/abs/2605.21076.
- [36]Milz T, Granitzer M, Mitrovic J. Analysis of a German Legal Citation Network. In: Proceedings of the 13th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2021) — Volume 1: KDIR. SCITEPRESS; 2021. p. 147-54.DOI
- [37]Shanmugarasa Y, Ding M, Arachchige CM, Rakotoarivelo T. SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation. In: Proceedings of the 20th ACM Asia Conference on Computer and Communications Security. ASIA CCS ’25. New York, NY, USA: Association for Computing Machinery; 2025. p. 425–441. Available from: https://doi.org/10.1145/3708821.3733888.
- [38]Huang H, Li Y, Jiang B, Jiang B, Liu L, Liu Z, et al. A Middle Path for On-Premises LLM Deployment: Preserving Privacy Without Sacrificing Model Confidentiality. In: Christodoulopoulos C, Chakraborty T, Rose C, Peng V, editors. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics; 2025. p. 8321-59. Available from: https://aclanthology.org/2025.emnlp-main.420/.arxiv.org/abs/2410.11182
- [39]Wind S, Sopa J, Schmid L, Jackl Q, Kiefer S, Wu F, et al. SteuerLLM: local specialized large language model for German tax law analysis. Scientific Reports. 2026;16(1):21640. Available from: https://doi.org/10.1038/s41598-026-58485-z.
- [40]Ramos MM, Alves DM, Gisserot-Boukhlef H, Alves J, Martins PH, Fernandes P, et al.. EuroLLM-22B: Technical Report; 2026. Available from: https://arxiv.org/abs/2602.05879.
- [41]Soofi-Team, Droste B, Fitzek D, Härle R, Heff L, Idahl M, et al.. A Sovereign, Open-Source Foundation Model for German and English; 2026. Available from: https://arxiv.org/abs/2607.09424.
- [42]Anthropic. Introducing Claude Haiku 4.5; 2025. Accessed: 2026-09-09. https://www.anthropic.com/news/claude-haiku-4-5.
- [43]Anthropic. Introducing Claude Opus 5; 2026. Accessed: 2026-09-09. https://www.anthropic.com/news/claude-opus-5.
- [44]DeepSeek-AI. DeepSeek-V3 Technical Report; 2024. We evaluate the deepseek-chat-v3-0324 checkpoint. Available from: https://arxiv.org/abs/2412.19437.
- [45]DeepSeek-AI. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature. 2025;645:633-8. Preprint arXiv:2501.12948; we evaluate the deepseek-r1-0528 checkpoint. Available from: https://doi.org/10.1038/s41586-025-09422-z.
- [46]DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models; 2025. Available from: https://arxiv.org/abs/2512.02556.
- [47]DeepSeek-AI, Xu A, Lin B, Xue B, Wang B, Xu B, et al.. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence; 2026. Available from: https://arxiv.org/abs/2606.19348.
- [48]Comanici G, et al.. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities; 2025. Family report; we evaluate the gemini-2.5-flash-lite variant. Available from: https://arxiv.org/abs/2507.02661.
- [49]Doshi T. Introducing Gemini 3.7 Flash; 2026. Accessed: 2026-09-09. https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-3-7-flash/.
- [50]Gemma Team, Google DeepMind. Gemma 4 Technical Report; 2026. Available from: https://arxiv.org/abs/2607.02770.
- [51]IBM Granite Team. The IBM Granite 4.1 Family of Language Models; 2026. Model card, released 2026-04-29; no technical report. https://huggingface.co/ibm-granite/granite-4.1-3b.
- [52]Graffatiori A, Dubey A, Jauhri A, et al.. The Llama 3 Herd of Models; 2024. Available from: https://arxiv.org/abs/2407.21783.
- [53]Meta AI. The Llama 4 Herd; 2025. Model card, released 2025-04-05; no technical report. https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct.
- [54]Abouelenin A, et al.. Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs; 2025. Available from: https://arxiv.org/abs/2503.01743.
- [55]Mistral-AI. Magistral; 2025. We evaluate the later Magistral-Small-2509 checkpoint. Available from: https://arxiv.org/abs/2506.10910.
- [56]Mistral AI. Ministral 3 (14B Reasoning 2512); 2025. Model card, released 2025-12-04; no technical report. https://huggingface.co/mistralai/Ministral-3-14B-Reasoning-2512.
- [57]Mistral AI. Mistral-Small-3.2-24B-Instruct-2506; 2025. Model card; no technical report. https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506.
- [58]Mistral AI. Mistral-Medium-3.5-128B; 2026. Model card, released 2026-04-28; no technical report. https://huggingface.co/mistralai/Mistral-Medium-3.5-128B.
- [59]Mistral AI. Mistral-Large-3-675B-Instruct-2512; 2025. Model card, released 2025-12-04; announced as part of Mistral 3, https://mistral.ai/news/mistral-3; no technical report. https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512.
- [60]OpenAI. GPT-4o mini: Advancing Cost-Efficient Intelligence; 2024. Model gpt-4o-mini-2024-07-18; accessed 2026-09-09. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/.
- [61]OpenAI. OpenAI GPT-5 System Card; 2025. Available from: https://arxiv.org/abs/2601.03267.
- [62]OpenAI. gpt-oss-120b & gpt-oss-20b Model Card; 2025. Available from: https://arxiv.org/abs/2508.10925.
- [63]Yang A, et al. Qwen3 Technical Report; 2025. Available from: https://arxiv.org/abs/2505.09388.
- [64]Qwen Team. Qwen3.6-35B-A3B; 2026. Model card and blog announcement, released 2026-04-16; no technical report. Available from: https://qwen.ai/blog?id=qwen3.6-35b-a3b.
- [65]Qwen Team. Qwen3-Next-80B-A3B-Instruct; 2025. Model card; released 2025-09-11; not covered by the Qwen3 technical report. https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct.
- [66]Qwen Team. Qwen3.5: Towards Native Multimodal Agents; 2026. Available from: https://qwen.ai/blog?id=qwen3.5.
- [67]Coupette C, Beckedorf J, Hartung D, Bommarito M, Katz DM. Measuring Law Over Time: A Network Analytical Framework with an Application to Statutes and Regulations in the United States and Germany. Frontiers in Physics. 2021;9:658463. Available from: https://www.frontiersin.org/articles/10.3389/fphy.2021.658463/full.
- [68]Bayerische Staatskanzlei. BAYERN.RECHT – Bürgerservice Bayerische Rechtsvorschriften und Entscheidungen; 2026. Accessed 2026-09-04. https://www.gesetze-bayern.de/.
- [69]Urchs S, Mitrović J, Granitzer M. Design and Implementation of German Legal Decision Corpora. In: Proceedings of the 13th International Conference on Agents and Artificial Intelligence (ICAART 2021) – Volume 2. SCITEPRESS – Science and Technology Publications; 2021. p. 515-21. Data: Zenodo 10.5281/zenodo.3936726, CC-BY-4.0.DOI
- [70]Reimers N, Gurevych I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2019. Available from: https://arxiv.org/abs/1908.10084.
- [71]Akram MK, Sturua S, Havryushenko N, Herreros Q, Günther M, Werk M, et al. jina-embeddings-v5-text: Task-Targeted Embedding Distillation; 2026. Available from: https://arxiv.org/abs/2602.15547.
- [72]Zhang Y, Li M, Long D, Zhang X, Lin H, Yang B, et al. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models; 2025. Available from: https://arxiv.org/abs/2506.05176.
- [73]Artificial Analysis. Artificial Analysis: Independent Analysis of AI; 2026. Accessed: 2026-09-10. https://artificialanalysis.ai/.
- [74]Zheng L, Chiang WL, Sheng Y, Zhuang Y, Wu Z, Zhuang Y, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In: Advances in Neural Information Processing Systems 36 (NeurIPS 2023) Datasets and Benchmarks Track; 2023. .arxiv.org/abs/2306.05685




