betaFeedback

WorksComputer Science

Show Your Work: Can Small Open Models Read SEC Filings with Evidence a Machine Can Check?

Abstract

A fluent answer from a language model is not a verified one. We ask whether small open models that run on a single consumer GPU can extract numbers from SEC filings with evidence a machine can check: the page and a verbatim quote containing the number. We introduce Filing-Evidence, a benchmark of 1,000 questions over 90 recent 10-K and 10-Q filings of 45 US companies, with answer keys taken from the filers' inline XBRL tags. We evaluate ten instruct models from ten developers (7-21B parameters) on one RTX 4090 in three setups: plain, check-and-retry, and the model's own reasoning mode. Accuracy with checkable evidence ranges from 17.1% to 82.7% in the plain setup, and the best result is 84.5% (Qwen3.5-9B). An automatic checker rejects many wrong answers, but between 16 and 167 answers per 1,000 give a wrong value for an answerable question and still pass every check. In the plain setup, 57% of these are scale or unit errors and 30% take another period or column of the right line. Reasoning significantly reduces such errors for 4 of 5 models that offer it. Retrying after a failed check raises accuracy by up to 3.9 points but never reduces them. Questions, model outputs, code and protocol are public.

Introduction

An analyst who copies a number from a filing can show where it came from. For a language model to be trusted with the same task, its answers should be auditable: each number should come with evidence that software can verify. This paper measures how far small open models are from that standard when they must name the page and quote the exact line containing the number.

We study models that fit on one 24 GB consumer GPU, because an organisation can run them on its own hardware and documents. Three research questions were fixed in a written protocol before the main runs:

RQ1. How reliably do these models extract a requested number with automatically checkable evidence, and answer “not found” when the number is absent from the pages provided?

RQ2. In which kinds of filing content, and with which kinds of error, do they fail?

RQ3. How much do two training-free interventions help, automatic checking with one retry and the model’s own reasoning mode, and at what cost?

Our contributions are: (1) a benchmark whose answer keys come from filers’ machine-readable tags; (2) a model-independent checker that separates wrong answers it can reject from wrong answers that pass; (3) a paired comparison of ten open models in up to three setups; and (4) an analysis showing that most errors passing the checker are misread units or columns on the correct line.

Related work

Financial question answering has been studied with FinQA ([2]), TAT-QA ([13]), FinanceBench ([5]) and DocFinQA ([9]). These benchmarks focus mainly on answer correctness and numerical reasoning. Work on attribution evaluates whether generated statements are supported by cited sources ([1, 3, 7]). Filing-Evidence combines the two concerns in a narrow, fully automatic setting: every answer must carry a verbatim quote, and correctness is graded against inline XBRL ([12]).

Benchmark design

Figure 1. Construction and evaluation pipeline.

Filings and answer key. We used the most recent 10-K and 10-Q filed on or before 15 November 2026 by 45 large US companies, 90 filings in total (89 filed in 2026), obtained from SEC EDGAR ((SEC, 2026)). Companies span technology, banking, insurance, REITs, utilities, healthcare, consumer, industrial and other sectors. Gold answers come from inline XBRL facts displayed on a page, with their value, unit, scale, period, dimensions and evidence line.

Questions. Each question names an item, company and period and is shown with three pages: the answer page, an adjacent page and a random distractor, in random order. The benchmark has eight slices of 125 questions (Table 1). In S8 the requested figure is absent from the pages and the correct response is “not found”.

SliceContentCompanies
S1Main statements, current period44
S2Main statements, prior period44
S310-Q quarter vs year-to-date36
S4Notes tables44
S5Segments29
S6Numbers in prose43
S7Units and per-share42
S8Not in the provided pages45

Table 1. Benchmark slices (125 questions each).

Quality control. Before the main runs, stratified samples of 100 questions were reviewed against the rendered filing. The review was performed by an AI assistant under the author’s direction. Rounds one to three scored 87, 76 and 96 of 100, failing on question wording; the builder was revised after each. Round four scored 98 of 100, meeting the protocol’s threshold of 98, and the dataset was frozen with a SHA-256 fingerprint (4fa8f252292167fb...). All 400 reviewed gold values were judged correct. The author did not independently verify the full sample (§ 7).

Checker and metrics. Models return JSON with a value, scale, page and quoted line, or not_found. An answer passes when it parses, cites a page that was shown, quotes text verbatim from that page, and the value appears in the quote (Figure 2). The prompt asks for the complete line, but the checker accepts any contiguous passage after normalising whitespace: 923 of the 5,293 correct plain-setup answers across the ten models quote something other than one whole line. Sign is not graded because filers display negative values inconsistently. Accuracy is the share of the 875 answerable questions answered correctly with passing evidence. The confident-wrong rate is the share of all 1,000 answers that give a wrong value for an answerable question and pass every check. Answers that invent a figure on S8 are counted separately, as failed abstentions. Abstention accuracy is the share of S8 questions answered “not found”. Rates carry 95% Wilson intervals ([11]). Paired comparisons use exact McNemar tests ([8]) with Holm correction ([4]) at α=0.05\alpha= 0.05. For each metric, the 23 within-model setup comparisons form one family (Appendix § 10), and the pairwise model comparisons within each setup form another.

Figure 2. The checker. It uses the answer and the pages only; comparison with the answer key happens afterwards, for scoring.

Models and setups

All models were served with vLLM 0.33.0 [6] on one NVIDIA RTX 4090 at temperature 0, using each model’s chat template and one shared prompt (Table 2). Qwen3-8B, the model of an earlier pilot, is a supplementary baseline and is excluded from model rankings. The setups are shown in Figure 3. C2 reuses C1’s first answer and retries only answers that fail a check; the retry message names the failed check and never reveals the answer. gpt-oss-20b has no non-reasoning mode, so its C1 and C2 use low reasoning effort and its C3 uses high effort.

ModelDeveloperParamsPrecisionCreatedReasoningLicence
Qwen3.5-9BAlibaba9.7Bbf162026-02yesApache 2.0
Granite-4.2-8BIBM8.8Bbf162026-08yesApache 2.0
Ministral-3-8BMistral8.9Bbf162025-10noApache 2.0
OLMo-3-7BAllenAI7.3Bbf162025-11noApache 2.0
Nemotron-Nano-9B-v2NVIDIA8.9Bbf162025-08yesNVIDIA Open Model
gpt-oss-20bOpenAI20.9Bmxfp42025-08yesApache 2.0
Phi-4Microsoft14.7Bfp82024-12noMIT
Gemma-3-12BGoogle12.2Bfp82025-03noGemma Terms
GLM-4.6V-FlashZhipu (Z.ai)10.3Bbf162025-12yesMIT
Llama-3.1-8BMeta8.0Bbf162024-07noLlama 3.1
Qwen3-8B*Alibaba8.2Bbf162025-04yesApache 2.0

Table 2. Models evaluated. Parameter counts are as published and may include unused vision components. * Supplementary baseline.

Figure 3. The three experimental setups.

Results

RQ1: Reliability

Table 3 and Figure 4 summarise the results. In the plain setup Qwen3.5-9B reaches 82.7% (95% CI 80.1–85.1%) and is significantly more accurate than every other model (Figure 5). Figure 6 shows where the answers go. Abstention accuracy ranges from 77.6% to 100.0%. Across the ten models, 73 invented answers to S8 questions passed every check in the plain setup (0 to 24 per model); these are not included in the confident-wrong counts.

Figure 4. Accuracy against confident-wrong rate for each model and setup. Up and to the left is better.

Figure 5. Pairwise differences in plain-setup accuracy (exact McNemar, Holm-corrected over 45 pairs).

ModelC1 Plain Acc.C1 Plain CWC2 Retry Acc.C2 Retry CWC3 Thinking Acc.C3 Thinking CWAbst.
Qwen3.5-9B82.7%3684.5%3674.6%16124
gpt-oss-20b74.4%11277.4%11880.7%56112
Granite-4.2-8B73.3%6476.2%7281.3%38121
GLM-4.6V-Flash70.9%11073.0%11182.3%61118
Ministral-3-8B66.7%7769.6%86––124
Gemma-3-12B61.4%11764.3%134––97
Llama-3.1-8B58.9%10262.7%124––110
Phi-457.3%15160.8%167––116
Nemotron-Nano-9B-v242.3%10542.4%10554.4%72125
OLMo-3-7B17.1%7017.3%70––121
Qwen3-8B*66.1%9467.3%10174.1%3788

Table 3. Accuracy (of 875), confident-wrong answers (of 1,000) and abstention (of 125, plain setup). “–”: no reasoning mode. * Supplementary baseline.

Figure 6. Outcomes of the 875 answerable questions in the plain setup.

RQ2: Where models fail

Pooled over the ten models, plain-setup accuracy is 79.2% on current-period statements, but 46.0% on segments, 41.4% on numbers in prose and 50.5% on units and per-share items (Figure 7).

Plain-setup accuracy by model and sliceVector (PDF)
S1 CurrentS2 PriorS3 Qtr/YTDS4 NotesS5 SegmentsS6 ProseS7 UnitsS8 Absent
Qwen3.5-9B9493968871587999
gpt-oss-20b9783708153736490
Granite-4.2-8B9683838655426797
GLM-4.6V-Flash9272868268385894
Ministral-3-8B9365787662415399
Gemma-3-12B8280736150414478
Llama-3.1-8B7071866533414788
Phi-47267706430475093
Nemotron-Nano-9B-v262475650282626100
OLMo-3-7B341021211081697

Figure 7. Plain-setup accuracy by model and slice (S8: correct “not found”).

Error types are assigned by fixed rules in the grading code: the right digits at the wrong scale, another figure from the answer key’s evidence line, or otherwise a wrong item. The 944 plain-setup confident-wrong answers across the ten models divide into scale or unit errors (57%), another period or column of the right line (30%) and wrong items (13%), as shown in Figure 8. The checker cannot detect the first two classes because the quote is genuine and contains the reported digits. Figure 9 shows a real case.

Confident-wrong answers by error typeVector (PDF)
ModelTotal
Qwen3.5-9B36
Granite-4.2-8B64
OLMo-3-7B70
Ministral-3-8B77
Llama-3.1-8B102
Nemotron-Nano-9B-v2105
GLM-4.6V-Flash110
gpt-oss-20b112
Gemma-3-12B117
Phi-4151

Figure 8. Confident-wrong answers by error type, plain setup.

Question. What was Bank of America Corporation’s net change in derivatives for the nine months ended September 30, 2025?

Quoted line. Net change in derivatives (1,496) 636 (2,874) 3,145

Model answer. -2874 (in millions), page 48. Answer key. 3,145

Checks. Parses, page shown, quote verbatim and value in quote all pass. Outcome. Confident-wrong.

fig:9: A confident-wrong answer from Qwen3.5-9B in the plain setup (Bank of America Corporation, 10-Q). The line is correct but the figure comes from another column.

RQ3: Retry and reasoning

Figure 10 shows the change from the plain setup. Check-and-retry raises accuracy by 0.1 to 3.9 points, significantly for 8 of the ten models. It never reduces confident-wrong answers, and it significantly increases them for 4 models (Ministral-3-8B, Phi-4, Gemma-3-12B, Llama-3.1-8B). A retry can make a wrong value pass by finding a line that contains it. For Phi-4, 16 of 177 retried answers became confident-wrong.

Figure 10. Change from the plain setup. Filled markers are significant after Holm correction.

Reasoning reduces confident-wrong answers for all 5 ranked models with a reasoning mode (4 significantly), and it raises accuracy for 4 of them. Qwen3.5-9B is the exception: accuracy falls from 82.7% to 74.6%. It reached the pre-specified 8,192-token limit on 180 questions, 88 of which produced no parseable answer; on the 699 answerable questions it finished within the limit, its accuracy was 85.1%, against 84.4% for the plain setup on the same questions. Reasoning multiplies mean generated tokens by 8 to 67 times (Figure 11).

Figure 11. Accuracy against mean generated tokens per answer (log scale). * Supplementary baseline.

Discussion

Requiring a verbatim quote exposes many errors at low cost, but it does not establish that the selected figure or its units are right. Because most passing errors occur on the correct line, a stronger checker could require the column header and units statement to be quoted and verify their relationship to the selected figure. Retrying should also be treated cautiously: a retried answer that passes may have been adapted to the checker rather than corrected.

Limitations

  • AI-assisted quality control. The dataset QC review was performed by an AI assistant under the author’s direction, and the author did not independently verify the full sample. The planned manual review of 20 error categorisations per model was not carried out.

  • Protocol timing. The protocol was maintained in version control and was not lodged with an external registry. Repository history was later rewritten to edit commit metadata, so its timestamps are not independent evidence of timing. Deviations are logged in the protocol. The protocol’s coverage-at-zero-confident-wrong metric was not computed, because every model and setup produced confident-wrong answers and the models report no confidence score to threshold.

  • Configuration on benchmark questions. Serving settings were checked on 24 questions from the benchmark before the main runs.

  • Units stated off-page. After the results, we identified 30 answerable questions whose evidence page shows no units statement, meaning no plural scale word such as “millions” and no scale written with the figure. Excluding them changes no model’s accuracy by more than 2.3 points (Appendix § 9) and lowers the scale or unit share of plain-setup confident-wrong answers from 57% to 54%.

  • Scope. Results reflect one prompt, one run per configuration, three supplied pages per question and text from large US filers. Model versions and quantisation may affect results.

  • Reasoning budget. The 8,192-token limit truncated some reasoning runs.

  • Contamination. Models may have seen earlier filings during training, although a memorised answer must still be supported by a verbatim quote.

Conclusion

Small open models can often extract figures from SEC filings with verifiable quotes, and model choice matters substantially. Verbatim evidence is necessary but insufficient: errors in units and columns can pass automatic checks. Future work should verify the relationship between a figure, its column and its units.

Data and code availability

Code and protocol:

https://github.com/pramana-research/filing-evidence

Questions, page texts, model outputs and server logs:

https://huggingface.co/datasets/pramana-research/filing-evidence

Acknowledgements and AI use

The author conceived and directed this study, defined its research questions and design, and made all decisions on scope, methods and publication. Claude (Anthropic) was used as an assistant for implementing code, running analyses, reviewing the quality-control samples and drafting text. The author reviewed the manuscript and takes full responsibility for its content. This work was done independently; it does not represent the views of any employer, and it is not financial or investment advice.

Sensitivity to off-page units

Accuracy on all 875 answerable questions and after excluding 30 questions without an on-page units statement.

ModelC1 allC1 excl.C2 allC2 excl.C3 allC3 excl.
Qwen3.5-9B82.7%82.7%84.5%84.5%74.6%76.1%
gpt-oss-20b74.4%76.0%77.4%79.1%80.7%82.4%
Granite-4.2-8B73.3%74.4%76.2%77.3%81.3%83.6%
GLM-4.6V-Flash70.9%72.3%73.0%74.6%82.3%83.0%
Ministral-3-8B66.7%68.8%69.6%71.7%––
Gemma-3-12B61.4%63.4%64.3%66.5%––
Llama-3.1-8B58.9%60.4%62.7%64.4%––
Phi-457.3%58.9%60.8%62.5%––
Nemotron-Nano-9B-v242.3%43.8%42.4%43.9%54.4%56.0%
OLMo-3-7B17.1%17.4%17.3%17.5%––

Table 4. Accuracy on all 875 answerable questions and after excluding 30 questions without an on-page units statement.

Within-model comparisons

Paired exact McNemar tests. “A/B” counts questions favouring only the first or second setup; pp values are Holm-adjusted.

ModelComparisonAcc. A/BAcc. ppCW A/BCW pp
Qwen3.5-9BC1 vs C20/150.0004270/01
Qwen3.5-9BC1 vs C3159/887.35e-0527/70.0107
Qwen3.5-9BC2 vs C3162/764.35e-0727/70.0107
Granite-4.2-8BC1 vs C20/264.77e-070/80.0579
Granite-4.2-8BC1 vs C391/1610.0001151/250.0422
Granite-4.2-8BC2 vs C396/1400.020156/220.00209
Ministral-3-8BC1 vs C20/257.75e-070/90.0422
OLMo-3-7BC1 vs C20/110/01
Nemotron-Nano-9B-v2C1 vs C20/110/01
Nemotron-Nano-9B-v2C1 vs C3100/2062.68e-0886/530.0579
Nemotron-Nano-9B-v2C2 vs C3101/2063.91e-0886/530.0579
gpt-oss-20bC1 vs C20/264.77e-070/60.156
gpt-oss-20bC1 vs C367/1220.00046276/201.32e-07
gpt-oss-20bC2 vs C374/1030.10579/172.09e-09
Phi-4C1 vs C20/311.96e-080/160.000458
Gemma-3-12BC1 vs C20/264.77e-070/170.000244
GLM-4.6V-FlashC1 vs C20/194.2e-050/11
GLM-4.6V-FlashC1 vs C349/1491.45e-1159/102.62e-08
GLM-4.6V-FlashC2 vs C352/1334.08e-0860/101.6e-08
Llama-3.1-8BC1 vs C20/342.56e-090/228.11e-06
Qwen3-8BC1 vs C20/110.004880/70.0938
Qwen3-8BC1 vs C366/1361.14e-0567/103.73e-10
Qwen3-8BC2 vs C370/1290.00027873/93.17e-12

Table 5. Paired exact McNemar tests. “A/B” counts questions favouring only the first or second setup; pp values are Holm-adjusted.

Confident-wrong rates

Figure 12. Confident-wrong rate with 95% Wilson intervals.

References

  1. [1]B. Bohnet et al. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv:2212.08037, 2022.arxiv.org/abs/2212.08037
  2. [2]Z. Chen et al. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of EMNLP, 2021.arxiv.org/abs/2109.00122
  3. [3]T. Gao, H. Yen, J. Yu, and D. Chen. Enabling large language models to generate text with citations. In Proceedings of EMNLP, 2023.
  4. [4]S. Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979.
  5. [5]P. Islam et al. FinanceBench: A new benchmark for financial question answering. arXiv:2311.11944, 2023.
  6. [6]W. Kwon et al. Efficient memory management for large language model serving with PagedAttention. In Proceedings of SOSP, 2023.
  7. [7]N. F. Liu, T. Zhang, and P. Liang. Evaluating verifiability in generative search engines. In Findings of EMNLP, 2023.arxiv.org/abs/2304.09848
  8. [8]Q. McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947.
  9. [9]V. Reddy et al. DocFinQA: A long-context financial reasoning dataset. In Proceedings of ACL, 2024.
  10. [11]E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927.
  11. [12]XBRL International. Inline XBRL Part 1: Specification 1.1, 2013.
  12. [13]F. Zhu et al. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of ACL-IJCNLP, 2021.

Paper details

vlnc:W_6ked83a4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude (Anthropic)

View statement
Looking for related papers…
Contents