betaFeedback

WorksComputer Science

Can You Trust Your Finance Agent? Drift, Diagnosis and Auditable Intervention on One Shipped Financial Research Agent

Abstract

Can you trust a finance agent you did not build? We take the reference agent shipped with the Vals Finance Agent Benchmark v2, leave it untuned, and ask four questions a leaderboard score does not answer. Is the score stable? Seven identical runs on 18 questions pass between 105 and 123 of 149 rubric criteria, and the bill varies by up to 17%. When the agent fails, who owns the fix? We classify 20 failed trajectories, from three runs of nine held-out questions, by the layer whose owner would remedy them, under a taxonomy frozen in advance: 13 belong to the agent's instructions, 3 to its tools, 4 to the model, none to its data. Can that diagnosis be trusted? Four blind readers from three model families, given only the traces, name the same fault as our LLM analyzer in 94% of cases, the rate at which they agree with one another (93%). Can a change be trusted? An optimization loop reads the traces, makes one targeted change per round and keeps it only if it clears a fixed threshold. Under a cost objective its final configuration costs 0.34x the baseline on the held-out questions (95% CI [0.22, 0.58]) at accuracy within noise. Under an accuracy objective it gains 14 points on the development questions and 1.5 on the held-out ones (CI [-5.3, +7.2]). Judged against the measured run-to-run noise, under a rule written before the test was opened, the first is a real saving and the second is not. This is a case study of one agent and nine test questions, not a general result.

Introduction

A financial research agent is in production. It has a leaderboard score. A compliance officer asks what it does when it is wrong. The score does not answer: it was computed once, on questions the officer has never seen, by a grader the institution does not control, and it says nothing about where the wrong answers came from or who in the building should fix them.

Benchmarks of financial agents have multiplied since 2025. The Vals Finance Agent Benchmark [5] scores answers to expert-written research questions against rubric criteria; BigFinanceBench [14] grades the visible derivation step by step. Both measure how much of the answer, or of its derivation, is right. Neither says why a wrong answer was wrong, and neither names the layer of the system that owns the remedy. DFAH-Bench [11] comes closest to the question we ask: replaying the same financial decision task, it finds that decisions agree 94–95% of the time while exact tool paths agree only 67–69%. The processes diverge; the benchmark establishes it and stops there.

We stop later. This paper is a controlled case study on one shipped agent that we did not write and did not tune, organised as five questions, each answering what the previous one leaves open.

  1. Is the score a property of the agent? No. Seven identical runs of the shipped configuration span 105–123 of 149 criteria (§3).

  1. When it fails, who owns the fix? On a pre-registered set of 20 failed trajectories, under our owner-of-the-remedy taxonomy: policy 13, tool 3, data 0, model 4 (§4).

  1. Can you trust that diagnosis? Blind readers of three model families name the same fault in 75 of 80 cases; the layer is a definition we supply; localisation is looser (§5).

  1. Can you trust a change? Same loop, two objectives: cost falls to 0.34× and holds on the frozen test; accuracy rises 14 points on dev and is indistinguishable from zero on the test (§6).

  1. What is trust, then? A protocol whose every step can be audited (§7).

The finance cut on the second question is the owner of the remedy. In an institution, compliance rewrites the agent’s policy, the platform engineer owns the tool, the data owner the retrieval pipeline, and model risk the model. We classify failures by that owner, not by symptom, and we measure rather than assert what the classification means (§5).

Scope. This is a case study of one agent: nine test questions and twenty failed trajectories give an anatomy rather than population shares, and every test-split interval is a paired bootstrap over the nine questions. The fourth question is where the first three put to work. Cost reduction is not the contribution of this paper; it is the intervention the diagnosis pointed at, on the axis where our instruments can resolve an effect (0.34 against a run-to-run noise band of 0.83). On accuracy the same instruments return noise, and that is a result too.

We contribute (i) a drift measurement of a shipped agent across seven identical baselines; (ii) an anatomy of its failures on a pre-registered set, under a taxonomy frozen before the test split was analysed; (iii) a blind, cross-family check of the diagnostic that separates fault identification (corroborated), remedy ownership (a definition, reproducible from the rule) and localisation (looser), with the diagnostic’s own errors declared; (iv) the optimization loop as shipped, under two objectives, each scored once on a frozen split under a written rule; (v) the protocol itself. We do not claim a better agent, a general method, or comparability with any leaderboard (§7.1).

Setting

Terms. The agent is the shipped configuration under evaluation. The analyzer is the component that reads a scored run’s traces and emits a diagnosis. The loop is the component that proposes one change per round from that diagnosis, applies it and accepts or rejects it. The protocol is the set of freezes, replicas, rules and test openings in Figure 1. The audit trail is the set of files the protocol produced.

The system and the protocol

Figure 1. The system and the protocol. Agent, judge, analyzer and loop form the optimization cycle of §6, run on the dev split only. Shaded boxes are frozen before use: the test split and its decision rule before any test number existed (§ 6.3), the taxonomy before the test trajectories were analysed (§ 4.1).

Models. For reference, the models in this paper and their roles: the agent runs on Claude Sonnet 4.5 [2], including its nested retrieval call, as shipped, and the cost arm of § 6.1 moves that call to Claude Haiku 4.5 [1]; the judge is Gemini 3.8 Flash [9]; the analyzer, and the loop’s proposer and applier, are Claude Sonnet 5 [4]; the blind readers of § 5.1 are Claude Opus 5 [3], Gemini 3.1 Pro (preview) [8], Gemini 3.8 Flash and GPT-5.5 [13], and the model that compares their justifications is Claude Opus 5. One reader, Gemini 3.8 Flash, is the same model as the judge.

The agent. The Vals Finance Agent Benchmark v2 ships a runnable agent: a tool-calling loop over Claude Sonnet 4.5 with seven tools (web search, EDGAR full-text search, HTML page parsing, a retrieval tool that reads stored filings through a nested LLM call, a calculator, price history, and final-answer submission) and a 2,767-character system prompt [5]. Its public split has 27 questions across nine categories, each with rubric criteria carrying a severity and a must-pass flag. We reproduce the shipped configuration with one documented backend substitution, required for reproducibility: the agent’s EDGAR search depends on a paid third-party index, which we replaced with the SEC’s own full-text search endpoint behind the same tool name, description and argument schema. On the 53 EDGAR queries of one baseline run replayed on both backends, the target company’s main filings were returned by both in 81 of 81 cases; the comparison is in Appendix E. Every run exports its trajectory in the Agent Trajectory Interchange Format [10], which is what the analyzer reads.

The split. We froze an 18/9 dev/test split, stratified across the nine categories and issuer-disjoint, before any analyzer ran. Every configuration change in this paper was decided on the 18 dev questions; the 9 test questions (Table 1) were opened twice, each time under a protocol written before any test number was seen, and are now closed.

idxcategoryissuers
1General qualitative analysisMAR, WH
4General quantitative analysisCZR
8Market analysisHUBS, NOW, TOST
9ComparablesAER, FTAI, WLFC
14PrecedentsPFE, SGEN
17AdjustmentsASPI
18Earnings analysisVSEC
21Disclosure analysisPFE
24Financial modelingCRM

Table 1. The nine test questions (Vals FABv2 public split).

The judge. We score against the benchmark’s verbatim criteria with our own judge: Gemini 3.8 Flash at temperature 0, three votes per criterion, a 0.5% tolerance on numeric matches, reading the final answer only. Its configuration fingerprint is identical across every report in this paper. Two facts bound the judge’s role. The cost results of §6 are sums of provider prices per call: they do not depend on the judge’s verdicts, so no judge error can move them. The pass-rate results do, and the judge is internally consistent under our fixed configuration: of the three votes per criterion, the vote split in 4 of 1,080 test cells and 12 of 1,043 dev baseline cells. This establishes within-configuration stability, not agreement with an external or human judge. Our numbers are internally comparable and not comparable with the Vals leaderboard, whose grader we have not used. We have not re-scored the test reports with a second judge model; the stored final answers allow it without re-running the agent, and § 7.1 says so.

The analyzer and the loop. We use ALoLab [7], an evaluation and optimization engine, as a black box: we configure it and read its outputs, and we did not modify it. The analyzer (Claude Sonnet 5) reads a scored run and emits failure categories, each with a location in the trace (a decisive step), a fix_level (prompt, skill, tool description, orchestration, architecture, or data issue) and a harness_component (observation, context, control, action, state, verification, or none). The loop proposes one change per round on one of six declared levers (the system prompt, tool descriptions, which tools are enabled, the agent’s model, the nested retrieval model, and that model’s instructions), applies it, re-runs the dev split, and promotes or rejects the change against the round’s parent configuration under a fixed gate. We wrote no agent and tuned nothing by hand.

The score is a draw

Before asking whether a change helped, one has to know how much the agent moves on its own. A score is, in this sense, a draw: one sample from the distribution over repeated runs of a fixed configuration, not a property of the configuration. We ran the shipped configuration seven times on the 18 dev questions and scored every run with the same judge (Table 2, Figure 2a).

Table 2: Seven identical runs of the shipped configuration on 18 dev questions, 149 rubric criteria. Twins are consecutive baselines; a flip is a criterion passed in one run and not the other.

runcells/149pass rateagent $
11190.79934.42
21160.77928.58
31200.80532.48
41230.82632.28
51050.70531.79
61160.77932.98
71090.73229.42
twinflipsratio of totalsgeomean
1→2190.8300.867
3→4210.9940.876
4→5280.9851.044
5→6151.0380.924
6→7190.8921.004

Table 2.

The four measurements

Figure 2. The four measurements. Top left on the dev split, the other three on the frozen test split. (a) Cells passed by the shipped configuration in seven identical dev runs. (b) Primary layer of the 20 failed test trajectories per baseline replica. (c, d) Both loop winners on the frozen test, three replicas each, paired bootstrap over the nine questions: the cost winner’s cost interval lies entirely below the drift band; every accuracy interval includes zero.

Cells passed span 105 to 123 of 149: 18 criteria, twelve points of pass rate, with nothing changed. Between two consecutive runs 15 to 28 criteria flip. The bill moves by up to 17% (ratio of run totals 0.830 to 1.038), and per question by far more: across the five twin pairs, a single question cost anywhere from 0.16× to 5.43× what it had cost the run before. Two consecutive runs, 6 and 7, scored 116 and 109.

The test split drifts the same way. Its three baseline replicas passed 54, 56 and 61 of 90 cells and cost $20.58, $23.67 and $26.91.

Two consequences follow for anyone who promotes a change. The loop’s default promotion bars sit inside this drift: a change is promoted if it cuts cost by at least 5% under the cost objective, or raises the pass rate by at least one point under the accuracy objective, each measured on a single run against its parent. In § 6.2 a +7-cell change is promoted, and +7 is the distance between baselines 6 and 7. And a median or geometric-mean gate cannot see what happens per question: between runs 3 and 4 the total bill moved by less than 1%, while per-question cost ratios ran from 0.16× to 2.52×.

A change that makes the twelve cheapest questions 30% cheaper and the three dearest 60% dearer passes a geometric-mean gate at 0.853 while the bill rises 11%; we built that case on the measured per-question costs of baseline 3, and the gate would approve a change that did nothing.

One failure class in the drift has a compliance name. The agent parses the wrong fiscal period’s filing: for one dev issuer (AMZN) in six of the seven baselines, for three others (DKNG, LULU, WYNN) depending on the draw. It is a property of the run, not of the configuration.

Who owns the fix

Taxonomy

We classify each failed trajectory by the layer that owns its remedy: policy (the instructions the agent is given; owner, whoever writes its policy), tool (interfaces, descriptions, output parsing, state between steps; owner, the platform engineer), data (what reaches the model’s context: sources, retrieval, truncation, staleness; owner, the data owner), and model (the model itself, given the right instructions, tools and facts; owner, model risk). The analyzer does not speak this vocabulary. It emits fix_level and harness_component, and we project the pair onto our four layers with a fixed table (Table 3) built from the analyzer’s own documentation and frozen before any analyzer run on the test trajectories.

Table 3: The projection, frozen before the test trajectories were analysed. fix_level is the primary key; harness_component decides only where the key is ambiguous.

fix_levelharness_componentlayer
promptany but nonepolicy
promptnonepolicy or model, by Rule M
skillanypolicy
tool descriptionanytool
orchestrationaction, observation, statetool
orchestrationcontextdata
orchestrationcontrol, verification, nonepolicy
architecturecontextdata
architectureother, nonetool
data issueanybenchmark artefact; outside

Table 3.

One cell needs a rule. A prompt category with no harness component is the analyzer’s “pure prompt or reasoning” finding, and it is exactly where the policy/model boundary lies. Rule M: the category maps to model when both hold, read from the trace and not from the analyzer’s prose: (a) the analyzer names no missing or wrong instruction, and (b) the decisive step is a reasoning, assistant or output event, and the facts it needed were already in the context at that step. Otherwise policy. Model is the residual: the case a model swap, and no harness change, would remedy.

The unit of count is the failed trajectory. One trajectory may carry several categories; its primary layer is the layer of the category containing its decisive step. A trajectory whose only category is a data issue is a benchmark artefact and leaves the denominator. Four choices in the table have defensible alternatives; we record them in Appendix A. Two fix_level values the analyzer emitted on the dev split (retrieval_llm, `tool_descriptions) are outside its documented vocabulary and are left unmapped; neither occurs on the test split.

The count

The pre-registered set is the three baseline replicas of the test split: 27 trajectories, 20 of them failed. Table 4 and Fig. 2b give the count; Appendix B gives every trajectory’s decisive step and verdict.

In this agent and under this projection, thirteen of twenty failures are owned by whoever writes the agent’s instructions, three by the tool layer, and four, one in five, would be remedied by a different model and nothing else; five under the sensitivity reading. Under either reading the model is the minority owner, for this agent.

What the count does not mean

Data is never primary, by construction. The plain-language “stale filing” failure exists in these traces: in one replica the agent parsed the FY2024 10-K while the FY2025 10-K sat in the same search result. It lands in policy, because the remedy the analyzer names is an instruction to check the period. The projection sends retrieval failures to data only when the analyzer says the material could not reach the window; when it says the agent did not look, the owner is the policy. Zero in the data column is a property of the projection and of this agent, not evidence that retrieval never failed.

The same question fails differently. Table 5 shows the primary layer of each test question across the three replicas. PFE is policy three times; CZR is tool, policy, tool; VSEC is model, policy, policy. Part of this is the agent failing in different ways (in one VSEC replica

questionr1r2r3
ASPIpolicytoolpolicy
CRMmodelpolicypolicy
CZRtoolpolicytool
FTAImodelpolicymodel
MAR/WHmodelpolicy–
PFE (disclosure)policypolicypolicy
VSECmodelpolicypolicy

Table 5. Primary layer of each failed test question across the three replicas (reading A). “–” = passed.

it rounds 60.41 to 60 basis points; in another it divides by segment revenue instead of consolidated), and part is the instrument (§5.2).

Can you trust the diagnosis?

The count in §4.2 is only as good as the analyzer that produced it. We checked it two ways: blind, against readers that never saw its output, and against itself. Three things are being checked and they come out differently: what failed, where in the trace, and which layer owns the remedy.

Blind panel

Four blind readers from three model families (Claude Opus 5, Gemini 3.1 Pro, Gemini 3.8 Flash, GPT-5.5) each received the same 20 failed trajectories with the question, the failed criteria and the numbered trace (for the three API readers, a length-capped rendering of it), and nothing from the analyzer. Each named the decisive event and the fault. A separate judge (Claude Opus 5), shown only pairs of justifications, unlabelled and in random order, judged whether the two described the same wrong quantity or the same wrong choice. The protocol, including how “same fault” is decided, was written before the panel ran (Appendix § 10).

What failed: corroborated. The analyzer names the same fault as a blind reader in 75 of 80 cases; the readers agree among themselves in 112 of 120 (Table 6). Every one of the 20 trajectories has its fault confirmed by at least three of the four readers.

Table 6: Blind panel, 20 failed test trajectories. Same fault: judged on unlabelled justification pairs; under the pre-registered rule, which also counts decisive events within ±5, the two totals are 75/80 and 113/120. Within ±5: decisive event within five trace events.

pairsame faultwithin ±5
analyzer vs Gemini Pro17/2010/20
analyzer vs Gemini Flash19/2011/20
analyzer vs GPT-5.520/2010/20
analyzer vs Opus19/2014/20
analyzer vs readers75/8045/80
readers vs readers (6 pairs)112/12084/120

Table 6.

Where: looser on both sides. The analyzer’s decisive event is within five trace events of a reader’s in 45 of 80 pairs; the readers’ are within five of each other’s in 84 of 120. The analyzer is as good as an independent reader at saying what went wrong and about as imprecise at saying where.

Which layer: a definition, not a perception. Asked to place the fault in our four layers, the blind readers said model in 66 of 80 labels and policy once. In 12 of the 20 Opus labels the reader quotes a violated instruction and still writes model. The readers answered who erred; the table answers who owns the remedy. We report this as a definition, not as a disagreement: the owner-of-the-remedy reading is ours, and it is not the reading a model gives unprompted. Given the rule, it is reproducible: a second reader with Rule M, the trace and the decisive index, and none of the analyzer’s prose, returned the same verdict on all eight cells where the rule applied.

The five misses are informative. Two are Gemini Pro calling ASPI and CZR benchmark artefacts rather than agent failures; on ASPI the reader is wrong (§5.2). Two are CRM, where the reader names a different arithmetic error that the analyzer did not report. One is CZR, where the reader blames a wrong-basis choice and the analyzer a corrupted extraction.

The analyzer’s own errors

An over-claim, caught by a sister replica. On CZR the analyzer wrote that “correcting this single decision point [. . . ] would flip every downstream EBITDAR/growth/CAGR check”. The sister replica had made that decision and still failed all eight checks. Three replicas took three different bases for the same figure and each was 0.6–2.1% off, against a 0.5% tolerance. No figure any replica retrieved reproduces the rubric. A reviewer reading one trajectory would have believed the analyzer; the replica did not.

Same input, two taxonomies. Given a byte-identical payload twice, the analyzer returned nine categories once and eight the next time, and the two categories outside the prompt layer became one, of a different kind. The layer distribution barely moved, which favours classifying by owner of the remedy over classifying by the analyzer’s category. But the instability reaches the count through one field: in one replica the analyzer attached verification to three of its four prompt categories, Rule M applied only to the fourth and returned policy, and the model count was zero; in another it attached none to six of eight, Rule M fired, and the model count was three. The per-replica spread in Table 4 is that.

replicafailedpolicytooldatamodel (A)model (B)
r1731034
r2761000
r3641011
total2013304 (20%)5 (25%)

Table 4. Primary layer per failed trajectory, three test baseline replicas. Reading A is Rule M as written; reading B additionally treats calculator arguments as model-authored and is a sensitivity only.

Same fault, four labels. In four of the five analyses of the accuracy arm (§6.2) the analyzer named the stale-period failure, each time under a different category label.

A benchmark artefact that was not. ASPI never passes its must-pass criteria under any configuration. The analyzer emits a data-issue category on it and one blind reader calls it a benchmark artefact, for a visible reason: the question asks for the gross margin between 2/28/2024 and 2/28/2025, and ASP Isotopes reports on calendar years. The periods are those of RenRengen, a South African producer that ASP Isotopes agreed in May 2025 to acquire, and whose audited statements for the years ended 28 February 2025 and 29 February 2024 ASP Isotopes filed as an exhibit to an 8-K. Those statements reproduce the rubric: plant depreciation in cost of sales rises from R1.5 million to R34.2 million, +2,149.47%, and the gross margin falls 88.62 points; one normalised figure in the rubric we could not reproduce. In one baseline replica the agent’s own EDGAR search returned that exhibit (event 39), and the agent chose ASP Isotopes’ quarterly figures instead (event 40). The failure is the agent’s, and the trajectory stays in the count, where the frozen rule had already kept it. An analyzer, a blind reader and, for a while, the agent chose to ask a hard question for a broken one; the trace said otherwise.

Replay. We replayed the analyzer twelve times over one scored dev run: two models, with the full run and with one failed question alone, three repetitions each (Appendix C). The stronger model returned six to seven categories where the production model returned three to five; given one question in isolation, both placed the AMZN stale-filing failure in a different fix_level in different repetitions (prompt, orchestration, architecture, tool description), and one repetition stated as trace evidence an inference the trace does not contain.

What this section supports is bounded. The analyzer is corroborated on what failed. The layer is our definition and is reproducible across model families when the rule is given. The instrument’s failure modes are known: it over-claims counterfactuals, it is unstable on the harness-component axis, and it names the same fault inconsistently.

Can you trust a change?

We report two runs of the loop as shipped on the 18 dev questions, one under a cost objective and one under an accuracy objective (Figure 1). Each round, the analyzer reads the traces of the current dev run; the loop picks one of its findings, proposes one change on one lever and states the trace evidence and the change answers; the change is applied, the dev split re-run once, and the gate compares the new run with its parent. Under the cost objective a change is promoted if the bill falls by at least 5% and the pass rate stays within 10 points of the session’s first run; under the accuracy objective, if the pass rate rises by at least one point and the bill rises by no more than 50%. A session stops after five rounds or two consecutive rejections. The gate reads one run, and § 3 shows how far one run moves: it filters changes, it does not prove them. What the loop guarantees is that every change answers something observed in a trace; whether it works is left to the test. The loop’s own final configuration, one configuration and no choice at numbers seen, then goes to the frozen test split, three replicas, against three baseline replicas, under a decision rule written before the test was opened (Appendix § 10). We read every promoted prompt change before the test for leakage: no company, ticker, figure, date or specific filing may appear. Both winners passed, with one declared exception noted below. The two arms are read together: the same rule, written in advance, and the same measured noise band are applied to both; the cost change clears the band and the accuracy change does not.

Cost arm

Five rounds (Table 7). The analyzer’s spend reading found the nested retrieval call at 74–96% of the cost of every question it sampled, and the agent’s own reasoning a minor share; the loop moved that call, a document extraction, from Sonnet 4.5 to Haiku 4.5 (round 1, cost 0.437×0.437 \times its parent, promoted). In traces where the agent queried the same stored document three to eight times, it then tried to teach the agent to batch its sub-questions into one retrieval call per document, first as a tool description (round 2, 0.996×, rejected) and then as a system-prompt paragraph (round 3, 0.844×, promoted). Round 4 tried to chain arithmetic into a single calculator call and saved four percent, rejected. Round 5 answered a trace with ten EDGAR searches, five or six of them near-repeats, for a 10-K that was never filed, before the agent found the answer in an 8-K; it added one sentence, to pivot to an 8-K exhibit or web search after two or three empty EDGAR queries, promoted.

roundlevercells$ (ratio) | verdict
baseas shipped10531.79
1retrieval model → Haiku 4.512013.90 (0.437)promoted
2tool desc.: batch per document10313.85 (0.996)rejected
3system prompt: batch per document11711.73 (0.843)promoted
4system prompt: chain arithmetic11711.24 (0.958)rejected
5system prompt: pivot after empty EDGAR10410.53 (0.898)promoted

Table 7. Cost arm, dev split: five rounds, gate = cost ratio against the round’s parent.

The same instruction saved nothing as a tool description and 16% of the bill as a system-prompt line. In this agent, the layer, not the wording, owned the remedy.

Accuracy arm

Four rounds, then a plateau (Table 8). Every change the loop chose was a system-prompt paragraph. Round 1 added scope qualifiers and gained seven cells, promoted; seven is the distance between baselines 6 and 7 in Table 2, and the gate promoted it. Round 2 answered five dev traces in which the agent had already retrieved the right figures and chose the wrong scope of an ambiguous term at one step (gross against net interest, all cash against cash in excess of the required minimum, equity price against enterprise value); it added five rules for these recurring ambiguities and reached 130 of 149, above every one of the seven shipped baselines. The gains sat where the rules pointed: TGT/WMT went from 3 to 8 cells on the net-interest rule, EPAM from 2 to 5 on the minimum-cash rule. On dev, the intervention landed where the diagnosis had pointed.

roundlevercells (Δ)$ | verdict
baseas shipped10929.42
1sys. prompt: scope qualifiers116 (+7)33.31promoted
2sys. prompt: five ambiguity rules130 (+14)38.09promoted
3sys. prompt: re-read item by item113 (−17)36.98rejected
4sys. prompt: reconcile multi-year117 (−13)36.81rejected

Table 8. Accuracy arm, dev split: four rounds then plateau, gate = cells against the round’s parent.

Two rounds later the loop stopped. Both rejected rounds kept the round-2 rules and scored TGT/WMT 3 of 8 both times and EPAM 2 and 1 of 5. We wrote, before the test ran, that 130 was one draw of a configuration whose level was unknown. The winning configuration’s second paragraph carries one example figure (“79 million shares”) that names no issuer; we declared it and sent the configuration to the test.

Both arms on the frozen test

tbl:9 and fig:2c–d give the test result for both arms: 9 questions, 3 replicas of the winner against 3 of the baseline, paired bootstrap over questions, the cost ratio judged against 0.83, the widest twin ratio of totals in the dev drift.

The cost winner spends 34% of the baseline, with the whole confidence interval below the noise band, at a pass rate within noise of the baseline. The saving is almost entirely on the nested retrieval side: $62.91 of the baseline’s $71.17 was nested-LLM spend, against $15.11 of the winner’s $24.27. Per question (Table 10), the disclosure question on PFE fell from $22.34 to $3.95 over three replicas; CRM, the financial-modeling question, is the one the winner makes dearer, because the agent spends 129 turns and 114 single-operation calculator calls on it, the defect the rejected round-4 change had targeted. Two questions, CZR and ASPI, are unsolved by every configuration. The cost protocol also declared, before the test, a secondary configuration: round 3, the promoted configuration with the most dev cells. We report it because it was declared. It returns the same verdicts as the winner, a real saving (0.397×) at a pass rate within noise.

Table 10: Cost arm, per test question, summed over three replicas.

questionbase cellswinner cellsbase ∣winner| winner
MAR/WH17/21176.631.60
CZR0/24013.012.35
HUBS/NOW/TOST27/27273.811.74
AER/FTAI/WLFC22/39203.811.73
PFE/SGEN27/27274.700.74
ASPI4/2769.265.33
VSEC23/27202.901.52
PFE (disclosure)33/392622.343.95
CRM18/39184.705.30

Table 10.

The accuracy winner gained four cells of 270. Its +14 points on dev became +1.5 on held-out questions, with a confidence interval that straddles zero and a sign-flip test at p=0.74p = 0.74; its cost is within noise too. The two paragraphs written on dev cases did not carry to nine questions about other issuers: VSEC lost cells in all three replicas, CRM gained five, then none, then lost four.

Read together

Same loop, same judge, same split, same rule, one pass and one fail. The structural remedy, a cheaper nested model and an instruction about how to use a tool, survived questions the loop had never seen. The case-specific remedy, two paragraphs about ambiguities in particular dev questions, did not. Without the drift measurement and the written rule we would have reported “+14 points on dev,” against the baseline replicas of the first opening, reused, so the pair does not share a time window. One example figure survived the leakage read of the accuracy winner’s prompt. The dev split’s analyses were produced under the cost objective and Rule M was not applied to them. Two substrate failures live in the traces and not in any score: the nested retrieval call fails with a prompt-too-long error when one call spans several large filings, and the agent’s provider client waits 57 minutes before noticing a dead connection.

tbl:9: Both loop winners on the frozen test split, 9 questions × 3 replicas, pooled. Intervals are paired bootstraps over the nine questions (10,000 resamples). The noise band for cost is [0.83, 1/0.83], the widest dev twin ratio; the accuracy rule is whether the interval excludes zero. Verdicts are those the pre-registered rule returns.

configurationcells/270pass rateΔ pts95% CImust-pass ok/27total $ | cost ratio | 95% CI | verdict (accuracy / cost)
baseline (shipped)1710.633971.17
cost winner (round 5)1610.596−3.7[−8.9, +1.1]1124.270.341[0.216, 0.580]within noise / real saving
cost arm, round 3 (secondary)1680.622−1.1[−7.4, +5.4]1228.260.397[0.325, 0.495]within noise / real saving
accuracy winner (round 2)1750.648+1.5[−5.3, +7.2]1176.981.082[0.844, 1.505]within noise / within noise

Table 9.

and it would have been true and worthless. The same instrument that passes one change fails the other, and that is what makes the 0.34 worth reporting.

Two boundaries on this reading. The cost result is consistent with the diagnosis having pointed at a real remedy; it does not isolate the diagnosis, because the loop has no control arm (§ 7.1), and the test split is now closed, so none can be added to this study. And cost is the axis on which the instrument has resolution, 0.34 against a band of 0.83; on accuracy the band is the result.

Trust is an audit trail

Limits, and what we do not claim

Our judge scores the benchmark’s criteria; the benchmark’s own grader would score differently, and no number here is comparable with the Vals leaderboard. The judge is a single model family, and we have not measured how the pass-rate verdicts move under a second judge. We have not run that check; the stored final answers allow the six test reports to be re-scored by another model without re-running the agent or reopening the split. What we can say now is that the cost result does not depend on the judge’s verdicts, and that the judge’s three votes were unanimous on all but 4 of 1,080 test cells, a statement about within-configuration stability and nothing more. Nine test questions and twenty failed trajectories make an anatomy, not an estimate of a population share; the inferential unit is the question, n=9n = 9. There is no control arm: the loop is the system as shipped, and the gain belongs to it end to end, not to the diagnosis in isolation; we validate the loop, not the causal efficacy of its diagnosis. One agent shows, and does not prove, that the approach transfers. The blind readers are language models of three families, not financial analysts; the owner-of-the-remedy question for a human analyst is open. The accuracy winner was compared

Related work

The Vals Finance Agent Benchmark [5] supplies our agent, questions and rubric; its paper, which describes the benchmark’s first version, reports the best model at 46.8% and a logarithmic accuracy–cost relation, and grades answers. BigFinanceBench [14] grades the visible derivation against point-weighted rubrics and localises failures along the analyst workflow; it does not name the layer that owns the fix. DFAH-Bench [11] replays financial decision tasks and finds tool paths diverging (67–69% agreement) under agreeing decisions (94–95%); our §3 measures the same instability on a shipped research agent and §4 says where, for this agent, it comes from. Khatchadourian and Franco [12] measure output drift across providers and propose deterministic harnesses with finance-calibrated invariants; our drift is at the level of a multi-step agent’s trajectory, where greedy decoding does not remove it. MAST [6] taxonomises multi-agent failures by symptom and finds that they often stem from system design rather than the model alone; our taxonomy is by owner of the remedy, on a single agent, and reaches a minority-model conclusion for that agent by a different route.

Conclusion

We asked whether a shipped finance agent can be trusted and answered with measurements rather than a cure. The score is a draw from a distribution 18 criteria wide. When this agent fails, under our taxonomy, the model owns one fix in five and the harness owns the rest. The diagnosis that says so is corroborated blind by readers of three model families on what failed, and its own failure modes are listed. A change to the agent can be trusted only against the measured drift and under a rule written in advance, and when we did that, the same instrument passed a structural change and failed a case-specific one. Trust, on this evidence, is the trail that lets a reader check each of those sentences against a file. A practitioner can build that trail with nothing from us: freeze a split, run the baseline more than once, write the promotion rule against the measured drift, and read the trace before believing the diagnosis.

The blind panel’s per-reader layer labels over the same 20 trajectories: Gemini 3.1 Pro model 14, tool 3, benchmark artefact 3; Gemini 3.8 Flash model 17, tool 2, data 1; GPT-5.5 model 17, data 2, tool 1; Claude Opus 5 model 18, data 1, policy 1. Localisation, same decisive event exactly (before the ±5 relabelation): analyzer readers 6, 7, 7, 6 of 20; readers vs readers 12, 9, 9, 10, 9, 11 of 20.

The analyzer’s errors: replay

Twelve analyzer calls over the same scored dev run (a shipped-configuration baseline, 18 trajectories): Claude Sonnet 5 (the analyzer used throughout the paper) and Claude Opus 5, each with the full run and with the AMZN trajectory alone, three repetitions of each condition. Categories returned, full run: Sonnet 5, 5; Opus 6, 7, 6. AMZN alone: Sonnet 2, 2, 2; Opus 4, 4, 3. The fix_level assigned to the AMZN stale-filing failure across the twelve calls: prompt (7 calls), orchestration (1), architecture (3), tool description (2), with some calls emitting two categories for it. Opus returned four category labels that Sonnet 5 never used:

  • INCOMPLETE_DISCLOSURE_COVERAGE

  • INCOMPLETE_SUPPORTING_DISCLOSURE

  • UNGROUNDED_PARAPHRASE

  • QUERY_MISMATCH

One full-run Sonnet call placed AMZN’s decisive step on a filing-selection inference that the trace does not record as an event.

Protocols

Test protocol, cost arm (written before any test measurement). 9 questions, 90 criteria. Judge: Gemini 3.8 Flash, temperature 0, 3 votes. Configurations: the shipped baseline; the round-5 configuration (primary: the loop’s last promoted configuration, chosen by the loop’s own gate); the round-3 configuration (declared secondary: the promoted configuration with the most cells on dev). Three replicas, interleaved: each replica runs every configuration back to back, order reversed on even replicas, so each pair shares a time window. Metrics declared per measurement (cells/90, severity-weighted, must-pass questions, cost split agent/nested, turns, flags), per replica paired (Δ\Delta cells, cost ratio Σ/Σ\Sigma/\Sigma, flips) and pooled. Decision rule: accuracy beats if the paired bootstrap 95% CI over questions of Δ\Delta pass rate lies above 0, worse if below, otherwise within run-to-run noise; the sign-flip pp is reported beside it and never used to rescue an interval that includes 0. Cost saving real if the CI of the cost ratio lies entirely below 0.83, the widest twin ratio of totals on dev; within noise inside [0.83, 1/0.83].

Test protocol, accuracy arm (written after the accuracy optimization ended and before any test measurement of its winner). Same split, judge, metrics and decision rule. One configuration, the loop’s final under the accuracy objective (two system-prompt paragraphs), three replicas, against the three baseline replicas of the first opening, reused. Declared as the second and last opening of the test split. Lever read under the leakage rule passed with one example figure declared.

Blind panel protocol (written before the panel ran). Each reader receives, per trajectory, the question, the system prompt, the frozen judge’s verdict on every criterion, and the numbered trace (length-capped for the API readers); nothing from the analyzer; no access

Taxonomy: declared choices

Four choices in Table 3 have alternatives we did not take. (1) Tool description maps to tool, not policy: both are text, but the description ships with the tool and the tool’s owner fixes it. (2) Orchestration with a control component maps to policy: defensible only because this agent is a thin prompt-driven loop with no coded planner; on a coded orchestrator the cell would be tool. (3) Rule M as written, rather than “model is whatever the declared model layer would remedy”: the simpler rule ties the taxonomy to the action space, and we kept it trace-based. (4) Data issue is outside the anatomy and reported separately, not a fifth bucket.

Two interpretations were declared when Rule M was applied. A submit_final_result tool call is the trace’s output event (verified on all 27 test trajectories: the output event’s content equals the last such call’s argument). Reading A, the count, treats any other tool call and any tool result as failing condition (b); reading B, the sensitivity, additionally treats a calculator call whose arguments carry the wrong figure as model-authored.

Anatomy tables

Table 11 lists every failed trajectory of the pre-registered set with its decisive event, the analyzer category holding it, the projected layer under reading A and, where it differs, reading B. Event indices are positions in the ATIF trace. Category names (EDGE_CASE, REASONING_ERROR, …) are the analyzer’s own labels; the layer columns are ours, obtained through Table 3 and Rule M. to any other file. Output per trajectory: earliest decisive event index and kind; layer in {policy, tool, data, model, benchmark artefact} defined as the owner of the remedy; secondary layers; a 2–5 sentence justification citing event indices; whether the needed facts were already in context; the instruction violated, if any; confidence. Readers: Claude Opus 5, Gemini 3.1 Pro (preview), Gemini 3.8 Flash, GPT-5.5. Same-fault agreement between two diagnoses: decisive steps within ±5 events, or a separate judge (Claude Opus 5, not one of the readers’ runs), shown only the two justifications, unlabelled and in random order, says they describe the same wrong quantity or choice. Reported: localisation per reader vs analyzer and per pair of readers; same-fault per pair; layer agreement as the secondary, definitional number.

replicaquestiondecisive eventprimary categoryfix × complayer Alayer B
r1ASPIassistant #20EDGE_CASEprompt × nonepolicy
r1CZRtool_result #18INTEGRATION_ERRORorchestration × observationtool
r1CRMsubmit #143INSTRUCTION_VIOLATIONprompt × nonemodel
r1FTAIcalculator #53REASONING_ERRORprompt × nonepolicymodel
r1PFEassistant #49KNOWLEDGE_GAPprompt × contextpolicy
r1MAR/WHsubmit #33REASONING_ERRORprompt × nonemodel
r1VSECsubmit #63INSTRUCTION_VIOLATIONprompt × nonemodel
r2CZRassistant #22REASONING_ERRORprompt × verificationpolicy
r2CRMcalculator #41REASONING_ERRORprompt × verificationpolicy
r2ASPIedgar_search #3TOOL_CALL_GAPtool_description × actiontool
r2FTAIcalculator #51REASONING_ERRORprompt × verificationpolicy
r2MAR/WHassistant #6REASONING_ERRORorchestration × controlpolicy
r2PFEtool_result #41KNOWLEDGE_GAPprompt × nonepolicy
r2VSECsubmit #101INSTRUCTION_VIOLATIONprompt × verificationpolicy
r3CZRtool_result #44INTEGRATION_ERRORtool_description × contexttool
r3ASPIassistant #17EDGE_CASEprompt × nonepolicy
r3FTAIassistant #110INSTRUCTION_VIOLATIONprompt × nonemodel
r3CRMcalculator #94REASONING_ERRORprompt × verificationpolicy
r3VSECtool_result #62REASONING_ERRORprompt × verificationpolicy
r3PFEcalculator #42REASONING_ERRORprompt × verificationpolicy

Table 11. Per-trajectory verdicts, three test baseline replicas. “fix × comp” is the analyzer’s fix_level × harness_component of the primary category.

EDGAR backend substitution

The shipped agent’s edgar_search tool queries a paid third-party index of EDGAR. We rebound the tool’s execution to the SEC’s own full-text search endpoint, keeping name, description, parameters and argument validation unchanged. Check: the 53 edgar_search calls of one baseline run, with the third-party results as recorded, replayed against the SEC endpoint. On the 40 queries non-empty on both backends: URL-set Jaccard mean 0.85, median 1.00; 21 of 40 identical result sets; same first result in 27 of 40. Of the third-party index’s main documents, 315 of 350 were returned by the SEC endpoint (90%); of its exhibits, 502 of 541 (93%); of the main documents of the target company named in the query, 81 of 81. Fourteen main documents were returned only by the SEC endpoint (amendments, PDF renditions, subsidiary lists). On shared records, accession number, CIK, form type and filing URL are identical; the filing date differs by one business day on 137 records because the third-party index reports the acceptance timestamp and the SEC endpoint the official filing date. One tokenisation difference: the SEC endpoint treats an apostrophe as part of the token. Six of the seven dev baselines and every test measurement ran on the SEC endpoint; the first dev baseline ran on the third-party index.

References

  1. [1]Anthropic. 2025. System Card: Claude Haiku 4.5. Retrieved September 23, 2026 from https://www.anthropic.com/claude-haiku-4-5-system-card
  2. [2]Anthropic. 2025. System Card: Claude Sonnet 4.5. Retrieved September 23, 2026 from https://www.anthropic.com/claude-sonnet-4-5-system-card
  3. [3]Anthropic. 2026. System Card: Claude Opus 5. Retrieved September 23, 2026 from https://www.anthropic.com/claude-opus-5-system-card
  4. [4]Anthropic. 2026. System Card: Claude Sonnet 5. Retrieved September 23, 2026 from https://www.anthropic.com/claude-sonnet-5-system-card
  5. [5]Antoine Bigard, Langston Nashold, Rayan Khrishnan, and Shirley Wu. 2025. Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks. arXiv:2508.00828. Describes v1 (537 questions); the agent and public split used here are v2, https://valis/benchmarks/fabv2.
  6. [6]Mert Cemri, Melissa Z. Pan, Shunyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657.arxiv.org/abs/2503.13657
  7. [7]Cosimo Galeone, Minsu Park, Giuseppe Ettorre, and Daniele Ligorio. 2026. When Correct Isn’t Usable: Improving Structured Output Reliability in Small Language Models. arXiv:2605.02363.arxiv.org/abs/2605.02363
  8. [8]Google DeepMind. 2026. Gemini 3.1 Pro Model Card. Retrieved September 23, 2026 from https://deepmind.google/models/model-cards/gemini-3-1-pro/
  9. [9]Google DeepMind. 2026. Gemini 3.8 Flash Model Card. Retrieved September 23, 2026 from https://deepmind.google/models/model-cards/gemini-3-8-flash/
  10. [10]Harbor Framework. 2026. Agent Trajectory Interchange Format (ATIF), v1.7. RFC 0001, https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md. Retrieved September 23, 2026 from https://www.harborframework.com/docs/agents/trajectory-format
  11. [11]Raffi Khatchadourian. 2026. DFA-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making. arXiv:2607.20491v3.
  12. [12]Raffi Khatchadourian and Rolando Franco. 2025. LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows. In Workshop on LLMs and Generative AI for Finance (AI4F) at ACM ICAIF ’25. arXiv:2511.07585.arxiv.org/abs/2511.07585
  13. [13]OpenAI. 2026. GPT-5.5 System Card. Retrieved September 23, 2026 from https://openai.com/index/gpt-5-5-system-card/
  14. [14]Alex Wang, Georg Meinhardt, Jacob Katz, Joseph H. Kim, Pratyush K. Chaudhary, Chase Blagden, and Eric Xu. 2026. BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents. arXiv:2606.03829.arxiv.org/abs/2606.03829

Paper details

vlnc:W_xrcctm5w
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents