betaFeedback

WorksComputer Science

AStockArena: Beyond Returns, Can We Entrust Real Capital to the Decisions of Financial Agents?

Abstract

Large language models (LLMs) are increasingly used in dynamic financial trading, yet real-time trading benchmarks mainly evaluate agents by cumulative returns and risk-adjusted performance. Such outcome metrics cannot show whether trades are faithfully reported, supported by available evidence, and consistent with account states. We introduce AStockArena, a real-time decision-auditing benchmark for continuous trading in China's A-share market. Under shared market conditions and execution constraints, we evaluate 13 open- and closed-source models over 3,198 sequential decisions. AStockArena reconstructsthe actions actually posted to accounts by aligning reports, trade requests, execution responses, and account states, and audits them with eight checks spanning execution traceability (O), evidence sufficiency (RK), and financial meta-awareness (B), combining record verification with consensus expert annotation. Of 1,070 trading points, 72.99% exhibit at least one audit gap; model-level rates range from52.2% to 91.2%, with evidence gaps most prevalent. Historical trade-filtering replay further shows thatportfolio outcomes are sensitive to these flagged trades. Decision-process auditing thus offers a verifiable complement to return-based evaluation of financial agents.

Introduction

The reasoning, tool-use, and long-horizon decision-making capabilities of large language models (LLMs) are moving autonomous agents from static tasks into dynamic environments {(citep}(Cheng and Chin, 2024; Song et al., 2026). Unlike one-off question-answering tasks such as FinQA {(citep}(Chen et al., 2021, 2022), these environments require agents to continuously acquire information, call tools, and act on evolving outcomes. Finance is a representative and high-stakes case:

BenchmarkCont.LiveDiag.Act.Evid.Acct.
StockBench {(citep}(2025)✓—————
AI-Trader {(citep}(2025)✓✓————
LiveTradeBench {(citep}(2025)✓✓————
TradeTrap {(citep}(2025)✓—✓———
CLQT {(citep}(2026)✓✓✓———
AStockArena✓✓✓✓✓✓

Table 1. Trading benchmark capabilities: continuous trading, real-time market data, process diagnostics, action verification, evidence auditing, and account auditing (left to right).

trading is continuous, and erroneous actions may be difficult to reverse {(citep}(Wu et al., 2023; Lee et al., 2024; Nie et al., 2024). Recent benchmarks report competitive cumulative returns from LLM trading agents, sometimes outperforming buy-and-hold {(citep}(Chen et al., 2025; Fan et al., 2025; Yu et al., 2025), but evaluate them mainly through realized portfolio performance.

Portfolio returns alone cannot establish whether the underlying trading decisions are reliable. An agent may report buying one million shares while submitting an order for only one hundred thousand, provide a plausible rationale without addressing material evidence, or misjudge its positions, cash, or profit and loss. Across successive trades, later actions must also remain understandable in light of earlier judgments and new information. Even under positive returns, such discrepancies determine whether decisions can be supervised and reviewed, yet they cannot be inferred from cumulative return or the Sharpe ratio {(citep}(Kong et al., 2026). We therefore ask: How can we assess the decision-process reliability of financial agents beyond their realized returns?

We introduce AStockArena, a real-time trading benchmark for auditing financial agents’ decisions.1 Agents trade under shared real-time market snapshots, uniform tools, and consistent execution constraints, each maintaining an independent account across successive decisions. Because stated intentions and submitted orders can diverge from recorded transactions, AStockArena first aligns model reports, trade requests, execution responses, and adjacent account states to reconstruct the actions actually posted to accounts.

On these records it defines eight checks in three dimensions: execution traceability (O) compares reports and execution records with observed account changes; evidence sufficiency (RK) checks trading rationales against materials available at decision time; and financial meta-awareness (B) checks account-state assessments and the coherence of successive judgments. The checks combine structured-record verification with consensus expert annotation.

We evaluate 13 open- and closed-source LLM trading agents in a unified A-share sandbox, collecting 3,198 sequential decisions. Of 1,070 trading points with formally annotated trades, 781 (72.99%) have at least one applicable audit gap; model-level joint gap rates range from 52.2% to 91.2%. Gaps include confirmed failures and cases where records cannot support a sufficiently verified judgment. Evidence gaps are prominent (pooled RK: 66.73%), alongside distinct execution and account-assessment gaps. Claude Opus 4.5 achieves a 10.84% cumulative return with joint gaps at 68.3% of annotated trading points, illustrating decision-process issues that returns alone cannot reveal.

Historical trade-filtering replay examines portfolio sensitivity to audit-flagged actions. Individual and combined filters replace flagged trades with No Trade, preserving retained trades’ original timing and target quantities subject to the resulting account constraints. Across the 12 actively trading models, separate O, RK, and B filters yield mean cumulative-return changes of +3.65, +8.86, and +6.11 percentage points, respectively. These retrospective comparisons complement process auditing with historical portfolio sensitivity analysis.

We make three contributions: 1) AStockArena, a continuous trading benchmark that reconstructs executed actions from reports, requests, execution responses, and account changes; 2) an eight-check, three-dimensional auditing framework combining record verification and consensus semantic annotation; and 3) an analysis of decision-audit gaps across 13 LLM agents, revealing reliability information beyond returns, supplemented by retrospective trade-filtering analysis.

Related Work

Continuous Trading Evaluation. Financial benchmarks cover numerical reasoning ([6] (Tang et al. 2025)), domain knowledge and analysis ([11, 25] (Shah et al. 2022; Xie et al. 2023; Xie et al. 2024; Son et al. 2025)), and multimodal or multilingual financial understanding ([2, 12] (Peng et al. 2026; Sugiura et al. 2026)). Trading agents use memory, reflection, and tools to adapt to market information and past experience ([21, 23, 31] (Zhang et al. 2024; Shang et al. 2025)). Historical and real-time trading benchmarks provide evolving account environments and primarily assess portfolio performance ([20]). Related environments address reinforcement learning, order-level market simulation, and economic competition ([26, 22, 32, 28] (Zheng et al. 2026; Weiss et al. 2024)). We complement these benchmarks and environments with trade-level audits of execution, evidence, and account facts.

Trading-Process Verification. Related evaluations examine forecasting consistency, financial safety, and deployment-relevant biases ([9, 29, 27] (Yang et al. 2023)). Financial-agent benchmarks use perturbations and closed-loop audits to assess trading reliability, strategy consistency, and alignment between stated and executed actions (Yan et al., 2025; Qu and Chen, 2026). General agent evaluations use traces, tool feedback, and environment states to verify action completion and statement–outcome consistency ([3] (Ye et al. 2026)). Runtime governance constrains trade authorization and execution gating (Tang et al., 2026), while provenance mechanisms bind claims, evidence, and execution records ([15, 17]). We extend this line of work by combining execution verification with separate audits of cited facts, key-fact coverage, and account judgments.

Rationale and Evidence Verification

Explanation-faithfulness research examines whether explanations reflect decision-relevant factors ([16] (Turpin et al. 2023)); financial argument evaluation examines evidence–claim support ((Sun et al., 2025)). Financial search and retrieval benchmarks assess access to supporting information ([15] (Tang and Yang 2025; Zhou et al. 2026; Zhao et al. 2025)), while financial knowledge and summarization studies assess factual and numerical reliability ([4] (Shah et al. 2025)). Symbolic references and executable financial reasoning enable finer-grained verification ([13] (Xie et al. 2026)). We jointly verify decision-time evidence and account facts, then use historical trade-filtering replay to assess the portfolio effects of flagged trades.

Method

AStockArena reconstructs executed actions, audits them along three dimensions, and converts consensus labels into trading-point comparisons and historical replay (Figure 1).

Figure 1. Overview of AStockArena. (a) Continuous trading, record alignment, independent annotation, and disagreement review; (b) execution, evidence, and account-fact auditing; (c) replacing flagged trades with No Trade and comparing replayed portfolio performance.

Reconstructing Executed Actions

Because reports and orders may differ from booked outcomes, we establish observed actions from account changes and execution records. In each decision window, models retrieve materials, trade or hold, and submit a structured report. The information pool is frozen at the window’s start; cash and positions carry forward.

The system saves reports, requests, execution responses, fill details, and account snapshots before and after each window (Table E5). With qm,t,s−q^{-}_{m,t,s} and qm,t,s+q^{+}_{m,t,s} the shares of security ss held by model mm before and after window tt, the net position change is Δqm,t,s=qm,t,s+−qm,t,s−\Delta q_{m,t,s} = q^{+}_{m,t,s} - q^{-}_{m,t,s}. Fills are aggregated by window and security, and shares bought minus shares sold are cross-checked against this change. The confirmed change provides the observed action for comparing net quantities in reports, requests, and responses; rejected requests are checked against execution responses.

Record typeRecord contentsCore audit use
Model reportStructured trade descriptions and rationalesCheck whether the report exists and has a valid structure; extract descriptions and rationales for subsequent verification
Trade requestSecurity code, buy/sell direction, and requested quantity submitted by the modelAggregate net trade quantities in requests and compare them with net account position changes
Execution responseProcessing status and execution records returned by the trading interfaceAggregate net trade quantities recorded in successful responses and compare them with net account position changes
Account snapshotPosition quantities, cash, sellable quantities, and other states before and after the windowVerify actual trade effects through net position changes and provide a reference for account-fact assessments

Table E5. Underlying verification records and their audit uses.

Auditing Decision Reliability

Eight checks audit the verified actions along three dimensions: execution traceability (O), evidence sufficiency (RK), and financial meta-awareness (B) (Table E6).

Audit questionAudit dimensionAssociated records
Was a valid report retained, and do reports, requests, and responses match outcomes posted to the ledger?O: Execution traceabilityModel reports, trade requests, execution responses, and actual booked trade effects
Are the materials supporting the current decision sufficient to support the stated trading claim?RK: Evidence sufficiencyCited materials, trading claims, and verified actual trade directions
Do decision rationales accord with account facts, and are successive trades reasonably explained?B: Financial meta-awarenessModel reports, account facts, and sequential trading records

Table E6. Core questions and associated records in the three-dimensional decision-auditing framework.

Execution traceability (O). The sandbox enforces trading rules; O checks the records. Report checks for a valid structured report; Rpt, Call, and Resp compare net quantities in report descriptions, trade requests, and successful execution responses with net position changes, respectively. Rejected requests are checked against execution responses.

Evidence sufficiency (RK). R checks for substantive citation fabrication, factual distortion, or direct contradiction between cited materials and the rationale or actual trade direction. A title mismatch alone is not fabrication; insufficient verification material yields unable to determine. Direction checks follow the specific claim: favorable news need not contradict a sale, and profit-taking or risk-adjustment rationales are checked against their evidence.

K uses an independently established reference of key facts. Facts are extracted from archived contemporaneous materials without model reports, trade directions, or subsequent market and return data. Favorable, unfavorable, and neutral facts share an importance threshold; the list is frozen before checking report coverage (Appendices § 7.§ 1 and § 7.§ 5).

K checks omissions or partial treatment of facts affecting the judgment, such as citing revenue growth for a purchase while overlooking substantial losses in the same material. Industry or macroeconomic sources constitute a gap when their assessment contradicts the trade and review finds them insufficient for the stock-specific decision. Coverage concerns treatment, not citation count: weighing counterevidence, giving an alternative interpretation supported by the materials, or explaining why a fact does not warrant changing the action counts as addressing it.

Mapping passes to 0 and confirmed or unable-to-determine gaps to 1, the evidence gap of trade event ee is RK⁡(e)=R(e)∨K(e)\operatorname{RK}(e)=R(e)\lor K(e), counted once per trade.

Financial meta-awareness (B). D compares reported positions, costs, profit and loss, and available funds with ledger records for the same security, time, and calculation basis. Reports without a verifiable account assessment are structurally inapplicable to D.

Temp complements D by checking transitions between judgments about the same asset. It pairs adjacent nonzero net trades in the same stock by the same model, using the earlier judgment to assess whether new information, market or account changes, trading constraints, or the prior rationale explain the later decision. Only coherent labels pass; partially coherent, incoherent, and unable-to-determine cases are gaps. The first trade is inapplicable, and direction reversals alone are not gaps (Appendix § 7.§ 5).

Forming Consistent Audit Labels

Record comparisons and semantic judgments need a common labeling basis. O is calculated from structured logs and account snapshots. Two authors with academic backgrounds in finance independently annotate R, K, D, and Temp under shared criteria by examining reports, contemporaneous materials, and account records. Two external fintech practitioners with securities research and financial analysis experience participate in disagreement review to finalize consensus labels (Appendix § 7.§ 1). Table 2 reports pre-consensus raw-label agreement: 95.77% on average, with mean Cohen’s κ\kappa of 0.814. Class imbalance and coherent/partially coherent boundary disagreements affect Temp’s κ\kappa (Appendix § 8.§ 5).

MetricnnA/BGPT-6 Pro vs. consensusGPT-6 Astra vs. consensus
Agree. %κ\kappaAC1Agree. %κ\kappaAC1Agree. %κ\kappaAC1
R137595.780.8950.94789.020.7510.85986.840.7070.830
K137597.090.9450.96198.400.9700.97896.070.9260.947
D137595.710.9280.94695.130.9180.93989.310.8290.865
Temp129094.500.4870.94392.560.3810.92294.260.6280.940
Mean-95.770.8140.94993.780.7550.92591.620.7730.895

Table 2. Annotator and model raw-label agreement. nn excludes fixed few-shot examples; R/K retain three categories and D/Temp four. κ\kappa/AC1 denote Cohen’s κ\kappa/Gwet’s AC1. Agreement is a percentage; Mean equally averages each statistic across the four metrics (Appendix § 7.6).

Consensus also supplies nine trades as fixed few-shot examples for R/K/D and three adjacent pairs for Temp to guide GPT-6 Pro and GPT-6 Astra annotation. Both models receive the same original materials, labels, and judgment rationales. Excluding these examples, mean agreement with consensus is 93.78% and 91.62%, respectively (Table 2; examples and merged-category results in Appendix § 7.6).

An audit gap denotes a confirmed defect or inability to verify an applicable check from archived records. Both lack a fully verified basis, so the main audit groups them as gaps while retaining raw labels; structurally inapplicable checks generate no gap. This shared definition supplies inputs to comparison and replay, including confirmed-error-only replay that retains unable-to-determine trades (§ 4.4).

Comparing Audit Gaps and Replaying Trades

Because checks assess windows, trades, account claims, or trade pairs, we map labels to a common comparison unit and to removable actions.

Trading-point comparisons. Main-text comparisons use windows containing formally annotated trades, termed annotated trading points. The gap rate is Gi=100 di/niG_i = 100\,d_i/n_i, where did_i counts annotated trading points with at least one applicable gap in check ii; nin_i is the model’s total annotated trading points. O, RK, B, and the eight-check joint audit take constituent-gap unions, counting each point once with this denominator. Appendix § 8.§ 1 reports rates over each check’s applicable records.

Action-level replay. Actions retain model, window, and security identities. Any O gap at a decision point flags all its net trades. R/K/D gaps flag their corresponding actions; Temp flags the later trade in the pair, restricted to formal R/K/D actions. Calibration actions remain in point-level auditing (Appendix § 9.§ 1). RK, B, and combined filters take action-level unions of R/K, D/Temp, and selected dimensions, respectively, counting each action once.

Flagged actions are replaced with No Trade. Retained actions keep their original timing, target quantities, and labels, executing subject to updated cash and sellable holdings. Results appear in § 4.4.

Experiments

Experimental Setup

AStockArena constructs a simulated trading environment using 10 STAR Market stocks spanning different industries to control the volume of information within each window and support item-by-item verification. The evaluation covers 82 trading days from March 2 to June 30, 2026. The system generates market snapshots each day at 10:30, 11:30, and 14:00 Beijing time, yielding 246 decision windows. The 13 models run independently under the same market conditions, producing 3,198 decisions.

Each model manages an independent simulated account with initial capital of CNY 1 million and no positions. Trading follows cash constraints, T+1, a short-selling prohibition, and a single-stock position limit of 50% of total account assets, with order quantities specified in units of 100 shares. Trading costs comprise a 0.03% commission on both purchases and sales (minimum CNY 5 per trade), plus a 0.05% stamp duty on sales.

The benchmark uses a shared ReAct agent architecture, allowing at most 30 tool-interaction steps per window. All models share prices, technical indicators, news, and company announcements available up to the current window, with textual materials restricted by publication time. The stock pool, data sources, and model configurations are detailed in Appendix § 11.

The simulator uses immediate full execution: orders that pass the constraint checks are filled at the current market snapshot price, while all other orders are rejected. Account states evolve through simulated trades, with position changes handled on a fixed share-count basis (Appendix § 9.§ 1).

The main experiment covers R, K, and D for 1,384 active trades and Temp for 1,293 adjacent trade pairs.

Original Investment Performance

Account performance is measured from daily marked-to-market net assets at 14:00 using cumulative return (CR), Sharpe, Sortino, maximum drawdown (MDD), and Calmar. Appendix Table B6b summarizes CR, Sharpe, and Calmar for the original accounts and reference benchmarks. Model CRs range from -3.59% to 10.84%; equally weighted buy-and-hold and the SSE 50 return 1.66% and -1.87%, respectively, while five technical strategies range from -5.50% to 3.90%. Benchmarks use the same trading constraints and costs; complete results and strategy parameters appear in Appendix § 8.§ 4 and Appendix § 11.

ModelORKBJoint
Claude Opus 4.52.465.08.968.3
GLM 4.7 Flash20.875.541.584.9
DeepSeek V3.210.872.515.876.7
Gemini 3.1 Pro14.376.817.083.0
Claude Haiku 4.511.676.953.191.2
Qwen3 Max2.355.816.359.3
DeepSeek V3.10.069.64.369.6
GLM 4.713.360.030.076.7
MiniMax M2.14.460.319.166.2
GPT-5.1N/AN/AN/AN/A
GPT-5.25.649.10.052.2
Qwen3 235B25.080.032.585.0
Gemini 2.5 Flash11.969.017.976.2

Table 3. Dimension and joint audit gap rates at annotated trading points (%, lower is better). Joint denotes a gap in any applicable check. The denominator is each model’s points containing formally annotated trades; gaps include confirmed failures and unable-to-determine labels, counted once per check per point. N/A denotes no annotated trades.

GPT-5.1 makes no active trades and returns zero; filtering replay therefore covers the remaining 12 active models, using each model’s original account as its baseline.

Decision Auditing

Tables 3 and 4 report results for the audit dimensions and eight checks, respectively, both using decision points containing formally annotated trades as the denominator. Of these 1,070 trading points, 781 have a gap in at least one check, yielding a trading-point joint gap rate of 72.99%. All-point results and detailed statistics for individual trades and adjacent trade pairs appear in Appendix § 8.

Most active models’ trading-point joint gap rates fall between 60% and 90%, with RK exceeding O and B: evidence omissions and citation conflicts are common. Trading-point joint rates also exceed all-point rates; including non-trading windows lowers the overall proportion and obscures defects concentrated in actual trades.

Complete report retention does not guarantee accurate trade descriptions. Qwen3 235B retains valid reports at all trading points, yet descriptions disagree with account postings at 25.0% of them, showing why report presence and action consistency require separate checks (Table 4).

R and K flag 714 trading points, with 185 triggering both (25.9%). Most points exhibit either citation problems or key-fact omissions, indicating that the two checks cannot substitute for each other. Pooling all 1,070 annotated trading points, confirmed RK gaps and undetermined-only cases account for 63.36% and 3.36%, respectively, giving a merged gap rate of 66.73% (Appendix § 8.§ 1).

Account assessments also vary: Claude Haiku 4.5 has a D gap rate of 49.0%, while low rates should be read alongside account-claim coverage (Appendix § 8.§ 1). Temp gaps occur at 8.79% of annotated trading points, reflecting explanation continuity. Formal pairs contain 70 partially coherent and 27 unable-to-determine cases, with no incoherent labels (Appendix § 8.§ 5).

Historical Trade-Filtering Replay

To assess how audit filtering affects existing trading paths, we replace flagged trades with No Trade, retain the original timing, target quantities, and labels of the other trades, and recompute account trajectories. Retained trades execute subject to the updated cash balance and sellable holdings. We replay O, RK, B, and their combinations separately, with each combined filter formed from the union of its flagged-trade sets. Execution and replay rules are detailed in Appendix § 9.§ 1.

Let V0V_{0} and VTV_{T} denote initial and final total account assets, respectively. Cumulative return is defined as:

CR=VTV0−1.(1)\mathrm{CR} = \frac{V_{T}}{V_{0}} - 1. \tag*{(1)}

The return change for model mm under filtering condition F\mathcal{F} is defined as:

ΔCRm,F=100(CRm,F−CRm,original).(2)\Delta\mathrm{CR}_{m,\mathcal{F}} = 100\left(\mathrm{CR}_{m,\mathcal{F}}-\mathrm{CR}_{m,\mathrm{original}}\right). \tag*{(2)}

Here, CR is a decimal and Δ\DeltaCR is in percentage points. Table 5 reports equally weighted mean changes across 12 active models for individual checks, dimensions, and combinations.

FilterΔ\DeltaCRΔ\DeltaSharpeΔ\DeltaSortinoΔ\DeltaCalmar
R+6.66 (8)+0.22 (8)-0.02 (8)+0.54 (8)
K+9.29 (10)+0.64 (9)+0.80 (8)+2.29 (9)
D+4.49 (10)+0.34 (9)+0.36 (9)+0.56 (9)
Temp+3.45 (9)+0.46 (7)+0.56 (8)+1.04 (8)
O+3.65 (11)+0.12 (7)-0.06 (7)+0.11 (7)
RK+8.86 (9)+0.61 (8)+0.82 (7)+2.68 (8)
B+6.11 (9)+0.44 (8)+0.43 (7)+0.84 (7)
O+RK+9.01 (9)+0.57 (9)+0.70 (7)+2.12 (9)
O+B+7.28 (10)+0.38 (8)+0.33 (7)+0.75 (7)
RK+B+8.25 (9)+0.59 (8)+0.70 (7)+2.05 (8)
O+RK+B+7.65 (9)+0.52 (8)+0.52 (6)+1.35 (9)

Table 5. Replay results for filtering. Δ\Delta is the equally weighted mean of filtered minus original results across 12 active models; parentheses indicate the number of models whose metric improves or remains unchanged. Filtering definitions appear in Appendix C.

All filters improve mean CR, with the largest gains from K and RK: 9.29 and 8.86 percentage points, respectively. All-dimension filtering yields 7.65 percentage points, improving returns for 9 models. Mean risk-adjusted return metrics also improve under RK and joint filtering (Table 5).

Confirmed-error-only filtering. Retaining unable-to-determine trades and removing confirmed gaps yields mean CR gains of 9.13 and 9.39 tbl:4: Eight-check audit gap rates at annotated trading points (%, lower is better). The denominator matches Table 3; multiple gaps in the same check at a point count once.

ModelReportRptCallRespRKDTemp
Claude Opus 4.50.02.42.42.443.943.97.32.4
GLM 4.7 Flash0.011.317.01.943.458.530.220.8
DeepSeek V3.20.810.01.70.829.263.38.310.8
Gemini 3.1 Pro3.611.610.78.045.557.18.013.4
Claude Haiku 4.50.011.611.611.656.543.549.07.5
Qwen3 Max0.02.32.32.323.341.911.65.8
DeepSeek V3.10.00.00.00.032.658.70.04.3
GLM 4.713.313.30.00.026.743.326.710.0
MiniMax M2.10.02.91.50.032.447.113.211.8
GPT-5.1N/AN/AN/AN/AN/AN/AN/AN/A
GPT-5.20.01.25.65.613.738.50.00.0
Qwen3 235B0.025.022.517.535.060.015.022.5
Gemini 2.5 Flash3.611.98.38.323.858.39.516.7

Table 3 (PDF p. 7).

percentage points for RK and all-dimension filtering, versus 8.86 and 7.65 under the merged definition (Figure 2). Both improve returns for 11 models; complete results and removal counts appear in Appendix Table C11.

Main versus confirmed-error-only filtering

Figure 2. Main versus confirmed-error-only filtering. Mean changes across 12 active models; All denotes O+RK+B.

Buy/sell-matched random-filtering control. To compare audit filtering with equal-count random deletion, we sample from each model’s full-period trades, match buy/sell removal counts, and repeat independently 1,000 times with a fixed seed (Figure 3). All four CR gains exceed random means and lie within their central 95% ranges. RK and joint filtering exceed the means by 3.55 and 2.75 percentage points; RK’s Sharpe, Sortino, and Calmar gains exceed the random 97.5th percentiles. Full ranges and multiple-metric adjustments appear in Appendix § 9.7.

Figure 3. Audit versus random filtering (mean Δ\DeltaCR). Random bars average 1,000 replays matched by model and buy/sell counts; All denotes O+RK+B.

At 20 basis points of slippage per side, mean CR gains from RK and all-dimension filtering are 10.17 and 9.21 percentage points, respectively; full cost-sensitivity results appear in Appendix § 9.6.

Historical Case Reruns

To observe defects after regenerating decisions, we test DeepSeek V3.2, GLM 4.7 Flash, GPT-6 Astra (max), and GPT-5.6 Sol on ten historical snapshots each. Each case is run independently after restoring contemporaneous information and the pre-decision account state. Independent snapshots exclude Temp (Figure 4).

Figure 4. Audit gap rates for ten reruns under each configuration (%, ↓\downarrow). Colors denote model and run configuration; hatching denotes metrics. Joint includes O, R, K, and D; budgets and audit-guidance conditions appear in Appendix § 10.§ 2.

Decision defects remain observable after rerunning. DeepSeek V3.2 has confirmed gaps in eight of ten cases, including all six trading points; GLM 4.7 Flash has three, comprising two key-fact omissions and one report-action mismatch. Neither GPT model has confirmed execution or account-fact gaps, but Astra and Sol have evidence gaps in two and five cases, respectively, with the remainder passing the seven-check joint audit. Cases under different configurations show that execution consistency and evidence sufficiency require separate checks.

Continuous Trading by a New Model

To examine the audit framework’s applicability to new-generation base models, we evaluate GPT-6 Sol (with medium reasoning) on the same historical snapshots. The model generates 96 active trades across 246 decision windows; its audit gap rates and filtering replay results appear in Table 6.

GPT-6 SolORKBJointTrading points
Gap rate (%)0.0042.710.0042.7196
ReplayCRSharpeSortinoCalmar
Original4.851.021.881.94
RK Filter38.282.865.2012.84
Change+33.43+1.84+3.33+10.90

Table 6. Audit and replay results for GPT-6 Sol. Gap rates use 96 trading points as the denominator; all-point results appear in Appendix § 10.§ 3. Replay compares the original and RK-filtered accounts.

New models may have encountered public market information from the evaluation period, limiting interpretation of historical investment performance. Auditing archived decision-time materials, account states, and execution records can still assess evidence use, account understanding, and trade execution.

No gaps are identified in GPT-6 Sol’s execution or account-fact checks, but 41 trades have evidence audit gaps. RK filtering increases CR by 33.43 percentage points and also improves risk-adjusted returns (Table 6).

Discussion and Conclusion

Discussion. The new model has no confirmed execution or account-fact gaps, yet still omits key evidence, suggesting a need for error-specific feedback. Reliable execution therefore needs to be accompanied by adequate consideration of decision-relevant evidence. AStockArena links execution discrepancies, evidence omissions, and account-assessment errors to trades and source materials as a basis for model correction and training. Future work can test whether revising rationales in response to this feedback corrects identified problems and how it affects subsequent trading decisions.

Conclusion. AStockArena audits continuous financial trading through structured-record verification and consensus semantic annotation, aligning reports, execution records, and external evidence with account changes to verify actions, rationales, and account assessments. Historical trade-filtering replay examines the relationship between audit gaps and portfolio performance. Together, these analyses provide a verifiable basis for evaluating and improving subsequent models’ decision processes and building more reliable trading agents.

Limitations

This study has not integrated audit feedback into a closed-loop trading system or evaluated subsequent model decisions following audit interventions. Due to running and annotation costs, each model follows a single continuous trajectory, and variability across repeated runs has not been characterized. The evaluation covers ten STAR Market stocks over a single period. The stock pool is sized to control the information load within each decision window and reduce the risk of dispersed attention when analyzing multiple stocks in parallel. Execution is immediate at snapshot prices, without partial fills, queuing, or latency. For models added later, timestamp constraints on tool inputs cannot rule out prior exposure to relevant historical information during training, which may affect trading rationales and historical investment performance.

References

Annotation and Consensus Formation

Annotation and Disagreement Resolution

Annotation and review personnel. Two paper authors independently completed the initial annotations A and B, respectively. We commissioned a fintech company serving the securities and fund-management sectors to review the annotation results and participate in discussions to resolve disagreements. Its products cover quantitative research, risk analysis, and financial agents. The institution provided two external reviewers for this process. Their backgrounds are described below.

External reviewer 1: Holds a master’s degree in finance and has over seven years of professional experience as a financial analyst, with a background in securities research. Their interdisciplinary experience spans finance and computer science, including financial-agent product development.

External reviewer 2: Holds a bachelor’s degree in economics and has over eight years of professional experience, including financial analysis and annotation projects. They have also participated in the development of financial-agent products and quantitative factor databases.

Initial annotator A (Author 1): Holds a bachelor’s degree with a double major in economics and finance and a master’s degree in economics. Their experience includes large language model research, financial data analysis, and economics research.

Initial annotator B (Author 2): Holds a dual degree in financial engineering and an MPhil, with research experience in AI for Finance.

Items with matching results in the two independent annotations are retained, while the two external experts and two authors jointly review the remaining items to form final consensus labels. For the 59 items with raw-label disagreements in DD, we retain both annotations and their rationales to trace the sources of disagreement. All reported rates and primary ablation results use final consensus labels, while independent annotations A and B are used to analyze pre-consensus agreement and differences in judgment.

Key-fact extraction and coverage assessment are conducted in two stages. First, annotators use only the target security, decision time, and external materials saved at that time to identify important facts that may affect trading judgments, including information that changes trading premises, provides important counterevidence, or constrains decision conditions. Model reports and actual trading directions are not provided during extraction, reducing the influence of model decisions on fact selection. Favorable, unfavorable, and neutral information is selected under the same criteria, and duplicate reports are merged into a single fact. After the fact list is determined and frozen, model reports and trading directions are introduced to check whether these facts have been adequately addressed. The entire process uses only materials available at decision time, without referring to subsequent market movements or investment returns.

Coverage assessment does not require reports to restate facts one by one; we examine whether they understand and address the implications of those facts for trading decisions. For example, a report may acknowledge unfavorable evidence and weigh it, offer an alternative interpretation based on other materials, or explain why a fact is insufficient to change the trading judgment. For materials saved only as titles, assessment is limited to information explicitly stated in the titles, retaining uncertainty qualifiers such as ’proposed’ and ’expected.’

Therefore, KK measures whether a decision adequately considers relevant information, rather than whether the model’s investment view agrees with the annotators’. Industry and macroeconomic materials are assessed in relation to the target stock, report arguments, and actual trading actions; the same fact may support different trading choices, provided that the report explains its decision implications and addresses important counterevidence. Company cash-flow and balance-sheet information constitutes external evidence, while cash, holdings, and sellable quantities in the model’s account are verified through DD and execution records, respectively. For disputed items, reviewers adjudicate using the original materials and shared judgment criteria, retaining the corresponding rationales.

R/KR/K and DD each cover 1,384 formal trades, while Temp covers 1,293 adjacent trade pairs.

ModelR A/BK A/BD A/BTemp A/B
Claude Opus 4.533.5/35.540.6/38.711.5/7.80.0/2.0
GLM 4.7 Flash26.4/29.948.3/46.056.2/56.26.3/13.9
DeepSeek V3.220.7/21.857.5/52.511.2/12.76.4/9.2
Gemini 3.1 Pro26.4/33.045.2/43.16.1/6.66.2/9.8
Claude Haiku 4.547.5/47.539.9/38.351.1/39.90.0/6.9
Qwen3 Max20.2/21.345.7/43.625.0/20.00.0/6.0
DeepSeek V3.129.4/29.454.9/52.90.0/0.02.3/4.7
GLM 4.730.3/24.239.4/39.434.6/26.911.5/11.5
MiniMax M2.126.2/28.642.9/38.125.5/18.05.3/12.0
GPT-5.1N/A/N/AN/A/N/AN/A/N/AN/A/N/A
GPT-5.212.6/12.638.3/36.00.0/0.00.0/0.0
Qwen3 235B27.1/29.256.2/56.243.8/37.57.5/25.0
Gemini 2.5 Flash20.4/21.455.1/54.117.4/17.03.3/15.2

Table A1. Gap rates from annotations A and B (%). R/K use formal trades, D uses applicable account assessments, and Temp uses adjacent trade pairs.

Agreement

MetricPairsAgreement %Cohen’s κ\kappaGwet’s AC1
R138495.810.8960.930
K138497.180.9430.944
D138495.880.9310.941
Temp129394.430.4830.938

Table A2. Agreement between annotations A and B. R, K, and Temp map to pass/gap, with unable-to-determine cases counted as gaps; D maps to pass/gap/not applicable and includes all 1,384 records to retain applicability disagreements. Agreement is the proportion of identical annotations under this mapping; Cohen’s κ\kappa and Gwet’s AC1 correct for chance agreement.

Agreement on the four raw D labels is 95.74%, with κ=0.929\kappa=0.929; merging confirmed errors and unable-to-determine cases removes two disagreements. Among the 1,293 Temp pairs, 1,221 have identical raw labels, while 72 differ between coherent and partially coherent (67 labeled coherent by A and partially coherent by B, and five in the reverse direction). Audit-category agreement is 94.43%, with κ=0.483\kappa=0.483; four-category raw-label agreement is 94.43%, with κ=0.490\kappa=0.490. For raw Temp labels, observed agreement is po=1221/1293=0.9443p_o = 1221/1293 = 0.9443, and chance agreement computed from the A/B category proportions is pe=0.8908p_e = 0.8908, giving κ=(po−pe)/(1−pe)=0.490\kappa= (p_o - p_e)/(1 - p_e) = 0.490. The prevalence of coherent labels raises chance agreement [1]; A/B also assign 15/77 partially coherent labels, reflecting different judgments at this boundary. Table 2 excludes three fixed examples and uses 1,290 pairs, hence its slightly different full-category coefficient.

Effects of Annotation Differences on Filtering Results

Label setFilterUp/down/unchangedMean Δ\DeltaCRActions removed
AR8/4/0+6.86372
AK10/2/0+9.12638
ARK9/3/0+9.13850
BR8/4/0+6.51396
BK11/1/0+10.46605
BRK10/2/0+8.87843
ConsensusR8/4/0+6.66399
ConsensusK10/2/0+9.29613
ConsensusRK9/3/0+8.86849

Table A3. Filtering results from annotations A and B and consensus labels. Mean Δ\DeltaCR is in percentage points.

Annotation for the New Model

GPT-6 Sol’s two R/K annotation rounds use the same frozen key-fact list. Raw agreement for citation risk, external support, and key-fact coverage is 98.96%, 95.83%, and 98.96%, respectively; six category disagreements are adjudicated before replay. The 96 D labels and 87 Temp labels agree in both rounds; both have constant categories, making Cohen’s κ\kappa undefined.

Annotation Inputs and Decision Records

Each annotation retains the target event, original statement, verification basis, category, and brief rationale. Materials are linked by decision time, and account statements are checked by security, pre- or post-trade timing, and calculation convention. Report text is supplied as content to be verified.

Annotation stageInput materialsDecision record
Key-fact extractionSecurity, decision time, and saved external materialsFact content, source, importance, and insufficient-material status
RReport citations, trading action, corresponding sources, and current contextClaims requiring verification, original passages providing support or contradiction, and citation risk
KFrozen fact list, report, and trading actionFacts addressed, partially addressed, or unaddressed, and coverage assessment
DReport, current price, pre- and post-trade accounts, and execution recordsConfirmed error, no confirmed error found, no verifiable account assessment, or indeterminate, with calculation bases retained
TempAdjacent trades by the same model in the same security, their reports, current materials, and account statesCoherent, partially coherent, incoherent, or indeterminate, with the basis for the change explained

Table A4. Inputs and outputs of semantic annotation.

R checks correspondence between cited facts and their sources; when titles do not match, readable facts are checked to determine citation risk. Insufficient materials are labeled unable to determine. K checks whether relevant facts are fully addressed: for example, citing revenue growth to support buying without addressing a major loss in the same material. K is also triggered when all citations concern industry or macroeconomic information, their assessments oppose the actual trade, and review finds them insufficient to support the stock-specific decision. D distinguishes daily price changes, unrealized position gains, sale proceeds, and realized profits before comparing values on the same basis. Temp checks whether trade changes are explained by the preceding and subsequent reports, applying the same criterion to continued and reversed directions.

Model Annotation and Expert-Consensus Agreement

After forming consensus, we select nine trades as fixed few-shot examples for R, K, and D, and

three adjacent trade pairs for Temp. Examples include original materials, labels, and rationales, with the same examples used across groups for both models. GPT-6 Pro and GPT-6 Astra independently produce labels and rationales from reports, decision-time materials, frozen key-fact lists, and relevant account records, which are then compared with expert consensus.

After excluding all examples, evaluation includes 1,375 records each for R, K, and D, and 1,290 pairs for Temp. Labels follow the main-text audit mapping: unable-to-determine cases count as gaps, only coherent Temp labels count as passes, while partially coherent and incoherent labels count as gaps, and D retains a not-applicable category. Table A5 reports author A/B agreement and agreement of both models with expert consensus on the same samples. Mean rows equally average agreement, κ\kappa, AC1, and F1 across R, K, D, and Temp. Within each metric, macro-F1 averages equally over categories occurring in either annotation, while gap F1 treats gap as the target category.

ComparisonMetricnAgree. %κ\kappaAC1Macro-F1Gap F1
A/BR137595.780.8950.930--
K137597.240.9440.945--
D137595.850.9300.941--
Temp129094.500.4800.939--
Mean-95.840.8120.939--
GPT-6 ProR137592.510.8230.8700.9120.877
K137598.470.9690.9700.9850.983
D137595.420.9220.9350.9260.833
Temp129092.950.4050.9200.7020.442
Mean-94.840.7800.9240.8810.784
GPT-6 AstraR137590.690.7830.8370.8910.850
K137596.150.9230.9230.9610.958
D137590.180.8400.8590.8560.673
Temp129094.810.6570.9390.8290.685
Mean-92.960.8010.8890.8840.792

Table A5. Annotator and model agreement under audit-category mapping. A/B compares the two independent author annotations before consensus; both models are compared separately with expert consensus. All examples are excluded. Agreement is reported as a percentage.

Under audit-category merging, mean agreement is 94.84% for GPT-6 Pro and 92.96% for GPT-6 Astra. Astra has higher agreement and gap F1 on Temp, whereas Pro has higher agreement and gap F1 on R, K, and D.

MetricA %B %Pro %Astra %Pro disagreementsAstra disagreements
R96.2999.4992.5190.69103128
K97.8299.4298.4796.152153
D96.6599.2095.4290.1863135
Temp95.0499.4692.9594.819167
Mean96.4599.3994.8492.96278383

Table A6. Agreement of independent author annotations and both models with expert consensus. The same samples as in Table A5 are used, with examples excluded. Agreement is reported as a percentage, and disagreements count records whose model labels differ from expert consensus in audit category. The Mean row equally averages agreement across the four metrics and sums disagreements. A and B both contribute to consensus; their agreement describes label changes during consensus formation.

Raw labels retain distinctions such as unable to determine, partially coherent, and not applicable. Main-text Table 2 reports agreement without category merging, using the same samples and model annotations as Table A5 and differing only in category merging.

ExampleCheckJudgment criterionLabels
1R/K/DThe repurchase remains a proposal; cash and zero-position statements match the ledgerP/P/P
2R/K/DForecast profit is described as realized earnings; the position percentage is verifiableG/P/P
3R/K/DTurnover is described as confirmed net fund inflowsG/P/P
4R/K/DFavorable facts do not support the sale direction, and key facts are not adequately addressedG/G/P
5R/K/DThe citation is not distorted, but key facts on declining installations and exports are omittedP/G/P
6R/K/DExternal facts and account assessments are checked separately; the latter contain a confirmed errorP/P/G
7R/K/DProfit/loss statements are checked against the cost of the same security’s positionP/P/G
8R/K/DGeneric risk language does not constitute a verifiable account statementP/P/N
9R/K/DThe formal report is missing, retaining an insufficient-material statusU/U/U
10TempNew market risk explains the transition from buying to sellingCoherent
11TempReentry is explained, but the previously stated position criterion is not addressedPartially coherent
12TempThe earlier formal report is missingUnable to determine

Table A7. Fixed few-shot examples and judgment criteria. Trade labels are listed in R/K/D order. P, G, U, and N denote pass, gap, unable to determine, and not applicable, respectively; Temp retains its raw labels.

Sample Statistics and Supplementary Results

Statistical Definitions

Formal R/K/D consensus labels cover 1,384 trades at 1,070 trading points, which define the point-level sample in main-text Tables 3 and 4. Table B1 and Table B1a report formal labels; Table B1b lists per-model annotation counts and D account-claim coverage.

ModelRKRKDTemp
Claude Opus 4.5155/0149/6151/4115/0148/0
GLM 4.7 Flash87/085/286/129/374/5
DeepSeek V3.2179/0171/8172/780/0168/5
Gemini 3.1 Pro194/3190/7190/7131/4186/7
Claude Haiku 4.5182/1176/7178/5173/5174/0
Qwen3 Max94/089/590/454/084/0
DeepSeek V3.151/050/150/112/043/0
GLM 4.732/131/231/225/123/3
MiniMax M2.184/082/282/250/073/2
GPT-5.10/00/00/00/00/0
GPT-5.2175/0166/9166/9122/0166/0
Qwen3 235B48/048/048/014/237/3
Gemini 2.5 Flash96/294/494/445/290/2

Table B1. Assessable/unable-to-determine counts for consensus labels. D records with no account assessment are excluded from both counts and reported in Table B1a. Temp counts are trade pairs. Across all models, unable-to-determine rates for R, K, RK, D, and Temp are 0.51%, 3.83%, 3.32%, 1.96%, and 2.09%, respectively, using the same denominators as their gap rates.

Main-text RK statistics pool all 1,070 annotated trading points: 356 pass, 678 have confirmed gaps, and 36 are undetermined only, accounting for 33.27%, 63.36%, and 3.36%, respectively. The latter two categories give 714 gap points (66.73%). A point with a confirmed R or K gap belongs to the confirmed category even if the other check is undetermined, avoiding overlap with the undetermined-only category. These pooled counts differ from equally averaging per-model percentages.

Tables 3 and 4 in the main text report results by trading point, using each model’s decision points with actual trades as the denominator and counting repeated gaps at a point once per check. All-point results appear in B.6. The detailed statistics below retain each check’s original sample unit: Report uses scheduled windows; Rpt, Call, and Resp use their applicable records; R/K is measured over formal active trades, D over events involving account assessments, and Temp over applicable adjacent trade pairs. Detailed gap rates equal (failed + unable-to-determine records)/applicable records, excluding structurally inapplicable items from the denominator. The table below separates assessable and unable-to-determine counts.

ModelConfirmed errorNo error foundNo account assessmentUnable to determine
Claude Opus 4.59106400
GLM 4.7 Flash1514553
DeepSeek V3.21070990
Gemini 3.1 Pro5126624
Claude Haiku 4.57110255
Qwen3 Max1143400
DeepSeek V3.1012390
GLM 4.771871
MiniMax M2.1941340
GPT-5.10000
GPT-5.20122530
Qwen3 235B410322
Gemini 2.5 Flash639512

Table B1a. Complete category counts for D. ’No account assessment’ denotes reports without verifiable account statements.

ModelFormal tradesAnnotated pointsD coverage (%)Calibration (excluded)
Claude Opus 4.515512374.192
GLM 4.7 Flash875336.782
DeepSeek V3.217912044.697
Gemini 3.1 Pro19711268.539
Claude Haiku 4.518314797.278
Qwen3 Max948657.450
DeepSeek V3.1514623.530
GLM 4.7333078.790
MiniMax M2.1846859.521
GPT-5.100N/A0
GPT-5.217516169.715
Qwen3 235B484033.331
Gemini 2.5 Flash988447.965

Table B1b. Formal annotation scope and account-claim coverage. D coverage divides applicable account assessments by formal trades; calibration trades do not enter R/K/D gap counts.

GPT-5.2 has 175 formally annotated trades at 161 points: 122 contain account assessments and 53 are inapplicable to D. Expert consensus identifies no confirmed errors or unable-to-determine cases, so D gaps are zero in both main-text Table 4 and Table B1a. Temp uses an independent cohort of 1,293 labeled pairs; first trades are structurally inapplicable. Expert unable-to-determine labels count as gaps. Appendix B.5 reports the full category distribution.

R/K Overlap and Call-Weighted Comparisons

ModelRKR∩KR \cap KR onlyK onlyR∪KR \cup K
Claude Opus 4.5556123323893
GLM 4.7 Flash264011152955
DeepSeek V3.23796191877114
Gemini 3.1 Pro6586263960125
Claude Haiku 4.59172296243134
Qwen3 Max20418123353
DeepSeek V3.115271051732
GLM 4.7813351018
MiniMax M2.1253313122045
GPT-5.1000000
GPT-5.222645175981
Qwen3 235B1427682135
Gemini 2.5 Flash215310114364
All399613163236450849

Table B2. Action counts flagged by final consensus labels. The union is deduplicated by actual action.

ModelRKRK
Claude Opus 4.537.036.458.4
GLM 4.7 Flash31.042.961.6
DeepSeek V3.220.452.464.4
Gemini 3.1 Pro33.946.866.2
Claude Haiku 4.551.139.973.4
Qwen3 Max23.340.754.7
DeepSeek V3.129.355.464.1
GLM 4.725.043.358.3
MiniMax M2.130.542.355.9
GPT-5.1N/AN/AN/A
GPT-5.212.438.047.3
Qwen3 235B33.855.875.8
Gemini 2.5 Flash20.854.865.7

Table B3. Call-weighted R/K gap-rate comparisons (%). Security-level events within each call are counted using the corresponding weights; the detailed event-level results weight events equally.

Per-Model Joint Eight-Metric Results

Joint statistics use decision points as the unit, combining all assessed applicable checks at each point. A point fails if any check has a confirmed failure; if no check has a confirmed failure but an unable-to-determine item exists, the point is undetermined; a point passes only when all assessed applicable checks pass. Unannotated items are excluded from semantic checks. At non-trading points, only O’s four checks are assessed. Table B4 covers all decision points and summarizes completed assessments at each point; Table B5 matches the main text and includes only annotated trading points. Gap rates combine failures and undetermined cases.

ModelPointsPass/fail/ undeterminedJoint gap rate %Undetermined %
Claude Opus 4.5246162/82/234.150.81
GLM 4.7 Flash246189/56/123.170.41
DeepSeek V3.2246151/90/538.622.03
Gemini 3.1 Pro246152/91/338.211.22
Claude Haiku 4.524697/146/360.571.22
Qwen3 Max246195/48/320.731.22
DeepSeek V3.1246214/31/113.010.41
GLM 4.7246202/44/017.890.00
MiniMax M2.1246196/48/220.330.81
GPT-5.1246246/0/00.000.00
GPT-5.2246161/77/834.553.25
Qwen3 235B246157/88/136.180.41
Gemini 2.5 Flash246164/81/133.330.41
All31982286/882/3028.520.94

Table B4. Joint results over all points.

ModelPointsPass/fail/ undeterminedJoint gap rate %Undetermined %
Claude Opus 4.512339/82/268.291.63
GLM 4.7 Flash538/44/184.911.89
DeepSeek V3.212028/87/576.674.17
Gemini 3.1 Pro11219/90/383.042.68
Claude Haiku 4.514713/131/391.162.04
Qwen3 Max8635/48/359.303.49
DeepSeek V3.14614/31/169.572.17
GLM 4.7307/23/076.670.00
MiniMax M2.16823/43/266.182.94
GPT-5.100/0/0N/AN/A
GPT-5.216177/76/852.174.97
Qwen3 235B406/33/185.002.50
Gemini 2.5 Flash8420/63/176.191.19
All1070289/751/3072.992.80

Table B5. Joint results at annotated trading points. The sample scope matches main-text Tables 3 and 4.

Complete Original Investment Performance

TypeModel / BenchmarkCR%MDD%Sharpe
LLMClaude Opus 4.510.84-7.981.79
GLM 4.7 Flash8.48-5.831.83
DeepSeek V3.27.50-3.921.81
Gemini 3.1 Pro7.38-9.561.06
Claude Haiku 4.56.00-9.081.05
Qwen3 Max5.46-6.411.20
DeepSeek V3.14.53-2.422.23
GLM 4.72.64-7.660.78
MiniMax M2.11.41-4.330.51
GPT-5.10.000.000.00
GPT-5.2-1.43-11.63-0.11
Qwen3 235B-1.85-4.58-0.72
Gemini 2.5 Flash-3.59-4.06-2.06
MarketEqual-weight buy-and-hold1.66-11.920.33
SSE 50-1.87-8.34-0.24
StrategyRSI3.25-2.251.45
MACD-1.85-6.02-0.42
SMA3.90-10.770.86
KDJ (closing price)-5.50-7.96-1.58
Mean reversion-0.92-3.77-0.25

Table B6a. Investment performance of models and reference benchmarks. CR and MDD are percentages; an MDD closer to zero indicates a smaller drawdown.

Model / baselineCR (%)SharpeCalmar
Claude Opus 4.510.841.794.66
GLM 4.7 Flash8.481.834.87
DeepSeek V3.27.501.816.34
Gemini 3.1 Pro7.381.062.56
Claude Haiku 4.56.001.052.16
Qwen3 Max5.461.202.77
DeepSeek V3.14.532.236.03
GLM 4.72.640.781.09
MiniMax M2.11.410.511.01
GPT-5.10.000.000.00
GPT-5.2-1.43-0.11-0.37
Qwen3 235B-1.85-0.72-1.22
Gemini 2.5 Flash-3.59-2.06-2.62
Mean (13 models)3.640.722.10
Equal-weight B&H1.660.330.43
SSE 50-1.87-0.24-0.68
RSI3.251.454.59
MACD-1.85-0.42-0.92
SMA3.900.861.16
KDJ-5.50-1.58-2.00
Mean reversion-0.92-0.25-0.75

Table B6b. Original account and benchmark performance. CR is reported as a percentage; Mean equally averages each metric across all 13 models. Full metrics appear in Appendix § 8.§ 4.

Temp Label Distribution and Boundary Cases

LabelAnnotator AAnnotator BFinal consensus
Coherent125111891196
Partially coherent157770
Incoherent000
Insufficient materials; indeterminate272727
Semantically unresolved000

Table B7. Temp label distribution for adjacent trade pairs. Each column contains 1,293 formal pairs; only coherent labels count as passing, and the other applicable labels count as gaps.

The final consensus comprises 1,196 coherent, 70 partially coherent, and 27 unable-to-determine pairs. Both partially coherent and unable-to-determine cases count as gaps, giving a Temp gap rate of 7.50% (97/1,293); unable-to-determine cases account for 2.09%.

A / BCoherentPartially coherentIncoherentUnable to determine
Coherent11846700
Partially coherent51000
Incoherent0000
Unable to determine00027

Table B7a. Expert A/B labels for Temp. All 1,293 pairs; rows denote A and columns B.

All 72 disagreements concern the boundary between coherent and partially coherent, with 60 occurring when the trading direction reverses. Joint review classifies 60 as partially coherent and 12 as coherent. In the original annotations, symmetric A/B F1 for the partially coherent category is only 0.217, reflecting substantial differences between the annotators’ judgments at this boundary. Disagreements mainly concern how to interpret local problems in trading rationales: missing or unsupported core grounds for a transition affect the explanation linking adjacent trades; problems limited to incidental facts do not lower the coherence judgment when the main transition grounds remain supported.

All-Point and Detailed Audit Results

Of all 3,198 decision points, 912 have a gap in at least one assessed check, yielding an all-point joint gap rate of 28.52%. Tables B8a and B8b summarize the complete decision trajectories without counting unannotated items as gaps; Table B9 retains statistics by applicable record, formally annotated trade event, and labeled adjacent trade pair. Main-text trading-point counts and joint gap counts appear in Table B5.

ModelReportRptCallRespRKRKDTemp
Claude Opus 4.50.01.21.21.235.539.460.07.82.0
GLM 4.7 Flash0.06.54.50.429.946.063.256.213.9
DeepSeek V3.21.25.70.80.420.753.663.712.58.1
Gemini 3.1 Pro1.65.34.93.733.043.763.56.78.3
Claude Haiku 4.50.013.07.77.749.739.373.242.76.9
Qwen3 Max0.00.80.80.821.343.656.420.46.0
DeepSeek V3.10.00.00.00.029.452.962.70.04.7
GLM 4.79.89.80.40.024.239.454.530.811.5
MiniMax M2.10.02.80.40.029.839.353.618.010.7
GPT-5.10.00.00.00.0N/AN/AN/AN/AN/A
GPT-5.20.01.24.14.112.636.646.30.00.0
Qwen3 235B0.025.65.34.129.256.272.937.522.5
Gemini 2.5 Flash1.211.03.73.321.454.165.317.015.2

Table B9. Detailed audit gap rates by applicable record, trade event, and trade pair (%, ↓). Report uses scheduled windows; Rpt, Call, and Resp use their applicable records; R, K, and RK use trade events; D uses events involving account assessments; Temp uses adjacent trade pairs. N/A denotes no applicable samples.

ModelORKBJoint
Claude Opus 4.51.232.54.534.1
GLM 4.7 Flash9.316.38.923.2
DeepSeek V3.26.135.48.138.6
Gemini 3.1 Pro6.535.08.138.2
Claude Haiku 4.513.045.931.760.6
Qwen3 Max0.819.55.720.7
DeepSeek V3.10.013.00.813.0
GLM 4.710.27.33.717.9
MiniMax M2.13.316.75.320.3
GPT-5.10.00.00.00.0
GPT-5.24.132.10.034.6
Qwen3 235B26.413.05.336.2
Gemini 2.5 Flash11.423.66.133.3

Table B8a. Dimensional and joint audit gap rates across all decision points (%, ↓). Each model’s 246 decision points form the denominator; each point is counted once per check.

ModelReportRptCallRespRKDTemp
Claude Opus 4.50.01.21.21.222.022.03.71.2
GLM 4.7 Flash0.06.54.50.49.312.66.54.5
DeepSeek V3.21.25.70.80.414.230.94.15.7
Gemini 3.1 Pro1.65.34.93.720.726.03.76.5
Claude Haiku 4.50.013.07.77.733.726.029.34.5
Qwen3 Max0.00.80.80.88.114.64.12.0
DeepSeek V3.10.00.00.00.06.111.00.00.8
GLM 4.79.89.80.40.03.35.33.31.2
MiniMax M2.10.02.80.40.08.913.03.73.3
GPT-5.10.00.00.00.00.00.00.00.0
GPT-5.20.01.24.14.18.925.20.00.0
Qwen3 235B0.025.65.34.15.79.82.43.7
Gemini 2.5 Flash1.211.03.73.38.119.93.35.7

Table B8b. Gap rates for the eight audit checks across all decision points (%, ↓). Each model’s 246 decision points form the denominator; each point is counted once per check.

Historical Replay and Ablation Analysis

Execution and Valuation

The main-experiment simulator uses current snapshot prices. Orders that pass the checks execute for the full requested quantity; rejected orders do not change cash or positions. Execution is immediate fill or rejection, without pending orders, partial fills, or cancellation of resting orders. Execution auditing aggregates the share quantities in fill records and compares them with position changes in the same window. Slippage costs are analyzed separately in C.6.

The simulated ledger updates cash and positions through fill records and updates sellable quantities across trading days; dividends and stock distributions are not entered as separate corporate-action ledger events.

All main-experiment replays start with CNY 1 million and use shared historical prices. Sales precede purchases within each window, with same-direction trades ordered by security. Retained actions keep their original net quantities as targets. Purchases are sized to available cash, 100-share lots, and fees; sales are sized to existing positions and T+1 sellable quantities. Accounts are continuously marked to market, and positions still open at the end are valued at historical prices. Each trade incurs a 0.03% commission with a minimum of CNY 5, plus a 0.05% stamp duty on sales.

All filters remove trades with failed or unable-to-determine checks. O removes all net trades at a decision point when any of its four record checks has a gap; RK takes the union of R/K removal sets; B removes 164 D gap actions or 93 Temp gap actions, with 32 overlapping actions and a union of 225. A Temp gap is assigned to the later trade in the labeled pair. No-account-assessment and first-trade cases do not independently trigger removal.

Of the 1,293 Temp pairs, 1,257 correspond to formal R/K/D actions: 1,164 are coherent, 66 partially coherent, and 27 unable to determine. The latter two categories give 93 removals. Another 36 pairs correspond to calibration actions, including four partially coherent pairs counted only in point-level audits. Increases and decreases require ΔCR\Delta\mathrm{CR} above +0.01 or below -0.01 percentage points, respectively; all other changes are classified as unchanged. Main results cover 12 actively trading models.

Complete Three-Dimensional Ablations for Each Model

ModelOriginalORKB
Claude Opus 4.510.8413.682.0414.54
GLM 4.7 Flash8.483.0427.537.70
DeepSeek V3.27.5011.5236.1920.19
Gemini 3.1 Pro7.388.2632.005.84
Claude Haiku 4.56.007.6810.6727.02
Qwen3 Max5.466.714.399.08
DeepSeek V3.14.534.5311.594.39
GLM 4.72.642.668.595.14
MiniMax M2.11.4112.428.618.26
GPT-5.10.000.000.000.00
GPT-5.2-1.430.96-3.32-1.43
Qwen3 235B-1.8523.1913.5613.79
Gemini 2.5 Flash-3.59-3.501.896.17

Table C1. Original CR and CR after single-dimension filtering (%). O, RK, and B represent execution traceability, evidence sufficiency, and financial meta-awareness, respectively.

ModelO+RKO+BRK+BO+RK+B
Claude Opus 4.52.0419.038.798.79
GLM 4.7 Flash22.48-2.4524.2517.42
DeepSeek V3.236.2318.0237.7537.80
Gemini 3.1 Pro30.489.3419.4816.12
Claude Haiku 4.517.5132.5911.3812.77
Qwen3 Max5.138.094.764.76
DeepSeek V3.111.594.3911.5911.59
GLM 4.79.164.677.858.42
MiniMax M2.18.798.688.848.69
GPT-5.10.000.000.000.00
GPT-5.2-3.500.96-3.32-3.50
Qwen3 235B14.2524.3712.0312.73
Gemini 2.5 Flash1.307.003.023.55

Table C2. CR after pairwise and three-dimensional joint filtering (%). Each cell is an independent replay result.

FilterMean CR %Buys/sells removedSharpeSortinoCalmar
Original3.950/00.781.552.27
R10.60126/2731.011.532.81
K13.23201/4121.422.354.56
D8.4341/1231.121.912.84
Temp7.4036/571.252.113.31
O7.6058/740.901.492.38
RK12.81286/5631.392.374.95
B10.0668/1571.221.983.12
O+RK12.96308/5851.352.254.39
O+B11.22115/2091.161.883.03
RK+B12.20315/5931.372.254.33
O+RK+B11.59337/6111.302.073.62

Table C3. Cumulative returns, trade removals, and risk-adjusted returns for submetrics and combinations. CR and risk-adjusted return metrics are equally weighted means over 12 active models. Higher Sharpe, Sortino, and Calmar values are better.

R/K construction, filtering analysis, and direction-matched random controls use the same historical trades, including exploratory analysis on that dataset and no separate held-out set. Results describe portfolio changes along fixed trading trajectories.

Replay Checks for the New Model

GPT-6 Sol’s 96 executed trades yield an original CR of 4.85%. RK flags 41 trades: 13 purchases and 28 sales, including 36 confirmed gaps and five unable-to-determine cases. The 28 flagged sales make exits a prominent part of the intervention. O, D, and Temp have no gaps, so their separate filters leave CR unchanged.

Decision-level RK filtering replaces flagged trades with No Trade and replays the others at their original times and target quantities under updated account constraints. Removing 41 trades prevents 22 additional orders from executing; 33 trades execute and CR reaches 38.28%. This result reflects the new cash and holding path, not the isolated payoffs of removed trades. Filtering the 36 confirmed gaps alone raises CR by 23.91 percentage points; SH688008 contributes a 28.50-point increase, exceeding the net portfolio change.

Risk-Adjusted Return Calculations and Complete Results

All main-experiment filtering conditions use daily 14:00 account assets and initial capital of CNY 1 million to calculate 82 daily returns, with an annualization factor of 252 and zero risk-free and downside target returns. Sharpe is the mean daily return divided by the sample standard deviation of daily returns, multiplied by 252\sqrt{252}; Sortino replaces the denominator with the square root of the mean squared negative returns over all trading days; Calmar is the evaluation period’s annualized return divided by the absolute daily maximum drawdown. Metrics are calculated for each model first, then equally averaged over 12 models.

Ratio increases and decreases use unrounded values, with absolute differences no greater than 10−810^{-8} treated as unchanged; return changes in percentage points retain the 0.01 threshold.

FilterCR%AR%SharpeSortinoCalmar
original3.9513.260.781.552.27
R10.6039.881.011.532.81
K13.2349.701.422.354.56
D8.4330.761.121.912.84
Temp7.4025.121.252.113.31
O7.6026.810.901.492.38
RK12.8150.251.392.374.95
B10.0636.211.221.983.12
O+RK12.9650.311.352.254.39
O+B11.2242.191.161.883.03
RK+B12.2046.411.372.254.33
O+RK+B11.5943.541.302.073.62

Table C4. Complete mean financial metrics for each filtering condition. Higher values are better for all metrics.

FilterSharpeSortinoCalmar
original0/0/120/0/120/0/12
R8/4/08/4/08/4/0
K9/3/08/4/09/3/0
D7/3/27/3/27/3/2
Temp6/5/17/4/17/4/1
O6/5/16/5/16/5/1
RK8/4/07/5/08/4/0
B7/4/16/5/16/5/1
O+RK9/3/07/5/09/3/0
O+B8/4/07/5/07/5/0
RK+B8/4/07/5/08/4/0
O+RK+B8/4/06/6/09/3/0

Table C5. Numbers of models with improved risk-adjusted returns. Each cell shows increased/decreased/unchanged counts, summing to 12.

ModelOriginalRK
SharpeSortinoCalmarSharpeSortinoCalmarSharpeSortinoCalmar
Claude Haiku 4.51.051.482.161.492.073.691.231.582.71
Claude Opus 4.51.793.314.660.600.920.891.512.553.17
DeepSeek V3.12.234.126.030.210.310.252.403.549.07
DeepSeek V3.21.813.976.341.381.783.542.394.467.93
Gemini 2.5 Flash-2.06-2.65-2.62-0.59-0.79-0.951.171.951.95
Gemini 3.1 Pro1.061.962.561.172.112.932.123.649.53
GPT-5.2-0.11-0.17-0.371.291.852.77-0.54-0.77-0.81
MiniMax M2.10.510.851.01-0.02-0.02-0.550.961.251.75
Qwen3 235B-0.72-0.94-1.221.692.315.741.241.653.16
Qwen3 Max1.201.912.771.331.973.840.380.560.38
GLM 4.70.781.281.091.652.265.391.773.644.60
GLM 4.7 Flash1.833.484.871.883.526.212.444.2211.27

Table C6. Per-model risk-adjusted returns. Each filtering condition is replayed independently, retaining original results for comparison.

ModelDTempO
SharpeSortinoCalmarSharpeSortinoCalmarSharpeSortinoCalmar
Claude Haiku 4.51.251.622.992.474.938.740.911.181.72
Claude Opus 4.52.144.026.361.793.495.231.983.785.75
DeepSeek V3.12.234.126.032.173.955.832.234.126.03
DeepSeek V3.22.555.077.321.542.455.831.542.464.91
Gemini 2.5 Flash0.791.021.530.791.031.48-1.97-2.56-2.58
Gemini 3.1 Pro1.622.414.280.901.452.150.951.542.28
GPT-5.2-0.11-0.17-0.37-0.11-0.17-0.370.240.370.16
MiniMax M2.1-0.59-0.86-0.801.011.322.011.061.402.23
Qwen3 235B0.040.06-0.031.001.302.001.481.974.33
Qwen3 Max1.031.451.631.322.093.240.911.431.48
GLM 4.71.161.982.210.941.531.300.821.321.10
GLM 4.7 Flash1.292.262.871.111.932.340.650.831.20

Table C6 (PDF p. 26). Per-model risk-adjusted returns (continued).

ModelRKBO+RK
SharpeSortinoCalmarSharpeSortinoCalmarSharpeSortinoCalmar
Claude Haiku 4.50.911.151.661.602.225.191.171.512.62
Claude Opus 4.50.360.510.311.863.385.310.360.510.31
DeepSeek V3.13.396.6111.392.173.955.833.396.6111.39
DeepSeek V3.21.972.747.942.273.947.441.962.737.96
Gemini 2.5 Flash0.390.590.450.771.001.420.300.450.31
Gemini 3.1 Pro2.394.6012.080.791.241.732.464.8510.81
GPT-5.2-0.25-0.34-0.52-0.11-0.17-0.37-0.30-0.40-0.55
MiniMax M2.10.750.961.130.730.961.010.760.981.16
Qwen3 235B1.391.863.881.131.502.471.441.944.13
Qwen3 Max0.540.740.631.081.541.820.590.820.73
GLM 4.72.013.845.851.161.982.212.134.186.28
GLM 4.7 Flash2.865.1914.631.212.173.361.952.797.55

Table C6 (PDF p. 26, second). Per-model risk-adjusted returns (continued).

ModelO+BO+BO+BRK+BRK+BRK+BO+RK+BO+RK+BO+RK+B
SharpeSortinoCalmarSharpeSortinoCalmarSharpeSortinoCalmar
Claude Haiku 4.51.672.315.630.981.272.041.081.402.46
Claude Opus 4.52.094.036.370.971.451.570.971.451.57
DeepSeek V3.12.173.955.833.396.6111.393.396.6111.39
DeepSeek V3.21.993.335.882.032.838.432.022.828.44
Gemini 2.5 Flash0.851.111.640.560.860.740.640.990.89
Gemini 3.1 Pro1.041.782.411.793.165.741.652.874.00
GPT-5.20.240.370.16-0.25-0.34-0.52-0.30-0.40-0.55
MiniMax M2.10.750.991.070.750.991.110.740.981.08
Qwen3 235B1.522.034.591.291.703.471.351.783.70
Qwen3 Max0.871.291.400.560.770.670.560.770.67
GLM 4.71.071.812.001.703.013.991.813.284.30
GLM 4.7 Flash-0.36-0.44-0.682.674.7113.281.662.295.47

Table C6 (PDF p. 27). Per-model risk-adjusted returns (continued).

ConditionCR%SharpeSortinoCalmar
Original4.851.021.881.94
RK38.282.865.2012.84

Table C7. Daily return and risk-adjusted metrics for GPT-6 Sol.

Confirmed Errors and Random Filtering

Confirmed-error-only replay retains trades labeled unable to determine. O removes 121 trades with confirmed record-verification discrepancies; RK removes 803 trades with confirmed R or K gaps; B removes trades with confirmed D errors or Temp non-passes, including partially coherent cases, for a union of 190 trades. Combinations take the union of removal sets.

Random filtering samples each model’s formal trades over the full March-to-June period, separately matching the numbers of purchases and sales removed. Sampling is without replacement and repeated 1,000 times; remaining trades are replayed under the same cash, T+1, lot-size, and fee rules. Table C12 reports mean return changes across the 12 actively trading models and the random distributions.

FilterTrades removedAudit Δ\DeltaCRRandom mean Δ\DeltaCRAbove randomRandom 95% range
O132+3.65+1.78+1.87[-1.37, +5.39]
RK849+8.86+5.32+3.55[-0.08, +11.26]
B225+6.11+4.09+2.02[+0.12, +8.32]
O + RK + B948+7.65+4.89+2.75[-0.89, +11.11]

Table C12. Random-filtering controls matched by model and buy/sell direction under the baseline replay rules. Audits and random controls use the removal counts and execution settings in Table 5; intervals are the 2.5th to 97.5th percentiles across 1,000 random controls. Values are in percentage points.

Slippage Sensitivity

Holding labels, trading times, and target quantities fixed, we set slippage at 0, 5, 10, 20, and 50 basis points per side. One basis point equals 0.01%; the purchase execution price is the snapshot price multiplied by 1+b/100001 + b/10000, and the sale execution price is the snapshot price multiplied by 1−b/100001 - b/10000, where bb denotes slippage in basis points per side. Account valuation continues to use snapshot prices, while cash, T+1, 100-share lot, and fee rules remain unchanged. Each filtered account is compared with an unfiltered account under the same slippage setting.

Slippage per side (bps)Unfiltered CRO CRRK CRB CRJoint CRRK Δ\DeltaCRJoint Δ\DeltaCR
03.957.6012.8110.0611.59+8.86+7.65
53.436.9512.539.5611.48+9.10+8.05
102.916.6212.419.2111.40+9.50+8.50
201.865.6112.038.5111.07+10.17+9.21
50-1.413.2410.985.9810.38+12.39+11.78

Table C9. Slippage sensitivity under the main replay setting. CR is the equally weighted mean across 12 actively trading models, expressed as a percentage; Δ\DeltaCR is the difference from the unfiltered account at the same slippage level, in percentage points.

Figure C1. Slippage sensitivity. (a) Mean CR across 12 actively trading models; (b) mean CR difference between filtered and unfiltered accounts at the same slippage level. The horizontal axis shows slippage in basis points per side.

Complete Financial Metrics for Random Controls

Random controls use the fixed seed 20261003, matching removal counts by model and buy/sell direction, and sample without replacement from full-period trades for 1,000 repetitions. Audit and random filtering both use the replay rules in Appendix C.1. Each repetition computes per-model changes from the original accounts and then takes the equally weighted mean across the 12 actively trading models.

Let kk count random changes at least as large as the audit change. Monte Carlo reference values use (k+1)/1001(k + 1)/1001, with Holm adjustment for multiple comparisons. For RK, Sharpe, Sortino, and Calmar adjusted reference values are 0.198, 0.076, and 0.040. These comparisons describe random deletions on the same historical trajectory, not confidence intervals across runs or market periods.

FilterMetricAudit changeRandom mean changeRandom 95% rangeHolm ref.
OΔ\DeltaCR+3.65+1.78[-1.37, +5.39]1.000
OΔ\DeltaSharpe+0.12+0.14[-0.30, +0.57]1.000
OΔ\DeltaSortino-0.06+0.12[-0.59, +0.95]1.000
OΔ\DeltaCalmar+0.11+0.58[-0.64, +2.19]1.000
RKΔ\DeltaCR+8.86+5.32[-0.08, +11.26]1.000
RKΔ\DeltaSharpe+0.61+0.03[-0.52, +0.52]0.198
RKΔ\DeltaSortino+0.82-0.31[-1.12, +0.45]0.076
RKΔ\DeltaCalmar+2.68+0.08[-1.23, +1.70]0.040
BΔ\DeltaCR+6.11+4.09[+0.12, +8.32]1.000
BΔ\DeltaSharpe+0.44+0.18[-0.25, +0.56]1.000
BΔ\DeltaSortino+0.43+0.06[-0.61, +0.92]1.000
BΔ\DeltaCalmar+0.84+0.63[-0.60, +2.72]1.000
O+RK+BΔ\DeltaCR+7.65+4.89[-0.89, +11.11]1.000
O+RK+BΔ\DeltaSharpe+0.52+0.02[-0.61, +0.57]0.831
O+RK+BΔ\DeltaSortino+0.52-0.28[-1.21, +0.61]0.713
O+RK+BΔ\DeltaCalmar+1.35+0.14[-1.35, +1.89]1.000

Table C10. Return and risk-adjusted metrics for audit filtering and random controls. The random 95% range comprises the 2.5th to 97.5th percentiles of the 1,000 replays. Δ\DeltaCR is in percentage points; other metrics are ratio differences. Higher values indicate more favorable changes relative to the original accounts.

Summary of Filtering Definitions and Random Controls

FilterTrades removedReturn (pp) Δ\DeltaCRRisk-adjusted ratios Δ\DeltaSharpeRisk-adjusted ratios Δ\DeltaSortinoRisk-adjusted ratios Δ\DeltaCalmar
O121+3.50 (9)+0.09 (6)-0.10 (6)+0.05 (6)
RK803+9.13 (11)+0.67 (9)+0.84 (8)+2.19 (8)
B190+4.63 (8)+0.30 (7)+0.26 (7)+0.49 (6)
O + RK + B898+9.39 (11)+0.64 (10)+0.69 (8)+1.56 (9)

Table C11. Replay results with only confirmed errors filtered out and unable-to-determine trades retained. Complete financial metrics appear in Table C8.

SettingFilterCR%AR%SharpeSortinoCalmar
Confirmed errors onlyO7.4526.280.871.452.33
Confirmed errors onlyRK13.0848.781.452.394.47
Confirmed errors onlyB8.5731.871.081.812.76
Confirmed errors onlyO + RK + B13.3349.751.422.243.84

Table C8. Selected mean financial metrics for confirmed-error-only filtering. All values are equally weighted means over the 12 actively trading models.

Reruns and Joint Gap Statistics

Complete Counts for the Three Dimensions

Counts below retain pass, fail, unresolved, not-applicable, and unannotated states by decision point. Gap rates combine failures and expert unable-to-determine labels; unannotated items do not enter the check. Unresolved proportions among assessed applicable points for O, RK, and B are 0.00%, 3.36%, and 3.12%, respectively; proportions for all-point and annotated-trading-point joint audits are 0.94% and 2.80%. D is inapplicable when no account statement exists.

ModelORKBJoint
Claude Opus 4.5243/3/0/0/043/78/2/122/1107/11/0/128/0162/82/2/0/0
GLM 4.7 Flash223/23/0/0/013/40/0/193/028/15/7/196/0189/56/1/0/0
DeepSeek V3.2231/15/0/0/033/81/6/123/399/16/4/127/0151/90/5/0/0
Gemini 3.1 Pro230/16/0/0/026/80/6/132/291/12/8/135/0152/91/3/0/0
Claude Haiku 4.5214/32/0/0/034/110/3/95/469/75/3/99/097/146/3/0/0
Qwen3 Max244/2/0/0/038/45/3/160/064/14/0/168/0195/48/3/0/0
DeepSeek V3.1246/0/0/0/014/31/1/200/038/2/0/206/0214/31/1/0/0
GLM 4.7221/25/0/0/012/16/2/216/017/7/2/220/0202/44/0/0/0
MiniMax M2.1238/8/0/0/027/39/2/177/151/11/2/182/0196/48/2/0/0
GPT-5.1246/0/0/0/00/0/0/246/00/0/0/246/0246/0/0/0/0
GPT-5.2236/10/0/0/082/71/8/80/5160/0/0/86/0161/77/8/0/0
Qwen3 235B181/65/0/0/08/32/0/206/021/10/3/212/0157/88/1/0/0
Gemini 2.5 Flash218/28/0/0/026/55/3/158/465/12/3/166/0164/81/1/0/0

Table D1. Pass/fail/unresolved/not-applicable/unannotated counts. Each cell sums to 246; applicable-check gap rates exclude not-applicable and unannotated items. Table B4 and Table B5 report all-point and annotated-trading-point joint results.

Individual Results for Ten-Case Reruns

The four models are each tested on ten independent historical snapshots, producing 41 net trade events in total. Each test restores market information and the pre-trade account state at the corresponding decision time and uses the same system prompt, interaction-step limit, and number of overall attempts to maintain consistent experimental settings. All outputs are assessed under the same O, RK, and D rules. Figure 4 presents the audit results for each model under the same experimental conditions.

Two rounds of independent blind annotation are performed for R/K/D across the 41 net trade events in the fixed ten-case sets. Raw agreement between the two rounds is as follows:

ItemAgreements/eventsRaw agreementCohen’s κ\kappaGwet’s AC1
R41/41100.0%1.0001.000
K40/4197.6%0.9590.965
D41/41100.0%1.0001.000

Table D1b. Annotation agreement for historical snapshot tests.

Raw agreement includes records both rounds label unable to determine; AC1 is calculated using each item’s complete label categories.

ModelReportRptCallResp
DeepSeek V3.260.060.020.020.0
GLM 4.7 Flash0.010.00.00.0
GPT-6 Astra(max)0.00.00.00.0
GPT-5.6 Sol0.00.00.00.0

Table D2. Gap rates for O’s four checks (%, ↓). Report uses 10 windows as its denominator; the other checks use applicable windows. Complete item states appear in Table D2a.

ModelReportRptCallResp
DeepSeek V3.24/6/0/04/0/6/08/0/2/08/0/2/0
GLM 4.7 Flash10/0/0/09/1/0/010/0/0/010/0/0/0
GPT-6 Astra(max)10/0/0/010/0/0/010/0/0/010/0/0/0
GPT-5.6 Sol10/0/0/010/0/0/010/0/0/010/0/0/0

Table D2a. Complete counts for O’s four checks. Counts are ordered as pass/confirmed failure/undetermined/not applicable.

The joint seven-check audit comprises Report, Rpt, Call, Resp, R, K, and D. O and joint rates use each group’s 10 decision points; RK and D use applicable points, and gaps combine confirmed failures and undetermined cases. D has 5, 2, 10, and 10 applicable points, including 4 unable-to-determine DeepSeek points. GLM’s D gap rate is 1/2=50.0%1/2=50.0\% by decision point and 33.3% by trade event (Table D3c).

ModelTrading windowsJoint passesConfirmed failuresUndetermined
DeepSeek V3.26280
GLM 4.7 Flash3730
GPT-6 Astra(max)10820
GPT-5.6 Sol10550

Table D3. Joint seven-check states. A point passes only when all applicable checks pass with no undetermined items; any confirmed failure takes precedence and determines joint failure; points without failures but with undetermined items remain undetermined.

ModelO pass/fail/undetermined/not applicableRK pass/fail/undetermined/not applicableD pass/fail/undetermined/not applicableJoint pass/fail/undetermined/not applicable
DeepSeek V3.24/6/0/00/2/4/41/0/4/52/8/0/0
GLM 4.7 Flash9/1/0/01/2/0/71/1/0/87/3/0/0
GPT-6 Astra(max)10/0/0/08/2/0/010/0/0/08/2/0/0
GPT-5.6 Sol10/0/0/05/5/0/010/0/0/05/5/0/0

Table D3a. Complete counts for O, RK, D, and the joint seven-check audit. Counts are ordered as pass/confirmed failure/undetermined/not applicable; applicable denominators equal totals minus not-applicable cases, including undetermined cases.

ModelRKD
DeepSeek V3.21/1/4/40/2/4/41/0/4/5
GLM 4.7 Flash2/0/1/71/2/0/71/1/0/8
GPT-6 Astra(max)10/0/0/08/2/0/010/0/0/0
GPT-5.6 Sol10/0/0/05/5/0/010/0/0/0

Table D3b. Decision-point states for R, K, and D. Multi-security events use the joint aggregation rules; no verifiable account assessment maps to D not applicable.

ModelNet trade eventsD pass/error/unable to determine/no account assessmentD gap rate %
DeepSeek V3.291/0/7/187.5
GLM 4.7 Flash42/1/0/133.3
GPT-6 Astra(max)1818/0/0/00.0
GPT-5.6 Sol1010/0/0/00.0

Table D3c. Net trade event counts for account facts. D gap rates combine confirmed errors and unable-to-determine cases and exclude events with no account assessment.

DeepSeek V3.2 lacks formal reports in four trading windows covering seven net trade events, whose R, K, and D records are labeled unable to determine. Statistics cover each model’s fixed ten historical windows; additional GLM attempts are archived separately.

Supplementary Results for Continuous Trading by the New Model

The continuous trading experiment uses the original 246 historical snapshots, carrying account cash and positions forward across windows. RK filtering raises CR from 4.85% to 38.28%, Sharpe from 1.02 to 2.86, and Sortino from 1.88 to 5.20. All-window maximum drawdown magnitude increases from 8.23% to 15.02%; daily financial metrics appear in Table C7.

ModelReportRptCallResp
GPT-6 Sol / medium0.00.00.00.0

Table D4. Gap rates for O’s four checks on GPT-6 Sol’s continuous trajectory (%, ↓). All denominators are 246.

Across all 246 points, 205 pass the joint audit, 36 have confirmed failures, and 5 are unable to determine, giving a joint gap rate of 16.67%. The corresponding trading-point counts are 55, 36, and 5, giving a trading joint gap rate of 42.71%. Temp is not applicable to the first trade, while other applicable checks must still pass. Among 87 trades for which all eight metrics apply, 48 pass, 34 have confirmed failures, and 5 are unable to determine, giving a combined gap rate of 44.83%.

Securities, Data Sources, and Runtime Configurations

Stock Pool and Data Sources

All ten stocks in Table E1 are listed on the Shanghai Stock Exchange’s STAR Market, with accounts and market prices denominated in CNY. Prices and technical indicators use TinySoft market data, while news and announcements are obtained through Tushare, AkShare, Sina Finance, the Shanghai Stock Exchange, and CNINFO, truncated at each decision time.

Security codeCompanyIndustryMarket
SH688008Montage TechnologySemiconductorsSSE STAR Market
SH688111Kingsoft OfficeSoftwareSSE STAR Market
SH688009China Railway Signal & CommunicationRail transportSSE STAR Market
SH688981SMICSemiconductorsSSE STAR Market
SH688256CambriconSemiconductorsSSE STAR Market
SH688271United Imaging HealthcareMedical devicesSSE STAR Market
SH688047Loongson TechnologySemiconductorsSSE STAR Market
SH688617APT MedicalMedical devicesSSE STAR Market
SH688303Daqo New EnergyPhotovoltaic materialsSSE STAR Market
SH688180Junshi BiosciencesBiopharmaceuticalsSSE STAR Market

Table E1. Stock pool.

DataSourceContents
Individual-stock market dataTinySoftPrices and hourly market data for the ten securities, from which technical indicators are calculated
Financial newsTushare, Sina FinanceFinancial information from Sina Finance, Tonghuashun, East Money, Wallstreetcn, Cailian Press, Yicai, JRJ, and other outlets
Stock-specific information and eventsAkShareIndividual-stock news, company developments, and financial events
Company announcementsShanghai Stock Exchange, CNINFOCompany announcements and available announcement texts

Table E2. Data sources.

News retrieval looks back 30 days, returning at most 20 items per security, deduplicated by security, publication time, and normalized title; different models receive the shared snapshot for the same window.

Model and Interaction Configurations

The main experiment calls models through an OpenAI-compatible interface, using a uniform Re-Act workflow with at most 30 interaction steps per window and 3 attempts per agent task. Table E3 lists model identifiers from the actual call records. Models use the interface’s default reasoning configuration.

ModelModel identifier in call recordsAdditional reasoning request
Claude Opus 4.5anthropic/claude-4.5-opus-20251124None specified
GLM 4.7 Flashz-ai/glm-4.7-flash-20260119None specified
DeepSeek V3.2deepseek/deepseek-v3.2None specified
Gemini 3.1 Progoogle/gemini-3.1-pro-preview-20260219None specified
Claude Haiku 4.5anthropic/claude-4.5-haiku-20251001None specified
Qwen3 Maxqwen/qwen3-maxNone specified
DeepSeek V3.1deepseek/deepseek-v3.1-terminusNone specified
GLM 4.7z-ai/glm-4.7; z-ai/glm-4.7-20251222None specified
MiniMax M2.1minimax/minimax-m2.1None specified
GPT-5.1openai/gpt-5.1-20251113None specified
GPT-5.2openai/gpt-5.2-20251211None specified
Qwen3 235Bqwen/qwen3-235b-a22b-04-28None specified
Gemini 2.5 Flashgoogle/gemini-2.5-flashNone specified

Table E3. Main-experiment model identifiers.

Parameters such as temperature and top_p use the interface defaults.

The uniform prompt asks models to inspect candidate stocks, account positions, and trading constraints, autonomously choose to buy, sell, or not trade, and submit structured reports containing material citations, trade descriptions, and decision rationales. Tools cover news retrieval, technical indicators, hourly market data, current prices, account-state queries, purchases, sales, No Trade recording, and report submission. Specific budgets for historical cases and continuous trading by the new model appear in Appendices D.2 and D.3.

Reference Benchmark Settings

The equally weighted buy-and-hold portfolio uses the same ten stocks; SSE 50 uses a normalized index series for the same period as a market reference. Five rule-based strategies start from empty accounts with CNY 1 million and calculate signals on shared snapshots; when a signal is active and the security is not already held, the target purchase value is 10% of pre-trade total assets; when the signal disappears, sellable holdings are sold. Strategies apply 100-share purchase lots, cash constraints, a short-selling prohibition, T+1, a 50% single-stock limit, and the same commissions and stamp duty as the main experiment.

StrategyParametersBuy signal
RSI14 periodsRSI below 30
MACD12/26/9Histogram value above 0
SMA5 and 20 periods5-period moving average above the 20-period moving average
KDJ (closing price)9 periodsK above D and below 80
Mean reversion60 periodsClosing price more than 3% below the rolling mean

Table E4. Strategy benchmark parameters. Periods are calculated over the shared snapshot sequence.

The KDJ benchmark constructs signals using only snapshot closing prices. All market and strategy benchmarks cover the same 82 trading days and use daily closing account values to calculate the financial metrics in Table B6a.

ModelCR%AR%AV%SharpeSortinoMDD%Calmar
Claude Opus 4.510.8437.2118.641.793.31-7.984.66
GLM 4.7 Flash8.4828.4114.241.833.48-5.834.87
DeepSeek V3.27.5024.8812.751.813.97-3.926.34
Gemini 3.1 Pro7.3824.4623.191.061.96-9.562.56
Claude Haiku 4.56.0019.6018.771.051.48-9.082.16
Qwen3 Max5.4617.7614.481.201.91-6.412.77
DeepSeek V3.14.5314.576.182.234.12-2.426.03
GLM 4.72.648.3310.960.781.28-7.661.09
MiniMax M2.11.414.399.250.510.85-4.331.01
GPT-5.10.000.000.000.000.000.000.00
GPT-5.2-1.43-4.3220.95-0.11-0.17-11.63-0.37
Qwen3 235B-1.85-5.577.59-0.72-0.94-4.58-1.22
Gemini 2.5 Flash-3.59-10.625.39-2.06-2.65-4.06-2.62

Table B6. Complete investment performance. GPT-5.1 makes no trades, with zero returns and volatility; its risk ratios are recorded as zero by computational convention.

Material Language and Saved Content

News, company announcements, and annotation instructions are primarily in Chinese, while model reports and tool interfaces contain English structured fields. Annotators view the titles, summaries, or full texts saved in the contemporaneous archives; citation checks and key-fact assessments retain the corresponding original text for reference.

The key-fact material packets used in the independent second round of annotation primarily contain news titles and summaries within titles, and key facts are extracted only from content explicitly stated in these materials. Materials corresponding to the current formal trades comprise 872 packets and 68,741 records. Publication times are no later than the corresponding decision times, and material identifiers cited by key facts can be located in the corresponding packets.

Verification Records and Audit Dimensions

Open-Source Resources

We will release four resource groups: evaluation code, including agents, tool interfaces, and account execution logic; expert annotations, including anonymized A/B labels, final consensus, rationales, and source references; analysis and replay scripts to reproduce audits, filtered replay, and figures; and prompts, including trading and semantic annotation instructions, fixed few-shot examples, and exclusion lists. Market data, news, and announcements are accessed through the system’s data interfaces; the README will document data sources, environment configuration, and execution commands.

References

  1. [1]Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596.
  2. [2]Gagan Bhatia, El Moatez Billah Nagoudi, Hasan Cavusoglu, and Muhammad Abdul-Mageed. 2024. FinTral: A family of GPT-4 level multimodal financial large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13064–13087, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2402.10986
  3. [3]Hongliu Cao, Ilias Driouich, and Eoin Thomas. 2026. Beyond task completion: Revealing corrupt success in llm agents through procedure-aware evaluation. Preprint, arXiv:2603.03116.arxiv.org/abs/2603.03116
  4. [4]Tianyu Cao, Natraj Raman, Danial Derdovic, and Chenhao Tan. 2024. Characterizing multimodal long-form summarization: A case study on financial reports. In First Conference on Language Modeling.arxiv.org/abs/2404.06162
  5. [6]Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.arxiv.org/abs/2109.00122
  6. [9]Yubin Choi, Yujin Yang, Subin Kim, Seokil Ham, Seungju Cho, Jungmin Son, Youngjun Kwak, and Changick Kim. 2026. FinHarmBench: Financial jailbreak benchmark and unsupervised safety fine-tuning via refusal steering distillation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 1714–1726, San Diego, California, USA. Association for Computational Linguistics.
  7. [11]Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, and Liwen Zhang. 2025. FinEval: A Chinese financial domain knowledge evaluation benchmark for large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6258–6292, Albuquerque, New Mexico. Association for Computational Linguistics.arxiv.org/abs/2308.09975
  8. [12]Xin Guo, Rongjuncheng Zhang, Guilong Lu, Xuntao Guo, Jia Shuai, Zhi Yang, and Liwen Zhang. 2026. BizFinBench v2: Towards reliable LLMs in finance via real-user data and Offline/Online bilingual evaluation. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 38642–38676. PMLR.DOI
  9. [13]Lucas Torroba Hennigen, Zejiang Shen, Aniruddha Nrusimha, Bernhard Gapp, David Sontag, and Yoon Kim. 2024. Towards verifiable text generation with symbolic references. In First Conference on Language Modeling.arxiv.org/abs/2311.09188
  10. [15]Miaobao Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Yina Sa, Daren Zha, and Jun Xiao. 2026. Actions with receipts: Jointly binding claims, evidence, and execution for replayable tool-agent auditing. Preprint, arXiv:2610.00327.arxiv.org/abs/2610.00327
  11. [16]Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online. Association for Computational Linguistics.arxiv.org/abs/2004.03685
  12. [17]Daehong Kim, Haichao Miao, and Shusen Liu. 2026. LEDGER: Claim-to-evidence trace graphs for auditing LLM agents. Preprint, arXiv:2608.18398.arxiv.org/abs/2608.18398
  13. [20]Changlun Li, Yao SHI, Chen Wang, Qiqi Duan, Runke RUAN, Weijie Huang, Haonan Long, Lijun Huang, Nan Tang, and Yuyu Luo. 2025a. Time travel is cheating: Going live with deepfund for real-time fund investment benchmarking. In Advances in Neural Information Processing Systems, volume 38, Main Conference. Curran Associates, Inc.arxiv.org/abs/2505.11065
  14. [21]Haohang Li, Yupeng Cao, Yangyang Yu, Shashidhar Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, K.P. Subbalakshmi, Jimin Huang, Lingfei Qian, Xueqing Peng, Jordan W. Suchow, and Qianqian Xie. 2025b. INVESTORBENCH: A benchmark for financial decision-making tasks with LLM-based agent. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2509–2525, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2412.18174
  15. [22]Junjie Li, Yang Liu, Weiqing Liu, Shikai Fang, Lewen Wang, Chang XU, and Jiang Bian. 2025c. Mars: a financial market simulation engine powered by generative foundation model. In International Conference on Learning Representations, volume 2025, pages 39490–39524.arxiv.org/abs/2409.07486
  16. [23]Yuan Li, Bingqiao Luo, Qian Wang, Nuo Chen, Xu Liu, and Bingsheng He. 2024. CryptoTrade: A reflective LLM-based agent to guide zero-shot cryptocurrency trading. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1094–1106, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2407.09546
  17. [25]Shu Liu, Shangqing Zhao, Chenghao Jia, Xinlin Zhuang, Zhaoguang Long, Jie Zhou, Aimin Zhou, Man Lan, and Yang Chong. 2025a. FinDABench: Benchmarking financial data analysis ability of large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 710–725, Abu Dhabi, UAE. Association for Computational Linguistics.arxiv.org/abs/2401.02982
  18. [26]Xiao-Yang Liu, Ziyi Xia, Jingyang Rui, Jiechao Gao, Hongyang Yang, Ming Zhu, Christina Wang, Zhaoran Wang, and Jian Guo. 2022. Finrl-meta: Market environments and benchmarks for data-driven financial reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pages 1835–1849. Curran Associates, Inc.
  19. [27]Xuan Liu, Haoyang Shang, and Haojian Jin. 2026a. CoBRA: Programming cognitive bias in social agents using classic social science experiments. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems.arxiv.org/abs/2509.13588
  20. [28]Xuan Liu, Haoyang Shang, and Haojian Jin. 2026b. Diagon: A programmable testbed for ai-agent cognitive labor markets. Preprint, arXiv:2604.06688.arxiv.org/abs/2604.06688
  21. [29]Xuan Liu, Jie Zhang, Haoyang Shang, Song Guo, Chengxu Yang, and Quanyan Zhu. 2025b. Exploring prosocial irrationality for LLM agents: A social cognition view. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2405.14744
  22. [31]Tianmi Ma, Wenxin Huang, Jiawei Du, Lin Li, Xian Zhong, and Joey Tianyi Zhou. 2026. Evolving quantitative reasoning through self-play in digital twin markets. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 84428–84445. PMLR.
  23. [32]Peer Nagy, Sascha Yves Frey, Kang Li, Bidipta Sarkar, Svitlana Vyetrenko, Stefan Zohren, Ani Calinescu, and Jakob Nicolaus Foerster. 2025. LOB-bench: Benchmarking generative AI for finance - an application to limit order book data. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 45437–45460. PMLR.arxiv.org/abs/2502.09172

Paper details

vlnc:W_fr3jyfvp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents