betaFeedback

WorksComputer Science

FinTradeBench: A Financial Reasoning Benchmark for LLMs

Abstract

Financial analysis requires reasoning over heterogeneous signals, including company fundamentals from regulatory filings and trading signals derived from market prices. However, existing financial benchmarks focus almost exclusively on regulatory filings, neglecting market dynamics and cross-signal interactions. We introduce FinTradeBench, a benchmark of 1,400 questions covering NASDAQ-100 stocks over a ten-year window. To holistically evaluate financial reasoning, FinTradeBench includes questions in three reasoning categories: fundamentals-focused, trading-signal-focused, and hybrid questions that require cross-signal reasoning. Our benchmark starts with 150 expert-authored seed questions, which are then scaled to 1400. To ensure reliable evaluation, we create a Core400 subset in which every item is expert-reviewed. In zero-shot (No-RAG) response generation, we observe that all models perform substantially worse on trading-focused questions compared to fundamental-focused questions. To identify failure causes, we evaluate LLMs under controlled settings including retrieval-based baselines, tool-assisted computation, oracle conditions, and time-series-specialized pipelines. LLMs' capabilities to do computation and evidence selection contribute most to reasoning failures. Additionally, retrieval significantly improves trading-signal reasoning but does not improve fundamental reasoning, where identifying relevant signals remains the dominant failure mode. Finally, rather than relying on a small, fine-tuned model for time-series reasoning, it is better to summarize the time-series data and leverage LLMs' reasoning capabilities.

Introduction

Real-world financial analysis requires reasoning from two complementary types of indicators (sources of information): company fundamentals and trading signals. Company fundamentals are accounting-based metrics derived from company

A trading signal-focused query showing a chart and model responses

Figure 1. A trading signal-focused query: NVDA did not pull back in July 2025, yet 3 of 4 frontier models accept the false premise and recommend buying.

balance sheets or Securities and Exchange Commission (SEC) filings, such as profitability and valuation ratios. They capture a company’s underlying financial health [29, 40]. In contrast, trading signals, computed from historical price and volume data, capture market dynamics and investor sentiment, including momentum, volatility, and trend reversals [10, 3, 18] (Jegadeesh and Titman 1993; Lo et al. 2000; Park and Irwin 2007; Zhang et al. 2024a). Effective financial analysis requires integrating these two perspectives rather than relying on either source in isolation. Additionally, synthesizing heterogeneous information sources, reasoning over numerical indicators, and interpreting market behavior under uncertainty make financial analysis an inherently challenging task, even for expert human analysts.

With recent advancements in large language models (LLMs), automated tools are increasingly applied to assist financial analysts tasks such as summarizing earnings calls, interpreting regulatory filings, and answering questions about company performance (Lee et al., 2025; Yang et al., 2023a,b; Djagba and Saley, 2025; Wu et al., 2023; Shah et al., 2022; Araci, 2019). Existing benchmarks such as FinQA [15], ConvFinQA [16], and TAT-QA (Zhu et al., 2021) focus on numerical reasoning over financial reports and tables. Recent benchmarks evaluate long-context reasoning and retrieval-based question answering [17] (Li et al. 2024; Reddy et al. 2024; Islam et al. 2023). For a comprehensive overview of FinLLMs and their benchmarking, see (Lee et al., 2025).

Nevertheless, these datasets rarely assess reasoning over trading signals derived from historical price dynamics and typically do not require models to integrate both sources of information. Consequently, it remains unclear whether current LLMs can answer financial questions that require joint reasoning of company fundamentals and market behavior. For example, answering the trading question “Is NVIDIA’s pullback in July 2025 a buying opportunity?” (see Figure 1) requires reasoning over both company fundamentals and market dynamics. An analyst must consider metrics such as return on assets and cash-flow strength while interpreting trading signals reflected in price momentum and trading volume to determine if the company’s stock is undervalued or trading at a premium. As Figure 1 shows, most LLMs fail to reason over the relevant trading signals and give incorrect answers. A related class of ambiguities arises when fundamentals and market behavior conflict. E.g., in April 2025, despite weak first-quarter earnings (Earnings per share (EPS) $0.27 vs. $0.42 expected and revenue $19.34B vs. $21B expected), and a generally cautious analyst consensus, Tesla’s stock still rallied by nearly 20% within days, driven by a forward-looking market narrative rather than contemporaneous fundamentals (Investing.com, 2025; McDade, 2025). This shows market narratives can move stock prices independently of fundamentals [23, 7, 11] (Shiller 2017).1 Understanding such nuanced situations is challenging as it requires retrieving relevant context, performing accurate computations, filtering redundant signals, and synthesizing heterogeneous data streams.

In this paper, we address these challenges by making the following contributions: (i) Fin-

Image not extracted

Figure 1 (PDF p. 2).

TradeBench: A Financial Reasoning Benchmark (§3). FinTradeBench evaluates financial reasoning over company fundamentals (from SEC filings) and trading signals (from historical price data). We curate a compact set of signals commonly used in financial analysis, such as valuation ratios and momentum indicators, and integrate them to support structured reasoning across heterogeneous data sources; see §D.3 Table 8 for the full set of signals. Questions are organized into fundamentals-focused, trading-signal-focused, and hybrid reasoning categories for granular model evaluation. The broader benchmark contains 1,400 questions, while the primary evaluation uses a fully audited Core400. (ii) A diagnostic evaluation framework (§4). We combine common-evidence RAG evaluation with controlled experiments (see § 4 and Table 15), allowing errors to be localized to evidence access, computation, evidence selection, or downstream synthesis. We also include time-series-specialized pipelines for Trading and Hybrid reasoning. (iii) Evidence-controlled and human-validated evaluation (§5). We use a human-aligned and validated LLM as a judge to evaluate models on the Core400 subset, which has itself undergone a complete item-level quality audit by a financial expert. (iv) Empirical evidence about financial reasoning bottlenecks (§5). Our empirical findings indicate that while retrieval improves factual grounding, evidence selection, numerical calculation, and time-series data integration remain persistent failure modes.

ConditionFundamental (FF)Trading (TT)Hybrid (FTFT)
No-RAGQuestion onlyQuestion onlyQuestion only
Raw-RAGPoint-in-time SEC evidencePoint-in-time raw OHLCVPoint-in-time SEC + raw OHLCV
Indicator-RAGSEC + precomputed fundamental indicatorsPrecomputed trading indicatorsSEC + fundamental + trading indicators
Tool-UseSEC evidenceRaw OHLCV + deterministic tool routingSEC + raw OHLCV + deterministic tool routing
OracleGold-selected indicator evidenceGold-selected indicator evidenceGold-selected indicator evidence

Table 15. Evidence conditions. Indicator-RAG does not use golden-indicator annotations for selection. Tool routing depends only on the question and declared tool inventory. Oracle supplies gold-selected evidence, not the benchmark reference response.

Background and Related Work

Financial QA benchmarks have advanced numerical reasoning over financial reports, tables, and textual disclosures [15] (Zhu et al. 2021), extending these tasks to conversational settings [16] and long-context reasoning and document retrieval [17] (Islam et al. 2023; Reddy et al. 2024). Broader evaluations cover financial NLP tasks (Xie et al., 2024a) and complex academic finance problems (Zhang et al., 2025), while agentic benchmarks examine financial decision-making through action-oriented simulations (Li et al., 2025a). Despite these advances, existing benchmarks focus mainly on textual financial disclosures and accounting-derived indicators. They rarely assess reasoning over trading signals derived from historical price movements or require models to integrate these signals with information from financial disclosures (Lee

DatasetPrimary evidenceReasoning targetRAG/Open-bookMarket TSF+TEvaluation style
FinQA (Chen et al., [15])Reports, tablesNumerical financial QA×××Program / exact match
TAT-QA (Zhu et al., 2021)Tables + textTabular-text numerical QA×××Exact match / derivation
ConvFinQA (Chen et al., [16])Reports, tablesConversational numerical QA×××Program / exact match
FinanceBench (Islam et al., 2023)SEC filings, earnings callsOpen-book financial QA✓××Human / reference answer
FinDER (Choi et al., [17])Financial documentsRetrieval-based financial QA✓××Retrieval + answer scoring
FinTextQA (Chen et al., [14])Long-form financial textLong-form financial QA✓××Human + GPT-4 scoring
DocFinQA (Reddy et al., 2024)Long financial documentsLong-context reasoning◦××Reference-based scoring
AlphaFin (Li et al., 2024)Retrieved stock contextFinancial analysis QA✓××LLM-oriented evaluation
FinBen (Xie et al., 2024a)Aggregated financial datasetsBroad financial NLP◦××Task-specific metrics
XFinBench (Zhang et al., 2025)Academic finance problemsComplex finance reasoning×××Program-of-thought / answer scoring
InvestorBench (Li et al., 2025a)News + market contextAgentic financial decisions◦◦◦Agent-action evaluation
FinTradeBenchSEC filings + historical market dataCross-signal financial reasoning✓✓✓Evidence-aware judge + human validation

caption: Comparison of financial QA and LLM benchmarks. emphasizing the core aspects of the work such as retrieval/open-book evaluation, explicit reasoning over historical market time series (TS), and joint reasoning over company fundamentals and trading evidence (F+T). ✓ denotes explicit support, × denotes no explicit support, and ◦ denotes partial or indirect support.

:::

et al., 2025). FinTradeBench addresses this gap by evaluating financial reasoning over both company fundamentals and market-derived indicators (trading signals) within a unified, calibrated framework. We discuss additional financial benchmarks with different objectives in §C; also, see Table 1.

FinTradeBench: Benchmark Design

We design FinTradeBench to evaluate financial reasoning over company fundamentals and trading signals (market behavior). We use a calibration-then-scaling framework that combines expert-defined financial evidence, diverse LLM candidate generation, numerical verification, and staged human quality control ([20] (Srivastava et al. 2023; Liang et al. 2022; Thrush et al. 2022)). Figure 2 summarizes the construction pipeline.

FinTradeBench construction pipeline

Figure 2. FinTradeBench construction pipeline. The benchmark is built through data and signal design, a three-way financial reasoning taxonomy, expert-guided seed construction and calibration, and finally scalable question generation and quality control.

(1) Scope and Data Sources. FinTradeBench covers companies represented in the NASDAQ-100 universe over 2015–2025, with questions anchored to historical reporting and market periods. Our principal data sources are: (i) Regulatory filings. We use SEC 10-K and 10-Q filings to derive accounting-based information including profitability, leverage, and valuation. (ii) Historical trading data. We use daily OHLCV (Open, High, Low, Close, Volume) data to derive market signals including momentum, realized volatility, and volume-based measures. We align all evidence by company/ticker and historical period. Each question additionally carries an explicit or recoverable point-in-time reference so that the evidence used to answer it does not rely on information from after the question’s historical cutoff. This temporal alignment is important in finance, where even numerically correct evidence can constitute leakage if it post-dates the decision period. Following asset-pricing and trading literature ([29, 40] (Jegadeesh and Titman 1993)), our signal selection follows three principles: interpretability, empirical relevance, and liquidity.

We group signals into: (i) Company fundamentals: accounting-derived measures such as return on assets (ROA); and (ii) Trading signals: price- and volume-derived measures such as volatility. The full signal inventory is provided in Table 8, and dataset-level statistics are reported in Table 7.

StatisticValue
Broader FinTradeBench collection1,400
Expert-authored seed questions150
Programmatically scaled questions1,250
Core candidate pool450
After canonical-data support screen403
Final human-validated Core400
Core400 Fundamental143
Core400 Trading122
Core400 Hybrid135
Core400 seed-origin147
Core400 company-scaled origin85
Core400 year-scaled origin168
Historical window2015–2025
Fundamental indicators10
Trading indicators9

Table 7. FinTradeBench dataset and final evaluation-set statistics. The 1,400-question collection provides benchmark breadth, whereas the human-validated Core400 is the frozen set used for the final controlled evaluation.

SignalFormulaDefinition / Description
Trading Signals *(Notation: PtP_t = Price at time tt, VtV_t = Volume, RtR_t = Return, NN, kk = Lookback periods, α\alpha = Smoothing factor)
MA (Moving Average)1N∑i=1NPt−i\frac{1}{N}\sum_{i=1}^{N} P_{t-i}Average price of stock over fixed lookback window. Smooths out short-term fluctuations to reveal longer-term trends.
EMA (Exp. Moving Average)αPt+(1−α)EMAt−1\alpha P_t + (1-\alpha)\mathrm{EMA}_{t-1}Weighted moving average emphasizing recent prices; reacts faster to new information (α\alpha is the smoothing factor).
MACDEMAshort−EMAlong\mathrm{EMA}_{\mathrm{short}}-\mathrm{EMA}_{\mathrm{long}}Momentum indicator based on difference between short- & long-term EMAs; captures trend strength & reversal signals.
RSI (Relative Strength Index)100−1001+RS100-\frac{100}{1+\mathrm{RS}} (RS=Avg GainAvg Loss\mathrm{RS}=\frac{\mathrm{Avg\ Gain}}{\mathrm{Avg\ Loss}})Scaled measure of recent gains vs. losses; high RSI suggests overbought conditions, low RSI indicates oversold conditions.
OBV (On-Balance Volume)OBVt−1+Vt⋅sgn⁡(Pt−Pt−1)\mathrm{OBV}_{t-1}+V_t\cdot\operatorname{sgn}(P_t-P_{t-1})Cumulative volume measure linking price movement and trading volume; rising OBV indicates accumulation, falling OBV suggests distribution.
One-Day ReversalPt−Pt−1Pt−1\frac{P_t-P_{t-1}}{P_{t-1}}Daily return from previous close to current close; captures immediate short-term reversals or shocks.
Max Return (20-day)max⁡1≤i≤20(Pt−i−Pt−i−1Pt−i−1)\max_{1\le i\le20}\left(\frac{P_{t-i}-P_{t-i-1}}{P_{t-i-1}}\right)Maximum single-day return observed over the past 20 trading days; indicates short-term volatility extremes.
Medium-Term Momentum∏i=1k(1+Rt−i)−1\prod_{i=1}^{k}(1+R_{t-i})-1Price persistence over weeks or months; positive values indicate sustained trends.
Long-Term Mean Reversal−(Pt−Pˉlong)-(P_t-\bar{P}_{long})Tendency of price to revert toward historical average, representing equilibrium-seeking behavior in markets.
Company Fundamentals
Cash Flow / AssetsOperating Cash FlowTotal Assets\frac{\text{Operating Cash Flow}}{\text{Total Assets}}Operating cash flow divided by total assets; measures how efficiently assets generate cash.
Book / Price (Quarterly)Book Value of EquityMarket Capitalization\frac{\text{Book Value of Equity}}{\text{Market Capitalization}}Ratio of book value to market price; high values may indicate undervaluation.
Earnings / Price (Quarterly)Earnings Per Share (EPS)Price Per Share\frac{\text{Earnings Per Share (EPS)}}{\text{Price Per Share}}Inverse of price-to-earnings ratio; higher values imply cheaper valuations relative to earnings.
Forecast Earnings / PriceExpected Future EPSPrice Per Share\frac{\text{Expected Future EPS}}{\text{Price Per Share}}Forward-looking E/P ratio using analyst forecasts; reflects expected profitability.
Sales / Assets (Quarterly)Total SalesTotal Assets\frac{\text{Total Sales}}{\text{Total Assets}}Asset turnover ratio; measures how effectively a company uses assets to generate revenue.
Debt / Assets (Quarterly)Total DebtTotal Assets\frac{\text{Total Debt}}{\text{Total Assets}}Leverage ratio showing the proportion of assets financed by debt.
Debt / Equity (Quarterly)Total DebtShareholders’ Equity\frac{\text{Total Debt}}{\text{Shareholders' Equity}}Ratio of total debt to shareholders’ equity; higher values indicate greater financial leverage.
Dividend Yield (Quarterly)Dividends Per SharePrice Per Share\frac{\text{Dividends Per Share}}{\text{Price Per Share}}Dividend per share divided by stock price; represent cash return to shareholders.
Return on Assets (Quarterly)Net IncomeTotal Assets\frac{\text{Net Income}}{\text{Total Assets}}Net income divided by total assets; gauges profitability relative to firm size.
Return on Equity (Quarterly)Net IncomeShareholders’ Equity\frac{\text{Net Income}}{\text{Shareholders' Equity}}Net income divided by shareholders’ equity; measures profitability relative to owners’ capital.

Table 8. Summary of trading signals and company fundamentals used in the question design.

(2) Question Taxonomy and Golden Indicators. We divide questions into three reasoning categories. (i) Fundamental (F): questions requiring reasoning primarily over accounting and valuation signals. (ii) Trading (T): questions requiring reasoning primarily over historical price, volatility, momentum, or other market signals, and (iii) Hybrid (FT): questions requiring joint reasoning over both company fundamentals and trading signals. This taxonomy supports aggregate evaluation and diagnosis of whether LLMs can integrate heterogeneous financial signals. Sample questions and reference responses are shown in Table 11. Each question is also associated with a set of golden indicators (G.I.). They are the set of expert-specified financial signals sufficient to answer the given question. Golden indicators are not treated as complete mandatory checklists. Instead, they identify the key evidence required for a grounded response. They are used during benchmark construction for evidence auditing and, in later experiments, as a secondary indicator-overlap diagnostic and to construct the gold-indicator Oracle condition (see §4).

(3) Calibration-Then-Scaling Framework. To ensure scalability and reliability, our calibration-then-scaling framework has three stages: (1) expert-guided seed construction, (2) candidate-response verification and calibration, and (3) automated scaling and human validation.

(3a) Phase 1: Seed Construction, Candidate Generation, and Self-Selection. We begin with 50 questions each for FF, TT, and FTFT reasoning questions, totaling 150 seed questions. For each question, candidate responses are generated using multiple LLMs and diverse prompt templates derived from the TELeR taxonomy (Karmaker and Feng, 2023). Each model produces multiple candidate responses under distinct reasoning-oriented prompt templates. This encourages within-model diversity while preserving a common financial task specification, following the intuition of best-of-NN generation and self-reflective refinement ([19] (Shinn et al. 2023)). Each model then independently selects its strongest candidate based on factual accuracy, reasoning completeness, and relevance. By restricting self-selection strictly to within-model candidates, we prevent cross-model preference leakage (Li et al., 2025b; Lee et al., 2024; Yuan et al., 2024; Wu et al., 2024). These candidates are checked against the structured financial data available for the corresponding company and historical period. Numerical claims are categorized as SUPPORTED, CONTRADICTED, or NOT FOUND. We also measure the G.I. coverage within the responses. For detailed prompt templates, audit rules, and construction-time metrics see Appendix § 12.

(3b) Phase 2: Expert Review and Development-Time Judge Calibration. Once each model submits its candidate answer, a financial-domain expert inspects each candidate response using four dimensions: (i) factual and numerical accuracy, (ii) completeness, (iii) relevance, and (iv) clarity, and then selects (and edits if needed) the best candidate response to get a viable ground truth answer for the benchmark. The model identities are hidden during review. An external LLM evaluator is then calibrated during benchmark development against these human judgments using the same four rubric dimensions. This development-time calibration is used to support scalable screening of generated benchmark material. After the automated judge configuration and scoring procedure are frozen, we validate the judge on a held-out blinded human-review set. This validation is used to assess evaluator reliability for robustness check of the scaling pipeline. It is not used to modify either the benchmark or the judge; see § 4.

(3c) Phase 3: Scaling the Benchmark. Using the calibrated construction pipeline, we scale question generation across companies and historical periods, producing a broader benchmark of 1,400 historically grounded financial-reasoning questions. Scaled questions preserve the same FF, TT, and FTFT taxonomy and are screened by a human for any catastrophic failures; see §D.9. For the controlled experiments in this paper, we construct a substantially stricter evaluation subset Core400 with stronger evidence-availability and full item-level human expert-verification requirements; see §D.6.

Experimental Setup

Evaluation Overview. Our experiments are organized into four studies. (i) General QA: No-RAG versus Raw-RAG evaluates whether retrieval-augmented generation improves financial reasoning when models receive point-in-time SEC and/or historical market evidence ([31] (Niu et al. 2024)). We evaluate models using point-in-time RAG rather than supervised fine-tuning to reflect realistic workflows where models must reason over continuously updating market and regulatory evidence without parameter retraining (see Appendix § 13.§ 2). (ii) Controlled evidence conditions (see Table 15) compare Indicator-RAG, Tool-Use, and gold-indicator Oracle evidence to diagnose whether errors arise from evidence representation, numerical computation, or evidence selection. (iii) Time-series specialist evaluation compares native time-series reasoning with a forecast-feature pipeline based on TimesFM 2.5. (iv) Judge validation evaluates whether the frozen automated scoring procedure agrees with blinded human judgments. Figure 3 shows the complete evaluation framework. To see models evaluated, their serving backends and inference configurations, see Table 16. Compute configuration and recorded runtime are reported in Table 17.

Figure 3. FinTradeBench evaluation framework. Models answer the final Core400 under general-QA, controlled-evidence, and time-series-specialist conditions. Generated responses and their evidence traces are evaluated using a common four-dimensional, evidence-aware judging framework. A financial expert then validates the automated judging.

System / runExecution modeInference hardwareClient allocationRecorded wall timecompleting-job
Qwen3.8-27B ((Qwen Team, 2026))Local vLLM ((Kwon et al., 2023))1× NVIDIA H100 80GB HBM316 CPU cores, 256 GB RAM21:58:41†
Ministral-3-8B ((Liu et al., 2026))Local vLLM ((Kwon et al., 2023))1× NVIDIA H100 80GB HBM316 CPU cores, 256 GB RAM02:27:38
Llama-4-Scout ((Meta AI, 2025))Azure AI Foundry APIProvider-managed; accelerator not exposed16 CPU cores, 64 GB RAM07:11:05†
GPT-5.4-mini ((OpenAI, 2026))Azure AI Foundry API, General QAProvider-managed; accelerator not exposed16 CPU cores, 64 GB RAM06:03:29†
DeepSeek-V4-Flash ([25])Azure AI Foundry APIProvider-managed; accelerator not exposed16 CPU cores, 64 GB RAM02:19:38†
Gemini-3.8-Flash ([34])Vertex AI APIProvider-managed; accelerator not exposed16 CPU cores, 64 GB RAM08:45:51†
Qwen2.5-32B ((Qwen et al., 2024)), R1-Distill-Qwen-32B ([24]), Llama-3.1-8B ([36]), Phi-4 ([1])Local controlled jobs ((Wolf et al., 2020))1× NVIDIA H100 80GB HBM3 per job8 CPU cores, 256 GB RAM per jobSee note‡
GPT-5.4-mini ((OpenAI, 2026))Azure AI Foundry API, controlled conditionsProvider-managed; accelerator not exposed16 CPU cores, 64 GB RAM01:55:03
ChatTS-8B ((Xie et al., 2024b))Local inference ((Wolf et al., 2020))1× NVIDIA H100 80GB HBM316 CPU cores, 256 GB RAM00:59:37
TimesFM 2.5 ([22, 35]) feature generationBigQuery AI.FORECASTProvider-managed; accelerator not exposed8 CPU cores, 64 GB RAM00:06:01
GPT-5.4-mini ((OpenAI, 2026)) + TimesFM 2.5 ([22, 35])Azure API synthesisProvider-managed; accelerator not exposed8 CPU cores, 64 GB RAM00:39:41
Claude Sonnet 4.5 ([5]): controlled-model judgingVertex AI APIProvider-managed; accelerator not exposed4 CPU cores, 24 GB RAM11:02:54 / 23:47:21§
Claude Sonnet 4.5 ([5]): time-series judgingVertex AI APIProvider-managed; accelerator not exposed4 CPU cores, 24 GB RAM02:33:58
Claude Sonnet 4.5 ([5]): common-evidence RAG judgingVertex AI API; parallel per-condition jobsProvider-managed; accelerator not exposed2 CPU cores, 16 GB RAM per job≤02:23:57 per job¶

Table 17. Compute configuration and recorded runtime. Times are Slurm wall-clock durations for the completing jobs associated with the frozen outputs; smoke tests and failed attempts are excluded. †These runners support resumption and may reuse rows generated by earlier jobs, so the reported time is the wall time of the completing job rather than cumulative compute expenditure. ‡The four locally controlled baseline models were dispatched as 12 independent model×condition jobs with generic Slurm job names (ftb403_resume). Their exact per-model wall times cannot be reconstructed reliably from the accounting export, so we do not assign potentially incorrect runtimes. §The two values correspond to the controlled-model judge batches for the two model groups. ¶Common-evidence RAG/No-RAG responses were judged in parallel model–condition jobs; the longest observed completing job was 02:23:57. For API-hosted models, the listed CPU/RAM values describe the Slurm client, not the provider-side inference hardware.

General QA: No-RAG and Raw-RAG. The General-QA experiment uses FinTradeBench’s TELeR-inspired multi-candidate generation and intra-model self-selection protocol. In the No-RAG condition, the model receives only the benchmark question. In the Raw-RAG condition, the model receives point-in-time external evidence appropriate to the reasoning lane: Fundamental questions receive SEC evidence, Trading questions receive historical raw OHLCV evidence, and Hybrid questions receive both. All retrieved evidence is restricted to information available by the question’s historical cutoff. Where an explicit question-level cutoff is available, that cutoff is used directly; period metadata are used when appropriate, and the benchmark-wide cutoff is used only as a fallback. Generation uses four No-RAG candidate prompts and two RAG candidate prompts, followed in both cases by intra-model self-selection. For detailed architecture, see Figure 8 and §E.3.

Overview of the RAG

Figure 8. Overview of the RAG. The pipeline features a dual-track retrieval engine that processes unstructured text and structured time-series separately. The generation phase utilizes TELeR taxonomy to produce multiple candidate responses across the model zoo, which are filtered via self-selection before final evaluation.

Common-Evidence Evaluation for RAG versus No-RAG. A central difficulty in evaluating RAG is that RAG outputs naturally carry richer source evidence than closed-book outputs. If the evaluator is given each system’s own generation-time evidence, a RAG response can receive a stronger factual anchor during scoring merely because it was generated with retrieved context. To remove this evaluator-side evidence asymmetry, we re-evaluate each paired No-RAG/RAG response against the same condition-independent canonical point-in-time evidence package. This evidence is constructed deterministically from the benchmark’s point-in-time fundamental and market context and is independent of the answer being scored.

Controlled Evidence Conditions. We next hold the question set, generator model, answer format, and judge rubric fixed while varying the evidence pathway. We introduce three controlled conditions to distinguish evidence selection from financial reasoning. (i) Indicator-RAG supplies retrieved SEC passages where applicable and the full set of pre-computed indicators (signals), leaving the model to identify the relevant signals. (ii) Tool-Use (Tool RAG) instead provides raw market data and allows a model-based router to request deterministic indicator values from the same underlying data, testing whether selective tool access improves evidence use. Neither condition receives the benchmark’s gold indicator annotations or reference response. (iii) Oracle condition supplies only the indicator evidence selected by those annotations, without revealing the reference answer. The model must still perform the financial reasoning and synthesis itself. This provides a diagnostic control for reasoning when relevant indicators are already identified. Together, these conditions help assess whether errors arise from selecting relevant evidence or interpreting that evidence to form an answer. Table 15 details the evidence provided for each reasoning question type for each of the controlled conditions.

Time-Series Specialist Evaluation. Since TT and FTFT questions involve time-series data, we evaluate models designed specifically for such tasks. This study is restricted to questions that require time-series reasoning: 122 TT and 135 FTFT questions. (i) ChatTS-8B. ChatTS (Xie et al., 2024b) receives the historical adjusted-close series directly and performs native time-series reasoning. For Hybrid questions, the time series is additionally accompanied by point-in-time SEC context. (ii) TimesFM 2.5 forecast-feature pipeline. For each question, TimesFM 2.5 [21] receives the most recent 128 adjusted-close observations available at the historical cutoff and produces a five-step forecast. Forecast-derived features are then provided to GPT-5.4-mini for natural-language financial synthesis. Hybrid questions additionally receive point-in-time SEC context. This helps us compare complete time-series-specialized pipelines and evaluate their overall performance.

Evidence-Aware LLM Judge. Generated responses are evaluated using human-aligned judge model (Claude Sonnet 4.5). The judge model does not belong to any evaluated model family to prevent bias [13] (Ye et al. 2024; Zheng et al. 2023). The judge produces structured deterministic output and scores from 1–5 on four dimensions: (i) Factual and Numerical Accuracy, (ii) Completeness and Context, (iii) Relevance and Utility, (iv) Clarity and Rationale. Our primary response-quality metric is the arithmetic mean of these four raw Likert scores. The judge never receives the benchmark reference response or answer_conclusion_summary as factual ground truth. Scoring is anchored to the supplied point-in-time evidence. Golden-indicator precision, recall, and F1 are retained only as ancillary diagnostics and are not the primary response-quality measure. The controlled and time-series studies use the evidence-aware judge protocol, while the No-RAG/RAG study uses the common-evidence protocol described in § 4. Full judge prompts, schema, validation rules, evidence-budget diagnostics, and statistical analysis are reported in E.7 and E.8.

Held-Out Human Validation of the Judge. After all automated evaluation, we freeze the judge configuration and conduct a separate blinded human validation. We use 24 paired questions (48 anonymous responses), balanced across FF, TT, and FTFT reasoning and stratified across four evaluation families: RAG versus No-RAG, Oracle versus Indicator-RAG, Tool-Use versus Indicator-RAG, and TimesFM+GPT versus ChatTS. The human reviewer independently scores each response using the same four 1–5 dimensions as the automated judge and additionally records an A/B/Tie pairwise preference and confidence score. Model identities, condition labels, and automated judge scores are hidden during review. The human labels are used to assess agreement with the frozen judge and agreement results are reported in Figure 5; see §F.1 for details about the expert-validated automated judge.

Figure 5. Figure 5: Held-out human–judge agreement on 48 responses (192 rubric ratings), stratified across 3 reasoning lanes and 4 evaluation families. Bars give the share of ratings where the judge matches the human score exactly, differs by 1, or differs by 2 points (no larger differences occur); ρ\rho is Spearman’s correlation and MAE the mean absolute error.

Research Questions. We address three research questions: (1) Retrieval: Does Raw-RAG improve financial reasoning when No-RAG and RAG outputs are evaluated against the same canonical evidence? (2) Reasoning Bottlenecks: After relevant financial information is available, how much error remains attributable to computation and evidence selection? (3) Time-Series Reasoning: Do time-series-specialized pipelines improve Trading and Hybrid financial reasoning?

Results and Discussion

We organize the results based on the research questions mentioned in § 4. Unless otherwise stated, reported scores are raw 1–5 mean judge Likert scores, and condition effects are paired question-level differences. Statistical conclusions use 5,000-sample paired bootstrap confidence intervals together with Holm-corrected Wilcoxon signed-rank tests.

RQ1: Does Retrieval Improve Financial Reasoning? Under common-evidence judging, RAG improves overall response quality for all six General-QA models (Table 2 and Figure 7). The overall paired gains are statistically reliable after Holm correction. RAG improves TT performance for every model, with paired effects between approximately +0.23 and +0.46, and improves FTFT performance by approximately +0.25 to +0.48. All TT and FTFT effects remain statistically significant after Holm correction. In contrast, FF effects are comparatively small and model-dependent; see Table 2. After multiple-comparison correction, only the Ministral Fundamental effect is statistically reliable.

ModelFundamental(F) NRFundamental(F) RFundamental(F) Δ\DeltaTrading(T) NRTrading(T) RTrading(T) Δ\DeltaHybrid(FT) NRHybrid(FT) RHybrid(FT) Δ\DeltaOverall NROverall ROverall Δ\Delta
DeepSeek-V4-Flash3.103.04−0.06 ↓2.042.50+0.46 ↑3.003.33+0.32 ↑2.752.97+0.23 ↑
Llama-4-Scout2.652.72+0.06 ↑1.932.34+0.41 ↑2.682.93+0.25 ↑2.442.67+0.23 ↑
Qwen3.8-27B2.882.85−0.03 ↓1.912.13+0.23 ↑2.813.29+0.48 ↑2.562.78+0.22 ↑
Gemini-3.8-Flash3.193.08−0.10 ↓2.112.44+0.33 ↑3.093.51+0.42 ↑2.823.03+0.20 ↑
GPT-5.4-mini3.203.06−0.14 ↓2.062.41+0.35 ↑3.083.41+0.33 ↑2.812.98+0.17 ↑
Ministral-3-8B2.452.58+0.14 ↑1.922.24+0.32 ↑2.382.69+0.31 ↑2.262.52+0.25 ↑

Table 2. Common-evidence RAG evaluation on Core400. For each reasoning category, Δ\Delta denotes the paired question-level RAG (R) minus No-RAG(NR) mean difference. Coloured arrows mark changes that are significant under a two-sided Wilcoxon signed-rank test, Holm-corrected (pHolm<0.05p_{\mathrm{Holm}} < 0.05); gray arrows are not significant. The test depends on how consistently questions move in one direction, not on the size of the mean change alone.

Figure 7. Lane-level paired effects: (a) RAG−No-RAG on the common-evidence Core400; (b) Tool−Indicator and (c) Oracle−Indicator for the seven controlled models. Points are mean paired changes in 1–5 judge Likert score with 95% bootstrap CIs; the vertical line marks no change. †Hollow marker: only 12 paired Trading items under Tool.

Retrieval benefit depends on the reasoning modality. With an identical factual anchor for paired outputs, retrieval is most consistently beneficial on TT and FTFT questions, where answers must connect historical market evidence to a financial conclusion. The common evidence package ensures that these gains are not explained by giving the RAG response a richer factual basis at scoring time. FF questions show a different behavior. Providing additional SEC evidence does not reliably improve the final answer for already-capable models. This suggests that for fundamental queries, performance is constrained by evidence selection and cross-statement synthesis rather than raw evidence availability. We test this explanation directly with the controlled evidence conditions next.

RQ2: What Limits Performance After Evidence Is Available? Gold-selected evidence consistently improves reasoning. Figure 10 and Table 3 compares Indicator-RAG, Tool-Use, and gold-indicator Oracle evidence across seven models spanning multiple model families and capability levels. The most consistent result is the gap between Oracle and Indicator-RAG. Oracle evidence improves the overall mean score for all models. Overall Oracle–Indicator improvements remain statistically significant after Holm correction; see Figure 7. The pattern is particularly strong on TT questions. Every model receives a positive Oracle–Indicator effect in this lane, ranging from approximately +0.33+0.33 for GPT-5.4-mini to more than +0.9+0.9 for Phi-4, with statistically reliable improvements across all models. This bottleneck shows that performance does not plateau once structured financial evidence is given as context. Models improve further when the relevant evidence is selected for them. This performance delta indicates that models frequently access sufficient context but fail to isolate the decisive signals. Here, Oracle is not a perfect-reasoning condition. It supplies gold-selected indicators but

ModelInd.ToolOrcl.ΔTI\Delta_{\mathrm{TI}}ΔOI\Delta_{\mathrm{OI}}
Qwen2.5-32B3.723.553.98−0.17-0.17 ↓+0.26+0.26 ↑
R1-Distill-Qwen-32B3.383.003.69−0.37-0.37 ↓+0.31+0.31 ↑
Llama-3.1-8B2.912.893.47−0.02-0.02 ↓+0.57+0.57 ↑
Phi-43.683.254.06−0.44-0.44 ↓+0.38+0.38 ↑
Llama-4-Scout3.533.354.19−0.18-0.18 ↓+0.65+0.65 ↑
Qwen3.8-27B†4.344.534.52+0.04+0.04 ↑+0.17+0.17 ↑
GPT-5.4-mini4.694.574.88−0.12-0.12 ↓+0.19+0.19 ↑

Table 3. Table 3: Controlled-condition results on Core400. Scores are 1–5 judge Likert means for Indicator (Ind.), Tool and Oracle (Orcl.) conditions; deltas are paired mean differences between Tool–Indicator (ΔTI\Delta_{\mathrm{TI}}) and Oracle–Indicator (ΔOI\Delta_{\mathrm{OI}}) conditions. Colored arrows are significant (Wilcoxon signed-rank, Holm-corrected, pHolm<0.05p_{\mathrm{Holm}} < 0.05), gray arrows are not. †Hollow marker: only 12 paired Trading items under Tool.

Figure 10. Oracle–Indicator effects across seven controlled models. Positive values indicate improved performance when question-specific gold-indicator evidence replaces the broader Indicator-RAG evidence-selection process.

Figure 6. Figure 6: Number of models whose score changes significantly under each intervention, by lane (two-sided Wilcoxon signed-rank test on per-question paired differences, Holm-corrected; pHolm<0.05p_{\mathrm{Holm}} < 0.05). Numbers inside bars are model counts. For details, see Table 22 and Table 23.

not the benchmark reference answer. The residual gap between Oracle performance and the maximum score, therefore, reflects remaining errors in interpretation, synthesis, and answer generation.

Tool access helps computation selectively, not universally. At the aggregate level, Tool-Use is below Indicator-RAG for six of the seven models, while Llama-3.1 is approximately unchanged. Thus, simply giving a system selective access to deterministic tools does not improve overall financial reasoning. The TT lane, however, shows the opposite tendency. All six complete-coverage models have non-negative Trading Tool–Indicator point estimates, and Qwen2.5, Llama-3.1, Phi-4, and Llama-4 show statistically reliable gains after multiple-comparison correction. In contrast, Tool-Use reduces Fundamental performance for every complete-coverage model; see Figure 7. This lane asymmetry clarifies the role of tools. Explicit computation is useful when the task requires deriving quantities from historical market data, but selective tool routing can remove broader context required for fundamental or cross-signal synthesis. Tool access should therefore be viewed as a task-dependent computational interface rather than a universally stronger form of retrieval.

RQ3: Do Time-Series Specialists Improve Financial Reasoning? Forecast-derived features substantially improve the time-series pipeline. This experiment evaluates the 257 Core400 questions requiring time-series reasoning. Figure 4 compares direct ChatTS reasoning with the GPT-5.4-mini + TimesFM 2.5 forecast-feature pipeline. The gain appears in both reasoning categories but is substantially larger for FTFT questions. All comparisons are statistically significant. The result suggests that forecast-derived numerical features can provide a useful intermediate representation between historical prices and language-based financial synthesis. The particularly large Hybrid gain is consistent with the value of combining summarized market dynamics with point-in-time fundamental evidence. Table 4 shows the performance of GPT-5.4-mini across all evaluation conditions (for qualitative responses, see Table 25–Table 27).

Figure 4. Time-series-specialized evaluation on the final 257-question eligible subset (122 Trading (T), 135 (FT)). Scores are raw 1–5 judge Likert means.

ConditionOverallFTFT
No-RAG2.813.202.063.08
RAG2.983.062.413.41
Indicator4.694.774.444.83
Tool4.574.424.614.69
Oracle4.884.904.774.95
GPT-5.4-mini + TimesFM2.54.12†–3.704.51

Table 4. GPT-5.4-mini across all experimental conditions. †TimesFM2.5 is evaluated only on the time-series-eligible questions, so its Overall excludes Fundamental.

GPT-5.4-mini Across Conditions: Fundamental
Q: As of Q3 2025, which generates more cash per dollar of assets: Netflix or Apple?
Golden answer: “As of Q3 2025, Apple generates more cash per dollar of assets than Netflix. According to the fundamental data, Apple’s Cash Flow to Assets ratio is 0.0841, meaning it generates roughly 8.4 cents of cash for every dollar of assets. In contrast, Netflix’s ratio is 0.0456, generating roughly 4.5 cents per dollar of assets.”
PER-CONDITION-EVIDENCE JUDGE
Indicator [5.00]: “From the supplied indicator data at 2025-09-30: Apple (AAPL) F Cash Flow/Assets: 0.084065; Netflix (NFLX) F Cash Flow/Assets: 0.045636. [. . . ] Apple generates more cash per dollar of assets than Netflix as of Q3 2025.”
Tool [3.00]: “The context provided does not include any Netflix Q3 2025 SEC evidence or balance sheet/cash flow figures [. . . ] The supplied evidence is insufficient to confidently determine whether Netflix or Apple generates more cash per dollar of assets as of Q3 2025.” (retrieved filings lacked Netflix data, so the model abstained)
Oracle [5.00]: “As of Q3 2025, Apple generates more cash per dollar of assets. Apple Cash Flow/Assets: 0.084065; Netflix Cash Flow/Assets: 0.045636. [. . . ] Apple’s value is higher, so Apple is the answer based on the supplied evidence.”
COMMON-EVIDENCE JUDGE
No-RAG [4.50]: “As of Q3 2025, Apple generates more cash per dollar of assets than Netflix. [. . . ] Apple has historically produced very large operating and free cash flows relative to its asset base. [. . . ] As a result, cash produced per dollar of assets is typically lower than Apple’s.” (correct conclusion, but no Q3 2025 figures cited)
RAG [2.50]: “The most recent Apple balance-sheet total assets in the context is from the 2023 filing: Total assets = $335,038 million [. . . ]” (divides 2025 cash flow by 2023 assets and gives no final verdict)

Table 25. GPT-5.4-mini responses to a Fundamental seed question under every evaluated condition (TimesFM2.5 is evaluated only on time-series-eligible Trading and Hybrid questions). Bracketed numbers are judge scores (1–5 Likert mean). Scores are comparable only within a judging protocol: the per-condition-evidence judge grades each answer against the evidence that condition supplied, the common-evidence judge grades both RAG and No-RAG against identical canonical evidence. Excerpts are verbatim apart from removed Markdown formatting; [. . . ] marks omitted text. Gray notes summarize the judge’s rationale.

GPT-5.4-mini Across Conditions: Hybrid
Q: Has Meta bottomed out after its decline in August 2025?
Golden answer: “Based on the August 2025 data, META has not clearly bottomed out after its -1.51% decline. The stock closed below key moving averages (MA 20 at 762.04 and EMA 20 at 750.93), with bearish short-term momentum (negative 5D/20D momentum, MACD below signal, neutral RSI at 39.23), indicating ongoing downtrend risks rather than reversal confirmation. [. . . ] Investors should monitor for price stabilization above EMA 20 and RSI improvement above 50 before confirming a bottom.”
PER-CONDITION-EVIDENCE JUDGE
Indicator [5.00]: “The trading indicators for 2025-08-29 still show weakness: Adj. Close: 738.194; MA_20: 762.043, so price is below the 20-day moving average; RSI: 39.23 [. . . ] the evidence does not support a confident ‘yes.’ ”
Tool [5.00]: “Using only the supplied evidence, I cannot conclude confidently that Meta had ‘bottomed out.’ [. . . ] Meta fell from an August 2025 high of $790.00 on 2025-08-12 to $738.70 on 2025-08-29. [. . . ] There was a rebound from $739.10 on 2025-08-21 to $754.79 on 2025-08-22, but that move did not hold [. . . ]”
Oracle [5.00]: “RSI = 39.23: This is below 50, indicating weak momentum, but it is not extremely oversold [. . . ] Momentum_5D = -0.0213 and Momentum_20D = -0.0151: Both short-term momentum measures are still negative [. . . ] there is not enough evidence to conclude that Meta has bottomed out after its August 2025 decline.”
+ TimesFM2.5 [4.50]: “Probably not enough evidence to say META has clearly bottomed out in August 2025. [. . . ] The TimesFM forecast is only mildly positive from the last actual close of 738.194 to about 742.7 over the next 5 trading observations, which suggests stabilization, not a strong rebound.”
COMMON-EVIDENCE JUDGE
No-RAG [2.50]: “If the stock stopped making lower lows and began consolidating after the August selloff, then the decline likely marked at least a near-term trough. But unless price action has clearly turned upward [. . . ] it is more accurate to describe the move as a potential bottom in progress [. . . ]” (conditional framework; no August data cited)
RAG [4.25]: “The stock fell from 790.00 on Aug. 12 to the 739 area by Aug. 21–29 [. . . ] the most defensible conclusion is possible bottoming, not confirmed bottoming.” (correct, but inside a ∼8k-character answer)

Table 27. GPT-5.4-mini responses to a Hybrid seed question under every evaluated condition. Notation as in Table 25.

Conclusion

We introduce FinTradeBench for evaluating financial reasoning across company fundamentals and trading signals. We use a decade of NASDAQ-100 financial data to curate our benchmark using a calibration-then-scaling construction pipeline and human validation. Evaluation uses human-validated Core400 questions. We find that financial reasoning failures cannot be explained by evidence access alone. Under common-evidence evaluation, RAG improves overall performance with the consistent gains on TT and FTFT questions, while FF effects are smaller and model-dependent. Controlled experiments further separate downstream bottlenecks. Tool-Use is beneficial when explicit numerical computation is required, particularly for TT questions, but does not universally improve FF or FTFT reasoning. In contrast, Oracle evidence improves all controlled models, showing that evidence selection remains a persistent bottleneck even when relevant financial information is available. The remaining gap under Oracle evidence further indicates that evidence selection is not the only limitation; interpretation and financial synthesis also remain imperfect. Our time-series experiments provide complementary evidence that the representation of numerical market history matters. Transforming historical price sequences into forecast-derived features can provide a useful interface between time-series models and language-based financial reasoning. Overall, financial reasoning errors cannot be reduced to a single retrieval problem. Retrieval remains useful, but providing more evidence alone is insufficient. Computation and evidence selection also cause financial reasoning errors. In time series pipelines, powerful reasoning model with summarized time series data outperforms smaller fine-tuned models for time-series reasoning. We validate benchmark quality and evaluator reliability through separate human-review procedures. Overall, we find that to make progress in financial LLMs, we must build systems that retrieve the right evidence, perform the required computation, identify which signals matter, and synthesize them into a grounded financial conclusion. FinTradeBench provides a framework for measuring each of these stages.

Limitations

(i) Market and asset-class coverage. FinTradeBench focuses on the NASDAQ-100 universe over 2015–2025. These firms are predominantly large, liquid, U.S.-listed companies, so the benchmark may not represent small-cap equities, international or emerging markets, or other asset classes such as fixed income, commodities, derivatives, and currencies.

(ii) Broader benchmark versus audited Core. The broader collection contains 1,400 questions, but the full item-level human quality audit is performed on the filtered Core403, producing Core400. The 99.3% acceptance rate therefore characterizes an already-screened evaluation subset and should not be interpreted as a quality estimate for all 1,400 questions. We perform a catastrophic error screening on the overall benchmark.

(iii) Golden indicators are not exhaustive. Golden indicators specify an expert-defined minimal evidence set. Financial questions can admit defensible reasoning paths using additional or alternative evidence, especially for subjective valuation or investment-style questions. For this reason, golden-indicator F1 is treated as an ancillary diagnostic rather than the primary quality score.

(iv) LLM-as-a-Judge limitations. The final evaluator is automated. Held-out human validation shows strong agreement, but the study uses financial expert only for validation of the evaluation and a PhD student to screen catastrophic errors. Relevance exhibits a modest positive judge bias, and automated scores remain an approximation of expert assessment rather than ground truth.

(v) No-RAG/RAG generation-budget asymmetry. The historical General-QA protocol uses four No-RAG candidates and two RAG candidates before self-selection. Common-evidence re-judging removes evaluator-side evidence asymmetry but does not make generation budgets identical. RAG–No-RAG is therefore a benchmark comparison, not a strict causal ablation.

(vi) Time-series comparison is not a pure ablation. ChatTS and the TimesFM 2.5 forecast-feature pipeline differ in their downstream language components. Their comparison evaluates complete systems and does not isolate the causal contribution of the forecasting architecture alone.

(vii) Point-in-time reconstruction. FinTradeBench uses question-specific historical cutoffs and point-in-time evidence whenever available, with a benchmark-wide cutoff as a fallback. Historical financial data can nevertheless contain complications such as filing availability dates, fiscal-period conventions, ticker changes, and market reaction timing. The benchmark should therefore be interpreted as a point-in-time historical reasoning benchmark rather than a live trading simulator.

(viii) Temporal contamination and model knowledge. Because the benchmark covers historical periods, some facts may already exist in a model’s pretraining or post-training data. Closed-book performance cannot be interpreted as proof that the model reconstructed an answer solely from reasoning. The controlled evidence conditions are intended to diagnose evidence-use behavior rather than to establish absence of memorization.

(ix) Hosted-model reproducibility. Several evaluated systems are accessed through hosted APIs. Even when model identifiers are fixed, providers may update infrastructure or serving behavior. We therefore preserve frozen generated outputs, judge outputs, model identifiers, and analysis scripts with the release.

(x) Task scope. FinTradeBench evaluates historical evidence-grounded question answering. It does not measure portfolio optimization, execution quality, risk management, market impact, profitability, or personalized investment suitability.

Ethical Considerations

(i) Financial decision-making risk. FinTradeBench evaluates language model reasoning over real financial data for named public companies. Scores on this benchmark should not be interpreted as endorsements of any model for live trading, investment advisory, or automated financial decision-making. Even the highest-performing models in our evaluation exhibit substantial error rates, and financial decisions based on LLM outputs carry real economic risk to end users. (ii) Benchmark misuse. While FinTradeBench is designed for research evaluation, we acknowledge that fine-tuning models specifically to maximise FinTradeBench scores without genuine improvement in financial reasoning could inflate reported performance. We encourage the community to treat benchmark results as one signal among many and to complement automated evaluation with human expert review before drawing strong conclusions about financial reasoning capability. (iii) Annotator and expert involvement. Human financial experts involved in seed question authoring were part of the research team. Evaluation and calibration were conducted with anonymized model outputs to reduce rater bias. Scaled benchmark responses were additionally screened for fatal flaws as a quality-control step rather than used as independent graded labels. (iv) Societal impact. Improvements in LLM financial reasoning could benefit retail investors and analysts by democratising access to structured financial analysis. However, the same capabilities could be exploited to automate misleading financial narratives or market manipulation at scale. We call for responsible disclosure norms and human-in-the-loop oversight in any deployment of LLM-based financial analysis tools. (v) Misinformation and market manipulation. Because FinTradeBench evaluates models on real public companies, there is a risk that financial reasoning systems trained or evaluated on similar data could be used to generate persuasive but misleading market narratives. This risk is especially important when LLM outputs are presented as authoritative financial analysis. We therefore recommend that any deployment of LLM-based financial analysis tools include source attribution, uncertainty disclosure, numerical verification, and human review before dissemination.

Organization of the Supplementary Material

This supplementary material expands the main paper in four parts: benchmark construction (§B–§D), experimental implementation (§E), evaluation evidence (§F–§I), and illustrations and responsible-use information (§J–§K). Limitations and ethical considerations appear in the main paper.

Motivating examples. Extended historical cases of fundamentals–market divergence that motivate joint reasoning over both signal types (cf. § 1).

Extended related work. Trading signals, LLM-as-a-judge evaluation, retrieval and tool use, and a comparison with existing financial benchmarks.

Benchmark design addendum. Validation gates, dataset statistics, the financial-signal reference, the TELeR prompt taxonomy, sample questions and golden answers, the 450→403→400 curation process with its provenance, item-level audit, and fatal-flaw screening, and benchmark quality control.

Experimental setup addendum. Models and serving details, the RAG architecture, evidence conditions, tool routing, time-series specialist pipelines, the evidence-aware judge, metrics and statistical testing, and compute resources.

Results addendum. Held-out judge validation, robustness to the final Core400 curation, and discussion, as referenced from the main-text results.

Human review and evaluator validation. The four human-review stages, held-out validation of the frozen judge, and pointers to the full annotation instruments (§L).

Evaluation integrity and sensitivity checks. Evidence coverage and QC, and sensitivity of all results to the Core403→Core400 transition.

Complete quantitative results. Full paired effects, bootstrap intervals, and Holm-adjusted tests for the common-evidence, controlled, and time-series studies.

Qualitative case studies. Per-condition response examples, presented as illustrations rather than as primary evidence.

Transparency and responsible use. Data transparency and privacy, safety and broader impact, data maintenance, and disclosure of AI assistance.

Annotation instruments. Reviewer rubrics, label sets, and record fields for the four human-review stages (reference material).

Extended Motivating Examples: Fundamentals–Market Divergence

The Tesla April 2025 case discussed in § 1 is illustrative of a broader and well-documented phenomenon in financial markets: sustained divergences between company fundamentals and market price dynamics driven by investor sentiment, narrative momentum, and forward-looking expectations. We present three additional historical examples that further motivate the need for benchmarks capable of reasoning over both types of signals jointly.

(i) Amazon (1999–2001). During the dot-com era, Amazon sustained extremely high market valuations despite persistent operating losses and negative earnings. Rather than anchoring on contemporaneous accounting metrics, investors priced in long-run platform dominance and e-commerce adoption narratives. This example illustrates how growth narratives can sustain valuations that are entirely disconnected from current fundamental signals ((Shiller, 2000)). A system reasoning solely from financial statements would have consistently flagged Amazon as financially distressed during this period, while the market priced in the opposite trajectory.

(ii) Nvidia (2016–2017). NVIDIA’s stock appreciated substantially in 2016-2017, well before its earnings reports fully reflected the revenue impact of GPU adoption in deep learning workloads. The rally was driven primarily by forward-looking narratives around artificial intelligence and autonomous vehicles, with price movements leading rather than following the fundamental confirmation ([37, 11]). This case illustrates the temporal asymmetry that motivates hybrid reasoning: trading signals captured the market’s forward expectation long before quarterly filings reflected it, meaning neither signal alone would have been sufficient to correctly make the investment decision.

(iii) Tesla (2020): Sentiment-Driven Rally Under Modest Profitability. Tesla’s stock rose dramatically in 2020 despite modest profitability at the time, as investors priced in narratives around electric vehicle adoption and autonomous driving at scale ([7] (Shiller 2017)). This example predates the main example we give in § 1 (the April 2025 earnings miss) and hence establishes that the fundamentals–market divergence is a recurring feature of Tesla’s price history rather than an isolated anomaly. It also shows how investor sentiment ([7]) and narrative economics ((Shiller, 2017)) can sustain multi-year divergences, not merely short-term reactions.

Takeaway. From these examples, we can see a consistent pattern emerging: price dynamics reflect investor expectations and narrative momentum that often precede or contradict the signal available in the company’s financial statements. Table 5 summarises the key characteristics of each case.

CompanyPeriodFundamental SignalDominant Signal
Amazon1999–2001Persistent losses, negative EPSMarket (narrative)
Nvidia2016–2017Moderate earnings, GPU revenue lagMarket (forward expectation)
Tesla2020Modest profitability, high P/EMarket (sentiment)
TeslaApr. 2025EPS miss (0.27vs.0.27 vs.0.42), revenue missMarket (narrative rally)

Table 5. Summary of fundamentals–market divergence episodes. “Dominant Signal” refers to which signal better characterised the market outcome at the time.

These cases collectively motivate FinTradeBench’s dual-signal design. A benchmark that evaluates reasoning over fundamentals alone would reward models that correctly identify Amazon, Nvidia, and Tesla as weak or fairly valued, missing the market trading signal entirely. A benchmark that evaluates trading signals alone would capture momentum but provide no mechanism for assessing whether that momentum is anchored in improving fundamentals or purely sentiment-driven. Only by evaluating both types of signals jointly, and by including hybrid questions that require reconciling conflicting signals, can we create a benchmark that reflects the reasoning demands of real-world financial analysis.

Extended Related Work

To understand the performance of LLMs across niche domain requires a comprehensive benchmark that covers different aspects of the domain ([41, 30, 12, 27, 33] (Liu et al. 2024b; Liu et al. 2024a; Kader and Karmaker 2025; Liang et al. 2022; Mialon et al. 2024; Singhal et al. 2023; Mazeika et al. 2024)). Recent financial-AI work includes text-centric financial QA benchmarks, chart- and table-reasoning benchmarks, as well as agentic and tool-use frameworks ([2, 15, 17, 16] (Xie et al. 2024a; Yang et al. 2023a; Xie et al. 2023; Klimaszewski et al. 2025; Peng et al. 2026; Peng et al. 2025; Larsen et al. 2026)). FinTradeBench is complementary to these efforts. Its evaluation is designed to diagnose distinct stages of a financial-reasoning pipeline which includes evidence access, computation, evidence selection, and synthesis, under a shared Fundamental/Trading/Hybrid taxonomy. The controlled conditions therefore focus on mechanism-level differences rather than treating a single RAG-versus-No-RAG score as a complete characterization of financial reasoning.

Trading Signals and Quantitative Finance. Trading signals derived from price and volume data play a key role in understanding stock market behavior and risk dynamics ((Lo et al., 2000)). Indicators such as momentum, volatility, moving averages, and drawdowns have been widely studied in asset pricing and quantitative finance ([29] (Jegadeesh and Titman 1993; Lo et al. 2000)). Volatility measures are also used to capture perceived market risk and regime changes ([28, 4, 9, 8]). While machine learning approaches have recently been applied to forecasting trading signals in finance ([39] (Mishra et al. 2024; Moreno-Pino and Zohren 2024; Li 2024)), there remains a gap in accurate financial analysis from multi-modal data and existing NLP benchmarks rarely evaluate whether LLMs can reason about these signals for financial analysis.

LLM Evaluation and Benchmark Design. Carefully designed benchmarks are essential for evaluating the reasoning capabilities of LLMs. Recent LLM benchmarks increasingly rely on calibrated automatic evaluation for open-ended outputs that are difficult to score with exact-match metrics. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (Zheng et al., 2023) established LLM-as-a-judge as a scalable evaluation paradigm for multi-turn, preference-oriented model outputs, while also identifying important evaluator biases such as position, verbosity, and self-enhancement bias. Subsequent work further studies reliability, bias, and calibration in LLM judging [13, 42, 38] (Ye et al. 2025). Financial QA presents additional challenges due to numerical fidelity, domain expertise, and the integration of heterogeneous data sources (Yang et al., 2023b; Ran et al., 2019; Zhang et al., 2024b). Open-ended answers often require factual grounding, numerical consistency, and domain-specific reasoning, making reference-only or lexical metrics insufficient. FinTextQA [14], for example, combines human ranking, automatic metrics, and GPT-4 scoring to evaluate long-form financial QA systems. These works provide a design of scalable evaluation pipeline while preserving domain-grounded controls over factuality, numerical fidelity, and reasoning quality thus motivating us to use a calibrated LLM-as-a-judge framework rather than treating LLM judgments as ground truth.

Comparison with Existing Benchmarks. Table 1 summarizes key differences between existing financial benchmarks and FinTradeBench. Prior financial QA benchmarks [16, 15, 14] (Islam et al. 2023) primarily emphasize reasoning over textual financial documents, while broad financial LLM benchmarks such as FinBen (Xie et al., 2024a) aggregate diverse NLP-style financial tasks. In parallel, quantitative finance research studies trading signals for forecasting, risk modeling, and asset pricing (Oberlechner, 2001; Jegadeesh and Titman, 1993), and recent agentic benchmarks evaluate simulated financial actions (Li et al., 2025a). Retrieval, Tools, and Heterogeneous Financial Evidence. Retrieval-augmented financial QA systems typically ground answers in external documents or structured context. FinanceBench, FinDER, FinTextQA, and AlphaFin [17, 14] (Islam et al. 2023; Li et al. 2024) illustrate the importance of grounding financial answers in retrieved information. Financial reasoning, however, also requires operations that are not purely retrieval problems, such as computing quantities from historical series, selecting among competing indicators, and reconciling evidence across modalities. FinTradeBench therefore evaluates several evidence pathways rather than a single RAG condition. Raw-RAG exposes point-in-time evidence, Indicator-RAG supplies precomputed structured signals, Tool-Use provides deterministic computation over raw market data, and Oracle supplies question-specific gold-selected indicators without providing the reference answer. This design makes retrieval, computation, and evidence selection separately observable.

LLM-as-a-Judge and Open-Ended Evaluation. Open-ended financial answers are difficult to score with exact match because quality depends on factual grounding, numerical consistency, completeness, relevance, and the quality of the reasoning used to connect evidence to a conclusion. LLM-as-a-judge work such as MT-Bench and Chatbot Arena (Zheng et al., 2023) established scalable model-based evaluation while also documenting biases such as position, verbosity, and self-enhancement. Subsequent work further examines judge reliability, bias, and calibration [13, 42, 38] (Ye et al. 2025). Financial QA adds domain-specific concerns because plausible language can mask incorrect quantities or unsupported interpretations (Yang et al., 2023b; Ran et al., 2019; Zhang et al., 2024b). FinTextQA [14], for example, combines human and model-based evaluation for long-form financial answers. Motivated by these concerns, FinTradeBench uses an evidence-aware LLM judge whose factual anchor is point-in-time benchmark evidence rather than the benchmark reference response. FinTradeBench separately validates the frozen judge on blinded human ratings to ensure the robustness of judge.

Addendum to Benchmark Design

Benchmark Summary and Validation Gates

Table 6 summarizes the distinct validation stages used in FinTradeBench. We distinguish construction-time checks from the final human audit of the Core evaluation set and from held-out validation of the automated judge.

CheckPurposeStage
Golden indicatorsProvide an expert-defined minimal evidence set for each question; used for construction auditing and controlled Oracle evidence, not as a complete answer template.Question design / controlled evaluation
Numerical auditFlag unsupported, contradicted, or unavailable numerical claims in construction-time candidate responses.Benchmark construction
Human–LLM alignmentVerify that automated judge scores approximate expert scoresJudge calibration
Aligned judge validationValidate the human-aligned LLM judge for benchmark scalingBenchmark Scaling
Scaled-set fatal-flaw screenIdentify catastrophic construction failures in the broader 1,400-question collection; 42 responses required regeneration during benchmark scaling.Broader benchmark construction
Canonical-data support screenRemove Core candidates whose required company/time-period evidence is absent from the frozen canonical data universe.450 → 403
Full item-level human auditReview question clarity, reference-answer correctness, canonical PIT evidence, golden-indicator adequacy, and entity/time alignment for every retained Core item.403 → 400
Common-evidence judgingScore paired RAG/No-RAG responses against the same canonical point-in-time evidence to remove evaluator-side evidence asymmetry.Final General-QA evaluation
Held-out human judge validationEvaluate the frozen Claude judge on blinded responses that were not used to modify its scoring prompt.Final evaluator validation

Table 6. Construction and validation checks in the FinTradeBench pipeline. The checks operate at different stages and should not be interpreted as interchangeable forms of validation.

Dataset Statistics

Table 7 summarizes the key FinTradeBench dataset and final evaluation statistics.

Financial Signal Reference

Table Table 8 summarizes the company-fundamental and trading signals used in question design. Trading signals are derived from historical OHLCV data, while fundamental signals originate from SEC 10-K/10-Q filings.

TELeR Prompt Taxonomy

FinTradeBench uses a TELeR-inspired prompt taxonomy for multi-prompt candidate generation. Table Table 9 describes the six TELeR-inspired prompt variants used for multi-prompt candidate generation during benchmark construction § 3. Prompts vary systematically in instruction richness and reasoning explicitness, from Level 0 (minimal context) to Level 6 (maximal justification).

Table Table 10 describes the TELeR taxonomy used during the General-QA study (§ 4). No-RAG uses four candidate prompt variants, and RAG uses two, followed by intra-model self-selection. Prompts vary systematically in instruction richness and reasoning explicitness, from Level 1 (single sentence high-level directive) to Level 6 (maximalist, evidence-citing, self-justified). Levels 1–4 are used without RAG; Levels 5–6 have RAG context.

LvlTypeStrategyRAG
L1BaselineSingle-sentence high-level directive.No
L2FocusRole specification + Paragraph style breakdown.No
L3CoTBulleted Step-by-Step Chain-of-Thought.No
L4AuditorAdds explicit evaluation criteria (e.g., coherence).No
L5ContextInjects retrieved evidence and citations.Yes
L6ExplainAdds self-justification for data usage.Yes

Table 10. The TELeR Prompt Taxonomy used for Benchmarking and evaluation for the RAG and No-RAG setting in § 4. The framework systematically increases prompt complexity, with Levels 5 and 6 designed to leverage RAG.

Sample Questions, Reference Answers, and Golden Indicators

Table Table 11 presents representative seed questions from each of the three FinTradeBench categories alongside their expert-authored golden answers. Each answer cites the specific golden indicators required for a complete response; these indicators serve as the reference set for Golden Indicator F1 scoring.

From 450 Candidates to the Human-Validated Core400.

For the main controlled evaluation, we first form a 450-item candidate Core pool. The pool contains the 150 seed questions together with company- and year-scaled questions selected for stronger historical coverage. We then apply two independent qual-

IDAttributesDescription (Example Behaviour)
L1Single-turn, low detail, instruction-style, role-specifiedInstructs the model in simple one-sentence instructions focusing on the high-level goal.
L2Single-turn, moderate detail, instruction-style, role-specifiedParagraph-style instructions expressing the high-level goal and sub-tasks that need to be performed to achieve the goal
L3Step-by-step reasoning, moderate detail, decomposedadds a structured reasoning template (bulleted-list-style): clarify goal, decompose into sub-questions, answer sub-parts using both contexts, and synthesize a final answer.
L4Step-by-step reasoning, moderate detail, decomposed“Level 3 Prompt” + “It provides a guideline on how LLMs will be evaluated.”
L5Step-by-step reasoning, high detail, decomposed“Level 4 Prompt” + “Provide additional relevant gathered via RAG”
L6Step-by-step reasoning, Maximalist, high detail, evidence-citing, justified“Level 5 Prompt” + “Provide explicit statement asking LLM to explain its own output.”

caption: Table 9: TELeR-inspired prompt variants used for multi-prompt generation during benchmark creation § 3.

:::

ity gates: (i) Canonical-data support screening. We verify that the frozen canonical financial-data universe can support each question and that the candidate contributes distinct evaluation coverage. 47 questions did not pass this screening because either the required company or historical evidence was insufficient or the question was redundant/repetitive with the retained pool. (ii) Full item-level human quality audit. An expert then audits all 403 retained questions to evaluate (i) question clarity and answerability, (ii) correctness of the reference answer, (iii) support from canonical point-in-time evidence, (iv) adequacy of the golden-indicator set, and (v) consistency of company identity and historical cutoff. The reviewer accepts 400 of 403 items (99.3%) thus giving us the final Core400: 143 Fundamental, 122 Trading, and 135 Hybrid questions. Table 12 summarizes this curation process. The human audit provides an additional content-quality check beyond the initial automated support screen. Across Core403, 400/403 questions receive a clean clarity pass, 401/403 reference answers are marked correct, 402/403 golden-indicator sets are judged adequate, and all 403 pass entity/time alignment. The full audit breakdown is reported in §D.8. Unless otherwise stated, all final quantitative experiments in this paper use the human-validated Core400. See §D.10 for detailed steps taken for

benchmark quality control.

Core Candidate Provenance

The auditable 450-item candidate pool consists of 150 seed-origin, 114 company-scaled, and 186 year-scaled items. Canonical-data support screening removes 47 items (29 company-scaled and 18 year-scaled), yielding Core403.

Full Human Quality Audit of Core403

After the canonical-data support screen, a financial expert audited all items in Core403 one at a time. For each item, the reviewer inspected the question, reference answer, expert-defined golden indicators, canonical point-in-time evidence, and entity/time alignment. The audit was a benchmark-quality review, not a model-response evaluation.

Three items are excluded after audit: FTB-0061 (Hybrid), FTB-0135 (Trading), and FTB-0149 (Trading). The final evaluation set therefore contains 143 Fundamental, 122 Trading, and 135 Hybrid (see Table 12). We preserve the pre-audit counts and exclusion log in the released benchmark metadata rather than silently overwriting the audit history. Automated evaluation coverage can occasionally be slightly below 400 when a model-specific evidence pipeline cannot construct a valid input; such cases are marked SOURCE_INVALID

Sample Golden Answers by Category

1. Company Fundamentals

Q: Is Nvidia’s profitability sustainable with its valuation in September 2025?

A: Nvidia’s profitability is highly sustainable and supported by strong operational efficiency. Its exceptional financial health is characterized by a ROE of 33.38% and a ROA of 21.31%, achieved with low leverage (Debt/Equity of 0.102). However, the valuation is extremely premium, evidenced by a Book/Price ratio of 0.023.

2. Trading Signals

Q: Based on trading in H1 2025, which stocks show the strongest EMA support?

A: INTC, MU, and LRCX show the strongest support, trading approx. 18.5%, 16.8%, and 14.4% above their 20-day EMAs. However, LRCX provides the most sustainable profile, as its technical trend is backed by strong fundamentals (18.7% ROE), whereas INTC’s support coincides with negative profitability metrics.

3. Hybrid Reasoning

Q: Is Microsoft overvalued based on Q3 2025 prices despite strong performance?

A: Microsoft is trading at a premium valuation, but it is not necessarily overvalued. While the Book/Price ratio of 0.0929 indicates the stock is expensive relative to its book value, this premium is supported by high operational efficiency (Cash Flow/Assets of 0.0689) and strong profitability (ROE of 0.0890). Technical indicators show strong momentum (RSI 65.42, MACD > Signal) without yet reaching extreme ”overbought” conditions (>70 RSI).

Table 11: Sample seed questions & corresponding golden answers; see details about key financial terms in Table 8.

StageTotalFTFT
Candidate pool450163124163
Canonical-data screening403143124136
Human audit400143122135

Table 12: FinTradeBench Core curation. We begin with 450 candidate items, then apply canonical-data support screening, followed by item-level human audit, producing the final human-validated Core400.

Human-audit criterionCountRate
Question clarity: Pass400/40399.3%
Question clarity: Pass or Minor403/403100.0%
Reference answer: Correct401/40399.5%
Canonical evidence: Full support399/40399.0%
Golden indicators: Adequate402/40399.8%
Entity/time alignment: Pass403/403100.0%
No fatal flaw401/40399.5%
Accepted into final Core400/40399.3%

Table 13: Full human quality audit of Core403. High pass rates reflect the preceding canonical-data support screen. The audit verifies the filtered Core and should not be interpreted as a quality estimate for the broader 1,400-question collection.

Sample bad responses by severity

1. Catastrophic failure

Q: Is Walmart a good defensive stock for September 2025?

Response: ...no direct assessment of its performance or defensive characteristics can be made. Proxy consumer staples and retail stocks (e.g., COST, PEP, KHC) exhibited mixed or negative returns during the period, with no clear defensive outperformance...

Reason: Does not talk about Walmart. Starts comparing different proxy stocks instead of focusing on Walmart.

2. Incomplete answer

Q: What companies have the most reliable moving average signals, based on 2025 back-testing?

A: ...The company displays a bearish Moving Average signal configuration. The last price (45.7400) is trading significantly below both the 20-day Simple Moving Average (49.6850) and the 20-day Exponential Moving Average (49.3665). This suggests that the short-term trend is negative. This technical weakness is corroborated by the fundamental data,

Reason: Focuses on one company only.

3. Sudden break

Q: During July 2025, what stocks are breaking out of long-term downtrends?

A: .......Additionally, AMD, Synopsys (SNPS), and The Trade Desk (TTD) are breaking out of medium-term corrections. These stocks showed negative momentum over the 2-to-12-month period but reversed sharply in July with gains ranging from 18% to 29%, supported by strong MACD crossover signals......

Reason: Breaks off suddenly without answering the main question.

Table 14: Sample bad responses caught during screening.

rather than assigned an artificial score.

Construction-Time Scaled-Set Screening

Benchmark scaling includes a separate fatal-flaw screen of the broader 1,400-question collection as a final screening before full benchmark release. For this, a Computer Science PhD student went through all the question-answer pairs. The idea was to screen for any catastrophic generation failures (e.g., wrong company, incomplete output, or abrupt truncation), not to serve as a full expert validation of financial reasoning. Any ambiguous responses were passed on to the financial expert for screening. Forty-two responses (3%) were regenerated after this screen. Some qualitative samples are provided in Table 14.

Benchmark Quality Control.

FinTradeBench applies quality control at multiple distinct levels. First, expert-defined golden indicators specify a minimal evidence set for grounded reasoning and reduce reliance on generic financial explanations. Second, candidate responses undergo structured numerical auditing against historical financial data. Third, the construction-time LLM evaluator is calibrated against human judgments on the expert seed material and then validated by a human using a scaled subset. Fourth, the 450-item Core candidate pool undergoes canonical-data support screening before experimental use. Fifth, all 403 support-screened Core items undergo a full human item-level audit, giving the final Core400 used in this paper. Finally, the human-aligned LLM judge used for model evaluation is frozen and validated separately on a held-out blinded human review.

Together, these checks separate evidence availability, annotation quality, benchmark-content validation, and evaluator reliability rather than treating them as a single quality-control stage. Detailed audit criteria, excluded-item documentation, numerical-audit definitions, and judge-validation statistics are provided in the appendix.

Addendum to Experimental Setup and Evaluation

Models Evaluated.

For the General-QA study, we evaluate six contemporary systems: Gemini-3.8-Flash, DeepSeek-V4-Flash, Llama-4-Scout-17B-16E-Instruct, Qwen3.8-27B, GPT-5.4-mini, and Ministral-3-8B-Instruct-2512. At the time of research, Gemini 3.8 Flash leads Vals Finance Agent v2 with 61.44 % accuracy [2]. These systems span API-hosted and open-weight models and provide a cross-model comparison without relying on parameter-count groupings.

For the controlled evidence study, we evaluate seven models spanning multiple model families and generations: Qwen2.5-32B-Instruct, DeepSeek-R1-Distill-Qwen-32B, Llama-3.1-8B-Instruct, Phi-4, Llama-4-Scout, Qwen3.8-27B, and GPT-5.4-mini. This broader controlled set tests whether evidence-pathway effects are consistent across heterogeneous model architectures and capability levels.

The time-series study evaluates ChatTS-8B and a GPT-5.4-mini + TimesFM 2.5 forecast-feature pipeline. Exact model identifiers, provider endpoints, inference parameters, hardware, and runtime details are reported in Appendix § 13.9.

Why Retrieval and Feature Augmentation over Parametric Fine-Tuning?

We prioritize retrieval-augmented generation and modular feature injection over parameter fine-tuning because financial analysis demands reasoning over exact, point-in-time evidence rather than static stylistic adaptation. Financial conclusions hinge on non-stationary numerical values across SEC filings and historical OHLCV series; relying strictly on parametric memory risks hallucination and prevents verifiable evidence auditing. Grounding parametric models with non-parametric external memory preserves provenance and supports factuality in knowledge-intensive domains (Lewis et al., 2020), while allowing continuous ingestion of newly released filings and market ticks without prohibitive retraining costs [32].

Recent empirical studies confirm that retrieval provides more dependable factual injection than supervised fine-tuning, which often adapts conversational tone without ensuring precise numerical recall (Ovadia et al., 2024).

Our empirical results in the time-series specialist evaluation (§ 5, RQ3) reinforce this distinction: a fine-tuned open-weight reasoning model on time-series representations, ChatTS underperformed a modular pipeline coupling a dedicated forecasting model TimesFM with a frozen frontier reasoning model GPTmini by 1.09 Likert points overall and by 1.60 points on Hybrid tasks (Figure 4). End-to-end parametric fine-tuning of smaller models often degrades complex cross-signal synthesis and numerical reasoning capabilities compared to modular pipelines that supply structured, up-to-date context to state-of-the-art general-purpose reasoners. Hence, our benchmark focuses on evaluating how effectively models reason over point-in-time financial evidence supplied dynamically at inference time.

RAG architecture explained

Here, we provide details for the architecture used for raw RAG vs No-RAG evaluation study described in § 4.

(1) Domain-Aware Hybrid Retrieval via RAG. We design a RAG-based architecture for the dual nature of financial analysis by integrating text and tabular data with numerical time-series data. Our four-part RAG implements a Dual-Track Retrieval Engine followed by TELER-guided generation and self-filtering before evaluation; see Figure 8. This design follows the general RAG principle of grounding generation in external evidence (Lewis et al., 2020), while adapting retrieval to the financial setting. We use a domain-aware dual-track retrieval architecture because financial evidence has two structurally different forms. (i) SEC track. Regulatory filings are parsed from HTML; non-content elements are removed, financial tables are converted into text-preserving representations, and filing sections are indexed hierarchically. Retrieval combines semantic and lexical matching followed by reranking, while larger parent sections are returned for generation to preserve financial context. (ii) Market-data track. Historical OHLCV observations are retrieved through a separate temporal path prioritizing ticker and date alignment rather than semantic similarity. This prevents natural-language retrieval models from underweighting structured numerical evidence.

(2) Data Ingestion and Indexing. Financial documents often contain high token density and complex tables that standard chunking corrupts. For regulatory filings, we parse SEC HTML documents, remove non-content elements, preserve section boundaries, and convert embedded financial tables into markdown-formatted text. To handle the complexities of the domain, we adopt two strategies: (a) Hierarchical Indexing: A parent-child strategy segments documents by logical boundaries (e.g., SEC “Item 7”) Shaukat et al., 2026; Zhou et al., 2026; Lewis et al., 2020. Each filing is segmented into parent sections (max 2,000 tokens) and child retrieval chunks of 300 tokens with overlap. Retrieval is performed over child chunks, while generation receives the corresponding parent context to preserve document-level coherence. (b) Metadata Injection: Structured metadata is prepended to every chunk embedding to mitigate temporal hallucination. For trading data, daily OHLCV histories are grouped into month-level, ticker-specific chunks, enabling temporal alignment between the question date and the relevant market evidence.

(3) Dual-Track Retrieval. We use dual-track RAG to handle the asymmetric structure of financial evidence; see Figure 8. In Track A, for each ticker, we build a FAISS dense vector index using BAAI/bge-large-en-v1.5 embeddings and maintain a BM25 lexical index for sparse matching.

At query time, the system first detects explicit tickers, company aliases, or market groups, and extracts temporal constraints such as the target year. It retrieves SEC evidence using dense similarity followed by cross-encoder reranking with ms-marco-MiniLM-L-6-v2. In Track B, trading-signal chunks are retrieved through a separate temporal query path that prioritizes ticker and date alignment. This separation is crucial because semantic rerankers trained primarily on natural language may undervalue structured numerical evidence. Retrieved SEC and trading contexts are merged under source-specific quotas and global context budget, with parent-level deduplication to avoid redundancies.

(4) Generation and self-selection. Given the retrieved evidence, each model generates multiple candidate answers using TELeR-inspired prompt variants ranging from simple direct answering (L1) to evidence-grounded, step-by-step reasoning (L6); see §D.4 Table 10. The model then performs intra-model self-selection among its own candidates against the retrieved context to identify the best RAG and best No-RAG response per model, consistent with the self-filtering approach used during benchmark construction; see § 3. This design reduces sensitivity to a single prompt template while avoiding cross-model preference leakage.

At inference time, evidence from the two tracks is merged under source-specific quotas and a shared context budget.

Evidence Conditions

Table 15 details the evidence provided for each evaluation pipeline.

Tool-Use Implementation

Tool-Use exposes a fixed inventory of deterministic financial computations. The router is deterministic and conditions only on the benchmark question and the available tool descriptions. It does not receive the reference answer, golden-indicator labels, or judge scores.

Time-Series Specialist Pipelines

The time-series evaluation contains 257 eligible Core400 questions: 122 Trading and 135 Hybrid. ChatTS-8B reasons directly over the historical adjusted-close series. The TimesFM 2.5 pipeline uses the latest 128 adjusted-close observations available at the point-in-time cutoff to generate a five-step forecast; forecast-derived features are then provided to GPT-5.4-mini for natural-language synthesis. Hybrid questions additionally receive point-in-time SEC evidence.

The comparison is between complete pipelines. It is not a pure architectural ablation because the downstream language components differ.

Evidence-Aware Judge and Common-Evidence Re-Judging

The final evaluator is Claude Sonnet 4.5 with deterministic structured output. Each answer is scored from 1–5 on Accuracy, Completeness, Relevance, and Clarity. The benchmark reference response and answer_conclusion_summary are never supplied to the judge as factual ground truth. Rubric scoring is anchored to point-in-time evidence.

For the General-QA comparison, both No-RAG and RAG answers are re-judged against the same condition-independent canonical evidence package. This removes evaluator-side evidence asymmetry while leaving the historical generation protocol unchanged.

Golden-indicator precision, recall, and F1 are ancillary diagnostics. They are not used as the primary response-quality score and should not be interpreted as proof that an answer using a different but defensible evidence set is wrong.

Evaluation Metrics and Statistical Testing

For response ii, let the four rubric scores be Ai,Ci,Ri,Li∈{1,…,5}A_i, C_i, R_i, L_i \in\{1,\ldots,5\} for Accuracy, Completeness, Relevance, and Clarity. The primary response score is

Si=Ai+Ci+Ri+Li4.(1)S_i = \frac{A_i + C_i + R_i + L_i}{4}. \tag*{(1)}

We report raw 1–5 scores as the primary metric. When useful for presentation, we additionally report the normalized judge score 100Si/5100S_i/5; this value is not called “absolute accuracy.”

For paired conditions XX and YY, the per-question effect is

di=Si(X)−Si(Y),dˉ=1N∑i=1Ndi.(2)d_i = S_i^{(X)} - S_i^{(Y)}, \qquad\bar{d} = \frac{1}{N}\sum_{i=1}^{N}d_i. \tag*{(2)}

We compute 95% confidence intervals for dˉ\bar{d} using 5,000 paired bootstrap resamples with a fixed random seed. Statistical significance is assessed with the paired Wilcoxon signed-rank test, and pp-values are corrected using the Holm procedure within each study family.

Rows without valid source evidence are marked SOURCE_INVALID; judge failures are recorded separately. We do not impute artificial scores for invalid rows.

Metric Extraction Precision (P(am∗a_m^*)), Recall (R(am∗a_m^*)), and F1 (F1(am∗a_m^*)) quantifies the overlap between the set of financial metrics cited in a generated response and the expert-defined reference set. For a self-selected response am⋆a_m^\star, let Mgen(am⋆)M_{\mathrm{gen}}(a_m^\star) are omitted from a paired comparison when a valid response is unavailable. denote the set of financial metrics mentioned in the response and MrefM_{\mathrm{ref}} the ground-truth reference set. Precision, recall, and F1 are then given by:

Figure 9. Controlled-condition evaluation. The question set, cutoff rules, model, answer budget, and judge are held fixed; only the evidence pathway changes. Indicator-RAG retrieves precomputed indicator and fundamental tables, Tool-Use lets the model query indicator values on demand, and Oracle supplies gold evidence as a probable upper bound, separating failures of evidence access from failures of reasoning.

P(am⋆)=∣Mgen∩Mref∣∣Mgen∣,R(am⋆)=∣Mgen∩Mref∣∣Mref∣,F1(am⋆)=2⋅P(am⋆)⋅R(am⋆)P(am⋆)+R(am⋆)(3)\begin{aligned} \mathrm{P}(a_m^\star) &= \frac{\lvert M_{\mathrm{gen}} \cap M_{\mathrm{ref}}\rvert}{\lvert M_{\mathrm{gen}}\rvert},\\ \mathrm{R}(a_m^\star) &= \frac{\lvert M_{\mathrm{gen}} \cap M_{\mathrm{ref}}\rvert}{\lvert M_{\mathrm{ref}}\rvert},\\ \mathrm{F1}(a_m^\star) &= \frac{2 \cdot\mathrm{P}(a_m^\star) \cdot\mathrm{R}(a_m^\star)}{\mathrm{P}(a_m^\star) + \mathrm{R}(a_m^\star)} \tag*{(3)} \end{aligned}

respectively. The macro-averaged F1 across all KK models is:

F1‾=1K∑m=1KF1(am⋆),(4)\overline{\mathrm{F1}} = \frac{1}{K}\sum_{m=1}^{K} \mathrm{F1}(a_m^\star), \tag*{(4)}

where KK is the total number of evaluated models and am⋆a_m^\star is the self-selected best response from model MmM_m.

These overlap metrics are secondary diagnostics rather than the primary measure of answer quality.

Statistical Analysis. All condition comparisons are performed at the question level using paired observations wherever both outputs are valid. We report the paired mean difference in raw judge Likert score together with a non-parametric 95% bootstrap confidence interval based on 5,000 paired resamples.

Statistical significance is assessed using the paired Wilcoxon signed-rank test, with pp-values corrected using the Holm procedure within each study family.

We do not assign artificial scores to generation, retrieval, or evidence failures. Such cases are marked as SOURCE_INVALID or judge errors and

Compute Resources and Reproducibility

Open-weight inference was run on GPU-enabled university cluster resources, while hosted systems and the automated judge were accessed through their corresponding cloud APIs.

Addendum to Results and discussions

The frozen judge closely tracks held-out human scores.

At the response level, the mean four-dimension score has an MAE of 0.214 Likert points relative to the human reviewer and Spearman correlation ρ=0.938\rho= 0.938. Across all 192 individual rubric judgments, exact agreement is 76.6%, 96.9% of scores fall within one Likert point, and quadratic-weighted κ=0.884\kappa= 0.884. Accuracy shows the strongest dimension-level agreement (ρ=0.952\rho= 0.952, 83.3% exact agreement), while Relevance has the largest residual discrepancy. The automated judge scores Relevance approximately 0.31 points higher on average than the human reviewer, suggesting mild positive calibration bias on this dimension. Pairwise evaluation leads to the same qualitative conclusion. Strict A/B/Tie agreement is 70.8%. The human reviewer expresses a directional preference on 15 of the 24 pairs; the automated judge agrees with all 15, with no directional reversals. Most disagreement therefore comes from tie calibration: the human reviewer sometimes considers two answers equivalent when the judge assigns a small numerical difference.

Robustness to Final Core400 Curation

As a sensitivity check, we recompute the analyses before and after the item-level Core audit. Excluding the three non-retained Core403 items leaves all controlled-condition and common-evidence RAG/No-RAG means and paired effects unchanged because those items were already invalid in the corresponding automated evidence pipelines.

The time-series comparison is the only analysis affected numerically: the eligible set changes from 259 to 257 questions and the paired TimesFM+GPT-ChatTS effect changes from approximately +1.08 to +1.09 Likert points. The substantive conclusions are unchanged. Full Core403–Core400 sensitivity results are reported in §H.2.

Discussion

Our experiments show that financial reasoning errors cannot be reduced to a single retrieval problem.

First, retrieval remains useful: under a shared factual evaluation anchor, RAG consistently improves Trading and Hybrid reasoning across the six-model General-QA set. However, the weak and inconsistent Fundamental effects show that providing more evidence alone is insufficient.

Second, the controlled conditions identify two distinct downstream bottlenecks. Deterministic tools are most useful when explicit numerical computation is required, whereas gold-selected evidence improves every model and is especially beneficial on Trading tasks. This indicates that both computation and evidence selection contribute to financial reasoning failures, with their importance varying by question type.

Third, specialized representations matter. The strong performance of the forecast-feature pipeline suggests that summarizing numerical time-series structure before language reasoning can be more effective than requiring a language-oriented model to infer all relevant market structure directly from the raw sequence.

stages may affect benchmark construction or evaluator development, whereas the final held-out study is performed only after the judge configuration is frozen and does not modify the evaluator. Table 18 summarizes these distinctions, and the reviewer instruments are reproduced in §L.

StageUnit reviewedHuman taskRole in pipeline
Seed construction and development calibrationSeed questions and generated responsesExperts author seed questions and golden indicators; generated responses are subsequently scored to calibrate the development-time automated evaluator.Benchmark construction and development evaluator
Scaled-set fatal-flaw screeningScaled question–answer pairsScreen for catastrophic generation failures such as wrong entity, incomplete answers, or abrupt truncation; ambiguous cases are escalated.Construction quality control
Core403 item-level auditBenchmark itemsInspect the question, reference answer, golden indicators, canonical point-in-time evidence, and entity/time alignment.Final benchmark curation: Core403 → Core400
Held-out judge validationAnonymous model responsesScore responses using the frozen four-dimension rubric and record blinded A/B/Tie preferences.Post-freeze evaluator validation only

Table 18. Distinct human-review stages in FinTradeBench. The stages operate on different objects and serve different purposes. The held-out judge-validation labels are not used to modify the frozen evaluation procedure.

Overall, FinTradeBench therefore functions not only as a model comparison benchmark but as a diagnostic framework for locating failure along the path from evidence access to computation, evidence selection, and final financial synthesis.

Human Review and Evaluator Validation

Human involvement occurs at four distinct stages, each serving a different methodological purpose: (1) expert-guided seed construction and development-time evaluator calibration; (2) construction-time fatal-flow screening of scaled responses; (3) full item-level quality auditing of the support-screened Core403; and (4) held-out validation of the frozen automated judge. These stages differ in both their unit of review and their role in the pipeline. In particular, the first three

Expert Seed Construction and Development-Time Evaluator Calibration

The first stage combines two closely related development activities. First, a domain expert constructs the seed portion of FinTradeBench, including the benchmark questions and their expert-defined golden indicators. These annotations establish the intended financial evidence needed to reason about each seed question and provide the foundation for subsequent benchmark scaling.

Second, during evaluator development, human experts score filtered model responses to seed questions on the same response-quality dimensions used by the automated evaluator. These annotations are used to inspect and calibrate the evaluation prompt during development. The corresponding human rubric includes factual and numerical ac-

└analysis ...The page contains a table top. Need starts_mid maybe table begins this page, not continuation? It has table header and full table, likely table starts on page, but preceding page maybe text and table starts here. "This page continues section Addendum..." means starts_mid=1 because paragraph continuation at bottom? Actually first thing is table, not continuation. Starts_mid applies first thing. Table appears likely begins at page top with header, not continuation. Set starts_mid=0? User skeleton says page continues section, not necessarily starts mid. Table caption then text starts with continuation from prior page? At bottom prior page likely paragraph ending and this page starts with table, then after table text "strong performance..." clearly continuation of paragraph from previous page? It appears left column starts "strong performance..." and right column continues sentence "stages may..." This paragraph likely began previous page and is interrupted by table float; page starts table but first body reading order table. starts_mid definition first thing page continues table/list/equation from previous. Table header suggests table likely starts here, no indication continuation. So 0 plausible. But page top table likely caption at bottom, full table. Set 0. Yet ends mid yes. Also table columns mismatch: header 6 but actual visual first header "Recorded wall time" and "completing-job" maybe last two columns? Rows time values in fifth? Data appears 5 columns? Header line shows Recorded wall time / completing-job split due narrow: likely single header "Recorded completing-job wall time". PDF text layer says "Recorded completing-job wall time" likely one column. Need 5 cols, not 6. fix table. Also "General QA" in execution. citation parentheses should exact; citation role includes parentheses. Good.

Need no prose. revise starts 0 and table 5 columns. Caption appears body text after. Also table starts likely page top and isn't continued. citation Qwen team blue parentheses yes.

цҳа final concise. curacy, completeness and context, relevance and utility, and clarity and rationale, together with a binary check of agreement with the construction-time numerical audit; see Table 28.

CriterionScaleReviewer instruction / anchors
Audit Validation Agreement0–1Does the independent human review agree with the automated numerical audit’s is_numerically_accurate flag? 1: Agree—the audit conclusion is correct. 0: Disagree—the audit missed an error or incorrectly flagged a correct claim.
Factual & Numerical Accuracy1–5Based on the reviewer’s own assessment, together with the numerical audit, assign the final accuracy score. 5: 100% correct. 1: Contains significant, misleading numerical errors.
Completeness & Context1–5Does the answer fully address the question and correctly use and contextualize the golden indicators? 5: Excellent; uses required metrics in a deep, integrated analysis. 1: Superficial; misses most required metrics or necessary context.
Relevance & Utility1–5Is the included information relevant, and does the response avoid fluff or potentially misleading tangents? 5: High precision; no fluff or harmful information. 1: Cluttered with irrelevant or misleading content.
Clarity & Rationale1–5Is the answer clear and well structured, and does it explain its reasoning? 5: Exceptionally clear and well reasoned. 1: Confusing, poorly written, or reasoning is opaque.

Table 28. Development-time human annotation rubric. This rubric was used during calibration of the construction-time automated evaluator. Reviewers additionally had optional fields for a human-written golden answer and free-text notes. Intermediate values between the endpoint anchors were available on the 1–5 scale but were not assigned additional verbal anchors in the original annotation instructions.

On this development sample, the human and automated scores have an overall MAE of approximately 0.40 on the 1–5 scale. Because the filtered development responses are heavily concentrated toward high scores, rank-based agreement measures are less informative for this sample. We therefore treat these measurements as evidence of development-time calibration, rather than as the final validation of evaluator reliability.

Crucially, the development annotations are not the held-out validation set used to support the final experiments. Reliability of the frozen judge is evaluated separately in §G.4.

Construction-Time Fatal-Flaw Screening of Scaled Responses

After scaling the benchmark, the generated question–answer pairs undergo a separate human quality-control screen. The purpose of this stage is deliberately narrower than response grading: the reviewer is not asked to re-evaluate the full financial reasoning quality of every answer. Instead, the screen targets clear construction failures that would make an item unsuitable for release.

Examples include answering for the wrong company or ticker, failing to answer a required part of the question, using an inconsistent historical period, or terminating before the requested analysis is complete. Responses that fail this screen are regenerated before inclusion in the broader benchmark, while ambiguous cases are escalated for financial-

expert review. In the original scaling process, 42 of the 1,400 generated responses (3%) required regeneration.

The operational screening categories are summarized in Table 29 and sample bad responses are provided in Table 14. We emphasize that this step is a construction quality-control gate, not an independent Likert-scale evaluation and not the same procedure as the later Core403 item-level audit.

Screening categoryOperational interpretationAction
Wrong entity / companyResponse analyzes a different company, ticker, or proxy rather than the entity requested by the question.Regenerate
Wrong period / ticker–period mismatchResponse is inconsistent with the requested historical entity or period.Regenerate
Unsupported material claimsResponse contains material claims that are not supported by the benchmark evidence available for the item.Regenerate / escalate if ambiguous
Conclusion inconsistent with supplied evidenceFinal conclusion conflicts materially with the information or indicators associated with the item.Regenerate / escalate if ambiguous
Incomplete answerResponse fails to answer a required part of the question; for example, a cross-company question is answered using only one company.Regenerate
Abrupt truncation / sudden breakGeneration terminates before completing the reasoning or answering the main question.Regenerate

Table 29. Construction-time fatal-flaw screening guide. This stage was a release-quality screen rather than an independent financial-reasoning score. Responses exhibiting catastrophic failures were regenerated and retained only after passing the screen; ambiguous cases were escalated for financial-expert review.

Full Item-Level Human Quality Audit of Core403

A substantially stricter human review is performed after the canonical-data support screen. At this stage, the unit of review is the benchmark item itself, rather than a model-generated response. All 403 retained Core candidates are examined individually.

For each item, the reviewer inspects four primary artifacts: the question, the benchmark reference answer, the expert-defined golden indicators, and the canonical point-in-time evidence. The reviewer then assesses question clarity, reference-answer correctness, evidence support, golden-indicator adequacy, and entity/time alignment. Fatal flaws and the final disposition of each item are recorded separately. The full categorical rubric is reported in Table 31.

DimensionLabelsRubric definition
Question ClarityPass; Minor issue; Major issuePass: clear and answerable at the historical point in time. Minor issue: small wording or precision issue that does not change the task. Major issue: ambiguous, inconsistent, based on an unanswerable or false premise, or not answerable from benchmark evidence.
Reference Answer CorrectnessCorrect; Minor correction; Major incorrect; UnverifiableCorrect: material claims and conclusion are supported. Minor correction: main conclusion remains valid but a small correction is needed. Major incorrect: material claims or conclusion conflict with evidence. Unverifiable: supplied evidence is insufficient. For subjective investment-style questions, the reviewer was instructed not to require a unique recommendation; the conclusion instead had to be defensible and appropriately qualified.
Evidence SupportFull; Partial; Unsupported; Evidence missingFull: evidence is sufficient for the material claims and conclusion. Partial: supports the main conclusion but not all secondary claims. Unsupported: material claims are unsupported or contradicted. Evidence missing: no usable canonical evidence.
Golden-Indicator AdequacyAdequate; Minor issue; Missing critical; Incorrect; N/AAdequate: captures the key evidence. Minor issue: basically correct with a non-critical omission or extraneous item. Missing critical: an essential indicator is absent. Incorrect: materially mismatched. N/A: an indicator checklist is not meaningful for the item. Golden indicators were explicitly treated as a minimal evidence set, not a mandatory wording checklist.
Entity / Time AlignmentPass; Unclear; FailPass: entity and cutoff are aligned and no post-cutoff evidence is required. Unclear: alignment cannot be verified confidently. Fail: wrong entity or historical period, or future information is required or used.
Record fields
Fatal flawNo; YesIf Yes, the reviewer was required to select a fatal-flaw type.
Fatal-flaw typeWrong entity/ticker; wrong period or future leakage; ambiguous or unanswerable question; reference answer incorrect or unsupported; golden indicators incorrect; canonical evidence missing; duplicate; OtherUsed to make exclusion and adjudication decisions auditable rather than silently modifying problematic items.
Overall dispositionAccept; Accept minor edit; Needs adjudication; ExcludeFinal item-level decision.
Human notesFree textOptional for Accept; strongly recommended for all other dispositions.

Table 31. Core403 item-level human quality-audit rubric and record fields. The unit of annotation is a benchmark item, not a model response.

This audit is intentionally different from model-response evaluation. For example, a question can fail because its reference answer is not sufficiently supported, because its historical cutoff is ambiguous, or because its golden indicators omit critical evidence even if a model could nevertheless produce a plausible answer.

DimensionMAEρ\rhoExact±1\pm1QWK
Accuracy0.170.9583.3%100.0%0.945
Completeness0.290.8875.0%95.8%0.882
Relevance0.400.8668.8%91.7%0.809
Clarity0.210.8679.2%100.0%0.857
All rubric ratings0.270.8976.6%96.9%0.884

caption: Table 19: Held-out human–judge agreement. Statistics are computed over 48 responses and 192 individual rubric ratings. Exact and ±1\pm1 denote the proportion of automated ratings identical to, or within one Likert point of, the human rating. QWK denotes quadratic-weighted Cohen’s κ\kappa.

:::

The reviewer accepts 400 of the 403 items into the final evaluation set. Three items—FTB-0061, FTB-0135, and FTB-0149—are excluded, yielding the final Core400. The resulting set contains 143 Fundamental, 122 Trading, and 135 Hybrid questions.

Held-Out Human Validation of the Frozen Judge

The final human-review stage evaluates the reliability of the frozen automated judge. This study is conducted only after the judge prompt, scoring schema, and evaluation procedure have been fixed. Its human labels are therefore used solely for validation and do not modify the judge.

We construct a blinded sample of 24 paired questions, corresponding to 48 anonymous responses. The sample spans all three FinTradeBench reasoning lanes and four experimental comparison families: RAG–No-RAG, Oracle–Indicator, Tool–Indicator, and TimesFM+GPT–ChatTS. The reviewer is blinded to model identity, experimental condition, and automated judge score.

Each anonymous response is independently scored from 1–5 on Factual and Numerical Accuracy, Completeness and Context, Relevance and Utility, and Clarity and Rationale, using the rubric in Table 30. After both responses in a pair have been scored independently, the reviewer additionally records an A/B/Tie preference and a 1–5 confidence score (Table 30).

Dimension54321
Factual & Numerical AccuracyFully supported by supplied evidence; calculations and entity/time references are sound.Mostly correct with a minor non-material issue.Mixed: useful core analysis but meaningful errors or unsupported details.Substantial factual or numerical problems.Materially wrong, contradicted by evidence, or wrong entity/period.
Completeness & ContextFully answers the question and covers evidence needed for the conclusion.Substantively complete with a minor omission.Partially complete; misses a meaningful aspect.Major omissions.Largely non-responsive.
Relevance & UtilityDirectly addresses the question with useful synthesis.Relevant with small digressions.Useful but noticeably generic, padded, or indirect.Weakly relevant.Mostly irrelevant or non-responsive.
Clarity & RationaleConcise, clear, and well justified.Clear with minor presentation issues.Understandable but wordy, repetitive, or mechanical.Difficult to follow or poorly justified.Confusing or disjointed.
Pairwise fields
Pairwise preference (A / B / Tie)Entered only after all four ratings were completed independently for both responses.
Pairwise confidence (1–5)Integer confidence in the preference.
Pair notes (free text)Optional.

Table 30. Held-out human response-scoring rubric and pairwise fields. Reference metrics were presented only as helpful context and were explicitly not treated as a mandatory checklist. For subjective investment-style questions, a conclusion differing from another response could still receive a high score when it was supported by the supplied evidence and appropriately qualified.

At the response level, the arithmetic mean across the four scoring dimensions yields MAE = 0.214 and Spearman ρ\rho = 0.938 between the human reviewer and the frozen judge. The mean Claude-minus-human difference is +0.099 Likert points. The largest dimension-level mean difference occurs for Relevance (+0.313), whereas Accuracy and Completeness exhibit little mean bias.

Pairwise agreement provides a complementary

ComparisonPairsDecisiveStrictDirect.
Overall241570.8%100.0%
Oracle vs. Indicator8462.5%100.0%
RAG vs. No-RAG8687.5%100.0%
TimesFM+GPT vs. ChatTS4375.0%100.0%
Tool vs. Indicator4250.0%100.0%

caption: Table 20: Pairwise preference agreement. Decisive denotes pairs for which the human reviewer selected A or B rather than Tie. Strict agreement requires identical A/B/Tie labels. Directional (Direct.) agreement is computed only over the 15 human-decisive pairs.

:::

test because it asks whether the judge preserves the human reviewer’s relative preference between two responses rather than merely reproducing absolute scores.

Strict A/B/Tie agreement is 70.8% overall. The lower strict agreement is primarily associated with pairs for which the human reviewer selects Tie while the automated scores imply a small numerical preference. Among the 15 pairs for which the human reviewer expresses a directional A/B preference, the judge agrees on all 15, with no preference reversal. Moreover, the sample is intentionally stratified across reasoning lanes, comparison families, and score-gap regimes to study evaluator behavior rather than to estimate a population-level model preference rate.

Evaluation Integrity and Sensitivity Checks

Beyond human review, we perform additional checks on evaluation coverage, evidence consistency, and sensitivity to the final Core403 to Core400 curation step. These analyses verify that the reported effects are not artifacts of evaluator-side evidence asymmetry, context truncation, or the final three benchmark exclusions.

Evaluation Coverage and Evidence QC

For the General-QA RAG–No-RAG comparison, both responses in each pair are scored against the same condition-independent canonical point-in-time evidence package. This removes evaluator-side evidence asymmetry between the two generation conditions. The final common-evidence and time-series judge inputs contain no evidence truncation.

The time-series study is restricted by design to questions requiring market time-series reasoning. Its eligible Core400 subset therefore contains 257 questions: 122 Trading and 135 Hybrid items. Full file-level coverage, evidence-size

QuantityCore403Core400Change
ChatTS overall3.03863.0321−0.0065
GPT+TimesFM overall4.11584.1235+0.0077
Paired time-series effect1.07721.0914+0.0142

caption: Sensitivity to the final human-audit exclusions. The final three exclusions produce only small numerical changes in the time-series study and do not change its substantive conclusion.

:::

diagnostics, source-validity counts, and ancillary metric-overlap analyses are included with the released analysis artifacts.

Sensitivity to the Core403–Core400 Transition

The final human audit excludes FTB-0061, FTB-0135, and FTB-0149. These items already lack valid paired observations in the controlled condition and common-evidence RAG–No-RAG analyses. Consequently, transitioning from Core403 to Core400 does not alter their reported means or paired-effect estimates.

The time-series study is the only analysis for which the exclusions change the numerical summaries, because excluded items overlap its eligible Trading/Hybrid subset. As shown in Table 21, the resulting changes are small and do not alter the substantive conclusion.

Complete Quantitative Results

The main paper reports compact summaries. This section contains the full paired effects, bootstrap intervals, and Holm-adjusted Wilcoxon tests.

Common-Evidence RAG vs. No-RAG

Table 22 reports the per-model, per-lane effect of retrieval under the common-evidence judge.

LaneNNΔ\Delta95% CIW/T/LpHolmp_{\mathrm{Holm}}
DeepSeek-V4-Flash
All399+0.23[.15, .30]53/17/31<.001
Fund.143−0.06[−.19, .07]39/15/46.683
Trad.121+0.46[.35, .57]69/16/16<.001
Hybrid135+0.32[.21, .44]53/19/27<.001
Llama-4-Scout
All399+0.23[.18, .28]55/22/23<.001
Fund.143+0.06[−.02, .14]39/26/35.551
Trad.121+0.41[.33, .48]73/18/9<.001
Hybrid135+0.25[.16, .34]55/22/23<.001
Qwen3.8-27B
All399+0.22[.15, .30]54/14/32<.001
Fund.143−0.03[−.16, .10]37/20/43.683
Trad.121+0.23[.13, .32]60/13/26<.001
Hybrid135+0.48[.34, .63]67/8/24<.001
Gemini-3.8-Flash
All398+0.20[.13, .28]55/15/30<.001
Fund.143−0.10[−.24, .03]40/15/45.551
Trad.121+0.33[.20, .45]62/11/27<.001
Hybrid134+0.42[.31, .54]65/19/16<.001
GPT-5.4-mini
All399+0.17[.09, .24]53/17/30<.001
Fund.143−0.14[−.27, −.02]37/18/45.480
Trad.121+0.35[.25, .44]63/20/17<.001
Hybrid135+0.33[.21, .45]60/14/26<.001
Ministral-3-8B
All399+0.25[.20, .30]59/20/21<.001
Fund.143+0.14[.06, .22]49/22/29.013
Trad.121+0.32[.24, .40]68/17/15<.001
Hybrid135+0.31[.23, .40]62/21/16<.001

Table 22. Effect of retrieval (RAG − No-RAG) on the common-evidence Core400, per model and lane (Fund. = Fundamental, Trad. = Trading). Δ\Delta: mean paired difference in 1–5 judge Likert score (multiply by 20 for the 0–100 scale); 95% CI: paired bootstrap; W/T/L: % of items where RAG wins, ties, or loses; pHolmp_{\mathrm{Holm}}: Wilcoxon signed-rank pp, Holm-corrected across all 24 tests. Bold: pHolm<0.05p_{\mathrm{Holm}} < 0.05.

Controlled Evidence Conditions

Table 23 reports the full paired Tool, Oracle, and Indicator-RAG comparisons for every model and lane.

LaneNNTool − Indicator Δ\DeltaTool − Indicator 95% CITool − Indicator pHolmp_{\mathrm{Holm}}Oracle − Indicator Δ\DeltaOracle − Indicator 95% CIOracle − Indicator pHolmp_{\mathrm{Holm}}Oracle − Tool Δ\DeltaOracle − Tool 95% CIOracle − Tool pHolmp_{\mathrm{Holm}}
Qwen2.5-32B
All399−0.17[−0.27, −0.06].004+0.26[0.17, 0.36]<.001+0.43[0.33, 0.53]<.001
Fund.143−0.74[−0.90, −0.58]<.001+0.01[−0.11, 0.13]1.00+0.75[0.61, 0.89]<.001
Trad.121+0.64[0.44, 0.85]<.001+0.82[0.62, 1.04]<.001+0.18[−0.05, 0.40]1.00
Hybrid135−0.28[−0.41, −0.15]<.001+0.03[−0.10, 0.15]1.00+0.31[0.15, 0.46]<.001
R1-Distill-Qwen-32B
All399−0.37[−0.46, −0.29]<.001+0.31[0.23, 0.41]<.001+0.69[0.59, 0.78]<.001
Fund.143−0.72[−0.87, −0.58]<.001+0.16[0.05, 0.28].046+0.89[0.72, 1.04]<.001
Trad.121+0.02[−0.12, 0.18]1.00+0.73[0.53, 0.93]<.001+0.70[0.51, 0.91]<.001
Hybrid135−0.35[−0.47, −0.24]<.001+0.10[−0.03, 0.24].462+0.46[0.30, 0.60]<.001
Llama-3.1-8B
All399−0.02[−0.11, 0.07]1.00+0.57[0.47, 0.66]<.001+0.58[0.48, 0.69]<.001
Fund.143−0.21[−0.36, −0.06].032+0.47[0.34, 0.59]<.001+0.68[0.52, 0.84]<.001
Trad.121+0.34[0.17, 0.51].002+0.87[0.67, 1.08]<.001+0.53[0.30, 0.75]<.001
Hybrid135−0.14[−0.28, 0.01].107+0.40[0.27, 0.53]<.001+0.54[0.37, 0.69]<.001
Phi-4
All399−0.44[−0.54, −0.33]<.001+0.38[0.29, 0.47]<.001+0.82[0.72, 0.91]<.001
Fund.143−1.03[−1.18, −0.90]<.001+0.13[0.01, 0.24].462+1.16[1.03, 1.29]<.001
Trad.121+0.35[0.16, 0.55].034+0.91[0.71, 1.10]<.001+0.56[0.36, 0.75]<.001
Hybrid135−0.51[−0.63, −0.39]<.001+0.17[0.05, 0.28].008+0.68[0.54, 0.81]<.001
Llama-4-Scout
All399−0.18[−0.28, −0.10].001+0.65[0.58, 0.73]<.001+0.84[0.75, 0.92]<.001
Fund.143−0.63[−0.77, −0.49]<.001+0.47[0.36, 0.57]<.001+1.09[0.94, 1.24]<.001
Trad.121+0.29[0.13, 0.45].006+0.67[0.51, 0.83]<.001+0.38[0.21, 0.55]<.001
Hybrid135−0.14[−0.27, −0.01].241+0.83[0.71, 0.95]<.001+0.97[0.85, 1.09]<.001
Qwen3.8-27B
All399+0.04†[−0.06, 0.13]1.00+0.17[0.10, 0.25]<.001+0.05†[−0.04, 0.14]1.00
Fund.143+0.06[−0.08, 0.20]1.00+0.06[−0.06, 0.19]1.00+0.00[−0.13, 0.13]1.00
Trad.121+0.56†[−0.15, 1.31]1.00+0.40[0.26, 0.55]<.001−0.25†[−1.00, 0.50]1.00
Hybrid135−0.03[−0.13, 0.08]1.00+0.09[−0.02, 0.20].538+0.13[0.01, 0.24].070
GPT-5.4-mini
All399−0.12[−0.19, −0.05]<.001+0.19[0.13, 0.24]<.001+0.31[0.25, 0.37]<.001
Fund.143−0.35[−0.47, −0.24]<.001+0.13[0.06, 0.20].003+0.48[0.38, 0.58]<.001
Trad.121+0.17[0.04, 0.31].538+0.33[0.20, 0.48]<.001+0.16[0.06, 0.27].010
Hybrid135−0.14[−0.22, −0.06].004+0.12[0.07, 0.18]<.001+0.26[0.19, 0.35]<.001

Table 23. Full paired effects for the controlled conditions on the final Core400, per model and lane (Fund. = Fundamental, Trad. = Trading). Δ\Delta: mean paired difference in 1–5 judge Likert score (multiply by 20 for the 0–100 scale); 95% CI: paired bootstrap; pHolmp_{\mathrm{Holm}}: two-sided Wilcoxon signed-rank pp on per-question paired differences, Holm-corrected within each experimental batch over all model ×\times lane ×\times comparison tests (48 tests for Qwen2.5-32B, R1-Distill-Qwen-32B, Llama-3.1-8B and Phi-4; 36 for Llama-4-Scout, Qwen3.8-27B and GPT-5.4-mini). Bold: pHolm<0.05p_{\mathrm{Holm}} < 0.05. NN is the number of paired questions. †Qwen3.8-27B’s Tool condition has incomplete coverage, so comparisons involving Tool use fewer pairs: 290 overall and only 12 on Trading.

Time-Series Specialist Effects

Table 24 reports the paired TimesFM+GPT versus ChatTS effects on the time-series-eligible Core400 subset.

LaneNNΔ(T+G)−C\Delta_{(T+G)-C}95% CI
Overall257+1.091[0.944, 1.245]
Trading122+0.525[0.277, 0.775]
Hybrid135+1.604[1.472, 1.739]

Table 24. Paired time-series specialist effects on the Core400 eligible subset. Here, Δ(T+G)−C\Delta_{(T+G)-C} represents (TimesFM+GPT)−ChatTS(TimesFM + GPT) - ChatTS values. All three comparisons remain significant after Holm correction.

Qualitative Case Studies

Tables 25–27 show one GPT-5.4-mini seed question per reasoning lane, evaluated under every evidence condition. The qualitative examples ill-

trate how different evidence representations can change a model’s reasoning path. They are not used to establish population-level effects; the controlled experiments in § 5 provide the quantitative evidence.

Data Transparency, Responsible Use, Maintenance, and AI Disclosure

Data Transparency, Privacy, and Reproducibility

FinTradeBench is constructed from public company-level financial information, including SEC EDGAR filings and historical market data. The benchmark is not constructed from private investor records, user conversations, material non-public information, or personal financial data. Questions are written at the company/ticker and historical-period level.

The released benchmark will include the 1,400-question collection, explicit origin metadata, the human-validated Core400, the Core403 audit/exclusion log, golden-indicator metadata, derived signals, frozen prompts, and evaluation scripts. Raw source-data redistribution remains subject to the terms of the original providers; where redistribution is restricted, reconstruction scripts, ticker/date identifiers, and derived features should be released instead.

All reported final results are tied to a frozen benchmark version and frozen model-output/judge files so that future changes to hosted APIs or source data do not silently alter published numbers.

Safety, Ethics, and Broader Impact

FinTradeBench is a research benchmark for historical financial reasoning. It is not a trading system, portfolio-management system, suitability engine, or investment-advisory product. Benchmark performance should not be interpreted as evidence that a model is safe or profitable for live financial deployment.

Real-world financial use would require independent numerical verification, human oversight, risk controls, source attribution, uncertainty disclosure, and appropriate regulatory/compliance review. Systems capable of producing fluent financial narratives may also amplify misleading or manipulative claims; therefore, high benchmark performance should not substitute for deployment governance.

Data Maintenance

FinTradeBench will be released upon acceptance and maintained as a versioned benchmark. Each release will freeze the question set, reference answers, golden indicators, derived signals, point-in-time metadata, prompts, judge version, and scoring scripts. Corrections to ambiguous questions, ticker-period mappings, evidence, or golden indicators will be documented in a changelog while prior releases remain available for reproducibility.

Future versions may broaden market coverage, extend the temporal window, and add additional evidence sources. Such expansions will be released as new versions rather than silently modifying Core400.

Disclosure of Substantive AI Assistance

In accordance with the ACL and ARR Policy on Substantive AI Assistance, we disclose the ways in which artificial intelligence tools were employed during the preparation of this work:

Writing and Editorial Assistance: We used large language models as an editorial assistant to polish draft prose, shorten complex phrasing (e.g., streamlining the abstract to meet length requirements), and improve readability in select sections. The authors conceived, verified, and finalized all substantive technical arguments, mathematical formalisms, experimental analyses, and conclusions.

Code Development and Debugging: The authors used AI coding assistants interactively to support pipeline implementation, specifically to debug data ingestion routines, troubleshoot Slurm batch dispatch scripts, and diagnose edge cases in evaluation parsing code. All generated scripts were reviewed, tested, and audited against ground-truth outputs by the research team prior to benchmark execution.

Figure and Visual Asset Generation: We generated or edited stylized icons for system architecture overviews (e.g., Figure 2 and Figure 8) with generative AI assistance and integrated them into human-designed vector schematics.

Experimental Evaluation Artifacts: We clarify that the use of language models as research objects (e.g., the evaluated baseline models detailed

GPT-5.4-mini Across Conditions: Trading
Q: As of September 2025, which stocks have the strongest volume confirmation for their price moves?
Golden answer: “The stocks with the strongest volume confirmation are TSLA, INTC, and LRCX. [. . . ] TSLA exhibits the strongest absolute volume confirmation, with nearly double the OBV of the next closest ticker, validating its 35% upward move. INTC shows strong volume confirmation for a recovery rally, despite weak underlying fundamentals. LRCX represents the healthiest volume confirmation, where high OBV and price momentum are aligned with strong fundamental operating metrics.”
PER-CONDITION-EVIDENCE JUDGE
Indicator [5.00]: “The clearest positive confirmations are: TSLA: PxChg +0.350255, OBVchg +1.223B; INTC: PxChg +0.385791, OBVchg +642.828M; MU: PxChg +0.412219, OBVchg +306.601M [. . . ]”
Tool [4.75]: “Using the supplied evidence, the best proxy for that is On-Balance Volume (OBV) change over the month [. . . ] TSLA: OBV_WINDOW_CHANGE +1,223,051,798; price rose from 329.36 on 2025-09-02 to 444.72 on 2025-09-30; NVDA: +910,704,556; price rose from 170.78 to 186.58 [. . . ]”
Oracle [5.00]: “Based on the largest OBV increases from 2025-09-02 to 2025-09-30: TSLA – OBV_WINDOW_CHANGE: +1,223,051,798; NVDA – OBV_WINDOW_CHANGE: +910,704,556; INTC – OBV_WINDOW_CHANGE: +642,827,861 [. . . ]”
+ TimesFM2.5 [2.50]: “[T]he strongest volume-confirmed price moves for September 2025 would be the stocks showing both: Meaningful forecasted price drift [. . . ] Large forecast uncertainty bands / implied move ranges [. . . ]” (no volume data supplied; forecast uncertainty used as a proxy)
COMMON-EVIDENCE JUDGE
No-RAG [1.50]: “[. . . ] led by: NVIDIA (NVDA); Broadcom (AVGO); Meta Platforms (META); Microsoft (MSFT); Amazon (AMZN). If you want the strongest raw volume spikes in more speculative movers, then Super Micro Computer (SMCI), Palantir (PLTR), and AMD also stand out.” (generic picks with no data, several outside the question’s stock universe)
RAG [3.00]: “As of September 2025, the strongest volume-confirmed price moves in the provided data appear to be: INTC – the clearest confirmation, with a strong price rise and an extreme volume surge. MU – very strong upward move with heavy and rising turnover. LRCX – sizable price appreciation with robust volume support. [. . . ]” (plausible ranking, but its daily price/volume figures are absent from the judge’s canonical evidence and were treated as unsupported)

Table 26. GPT-5.4-mini responses to a Trading seed question under every evaluated condition. Notation as in Table 25.

Table 26: GPT-5.4-mini responses to a Trading seed question under every evaluated condition. Notation as in Table 25.

in Table 16, candidate generators in the calibration pipeline (§ 3), and Claude Sonnet 4.5 as an automated evaluation judge (§ 4, §E.7) constitute core scientific artifacts and methodology of this study rather than external authorial assistance. The automated judge configuration was systematically aligned with human expert rubrics and independently validated against blinded human evaluations (§F.1).

SystemStudyExact model identifierProvider / endpointInference configuration
DeepSeek-V4-Flash [25]General QADeepSeek-V4-FlashAzure AI Foundry, OpenAI-compatible v1; deployment ftb-deepseek-v4-flashMax output 1,200 tokens. The endpoint is invoked without custom temperature or top-p. Visible <think> blocks are removed before scoring. API timeout 900 s with up to six retries.
GPT-5.4-mini (OpenAI, 2026)General QAgpt-5.4-miniAzure AI Foundry, OpenAI-compatible v1; deployment ftb-gpt54miniMax completion 1,200 tokens; reasoning effort=medium. No custom temperature or top-p is supplied to the endpoint. API timeout 900 s with up to six retries.
Gemini-3.8-Flash [34]General QAgemini-3.8-flashGoogle Vertex AI v1 (google-genai); location globalMax output 1,200 tokens; thinking level=MEDIUM. Deprecated temperature/top-p controls are not supplied. API timeout 900 s with up to six retries.
Llama-4-Scout (Meta AI, 2025)General QA + ControlledLlama-4-Scout-17B-16E-InstructAzure AI Foundry, OpenAI-compatible v1; deployment ftb-llama4-scoutGeneral QA: max output 10,000 tokens; candidate generation uses temperature 0.1, top-p = 0.9; self-selection is deterministic (temperature 0, top-p = 1). Controlled conditions use a 10,000-token answer ceiling; Tool-Use router ceiling 1,024 tokens.
Qwen3.8-27B (Qwen Team, 2026)General QA + ControlledQwen3.8-27BLocal vLLM (Kwon et al., 2023), OpenAI-compatible loopback endpointGeneral QA: max output 1,200 tokens; candidate generation temperature 0.1, top-p = 0.9; deterministic self-selection; seed 42; thinking explicitly disabled. Controlled: max output 1,200 tokens; deterministic generation; router ceiling 120 tokens. Serving: BF16, max model length 32,768, GPU-memory utilization 0.90.
Ministral-3-8B (Liu et al., 2026)General QAMinistral-3-8B-Instruct-2512Local vLLM (Kwon et al., 2023), OpenAI-compatible loopback endpointMax output 1,200 tokens; candidate generation temperature 0.1, top-p = 0.9; deterministic self-selection; seed 42. Serving: BF16, max model length 32,768, GPU-memory utilization 0.90, Mistral tokenizer/config/load format.
Qwen2.5-32B (Qwen et al., 2024)ControlledQwen2.5-32B-InstructLocal Hugging Face / Transformers (Wolf et al., 2020)4-bit model loading; max output 1,200 tokens; deterministic greedy generation (do sample=False); seed 42; Tool-Use router ceiling 120 tokens.
R1-Distill-Qwen-32B [24]ControlledDeepSeek-R1-Distill-Qwen-32BLocal Hugging Face / Transformers (Wolf et al., 2020)4-bit model loading; max output 1,200 tokens; deterministic greedy generation (do sample=False); seed 42; Tool-Use router ceiling 120 tokens.
Llama-3.1-8B (Grattafiori et al., 2024)ControlledLlama-3.1-8B-InstructLocal Hugging Face / Transformers (Wolf et al., 2020)4-bit model loading; max output 1,200 tokens; deterministic greedy generation (do sample=False); seed 42; Tool-Use router ceiling 120 tokens.
Phi-4 [1]Controlledphi-4Local Hugging Face / Transformers (Wolf et al., 2020)4-bit model loading; max output 1,200 tokens; deterministic greedy generation (do sample=False); seed 42; Tool-Use router ceiling 120 tokens.
GPT-5.4-mini (OpenAI, 2026)Controlledgpt-5.4-miniAzure AI Foundry, OpenAI-compatible v1; deployment ftb-gpt54miniMax completion 3,000 tokens; reasoning_effort=none; no custom temperature/top-p; Tool-Use router ceiling 120 tokens.
ChatTS-8B (Xie et al., 2024b)Time seriesbytedance-research/ChatTS-8BLocal Hugging Face / Transformers (Wolf et al., 2020)FP16 CUDA inference; deterministic decoding (do.sample=False); 128 adjusted-close observations per series; minimum usable history 64 points; final-answer cap 2,000 tokens; chunk-analysis cap 600 tokens.
TimesFM 2.5 [22, 35] + GPT-5.4-mini (OpenAI, 2026)Time seriesTimesFM 2.5 + gpt-5.4-miniTimesFM via BigQuery AI.FORECAST; GPT via Azure AI Foundry, deployment ftb-gpt54miniTimesFM receives the latest 128 point-in-time daily adjusted-close observations and forecasts the next 5 trading steps. Forecast-derived features are passed to GPT-5.4-mini; GPT max completion 3,000 tokens; reasoning_effort=none.
Claude Sonnet 4.5 [5]Automated judgeclaude-sonnet-4-5@20250929Google Vertex AI; region us-east5Temperature 0; no extended thinking; structured output; max judge output 2,000 tokens; maximum evidence payload 70,000 characters. The benchmark reference answer is not supplied as factual ground truth.

Table 16. Model identifiers, serving backends, and inference configuration. General-QA models generate four No-RAG candidates and two RAG candidates before intra-model self-selection. Controlled Indicator-RAG, Tool-Use, and Oracle conditions use a single final generation; Tool-Use additionally invokes the deterministic router. Llama-4-Scout is an open-weight Meta model evaluated through Azure AI Foundry. TimesFM 2.5 is used only to generate forecast features and does not directly answer FinTradeBench questions. At the time of research, Gemini 3.8 Flash leads vals Finance Agent v2 with 61.44% accuracy [2]

Annotation Instruments

This section reproduces the instruments used in the four human-review stages of §G: the development-time annotation rubric (Table 28), the construction-time fatal-flaw screening guide (Table 29), the Core403 item-level held-out response-scoring rubric with its pairwise fields (Table 30), and the Core403 item-level audit rubric with its record fields (Table 31).

References

  1. [1]Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905.arxiv.org/abs/2412.08905
  2. [2]Vals AI. 2026. Finance agent v2 benchmark. https://www.vals.ai/benchmarks/fabv2. Accessed: 2026-09-20.
  3. [3]Torben G Andersen, Tim Bollerslev, Francis X Diebold, and Paul Labys. 2003. Modeling and forecasting realized volatility. Econometrica, 71(2):579–625.
  4. [4]Andrew Ang and Allan Timmermann. 2012. Regime changes and financial markets. Annu. Rev. Financ. Econ., 4(1):313–337.
  5. [5]Anthropic. 2025. Claude sonnet 4.5 system card. Anthropic system card.
  6. [7]Malcolm Baker and Jeffrey Wurgler. 2006. Investor sentiment and the cross-section of stock returns. The journal of Finance, 61(4):1645–1680.
  7. [8]Tim Bollerslev, Benjamin Hood, John Huss, and Lasse Heje Pedersen. 2018. Risk everywhere: Modeling and managing volatility. The Review of Financial Studies, 31(7):2729–2773.
  8. [9]Tim Bollerslev, Lai Xu, and Hao Zhou. 2015. Stock return and cash flow predictability: The role of volatility risk. Journal of Econometrics, 187(2):458–471.
  9. [10]William Brock, Josef Lakonishok, and Blake LeBaron. 1992. Simple technical trading rules and the stochastic properties of stock returns. The Journal of finance, 47(5):1731–1764.DOI
  10. [11]Leland Bybee, Bryan Kelly, and Yinan Su. 2023. Narrative asset pricing: Interpretable systematic risk factors from news text. The Review of Financial Studies, 36(12):4759–4787.
  11. [12]Ron Campos, Ashmal Vayani, Parth Parag Kulkarni, Rohit Gupta, Aritra Dutta, and Mubarak Shah. 2025. Gaea: A geolocation aware conversational model. arXiv e-prints, pages arXiv–2503.arxiv.org/abs/2503.16423
  12. [13]Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024a. Humans or llms as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327.
  13. [14]Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024b. Fintextqa: A dataset for long-form financial question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 6025–6047. Association for Computational Linguistics.
  14. [15]Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, and 1 others. 2021. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711.arxiv.org/abs/2109.00122
  15. [16]Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 conference on empirical methods in natural language processing, pages 6279–6292.arxiv.org/abs/2210.03849
  16. [17]Chanyeol Choi, Jihoon Kwon, Jaeseon Ha, Hojun Choi, Chaewoon Kim, Yongjae Lee, Jy-Yong Sohn, and Alejandro Lopez-Lira. 2025. Finder: Financial dataset for question answering and evaluating retrieval-augmented generation. In Proceedings of the 6th ACM International Conference on AI in Finance, pages 638–646.arxiv.org/abs/2504.15800
  17. [18]Jaehyung Choi. 2021. Maximum drawdown, recovery, and momentum. Journal of Risk and Financial Management, 14(11):542.
  18. [19]Yinlam Chow, Yu Li, Xuchao Han, Prateek Jain, Ruiqi Sun, Huazhe Xu, Jiawei Gao, Dong Zhou, and Bin Yu. 2025. Inference-aware fine-tuning for best-of-n sampling in large language models. In Proceedings of the 13th International Conference on Learning Representations (ICLR).arxiv.org/abs/2412.15287
  19. [20]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.arxiv.org/abs/2110.14168
  20. [21]Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. 2023. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688.arxiv.org/abs/2310.10688
  21. [22]Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. 2024. A decoder-only foundation model for time-series forecasting. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 10148–10167. PMLR.arxiv.org/abs/2310.10688
  22. [23]Werner FM De Bondt and Richard Thaler. 1985. Does the stock market overreact? The Journal of finance, 40(3):793–805.DOI
  23. [24]DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, and 1 others. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948.DOI
  24. [25]DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxyuan Wang, Bingzheng Xu, Bochao Wu, and 1 others. 2026. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348.arxiv.org/abs/2606.19348
  25. [27]Aritra Dutta, Srijan Das, Jacob Nielsen, Rajatsubhra Chakraborty, and Mubarak Shah. 2024. Multiview aerial visual recognition (mavrec): Can multi-view improve aerial visual perception? In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22678–22690.arxiv.org/abs/2312.04548
  26. [28]Robert Engle. 2004. Risk and volatility: Econometric models and financial practice. American economic review, 94(3):405–420.DOI
  27. [29]Eugene F. Fama and Kenneth R. French. 1992. The cross-section of expected stock returns. The Journal of Finance, 47(2):427–465.DOI
  28. [30]Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, and 1 others. 2024. Law-bench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 7933–7962.
  29. [31]Robert Friel, Masha Belyi, and Atindriyo Sanyal. 2024. Ragbench: Explainable benchmark for retrieval-augmented generation systems. arXiv preprint arXiv:2407.11005.arxiv.org/abs/2407.11005
  30. [32]Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, Haofen Wang, and 1 others. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1):32.arxiv.org/abs/2312.10997
  31. [33]Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Arno Jacobsen. 2013. Bigbench: Towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 ACM SIGMOD international conference on Management of data, pages 1197–1208.DOI
  32. [34]Google DeepMind. 2026. Gemini 3.8 Flash model card. Google DeepMind Model Card. Accessed: 2026-10-12.
  33. [35]Google Research. 2025. TimesFM 2.5 (200m, pytorch) model card. Hugging Face model card. Accessed: 2026-10-12.
  34. [36]Aaron Grattafori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and 1 others. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  35. [37]Robin Greenwood, Samuel G Hanson, and Gordon Y Liao. 2018. Asset price dynamics in partially segmented markets. The Review of Financial Studies, 31(9):3307–3343.DOI
  36. [38]Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. The Innovation.
  37. [39]Beining Han, Anqi Liu, Jing Chen, and William Knott-tenbelt. 2025. Can machine learning models better volatility forecasting? a combined method. The European Journal of Finance, pages 1–22.DOI
  38. [40]Campbell R. Harvey, Yan Liu, and Heqing Zhu. 2016. ...and the cross-section of expected returns. The Review of Financial Studies, 29(1):5–68.
  39. [41]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  40. [42]MD Kamrul Hossain, Rupeng Zhang, Haoran Hu, and Kelvin Lo. 2025. Llms as meta-reviewers’ assistants: Benchmarking reliability, calibration, and bias in automated paper evaluation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. To appear.

Paper details

vlnc:W_5f4mrgmn
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents