betaFeedback

WorksComputer Science

SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models

Abstract

System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.

Figure 1. Highlights of SO-Arena. SO-Arena evaluates System One models on capability and reliability. Compared with other benchmarks, it verifies reference labels, stratifies tasks by difficulty to discriminate between models, evaluates long-horizon agentic ability in sandboxes built for System One, and tests reliability under reordered options and a missing correct option.

Introduction

System Two reasoning has substantially advanced language model capabilities in mathematics, coding, and science [3, 23, 34, 46, 55, 62]. These gains often come at a higher inference cost, as models generate additional reasoning tokens before answering [12, 27, 29, 37, 60]. Yet many routine decisions in software workflows, such as routing requests, selecting tool actions, and verifying task completion, can be made without generating explicit reasoning traces. System One models [2, 28, 64, 67, 86, 86, 95] such as Jev [2] predict a distribution over predefined outcomes in a single forward pass, substantially reducing inference latency and providing structured outputs that downstream software can use directly.

Recent benchmarks evaluate System One models across a range of decision tasks and settings [1, 5, 7, 11, 81]. Building on these evaluations, we argue that a benchmark designed to characterize the capabilities and reliability of System One models should provide trustworthy reference labels supported by model-visible evidence, distinguish performance across increasing task complexity, evaluate models both as actors and verifiers in interactive settings, and assess decision reliability under controlled changes to the available options.

To address these requirements, we introduce SO-Arena, a benchmark comprising 2,919 tasks across eleven domains organized into five capability families: semantic understanding, knowledge and reasoning, safety and policy compliance, evidence assessment, and agent task completion. The benchmark brings together static and environment-grounded tasks under a unified decision interface, with performance evaluated at the level of the original task. We curate tasks from traceable sources, manually verify that reference labels are supported by the information available to the model, and keep evaluation references separate from model inputs. To characterize the capability spectrum beyond aggregate accuracy, we organize nine domains into difficulty tiers based on domain-specific criteria for task complexity. We report performance across these tiers and construct a capability leaderboard that weights all eleven domains equally while emphasizing the most demanding tasks. In agentic settings, we further distinguish between action selection and completion verification. The former evaluates whether models can accomplish goals through sequential decisions in web and tool environments, while the latter assesses whether they can correctly judge recorded executions, including cases where identical final goal states conceal different compliance with intermediate requirements. We also incorporate reasoning-intensive tasks to compare direct System One decisions with explicit System Two reasoning under matched task conditions. Beyond task performance, we examine decision reliability under changes to the available options. Specifically, we test whether models maintain consistent decisions when options are reordered and recognize the absence of a valid answer when the correct option is removed. These two aspects are combined into a separate reliability score, allowing us to assess whether strong capability performance is accompanied by reliable decision-making.

We evaluate 21 models on SO-Arena in terms of capability, inference latency, and decision reliability. The two System Two references lead the capability leaderboard, while GLiDE [28] and KnowLine-31B [67] achieve the highest scores among System One models, illustrating the remaining performance gap despite the latter’s substantial speed advantage. Reliability, however, does not follow the same ranking: JevAny-27B [86] achieves the highest reliability among the evaluated System One models despite ranking eleventh in capability. Beyond these aggregate results, we find that performance declines with increasing task difficulty, particularly in agentic settings, and that the benefits of stronger backbones and additional reasoning vary substantially across domains.

These findings demonstrate the importance of characterizing the capability spectrum and decision reliability of System One models as complementary dimensions.

Our main contributions are summarized as follows:

  • We introduce SO-Arena, comprising 2,919 tasks across 11 domains and five capability families. We establish a traceable evaluation process and manually verify that reference labels are supported by the information available to the model.

  • We introduce domain-specific difficulty tiers to distinguish model capabilities and extend evaluation beyond static judgments to interactive execution and history-dependent task verification.

  • We develop a separate reliability assessment combining decision consistency under option reordering and the ability to recognize missing correct answers, revealing aspects of decision-making overlooked by conventional accuracy metrics.

  • We evaluate 21 models to reveal substantial capability differences across domains and difficulty levels, uneven benefits from model capacity and additional reasoning, and a notable divergence between capability and reliability rankings.

Preliminaries

We use System One decision models to denote models that map a supplied state and a decision specification to structured predictions over admissible outcomes. Following existing decision-task formulations [2, 26], we represent an instance as

x=(s,q,C),(1)x = (s, q, C), \tag*{(1)}

where ss contains the supplied information, qq specifies the decision objective, and C={c1,…,cK}C = \{c_1, \ldots, c_K\} defines the admissible outcomes and their descriptions. The supplied state may contain text, structured records, retrieved passages, or interaction histories. A decision model returns a distribution pθ(⋅∣x)p_\theta(\cdot\mid x) satisfying

pθ(ck∣x)≥0,∑k=1Kpθ(ck∣x)=1.(2)p_\theta(c_k \mid x) \ge0, \qquad\sum_{k=1}^{K} p_\theta(c_k \mid x) = 1. \tag*{(2)}

This distribution can be obtained through a learned decision head or a readout from language-model logits, among other implementations [32, 65]. Across these implementations, the decision type specifies the structure of the outcome space over which the model assigns probabilities. We consider three output formats supported by the Jev-style interface [83]:

  • Categorical selection (choice). The outcomes are explicitly specified alternatives without an imposed ordinal structure. The predicted outcome is

y^(x)=arg⁡max⁡c∈C pθ(c∣x).(3)\hat{y}(x) = \underset{c \in C}{\arg\max}\ p_\theta(c \mid x). \tag*{(3)}
  • Binary judgment (noul, the interface’s yes/no question type). The outcome space is {0,1}\{0,1\}, and the model returns pθ(1∣x)p_\theta(1 \mid x), interpreted as the probability assigned to the affirmative outcome.

  • Ordinal assessment (score). The outcomes are ordered descriptive levels with grades g(ck)g(c_k) on the source’s native scale. The predicted grade is that of the most probable level, g^(x)=g(y^(x))\hat{g}(x) = g(\hat{y}(x)). An ordinal ordering alone does not establish equal semantic distances between levels, so we compare grades only through a bounded credit for adjacent levels.

These formats describe the output space. They do not determine the task domain or the reasoning required to answer a question.

Benchmark Design

This section describes how we construct and evaluate our benchmark for System One decision-making. We first collect and derive tasks from public sources and control their quality (§ 3.1). We then organize these tasks by domain and, where applicable, difficulty tier, and formulate static and environment-based tasks using a common decision interface (§ 3.2). Given these task formulations, we define metrics for capability across domains and decision reliability (§ 3.3). Figure 3 summarizes the pipeline.

Figure 3. Overview of SO-Arena. (1) Tasks are drawn or derived from 40 established public sources; items with questionable labels are dropped after human and LLM review. (2) The 2,919 tasks span 11 domains in five capability families, nine of which have domain-relative difficulty tiers, and are posed as static or environment-based tasks. (3) The capability score CC averages the hardest tier of each domain with equal domain weights, and the reliability score RR combines the option-reordering and missing-correct-option tests.

Data Sources and Quality Control

We draw on 40 distinct public datasets, benchmarks, and environments across 11 domains. Representative sources include GoEmotions [24] for emotion understanding, MATH500 [54] for mathematics, GPQA Diamond [73] for scientific reasoning, Mind2Web [25] for web interaction, and BFCL [66] and ToolSandbox [57] for tool use. We convert the source data to our decision formats or derive new tasks from them, such as completion checks on BrowseComp-Plus [13] search runs and VirtualHome [71] trajectories. For quality control, we keep source labels as references, review them manually and with LLMs, and exclude items with questionable references. We record each task’s source, version, and original identifier where applicable, and flag model-derived references in the metadata.

Task Organization, Formulation, and Evaluation Settings

We group the 11 domains into five capability families covering semantic understanding, knowledge and reasoning, safety and policy compliance, evidence assessment, and agent task completion. Of the 11 domains, nine have difficulty tiers. The remaining two, Sentiment & Emotion Understanding and Knowledge & Commonsense, are untiered because their tasks offer limited scope for systematically varying decision complexity. In the tiered domains, difficulty is relative to each domain and reflects the demands of its tasks. For example, Math uses source difficulty levels or the number of calculation steps, depending on the source. Task Completion Verification distinguishes direct state checks, judgments that require integrating evidence, and judgments that depend on the execution history. Figure 4(b) shows the distribution of tasks across tiers, and Appendix § 7 specifies the criteria for each tiered domain.

Figure 4. Composition of the benchmark. (a) Natural tasks by capability family (inner ring) and domain (outer ring). (b) Share of each domain’s tasks in each difficulty tier, with task counts on the bars; side rules mark the families in (a). Tiers are defined within each domain and are not calibrated across domains.

To evaluate these capabilities in settings that reflect real-world applications, we consider how the context of a task shapes the model’s decision. Some decisions can be made solely from a fixed input, while others are made from the current state of an environment, such as a website or a sandbox with tools; in sequential tasks, each decision changes the state for the next, so later decisions depend on earlier ones being correct. We therefore organize the benchmark into two task settings, static and environment-based.

Static Tasks. In static tasks, the model judges a fixed input. We convert each source task into one of the three formats and keep the source’s answer as the reference. For choice, multiple-choice questions are used as they are, and classification tasks offer the source’s labels as options. Open-answer math problems have no options, so we build distractors from wrong answers of a base model, intermediate results of the solution, and rule-based near misses. For noul, binary judgments are posed as yes/no questions, and a multi-label task becomes one yes/no question per candidate label. For score, ordinal ratings keep the source’s native scale. For tasks we build ourselves, we establish the reference by execution, verified construction, or annotation (Appendix § 7). We exclude tasks that require free-text responses.

Environment-Based Tasks. In environment-based tasks, the model assesses or selects actions given a task goal and an observation from a web or tool environment. Single-step web tasks show one recorded page state, on which the model judges whether a proposed action advances the goal or selects the best candidate action. Sequential tasks require a series of decisions, each a choice from a menu of candidate actions. In multi-step web tasks, the model acts at successive recorded states along a reference route that replay has verified to complete the task.3 In interactive tool-use episodes, the BFCL [66] or ToolSandbox [57] simulator executes each selected action and returns the observation on which the next decision is made. We score each episode based on the corresponding benchmark’s success criteria (Appendix § 7.§ 6).

The resulting benchmark comprises 2,300 static tasks and 619 environment-based tasks. The latter include 337 web tasks (100 single-step judgments and 237 reference-route tasks) and 282 interactive tool-use episodes. Figure 4(a) shows the distribution of tasks across capability families and domains. Table 3 summarizes the sources, task counts, difficulty tiers, and decision formats for each domain.

DomainTasksTiersFormatSources
Semantic understanding
Sentiment & Emotion Understanding200–choice, scoreAmazon Reviews 2023 [42], GoEmotions [24], SST-5 [74], DynaSent [70], ARTS [88]
Intent & Routing300100/100/100choice, noulCLINC150 [48], BANKING77 [8], SGD [72] → NatCS [35] → NLU++ [9]
Knowledge and reasoning
Math20050/50/70choiceGSM8K-Platinum [18, 84], MATH500 [40, 54], AIME 2025–2026 [4]
Code Understanding20060/70/70noulCodeJudgeBench [44], LiveCodeBench [43], LeetCodeDataset [87]
Scientific Reasoning20050/70/80choiceARC-Challenge [15] → MMLU-Redux [31] → GPQA Diamond [73]
Knowledge & Commonsense200–choiceMMLU-Redux [31, 39], HellaSwag [92], CommonsenseQA [79]
Safety and policy compliance
Safety500177/165/158choice, noulRAGTruth [59], ReasoningShield [49], HarmThoughts [45], DynaBench [41], Aegis 2.0 [33], OR-Bench [22]
Evidence assessment
Retrieval & Reranking300100/100/100noul, scoreTREC DL [20, 21] → BRIGHT [77], Touché 2020 [6, 82] → ACORD [85]
Task Completion Verification20091/89/20choice, noulBrowseComp-Plus [13], CALVIN [58], τ\tau-bench [89], MiniGrid [14], EAI-VirtualHome [50, 71]
Agent task completion
Tool & API Use282100/78/104live seq.BFCL [66] → ToolSandbox [56] → BFCL
Text-based Web Interaction337100/137/100choice, noul; seq.Mind2Web [25] → BreakingWeb [91]

Table 3. Composition of SO-Arena. Counts are natural tasks. Tiers give easy/medium/hard counts; Math additionally has 30 very hard AIME problems, and “–” marks domains without tiers. Where tiers draw on different sources, sources are listed from easy to hard, separated by →. “Seq.” denotes a sequence of choice requests.

Evaluation Metrics

Capability across domains.

We use the original task as the unit of scoring and weight domains equally so that larger domains do not dominate the leaderboard. Let ci(m)∈[0,1]c_i(m) \in[0, 1] denote model mm’s credit on task ii. For domain dd, let TdT_d contain the hardest tier, both the hard and very hard tiers for Math, or all tasks for untiered domains. Focusing on the upper tiers makes the leaderboard more sensitive to performance differences on demanding tasks. The capability score is

C(m)=100D∑d=1D1∣Td∣∑i∈Tdci(m),(4)C(m) = \frac{100}{D}\sum_{d=1}^{D}\frac{1}{\lvert\mathcal{T}_d\rvert}\sum_{i\in\mathcal{T}_d}c_i(m), \tag*{(4)}

where DD is the number of domains.

Reliability

Option order stability. A reliable model should pick the same option however the options are ordered. We show each item ii in its original order and in 12 reorderings, and identify the model’s answer by the option’s content, not its position. A(m)A(m) is the share of reorderings in which model mm keeps its original answer; S(m)S(m) rescales it so that random picks score 0 and full agreement scores 1:

S(m)=A(m)−pe1−pe,A(m)=112M∑i=1M∑r=1121{ai(r)=ai(0)},pe=1M∑i=1M1Ki,(5)S(m) = \frac{A(m)-p_e}{1-p_e},\qquad A(m) = \frac{1}{12M}\sum_{i=1}^{M}\sum_{r=1}^{12}\mathbf{1}\{a_i^{(r)}=a_i^{(0)}\},\qquad p_e = \frac{1}{M}\sum_{i=1}^{M}\frac{1}{K_i}, \tag*{(5)}

where ai(0)a_i^{(0)} and ai(r)a_i^{(r)} are the answers in the original order and in reordering rr, MM is the number of items, and pep_e is the agreement of random picks among item ii’s KiK_i options. SS measures consistency, not correctness.

Recognition of missing correct options. A model should assign a higher probability to the absence of a correct answer (Null) when none of the offered options is correct. We define this as pnull=1−p(inc)p_{\mathrm{null}} = 1-p(\mathrm{inc}), where p(inc)p(\mathrm{inc}) is the model’s estimated probability that at least one correct option is included (Appendix § 8). We show each item twice, as it is and with its correct option removed, and use pnullp_{\mathrm{null}} to predict which copies lack a correct option4. Null discrimination is

N(m)=2AUC⁡(m)−1,(6)N(m) = 2\operatorname{AUC}(m)-1, \tag*{(6)}

where AUC⁡(m)\operatorname{AUC}(m) is the area under the ROC curve of this prediction over all copies: the probability that a random copy without a correct option receives a higher pnullp_{\mathrm{null}} than a random copy with one, with ties counted as half. NN is 0 at chance and 1 when every copy without a correct option scores higher.

Combined reliability. Reliable decisions should be stable under reordering and sensitive to whether a correct option exists. We combine the two components using their geometric mean, with a positive score only when both exceed chance:

R(m)={S(m)N(m),S(m)>0 and N(m)>0,0,otherwise.(7)R(m) = \begin{cases} \sqrt{S(m)N(m)}, & S(m)>0\ \text{and}\ N(m)>0,\\ 0, & \text{otherwise}. \end{cases} \tag*{(7)}

We report the reliability score R(m)R(m) as a percentage alongside the capability score C(m)C(m) in Figure 2.

Figure 2. SO-Arena at a glance. (a) Leaderboard scores for 21 models; greater depth indicates a higher score, and models are grouped horizontally by size and access. (b) Performance (the leaderboard score) versus stability under option reordering (100S) for the 19 models with stability measurements; the red line marks the Pareto frontier.

Experiments

System One models decide much faster than System Two models, but it remains unclear whether, and in which domains, they lose performance. Meanwhile, their speed suits high-volume, repeated daily tasks such as classification, routing, and real-time responses, where decisions must also be consistent. Our experiments therefore measure how well and how fast current System One models decide and how reliable their decisions are, across 11 major domains. § 4.1 reports the main results; § 4.2 examines where System One models match a System Two reference, what narrows the gap where they do not, and which factors shape reliability.

Main Results

We evaluate 21 models shown in Table 1: 16 open-weight System One models from 0.4B to 31B, one model that switches automatically between System One and Two (StartLux Auto), two closed decision APIs (Jev API 1.14 and OpenAI Decisions), and two closed System Two references (GPT-6 Luna and Sol). Each model runs in the settings documented by its developer5 (Appendix § 8). We evaluate them along two dimensions. Performance is paired with latency, since speed is what System One models are built for (§ 4.1.1). Reliability asks whether decisions hold when the options are reordered or the correct option is absent (§ 4.1.2).

Table 1: The 21 evaluated models. Open weights: ✓ available, ✗ closed API. Reasoning: whether the model reasons before answering; auto lets the model decide for each request. Releases and the settings of our latency and reliability runs are listed in Table 4.

ModelDeveloperSizeOpen weightsReasoningModelDeveloperSizeOpen weightsReasoning
GLiDE [28]Fastino27B✓NoDecider Flash [69]Perplexity4B✓No
KnowLine-31B [67]PelaAI31B✓NoJevAny-4B [86]SimpleJev4B✓No
Sieve-31B-v3 [76]Sthanika AI31B✓NoSII-4B [26]SII4B✓No
Decider-27B [68]Perplexity27B✓NoKev-4B [65]J. Palmer4B✓No
Kev-27B [64]J. Palmer27B✓NoJeva-4B [51]NUS4B✓No
JevAny-27B [86]SimpleJev27B✓NoLaya [19]ConvAI Innovations0.4B✓No
StartLux Auto [95]StartLux4B✓AutoGPT-6 Sol [63]OpenAI–✗Medium effort
StartLux Instant [95]StartLux4B✓NoGPT-6 Luna [63]OpenAI–✗Medium effort
Clef-27B [16]Cloudflare27B✓NoJev API 1.14 [2]TypeSafe–✗Not disclosed
H2O Lightning [38]H2O.ai4B✓NoOpenAI Decisions [61]OpenAI–✗Not disclosed
Clef-Flash-9B [17]Cloudflare9B✓No

Table 1.

Performance and Latency

As shown in Table 2 and Figure 5, System Two models still lead in overall performance: GPT-6 Sol and Luna outperform every System One model, and the strongest System One models, GLiDE and KnowLine-31B, trail Luna by about ten points. § 4.2.1 shows that this lead does not hold in every domain. In exchange, System One models are much faster. Every open-weight System One model answers within 0.3 seconds per task on one H100 GPU, tens of times faster than GPT-6 Sol and Luna (Figure 5).

ModelScoreSent. AllIntent EIntent MIntent HMath EMath MMath HMath VCode ECode MCode HScience EScience MScience HKnow. AllSafety ESafety MSafety HRetrieval ERetrieval MRetrieval HVerification EVerification MVerification HTools ETools MTools HWeb EWeb MWeb H
Closed-source APIs
GPT-6 Sol82.38810078821001001001009896939896919484827599916210099100998866987653
GPT-6 Luna79.785997781100100100979891869896889082807098916210098100968668997748
Jev API 1.1460.48699785610084633083705996947192837369998360888820977559997341
OpenAI Decisions53.6749977587446343088716694814686716763988455797540866138986831
Open-weight models: 31B
KnowLine-31B70.288100837898987140927967989452938683779989649796909784561007444
Sieve-31B-v367.78899816810094664793776698935593868278988464979870968660996043
Open-weight models: 27B
GLiDE70.2901008476988664309369579687559587847599845710099100978165998149
Decider-27B66.78998837096906133877361100905295868175998856999765978467997950
Kev-27B61.484978268100785630877154988754927973649887598989459781621007845
JevAny-27B58.484988152100825327877350100876094817565927057939035988458997141
Clef-27B56.9859779649674471792706098915194787559978861888315928655987343
Open-weight models: 9B
Clef-Flash-9B51.6839942649468511770594198944896706646998358765835915032905525
Open-weight models: 4B
StartLux Instant59.0889471709888434080695696844688827363998254868055967954983931
StartLux Auto58.6849671709890814780635490865185746662988256887635927646925031
H2O Lightning54.9859466639872444370544692804491826864988761858040966638943628
Decider Flash52.0889875599676402077645392834091796561998056888315956841945634
Jeva-4B51.6818761628462532382645082736076686451968564857945733217856118
JevAny-4B50.9839767619260392073664796844490696349917252716345937241944819
SII-4B48.0789164406242392770595190814584716045958454827160844320914915
Kev-4B46.1749264509058332067665688744182565832938054767250814622833916
Open-weight models: 0.4B
Laya30.9658273042422320585151262924434939358662385244301104462

Table 2. Scores by domain and difficulty tier. E, M, H, and V denote the easy, medium, hard, and very hard tiers; All marks untiered domains. Score is the leaderboard score (Equation (4)). Rows are grouped by access and size and sorted by Score within each group; the two System Two references are shaded orange. Darker is higher; bold is the best in each column. Tiers are not calibrated across domains.

Figure 5. Performance against latency. Performance is given by the leaderboard score (Equation (4)). The Pareto frontier is formed by the System One models with optimal performance at each latency. Open-weight models run on one H100 GPU each, one task at a time. Closed APIs are timed end to end over the network (Appendix § 9).

Reliability

We measure reliability R (Equation (7)) and its components, order stability S (Equation (5)) and null discrimination N (Equation (6)), for the 18 System One models (Figure 6). While performance and reliability are positively correlated sometimes, higher performance does not always imply higher reliability: the two most reliable models, JevAny-27B and Jev API 1.14, rank only eighth and sixth of the 18 in performance (Table 2).

Figure 6. Reliability and its two components for the 18 System One models. (a) Reliability RR (Equation (7)), with each model’s performance (Perf.), its leaderboard score (Equation (4)), to the left of its bar. (b) NN against SS; 0 is chance level for both, and dashed curves mark constant RR.

Option order stability. Most models keep their answer when only the order of the options changes: 14 of the 18 reach S>0.8S > 0.8 (Figure 6b). Apart from Jeva-4B, at S=1S = 1 by construction (Appendix § 8), these 14 lie between 0.83 and 0.94, while the remaining four, Laya (0.35), SII-4B, Kev-4B, and OpenAI Decisions (0.68 to 0.72), are sensitive to the order.

Null discrimination. Null discrimination, in contrast, varies widely, from near chance to about 0.8 (Figure 6b). Size alone does not decide it: within each family released in two sizes the larger model does score higher on null discrimination, yet some smaller 4B models exceed some larger 9B and 27B models.

Discussion of Results

The main results raise two questions. Since System One models trail the System Two references on general performance, Section 4.2.1 asks in which domains the gap lies and whether a more capable backbone or reasoning closes it. Section 4.2.2 asks how far performance explains each reliability component.

Where System One Models Match System Two, and What Closes the Gap

The gap to System Two models across domains. How close a System One model’s performance comes to a System Two model depends on the domain. We compare GPT-6 Luna with the best-performing open-weight System One models of two sizes: StartLux Instant (4B) and GLiDE (27B) (Figure 7a). By their nature and by how the models perform on them, the domains fall into three categories: direct semantic judgment, where System One models perform comparably to System Two or even better; fine-grained grounded decision, where only the System One model with a larger, more capable backbone comes close to System Two; and multi-step derivation, where both System One models fall far behind.

Figure 7. The gap to a System Two model, and what narrows it, differ by domain. (a) Score difference of two System One models from GPT-6 Luna on the medium and hard tiers, with 95% confidence intervals (Appendix § 10). (b) Gain from the smaller to the 27B model of each family. (c) Gain of StartLux Auto from always thinking over never thinking. Rows are grouped by the three categories of § 4.2.1.

To see what drives the gap in each category, we measure two gains (Figure 7b and c). The gain from backbone capability compares the smaller and the larger model of each family released in both sizes: every smaller model builds on Qwen3.5, every larger model on the newer Qwen3.8-27B. The gain from thinking compares StartLux Auto with thinking always on against never on (Appendix § 10).

Direct semantic judgment. In Knowledge & Commonsense, Sentiment & Emotion Understanding, and Retrieval & Reranking, System One models perform comparably to Luna (Figure 7a). Neither a more capable backbone nor thinking adds much (Figure 7b and c).

Fine-grained grounded decision. In Intent & Routing, Safety, Task Completion Verification, Tool & API Use, and Web Interaction, GLiDE comes within 4 points of Luna, while StartLux Instant trails by up to 29 (Figure 7a). The model with larger and newer backbone of every family scores higher in all five domains, by far more than thinking adds (Figure 7b and c).

Multi-step derivation. In Math, Scientific Reasoning, and Code Understanding, both trail Luna significantly (Figure 7a). Thinking helps most in Math and Scientific Reasoning (Figure 7c): in Math it lifts StartLux Auto from 50.8 to 92.5, close to Luna, in line with the finding that chain-of-thought helps mainly on math and symbolic reasoning [75].

Factors Behind Reliability

Option order stability. Order stability largely follows performance (Figure 8a): it rises with performance, and most models lie close to the trend. The flip rates further demonstrate this: the model changes its answer much more frequently on the reorderings of questions it answers wrongly, against those it answers correctly (Figure 8b). Instability thus arises mostly where a model performs bad.

Figure 8. Order stability rises with performance; null discrimination follows it only loosely. In (a) and (c), each point is one of the 18 System One models, the line is a quadratic trend, and the band holds 90% of the fitted models (Appendix D). (a) Order stability SS (Equation (5)) against performance. (b) Share of option reorderings that change the answer, for originally right and wrong answers: median and interquartile range over the models. (c) Null discrimination NN (Equation (6)) against performance.

Null discrimination. Null discrimination (NN) follows performance only loosely (Figure 8c): around the same performance, it varies far more than option order stability. Kev-27B, Jev API 1.14, and JevAny-27B each exceed the four best-performing System One models in NN (Appendix § 10). A performance score therefore says much about a model’s order stability but little about whether it notices a missing answer, which has to be measured directly.

Related Work

System One Models and Decision Benchmarks. System One models expose predictions over explicitly specified outcomes, enabling classification, routing, scoring, and verification without open-ended text generation. Jev introduced this interface as a general-purpose service [2], while open releases such as Kev and JevAny provide language-model-based implementations [65, 86]. Recent research investigates how these capabilities can be obtained from pretrained models: SII-Decision distills reasoning-informed predictions into a single-pass readout, whereas Releasing the Inner Jev studies answer-token readout and supervised adaptation [26, 32]. Evaluation efforts already extend beyond accuracy. JevBench measures capability, calibration, latency, and cost [5]; JEVal covers ten application domains and examines how local decisions compose into agent workflows and social simulations [26]. OpenDecisions additionally includes difficulty tiers and controlled perturbations [32]. Our work complements these efforts with within-domain difficulty tiers, controlled history-dependent verification, and a separate reliability score within a broader evaluation of decision competence.

Generalist Capability Evaluation. MMLU measures knowledge and problem solving across academic and professional subjects [39], while BIG-Bench Hard probes challenging tasks and the effects of explicit reasoning [78]. Humanity’s Last Exam extends evaluation toward expert-level academic questions [10]. These benchmarks establish useful measures of capability breadth and demanding problem solving. HELM further emphasizes standardized evaluation across scenarios and multiple desiderata, including calibration and efficiency [53]. SO-Arena adopts a complementary focus: assessing generalist competence through decisions over supplied evidence and specified alternatives, while also examining whether judgments remain reliable as their informational and temporal requirements change. Academic reasoning is therefore one component of our evaluation, alongside semantic, policy, evidence, and environment-dependent decisions.

Judges, Reward Models, and Verification Reliability. Research on LLM-as-a-judge establishes the usefulness of automated evaluation while documenting position, verbosity, and self-enhancement biases [93]. RewardBench evaluates preferences across chat, safety, and reasoning, whereas JudgeBench constructs response comparisons grounded in factual and logical correctness [47, 80]. VerifyBench examines agreement between candidate responses and reference answers across scientific domains, including sensitivity to response presentation [52]. These studies motivate scrutinizing the evaluator itself. Separately, calibration research distinguishes predictive accuracy from the quality of confidence estimates, and selective classification studies the trade-off between error and coverage [30, 36]. We connect these perspectives to general-purpose decision evaluation, encompassing judgments whose evidence consists of documents, structured states, or interaction histories rather than only candidate answer pairs.

Evaluation in Agentic Environments. Mind2Web evaluates action prediction across tasks, websites, and domains [25]. WebArena provides reproducible web environments and functional evaluation using answers and programmatic state checks [94]; τ\tau-bench evaluates policy-constrained tool interaction using database outcomes and repeated-trial reliability [89]. More recently, VerifierBench evaluates the generation of executable checkers against valid solutions, ordinary errors, and adversarial cases [90]. These works address complementary evaluation targets: executing tasks, maintaining reliable behavior, or constructing verification programs. We evaluate decision models in two of these roles: as the policy that chooses each action of a tool or web episode from a menu of public options, and as the verifier that judges recorded executions, including trajectories whose final goal projections agree while their histories differ (Appendix A.4).

Conclusion

We introduce SO-Arena, a benchmark assessing generalist competence and decision reliability in System One models across eleven domains. It combines difficulty-stratified tasks, action selection and completion verification, and controlled option-set interventions to evaluate performance beyond aggregate accuracy. Our evaluation of 21 models reveals substantial capability gaps on demanding tasks, especially in agentic settings, despite the latency advantages of System One models. Stronger backbones and additional reasoning offer domain-dependent gains rather than universal improvements. More importantly, strong competence does not guarantee reliable judgment: option-order stability largely tracks performance, while recognition of missing correct options varies considerably among similarly capable models. These findings highlight the need to advance both capability and reliability in future System One models.

Benchmark Construction Details

This appendix complements § 3 with per-domain construction details. Every item records its source, pinned version, original split, and original identifier; these records, together with the reference of each item, are kept separate from the model inputs.

Semantic Understanding

Sentiment & Emotion Understanding (200). We construct 200 items from five public datasets. From the All Beauty category of Amazon Reviews 2023 [42], we sample 50 reviews, with 10 for each recorded star rating. We sample 40 single-label test comments from GoEmotions [24], with four per target emotion across ten emotions. For sentiment classification, we sample 30 SST-5 [74] sentences (six per class) and 50 DynaSent [70] test sentences (18 negative, 18 positive, and 14 neutral). We also construct 30 aspect-level sentiment items from 10 restaurant sentences in ARTS [88]. The items use two decision formats. In ordinal score items, the model reads a review title and text and predicts the reviewer’s rating on the original 1–5 star scale. In single-label classification items, the model selects one label from a shuffled list of options. GoEmotions items use the full set of 28 labels, including labels outside the ten emotions used for sampling. We score predictions against the recorded star rating or the source label, depending on the format. The resulting scores reflect agreement with the reviewer or annotator, since a review may leave the exact star rating ambiguous and a comment may admit more than one plausible emotion label.

Intent & Routing (300). We construct 300 items, with 100 per difficulty tier. For the easy tier, we sample 42 single requests from CLINC150 [48], 40 from BANKING77 [8], and 18 from SGD [72]. The BANKING77 sample contains 29 in-scope and 11 out-of-scope requests relative to a 50-category service catalog. For the medium tier, we select 100 complete NatCS [35] customer-service conversations (36 insurance, 30 banking, and 34 finance). For the hard tier, we select 100 NLU++ [9] utterances (73 banking and 27 hotels). We rank medium and hard candidates using a deterministic hash and screen them using only the source data. The audits set aside 30 medium and 20 hard candidates because of label uncertainty or poor task fit. The tiers differ in what the model must identify. Easy items ask the model to judge the intent of a single request. Medium items require tracking exactly two customer goals across a full conversation. The goals may appear far apart, and the model must distinguish them from goals suggested only by the agent. Hard items require identifying which components of a larger domain ontology apply to an utterance. The easy tier uses 47 four-way choice items and 53 binary noul items. Each choice item presents the native intent and three neighboring native classes, while each noul item asks whether one candidate intent applies. In the medium tier, we pose one noul question per native goal label, yielding 22–39 questions per conversation. In the hard tier, we pose one noul question per ontology component, yielding 48 questions for banking and 40 for hotels. We use the native component definitions verbatim. For medium and hard items, we form the predicted label set from all labels whose affirmative probability exceeds 0.5. We score each conversation or utterance using the Jaccard index between this set and the source-annotated label set.

Knowledge and Reasoning

Math (200). We sample 50 problems from GSM8K-Platinum [18, 84], 120 from MATH500 [40, 54], and 30 from AIME 2025 and 2026 [4]. We restrict the MATH500 sample to problems with numeric answers and no figures. We assign difficulty using calculation-step counts for GSM8K-Platinum and source levels for MATH500. The easy tier combines 30 GSM8K-Platinum problems requiring at most three steps and 20 MATH500 Level 1 problems. The medium tier combines 20 GSM8K-Platinum problems requiring four or more steps and 30 MATH500 Levels 2–3 problems. The hard tier contains 70 MATH500 Levels 4–5 problems, and the very hard tier contains 13 AIME 2025 and 17 AIME 2026 problems. We convert each problem into a multiple-choice item. Of the 600 distractors, 350 are incorrect answers obtained by sampling Qwen3.5-4B-Base 16 times per problem without reasoning, 22 are intermediate results from GSM8K solutions, and 228 are rule-based near misses. We sort options by numeric value, balance the position of the correct answer, and exclude items with duplicate options, doubtful labels, or a correct answer that can be identified from the options alone. The correct numeric answer serves as the scoring reference.

Code Understanding (200). We take 156 solutions written by seven LLMs from CodeJudgeBench [44] and sample 44 additional solutions from Qwen3.5-4B-Base for 2024–2025 LeetCodeDataset [87] Easy problems outside CodeJudgeBench. We compute a size score as the mean of the standardized log line count and standardized log branch and loop count. For correct solutions we add the LeetCode difficulty, encoded as −1, 0, or 1. For incorrect solutions we add 1.5 times the standardized fraction of tests passed. Within each label, the bottom quartile forms the easy pool, the second quartile the medium pool, and the top quartile the hard pool. An integer program selects 60, 70, and 70 items from these pools, with equal numbers of correct and incorrect solutions per tier and one item per problem family. It matches the labels within each tier by LeetCode difficulty, generator, and code length. Each item presents a LeetCode problem statement and a Python solution with comments and docstrings removed, and asks whether the solution returns the correct answer for every valid input. We establish the reference label by re-executing each solution under its original import header on the public and private LiveCodeBench [43] tests in a network-isolated sandbox. For solutions labeled correct, we additionally require either a pass on the corresponding LeetCodeDataset tests or a recorded manual check, and compare each solution with an independent brute-force implementation on thousands of valid random inputs. For solutions labeled incorrect, we require a wrong answer or runtime error on a problem example or a test whose expected output is independently confirmed. We exclude solutions whose only failures arise from machine-dependent time, memory, or recursion limits.

Scientific Reasoning (200). We sample 50 four-option questions from the ARC-Challenge [15] test set, 70 natural-science questions marked error-free in MMLU-Redux [31], and 80 GPQA Diamond [73] questions (36 chemistry, 35 physics, and 9 biology). We exclude items with duplicate option texts and those flagged as incorrect or ambiguous in an earlier internal label audit. We assign difficulty by source, with ARC-Challenge in the easy tier, MMLU-Redux in the medium tier, and GPQA Diamond in the hard tier. Each item asks the model to select one answer from the available options. We shuffle the GPQA Diamond options with a fixed seed, balance the correct-answer position within each source, and use the audited answer key as the scoring reference.

Knowledge & Commonsense (200). We sample 120 MMLU-Redux [31] questions marked error-free across 36 non-STEM subjects, with at most six per subject, along with 40 HellaSwag [92] and 40 CommonsenseQA [79] validation items. Within the MMLU-Redux material, we reserve natural-science subjects for Scientific Reasoning and exclude mathematics, computer science, and logic from both domains. We also exclude items with duplicate option texts and those flagged as incorrect or ambiguous in an earlier internal label audit. We leave this domain untiered because its tasks offer limited scope for systematically varying decision complexity. Each item asks the model to select one answer from the available options. For CommonsenseQA, we randomly remove one distractor to leave four options. We balance the correct-answer position within each source and use the audited answer key as the scoring reference.

Safety and Policy Compliance

Safety (500). We construct 500 items from six sources, drawing 150 from RAGTruth [59], 75 from ReasoningShield [49], 50 from HarmThoughts [45], 100 from DynaBench [41], 100 from Aegis 2.0 [33], and 25 from the hard-benign subset of OR-Bench [22]. During curation, we assign 177 easy, 165 medium, and 158 hard items based on the reasoning each item requires, without using model predictions or independent calibration. For example, a refusal decision can involve a direct request or one whose intent is disguised by context, while a policy judgment can require a direct rule match or depend on multi-hop references and interacting exceptions. The decision format varies by source. In RAGTruth, the model either judges whether a response contains a hallucination relative to the supplied material (90 items) or identifies which segment, if any, contains the sole hallucinated span (60). ReasoningShield items ask the model to classify a reasoning trace as harmful or safe (45) or assign one of three risk levels (30), using human majority labels as references. HarmThoughts items ask the model to classify a target step into one of 16 behavior categories. DynaBench items ask either whether a dialogue complies with a given policy (50) or which rule, if any, it violates (50). Aegis items assess prompt safety (50) or response safety (50), and OR-Bench items test over-refusal on hard benign prompts. We score each decision against its reference label. Of the 500 references, 100 are model-derived, comprising 50 original HarmThoughts LLM annotations, 25 Aegis response labels from an LLM jury, and 25 OR-Bench labels based on hard-benign subset membership.

Evidence Assessment

Retrieval & Reranking (300). We construct 300 pairs from four sources. From TREC Deep Learning 2019/2020 [20, 21], we sample 100 query–passage pairs, split evenly between the endpoint grades of perfectly relevant and irrelevant. From BRIGHT [77], we select 70 pairs across biology, economics, sustainable living, and earth science. Of these, 35 are relevant, and 35 pair a query with a non-gold corpus passage retrieved by BM25 and screened with LLM assistance. From Touché 2020 [6, 82], we sample 30 pairs across ten topics, with one pair per topic at each native grade from 0 to 2. From ACORD [85], we sample 100 requirement–clause pairs, with 25 per grade from 1 to 4 on the native 0–4 scale. We assign TREC to the easy tier, BRIGHT and Touché to the medium tier, and ACORD to the hard tier. Each item presents a single pair and asks for a pointwise relevance judgment. We score predictions against the reference judgments and award partial credit for predictions one grade away from the reference on ACORD. Pairs that share a query are statistically dependent.

Task Completion Verification (200). We construct 200 items from five sources. For BrowseComp-Plus [13], we define new subgoals and use snippets from a fixed historical search run as evidence. Each of 25 search prefixes yields one fully supported and one partially supported item, for a total of 50. We also include 8 CALVIN [58] light-state checks on official debug states, 56 τ\tau-bench [89] checks of backend records after replaying historical tool calls, and 46 MiniGrid [14] state-predicate checks on newly collected seeded rollouts. The Embodied Agent Interface [50] supplies the base tasks for 40 VirtualHome items. We assign difficulty by the minimum information and operations needed for a decision. Easy items require a direct check of the current state, medium items require integrating a bounded set of records or evidence, and hard items require inspecting the execution history. The VirtualHome subset contains 10 easy and 10 medium current-state checks and 20 hard history-dependent items. Each hard item starts with a successful execution of one of five base tasks. We append an executable sequence, called the audit suffix, that changes appliance states, removes two target objects, and restores zero, one, or both. A trajectory can therefore end with the objects in their required positions even if they were misplaced when an appliance was operated. The model judges three conditions. C1 requires at least one switch-on in the suffix and checks that all required object placements hold immediately before every such action. C2 requires at least one close action in the suffix and checks that all required placements hold immediately before the first close. C3 checks whether all original state and placement goals hold after the last action. The model selects one of four three-bit profiles, and each distractor flips exactly one bit of the correct profile. For each base task, we generate and verify all eight profiles, retaining four for evaluation and four as counterfactual witnesses. For every evaluated trajectory, a verified counterfactual reaches the same final goal state while changing the value of C1 or C2. Each condition is true in 10 of the 20 evaluated items. We use the verified profiles as references for the hard items and new model-derived annotations as references for the BrowseComp-Plus items. For evidence-sufficiency questions, a negative answer indicates that the visible evidence is insufficient to establish the requirement. We use the base task as the resampling unit when estimating uncertainty for the 20 hard items.

Agent Task Completion

Tool & API Use (282). We construct 282 episodes from 204 BFCL [66] requests and 78 ToolSandbox [56] scenarios. The easy tier contains 100 BFCL requests that specify an operation explicitly or require a direct lookup. Of these, 26 start at the first turn and 74 start at a later turn. The medium tier contains all 78 ToolSandbox scenarios, each presented as a single self-contained request. The hard tier contains 104 original first-turn requests from multi-turn BFCL tasks. Across the tier, these requests involve dependent lookups, conditional actions, or several coordinated outcomes. For episodes that start partway through an interaction, we replay the earlier BFCL turns or execute the earlier ToolSandbox turns in the simulator and show the resulting history to the model. We automatically resolve decisions with a single available option. Each episode has separate budgets for model decisions and native tool calls, with limits of 96 decisions and 12 calls for hard BFCL tasks. Tool calls that exceed time, memory, or output limits return an error and have no effect. Appendix A.6 describes how the episodes are scored.

Text-based Web Interaction (337). We construct 337 items from 100 single steps of Mind2Web [25] demonstrations and 237 BreakingWeb [91] tasks across seven simulated applications. The Mind2Web steps form the easy tier and test whether the model can identify the relevant page element given the operation and any text to enter. The medium and hard tiers contain 137 and 100 BreakingWeb tasks, with 361 and 969 scored decisions, respectively. To enter the hard tier, a task must have at least five scored decisions or at least three entity choices whose targets are not named in the instruction. It must also include at least one decision that requires recalling a value from an earlier page, choosing from a menu with more than 60 options, or inferring several entities. In the easy tier, 50 items ask whether a supplied action advances the goal and 50 ask the model to select the best of four actions, using other controls on the same page as distractors. In the medium and hard tiers, the model makes decisions about which entity to act on, what exact value to enter, and when to stop. To fill a field, the model first selects the field, then selects a group if there are more than 64 candidate values, and finally selects the exact text span to enter. Reference routes are found by an external LLM or written by hand, and we replay each route to successful completion under the original checker. We also accept alternative action orders and detours when replay confirms that the task can still be completed.

Scoring of Tool-Use Episodes

Each tool-use episode is scored by the checker of its source benchmark.

BFCL. An episode scores 1 if BFCL’s multi-turn checker accepts it and 0 otherwise. The checker compares the state of the simulated backend and the results of the model’s tool calls with those of the reference calls [66].

ToolSandbox. ToolSandbox specifies each scenario by milestones, events that must occur in order, and minefields, events that must not occur. Its native score measures how closely the episode matches the milestones and is 0 if any minefield occurs [56]. We make two adaptations. We omit the wording checks on final acknowledgements, because our interface accepts only structured decisions, and we award a score of 1 whenever the full goal state is reached without a minefield violation.

Exhausted budgets. Episodes that exhaust their decision or tool-call budget (Appendix § 7.§ 5) are scored in the same way.

Model Details

Table 4 lists the release of each open-weight model with the settings of our latency and reliability runs, and the model identifier and settings of each API.

ModelReleaseSettings
GLiDEfastino/GLiDE @ d199b6dIn process (Glide.decide), BF16, SDPA attention, flash-linear-attention
KnowLine-31BPelaAI/KnowLine-31B-Gen1 @ 8f052a8SGLang 0.5.21, BF16 with an FP32 output layer, the documented read-twice front end
Sieve-31B-v3sthanika-ai/Sieve-31B-v3 @ 44aa517In process (load_sieve), BF16, eager; the released policy, which scores each request in two option orders in one pass
Decider-27Bperplexity-ai/pplx-decider-v1.2 @ cce0f3fDocumented fast path (layer compilation, CUDA graphs, length buckets) with FP8 matrix multiplication
Kev-27Bjaredpalmer/kev-27b @ af0e6d5Documented serving defaults in process: BF16, fused kernels, CUDA graphs
JevAny-27BSimpleJev/JevAny-Qwen3.8-27B-LoRA @ 09c9e91 on Qwen/Qwen3.8-27B @ 1d4bf0fAdapter merged in BF16, SDPA attention, flash-linear-attention, CUDA graphs
StartLux Autostartlux-models/StartLux-Decision-v1.1-4B-AutoThink @ 23dd4c5Documented server on vLLM 0.19.1 with FP8; auto thinking with gate 0.6 and at most 4,096 thinking tokens
StartLux Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant @ 416088fDocumented server on vLLM 0.19.1 with FP8
Clef-27BCloudflare/clef @ a30162bDocumented SGLang build (dev-clef)
H2O Lightningh2oai/h2o-lightning-4b @ 08e1da5The developer’s vLLM 0.30.0 build with FP8
Clef-Flash-9BCloudflare/clef-flash @ f0b7d3cAs Clef-27B
Decider Flashperplexity-ai/pplx-decider-flash-v1.2 @ 9ff7cf9Documented fast path, BF16
JevAny-4BSimpleJev/JevAny-Qwen3.5-4B-LoRA @ 1c7aa9b on Qwen/Qwen3.5-4B @ 851bf6eAs JevAny-27B
SII-4Bjylin001206/SII-Decision-4B-LoRA @ 5c77219DecisionEngine, BF16; flash-linear-attention added in our runs (SII-4B + fla)
Kev-4Bjaredpalmer/kev-4b @ 6cfce5c on Qwen/Qwen3.5-4B-Base @ 1001bb4As Kev-27B, with the adapter merged
Jeva-4BLIQIIIII/Jeva-4B @ f78e14epredict() with the documented defaults; option cache warm or cold (Appendix § 9)
Layaconvaiinnovations/laya @ 7b928d8TileLang fast path, 8,192-token input window; options not cut at 48 tokens
GPT-6 Solgpt-6-solResponses API, medium reasoning effort, at most 25,000 output tokens
GPT-6 Lunagpt-6-lunaAs GPT-6 Sol
Jev API 1.14jev-1.14.0TypeSafe decision API
OpenAI Decisionsgpt-6-lunaOpenAI decision API

Table 4. Releases and settings. Open-weight models: the Hugging Face repository at the revision used, and the inference path of our latency and reliability runs, which follows each developer’s documented fast path. APIs: the model identifier and settings of the evaluation runs.

Exceptions to the documented settings. Laya’s official package cuts each option text at 48 tokens; all runs use the package without this cut, so that every option is read in full. In our latency and reliability runs, SII-4B + fla adds flash-linear-attention to the documented setup of SII-4B; its leaderboard score is that of the documented setup.

Reliability runs. Each open-weight System One model ran in the settings of the latency measurements (Appendix § 9), one request at a time per model copy, on NVIDIA H100 80 GB GPUs; the two System One APIs ran with four requests in flight. Eight of the 18 corrected references belong to the reliability sets. In the option-order test (six of the eight), we rescored the existing predictions; in the null-awareness test (all eight), we reran the affected items with the corrected option removed. Jeva-4B and StartLux Instant also expose their own null output, and we ran the null-awareness test a second time with it switched on, in the original choice format. For each of the two, NN uses the format that separates the two versions better on the 597 items: the own null output for StartLux Instant and the yes/no question for Jeva-4B. The comparison of StartLux Instant with larger models in § 4.1.2 therefore rests on its own null output. Jeva-4B encodes each option separately, so the order of the options cannot change its scores: its SS is 1 by construction.

Unanswered requests and scoring. We distinguish three cases. (1) Context-window limits. In 30 Web tasks, at least one request exceeds Laya’s 8,192-token window. We retain these tasks without truncating their inputs and assign zero task credit under the context-window rule. (2) Static-task and tool-episode refusals. OpenAI Decisions refuses four static tasks and twelve environment-based episodes. The Safety requests contain harmful traces and instructions to bypass safety restrictions, which may trigger refusal even though the evaluation asks only for classification (Figure 9). An explicit refusal leaves a required decision unanswered, so the affected static task or tool episode receives zero credit under the refusal rule. (3) Web-step refusals. OpenAI Decisions refuses six exact-value requests in Web tasks involving a learning-management system, a patient portal, and Reddit. Potential triggers include sensitive page content; for example, the patient-portal request includes diagnoses and immunization records while asking the model to select a visit reason (Figure 10). Each refused decision is scored as incorrect, and route progress stops at the first incorrect decision. Credit for the preceding correct decisions is retained.

A safety classification refusal showing safety-related request excerpts, an API response, and task credit

Figure 9. A Safety classification refusal. The supplied context combines instructions to disregard safety restrictions with a harmful request, while the evaluation asks only for classification of a reasoning step. OpenAI Decisions returns a refusal rather than a category, yielding zero task credit.

A refusal on a Web appointment task, showing a Web exact-value request excerpt and a saved refusal record

Figure 10. A refusal on a Web appointment task. The request asks for an exact value from the task or page to fill a visit-reason field; the available context includes diagnoses, immunizations, and overdue-status information. OpenAI Decisions refuses this selection. The first three scored decisions retain 3/17=17.65%3/17 = 17.65\% progress; the fourth includes both a wrong group choice and the refusal.

Scored runs and our runs. The scores of Section 4.1 come from runs separate from those behind the latency and reliability results. On the 939 option-order items in their original order, the two give the same credit on 907 to 938 items for each open-weight model except StartLux Auto, which samples its thoughts and agrees on 875. For SII-4B, the comparison is between its documented setup (scored) and SII-4B + fla (ours).

Latency Measurement

Tasks. The 1,109 timed tasks are every single-question choice task of seven static domains, across all tiers: Math 200, Scientific Reasoning 200, Knowledge & Commonsense 200, Safety 190, Sentiment & Emotion Understanding 150, Task Completion Verification 122, and Intent & Routing 47.

Model settings. Each model runs the fast path documented by its authors (Table 4), at their settings for precision, fused kernels, and CUDA graphs, and uses FP8 only where its documentation specifies it. Jeva-4B caches option encodings across requests; we time it with the cache kept within a pass (warm) and with the cache cleared before every request (cold). Figure 5 shows the warm time.

Timing. Each model runs on one NVIDIA H100 80 GB GPU used by no other process, with one request per call. The clock starts when the request is handed to the model’s documented entry point and stops when the option probabilities are back on the host; it therefore includes tokenization and, for server-based models, the local HTTP round trip. After one untimed warm-up pass over all tasks, each model makes three timed passes, and a task’s latency is the mean of its three timings. Two copies of each model ran at the same time on two GPUs, each on its own half of the tasks and pinned to its own 24 CPU cores.

Results. Table 5 lists the latency of every timed model.

ModelSizeMean90th percentile99th percentile
Laya0.4B6.010.237.2
Kev-4B4B25.261.0157
JevAny-4B4B25.256.2173
H2O Lightning4B26.548.4155
StartLux Instant4B35.163.2138
Decider Flash4B39.839.2149
SII-4B + fla4B40.648.5157
Clef-Flash-9B9B53.569.7230
Jeva-4B4B60.972.9142
Jeva-4B, cold cache4B79.088.2180
Kev-27B27B90.3181598
JevAny-27B27B93.9233697
Clef-27B27B108184533
GLiDE27B113214774
Decider-27B27B153145584
KnowLine-31B31B177515959
Sieve-31B-v331B2656682,475
StartLux Auto4B3,08410,03221,870

Table 5. Latency per task (ms) on the 1,109 timed tasks: mean, 90th and 99th percentiles over tasks. Jeva-4B is timed with a warm and a cold option cache.

Closed APIs. The closed APIs cannot be timed on this hardware. Figure 5 places them at their end-to-end times from the evaluation runs, which include the network and the provider’s queue, with four requests in flight.

Analysis Details

Task selection. The analysis of § 4.2.1 uses the medium and hard tiers of each tiered domain (for Math, its 50 medium and 70 hard tasks) and all tasks of the two untiered domains: 2,061 tasks, with 109 to 323 per domain (Table 6). The leaderboard selection (Equation (4)) has as few as 20 tasks in one domain (Task Completion Verification).

DomainTasksLunaGLiDEStartLux InstantBackbone gainNeverAlwaysAuto
Knowledge & Commons.20090.0+5.0 [+1.5, +9.0]−1.5 [−6.0, +3.0]+4.485.088.087.5
Sentiment & Emotion20084.8+4.8 [+1.2, +8.5]+3.0 [−2.0, +8.0]+3.488.085.886.0
Retrieval & Reranking20076.5−5.8 [−11.2, −0.2]−8.2 [−14.5, −1.8]+3.969.070.871.2
Intent & Routing20079.5+0.1 [−2.4, +2.9]−8.9 [−12.0, −5.8]+12.068.870.871.1
Safety32375.2+4.0 [−1.2, +9.0]−7.1 [−12.7, −1.5]+16.260.463.863.5
Task Completion Verif.10998.2+0.9 [+0.0, +2.8]−22.9 [−31.2, −15.6]+17.458.766.173.4
Tool & API Use18275.8−3.6 [−9.1, +1.8]−11.2 [−17.1, −5.2]+25.661.964.561.9
Web Interaction23764.4+3.1 [−0.4, +6.5]−28.8 [−33.8, −23.8]+23.940.842.640.4
Math120100.0−26.7 [−34.2, −19.2]−38.3 [−47.5, −30.0]+14.450.892.587.5
Scientific Reasoning15091.3−21.3 [−28.7, −14.7]−27.3 [−35.3, −19.3]+8.358.770.770.7
Code Understanding14088.6−25.7 [−34.3, −17.1]−26.4 [−35.7, −17.1]+7.755.761.459.3

Table 6. Per-domain values behind Figure 7. Tasks: medium and hard tiers, or all tasks of an untiered domain. Luna: GPT-6 Luna’s score. Differences from Luna with 95% bootstrap intervals over tasks. Backbone gain: mean over the four families. StartLux-4B-AutoThink: its scores in the three thinking modes.

Per-domain values. Table 6 lists the values plotted in Figure 7.

Thinking control. StartLux-4B-AutoThink ran in each of its three thinking modes on all 2,061 tasks: the 1,642 static tasks (8,167 questions per mode), sent as batched requests, and the 419 Tool & API Use and Web Interaction tasks through the benchmark’s runtime (5,500 to 5,800 model calls per mode). Our auto run gives the same credit as the benchmark’s own auto run on 1,507 of the 1,642 static tasks (92%) and on 376 of the 419 tool and web tasks (90%); thinking is sampled, and our requests were batched.

Hard tiers only. Table 7 repeats the backbone and thinking gains on the hard tier of each tiered domain, together with all tasks of the two untiered domains (1,202 tasks, with Math’s 70 hard tasks). The patterns of § 4.2.1 are unchanged except in three domains. In Intent & Routing, the backbone gain falls from 12.0 to 4.9 points. In Task Completion Verification, whose hard tier has 20 tasks, the backbone gain falls from 17.4 to 3.8 points and the thinking gain from 7.3 to −15.0. In Code Understanding, the backbone gain is 7.7 points on medium and hard combined and 7.1 on hard alone, and the thinking gain is 5.7 and 8.6.

DomainTasksBackboneThinkingTasksBackboneThinking
Knowledge & Commons.200+4.4+3.0untiered
Sentiment & Emotion200+3.4−2.2untiered
Retrieval & Reranking200+3.9+1.8100+3.4+4.0
Intent & Routing200+12.0+2.0100+4.9+3.6
Safety323+16.2+3.4158+19.8+4.4
Task Completion Verif.109+17.4+7.320+3.8−15.0
Tool & API Use182+25.6+2.6104+26.2+0.0
Web Interaction237+23.9+1.8100+21.5+2.7
Math120+14.4+41.770+13.6+52.9
Scientific Reasoning150+8.3+12.080+11.2+13.8
Code Understanding140+7.7+5.770+7.1+8.6

Table 7. Backbone and thinking gains on the medium and hard tiers and on the hard tier alone. Gains as in Figure 7b and Figure 7c (thinking: always minus never). The untiered domains are identical in both selections.

Reliability against performance. In Figure 8a and c, the trend is a quadratic fit of SS (or NN) on the leaderboard score over the 16 System One models other than Laya and Jeva-4B. The shaded region extends the trend by a constant half-width, the 90th percentile of the absolute residuals, over scores from 40 to 74; its half-width is 0.07 for SS and 0.18 for NN. For NN, we estimate 95% intervals by resampling the 597 items 2,000 times, with the same resample for every model. Each of Kev-27B, Jev API 1.14, and JevAny-27B exceeds each of the four highest (GLiDE, KnowLine-31B, Sieve-31B-v3, and Decider-27B), with the interval of the difference above zero in all 12 pairs.

References

  1. [1]Ali Alhashimi. System One Bench: An independent benchmark for decision models. GitHub repository, 2026. URL https://github.com/alilibx/system-one-bench.
  2. [2]Diogo Almeida. Introducing System One Models & Jev. TypeSafe AI Blog, 2026. URL https://typesafe.ai/blog/introducing-system-one-models-and-jev.
  3. [3]Anthropic. System card: Claude Opus 5.5. Anthropic, 2026. URL https://www.anthropic.com/system-cards.
  4. [4]Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. MathArena: Evaluating LLMs on Uncontaminated Math Competitions. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.23281
  5. [5]Benchmark Heaven. JevBench: Benchmark and leaderboard for AI decision models. GitHub repository, 2026. URL https://github.com/fstandhartinger/jevbench.
  6. [6]Alexander Bondarenko, Maik Fröbe, Meriem Beloucif, Lukas Gienapp, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, et al. Overview of Touché 2020: Argument retrieval. In International Conference of the Cross-Language Evaluation Forum for European Languages, 2020.DOI
  7. [7]Brida. ReflexBench v1: An open benchmark for system one models, 2026. URL https://www.brida.ai/blog/reflexbench-v1-system-one-models.
  8. [8]Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient Intent Detection with Dual Sentence Encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, 2020.arxiv.org/abs/2003.04807
  9. [9]Iñigo Casanueva, Ivan Vulić, Georgios Spithourakis, and Pawel Budzianowski. NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue. In Findings of the Association for Computational Linguistics: NAACL 2022, 2022.
  10. [10]Center for AI Safety, Scale AI, and HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. Nature, 649:1139–1146, 2026. doi: 10.1038/s41586-025-09962-4. URL https://www.nature.com/articles/s41586-025-09962-4.
  11. [11]Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, and Mingyuan Zhou. System1-Arena: Which decisions can be left to system 1?, 2026. URL https://arxiv.org/abs/2610.21244.
  12. [12]Zigeng Chen, Xinyin Ma, Gongfan Fang, Ruonan Yu, and Xinchao Wang. Verithinker: Learning to verify makes reasoning model efficient. Advances in Neural Information Processing Systems, 2025.
  13. [13]Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Sahel Sharifymoghaddam, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Yanxi Li, Haoran Hong, Xinyu Shi, Xue Liu, Hosna Oyarhoseini, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026.
  14. [14]Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and J. K. Terry. Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks. In Advances in Neural Information Processing Systems, 2023.
  15. [15]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  16. [16]Cloudflare. Clef. Hugging Face model card, 2026. URL https://huggingface.co/Cloudflare/clef.
  17. [17]Cloudflare. Clef-flash. Hugging Face model card, 2026. URL https://huggingface.co/Cloudflare/clef-flash.
  18. [18]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  19. [19]ConvAI Innovations. Laya. Hugging Face model card, 2026. URL https://huggingface.co/convaiinnovations/laya.
  20. [20]Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the TREC 2019 Deep Learning Track. arXiv preprint arXiv:2003.07820, 2020.arxiv.org/abs/2003.07820
  21. [21]Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the TREC 2020 Deep Learning Track. arXiv preprint arXiv:2102.07662, 2021.arxiv.org/abs/2102.07662
  22. [22]Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. OR-Bench: An Over-Refusal Benchmark for Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning, 2025.arxiv.org/abs/2405.20947
  23. [23]DeepSeek-AI. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. URL https://arxiv.org/abs/2512.02556.
  24. [24]Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoEmotions: A Dataset of Fine-Grained Emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.arxiv.org/abs/2005.00547
  25. [25]Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/5950bf290a1570ea401bf9882128160-Abstract-Datasets_and_Benchmarks.html.arxiv.org/abs/2306.06070
  26. [26]Feiyu Duan, Jiayu Lin, Jia Wang, Jun Xiang, Jialiang Wu, Xinnong Zhang, Hanqi Yan, Siyuan Wang, and Zhongyu Wei. General decision models: Benchmarking and insights beyond Jey, 2026. URL https://arxiv.org/abs/2610.03935.
  27. [27]Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: Llm learns when to think. Advances in neural information processing systems, 2025.arxiv.org/abs/2505.13379
  28. [28]Fastino. GLiDE 1.2. Hugging Face model card, 2026. URL https://huggingface.co/fastino/GLiDE.
  29. [29]Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. Efficient reasoning models: A survey. Transactions on Machine Learning Research, 2025. URL https://openreview.net/forum?id=sySqlxj8EB.arxiv.org/abs/2504.10903
  30. [30]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://papers.nips.cc/paper_files/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html.
  31. [31]Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. Are We Done with MMLU? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025.
  32. [32]Jonathan Geuter, Andrew Cheng, and David Alvarez-Melis. Releasing the inner Jev: A simple recipe for open decision models, 2026. URL https://arxiv.org/abs/2610.17205.
  33. [33]Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, and Christopher Parisien. AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025.
  34. [34]Google DeepMind. Gemini 3 Pro model card. Google DeepMind, 2025. URL https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf.
  35. [35]James Gung, Emily Moeng, Wesley Rose, Arshit Gupta, Yi Zhang, and Saab Mansour. NatCS: Eliciting Natural Customer Support Dialogues. In Findings of the Association for Computational Linguistics: ACL 2023, 2023.arxiv.org/abs/2305.03007
  36. [36]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330, 2017. URL https://proceedings.mlr.press/v70/guo17a.html.arxiv.org/abs/1706.04599
  37. [37]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.DOI
  38. [38]H2O.ai. H2O-Lightning-4B. Hugging Face model card, 2026. URL https://huggingface.co/h2oai/h2o-lightning-4b.
  39. [39]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2009.03300.
  40. [40]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
  41. [41]Monte Hoover, Vatsal Baherwani, Neel Jain, Khalid Saifullah, Joseph James Vincent, Chirag Jain, Melissa Kazemi Rad, C. Bayan Bruss, Ashwinee Panda, and Tom Goldstein. DynaGuard: A Dynamic Guardian Model With User-Defined Policies. In International Conference on Learning Representations, 2026.arxiv.org/abs/2509.02563
  42. [42]Yupeng Hou, Jiacheng Li, Xiangjun Fu, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026.
  43. [43]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In International Conference on Learning Representations, 2025.arxiv.org/abs/2403.07974
  44. [44]Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, and Robby T. Tan. CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026.
  45. [45]Ishita Kakkar, Enze Zhang, Rheeya Uppaal, and Junjie Hu. HarmThoughts: A Benchmark for Fine-Grained Harmful Behavior Detection in Reasoning Traces. In Conference on Language Modeling, 2026.arxiv.org/abs/2604.19001
  46. [46]Runze Kong, Haibo Jin, Xucheng Yu, Xiaoqi Wang, Huimin Zeng, and Haohan Wang. Do geo gains survive the live web? evaluating in-the-wild visibility gains of generative engine optimization via live-geo. arXiv preprint arXiv:2611.01126, 2026.arxiv.org/abs/2611.01126
  47. [47]Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannah Hajishirzi. RewardBench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1755–1797. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-naacl.96. URL https://aclanthology.org/2025.findings-naacl.96/.DOI
  48. [48]Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjiao Tang, and Jason Mars. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019.arxiv.org/abs/1909.02027
  49. [49]Changyi Li, Jiayi Wang, Xudong Pan, Geng Hong, and Min Yang. ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models. arXiv preprint arXiv:2505.17244, 2025.arxiv.org/abs/2505.17244
  50. [50]Manling Li, Shiyu Zhao, Qiner Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking LLMs for embodied decision making. In NeurIPS 2024, 2024.arxiv.org/abs/2410.07166
  51. [51]Qi Li and Xinchao Wang. Jeva: System one models from first principles, 2026. URL https://arxiv.org/abs/2610.18773.
  52. [52]Xuzhao Li, Xuchen Li, Shiyu Hu, Yongzhen Guo, and Wentao Zhang. VerifyBench: A systematic benchmark for evaluating reasoning verifiers across domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 31796–31804, 2026. doi: 10.1609/aaai.v40i38.40448. URL https://ojs.aaai.org/index.php/AAAI/article/view/40448.DOI
  53. [53]Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhui Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. URL https://openreview.net/forum?id=i04LZibEqW.DOI
  54. [54]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step. In International Conference on Learning Representations, 2024.arxiv.org/abs/2305.20050
  55. [55]Chenghao Liu, Wei Fan, Jindong Han, Dongjie Wang, Aniket Mahanti, Pengyang Wang, and Hao Liu. Timesieve: Thinking fast on time series with system 1 sufficiency. arXiv preprint arXiv:2610.26002, 2026.arxiv.org/abs/2610.26002
  56. [56]Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160–1183, 2025. doi: 10.18653/v1/2025.findings-naacl.65. URL https://aclanthology.org/2025.findings-naacl.65/.DOI
  57. [57]Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, 2025.
  58. [58]Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks. IEEE Robotics and Automation Letters, 2022.arxiv.org/abs/2112.03227
  59. [59]Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.arxiv.org/abs/2401.00396
  60. [60]OpenAI. OpenAI o1. https://openai.com/o1/, 2024.
  61. [61]OpenAI. Build more responsive apps with the Decisions API. OpenAI, October 2026. URL https://openai.com/index/decisions-api/.
  62. [62]OpenAI. GPT-6 Astra system card. OpenAI Deployment Safety Hub, 2026. URL https://deploymentsafety.openai.com/gpt-6-astra.
  63. [63]OpenAI. Introducing GPT-6 Sol and Luna. OpenAI, 2026. URL https://openai.com/index/introducing-gpt-6-sol-and-luna/.
  64. [64]Jared Palmer. Key-27B: A calibrated decision model on Qwen3.8-27B. Hugging Face model card, 2026. URL https://huggingface.co/jaredpalmer/kev-27b.
  65. [65]Jared Palmer. Kev-4B. Hugging Face model card, 2026. URL https://huggingface.co/jaredpalmer/kev-4b.
  66. [66]Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
  67. [67]PelaAI. KnowLine-31B-Gen1: A 31b decision model. Hugging Face model card, 2026. URL https://huggingface.co/PelaAI/KnowLine-31B-Gen1.
  68. [68]Perplexity. pplx-decider-v1.2. Hugging Face model card, 2026. URL https://huggingface.co/perplexity-ai/pplx-decider-v1.2.
  69. [69]Perplexity. pplx-decider-flash-v1.2. Hugging Face model card, 2026. URL https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.2.
  70. [70]Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. DynaSent: A Dynamic Benchmark for Sentiment Analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
  71. [71]Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtu-alHome: Simulating household activities via programs. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8494–8502. IEEE, 2018.
  72. [72]Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.arxiv.org/abs/1909.05855
  73. [73]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling, 2024.arxiv.org/abs/2311.12022
  74. [74]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013.DOI
  75. [75]Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To CoT or not to CoT? chain-of-thought helps mainly on math and symbolic reasoning. In ICLR, 2025.arxiv.org/abs/2409.12183
  76. [76]Sthanika AI. Sieve: Decision models. GitHub repository, 2026. URL https://github.com/sthanika-ai/Sieve.
  77. [77]Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Haifu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan Ö. Arik, Danqi Chen, and Tao Yu. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval. In International Conference on Learning Representations, 2025.arxiv.org/abs/2407.12883
  78. [78]Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-acl.824. URL https://aclanthology.org/2023.findings-acl.824/.DOI
  79. [79]Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.arxiv.org/abs/1811.00937
  80. [80]Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica. JudgeBench: A benchmark for evaluating LLM-based judges. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/9e720fce64f91114c49cfd640d821da3-Abstract-Conference.html.arxiv.org/abs/2410.12784
  81. [81]Yuichi Tateno. S1MB: Comparing system one decision models across 100+ benchmarks. Hugging Face, 2026. URL https://huggingface.co/blog/hotchpotch/system-one-mosaic-benchmark.
  82. [82]Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
  83. [83]TypeSafe AI. Primitives (questions). Official API documentation, n.d. URL https://docs.typesafe.ai/primitives. Accessed November 3, 2026.
  84. [84]Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do Large Language Model Benchmarks Test Reliability? arXiv preprint arXiv:2502.03461, 2025.arxiv.org/abs/2502.03461
  85. [85]Steven H. Wang, Maksim Zubkov, Kexin Fan, Sarah Harrell, Yuyang Sun, Wei Chen, Andreas Plesner, and Roger Wattenhofer. ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025.arxiv.org/abs/2501.06582
  86. [86]Tianxin Wei. JevAny. GitHub repository, 2026. URL https://github.com/SimpleJev/JevAny.
  87. [87]Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leet-CodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs. arXiv preprint arXiv:2504.14655, 2025.arxiv.org/abs/2504.14655
  88. [88]Xiaoyu Xing, Zhijing Jin, Di Jin, Bingning Wang, Qi Zhang, and Xuanjing Huang. Tasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020.arxiv.org/abs/2009.07964
  89. [89]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/1b126cc38b8638e07bef37e7b2bb72bf-Abstract-Conference.html.DOI
  90. [90]Jianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun, Zijian Jin, Dawn Song, and Daoan Zhang. VerifierBench: Benchmarking the generation of executable verifiers for LLM agents, 2026. URL https://arxiv.org/abs/2610.28679.
  91. [91]Xunjian Yin, Tianchen Guan, Jinao Wang, Weili Cao, Daisy Xinlei Lin, Royce Cheng-Yue, Keagan Long, Kyle Wong, Bhuwan Dhingra, Xiangjun Wang, and Shuyan Zhou. Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions. arXiv preprint arXiv:2609.35814, 2026.arxiv.org/abs/2609.35814
  92. [92]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.arxiv.org/abs/1905.07830
  93. [93]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html.arxiv.org/abs/2306.05685
  94. [94]Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2307.13854.
  95. [95]Xialie Zhuang, Zunhai Su, Jia Pan, Ngai Wong, Guan-hua Chen, and Chuan-Wei Kuo. StartLux-Decision: AutoThink decision models with reinforcement learning for calibrated deliberative decisions, 2026. URL https://arxiv.org/abs/2610.17309.

Paper details

vlnc:W_4khn8ddq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents