betaFeedback

WorksComputer Science

TandemBench: Benchmarking Decision Models for Multi-Agent Collaboration

Abstract

Decision models choose among predefined options in tasks such as routing and action selection. In multi-agent collaboration, a decision determines how members divide work, exchange resources, and act in parallel. We introduce TandemBench, a benchmark of 1,200 questions in multi-agent cooking scenarios where coordination choices affect goal completion. Models receive complete global information and candidate team plans, with reference answers established through execution and exact search. Models select an optimal team plan and, in paired variants that retain or remove it, judge whether better arrangements exist beyond the supplied options. We evaluate 40 native decision models and compare them with 10 general-purpose language models. The best native decision model achieves 62.50% decision accuracy. Native decision models remain close to chance in optimality verification. Paired tests reveal that some report better alternatives regardless of whether an optimal plan is already present. These findings show that team-plan decision accuracy alone can conceal systematic errors in judging whether the supplied options contain an optimal arrangement. The dataset and evaluation code are publicly available at https://github.com/lililibaohang/TandemBench.

Introduction

Decision models choose among predefined options in tasks such as routing and action selection [5, 6]. In multi-agent collaboration, a single decision can determine how team members coordinate their actions toward a shared goal. For example, an assistant can supply an ingredient before preparing plates, allowing the chef to cook while the assistant continues preparation. Preparing plates first delays the handoff and leaves the chef waiting (Figure 1). Judging a team plan therefore requires considering how members’ actions depend on one another and jointly contribute to the goal.

Figure 1. One member’s action order affects the whole team. (a) Handing over ingredients first enables parallel cooking and plate preparation. (b) Preparing plates first leaves the chef waiting.

Existing decision benchmarks examine accuracy, calibration, and robustness [5, 8] (Zhuang et al. 2026), while AgentDecisionBench evaluates tool and action choices in agent workflows (Zhu et al., 2026). We focus on decisions that specify joint plans for multiple team members. Several plans may be feasible, yet their coordination choices can lead to different outcomes for the team. Evaluating these choices requires a reference for the best achievable outcome from the given state, since even the best supplied plan may be suboptimal. We therefore assess whether models select an optimal team plan and recognize when better arrangements exist beyond the supplied options.

We introduce TandemBench, comprising 1,200 questions from 200 scenarios across 30 recipe tasks in Collab-Overcooked (Sun et al., 2025). We select states where action ordering or resource handoffs affect task completion even when the remaining actions are optimized. Each question provides complete global information and a candidate set containing multiple feasible team plans. Execution and exact search establish candidate outcomes. comes and the best achievable result from each state. Paired variants retain or remove the optimal plan while keeping the state and goal fixed, allowing us to distinguish correct reports of better alternatives from false alarms.

We evaluate 40 native decision models and compare them with 10 general-purpose language models. KnowLine leads the native models in decision accuracy (DA) at 62.50% (Figure 2; Table 1 and Table 2), while their verification accuracy (VA) remains near chance (50%). Paired tests reveal two contrasting errors: some models report a better plan whether or not the optimum is supplied, while others select a supplied plan even when it is suboptimal. Thus, DA alone misses systematic errors in optimality judgments.

Figure 2. Decision accuracy (DA) and verification accuracy (VA), with models sorted by increasing DA within each family.

RankModelParams. (B)DA ↑\uparrowVA ↑\uparrow
1🤗 KnowLine-31B-Gen13162.5051.44
2🤗 Basal-Omni-1.0-Ultra3160.7551.97
3🤗 Quyet-1.0-Large3160.7552.06
4🤗 DOPEST-1.0-31B3160.3852.16
5🤗 PPLX-Decider-v1.22760.3855.31
6🤗 PPLX-Decider-v1.1-27B2759.4454.28
7🤗 Torchcast-4B-v1.2458.9450.66
8🤗 H2O-Lightning-31B (BF16)3158.6950.19
9🤗 D32758.6952.19
10🤗 GLiDE-1.2-Open2758.1354.72
11🤗 StartLux-v1.1-27B-Instant2757.4454.00
12🤗 René-31B (dequant. BF16)3155.5653.41
13🤗 PPLX-Decider-Flash-v1.2452.9453.19
14🤗 D3-Turbo2752.9449.97
15🤗 Torchcast-4B-v1.3452.7551.28
16🤗 deck31b (BF16)3152.1949.72
17🤗 Wald-26B-A4B26/450.6351.09
18🤗 Jeff-1.1-Medium1250.5050.69
19🤗 Vega-27B2750.2552.28
20🤗 Tacit-9B948.6349.94
21🤗 D3-Ultra2747.5056.28
22🤗 MeTask-Rain-31B3147.4450.56
23🤗 MeTask-Rain-12B1247.3850.66
24🤗 Liquid AI d1-3B3.1247.1953.47
25🤗 Torchcast-12B1246.7552.69
26🤗 Tacit-4B446.3150.00
27🤗 Mapika-Decider-12B1244.2550.78
28🤗 Mammoth-12743.8852.34
29🤗 Torchcast-4B-v1.1443.5052.34
30🤗 Clef2740.9452.91
31🤗 Joey-1240.6950.53
32🤗 Tacit-2B239.4450.25
33🤗 H2O-Lightning-4B439.0651.19
34🤗 Nox-4B439.0050.00
35🤗 Clef-Flash935.8149.91
36🤗 Lux-9B934.5651.97
37🤗 Tacit-1.7B1.729.4449.59
38🤗 Laya-421M0.42125.8849.69
39🤗 Kai-0.6B0.621.8150.00
40🤗 CLM-8B815.4445.94

Table 1. DA and VA (%) for 40 native decision models, averaged over four option rotations. Parameters are in billions; “26/4” denotes total/active parameters.

RankModelAvg. output tokensDA ↑\uparrowVA ↑\uparrow
1🤗 Qwen3.8-27B865282.8071.88
2🤗 Gemma4-31B107273.1157.03
3🤗 Gemma4-26B-A4B151660.5853.66
4🤗 Muse-Glimmer-30B141559.6460.35
5🤗 Qwen3-32B92546.3349.45

Table 2. DA and VA (%) and mean output length for the five highest-DA general-purpose models.

Our contributions are as follows.

  • We introduce TandemBench, a 1,200-question benchmark for evaluating decision models on how multiple agents coordinate their actions toward a shared goal, given complete global information.

  • We ground comparisons among candidate coordination plans in execution and exact search, and use paired tests to assess whether models recognize when the supplied options omit the optimal arrangement.

  • We show that decision accuracy alone misses systematic errors in optimality verification, and analyze how these decisions vary across short- and long-horizon goals and option orders.

Related Work

Decision Models. Decision-model evaluation covers both individual choices and decisions within agent workflows. JEVal and OpenDecisions assess accuracy, calibration, and inference efficiency ([5, 6]), while DecisionRobust and DecisionPivot examine adversarial robustness and controlled input changes, respectively ([8] (Zhuang et al. 2026)). AgentBench and AgentBoard evaluate sequential decisions through task success and progress ([10, 11]); Agent-DecisionBench spans isolated choices, recorded agent states, and online control ((Zhu et al., 2026)). TandemBench takes a joint team plan as the decision unit, so one choice specifies actions for multiple members toward a shared goal.

Multi-Agent Collaboration. CoELA and RoCo study communication and coordinated execution, while MultiAgentBench and PARTNR evaluate collaboration in interactive settings ([12, 2] (Zhang et al. 2024; Zhu et al. 2025)). CoordQA and RoCoBench-Text assess coordination choices and reasoning through static questions ([1, 12]). ParaCook supplies CP-SAT optimal references for abstract scheduling ((Zhang et al., 2026)), while CoCoBench evaluates coordination against optimal makespan under a skill abstraction ([3]). TandemBench builds on CollaborationOvercooked’s asymmetric cooking environment ((Sun et al., 2025)). It evaluates a single model’s choices among executable team plans under complete global information, using an exact optimality reference from the same state.

Planning and Verification. Planning benchmarks assess plan generation and reasoning about actions, state transitions, and constraints ([7] (Valmeekam et al. 2023; Xie et al. 2024; Zheng et al. 2024)). Plan-optimality verification asks whether a given plan is optimal ([9]), while optimality certificates provide independently checkable evidence ([4]). DecisionPivot uses controlled contrasts to examine how decisions respond to input changes ((Zhuang et al., 2026)). TandemBench asks whether any supplied joint team plan attains the state optimum. Paired variants retain or remove the optimal plan while keeping the state and goal fixed, testing whether model judgments track its presence.

Multi-Agent Collaborative Decision-Making

Task Definition

A single decision model receives a complete global state, the environment rules, a shared goal, and candidate team plans. Each candidate specifies an action program for each of two team members. The global state records both members’ configurations, task-relevant resources, and ongoing processes.

The two programs execute concurrently under the environment rules. Their dependencies and resource handoffs determine when each member can proceed (Figure 1). A candidate may cover the full task or prescribe a stage that ends before the final goal. Appendix § 8 specifies the execution semantics.

Decision Objectives

We evaluate a team plan against either an intermediate milestone or the final task goal, referred to as short- and long-horizon goals. Completion cost is measured in joint execution steps.

Short-horizon goals. We count the steps until the specified intermediate milestone is first reached.

Long-horizon goals. We count the steps until the final goal is reached. If the prescribed stage ends before completion, the cost is its executed steps plus the minimum remaining steps from the resulting state.

Both goal settings apply to team-plan decisions and optimality verification. Short- and long-horizon questions from the same state use their respective execution protocols and may have different candidate programs. Appendix § 8.§ 2 provides the formal definitions.

Optimal Decisions

Both tasks use the state optimum: the minimum completion cost of any legal joint execution from the given state under the specified goal and action domain.

For team-plan decision-making, the model chooses among four candidate plans. Exactly one attains the state optimum and is the correct answer.

Plan-optimality verification asks whether a given plan is optimal (Lin et al., [9]). We evaluate whether the supplied set contains such a plan. Each verification question includes three concrete plans and a fourth option stating that a feasible plan can reach the goal in fewer joint steps than all three. This statement is true exactly when none of the supplied plans attains the state optimum.

The two tasks are scored separately. Decision accuracy scores the optimal concrete choice. Verification accuracy scores whether the better-plan statement is correctly accepted or rejected; any valid concrete choice counts as rejection. § 6 defines the metrics, and Appendix § 8.§ 2 gives the formal decision criteria.

TandemBench

We construct TandemBench from states where coordination choices affect task completion. Alternative team plans make these choices explicit, and execution and exact search establish their outcomes and the state optimum. Each verified base question yields paired variants that retain or remove the optimal plan (Figure 3).

Construction of TandemBench

Figure 3. Construction of TandemBench. We select scenarios with consequential coordination choices and construct candidate team plans. Execution and exact search establish reference answers. Retaining or removing the optimal plan yields paired verification questions.

Environment and Tasks

We use 30 recipe tasks from Collab-Overcooked (Sun et al., 2025), where a chef and an assistant work in separate kitchen regions with asymmetric access to ingredients, appliances, and counters. Access restrictions make some actions depend on resources supplied by the other member, while recipe dependencies constrain action order. Automatic processing allows preparation to overlap with cooking (Figure 1). Carrying limits and shared-counter availability constrain handoff locations and timing. Appendix § 8 gives the execution rules.

Scenario Selection

We select states where fixing a coordination choice changes the best achievable completion cost. Candidate states come from verified optimal delivery trajectories, at events that change available materials, processing status, or handoff opportunities. Execution prefixes establish reachability.

At each state, we compare alternative orders of two required events performed by the same member or alternative locations for a required handoff. Exact search determines the minimum delivery cost subject to each choice, optimizing all remaining actions. A state qualifies when two feasible choices are established: one attains the state optimum, while the other requires at least three additional joint steps.

We select 200 qualifying scenarios across all 30 tasks, prioritizing coverage of event orderings, handoff directions, comparison families, and material stages. Screening uses the final delivery goal and constraints on individual coordination choices. Concrete candidate programs are constructed and evaluated separately for the short- and long-horizon questions. Algorithm 1 in Appendix § 9 details screening and coverage.

statement that a feasible plan can reach the goal in fewer joint steps than all three remaining plans. The state, goal, execution rules, and retained programs remain unchanged. The resulting four-option pair supports comparison of correct reports when the optimum is missing and false alarms when it is present.

Instance Construction

Each scenario supplies a short-horizon and a long-horizon base question. From each verified base question, we derive a pair of optimality-verification variants.

Question Construction

Candidate plans express different ways to coordinate the same team toward the specified goal. They vary in work order, handoff location, preparation scope, work overlap, or waiting conditions. For example, changing the order of ingredient supply and utensil preparation affects when processing can begin and which work can overlap. Handoff choices also depend on members’ positions and counter availability. Appendix § 8.§ 3 gives concrete candidate descriptions and costs.

Each base question contains four plans, with at least three feasible and exactly one attaining the state optimum. Multiple feasible alternatives require comparing arrangements that can complete the goal. We aim to include alternatives with small cost gaps when they express distinct coordination choices. Candidate programs may differ between the short- and long-horizon objectives.

To construct a verification pair, we remove the optimal plan in one variant and a predetermined nonoptimal plan in the other. Both add the same statement that a feasible plan can reach the goal in fewer joint steps than all three remaining plans. The state, goal, execution rules, and retained programs remain unchanged. The resulting four-option pair supports comparison of correct reports when the optimum is missing and false alarms when it is present.

Answer Annotation

Candidate execution records the steps taken, goal attainment, and resulting state. Exact search establishes the state optimum and any optimal continuation needed for long-horizon evaluation. Together, these determine candidate costs under the objectives in § 3. Matching candidate costs to the state optimum identifies the correct base answer. In the verification pair, the better-plan statement is false when the optimum is retained and true when it is removed.

Noncompletion requires evidence under the prescribed program and protocol; unresolved searches do not supply answer labels. Appendix § 9.§ 2 details candidate validation and content review.

Benchmark Statistics

TandemBench contains 1,200 questions from 200 scenarios across 30 tasks: 400 team-plan decisions and 800 optimality-verification variants. The 200 short- and 200 long-horizon questions form 200 pairs sharing the initial state. The variants form 400 pairs, each containing one question with the optimum retained and one with it removed. Both types of pairs derive from the same 200 scenarios. The two evaluation dimensions are scored separately, and option rotations do not count as additional questions.

In 64 scenarios, the short- and long-horizon questions use identical candidate programs and retain the same optimal choice, supporting comparisons across goals and execution protocols. In 15 long-horizon questions, a feasible alternative finishes its prescribed stage sooner than the optimal candidate but has a higher total completion cost. These cases support analysis of choices where stage length and final cost favor different plans. Appendix § 9.§ 3 reports structural groups for related recipe variants and programs, with grouping rules and full counts.

Experiments and Analysis

Experimental Setup

TandemBench uses English inputs. We evaluate 40 native decision models and report reference results from 10 general-purpose language models. Unresolved answers from these models count as incorrect under the scoring policy. Each general-purpose model contributes 19,200 outputs across questions, option rotations, and samples.

Each question is evaluated under four cyclic option rotations, so each candidate appears once in each position. Native decision models produce one response per rotation, and general-purpose models produce four. Each response is scored separately, without voting. Models use their native decision or generation interfaces; Appendix § 10 details reasoning modes and sampling parameters.

Decision accuracy (DA) measures selection of the optimal concrete plan. Verification accuracy (VA) averages the correct-detection rate when the optimum is removed and the correct-rejection rate when it is retained. Any valid concrete-plan choice counts as rejecting the better-plan statement. Invalid or truncated responses count as incorrect for both metrics. Uniform option selection yields 25% DA and 50% VA. Appendix § 11 gives the metric formulas.

Overall Performance

Native decision models vary widely in team-plan decision accuracy, from 15.44% to 62.50% (Table 1). Their verification accuracy remains near chance, ranging from 45.94% to 56.28%.

The highest-DA model, KnowLine, does not lead in verification: D3-Ultra achieves the highest VA despite lower DA. Thus, plan-selection accuracy alone does not capture how well models judge whether better arrangements exist beyond the supplied options.

Among general-purpose models (Table 2; Appendix § 11.§ 2), Qwen3.8-27B leads with 82.80% DA and 71.88% VA.

whether better arrangements exist beyond the supplied options.

Among general-purpose models (Table 2; Appendix § 11.§ 2), Qwen3.8-27B leads with 82.80% DA and 71.88% VA.

Optimality Verification

Many models report a better plan at similar rates whether the optimal plan is present or absent (Figure 4(a)). Tacit-2B detects 98.38% of cases where the optimum is absent, but also raises false alarms in 97.88% of cases where it is present.

Detection and false-alarm rates and paired responses

Figure 4. Optimality-verification behavior. (a) Detection versus false-alarm rates. (b) Paired responses when the optimal plan is retained or removed, in that order. Yes reports a better plan; No selects a concrete plan.

Paired responses show two recurring patterns: some models report a better plan in both variants, while others select a concrete plan in both (Figure 4(b)). Both patterns produce a correct judgment in one condition and an incorrect judgment in the other.

The better-plan probabilities also discriminate weakly between the two conditions, with a median AUROC of 0.537 and a maximum of 0.598 across native models (Appendix § 14).

Short- and Long-Horizon Goals

We compare 64 matched pairs that share the initial state, candidate programs, and optimal choice, but differ in their goal and execution rules.

Across native decision models, mean DA decreases from 49.58% for short-horizon goals to 46.75% for long-horizon goals, although individual models vary (Figure 5). Full model-level results appear in Appendix § 12.§ 2.

Short- and long-horizon decision accuracy

Figure 5. Short- and long-horizon DA with identical candidate programs but different goals and execution rules. The three models with the highest overall DA in each family are shown. Changes are long minus short, in percentage points.

Earlier stage completion can leave more work for the team. Appendix § 12 reports model-level choices on 15 long-horizon questions where an alternative finishes its prescribed stage sooner but requires more joint steps to reach the final goal.

Position Sensitivity

We measure position sensitivity as the difference between a model’s highest and lowest accuracy across four option rotations. The median range across native models is 10.38 percentage points, indicating that option order affects measured accuracy (Figure 6(a)).

Figure 6. Sensitivity to option order. (a) DA range across four rotations; bars mark group medians. (b) DA versus choice consistency, the percentage of questions selecting the same underlying plan across all rotations. General-model scores average four samples per rotation and include sampling variability.

Choice consistency measures how often the same underlying plan is selected across all four rotations (Figure 6(b)). CLM selects the same plan in 92.00% of questions despite a DA of 15.44%. Stable choices can therefore still be consistently incorrect.

Conclusion

TandemBench evaluates decision models on team plans involving task dependencies, resource handoffs, and parallel work. Execution and exact search establish reference answers, while paired candidate sets test whether models recognize when better arrangements exist beyond the supplied options. Native decision models differ widely in plan-choice accuracy, but their reports of better alternatives often fail to distinguish whether an optimal plan is present. Performance also varies across goal settings and option orders. TandemBench therefore supports comparisons that assess both the selected team plan and judgments about the available alternatives.

Limitations

Our findings concern team-plan decisions for two agents in one kitchen with global information. Questions share scenarios, and inference budgets vary across models.

Execution and Decision Criteria

Execution Rules

Each joint step resolves chef interaction, assistant interaction, movement, then appliance processing. Processing includes its start step; finished products await retrieval. Waits are checked at step boundaries. An assistant interaction can retrieve an item placed by the chef in the same step; an explicit wait cannot.

Domains define role permissions, retrieval, and counter access. The canonical processing-chain domain permits retrieval before processing starts and plated-product transfers. Search records require matching states, goals, domains, and execution rules.

Goal-Dependent Costs

Let stps_t^p be the state after tt joint steps of plan p=(p(1),p(2))p = (p^{(1)}, p^{(2)}), and Vg∗(s;D)V_g^*(s; \mathcal{D}) the minimum legal cost to reach goal gg in domain D\mathcal{D}. Short-horizon cost stops at the intermediate goal:

CS(s,p)=min⁡{t≥0:gS(stp)=1}.(1)C_{\mathrm{S}}(s,p) = \min\{t \ge0 : g_{\mathrm{S}}(s_t^p) = 1\}. \tag*{(1)}

If both programs finish at step τp\tau_p before final delivery, long-horizon cost includes optimal continuation:

CL(s,p)=τp+VGL∗(sτpp;D).(2)C_{\mathrm{L}}(s,p) = \tau_p + V_{\mathrm{GL}}^*(s_{\tau_p}^p;\mathcal{D}). \tag*{(2)}

Delivery during the prescribed stage instead uses its first hitting time. Members remain idle after their programs end, while appliances advance each step. A plan that cannot complete under this protocol has Cg=+∞C_g = +\infty, even if abandoning it would permit recovery.

For the four base candidates, i∗=arg mini∈{1,2,3,4}Cg(s,pi)i^* = \mathop{\mathrm{arg\,min}}_{i\in\{1,2,3,4\}} C_g(s,p_i) is unique and satisfies

Cg(s,pi∗)=Vg∗(s;D).(3)C_g(s,p_{i^*}) = V_g^*(s;\mathcal{D}). \tag*{(3)}

State optimality ranges over all legal joint executions, with uniqueness required among the supplied candidates. For three retained plans P′\mathcal{P}', the better-plan statement is true exactly when

min⁡p∈P′Cg(s,p)>Vg∗(s;D).(4)\min_{p\in\mathcal{P}'} C_g(s,p) > V_g^*(s;\mathcal{D}). \tag*{(4)}

Concrete Example

In the long-horizon point024 example (taro–bean–bell-pepper pie), B attains the 13-step optimum; C ends its stage sooner but requires 15 steps overall (Table 3). Inputs contain full programs without costs; Figure 1 is schematic.

PlanProgram IDStageRemainingTotal
Aopt0413821
Bopt0110313
Copt027815
Dopt039615

Table 3. Long-horizon plan costs in joint steps. Stage is the prescribed-program cost; Remaining is the optimal continuation cost. Bold marks the minimum total.

Construction and Dataset Details

Scenario Screening

Algorithm 1 selects states where coordination choices affect completion cost even when all remaining actions are optimized. A constraint cc fixes the order of two required events by one member or the location of a required handoff. Alternatives for the same events form a comparison family. With all remaining actions optimized, their cost penalty is

Δ(s,c)=J(s,c)−VgL∗(s;D).(5)\Delta(s,c) = J(s,c) - V_{\mathrm{gL}}^*(s;\mathcal{D}). \tag*{(5)}

where J(s,c)J(s,c) is the constrained optimal delivery cost. An examined family must contain two proven feasible choices satisfying

Δ(s,c+)=0,Δ(s,c−)≥3.(6)\Delta(s,c^+) = 0,\qquad\Delta(s,c^-) \ge3. \tag*{(6)}

States match by positions, orientations, carried items, material locations, and processing progress. Up to eight comparison families alternate member-specific event orders and both handoff directions at dependency depths 1–3. Alternatives are fixed before cost queries.

Grouping preserves task, domain, goal, comparison family, alternatives, feasibility, penalties, and timing relative to partner events. Materials and processing progress remain fixed across intervening trajectory states; positions may vary. The representative is nearest a certified divergence, with earlier trajectory position breaking ties.

Algorithm 1 Scenario Selection

Input: Reachable states $S$ with domains $\{D_s\}$; trajectory lengths $\{T_i\}$, budget $B = 200$; comparison-family limit $K = 8$

Output: Selected scenarios $Q$

1: $R \leftarrow \varnothing$
2: for $s \in S$ do
3:   $v_s \leftarrow V^{*}_{\mathrm{gl}}(s, D_s)$
4:   $\mathcal{F}_s \leftarrow \operatorname{ComparisonFamilies}(s, K)$
5:   for $F \in \mathcal{F}_s$ do
6:     $J(s,c) \leftarrow$ optimal delivery cost subject to $c$, for $c \in F$
7:     $P_F \leftarrow$ choices in $F$ with proven finite costs
8:     $\Delta(s,c) \leftarrow J(s,c) - v_s$ for $c \in P_F$
9:     if $\exists c^{+}, c^{-} \in P_F$ satisfying {xref}`Eq. 6` then
10:       $R \leftarrow R \cup \{s\}$
11: $(G_i) \leftarrow \operatorname{GroupComparisons}(R)$
12: $(q_i) \leftarrow \operatorname{AllocateQuotas}((G_i), \{T_i\}, B)$
13: $Q \leftarrow \varnothing$
14: for each task $i$ do
15:   $Q \leftarrow Q \cup \operatorname{GreedyCoverage}(G_i, q_i)$
16: return $Q$

Coverage and Candidate Verification

For representative states SiS_i and optimal trajectory length TiT_i, task capacity is

bi=min⁡(∣Si∣,⌊Ti/3⌋).(7)b_i = \min\left(\lvert S_i\rvert, \left\lfloor T_i/3 \right\rfloor\right). \tag*{(7)}

Capacity limits counts, not spacing. Each task receives one scenario; the remaining 170 follow largest-remainder allocation proportional to bi−1b_i - 1, with ties by task ID. Coverage prioritizes event-ordering or handoff directions, family/optimal-arrangement-set combinations, then material stages. Keys include all optimal arrangements; processing progress alone does not define a stage.

Ties use trajectory distance, proximity grade, earlier position, then candidate ID. Grades order certified common optimal prefixes of 0, 1, 2, or ≥3\ge3 steps, then unknown and uncomputed cases. Source levels L1–L6 contribute 10, 5, 40, 35, 50, and 60 scenarios; these are recipe-complexity labels, not calibrated item difficulties.

Base questions contain a unique state-optimal plan and at least two feasible alternatives. Suboptimal gaps are at least one joint step for short goals and two for long goals; short-goal contrasts act before first attainment. Rationales link state facts and program differences to execution outcomes. Parallelism tags require different productive overlap.

Dataset Composition

Table 4 separates questions, scenarios, and structures. Of 200 cross-goal pairs, 64 share programs and the optimum; 136 use different programs. Normalization replaces ingredient/product names while preserving dependencies, states, programs, goals, and domains.

Unit or propertyCount
Recipe tasks30
Decision scenarios200
Short-/long-horizon question pairs200
Total evaluation questions1,200
Base team-plan decision questions400
Optimality-verification variants800
Base short-/long-horizon questions200 / 200
Base candidate plans1,600
Base questions with 3 / 4 feasible plans246 / 154
Normalized question structures181
Normalized paired-scenario structures97
Pairs with identical candidate programs64
Long-horizon questions with stage/total rank reversals15

Table 4. Dataset composition. Candidate and feasibility counts cover base questions; normalized structures cover base questions and short-/long-horizon pairs. Option rotations do not create additional questions.

The public dataset release is tandembench-en-v1.0.0. Original gold positions A–D contain 200, 69, 62, and 69 questions; cyclic rotations balance positions.

Models and Inference Protocols

Native interfaces. Models retain native inputs and readouts: Tacit inference; Decision-2 for Vega/Lux/Kai/Nox; CLM’s prediction head and last-token pooling; Laya state inputs and short labels; KnowLine’s read_twice; and GLiDE’s non-causal text interface. Mammoth, PPLX, and D3 retain calibration; StartLux uses Instant inference. MeTask-Rain-31B uses W4A16 with unmerged LoRA, routing, and calibration.

Weights and reasoning. H2O-Lightning-31B and deck31b use BF16; René uses FP8 dequantized to BF16. H2O disables online FP8 quantization. D3 sizes round 26.09B to 27B; Wald lists total/active parameters. Muse-Glimmer-30B uses low reasoning, and the evaluated Qwen conditions disable thinking.

Response scoring. Truncated, missing-choice, and multiple-choice outputs count as incorrect.

Evaluation Metrics and Full Results

Coverage and Scoring

For NN questions, decision accuracy (denoted Acc⁡4,S\operatorname{Acc}_{4,S} below) compares stable candidate IDs i^qrs\widehat{i}_{qrs} with gold iq∗i_q^*:

Acc⁡4,S=14SN∑q=1N∑r=03∑s=1SI[i^qrs=iq∗].(8)\operatorname{Acc}_{4,S} = \frac{1}{4SN} \sum_{q=1}^{N} \sum_{r=0}^{3} \sum_{s=1}^{S} \mathbb{I}[\widehat{i}_{qrs}=i_q^*]. \tag*{(8)}

Each model contributes 1,600 native or 6,400 general-model base outputs. Uniform selection or a fixed display letter yields 25% DA. Truncated, missing-choice, multiple-choice, and infrastructure-failure outputs count as incorrect and are recorded separately.

Verification uses 800 paired variants. Reporting a better plan is TP when the optimum is removed; selecting any valid concrete plan is TN when it is retained:

VA=12(TPNremove+TNNretain).(9)\mathrm{VA} = \frac{1}{2}\left(\frac{\mathrm{TP}}{N_{\mathrm{remove}}}+\frac{\mathrm{TN}}{N_{\mathrm{retain}}}\right). \tag*{(9)}

VA is balanced binary accuracy and, with equal group sizes, pooled judgment accuracy. Invalid responses count as neither TP nor TN. Uniform selection, always reporting, or always choosing a concrete plan yields 50% VA. TN does not require selecting the optimal retained plan.

Full Model Comparison

Table 5 reports results for the ten general-purpose models evaluated on the English inputs.

RankModelAvg. output tokensDA ↑VA ↑
1Qwen3.8-27B865282.8071.88
2Gemma4-31B107273.1157.03
3Gemma4-26B-A4B151660.5853.66
4Muse-Glimmer-30B141559.6460.35
5Qwen3-32B92546.3349.45
6Gemma4-E4B256445.0351.16
7Qwen3-8B109241.0349.85
8Ministral 3 14B Instruct176239.8951.45
9Llama3.3-70B66139.5048.92
10Qwen3-0.6B2517.1940.61

Table 5. Table 5: DA and VA (%) and mean output length for ten general-purpose models, ranked by DA. Unresolved answers count as incorrect; shading marks the top three and bold marks the best score in each metric.

Interpretation Scope

The benchmark evaluates centralized plan decisions from complete global information; it does not test communication or adaptation to independent partners. Option rotations balance answer positions, while program length and waiting patterns remain observable cues. Models use their native interfaces and inference budgets, so results compare evaluated systems under their respective protocols.

Costs and Goal Comparisons

Decision Costs and Choice Stability

Figure 7 covers 40 native models and 10 general-purpose models; Table 6 lists six native models. For feasible selections, regret is

Figure 7. Execution consequences of selected plans. Left: mean extra joint steps above the state optimum among feasible selections. Right: the percentage of all outputs selecting a noncompleting plan; invalid responses form a separate category.

ModelFeasible (%)Noncompletion (%)RegretConsistency (%)Range (pp)
Basal-Omni-1.0-Ultra99.750.251.4431.7511.25
KnowLine-31B-Gen199.940.061.6845.2511.75
Quyet-1.0-Large98.751.251.2237.2525.75
PPLX-Decider-v1.1-27B100.000.001.7437.009.00
Torchcast-4B-v1.295.694.311.2543.252.00
CLM-8B78.5621.444.5092.001.00

Table 6. Execution outcomes and choice stability. Regret is mean extra joint steps among feasible selections. Consistency is the percentage of questions selecting the same plan across all rotations; Range is maximum minus minimum rotation DA.

Rq=Cgq(sq,p^iq)−Vq∗(sq;Dq).(10)R_q = C_{g_q}(s_q,\widehat{p}_{i_q}) - V_q^*(s_q;D_q). \tag*{(10)}

Regret includes optimal choices at zero. Feasible, noncompletion, and invalid rates use all outputs and sum to one. The 246 questions with noncompleting candidates give exposure counts of 984 (S=1S=1) or 3,936 (S=4S=4).

Position sensitivity. For general-purpose models, averaging four samples per position yields a median DA range of 7.34 points. Averaging the four within-sample ranges for each model and taking the median across models gives 8.47 points. Consistency tracks the same candidate across rotations within each sample index. Both include generation randomness (Figure 6; Table 6).

Goal Comparisons

Figure 8 compares 64 pairs sharing initial states, programs, and optimal choices, with different goals and execution rules. Mean short/long DA is 49.58/46.75% for 40 native models and 56.68/48.04% for 10 general-purpose models, weighting models equally.

Figure 8. Short- and long-horizon DA with identical candidate programs but different goals and execution rules. Models are ordered by decreasing long-horizon DA within each family.

A disjoint set of 15 long-horizon questions offers feasible alternatives with τi<τi∗\tau_i < \tau_{i^*} but

CL(s,πi)>CL(s,πi∗)C_{\mathrm{L}}(s,\pi_i) > C_{\mathrm{L}}(s,\pi_{i^*}). Table 7 reports their DA and selection rates. Basal selects these earlier-finishing but higher-total-cost alternatives in 22/60 presentations; Qwen3.8-27B in 13/240, including rotations and samples. AUROC is 0.537/0.598 (GLiDE); Basal scores 0.520. Panel (a) averages all pairs. For pairs reporting twice in (b), mean shifts are +3.26 points for Basal (240 pairs), +2.05 for PPLX (504), and +0.55 for Tacit-2B (1,557). Figure 4(b) gives unconditional choices.

ModelSame-plan shortSame-plan longConflict DAFast-stage choice
Basal-Omni-1.0-Ultra72.2765.2335.0036.67
KnowLine-31B-Gen172.2759.7755.000.00
Quyet-1.0-Large56.6461.3348.3310.00
PPLX-Decider-v1.1-27B58.2057.8145.0035.00
Torchcast-4B-v1.255.4747.6673.3315.00
CLM-8B4.305.860.0066.67

Table 7. DA and fast-stage choice rates (%). Same-plan columns compare short and long goals with identical programs. Conflict columns cover a separate set where an earlier stage finish can require more total steps; Fast-stage choice is the rate of selecting such alternatives.

Coordination-Group Analysis

Figure 9 compares five native models across overlapping coordination groups defined by the union of candidate tags.

Figure 9. DA (%) across coordination groups. Darker cells indicate higher accuracy. A question may belong to more than one group.

Work order covers processing/supply order (300 questions, 156 scenarios); handoffs cover transfer locations/conditions (376, 188); parallel work covers productive overlap (355, 182); resource constraints cover carrying/counter capacity (141, 77). These tags capture intended coordination contrasts. Because groups may overlap and differ in task and goal composition, their scores describe performance on these subsets rather than isolated mechanism effects.

Probability Analysis

Figure 10 uses 128,000 outputs: 40 models, 400 pairs, four rotations. Probabilities are aligned by option letter or text, correcting sums within 10−410^{-4} rounding tolerance and excluding interface-specific confidence fields.

Figure 10. Better-plan probabilities. (a) AUROC versus mean shift, colored by VA. (b) Shifts for pairs reporting twice; boxes show quartiles and central lines the median. (c) AUROC by rotation. Shifts are removed minus retained, in percentage points.

AUROC ranks removed- against retained-optimum examples by better-plan probability, with half credit for ties. Median/maximum AUROC is 0.537/0.598 (GLiDE); Basal scores 0.520. Panel (a) averages all pairs. For pairs reporting twice in (b), mean shifts are +3.26 points for Basal (240 pairs), +2.05 for PPLX (504), and +0.55 for Tacit-2B (1,557). Figure 4(b) gives unconditional choices.

Task Weighting

ModelItem-weightedTask-macro
Basal-Omni-1.0-Ultra60.7565.45
KnowLine-31B-Gen162.5066.96
Quyet-1.0-Large60.7561.00
PPLX-Decider-v1.1-27B59.4462.90
Torchcast-4B-v1.258.9456.95
CLM-8B15.4410.52

Table 8. DA (%) under two weighting schemes. Item-weighted gives equal weight to questions; Task-macro averages task-level accuracies equally.

Task weighting and uncertainty. After averaging four rotations per question, we bootstrap 200 scenarios with both goals retained (10,000 replications; seed 20261027). KnowLine minus Basal is 1.75 points: the percentile 95% interval is [-1.56, 5.00], or [-2.10, 5.56] when resampling the 30 tasks. Intervals condition on the recorded runs; structure IDs record further cross-scenario relationships.

References

  1. [1]Saaket Agashe, Yue Fan, Anthony Reyna, and Xin Eric Wang. 2025. LLM-coordination: Evaluating and analyzing multi-agent coordination abilities in large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 8053–8072.
  2. [2]Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramraksha, Daniel Tran, Joanne Truong, John Turner, Eric Undersander, and Tsung-Yen Yang. 2025. PARTNR: A benchmark for planning and reasoning in embodied multi-agent tasks. In International Conference on Learning Representations.arxiv.org/abs/2411.00081
  3. [3]Yang Chen, Ye-Xin Xie, Lirong Che, Danyang Peng, Yuzhe Yang, Peiwen Lin, Xu Cao, Chuang Wang, Lei Yuan, Jian Su, and Lan-Zhe Guo. 2026. CoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning. Preprint, arXiv:2608.28266.arxiv.org/abs/2608.28266
  4. [4]Simon Dold, Malte Helmert, Jakob Nordström, Gabriele Röger, and Tanja Schindler. 2025. Pseudo-boolean proof logging for optimal classical planning. Preprint, arXiv:2504.18443.arxiv.org/abs/2504.18443
  5. [5]Feiyu Duan, Jiayu Lin, Jia Wang, Jun Xiang, Jialiang Wu, Xinnong Zhang, Hanqi Yan, Siyuan Wang, and Zhongyu Wei. 2026. General decision models: Benchmarking and insights beyond J̌ev. Preprint, arXiv:2610.03935. Version 2, 12 October 2026.arxiv.org/abs/2610.03935
  6. [6]Jonathan Geuter, Andrew Cheng, and David Alvarez-Melis. 2026. Releasing the inner Jev: A simple recipe for open decision models. Preprint, arXiv:2610.17205. Version 1, 12 October 2026.arxiv.org/abs/2610.17205
  7. [7]Harsha Kokel, Michael Katz, Kavitha Srinivas, and Shirin Sohrabi. 2025. ACPBench: Reasoning about action, change, and planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26559–26568.
  8. [8]Yifan Lan, Zhongye Liu, Yurui Chang, Hanyu Wang, Yongkang Du, Lu Lin, and Jinghui Chen. 2026. DecisionRobust: Benchmarking robustness for JEV style decision models across text and vision. Preprint, arXiv:2610.19118. Version 1, 13 October 2026.arxiv.org/abs/2610.19118
  9. [9]Songtuan Lin, Conny Olz, Malte Helmert, and Pascal Bercher. 2024. On the computational complexity of plan verification, (bounded) plan-optimality verification, and bounded plan existence. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20203–20211.
  10. [10]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Ao-han Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, and 3 others. 2024. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations.arxiv.org/abs/2308.03688
  11. [11]Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. AgentBoard: An analytical evaluation board of multi-turn LLM agents. In Advances in Neural Information Processing Systems, volume 37.arxiv.org/abs/2401.13178
  12. [12]Zhao Mandi, Shreeya Jain, and Shuran Song. 2024. RoCo: Dialectic Multi-Robot Collaboration with Large Language Models. In IEEE International Conference on Robotics and Automation.

Paper details

vlnc:W_hwzn9ne4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents