ABACUS: Program-Guided Multistep Reasoning over Frozen Judgment Models

Introduction

Judgment models score finite candidate outcomes given evidence and a question. Their local decisions provide useful building blocks for reasoning: recognizing movements, identifying claims about truthfulness, or distinguishing observations from temporal boundaries. Complete solutions additionally require accumulating movements, propagating truth values, or combining interval constraints. These computations often share a reusable structure across instances.

This motivates a general research question: can offline program construction equip frozen judgment models with strong multistep reasoning? Keeping the judgment model fixed places the design effort in the reasoning procedure: which local questions to ask, what their answers represent, and how to compose those answers. A useful procedure connects the model’s semantic decisions to the state updates and constraints required by the task.

We introduce ABACUS, a framework for program-guided multistep reasoning over frozen judgment models. A generative teacher jointly designs reusable judgment schemas and a task-level composition program offline. For each new input, the judgment model scores the declared local outcomes, and the saved program computes the answer. The same procedure is reused with fixed model parameters.

The central design problem is to find an intermediate representation that the model can recognize and the program can use. In navigation, distinguishing turns from steps enables position and heading updates. In a truth chain, identifying the form of each claim enables Boolean propagation. Joint construction lets the teacher design this semantic vocabulary together with the computation that consumes it.

We instantiate ABACUS with Kev, which returns probabilities over declared outcomes for typed questions ([10]), and GLM as the offline teacher. To our knowledge, ABACUS is the first framework to jointly construct reusable finite judgment schemas and composition programs specifically for Kev.

We study three allocations of procedure design. In human-designed workflows, a designer specifies the local questions, state, and update rules. In task-specific interfaces, humans expose domain objects such as people, tasks, and pairs; the teacher designs semantic judgments and a scoring program. With a shared interface, humans provide common constructors and execution constraints; the teacher selects the units, designs the judgments, and writes the solver. Each allocation makes the human–teacher division of procedure design explicit. We evaluate each with frozen Kev on its corresponding tasks.

Our main construction procedure generates complete joint proposals from a task description, four solved examples, and the shared interface. Each proposal uses the same task specification; the first passing structural, read-sensitivity, and solved-example checks becomes the reusable artifact.

The shared-interface stage is our main empirical focus, with separate construction and deployment studies. The construction study evaluates three independently generated programs per task on the selected web-of-lies, navigation, and temporal-sequence tasks. All nine are admitted, with 100% accuracy over 450 executions of 150 development inputs. The deployment study uses three separately frozen programs selected with a reference-agreement refinement step and evaluates them on 600 held-out inputs. With the same online Kev model, ABACUS raises macro accuracy from 64.00% to 99.67%. It also achieves comparable observed accuracy to Qwen3.5-4B’s unrestricted response policy at substantially lower online latency: 0.255 versus 2.150 seconds per question.

Our contributions are:

• Program-guided reasoning over frozen judgment models. ABACUS combines finite-candidate semantic decisions with reusable computation, separating offline procedure design from online reasoning.

• Joint design of judgments and computation. A shared interface lets the teacher construct both the intermediate semantic vocabulary and its composition program, with explicit checks for admission and reuse.

• Strong reasoning with a compact, fixed model. Our Kev implementation connects three allocations of procedure design to their evaluations, culminating in near-perfect held-out accuracy and a strong accuracy–latency tradeoff against direct Kev and seven Qwen3.5-4B answer budgets.

Related Work

Programs for language-model reasoning. Chain-of-thought prompting represents intermediate reasoning in text ([15]). PAL and Program of Thoughts generate programs that delegate computation to an interpreter ([6, 3]). Binder incorporates model operations into symbolic execution ([5]). These approaches establish the value of combining learned interpretation with explicit computation. ABACUS constructs reusable task-level programs whose semantic operations are supplied by a frozen model through finite-candidate judgments.

Reusable tools and structured composition. LATM separates tool creation from use and pairs capable tool makers with lighter users (Cai et al., 2024). Analytica combines grounded propositions through soft reasoning trees ([4]); TACs learn typed probabilistic workflows ([8]); NePTune composes perceptual scores and program execution ([7]); Deep-ProbLog integrates neural predicates with probabilistic logic ([9]). ABACUS develops reusable synthesis around a finite-candidate judgment interface: the artifact jointly defines semantic questions and their executable composition, and the frozen model supplies the declared judgments.

Compositional evaluation. BIG-Bench Hard (BBH) provides diverse language reasoning tasks ([13]), while MuSR provides narrative problems requiring multiple reasoning steps ([12]). We study truth propagation, navigation, and temporal constraints through the shared interface, and use MuSR for supporting workflow and task-specific construction studies. The explicit intermediate decisions also make the computation inspectable, complementing research on the faithfulness of generated reasoning traces ([14]).

Method

Frozen Judgment Models and Reusable Programs

An instance is $x = (e, q, \mathcal{A})$, comprising evidence, a question, and finite answer choices. A frozen judgment model $J_\theta$ maps evidence and a local question to a distribution over declared candidate outcomes; its parameters $\theta$ remain fixed. Task identity $\tau$ selects a reusable artifact. Each local judgment takes the form

where $e_i$ is the specified evidence view, $q_i$ a local question, and $\mathcal{Y}_i$ a finite set of named outcomes with descriptions. We instantiate $J_\theta$ with Kev in our experiments. The reusable artifact is

with input constructors $C_\tau$, judgment schemas $R_\tau$, and a composition program $g_\tau$.

Let $H_\tau(x)$ denote structural information exposed by the interface and $z^*(x)$ the intended local facts. Construction seeks a sufficient decomposition,

The local facts must also be recognizable by the judgment model from their assigned evidence. The teacher therefore designs the semantic representation and its composition program together.

Three Levels of Procedure Design

Table 1 describes three allocations of procedure design. Each combines local judgments with explicit computation, using an inference rule suited to its interface. The implementations below use GLM as the teacher and Kev as the judgment model.

Human-designed workflow. A designer specifies the local questions, state variables, and update rules. In object placement, judgments about movements and observations update object locations and actors’ beliefs. GLM has no construction role. This stage tests the capability of Kev within a carefully specified reasoning procedure.

Task-specific interface. Humans expose the task’s objects and query scopes, such as people, tasks, and unordered pairs in team allocation. GLM generates the semantic attributes, candidate labels, and scoring program. Kev evaluates the instantiated judgments, and the program combines them to rank candidate answers. This stage delegates the core judgment and scoring design to the teacher.

Shared interface. Humans supply common input constructors, execution tools, and admission checks. Given the task description and solved examples, GLM chooses the units, specifies the semantic questions, and generates their composition program. This stage is the main ABACUS construction procedure, detailed below and illustrated in Figure 1.

A Shared Interface for Judgment and Execution

Evidence and structural information. The shared interface provides constructors for sentences, enumeration items, capitalized-token candidates, dates, times, and answer choices. Each constructor yields an ordered list of occurrence handles and a small set of lexical operations: extracting numbers, dates, and times; checking normalized literal equality; and locating literal mentions. The frozen model supplies semantic interpretation through declared judgments. Structural information and semantic decisions then provide distinct inputs to the composition program.

Judgment schemas. Each schema defines a question, candidate outcomes and descriptions, an evidence view, and a scope over zero, one, or two unit lists. A local view supplies the referenced unit; a full view supplies the instance context. Instantiating a schema over its selected lists produces a finite query plan:

Questions that share evidence are grouped into requests to the judgment model. The complete query plan is instantiated and evaluated before the program consumes the recorded judgments.

Composition. The main decoder selects the most likely local outcome and executes the saved program:

The program can update state, branch on local outcomes, enumerate assignments, and apply numerical or combinatorial operations. It returns an answer from $\mathcal{A}$. The same artifact is reused across instances of task $\tau$.

Joint Construction and Admission

The teacher receives the task description, four solved examples, and the shared interface. It jointly proposes the selected input units, judgment schemas, and composition program. ABACUS uses independent complete proposals: every proposal receives the same base task specification, and the first passing proposal is retained.

Admission screens executability and observable candidate behavior through three components. Structural and runtime checks validate the schema, query plan, permitted operations, and returned answers on development inputs. Read-sensitivity checks evaluate each candidate on 20 development inputs with 64 randomized judgment assignments per input, measuring whether its answers respond to its semantic inputs. Solved-example checks obtain local annotations in separate teacher calls and require correct solutions to at least three of the four examples. The annotator receives evidence and local questions; the final target answers are used by the admission check.

Each trajectory permits up to four complete proposals within a 256,000-token teacher input-plus-output budget that includes annotation. The retained artifact is frozen for evaluation. Algorithm 1 in the appendix gives construction and online execution in a single procedure. The runtime configuration and quantitative admission thresholds are specified in Appendix § 10.

Deployment program selection

The separate deployment study adds reference-agreement refinement before freezing. For an admitted candidate, Kev and the teacher independently label local questions on ten development inputs. A judgment type with at least five observations and agreement below 0.8 triggers one proposed revision, which is retained if it passes admission. The deployed truth-chain program uses this revision; navigation and temporal sequences retain their initial admitted candidates. Both studies use the online decoder in Eq. (5).

A Concrete Reasoning Procedure

Consider the navigation instruction “Take 2 steps.” A constructed judgment asks which action the sentence describes, with candidates for forward or backward movement, lateral movement, turns, and other statements. Kev supplies the action label, while lexical extraction supplies the distance. The program maintains a state $(x,y,h)$ consisting of position and heading. If $v(h)$ is the unit vector in heading $h$, a forward move of distance $d$ gives

A right turn updates the heading by a quarter turn. After the final instruction, the program checks whether the position equals the origin and selects the corresponding answer choice.

Figure 1 traces a rectangular route. The same procedure handles different step lengths, turn sequences, and numbers of instructions by instantiating the saved schemas on the new input. The local questions carry the semantics; the accumulated state carries the multistep computation. Table 2 summarizes the constructed interfaces for all three tasks.

Task	Semantic decisions supplied by Kev	Computation supplied by the program	Judgments
Web of lies	Truth or lie in each claim; affirmative or negative answer meaning	Propagate the current truth state and match the final query	8.0
Navigation	Forward, backward, lateral movement, or turn; answer meaning	Accumulate coordinates and heading; check return to the origin	8.8
Temporal sequences	Waking time, observed activity, closing time, or other statement	Compare candidate intervals with event and boundary constraints	9.9

Online Execution and Reuse

For a fixed artifact, online work consists of query construction, grouped judgment requests, and program execution:

The reported client latency measures this complete online procedure. Construction cost is accounted for separately; over $N$ uses, average cost in a common additive unit is $C_{\mathrm{build}}/N + \mathbb{E}[C_{\mathrm{online}}]$. This makes reuse an explicit property of the deployment workload.

Experiments

We evaluate the framework with Kev as the frozen judgment model and GLM as the construction teacher. The experiments examine the three allocations of procedure design in § 3.2. The first two study manual workflows and task-specific teacher construction on narrative problems. Our main study evaluates shared-interface construction on three selected BBH tasks and measures held-out deployment against direct Kev and Qwen3.5-4B. Each design regime uses its corresponding task and cohort.

Human-Designed Workflow

Explicit event and belief tracking. For object placement, a human designer specifies the questions, event ledger, object locations, and belief updates. Kev judges movements and observations, and the workflow propagates their effects through the story. Across 208 questions in 52 stories, the workflow reaches 162/208 (77.9%), compared with direct Kev’s 100/208 (48.1%). The 29.8-point gain demonstrates the value of placing Kev’s judgments inside an explicit stateful computation. Appendix § 13 gives the workflow and evaluation details.

Task-Specific Teacher Construction

Teacher-designed semantic interfaces. On MuSR team allocation and murder mystery (Sprague et al., [12]), humans provide domain objects and query scopes. GLM supplies the semantic attributes, candidate values, and scoring program. Kev evaluates the resulting local judgments, and factorized local sampling selects the candidate with the greatest estimated winning frequency. Team allocation reaches 187/200 (93.5%) versus 130/200 (65.0%) for direct Kev. Murder mystery reaches 152/200 (76.0%) versus 111/200 (55.5%). These gains of 28.5 and 20.5 points show that useful reusable procedures can be constructed by the teacher within a task-specific interface. Each task uses its first registered build and 200 evaluation stories; Appendix § 13 details the human inputs and generated artifacts.

Shared-Interface Construction

One interface for distinct reasoning procedures. The shared interface gives GLM responsibility for choosing input units, semantic questions, and their composition. We study this regime through independently generated artifacts and a separate held-out deployment evaluation.

Reusable Program Construction

The construction study covers web of lies, navigation, and temporal sequences (Suzgun et al., [13]). Each task has 50 development inputs and four solved examples. For each task, we independently construct three programs using the joint-proposal procedure. The same development inputs are evaluated for each artifact.

All nine trajectories admit their first proposal. Each task obtains 150 correct executions from three programs evaluated on 50 distinct inputs, for 450/450 overall (Table 3). Removing the four shown examples per task yields 414/414 executions on 138 distinct development inputs. All executions complete through their constructed programs. The nine trajectories consume 322,117 teacher tokens in total, including annotations.

Task	Admitted	Correct	Seconds
Web of lies	3/3	150/150	0.245
Navigation	3/3	150/150	0.265
Temporal sequences	3/3	150/150	0.252
All tasks	9/9	450/450	0.254

The generated interfaces encode compact task computations. Truth-chain programs identify assertions and propagate truth values. Navigation programs recognize actions and maintain position and heading. Temporal programs identify event roles and compare candidate intervals with the resulting constraints. These are distinct procedures constructed through the same shared interface.

Held-out deployment protocol

Programs and data. The deployment study uses a separate frozen artifact for each of the same three tasks. Program selection combines joint proposals and admission with a local reference-agreement refinement step on ten development inputs. Navigation and temporal sequences retain their initial admitted proposals; the truth-chain program retains an agreement-refined proposal. Appendix § 11 specifies this selection protocol. The held-out cohort contains 200 distinct-context questions per task, 600 in total, after excluding development inputs and duplicate contexts. Programs are frozen before held-out inference.

Models and answer policies. The construction teacher is GLM-5.3-Flash ([16]). Online judgments use Kev4B, built on Qwen3.5-4B-Base ([10]). Direct Kev selects the final answer from the full problem; a second Kev control uses a teacher-written task prompt. The generative comparison uses post-trained Qwen3.5-4B ([11]), with the same four solved examples. Seven answer policies request a final answer only, concise reasoning targeting 16/32/64/128/256 words, or free reasoning. All seven policies use Qwen’s non-thinking mode and request a structured final answer. Here, free reasoning means an unrestricted explanation within the 8,192-token output capacity. Actual generated output and elapsed time are charged.

Measurement and statistics. Each request runs on an individual H20 GPU with BF16 weights and a local client. Cross-question prefixes are cold; shared evidence can be reused within a question. We repeat measurement five times, keeping programs fixed throughout. The full grid contains 30,000 predictions across ten policies. We report equal-task macro accuracy and mean online client latency. Paired crossed bootstrap resamples questions within tasks and collection rounds for 5,000 draws. Appendix § 12 gives the serving, parsing, and accounting details.

Policy	Accuracy (%)	Seconds
Kev direct	64.00	0.022
Kev teacher prompt	71.67	0.053
Qwen4B final only	68.80	0.108
Qwen4B brief 32	79.57	0.294
Qwen4B brief 64	88.70	0.436
ABACUS	99.67	0.255
Qwen4B brief 128	94.83	0.689
Qwen4B free	99.07	2.150

Near-Perfect Accuracy with Compact Judgments

ABACUS reaches 99.0% accuracy on web of lies, 100.0% on navigation, and 100.0% on temporal sequences (Figure 2). Direct Kev obtains 59.5%, 63.0%, and 69.5%, respectively. The gains are 39.5, 37.0, and 30.5 percentage points, achieved with fixed Kev weights and saved task programs.

Across tasks, ABACUS achieves 99.67% macro accuracy [99.17,100.00], compared with 64.00% for direct Kev and 68.80% for final-answer-only Qwen3.5-4B. A teacher-written direct Kev prompt reaches 71.67%. The reusable program therefore yields large gains over both direct answer selection and the tested task prompt. All 3,000 ABACUS executions use the saved programs, and offline replay reproduces their recorded outputs. The local judgments also support a precise diagnosis of the two erroneous inputs (Appendix § 12.§ 1).

Near-Perfect Accuracy at Low Online Latency

ABACUS provides a strong operating point at 99.67% accuracy and 0.255 seconds per question. It exceeds Qwen’s 32-word policy at slightly lower latency and its 64-word policy at substantially lower latency. We additionally compute an empirical envelope that allows the external policies to use known task identity, as ABACUS does. At the ABACUS latency, this task-conditioned envelope reaches 89.90%; ABACUS leads by 9.77 percentage points, with a paired 95% interval of [7.50,11.73]. It is a vertex of the all-policy task-conditioned envelope in all 5,000 bootstrap draws. This post-hoc analysis uses the existing predictions and timing records; Appendix § 11.§ 3 gives the mixture and the original task-independent comparison.

Free Qwen reasoning reaches 99.07% at 2.150 seconds per question. ABACUS combines comparable observed accuracy with 8.43 times faster online execution. Its observed accuracy difference is 0.60 points, with a paired interval of [−0.13,1.37]. The task-level results show complementary strengths: Qwen free reaches 99.9% on web of lies and 97.3% on navigation, while ABACUS reaches 99.0% and 100.0%; both obtain 100.0% on temporal sequences. Across the tested answer budgets, reusable computation delivers a strong accuracy–latency tradeoff. Appendix § 14 accounts separately for online latency, token work, and offline construction.

Discussion

Structure as a capability multiplier. With fixed Kev weights, reusable programs lift direct-answer accuracy from 64.00% to 99.67% on the selected deployment tasks. Local decisions acquire precise computational roles: update a state, propagate a constraint, or evaluate a candidate. The teacher contributes a reusable procedure that connects these decisions across new instances.

An interface that matches the model. Jointly constructing judgments and the solver makes the intermediate representation a central design choice. The three BBH programs pair a compact semantic vocabulary with explicit state updates and constraints. Their saved judgments make that division inspectable: the reference-label diagnostic identifies the exact local decision that changes the two incorrect outputs.

Reuse and deployment. Online execution combines compact semantic judgments with saved computation. Separate construction and deployment studies characterize artifact generation and online performance. ABACUS retains a strong operating point when baseline policies also use task identity to allocate their budget, supporting reusable procedure design as a practical way to extend compact judgment models.

Conclusion

ABACUS equips frozen judgment models with multistep reasoning by jointly constructing local judgment schemas and reusable programs. Three design allocations specify the roles of humans and the teacher. With Kev, the main construction procedure yields nine admitted programs on three selected tasks. A separate study evaluates three programs frozen after reference-agreement refinement on held-out inputs: 99.67% accuracy and an 8.43-fold online speedup over free Qwen3.5-4B at comparable observed accuracy. Reusable programs thus offer a practical route to strong reasoning with compact, fixed judgment models.

Impact Statement

Reusable reasoning programs can make compact models more useful and their computation easier to inspect. Explicit judgments and execution traces support auditing, error analysis, and task-specific validation. Applications should evaluate the accuracy, biases, and consequences of the local decisions and the resulting outputs in their intended setting.

References

Experimental Design

The experiments examine program-guided reasoning with a frozen Kev model through joint construction, held-out deployment, and supporting narrative workflows. Table 5 states the program-selection and evaluation protocol for each. The three-task deployment study evaluates the programs selected by the procedure in Appendix § 11.

Study	Program selection	Evaluation
Joint construction	Independent complete proposals; structural, sensitivity, and example checks; first passing proposal retained	Three independent constructions per task, three tasks, 50 development inputs per task: 450 executions of 150 distinct inputs
Held-out deployment	Joint proposals and admission followed by reference-agreement refinement on ten development inputs	Three frozen programs, 200 held-out inputs per task, five collection rounds: 3,000 ABACUS predictions
Task-specific construction	Teacher-generated attributes and scoring programs; first registered build	MuSR team allocation and murder mystery, 200 stories each
Manual workflow	Human-designed event and belief updates	208 questions in 52 object-placement stories

Joint Construction and Online Execution

Algorithm

Algorithm 1 expresses the main joint-construction procedure and its online use. Each teacher proposal defines both the local questions and the computation that composes their answers. The four solved examples are available to the constructing teacher. Local-annotation requests contain the evidence, question, and candidate descriptions, and the admission evaluator compares the resulting program answers with the example targets.

Algorithm 1 ABACUS: joint program construction and online execution

Require: Task description $D$, solved examples $E$, development inputs $V$, shared interface $\mathcal{U}$, proposal budget $K$, teacher budget $B$

1. $\mathcal{P} \leftarrow\varnothing$; initialize budget accounting

2. for $k = 1, \ldots, K$ while teacher budget remains do

3. Generate $(C_k, R_k, g_k)$ jointly from $(D, E, \mathcal{U})$ with independent proposal randomness

4. Validate structural and runtime compatibility on $V$; continue to the next proposal if it fails

5. Evaluate read sensitivity with randomized local labels; continue to the next proposal if it fails

6. Obtain teacher annotations for the local judgments on $E$

7. Execute $g_k$ with those annotations and score example answers

8. if all admission criteria are satisfied then

9. Freeze $\mathcal{P} \leftarrow(C_k, R_k, g_k)$; stop construction

10. end if

11. end for

12. For each instance $x = (e, q, \mathcal{A})$:

13. if $\mathcal{P} = \varnothing$ then

14. Obtain a direct Kev judgment over $\mathcal{A}$

15. else

16. Construct units and instantiate all questions in $\mathcal{I}(x)$

17. Group shared evidence and obtain $p_i = J_{\theta}(e_i, q_i, \mathcal{Y}_i)$

18. Select $\hat{z}_i = \operatorname*{arg\,max}_{z \in\mathcal{Y}_i} p_i(z)$ for each question

19. Return $g(H(x), \hat{z})$; use direct Kev if program computation fails

20. end if

Admission and Resource Settings

The constructing teacher uses temperature 1 and top-p 0.95. Three independent runs generate complete proposals from the same task specification. The total budget includes input and output tokens for both generation and local annotation. Table 6 gives the shared configuration used for the reported construction study.

Setting	Value
Solved examples per task	4
Development inputs per task	50
Maximum complete proposals	4
Total teacher token budget	256,000
Sensitivity inputs / draws per input	20 / 64
Maximum constant-output fraction	0.5
Required solved examples	3 of 4
Retained units per ordinary list	First 40
Maximum judgments per input	60
Outcomes per judgment	2–12
Solver time limit	10 seconds

The read-sensitivity check measures answer variation under synthetic local judgments. Runtime checks exercise the selected constructors and operations on development inputs. The example check evaluates the complete composition using teacher-annotated local values. These checks select an executable artifact whose use of semantic decisions and example behavior are observable.

Randomness settings. Construction runs 1–3 use random seeds 17, 29, and 43; Table 2 illustrates the first run. Deployment measurement rounds use seeds 17, 29, 43, 71, and 101, with the programs fixed throughout.

Independent Construction Results

Every task admits its first proposal in all three construction runs. Table 7 reports the complete construction-study results. All 450 executions return program-backed answers, with zero host-level direct-answer fallbacks. Excluding the four construction examples per task leaves 46 inputs per task and run, all answered correctly. These inputs belong to the development cohort used for construction checks.

Run	Task	Correct	Mean seconds	Teacher tokens	Proposals
1	Web of lies	50/50	0.195	36,817	1
1	Navigation	50/50	0.246	37,403	1
1	Temporal sequences	50/50	0.225	25,932	1
2	Web of lies	50/50	0.332	55,101	1
2	Navigation	50/50	0.257	31,197	1
2	Temporal sequences	50/50	0.270	27,865	1
3	Web of lies	50/50	0.208	38,190	1
3	Navigation	50/50	0.293	45,272	1
3	Temporal sequences	50/50	0.262	24,340	1

Held-Out Deployment Study

Program Selection

The deployment study constructs one program per task using joint proposals, the admission checks, and a reference-agreement refinement step. It permits up to three initial proposals, retaining the first admitted candidate. The candidate is evaluated on ten additional development inputs using both Kev judgments and independently requested teacher-reference annotations. A judgment type with at least five observations and reference agreement below 0.8 triggers one proposed revision. A revision passing structural, sensitivity, and example checks is retained; otherwise the admitted candidate remains selected. Each generation request has an output capacity of 262,144 tokens.

The three deployed artifacts consist of first-admitted navigation and temporal programs and a revised truth-chain program. In the truth-chain construction cohort, the answer-meaning judgment’s reference agreement increases from 10/20 to 20/20 after refinement. All three programs are frozen before evaluation on the 600 held-out inputs. This study measures their deployment behavior; the independent joint-proposal construction study is specified in Appendix § 10.

The deployed programs use sentence-level semantic judgments and explicit computation. Web of lies uses eight judgments per held-out input, navigation averages 8.555, and temporal sequences averages 10.025. Their respective maximum counts are 8, 13, and 11. All retained ordinary unit lists in this cohort are shorter than the 40-unit limit.

Complete Accuracy and Latency Results

Tables 8 and 9 report every task and answer policy. Table 10 reports macro averages and paired confidence intervals. Each task has 200 distinct held-out inputs, and each policy is evaluated in five collection rounds. Task averages receive equal weight.

Accuracy–Latency Envelopes

For policy $m$, let $c_m$ be mean client latency and $a_m$ macro accuracy. A task-independent randomized mixture has

We compute the upper concave envelope in linear latency and display the resulting segments on logarithmic axes. The external envelope contains the two Kev controls and seven Qwen policies. The full envelope additionally includes

ABACUS.

Under this original task-independent protocol, the external envelope reaches 80.66% at the ABACUS latency. The difference is 19.01 percentage points [16.76,21.08]; ABACUS is an all-policy envelope vertex in every measurement round and bootstrap draw.

Task-conditioned empirical comparison. Known task identity also permits a separate policy mixture for each task. Using per-task accuracy $a_{\tau m}$ and latency $c_{\tau m}$, define

We compute this post-hoc empirical reference from the unrounded records, keeping all nine external policies. At $b = 0.255085$ seconds, its optimum is 89.8978%: Qwen brief 128 for web of lies; Kev teacher prompt for temporal sequences; and a navigation mixture of 87.9424% Qwen brief 32 and 12.0576% direct Kev. The navigation mixture obtains 79.0935% at 0.239703 seconds. The three task costs average to the ABACUS budget.

Paired crossed bootstrap resamples collection rounds and questions within each task for 5,000 draws, retaining all policies and reoptimizing the envelope in each draw. The ABACUS advantage is 9.77 percentage points [7.50,11.73]. The all-policy task-conditioned envelope includes ABACUS as a vertex in all draws. The observed accuracy difference relative to free Qwen reasoning remains 0.60 points [-0.13,1.37], with a mean latency ratio of 8.43. The uncertainty estimates are conditional on the stated tasks and fixed artifacts.

Policy	Lies	Temporal	Navigation	Macro
Kev direct	59.50	69.50	63.00	64.00
Kev teacher prompt	56.50	94.00	64.50	71.67
Qwen4B final only	52.30	89.80	64.30	68.80
Qwen4B brief 16	64.30	89.60	74.60	76.17
Qwen4B brief 32	68.40	89.00	81.30	79.57
Qwen4B brief 64	85.00	92.90	88.20	88.70
ABACUS	99.00	100.00	100.00	99.67
Qwen4B brief 128	96.60	96.10	91.80	94.83
Qwen4B brief 256	98.70	98.50	95.80	97.67
Qwen4B free	99.90	100.00	97.30	99.07

Models, Measurement, and Execution

Models and serving. The teacher is the GLM-5.3-Flash service [16]. Kev uses Qwen3.5-4B-Base as its backbone, and the generative comparator is post-trained Qwen3.5-4B. Both online services use BF16 weights on individual H20 GPUs with local clients. Kev’s probability temperature is 2.40605. Its serving implementation uses CUDA graphs and shared-state caching; Qwen uses vLLM with a 262,144-token context limit. Input truncation and date-fact augmentation are disabled for Kev. The two systems use their respective serving implementations on the same GPU class.

Qwen sampling uses temperature 0.7, top-$p$ 0.8, top-$k$ 20, presence penalty 1.5, and repetition penalty 1. The response capacity is 8,192 tokens. The word targets specify requested reasoning lengths, and accounting includes the complete realized response. Requests disable the optional thinking mode. Two responses nevertheless include a nonempty additional reasoning field, one under free reasoning and one under the 32-word policy; both remain in the analysis with their full generated output and elapsed time charged.

Timing and scoring. The client measures input preparation, requests, parsing, and solver execution. Measurement-journal writing is excluded from the timed procedure. Cross-question caches are cleared before collection; setup work is recorded separately. Shared evidence can be reused within a question. Qwen replicas collect independent requests, with one active measurement stream per physical GPU. Final answers are parsed with a fixed rule; unparseable outputs are scored as incorrect, and all generated tokens and elapsed time remain charged.

Execution accounting. The three-task grid contains 30,000 predictions with complete question–policy–round coverage. All retained requests complete successfully with isolated service accounting. ABACUS has zero host-level direct-answer fallbacks in its 3,000 executions. Replaying the saved local judgments reproduces every program output. Two web-of-lies inputs invoke the program’s default truth-state initialization in each of the five rounds; all ten outputs are scored as incorrect and retained in the denominator. The 99.67% macro accuracy therefore includes these execution outcomes. Navigation and temporal sequences complete their audited computation paths on all measured inputs.

A Local-Judgment Error and Its Computational Effect

The two incorrect truth-chain inputs share the initial sentence “Question: Conception lies.” The local question asks which description best fits that sentence. Its candidates are a direct truth statement, a direct lie statement, a report of another person’s truthfulness, a report of another person’s truthfulness, a report of another person’s lying, a question, and other. Kev selects other in all five recorded rounds for both inputs. The intended category is direct lie statement.

Policy	Lies	Temporal	Navigation	Macro
Kev direct	0.018	0.029	0.018	0.022
Kev teacher prompt	0.048	0.061	0.050	0.053
Qwen4B final only	0.097	0.131	0.097	0.108
Qwen4B brief 16	0.215	0.275	0.182	0.224
Qwen4B brief 32	0.269	0.344	0.270	0.294
Qwen4B brief 64	0.383	0.476	0.449	0.436
ABACUS	0.209	0.274	0.282	0.255
Qwen4B brief 128	0.465	0.773	0.829	0.689
Qwen4B brief 256	0.548	1.324	1.175	1.016
Qwen4B free	1.401	2.580	2.468	2.150

Policy	Accuracy (%)	95% interval	Seconds [95% interval]	Token proxy
Kev direct	64.00	[60.33,67.83]	0.022 [0.022,0.022]	120.3
Kev teacher prompt	71.67	[68.33,74.83]	0.053 [0.053,0.053]	557.0
Qwen4B final only	68.80	[65.77,71.80]	0.108 [0.108,0.108]	693.0
Qwen4B brief 16	76.17	[73.47,78.80]	0.224 [0.219,0.234]	722.3
Qwen4B brief 32	79.57	[77.07,82.07]	0.294 [0.286,0.309]	738.8
Qwen4B brief 64	88.70	[86.70,90.77]	0.436 [0.415,0.463]	772.2
ABACUS	99.67	[99.17,100.00]	0.255 [0.252,0.258]	1831.3
Qwen4B brief 128	94.83	[93.50,96.17]	0.689 [0.660,0.722]	832.6
Qwen4B brief 256	97.67	[96.60,98.57]	1.016 [0.980,1.053]	909.1
Qwen4B free	99.07	[98.40,99.57]	2.150 [2.047,2.267]	1166.7

One complete input is: “Question: Conception lies. Rashida says Conception tells the truth. Alejandro says Rashida tells the truth. Sherrie says Alejandro lies. Amberly says Sherrie tells the truth. Does Amberly tell the truth?” The correct answer is Yes. The remaining statement labels correctly encode, in order, reports of truth, truth, lying, and truth; the final sentence is recognized as a question, and the Yes/No options are correctly identified.

The program normally initializes a Boolean truth state from the first statement. Receiving other leaves that state unset; the next report of truth activates its default initialization to true. Propagating the later labels then yields No. In an offline diagnostic, replacing only the first local label with direct lie statement, while holding the entire program and all other saved judgments fixed, yields Yes. The same single-label replacement corrects the second input, whose correct answer is No. This holds for all ten saved executions of the two inputs. The original ten errors remain in the reported scores. This intervention localizes these failures to the initial semantic judgment and its interaction with the default state update.

Supporting Narrative Studies

Task-Specific Teacher Construction

Team allocation and murder mystery each use 50 development stories and 200 evaluation stories, with four solved development examples supplied to the teacher. The first registered build is the evaluation artifact. These narrative cohorts are reused for supporting analyses.

For team allocation, the host supplies people, tasks, candidate assignments, and query scopes over a person, a task, a person–task combination, or an unordered pair. The teacher supplies semantic attributes, finite labels, numerical values, and their scoring function. The resulting program uses 14 judgments: six task-fit assessments, three pair relationships, three work-style preferences, and two staffing preferences. Kev sees the complete story with the instantiated questions and candidate descriptions.

For assignment $A$, the generated score is

where $f$ denotes task fit, $w$ work style, $c$ pair compatibility, $b$ staffing preference, and $s_1=-1$, $s_2=1$. Task-fit values are 3, 1, $-1$, $-4$; pair-compatibility values are 2, 0, $-2$, $-5$; work-style and staffing values are $-1$, 0, 1. The decoder uses a factorized distribution over local judgments and selects the assignment with the highest estimated winning frequency. It enumerates all assignments when their support has at most 200,000 combinations; otherwise it draws 20,000 independent samples. Ties within a sample split winning mass equally; final ties use the fixed answer order. This inference rule is also used for murder mystery.

For murder mystery, the host supplies suspect and story scopes. Six ratings per suspect describe motive $m$, material gain $g$, weapon connection $w$, opportunity $o$, concrete evidence $e$, and irrelevant suspicious traits $r$. Two shared ratings describe weapon commonness $u$ and premeditation $p$. The generated score for suspect $c$ is

Weapon commonness ranges from specialist (0), through uncommon (1), to everyday (2); premeditation ranges from spontaneous (0), through ambiguous (1), to planned (2). Motive takes values 0/1/3 for none, grievance, or a life-changing stake. Gain takes 0/1/2 for none, minor, or major benefit. Weapon connection takes 0/1/2/3 for none, familiarity, possession or regular use, or recent preparation. Opportunity takes 0/1/2 for unplaced, loosely placed, or directly placed at the scene. Evidence takes 0/1/2 for none, circumstantial, or documentary evidence. Irrelevant suspicious traits take 0/1/2 for crime-linked, mixed, or predominantly irrelevant suspicion. The two-suspect instances require 14 judgments. The host evaluates the score using the factorized winning-frequency rule above. Table 11 reports task-specific and manual results separately.

Task	Direct Kev	Program + Kev
Team allocation	130/200	187/200
Murder mystery	111/200	152/200
Manual object placement	100/208	162/208

Manual Event and Belief Updates

The object-placement workflow uses an event ledger, object locations, observations, and beliefs. Local judgments identify movements and observations; explicit updates track the resulting world state and each observer’s belief. The workflow includes local rereading, probability sharpening of 1.5, a decision threshold of 0.5, and a maximum move count of eight. Its 208 questions are grouped in 52 stories, and it is evaluated separately from teacher-constructed programs.

Cost and Reuse

Let offline construction cost be $C_{\mathrm{build}}$ and average online cost be $c_{\mathrm{online}}$, measured in the same additive unit. Across $N$ uses,

Relative to a policy costing $c_{\mathrm{base}} > c_{\mathrm{online}}$, the amortization threshold is

The joint-construction study records 322,117 teacher tokens across nine admitted programs. The deployment results measure client latency for the fixed online procedure. The token count belongs to the nine-program construction study, while the latency measurements belong to the three frozen deployment programs. The amortization formula applies when construction and online costs are available in a common unit for the same artifacts.

The token-work statistic counts Kev input processing, including repeated evidence, and Qwen prompt plus generated tokens, scaled by nominal parameter count. ABACUS uses 1,831.32 units in the three-task deployment study, compared with 832.61 for Qwen’s 128-word policy and 1,166.66 for free reasoning. This accounting complements the measured latency advantage: reusable programs improve the online time to an answer through the combination of finite scoring and explicit execution, while total workload cost depends on the deployment volume and resource measure.

Reproduction Materials

The release provides construction and annotation prompts, nine joint-construction programs, three deployment programs, two task-specific narrative scoring structures, judgment schemas, and split identifiers. Saved programs require Kev; generative comparisons additionally require Qwen. Upstream links identify the original data and pretrained-weight providers; deployment instructions support readers’ own inference services. Fresh construction uses a task description and four solved development examples to jointly specify units, finite questions and candidate descriptions, and a solver. Local annotation receives the evidence, instantiated questions, and candidates.

The released BBH split fixes development and example membership, excludes their inputs and duplicate contexts, and retains 200 held-out inputs per selected task. Teacher requests use development data; held-out targets are used for scoring. Frozen evaluation and fresh construction have separate entry points.

Limitations

Task selection and scope. The three-task presentation was selected after inspection of the broader evaluation, including held-out results. The original six-task deployment plan covered 1,135 distinct inputs and an equal-task six-task average; the three-task study is a post-hoc subset analysis of 600 of those inputs. The programs were frozen before inference, so these are new instances for the frozen programs, but task selection is informed by observed performance. Accordingly, the three-task estimates describe selected task families and do not estimate benchmark-wide generalization. Table 12 records the completed original scope for transparency; the supporting narrative cohorts were also reused during development and analysis.

Task	Inputs	Direct Kev	Program policy
Web of lies	200	59.50	99.00
Navigation	200	63.00	100.00
Temporal sequences	200	69.50	100.00
Logical deduction	200	64.50	68.50
Causal judgment	135	54.81	54.81
Date understanding	200	77.50	77.50
Six-task macro	1,135	64.80	83.30

Construction and deployment. The nine independently constructed programs are evaluated on development inputs, including inputs used for admission checks. The 99.67% held-out result belongs to the three separately frozen deployment artifacts, selected with reference-agreement refinement. It does not establish that independent joint proposals alone consistently attain that held-out accuracy. The truth-chain artifact changes after agreement feedback, making the distinction consequential. The full construction cost of the deployment artifacts is not measured in the same budget convention as the nine-program study.

Mechanism and comparison coverage. The direct-Kev and generative-Qwen comparisons establish benefits over the stated answer policies. They do not isolate joint interface construction from other reusable-program strategies. All nine reported construction trajectories pass on their first proposal, so their outcomes do not estimate the benefit of rejecting candidates. Admission checks establish permitted execution and observable candidate behavior, not semantic verification. The three human–teacher allocations use different tasks and cohorts, and therefore provide a design taxonomy rather than a controlled test of progressively reducing human involvement. The narrative decoders and the shared-interface decoder also differ, as stated in Table 1.

Design level	Human design	GLM design	Online inference
Human-designed workflow	Judgment questions, state, and update rules	None	Event and belief updates with local rereading and thresholds
Task-specific interface	Domain objects and judgment scopes	Semantic attributes, candidate labels, and scoring program	Candidate winning frequencies under factorized local judgments
Shared interface	Common constructors, tools, and admission checks	Unit selection, judgment schemas, and composition program	Local argmax followed by reusable program execution

Empirical routing and uncertainty. Both envelopes are optimized from observed evaluation records. The task-conditioned envelope is a post-hoc empirical reference, not the independent test performance of a router frozen on development data. Its weights use task identity, not per-input target answers, but their selection uses aggregate test outcomes. The bootstrap quantifies question and measurement variation conditional on the fixed tasks and artifacts; it excludes task-selection and fresh-construction uncertainty. Repeated measurements do not increase the number of distinct inputs. The accuracy difference from free Qwen has an interval containing zero, supporting a claim about comparable observed accuracy rather than demonstrated superiority or statistical equivalence.

Representation and distribution. The reported programs are tested within the stated input formats and sizes. Their maximum judgment counts of 8, 13, and 11 are below the 60-judgment budget, and no retained unit list reaches the 40-unit cap. Longer chains, systematic paraphrases, entity substitutions, and option reorderings require separate validation. The two truth-chain errors expose sensitivity to an initial local classification; the offline reference-label intervention diagnoses this failure but is not a deployed correction. Replay consistency verifies execution reproducibility, not the correctness of every local interpretation. Factorized local sampling in the narrative studies does not model all dependencies between judgments, and its winning frequencies are not calibrated answer confidence.

Resources and model configurations. The latency advantage is measured for the specified H20 hardware and serving stacks. It excludes offline construction and does not imply lower total FLOPs, energy, or financial cost. The token-work proxy is higher for ABACUS than free Qwen in this deployment. A numerical amortization threshold requires corresponding costs for the same programs in a common unit. The Qwen comparison uses non-thinking mode throughout; free reasoning describes its response policy rather than its optional native thinking configuration. Kev and Qwen share nominal size but differ in post-training and serving implementation.