What Can't Jev Decide? Probing the Limits of One-Pass Decision Models

Introduction

Jev (TypeSafe AI, [30]) reads a state, a question and a list of options supplied by the caller, and returns a probability for each option from a single forward pass, in about 0.2 s. It is meant to make the decisions in agent loops (Yao et al., [36]) while an LLM does the writing. Open replicas followed within weeks (Palmer, 2026; alibiserikbay, 2026; Mapika, 2026; Zefan-Cai, 2026; OpenJev, 2026; ZhengxuYu, 2026; tic-top, 2026), and public benchmarks report high accuracy and calibrated probabilities (Benchmark Heaven, [2]; Jevals [8]). Aggregate accuracy, however, leaves the question that matters to a system built on Jev unanswered: what can a single forward pass not decide?

The limit we find is simple to state: Jev cannot make a decision that an LLM makes only by writing one record: what does it say? several records: what do they imply? a given state: is it correct?

out intermediate steps. It decides what one record says, but not what several records imply together, nor whether the state it is given is correct, and it does not know when it has crossed this limit. We show this for Jev 1.13.0 and 23 open configurations of Jev-style models in three steps.

Locating the limit (§ 3). On a benchmark that varies one factor at a time, Jev finds and selects records without error, up to 20k tokens and 64 options. Once the answer depends on two or more updates of one user, accuracy drops, to 53–80% at sixteen updates, because records of other users interfere; neither length nor the wording of the question is the cause. Given a precomputed summary of the state, Jev follows it even when it is stale. An LLM that writes its steps does both tasks: with thinking, Qwen3.8-27B answers 96% of the multi-update items and rejects 95% of the stale summaries up to 8k tokens, against 77% and 74% in one pass.

The same limit on real tasks (§ 4). We grade 1,131 real decision items by how many thinking tokens Qwen3.8-27B needs to solve them, and keep the 738 with reliable labels. Jev’s accuracy falls from 99% on items that need fewer than 250 tokens to 50% on items that need more than 4k; overall it answers 97.5% of the items one forward pass of Qwen3.8-27B solves, but only 74% of the others. Every open model shows the same gap, and a backbone trained to reason does not close it.

Jev does not know its limit (Section 5). Its confidence flags the errors it makes itself (AUROC 0.83) but not those it inherits from a stale input (0.53), and it almost never takes an offered hand-off option. A hand-off option trained in without new data catches more of its own errors, and is the steadier routing signal where no labelled data can tune a threshold, but it does not reach the inherited errors either.

Current Jev-style models are therefore selectors over a state computed elsewhere: decisions that need intermediate steps, including checking that state, belong to a model that reasons. We contribute a controlled benchmark that locates where one-pass decision models fail, a measure of reasoning demand for real decision items, and an analysis of whether these models notice and hand off their errors.

Jev and Jev-Style Decision Models

Decision models. A decision model reads a state $s$, a question $q$ and a set of options $\mathcal{O}$ supplied by the caller, and returns a probability distribution $p(\cdot\mid s,q,\mathcal{O})$ over $\mathcal{O}$ from a single forward pass, without generating text. Its answer is $\hat{o} = \operatorname*{arg\,max}_{o\in\mathcal{O}} p(o)$ and its confidence is $p_{\max}=p(\hat{o})$.

Jev. We evaluate Jev 1.13.0 (TypeSafe AI, [30]) through its API, which accepts up to 32k tokens; its weights and training data are not public.

Jev-style models. To tell whether a failure belongs to Jev or to the model class, we also evaluate 23 open configurations that serve the same interface (listed in Table 5, Appendix § 10): 18 from nine released projects and five plain Qwen models read through their option-letter logits. Most fine-tune a Qwen backbone (Qwen Team, [23, 24]) on decision data (Palmer, 2026; alibiserikbay, 2026; Mapika, 2026; Zefan-Cai, 2026; OpenJev, 2026); the others wrap an unmodified LLM (ZhengxuYu, 2026; tic-top, 2026; ekzhang, 2026), read the option-letter logits of the base LLM (Octalab-Inc, [20]; TheoLeeCJ [28]), or score options contrastively (Kwok et al., [14]).

Model	Backbone	How the answer is read	Input limit
Jev 1.13.0 ([30])	undisclosed	undisclosed (hosted API)	32k, state + longest question
Kev 0.8B, 4B, 9B, 27B ([22])	Qwen3.5-Base; Qwen3.8-27B	LoRA and a pointer head over the options	4B, 27B: <8k
JevK5 4B v0.2, v0.3; 9B v0.3.3 ((alibiserikbay, 2026))	Qwen3.5-4B, 9B	merged LoRA, option-letter logits	–
Decider 0.8B–35B-A3B ([16])	Qwen3.5-Base	fine-tuned, one forward pass	32,768 per row
Open-Jev-27B v1.1 ([37])	Qwen3.8-27B	fine-tuned, one forward pass	16,384
OpenJev-27B ([21])	Qwen3.8-27B	fine-tuned, one forward pass	16,384; ≤52 options
LitJev-27B ((ZhengxuYu, 2026))	Qwen3.8-27B	training-free decision layer	16,384
llm2jev 4B, 27B ((tic-top, 2026))	Qwen3.5-4B; Qwen3.8-27B	label-token log-probabilities after one prefill	–
OpenJev-SGLang ((ekzhang, 2026))	Qwen3.6-35B-A3B (bf16)	label-token log-probabilities after one prefill	32,768
Qwen, letter logits (ours)	Qwen3.5-4B, 9B; Qwen3.8-27B	option-letter logits after one forward pass	32,768; ≤26 options
CLM-v0.1-8B ([14])	Qwen3-8B (frozen)	similarity of contrastive state and option embeddings	40,960

Protocol. Every model sees the same state, question, options and option order. Inputs beyond a model’s context limit are recorded as unsupported rather than truncated, and cross-model comparisons use only the items every compared model supports.

Where Jev Breaks: A Controlled Diagnosis

A benchmark that varies one factor at a time

Design. Each item consists of a context, a question and 2–64 options, and its answer is computed by a program rather than labelled by a model (Figure 2). Contexts range from 1k to 20k tokens. For each scenario, shorter contexts are prefixes of longer ones and the target records keep their relative position, so neighbouring cells differ only in the factor under test. Every wrong option also appears in the context, so no option can be ruled out without reading it. Each cell contains 10 scenarios, each with 3 option orders.

Tasks. Retrieval asks for the access code of one user, hidden among other users’ records or among the essay paragraphs of the needle-in-a-haystack test [11], at 10%, 50% or 90% depth. Selection asks the same question with 2–64 candidate codes, all of which occur in the context. State tracking gives a day-ordered log in which the target user joins and leaves projects $k$ times ($k \in\{1, 2, 4, 8, 16\}$), interleaved with other users’ logs. Each log is asked two questions: event lookup asks what happened on one day, and final state asks which projects the user belongs to after the last entry. The two questions share the context and differ only in whether the $k$ updates must be combined.

Length and the number of options are not the bottleneck

We first test whether finding and selecting a record degrade with length, depth or the number of options. They do not: Jev answers all 105 retrieval, selection and event-lookup cells correctly, up to 20k tokens and 64 options, with the target at any depth and in either haystack, and puts a probability close to 1 on the correct option. Long contexts, buried targets and many options, the usual explanations for failures of long-context models [11, 15, 7], therefore do not explain the failures below.

State tracking fails from two updates on

We next ask for the user’s current state, which depends on all $k$ updates in the same log. Accuracy drops as soon as $k = 2$ and falls to 53–80% at $k = 16$ (Figure 3b), although Jev answers every event lookup on these logs correctly. It finds each update when asked about it, but does not use all of them when the answer depends on all of them.

The errors show which updates are lost: the later ones. Of 145 wrong answers, 134 are an earlier state of the same user and 137 contain fewer projects than the correct set; only 3 confuse users. The probability on the correct option falls before the accuracy does: it averages 0.93 in the multi-update cells that are still answered perfectly, and about 0.5 at $k = 16$.

Prompting does not help; similar records are the cause

We first ask whether a better question fixes the failure. It does not (Table 1, rows P1–P3): asking for step-by-step thinking (P1), spelling out the procedure of going through the user’s entries in order (P2), or naming the failure observed above

Variant	$k \ge2$ Jev	$k \ge2$ K-27B	$k \ge2$ K-4B	$k = 16$ Jev	$k = 16$ K-27B	$k = 16$ K-4B
P0 original	80	90	66	61	86	41
P1 step by step	81	90	67	61	88	39
P2 procedure	79	92	65	54	87	38
P3 name failure	78	88	62	64	86	38
R1 target only	100	100	85	100	100	67
R2 newest first	72	88	20	61	80	4
R3 target and essay	89	97	61	64	100	29
R4 R3 + essays	100	99	78	99	99	67

(P3) leaves Jev between 54% and 64% at $k = 16$, against 61% for the original item.

We then change the log, keeping the question and options, to find which of its properties causes the failure (rows R1–R4). With only the target user’s entries left (R1), Jev answers every item correctly, so it can combine 16 updates. Moving these entries, in order, into one block at the end of the log while keeping every other entry (R3) leaves the length and the other users’ records unchanged, and accuracy at $k = 16$ stays at 64%. Replacing the other users’ entries of R3 by essay text of the same length (R4) raises it to 99%. The failure is caused neither by scattered entries nor by length, but by the similar records around them, a sharper form of the distraction by irrelevant context known from LLMs [27]. Position does not explain it either: listing the log newest first (R2) lowers accuracy at $k \ge2$ from 80% to 72%.

This interference differs from the one reported for LLMs, where earlier values of the same key displace the latest one [31]: R1 contains all of the target’s earlier updates and is answered perfectly. What Jev fails at is keeping one user’s updates apart from other users’ records. Kev-27B behaves alike, except that R3 already suffices; Kev-4B reaches only 67% at $k = 16$ even in R1 and R4, so at this scale the combination itself is limited as well.

In practice the target’s history cannot be pulled out of a real log by hand; the usual way to spare the model the search is to state the result in advance, which we test next.

A state summary is used, trusted, and not checked

We next test the obvious remedy: compute the state in advance and append it to the log as a one-line summary. Jev uses the summary, but also trusts it. With a correct summary, it answers the final-state question correctly in all 30 cells. With a stale summary, which omits the last update, Jev picks the stale state in 61–100% of the items that offer it as an option, even though the log contradicts it, and more often in longer logs (68% at 1k, 93% at 20k; Figure 3c).

Asked directly, Jev cannot tell a correct summary from a stale one. It accepts 91% of correct summaries but rejects only 62% of stale ones over all context lengths (92% and 70% up to 8k, Table 2), and almost none (0–7%) when the summary omits one of two updates in a 16k–20k log. The remedy is therefore only as good as the summary.

The failures are shared by the model class

We finally ask whether these failures are specific to Jev. They are not: on contexts up to 8k tokens, every open model in Table 2 shows the same pattern: retrieval and selection are close to perfect, final state drops once updates must be combined, and a stale summary is followed. No decision model verifies summaries reliably: averaged over correct and stale summaries, none exceeds 85% (chance 50%), and the models that reject the most stale summaries reject most correct ones as well (Qwen3.5-4B-Base rejects 98% of stale and accepts 2% of correct ones). Neither scale (0.8B to 35B), training on questions written by a reasoning teacher (JevK5), nor reading the logits of an untuned LLM changes this pattern. Two models in Appendix § 11 fail earlier, at retrieval itself: Key-0.8B and the contrastive CLM-8B.

Jev thus decides what one record says, but not what several records imply together once similar records surround them, nor whether a state it is given is correct. Both are decisions that an LLM makes by writing intermediate steps: with thinking, Qwen3.8-27B answers 96% of the multi-update items and rejects 95% of the stale summaries (Table 2). § 4 asks whether the same limit appears on real decision tasks.

Where Jev Breaks on Real Tasks

Real tasks offer no factor to vary, so we measure how much reasoning each item demands and relate Jev’s accuracy to it.

Items and reasoning demand

Items. We test on 1,131 real decision items with a fixed option list: the public JevBench tiers (easy, original and hard; 231 items; [2]) and 300 items each from PubMedQA ([9]), Banking77 ([3]) and HelpSteer2 ([32]) as packaged by Jevals ((Jeyals, 2026)). Questions are yes/no, a choice among 2–77 labels, or a score level.

Reasoning demand. To measure how much reasoning an item needs, we run Qwen3.8-27B on it in three ways: reading the option probabilities after one forward pass without generating (direct readout), answering after generation without thinking, and answering after thinking, the latter two with 10 samples each. An item is one-pass solvable if the direct readout is correct, and needs reasoning otherwise. Its demand is the median number of thinking

caption: Accuracy in % of Jev and representative open models on contexts up to 8k tokens. “Select” pools 2–64 options. “With summary” is final-state accuracy when a correct or a stale summary is appended; “reject stale” is the share of stale summaries judged incorrect (chance 50%). †Kev-4B and Kev-27B accept at most 4k tokens and are evaluated up to 4k. ‡Not a decision model: the same LLM generates its reasoning before answering; shown for reference and run on the state-tracking items only. All models are in Appendix § 11.

:::

tokens in the correct thinking samples. Qwen3.8-27B is only a reference for how much reasoning an item requires, not a model of Jev, whose backbone is not public.

Excluded items. We set aside items whose labels cannot separate the models. HelpSteer2 asks for one of five helpfulness levels, and no model clearly beats always choosing the most frequent level (42%): the decision models reach 31–49% and Qwen3.8-27B with thinking 33%, so we report it only in Appendix § 11. We further exclude the 93 items that Qwen3.8-27B with thinking answers correctly in at most 5 of 10 samples: 65 from Banking77, 25 from PubMedQA and 3 from JevBench-hard. Their gold labels are disputed rather than hard: Jev answers 76 of them incorrectly, and in 68 of these it picks the same wrong answer as the majority of the thinking samples (example in Figure 4b). The remaining 738 items comprise 646 one-pass solvable items and 92 that need reasoning. Results on all items are in Appendix § 11.

Jev loses the items that need reasoning

Jev loses accuracy where an item needs reasoning, by every measure we use. Its accuracy falls with the demand of the item (Figure 5), from 99% on items that need fewer than 250 thinking tokens to 50% on items that need more than 4k, while Qwen3.8-27B with thinking stays above 94%. Jev answers 97.5% of the one-pass solvable items but only 74% of those that need reasoning. The gap sits in one source (Table 3): on JevBench-hard,

Model	One pass 646	Reasoning 92	J-easy 48	J-original 72	J-hard 108	PubMedQA 275	Banking77 235	All 738
Jev 1.13.0	97.5	73.9	100.0	98.6	73.1	98.5	97.4	94.6
Kev-27B	97.4	69.6	100.0	100.0	72.2	96.7	97.4	93.9
llm2jev-27B	95.4	68.5	100.0	95.8	74.1	94.5	94.5	92.0
LitJev-27B	95.4	63.0	100.0	98.6	75.9	93.5	91.9	91.3
Open-Jev-27B	83.4	62.0	100.0	95.8	74.1	75.6	81.3	80.8
JevK5-9B	94.6	59.8	100.0	95.8	77.8	94.2	87.7	90.2
Kev-4B	91.2	53.3	100.0	93.1	52.8	88.4	94.9	86.4
Reference: Qwen3.8-27B
Direct readout	100.0	0.0	97.9	94.4	68.5	96.4	81.7	87.5
Thinking	99.3	97.3	100.0	99.7	99.2	98.9	98.6	99.0

whose items apply policies, dates and multi-step rules, Jev reaches 73% against 99% for the thinking model, and comes within about one point of it on the other four sources. Of Jev’s 40 errors, 24 are on items that need reasoning; of the other 16, 8 are JevBench-hard items that the direct readout happens to answer correctly.

The model class shares it

We next ask whether the open models show the same gap. They do (Table 3): every model is close to its best on the one-pass solvable items and makes most of its errors on the items that need reasoning: the 27B models answer 62–70% of them, and the 4B and 9B models 53–60%. For every model the gap between the two kinds of items lies between 21 and 38 points, and Jev has one of the smallest.

A reasoning backbone does not help

We finally test whether a backbone trained to reason closes the gap. It does not. We train Kev with its released recipe twice, once from Qwen3.5-Base, as released, and once from the post-trained Qwen3.5 that thinks before answering, three seeds each (Table 9 in Appendix § 11). On the real-task items the two backbones are indistinguishable: Kev-4B reaches 87.2% and 87.4%, Kev-9B 90.1% and 90.1%, and on the items that need reasoning the difference is comparable to the spread across seeds. On the final-state items of § 3 up to 4k tokens, the reasoning backbone is worse: 70% vs. 50% at 4B and 76% vs. 71% at 9B for $k \ge2$. Whatever reasoning the backbone has learned is exercised when it generates tokens; a model that answers in one forward pass does not draw on it.

	Backbone	Real tasks			Final state
		All	One pass	Reasoning	$k \ge2$
4B	Base	$87.2 \pm0.5$	$91.8 \pm0.4$	$54 \pm3$	$70 \pm5$
	Reasoning	$87.4 \pm0.4$	$91.4 \pm0.3$	$58 \pm1$	$50 \pm3$
9B	Base	$90.1 \pm0.2$	$94.1 \pm0.2$	$60 \pm3$	$76 \pm0$
	Reasoning	$90.1 \pm0.5$	$94.2 \pm0.5$	$59 \pm2$	$71 \pm2$

On real tasks as on synthetic logs, then, Jev loses exactly the decisions that an LLM makes by writing intermediate steps; a backbone trained to reason does not recover them. Section 5 asks whether Jev can tell when it meets such a decision.

Does It Know, and Can It Hand Off?

A system built on Jev must send the items Jev would get wrong to a stronger model, either by reading Jev’s confidence $p_{\max}$ or by offering it an option to hand the question off.

Confidence flags Jev’s own errors, not inherited ones

We first ask whether $p_{\max}$ separates Jev’s wrong answers from its right ones. It does when Jev makes the error itself: AUROC is 0.83 on the final-state items, with an expected calibration error of 0.03, and 0.94 on the real tasks. When the error comes from a stale summary, however, the signal is gone: AUROC is 0.53, and at $k = 2$ it is 0.26, so Jev is more confident when it follows the stale summary than when it does not. Every model shows the same split (Figure 6a): on the synthetic logs its AUROC is 0.68–0.93 for its own errors and 0.18–0.66 after a stale summary, below chance for 7 of 11 models.

Jev does not hand off

We next offer an explicit way out, and Jev almost never takes it. We add one option, “Cannot be determined reliably from the state; hand this question to a stronger model”, and one sentence asking the model to choose it when unsure. Jev chooses it for 1.6% of the real-task items and 3 of its 40 errors, none of them among its 29 errors on JevBench-hard; on the synthetic logs it hands off 3 of 145 wrong answers. The open models hand off more, but for every model except Open-Jev-27B a confidence threshold that routes the same share of items leaves a more accurate remainder (Table 4, top). The models carry the signal but do not act on it.

Hand-off training helps where a threshold cannot be tuned

We then ask whether the option can be trained in without new data, keeping the training data and number of steps of the original recipes. The training set is split in two halves; a model trained on one half answers the other, and every question it answers wrongly is marked as unlearnable. In the final run each question carries the hand-off option with probability 0.5, worded with paraphrases disjoint from the evaluation wording, and its target is the hand-off option if the question is unlearnable and the answer otherwise. We train Kev-4B, Kev-9B and JevK5-4B, three seeds each.

On the task the threshold was tuned for, training makes the option useful but not better than the threshold (Table 4, bottom). Accuracy without the option is unchanged, and the share of errors handed off doubles, from 23% to 47% for Kev-4B and from 22% to 44% for Kev-9B. A threshold on the untrained model that routes the same share of items is still more accurate, by 1.3–1.7 points.

Without labelled data from the target task, however, the threshold cannot be set, and the trained option is the steadier signal (Figure 6b). Choosing the threshold on two sources so that it routes the same share as the trained model and applying it to the third routes 9–32% of the items for Kev-4B and Kev-9B instead of about 15%; for JevK5-4B on PubMedQA, whose yes/no questions never give $p_{\max}<0.5$, it routes none and catches 2% of the errors, where the trained option catches 71%. The trained option routes 6–25% on every source and needs no tuning.

Neither mechanism reaches the inherited errors (Figure 6c). On the synthetic logs up to 4k tokens, training raises the share of final-state errors that the models hand off from 0–14% to 30–40%, but the share of errors caused by a stale summary only from 0% to 3–9%: the training labels mark questions the model cannot answer, not inputs that are wrong. In our experiments only generated reasoning checks the input: with thinking, Qwen3.8-27B rejects 95% of the stale summaries (Table 2).

Jev therefore does not recognise its limit reliably, and least of all when the error comes from its input. The division of labour for a system built on Jev follows: where labelled data from the task exist, a threshold on $p_{\max}$ routes Jev’s own errors best; where they do not, the trained hand-off option is more robust; and decisions that need intermediate

caption: Hand-off on the 738 real-task items; all columns in %. Acc.: accuracy without the hand-off option. Rate: share of items handed off when the option is offered; Errors: share of the model’s errors (without the option) that it hands off. Acc. of the rest: accuracy on the items not handed off, by the option or by a threshold on the confidence of the untrained model that routes the same share. Bottom: our training runs, mean over three seeds; “(ours)” reproduces the released recipe and “+ hand-off” adds hand-off training. Kev-0.8B is in Appendix § 11.

:::

steps, above all checking an input Jev did not compute itself, have to be made by a model that writes them.

Related Work

Decision models. Jev (TypeSafe AI, [30]) introduced the interface we study; its replicas fine-tune an LLM, wrap one, or read its option logits (§ 2). Reading option scores from an LLM is also how multiple-choice benchmarks are commonly evaluated, and such scores depend on the prompt, the option labels and their order [39, 40]. Public evaluations of decision models report accuracy and calibration over mixed tasks [2, 8]; we instead isolate the inputs on which they fail.

Long contexts and state tracking. Retrieval from long contexts degrades with length, the position of the target and the difficulty of the task [11, 15, 7], and reasoning over facts spread through them degrades further [13]. Language models track entity states only partly [12], and recalling the latest value of a key suffers from interference by its earlier values [31]. Decision models retrieve without error up to their limit, and in our logs the target’s own

earlier updates are harmless (R1); the failure comes from similar records of other users.

What one forward pass can compute. Writing intermediate steps improves LLM reasoning [33], and theory shows why: bounded-depth transformers probably cannot solve some problems without them [5, 18], state tracking among them [17]; LLMs also complete multi-hop inferences only partly within one pass [35]. Decision models give these steps up by design; we measure what this costs on practical decisions, on synthetic logs as well as on real tasks.

Knowing when to abstain. Large LLMs are well calibrated on multiple-choice questions in a suitable format [10], and a threshold on the maximum probability is the standard way to abstain [6]. Learning to defer trains a model to pass cases to an expert [19], and refusal tuning labels questions beyond the model’s knowledge as ones to decline [38], much as our hand-off training labels questions a held-out model answers wrongly. In our experiments both catch only the errors a model makes itself: like LLMs that adopt coherent but conflicting context [34], Jev adopts a stale summary, although it contradicts the log rather than the model’s memory, and neither mechanism detects it.

Conclusion

We asked what Jev cannot decide, and found a single limit: Jev, like the open Jev-style models we evaluated, cannot make a decision that an LLM makes only by writing out intermediate steps. These models find and select records reliably, but they fail to track a state among similar records, follow a precomputed state without checking it, and lose the real decision items that need reasoning, the more so the more reasoning an item needs. Their confidence flags the errors they make themselves but not those they inherit from a wrong input, and neither an explicit nor a trained hand-off option closes this gap. One-pass decision models are therefore best used as fast selectors over a state that a reasoning model has computed and checked. Whether training on such decisions, or a few internal steps before the readout, can move the limit remains open.

Limitations

Our items are isolated decisions: each is posed once, answered once, and scored once. In a deployed agent loop a decision model is called many times along a trajectory, its errors compound, and they interact with the surrounding system, with retries, with memory, and with the state summaries that earlier steps wrote, the very inputs Section 5 shows it does not check. Measuring these effects requires an end-to-end evaluation of a full agent system, which no single-model benchmark provides. Our results give the per-decision error rates and the routing signals that such an evaluation would build on.

References

Models and Serving Details

Decision models

Table 5 lists every model we evaluate. Each model is served by the code its authors released, behind an endpoint that accepts a state, a question and an option list and returns a probability per option; the two exceptions are the letter-logit readout of plain Qwen models, which we implement ourselves, and CLM, which scores options by embedding similarity. All models receive the same items with the same option order. An input that exceeds a model’s limit is recorded as unsupported and never truncated: Decider is started with a state limit above its 32,768-token row limit so that it refuses instead of cutting, and CLM is run with its truncation disabled (–max-tokens 0). Requests that fail with HTTP 429, 503 or 529 are retried with exponential backoff; any other failure is recorded as an error and excluded from accuracy.

For the open checkpoints we pin the Hugging Face revision. The revisions behind our numbers are 139fdd94 (Kev-4B), 2629c06a (Kev-9B), 01b81998 (Kev-27B, the snapshot with a separate adapter) and 27d2d6b8 (JevK5-4B v0.2). The main branch of the Kev repositories was replaced on 2026-10-10; the new Kev-9B weights differ from the ones we report (the same answer on 82% of the final-state items up to 4k tokens), whereas the new Kev-4B weights are identical to ours.

Letter-logit readout. For plain Qwen models (Qwen3.5-4B and 9B, each with its Base model, and Qwen3.8-27B) we follow the recipe of jqv (Octalab-Inc, [20]) and SemIf (TheoLeeCJ, [28]). The prompt, in the Qwen chat format with thinking disabled, contains the document, the question and the options labelled A, B, . . . , asks for the letter only, and ends with Answer:. The next-token probabilities of the offered letters, renormalised, are the answer distribution; nothing is generated and no temperature is fitted. Items with more than 26 options are unsupported, which removes the 32–64-option selection cells.

Thinking reference. Qwen3.8-27B with thinking is served with SGLang and queried through the OpenAI-compatible chat API with the sampling settings of its model card (temperature 1.0, top-p 0.95, top-k 20) and at most 24,000 new tokens; a reply cut off at that limit counts as wrong. The answer is parsed from the reply, so it has no probabilities and no AUROC.

Real-task items and reasoning demand

JevBench items are the public easy, original and hard splits of the repository (Benchmark Heaven, [2]). Jevals items are rebuilt from the pinned Hugging Face rows of suite 0.1.0 (Jevals, [8]) exactly as the Jevals harness builds them, and each state is checked against the SHA-256 hash in the suite. Every item is reduced to a closed option list: yes/no questions get the options yes and no, with the criteria as their descriptions, and score questions their levels.

Reasoning demand is measured with Qwen3.8-27B served by SGLang. The direct readout disables thinking, forces the reply to begin with Answer:, and scores each option label by the log-probability of its tokens followed by the end of the label, normalised over the options; nothing is generated. The no-thinking and thinking modes draw 10 samples each with the sampling settings of the model card (no thinking: temperature 0.7, top-p 0.8, top-k 20, presence penalty 1.5; thinking: temperature 1.0, top-p 0.95, top-k 20) and at most 4,096 and 32,768 new tokens; a truncated sample counts as wrong. The demand of an item is the median number of thinking tokens over the thinking samples that answer it correctly.

Training runs

Kev. All Kev runs use the decision-v7 configuration released with Kev: the decision-v7 training suite with at most 1,000 records per source, two epochs, learning rate $5 \times10^{-5}$ ($10^{-4}$ at 0.8B), LoRA rank 16, an effective batch of 8, and a pinned base revision; seeds 2, 3 and 4. The backbone comparison of § 4 changes only the base model, from Qwen3.5-4B-Base and Qwen3.5-9B-Base to the post-trained Qwen3.5-4B and Qwen3.5-9B (Qwen Team, [24]).

JevK5. The JevK5 runs reproduce the v0.2 recipe. Its teacher-written questions are not released, so we regenerate them with the released teacher script and settings and add the same public training splits; we then train with LoRA rank 16, learning rate $3 \times10^{-5}$, two epochs, inputs up to 2,048 tokens, 20 warm-up steps, and a softmax temperature fitted after training; seeds 0, 1 and 2.

Hand-off training. The training records are split into two halves (6,334 and 6,242 records for Kev; 3,272 each for JevK5). A model trained with the unchanged recipe on one half answers every question of the other half, and a question it answers wrongly is marked unlearnable. A separate pair of models is trained for each backbone, so the labels are the backbone’s own: 2,220 Kev records contain an unlearnable question for the 4B labels, 2,069 for 9B and 2,939 for 0.8B, and 1,331 of the 6,544 JevK5 training items are marked unlearnable. The final run sees each question once per epoch; with probability 0.5 it is rewritten as a choice with an added hand-off option at a random position and a sentence asking the model to choose it when unsure. Option text and sentence are drawn from six and four paraphrases that exclude the evaluation wording. The target is the hand-off option for an unlearnable question and the gold answer otherwise. Everything else, including the number of steps, follows the recipe above.

CLM-v0.1-8B, which scores options by embedding similarity, is at chance on retrieval with four options (25%) and answers no single-update final-state item; it accepts every summary, correct or stale, and its 80% at $k = 16$ next to 0% at $k = 1$ points to a preference for particular options rather than to reading the log. Several models reach a high rejection rate for stale summaries only by rejecting nearly every summary: Qwen3.5-4B-Base rejects 98% of stale summaries and accepts 2% of correct ones.

Full Benchmark Results

Controlled benchmark

Table 6 extends Table 2 to every model. Jev answers every retrieval item correctly in both haystacks, at every depth and up to 28k tokens of essay text, and every selection item with 2 to 64 options; its final-state accuracy by context length is shown in Figure 3. Two models fail before the final-state question. Kev-0.8B already loses event lookup (66%).

Prompt and log variants

The variants of Table 1 are built from each final-state item without changing its options or their order. P1–P3 append one sentence to the question. R1 keeps only the log lines that mention the target user. R2 reverses the log and replaces “After the last log entry” in the question by “The log is listed from the newest entry to the oldest. After the most recent log entry”. R3 moves the target user’s lines, in order, to the end of the log and keeps every other line. R4 replaces the other users’ lines of R3 by paragraphs of the essay corpus with the same total length in tokens. R3 and R4 therefore have the length of the original item, and the last $k$ lines of both are exactly the target user’s updates. Table 7 gives Kev-9B, which follows Kev-4B.

Model	Retr.	Sel. ≤26	Sel. 32–64	Event	$k=1$	$k\ge2$	$k=16$	correct	stale	accept	reject	own	stale
Jev 1.13.0	100	100	100	100	100	84	64	100	58	92	70	0.83	0.57
Kev-27B†	100	100	100	100	100	91	88	100	60	100	62	0.90	0.62
Kev-9B	100	100	100	100	100	70	54	100	55	81	26	0.70	0.38
Kev-4B†	100	100	100	100	98	66	41	100	53	88	38	0.76	0.29
Kev-0.8B	86	99	97	66	18	32	18	91	25	66	41	0.56	0.46
JevK5-4B v0.2	100	100	100	99	98	53	56	100	47	73	52	0.69	0.38
JevK5-4B v0.3	100	100	99	99	100	61	65	99	55	84	32	0.75	0.49
JevK5-9B v0.3.3	100	100	100	100	100	68	86	98	64	92	26	0.79	0.50
Decider-35B-A3B	100	100	100	99	83	65	60	100	39	85	31	0.70	0.27
Decider-4B	100	100	100	98	100	67	80	100	39	100	0	0.71	0.21
Decider-2B	100	100	100	98	95	69	95	100	42	85	21	0.67	0.63
Decider-0.8B	97	100	100	82	67	54	68	100	37	89	17	0.63	0.59
Open-Jev-27B v1.1	100	100	100	100	100	89	78	100	72	96	74	0.87	0.58
OpenJev-27B	100	100	100	100	100	95	93	100	66	98	56	0.87	0.62
LitJev-27B	100	100	100	100	98	79	76	100	41	100	7	0.86	0.16
llm2jev-27B	100	100	100	100	100	86	81	100	43	51	73	0.90	0.20
llm2jev-4B	100	100	77	99	83	37	25	100	38	10	89	0.70	0.23
OpenJev-SGLang-35B-A3B	100	100	98	100	87	50	61	100	38	59	54	0.74	0.24
Qwen3.8-27B	100	100	–	100	100	77	77	100	42	51	74	0.82	0.28
Qwen3.5-9B	100	100	–	100	98	58	79	99	40	98	5	0.74	0.26
Qwen3.5-9B-Base	100	100	–	100	95	69	83	100	48	53	52	0.77	0.39
Qwen3.5-4B	100	100	–	100	88	39	30	100	41	10	88	0.68	0.22
Qwen3.5-4B-Base	100	100	–	99	83	63	79	100	49	2	98	0.76	0.39
CLM-v0.1-8B	25	36	10	26	0	50	80	24	24	100	0	0.62	0.42
Qwen3.8-27B, thinking‡	–	–	–	100	100	96	100	100	83	99	95	–	–

Variant	$k\ge2$	$k=16$
P0 original	69	48
P1 step by step	69	56
P2 procedure	69	56
P3 name failure	67	48
R1 target only	71	50
R2 newest first	20	7
R3 target at end	66	44
R4 R3 + essays	90	83

Real tasks

Table 8 gives every model on every source before any exclusion, the 831 items without HelpSteer2, and the 738 items of § 4. On HelpSteer2 every item asks for one of five helpfulness levels, and always answering the most frequent level (4) is correct for 42% of the items; no model, with or without thinking, is clearly above that. The 93 excluded items are those Qwen3.8-27B with thinking answers correctly in at most 5 of 10 samples: 65 from Banking77, 25 from PubMedQA and 3 from JevBench-hard.

Model	JevBench			PubMedQA	Banking77	HelpSteer2	Pooled
	easy	original	hard				831 items	738 items
	48	72	111	300	300	300
Jev 1.13.0	100.0	98.6	71.2	92.0	80.3	41.7	86.0	94.6
Kev-27B	100.0	100.0	70.3	89.7	83.3	44.3	86.3	93.9
Kev-4B	100.0	93.1	51.4	83.7	82.7	44.0	80.7	86.4
JevK5-4B v0.2	100.0	94.4	73.9	85.0	69.7	35.7	79.7	87.7
JevK5-4B v0.3	100.0	94.4	74.8	83.7	71.7	46.3	80.0	88.1
JevK5-9B v0.3.3	100.0	95.8	76.6	88.3	72.7	47.0	82.4	90.2
Decider-4B	100.0	98.6	64.0	87.0	85.0	49.0	85.0	90.4
Open-Jev-27B v1.1	100.0	95.8	73.0	71.7	65.3	39.3	73.3	80.8
LitJev-27B	100.0	98.6	73.9	88.0	74.3	36.0	82.8	91.3
llm2jev-27B	100.0	95.8	72.1	88.7	75.7	30.7	83.0	92.0
llm2jev-4B	100.0	76.4	59.5	84.7	67.7	32.3	75.3	82.0
Qwen3.8-27B, direct	97.9	94.4	66.7	90.3	67.7	36.3	79.8	87.5
Qwen3.8-27B, no thinking	100.0	98.6	97.6	89.0	78.0	34.0	87.6	96.9
Qwen3.8-27B, thinking	100.0	99.7	97.3	91.6	78.6	32.7	88.9	99.0

Reasoning backbone

Table 9 gives the backbone comparison of § 4.

Hand-off

Kev-0.8B, trained and evaluated like the larger Kev models in Table 4, follows the same pattern with larger effects. Hand-off training lowers accuracy without the option from 79.3% to 77.9%, raises the share of items handed off from 7.7% to 32.8% and the share of errors handed off from 15% to 58%. The items it answers are then correct 86.5% of the time, against 91.8% for a threshold on the untrained model that routes the same share.