betaFeedback

WorksComputer Science

IJev: Order-Invariant Decisions via Iterated Cyclic Reordering

Abstract

System One decision models answer a choice question in one forward pass, but their answer depends on the order in which options are listed, and their probabilities carry a prior over answer labels. An invariant ordering operator is any fixed function that returns the same ordered option list for every input order of a fixed option set. Starting from any such order and averaging a listwise readout over its K cyclic shifts removes the input order, but the decision still depends on the chosen cyclic class: those KK shifts are only one of the (K-1)! classes, and different invariant operators can select different classes. We propose IJev, which repeatedly reorders the options by the cyclic-averaged, label-prior-corrected distribution and runs another cyclic average. Every iterate is exactly input-order-invariant. When the iteration does not converge, we average the round distributions rather than return the last round. On five benchmarks and five models, IJev has flip rate zero in every cell. The default uniform average over T=20 rounds substantially improves calibration over a single cyclic average, and its accuracy is comparable. A descriptive choice of horizon and averaging rule is best or tied-best for accuracy in all 25 settings.

Introduction

System One decision models turn a state and a typed question—yes/no, a choice among options, or an ordered level—into a probability distribution over answers without generating text ([13, 2]). Open implementations include purpose-trained decision models and readout heads on frozen language models ([18, 8, 3] (Zhang et al. 2026)). However, two biases limit the reliability of these decisions. First, for a choice question the answer depends on the order in which the options are listed, and the dependence grows with the number of options ([11, 5] (Zheng et al. 2024; Zhang et al. 2026)). In an agent, the order is typically inherited from a tool registry or a page layout; a change there can silently change the prediction, and whoever sets the order can steer it. Second, the next-token probabilities carry a prior over answer labels: some option letters are favored independently of their content, which distorts both the answer and its confidence ((Zheng et al., 2024; Zhao et al., 2021; Zhou et al., 2024; Zhang et al., 2026)).

AnyJev’s L0 readout ((Zhang et al., 2026)) addresses both problems. It reads the options in all KK cyclic shifts, so every option sits at every position once, and divides out a label prior estimated without labels. Two limitations remain. First, L0 is not input-order-invariant: the KK listings it averages are the rotations of the input order, so that order decides which KK of the K!K! listings enter the average. Second, those KK listings form a single cyclic class. Classes that are never read can still change the decision.

Invariant ordering. The input-order dependence can be removed before any model call by an invariant ordering operator: a fixed function that maps an option set, a state, and a question to an ordered list of the same options, and that returns the same list for every input order of that set. Sorting by option text is one such operator. Sorting by pointwise yes/no scores is another: each option is scored alone, so the score and the induced order do not depend on the other options or on their order. Given any invariant order τ\tau, averaging the listwise readout over the KK cyclic shifts of τ\tau is input-order-invariant, because the input order has already been discarded.

Residual cyclic-class bias. Input-order invariance is not the end of the story. The K!K! orders of KK options partition into (K−1)!(K-1)! cyclic classes, where a class is the set of cyclic shifts of one order. Cyclic averaging uses one class. Different invariant operators—sorting by text, sorting by pointwise scores, or sorting by a previous decision—can select different classes. A model that is not symmetric under rotation can return a different distribution on each class. That residual position bias is what one cyclic average leaves in place. Full permutation marginalization would average every class, but it costs K!K! listings.

This paper. We propose IJev (iterative invariant Jev), which starts from an invariant order, applies AnyJev’s prior-corrected cyclic average, and reorders the options by the resulting distribution. The reorder step never reads the input order, so every iterate is input-order-invariant. The iteration may reach a fixed order, enter a cycle, or still be moving when the round budget ends. We therefore average the round distributions—uniformly, in log space, or with exponential weights—instead of returning the last one. That average is an ensemble over the cyclic classes the iteration visits. Figure 1 draws the argument for K=3K=3, the case in which both cyclic classes fit on the page.

Figure 1. The argument, drawn in full for K=3K=3, where there are exactly two cyclic classes. (a) The same options in two orders can yield different top choices. On the five benchmarks, one shuffle flips a raw listwise answer on between 7.9% and 59.7% of items. (b) The KK cyclic shifts of an order stay inside one class, and there are (K−1)!(K-1)! classes. AnyJev-L0 averages the class of the input order and divides out a position prior, so a shuffle still selects a different class. (c) Pointwise yes/no scores induce one order τ1\tau_1 for every input order. That removes the input order and leaves the class. (d) IJev reorders by the prior-corrected cyclic average. For this τ1\tau_1, the next order lies in the other class. The returned distribution is the uniform average of the round distributions, not the last round. Because τ1\tau_1 ignores the input order, the walk and PTP_T do too, so the flip rate is zero.

Contributions.

  • A definition of invariant ordering operators and cyclic classes for System One choice decisions, with text sorting and pointwise scoring as examples. The definition separates input-order invariance from the residual dependence on the chosen cyclic class.

  • IJev, which iterates reordering and prior-corrected cyclic averaging. Every iterate and every round average is exactly input-order-invariant, and on a finite set of orders the iteration reaches a fixed point or a cycle.

  • Experiments on five benchmarks and five models. IJev has flip rate 0 in every cell. Uniform averaging over T=20T=20 rounds improves calibration over AnyJev-L0 in nearly every setting, while accuracy stays comparable. A post-hoc choice of horizon and rule is best or tied-best for accuracy in all 25 model–benchmark cells; that row is a descriptive reference, not a confirmatory result.

Related Work

One forward pass of a System One model returns a distribution over the answer space, whether the question is yes/no, a choice, or a graded score [2, 13]. Public systems either fit a dedicated decision model [10, 3, 14] (Yu and Yao 2026) or attach a readout to a frozen language model [8]. The readout may score options independently [18] or decode the labels of a single listing [15] (Zhang et al. 2026).

Placing the same options in a different order changes these predictions [11, 17, 5] (Zong et al. 2024; Zhang et al. 2026). A substantial part of the effect is a learned preference for particular answer labels (Zheng et al., 2024; Zhang et al., 2026). Such priors are usually removed by dividing out a content-free or contextual estimate (Zhao et al., 2021; Zhou et al., 2024). AnyJev estimates its prior on the batch being decided, or on empty probe states, and does not use labels. We keep that estimator fixed and change only the orders that enter the average.

Enumerating every permutation removes order dependence but requires K!K! readings [4]. Cheaper alternatives, including KK cyclic shifts [5] (Iてazard et al. 2023; Zheng et al. 2024; Zhang et al. 2026), remove only the part of the bias that lives inside one cyclic class. The concurrent method Runoff [5] combines pointwise scoring with cyclic comparison: it canonicalizes a candidate list once, with pointwise decisions, and then compares those candidates in the cyclic listing of that one order. The construction is input-order-invariant, and it is cheaper when the candidate set is small, but it tends to lose accuracy and its guarantee covers only one cyclic class. IJev does not prune. It iterates the canonical order and averages the distributions of the classes it visits, trading additional reads for accuracy and calibration. Pruning is orthogonal and can be composed with the iteration when a smaller pool is required.

Preliminaries

Consider a choice question qq posed about a state xx. The candidate answers form a set O={o1,…,oK}\mathcal{O} = \{o_1,\ldots,o_K\} containing exactly one correct option. An order σ\sigma is a list of all options in O\mathcal{O}. The listwise readout g(x,q,λ)∈Δ(O)g(x,q,\lambda) \in\Delta(\mathcal{O}) renormalizes the model’s next-token probabilities of the letters in listing λ\lambda. Nothing is generated or parsed. We write gλ(o)g_{\lambda}(o) for the probability of option oo.

AnyJev. For a choice question, AnyJev-L0 (Zhang et al., 2026) averages the listwise readout over the KK cyclic shifts of the input order and divides out a position prior inside each listing. Let σs\sigma_s be the ss-th cyclic shift of σ\sigma, and let pos⁡s(o)\operatorname{pos}_s(o) be the position of option oo in σs\sigma_s. Then

pσL0(o)∝exp⁡(1K∑s=1Klog⁡gσs(o)bs(pos⁡s(o))β),(1)p_{\sigma}^{\mathrm{L0}}(o) \propto\exp\left(\frac{1}{K}\sum_{s=1}^{K}\log\frac{g_{\sigma_s}(o)}{b_s(\operatorname{pos}_s(o))^\beta}\right), \tag*{(1)}

where bsb_s is a distribution over the KK positions in shift ss. The prior is not a distribution over option identities. It can be estimated from the batch being decided—the mean listwise distribution over positions at the same shift—or from content-free probe states with the same question and listing. Neither estimate uses answer labels.

Invariant ordering operators.

Definition 1 (Invariant ordering operator). Let O\mathcal{O} be a finite option set and let Σ(O)\Sigma(\mathcal{O}) be the set of all orders of O\mathcal{O}. A map Φ(x,q,O,σ)∈Σ(O)\Phi(x,q,\mathcal{O},\sigma) \in\Sigma(\mathcal{O}) is invariant if Φ(x,q,O,σ)=Φ(x,q,O,σ′)\Phi(x,q,\mathcal{O},\sigma)=\Phi(x,q,\mathcal{O},\sigma') as ordered sequences of option identities for all σ,σ′∈Σ(O)\sigma,\sigma' \in\Sigma(\mathcal{O}).

Two examples are used below. Text sort orders options lexicographically by their text. Pointwise sort scores each option alone, so the score does not depend on the remaining options or on their order. One such score is the yes/no log-odds

f(x,q,o)=log⁡P(yes∣x,q,o)−log⁡P(no∣x,q,o),f(x,q,o)=\log P(\mathrm{yes}\mid x,q,o)-\log P(\mathrm{no}\mid x,q,o),
π(o)=ef(x,q,o)∑o′∈Oef(x,q,o′),(2)\pi(o)=\frac{e^{f(x,q,o)}}{\sum_{o'\in\mathcal{O}}e^{f(x,q,o')}}, \tag*{(2)}

and sort⁡↓(π)\operatorname{sort}_{\downarrow}(\pi) denotes sorting by decreasing π\pi with ties broken by option text. Both operators return the same ordered list under every input order.

Iterative Invariant Jev

IJev improves a System One choice decision by iterating a step that is already order-invariant. Pointwise sort supplies the initial order. Each round replaces that order by the ranking of a prior-corrected cyclic average, then repeats. If the orders reach a fixed point we stop; otherwise we run a fixed budget of rounds. The value we return is an average of the round distributions, not the last round. Algorithm 1 is the shared-pool procedure. When each item has its own option set, the same updates apply per item and the prior is estimated inside groups that share a question group, an option count, and a shift (§ 5.1).

Algorithm 1 Iterative Invariant Jev (IJev)
Require: states X = {x_i}_{i=1}^N, question q, options O (K = |O|), round budget T, strength β, prior type
Ensure: round-averaged distribution P^{(i)}
1: for i ← 1, . . . , N do
2:     f_i(o) ← log P(yes | x_i, q, o) − log P(no | x_i, q, o)
3:     π_i ← softmax_{o∈O} f_i(o)
4:     τ_1^{(i)} ← sort_↓(π_i), ties by option text
5: end for
6: for t ← 1, . . . , T do
7:     for i ← 1, . . . , N do
8:         for s ← 1, . . . , K do
9:             λ_{t,s}^{(i)} ← s-th cyclic shift of τ_t^{(i)}
10:            read g_{λ_{t,s}^{(i)}}(o) for all o ∈ O
11:        end for
12:    end for
13:    b_{t,s}(j) ← label-free prior on position j at shift s
14:    for i ← 1, . . . , N do
15:        ℓ_t^{(i)} ← log-average in {xref}`Eq. (3)`
16:        r_t^{(i)} ← softmax(ℓ_t^{(i)})
17:        τ_{t+1}^{(i)} ← sort_↓(r_t^{(i)}), ties by option text
18:    end for
19:    if τ_{t+1}^{(i)} = τ_t^{(i)} for every i then
20:        break ▷ fixed point; r_t is stationary
21:    end if
22: end for
23: T_run ← number of executed rounds
24: P^{(i)} ← \frac{1}{T_{\mathrm{run}}} \sum_{t=1}^{T_{\mathrm{run}}} r_t^{(i)}
25: return P^{(i)} and {(τ_t^{(i)}, r_t^{(i)})}_{t≤T_{\mathrm{run}}}

Cyclic classes. For an order τ=(t1,…,tK)\tau=(t_1,\ldots,t_K), the cyclic class [τ]={τs:s=1,…,K}[\tau]=\{\tau_s:s=1,\ldots,K\} is the set of its cyclic shifts. The K!K! orders partition into (K−1)!(K-1)! classes, and a cyclic average over [τ][\tau] uses exactly one of them. If Φ\Phi is an invariant ordering operator, [Φ(x,q,O)][\Phi(x,q,\mathcal{O})] does not depend on the input order, so the cyclic average is input-order-invariant. It still depends on which operator was used. Text sort and pointwise sort generally select different classes, and a model that is not rotation-symmetric can assign them different distributions. Iteration is a way to move off that arbitrary initial class.

Initial order. The default operator is pointwise sort. Compute π\pi from Eq. (2) and set τ1=sort⁡↓(π)\tau_1=\operatorname{sort}_{\downarrow}(\pi), with ties broken by option text, in the initialization of Algorithm 1. Text sort is a zero-cost alternative. The invariance argument applies to any operator in Definition 1; only the class it selects changes.

One cyclic round. At round tt, let λt,s\lambda_{t,s} be the ss-th cyclic shift of τt=(t1,…,tK)\tau_t=(t_1,\ldots,t_K), and let pos⁡t,s(o)\operatorname{pos}_{t,s}(o) be the position of option oo in that listing. Estimate a position prior bt,s(j)b_{t,s}(j) without labels: the batch prior is the mean listwise position distribution of the evaluation batch at shift ss, and a content-free prior is the same mean on probe states. Correct each listing and average in log space,

rt(o)∝exp⁡(1K∑s=1Klog⁡gλt,s(o)bt,s(pos⁡t,s(o))β).(3)r_t(o)\propto\exp\left(\frac{1}{K}\sum_{s=1}^{K}\log\frac{g_{\lambda_{t,s}}(o)}{b_{t,s}(\operatorname{pos}_{t,s}(o))^\beta}\right). \tag*{(3)}

This is AnyJev-L0, Eq. (1), applied to the current invariant order rather than to the input order. We use β=1.0\beta=1.0 and the batch prior. Here gλ(o)g_\lambda(o) is the renormalized next-token probability of the letter in front of oo when the model is shown listing λ\lambda.

Reordering. The next invariant order ranks the just-computed distribution,

τt+1=sort⁡↓(rt),(4)\tau_{t+1}=\operatorname{sort}_{\downarrow}(r_t), \tag*{(4)}

again with ties broken by option text. Round t+1t+1 averages the cyclic shifts of τt+1\tau_{t+1}. Because rtr_t does not depend on the input order, neither does τt+1\tau_{t+1}.

Round averaging. A fixed point makes rtr_t stationary, and the last round is then enough. A cycle repeats a finite set of distributions. A trajectory that is still moving makes the last round an arbitrary sample. We therefore return an average over the rounds that were run. Writing TT for that number, the default is the uniform average

PT(o)=1T∑t=1Trt(o).(5)P_T(o)=\frac{1}{T}\sum_{t=1}^{T}r_t(o). \tag*{(5)}

We also evaluate the log-average

PTlog⁡(o)∝exp⁡(1T∑t=1Tlog⁡rt(o)),(6)P_T^{\log}(o)\propto\exp\left(\frac{1}{T}\sum_{t=1}^{T}\log r_t(o)\right), \tag*{(6)}

using the same probability floor as the prior correction, and the exponential average PT(o)∝∑t=1Twtrt(o)P_T(o)\propto\sum_{t=1}^{T}w_t r_t(o) with wt∝e−(T−t)/τww_t\propto e^{-(T-t)/\tau_w}.

Proposition 1 (Invariance of every iterate and of the round average). For a deterministic model, if the initial order τ1\tau_1 is produced by an invariant ordering operator, then every τt\tau_t, every rtr_t, and every round average PTP_T is independent of the input order of O\mathcal{O}. (Proofs are in Appendix § 7.)

Proposition 2 (Termination or cycle). With a fixed prior, the map τt↦τt+1\tau_t\mapsto\tau_{t+1} is deterministic on the K!K! orders of one state, so the sequence reaches a fixed point or a cycle within K!+1K!+1 rounds. With the batch prior, the prior at round tt depends on all NN orders, the state is the joint order vector, and the same counting argument gives a cycle within (K!)N+1(K!)^N+1 rounds.

What iteration changes. One cyclic average depends on the class [τ1]. Reordering by rτr_{\tau} and averaging again moves to another class, or returns to the same one. The round average is an ensemble over the classes that were visited. It does not cover all (K−1)!(K-1)! classes, so it is not full permutation marginalization. It does remove the privilege of whichever class the initial operator happened to select. In our runs the trajectory is still moving at T=20T=20, which is why the last round is not a stable rule.

Cost. Pointwise sort costs KK yes/no reads. Each round costs KK listwise reads and K2K^2 option slots, so TT rounds cost TKTK listwise reads. For K=20K=20 and T=20T=20 that is 400 listwise reads, twenty times the 20 reads of L0. The batch prior is computed from those reads and adds none. A content-free prior would add probe reads. IJev spends this compute on invariance and calibration; it is not a cheaper readout. Future work could develop cheaper methods while maintaining the accuracy.

Experiments

Setup

Benchmarks. We evaluate five single-label choice benchmarks. Banking20 is the BANKING77 test split restricted to the 20 most frequent intents: 799 items with a shared set of 20 intent options, all evaluated in the seed-0 order. Newsgroups is the standard 300-item test sample of the 20-way topic classification task, again with a shared option set. MMLU-Pro (Wang et al., [16]) contributes 1,400 multiple-choice questions, 100 per category, with 3–10 options each. Typed Decisions (LocalLLaMA, [9]) contains 600 choice questions with 4–5 options, and the public items of JevBench (Standhartinger, [12]) contain 139 choice questions with 3–6 options. The last three assign every question its own option set and option count; there we use the per-item form of the operator, with cyclic shifts of each question’s own order and a prior grouped by question group, option count, and shift. There is no training and no calibration set; every prior is estimated without labels.

Models. We use five instruction-tuned models from three families, spanning 1.7B–8B parameters: Qwen3-1.7B, Qwen3-4B and Qwen3-8B (Yang et al., [19]); Mistral-7B-Instruct-v0.3 (Jiang et al., [7]); and Phi-3.5-mini-instruct (Abdin et al., [1]). All runs are fp16 on a single NVIDIA RTX 4090 D (24 GB), with PyTorch 2.6.0 and CUDA 12.4, and use the model’s chat template.

Implementation. The listwise readout gλg_{\lambda} reads the next-token probabilities of the option letters and renormalizes them. The pointwise probe reads the yes/no next-token log-odds for one candidate at a time. Every prompt is rendered with the chat template, sorted by prompt text within a batch, and read with a single forward pass, so the batch composition does not depend on the option order. Decoding is deterministic on a fixed model and prompt set. Appendix § 8 gives the prompt, prior, and precision settings.

Metrics. Accuracy (Acc) is the fraction of items whose top-1 option is correct and measures decision quality on the original listing. Flip is the fraction whose top-1 option changes under one independent shuffle; it measures input-order sensitivity directly, and an order-invariant readout has flip 0. ECE compares confidence with accuracy over B=15B=15 equal-mass bins SbS_b, so ECE is the bin-size-weighted average calibration gap (0 is perfect calibration). Brier is the mean squared error of the whole predicted distribution over the KiK_i options, a proper score that rewards distributions that are both calibrated and sharp. NLL is the mean negative log-probability of the correct option, also a proper score; it penalizes confident mistakes most heavily and is unbounded above. Accuracy and flip rate are percentages; lower ECE, Brier, and NLL are better. Details are in Appendix § 9.

Methods. We compare the one-pass listwise readout on the input order (LISTWISE), AnyJev-L0 with the label-free batch prior (Eq. (1)), and IJev started from the pointwise order with β=1.0\beta=1.0. AnyJev-L0 and the iterative runs both use β=1.0\beta=1.0 throughout, so the prior correction is identical across all five benchmarks. The iteration has one distribution per round; we report the last round, the uniform average, the log-average, and the exponential averages with τw=10\tau_w=10 and 20, each at T=20T=20. For every model and benchmark we also compute each rule at every horizon T=1,…,20T=1,\ldots,20 and define IJev (best) as the (rule, TT) pair with the highest accuracy, breaking ties by lower NLL. That row selects the horizon on the same test set, so it is a descriptive upper reference rather than a confirmatory result; the fixed T=20T=20 uniform average is the default rule in the detailed tables.

Main results

Table 1 compares the listwise readout, AnyJev-L0, and IJev on five models and five benchmarks. Flip rate is the clean separation. Listwise flip runs from 7.9% to 59.7%, and AnyJev-L0 reduces it to 1.4–39.4%, but never to zero, because L0 still rotates the input order. Every IJev cell has flip 0.0, as Proposition 1 guarantees. Consistent with that proposition, the 20-round runs were not repeated under a shuffle (§ 5.3).

Calibration, not accuracy, is the reliable gain of the default rule. Table 2 reports ECE, Brier, and NLL for the same systems. Relative to AnyJev-L0, the uniform average at T=20T=20 lowers ECE in all 25 cells, Brier in 23, and NLL in 24. The misses are on Typed Decisions: Brier ties AnyJev-L0 for Qwen3-4B, and both Brier and NLL are worse for Phi-3.5-mini. The uniform rule has the lowest NLL of the five methods in 24 of the 25 cells, and the log-average is less calibrated than the uniform average even when the two accuracies are close.

ModelMethodBanking20 ECE↓Banking20 Brier↓Banking20 NLL↓Newsgroups ECE↓Newsgroups Brier↓Newsgroups NLL↓MMLU-Pro ECE↓MMLU-Pro Brier↓MMLU-Pro NLL↓Typed Dec. ECE↓Typed Dec. Brier↓Typed Dec. NLL↓JevBench ECE↓JevBench Brier↓JevBench NLL↓
Qwen3-1.7BListwise (raw)0.360.736.560.390.797.030.631.317.590.511.027.290.400.814.14
AnyJev-L00.270.573.380.350.724.720.461.084.700.370.833.030.360.763.71
IJev (uniform)0.190.471.840.280.642.980.310.943.320.250.722.500.280.642.13
IJev (log-avg)0.250.532.620.320.694.120.371.003.770.340.752.860.330.722.63
IJev (best)0.190.492.280.310.664.110.330.973.600.250.722.500.310.693.05
Qwen3-4BListwise (raw)0.260.525.070.340.696.170.501.066.800.400.835.960.320.644.01
AnyJev-L00.210.452.990.310.634.720.400.914.580.290.672.470.320.643.58
IJev (uniform)0.160.412.060.240.552.700.290.833.390.280.672.290.230.522.33
IJev (log-avg)0.200.432.880.300.613.770.350.883.890.350.742.890.290.582.69
IJev (best)0.160.412.160.280.594.140.350.873.890.290.682.310.220.522.43
Qwen3-8BListwise (raw)0.230.474.300.330.685.720.470.995.890.370.764.930.270.553.61
AnyJev-L00.180.362.480.310.634.270.370.843.980.270.612.820.250.533.10
IJev (uniform)0.140.331.690.260.582.920.240.742.770.230.592.680.200.492.04
IJev (log-avg)0.170.352.430.320.644.090.310.783.250.280.623.040.230.502.28
IJev (best)0.130.331.700.250.583.100.310.783.250.280.623.040.200.492.09
Mistral-7BListwise (raw)0.250.523.490.320.664.960.541.206.600.350.773.760.330.713.35
AnyJev-L00.200.431.950.290.613.700.481.094.540.250.681.870.340.723.12
IJev (uniform)0.110.361.100.190.511.950.280.913.050.230.641.620.260.642.09
IJev (log-avg)0.180.401.530.270.582.610.360.983.510.280.671.850.290.672.32
IJev (best)0.110.371.260.190.522.330.350.983.520.230.651.690.290.652.41
Phi-3.5-miniListwise (raw)0.190.412.700.310.664.670.430.994.470.360.763.330.360.774.75
AnyJev-L00.160.341.590.330.673.700.350.883.390.250.651.710.370.764.25
IJev (uniform)0.090.301.040.200.521.750.200.782.440.240.671.870.300.673.04
IJev (log-avg)0.130.311.410.260.592.570.250.812.690.320.712.250.350.723.39
IJev (best)0.120.311.430.200.532.090.210.792.510.240.671.870.320.703.32

Table 2. ECE, Brier score, and NLL for the five methods on all five benchmarks. Lower is better in every column. The iterative rows are the uniform and log-average rules at T=20T = 20, and “best” selects the post-hoc accuracy-maximizing horizon. Bold marks the lowest value per model and column.

Accuracy should be read in two layers. The confirmatory rule is the uniform average at the pre-specified budget T=20T=20. Averaged over the five benchmarks it matches or exceeds AnyJev-L0 for every model (Table 1, Avg column): the margins are +1.8, +0.4, 0.0, +1.8, and +1.0 points on Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Mistral-7B, and Phi-3.5-mini. Individual cells move both ways. The largest gain is Typed Decisions on Qwen3-1.7B (49.0 to 57.7); JevBench loses a point or two for three of the five models. The best row, which selects the horizon and the averaging rule on the same test set, is the highest-accuracy readout in all 25 model–benchmark cells and ties AnyJev-L0 on Typed Decisions with Qwen3-4B. Averaged over models, that descriptive row gains 1.7 points over L0 on Banking20, 2.9 on Newsgroups, 2.2 on MMLU-Pro, 2.5 on Typed Decisions, and 3.5 on JevBench. We do not treat those gains as confirmed. § 5.3 reports the paired tests for the fixed rule, which are positive for accuracy, and large for calibration.

ModelMethodBanking20 Acc↑Banking20 Flip↓Newsgroups Acc↑Newsgroups Flip↓MMLU-Pro Acc↑MMLU-Pro Flip↓Typed Dec. Acc↑Typed Dec. Flip↓JevBench Acc↑JevBench Flip↓Avg Acc↑Avg Flip↓
Qwen3-1.7BListwise (raw)62.133.759.028.727.659.747.339.054.728.150.137.8
AnyJev-L069.511.861.319.733.231.249.025.759.09.454.419.5
IJev (uniform, T=20T=20)70.30.062.00.033.20.057.70.057.60.056.20.0
IJev (log-avg, T=20T=20)70.50.063.00.033.30.057.50.058.30.056.50.0
IJev (best)72.00.065.30.034.60.057.70.061.90.058.30.0
Qwen3-4BListwise (raw)73.121.963.720.040.948.655.824.765.512.959.825.6
AnyJev-L076.313.066.711.746.423.959.815.764.75.062.813.8
IJev (uniform, T=20T=20)77.10.067.30.046.00.059.30.066.20.063.20.0
IJev (log-avg, T=20T=20)77.20.067.70.046.00.059.20.064.70.063.00.0
IJev (best)77.60.069.70.046.90.059.80.069.80.064.70.0
Qwen3-8BListwise (raw)75.818.464.320.744.546.560.217.071.27.963.222.1
AnyJev-L081.27.666.312.750.423.765.29.571.91.467.011.0
IJev (uniform, T=20T=20)81.70.066.00.051.90.065.00.070.50.067.00.0
IJev (log-avg, T=20T=20)81.90.065.30.052.30.065.80.071.90.067.40.0
IJev (best)82.00.066.70.052.30.066.20.073.40.068.10.0
Mistral-7BListwise (raw)72.631.964.730.030.655.356.022.259.012.956.630.5
AnyJev-L076.313.667.011.332.839.458.321.359.05.858.718.3
IJev (uniform, T=20T=20)76.80.069.00.036.60.059.70.060.40.060.50.0
IJev (log-avg, T=20T=20)77.50.067.70.037.10.059.50.059.70.060.30.0
IJev (best)78.00.070.00.037.60.060.30.062.60.061.70.0
Phi-3.5-miniListwise (raw)76.826.264.328.340.447.156.216.258.312.259.226.0
AnyJev-L080.29.664.311.343.327.458.816.058.35.061.013.9
IJev (uniform, T=20T=20)82.00.066.00.045.10.059.20.057.60.062.00.0
IJev (log-avg, T=20T=20)81.70.066.70.045.30.058.80.056.80.061.90.0
IJev (best)82.50.068.30.045.70.059.50.062.60.063.70.0

Table 1. Accuracy and flip rate on five decision benchmarks. Avg is the unweighted mean over the five datasets. Bold marks the best value for each model and column.

Banking20: Analysis

Banking20 is the only benchmark with the full 799-item pool, so the paired tests are reported here. Appendix Table 6 lists every averaging rule at T=20T=20. The 20-round runs were executed on the original option order only. A second 20-round pass would have doubled an already expensive sweep, and it would have been identical: the initial order is invariant, prompts are sorted canonically, and the cyclic average does not read the input order (Proposition 1). Listwise and AnyJev-L0 flips were measured under one per-item shuffle.

ModelMethodAccFlipECEBrierNLL
Qwen3-1.7B, Banking20, full 799
Listwise (raw)62.133.70.3550.7286.560
AnyJev-L069.511.80.2700.5713.382
IJev (last round)69.10.00.2710.5682.979
IJev (uniform average)70.30.00.1900.4711.839
IJev (log-average)70.50.00.2540.5322.619
IJev (exp. average, τw=10\tau_w = 10)71.00.00.1860.4711.864
IJev (exp. average, τw=20\tau_w = 20)70.80.00.1860.4701.848
Best accuracy (post hoc, avg. last 3, T=18T = 18)72.00.00.1930.4872.276
Qwen3-4B, Banking20, full 799
Listwise (raw)73.121.90.2570.5245.073
AnyJev-L076.313.00.2140.4492.988
IJev (last round)75.50.00.2260.4683.102
IJev (uniform average)77.10.00.1560.4072.064
IJev (log-average)77.20.00.2030.4352.880
IJev (exp. average, τw=10\tau_w = 10)76.60.00.1640.4112.074
IJev (exp. average, τw=20\tau_w = 20)77.10.00.1600.4092.065
Best accuracy (post hoc, uniform, T=14T = 14)77.60.00.1570.4062.155
Qwen3-8B, Banking20, full 799
Listwise (raw)75.818.40.2260.4674.300
AnyJev-L081.27.60.1750.3622.484
IJev (last round)79.70.00.1860.3782.630
IJev (uniform average)81.70.00.1350.3321.685
IJev (log-average)81.90.00.1660.3502.425
IJev (exp. average, τw=10\tau_w = 10)81.60.00.1350.3351.707
IJev (exp. average, τw=20\tau_w = 20)81.70.00.1360.3331.693
Best accuracy (post hoc, uniform, T=18T = 18)82.00.00.1330.3321.697

Table 6. Banking20 full-pool results (799 items). Acc and Flip are percentages; ECE, Brier, and NLL are lower-is-better. AnyJev-L0 uses the library’s full batch prior; iterative runs use β=1.0\beta= 1.0 and T=20T = 20. Iterative flip is 0.0 by Proposition 1; the 20-round runs were executed on the original order only, while the AnyJev flip rates are measured under the standard shuffle. The last round is shown as a control and is not the recommended rule. The uniform T=20T = 20 row has the best NLL; the best-accuracy rows select the horizon on the same test set and are descriptive.

Accuracy. Under the fixed T=20T = 20 rule the uniform average gains 0.8, 0.8, and 0.5 points over AnyJev-L0 on 1.7B, 4B, and 8B (7, 7, and 4 items). Exact McNemar tests give 29 fixes and 22 breaks (p=0.40p = 0.40), 28 and 22 (p=0.48p = 0.48), and 21 and 17 (p=0.63p = 0.63); the gains are positive but not significant. The exponential averages stay within 0.3 points of the uniform rule. The post-hoc best rows add a further 0.3–1.7 points, but they choose the horizon on this test set. Their McNemar tests (38/18, 30/20, and 21/15 fixes/breaks) give p=0.01p = 0.01, 0.20, and 0.41 and are descriptive, not confirmatory. Only the 1.7B descriptive comparison would clear a conventional threshold, and it was not pre-specified.

Calibration. The calibration gains are larger and statistically reliable. Paired bootstrap intervals exclude zero on all three models. Brier improves by 0.100 (95% CI 0.071–0.129), 0.042 (0.013–0.071), and 0.030 (0.007–0.054) on 1.7B, 4B, and 8B, and NLL by 1.543 (1.292–1.799), 0.924 (0.711–1.141), and 0.799 (0.616–0.991). ECE falls in every cell (Table 2). Exponential averages stay within 0.03 NLL of the uniform rule. The log-average is consistently worse on both Brier and NLL.

Round-count ablation

Table 3 reports accuracy of the uniform and log-average rules at T=1,2,5,10,15,20T = 1, 2, 5, 10, 15, 20 for every model and benchmark. Appendix § 11 adds ECE and NLL on Banking20, and Appendix § 11 records every horizon.

ModelT=1T=1T=2T=2T=5T=5T=10T=10T=15T=15T=20T=20T=1T=1T=2T=2T=5T=5T=10T=10T=15T=15T=20T=20
Banking20
Qwen3-1.7B69.570.570.670.870.270.369.570.870.770.870.170.5
Qwen3-4B76.075.876.876.277.377.176.076.277.076.677.277.2
Qwen3-8B80.481.482.081.481.581.780.481.681.781.581.581.9
Mistral-7B73.375.176.776.877.076.873.376.877.077.077.677.5
Phi-3.5-mini78.281.080.981.681.682.078.281.682.082.081.681.7
Newsgroups
Qwen3-1.7B65.064.363.061.361.762.065.065.363.062.762.063.0
Qwen3-4B65.768.067.066.367.367.365.767.767.766.767.367.7
Qwen3-8B64.765.065.766.365.766.064.765.366.366.065.765.3
Mistral-7B66.067.368.068.068.769.066.066.768.068.068.067.7
Phi-3.5-mini64.365.767.066.366.366.064.366.367.067.367.066.7
MMLU-Pro
Qwen3-1.7B32.832.932.532.933.533.232.833.032.733.633.333.3
Qwen3-4B45.145.646.146.446.046.045.146.146.046.446.546.0
Qwen3-8B49.650.650.851.752.051.949.651.151.951.951.952.3
Mistral-7B32.935.435.336.036.636.632.935.536.636.737.637.1
Phi-3.5-mini41.643.144.045.045.145.141.643.645.044.945.245.3
Typed Decisions
Qwen3-1.7B55.254.356.556.856.557.755.255.057.057.257.557.5
Qwen3-4B59.259.559.559.559.259.359.259.259.759.259.059.2
Qwen3-8B64.064.065.364.565.765.064.064.365.565.866.065.8
Mistral-7B56.858.759.258.859.259.756.859.259.759.359.359.5
Phi-3.5-mini56.258.758.559.058.559.256.258.858.859.058.858.8
JevBench
Qwen3-1.7B60.456.858.356.858.357.660.458.359.058.358.358.3
Qwen3-4B67.665.566.265.566.266.267.664.764.064.064.064.7
Qwen3-8B69.872.771.971.271.970.569.871.972.770.571.971.9
Mistral-7B57.659.059.759.060.460.457.659.059.760.459.759.7
Phi-3.5-mini56.859.756.857.657.657.656.859.058.356.856.856.8

Table 3. Accuracy (%) of IJev at T=1,2,5,10,15,20T = 1, 2, 5, 10, 15, 20 for the uniform and log-average rules, every model and benchmark. All iterative cells are input-order-invariant (flip 0). Bold marks the best cell in each model row.

Calibration moves first. Most of the NLL reduction is present by five to ten rounds, and for the uniform average the best NLL horizon is T=20T = 20 in every setting we checked. Accuracy does not move the same way. Individual rounds are uneven, so a short window can dip below the cumulative average. On Banking20 the uniform average is already within 0.5 points of its T=20T = 20 value by T=5T = 5 for all three Qwen models, and on Newsgroups the curves are flat or mildly non-monotone after T=2T = 2. The single best accuracy, however, often sits at an intermediate horizon rather than at the endpoint (T=18T = 18 on 1.7B, T=14T = 14 on 4B, and T=18T = 18 on 8B for Banking20). That is why T=20T = 20 uniform is the default: it is the stable calibration rule, not the accuracy maximizer. The accuracy-maximizing horizon is chosen after seeing the test set. The log-average matches uniform accuracy at its own best horizon but remains less calibrated (Table 2); its best NLL is T=15T = 15 on Banking20 with the 4B model and T=10T = 10 on Newsgroups with the 1.7B model, rather than T=20T = 20.

Runtime

Table 4 reports seconds per decision on one RTX 4090 D, wall time divided by the number of states, including batching and the pointwise pass. Listwise is one forward pass per state. AnyJev-L0 performs KK cyclic reads and reuses them for the batch prior. IJev adds the pointwise reads and TKTK listwise reads, and the tabulated time includes every round up to that TT. At T=20T = 20 the iterative readout costs 20–22× AnyJev-L0, in line with TKTK versus KK reads; the pointwise pass is only 2–4% of the total. Newsgroups is slower than Banking20 because its prompts are longer. The method is a compute-for-calibration trade-off. § 5.4 discusses the practical consequence: five to ten rounds already capture most of the NLL gain when a full budget of 20 is too expensive.

ModelTaskListwiseAnyJev-L0T=1T=1T=2T=2T=5T=5T=10T=10T=15T=15T=20T=20
1.7BBanking200.0080.1410.2110.3630.8191.5782.3373.096
4BBanking200.0170.3330.4830.8251.8503.5575.2656.972
8BBanking200.0300.5890.8211.4133.1936.1469.09612.045
1.7BNewsgroups0.0140.2440.4450.6971.4542.7153.9785.239
4BNewsgroups0.0310.5571.0031.5713.2736.1108.94711.785
8BNewsgroups0.0490.9181.6172.5345.2849.86914.45519.041

Table 4. Seconds per decision (wall time / number of states). Iterative columns use the uniform round average. Banking20 uses 799 states; Newsgroups uses 300.

Conclusion

To improve a System One choice decision, this paper introduces IJev, which iterates that reorder-and-average step and returns an average of the round distributions, so the decision does not rest on the last round of a trajectory that, in our runs, has not converged. On five benchmarks and five models the procedure has a flip rate of zero everywhere. Experiments also show IJev’s advantage in calibration and accuracy.

Limitations

The iteration costs about TT times one cyclic average. The round average is an ensemble over the classes the trajectory happens to visit, not a proof of full permutation symmetry, and Proposition 2 only guarantees that a cycle exists somewhere in a state space too large to enumerate. The batch prior assumes a representative evaluation batch; a pathological batch can distort it, and items inside the batch are coupled through that prior. The invariance proof assumes a deterministic model and canonically sorted prompts. An interface that returns nondeterministic logits can still flip. The method also inherits the pointwise initialization: when π\pi is nearly flat, the initial order carries little information and later rounds can lose accuracy.

Proofs

Proof of Proposition 1. The initial order τ1\tau_1 is invariant by Definition 1. Assume τt\tau_t is invariant for every state. Every cyclic shift λt,s\lambda_{t,s} is a function of τt\tau_t. With a fixed prior, bt,sb_{t,s} does not depend on the data order. With the batch prior, bt,sb_{t,s} is the mean over a canonically sorted batch, or over content-free probes with the same listing, so it is a function of the current invariant orders. Equation (3) is therefore a function of (x,q,O)(x,q,\mathcal{O}) and is invariant. Equation (4) reorders that distribution and breaks ties by option text, so τt+1\tau_{t+1} is invariant. By induction every iterate is invariant. Any convex combination of invariant distributions, including the uniform and exponential averages, is invariant. The log-average is a deterministic function of the same distributions and is invariant as well. □

Proof of Proposition 2. With a fixed prior, the transition is a deterministic map on a finite set of K!K! orders. Among any K!+1K!+1 consecutive orders, one repeats, so the sequence enters a cycle. A cycle of period one is a fixed point. With the batch prior, the prior at round tt depends on every state’s order, so the deterministic state is the joint vector of NN orders. There are (K!)N(K!)^N such vectors, and the same counting argument yields a cycle within (K!)N+1(K!)^N+1 rounds. Recomputing cyclic shifts from the current order does not enlarge that set. □

Implementation details

Prompts and readouts. The listwise prompt renders the question and the KK options under the letters A, B, …, and we read the next-token probabilities of those letters at the first answer position, renormalizing them over the KK letters to obtain gλg_\lambda. The pointwise prompt shows one candidate and asks whether it is the correct answer; we read the yes/no next-token log-odds f(o)=log⁡P(yes)−log⁡P(no)f(o)=\log P(\text{yes})-\log P(\text{no}) and set π=softmax⁡of(o)\pi=\operatorname{softmax}_o f(o). All prompts in a batch are rendered with the model’s chat template and sorted by prompt text before the forward pass, so the batch composition and padding do not depend on the option order; ties in every sort are broken by option text. Runs use fp16 with one forward pass per prompt (only the first answer token is scored), batch sizes of 2–64 chosen per model and benchmark to fit the 24 GB device, seed 0 for the data order, and shuffle seed 1234 for the flip measurements. The iteration is deterministic on a fixed model and prompt set; the iterative flip-0 result follows from Proposition 1, while the Listwise and AnyJev flips are measured under one per-item shuffle.

Cost per decision. A listwise pass creates KK prompts per state, one per cyclic shift, and scores the KK letters in each; the pointwise pass creates KK prompts. AnyJev-L0 therefore costs KK prompts per state and IJev costs K(T+1)K(T+1): the measured 20–22× ratio at T=20T=20 is consistent with TKTK versus KK, and the pointwise pass contributes only 2–4% of the total.

Label priors. Both prior variants are distributions over list positions, not over option identities, and neither uses labels. For a listing λ\lambda with KK positions, the prior is bλ(j)=1∣R∣∑r∈Rgλ(r)(j)b_\lambda(j)=\frac{1}{|R|}\sum_{r\in R}g_\lambda^{(r)}(j), the mean probability mass that the listwise readout places on position jj over a label-free reference set RR.

  • Batch prior (used in every experiment): RR is the evaluation batch itself, that is, the states evaluated together, rather than a running estimate. It is re-estimated for every cyclic shift λt,s\lambda_{t,s} of the current order and for every round, from the same listings that produce gg, so it adds no forward passes.

SettingValue
Prior typebatch position prior
Reference setevaluation batch
Per-item grouping(group, KK, shift), ≥8\ge8 states
FallbackKK-pooled; none if <2< 2 states
Content-free probesavailable, not used
Prior strength β\beta1.0 (AnyJev-L0 and IJev)
Clipping ε\varepsilon10−1210^{-12} (float64)
Extra forward passesnone
Iterative horizonT=20T=20; best over T=1,…,20T=1,\ldots,20
Data / shuffle seed0 / 1234
Precisionfp16, one pass per prompt

Table 5. Implementation and label-prior settings.

  • *Per-item benchmarks: option sets differ across questions, so states are grouped by (question group, KK, shift) and a separate prior is estimated for each group with at least 8 states; smaller groups fall back to a prior pooled over all states with the same KK, and a shift with fewer than 2 states receives no correction. In MMLU-Pro each of the 14 categories mixes several option counts, so only 22 of its 83 (group, KK) cells reach 8 states and the other 61 use the KK-pooled prior; Typed Decisions and JevBench have at most one option count per group and use group priors throughout.

  • *Content-free prior (available, not used): RR consists of probes with an empty state and the same question and listing. All reported AnyJev-L0 and IJev numbers use the batch prior.

For each shift the prior is applied as p′(j)∝gλ(j)/bλ(j)βp'(j) \propto g_\lambda(j)/b_\lambda(j)^\beta with β=1.0\beta=1.0, evaluated in float64 with probabilities and priors clipped at ε=10−12\varepsilon=10^{-12} and renormalized; β=0\beta=0 recovers the plain cyclic average:

pσperm(o)∝exp⁡(1K∑s=1Klog⁡gσs(o)).(7)p_\sigma^{\mathrm{perm}}(o) \propto\exp\left(\frac{1}{K}\sum_{s=1}^{K}\log g_{\sigma_s}(o)\right). \tag*{(7)}

The corrected distributions enter the log-space geometric mean over shifts in Eq. (3). Because the prior is recomputed from the current listings, it remains a function of the batch and of the (invariant) current order; no labels or held-out data enter at any step.

Metrics

Let pip_i be the predicted distribution for item ii, yiy_i its label, y^i=arg⁡max⁡kpi(k)\hat{y}_i=\arg\max_k p_i(k), and y^i′\hat{y}'_i the prediction after one independent per-item shuffle of the options. We report

Acc=1N∑i=1N1[y^i=yi],(8)\mathrm{Acc}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat{y}_i=y_i], \tag*{(8)}
Flip=1N∑i=1N1[y^i≠y^i′],(9)\mathrm{Flip}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat{y}_i\ne\hat{y}'_i], \tag*{(9)}
ECE=∑b=1B∣Sb∣N∣acc⁡(Sb)−conf⁡(Sb)∣,(10)\mathrm{ECE}=\sum_{b=1}^{B}\frac{|S_b|}{N}\left|\operatorname{acc}(S_b)-\operatorname{conf}(S_b)\right|, \tag*{(10)}
Brier=1N∑i=1N∑k=1Ki(pi(k)−1[yi=k])2,(11)\mathrm{Brier}=\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{K_i}\left(p_i(k)-\mathbf{1}[y_i=k]\right)^2, \tag*{(11)}
NLL=−1N∑i=1Nlog⁡pi(yi),(12)\mathrm{NLL}=-\frac{1}{N}\sum_{i=1}^{N}\log p_i(y_i), \tag*{(12)}

where KiK_i is the number of options of item ii. Accuracy (Acc) is the fraction of items whose top-1 option is correct and measures decision quality on the original listing. Flip is the fraction whose top-1 option changes under one independent shuffle; it measures input-order sensitivity directly, and an order-invariant readout has flip 0. ECE compares confidence with accuracy over B=15B=15 equal-mass bins SbS_b: conf⁡(Sb)\operatorname{conf}(S_b) is the mean top-1 probability in the bin and acc⁡(Sb)\operatorname{acc}(S_b) the empirical accuracy there, so ECE is the bin-size-weighted average calibration gap (0 is perfect calibration). Brier is the mean squared error of the whole predicted distribution over the KiK_i options, a proper score that rewards distributions that are both calibrated and sharp. NLL is the mean negative log-probability of the correct option, also a proper score; it penalizes confident mistakes most heavily and is unbounded above. Accuracy and flip rate are percentages; lower ECE, Brier, and NLL are better.

Additional full tables

Tables 6 and 7 list every averaging rule at T=20T=20 for Banking20 and Newsgroups; Tables 8 and 9 give the uniform and log-average rules at six selected horizons. The compact versions in the main text are Tables 1 and 2.

Per-round results

Tables 10, 11, 12, 13, 14 and 15 record the average over rounds 1,…,T1,\ldots,T for every horizon. “Best at TT” is the rule with the highest accuracy at that horizon; the value after the slash is its NLL. The last column is seconds per decision including the pointwise pass.

MethodAccFlipECEBrierNLL
Qwen3-1.7B, Newsgroups, 300
Listwise (raw)59.028.70.3870.7907.026
AnyJev-L061.319.70.3490.7204.719
IJev (last round)60.00.00.3580.7324.484
IJev (uniform average)62.00.00.2750.6352.983
IJev (log-average)63.00.00.3240.6894.124
IJev (exp. average, τw=10\tau_w = 10)62.30.00.2790.6443.021
IJev (exp. average, τw=20\tau_w = 20)62.30.00.2760.6402.997
Best accuracy (post hoc, logavg, T=2T = 2)65.30.00.3120.6614.108
Qwen3-4B, Newsgroups, 300
Listwise (raw)63.720.00.3400.6926.171
AnyJev-L066.711.70.3100.6314.717
IJev (last round)66.30.00.3020.6304.126
IJev (uniform average)67.30.00.2380.5492.702
IJev (log-average)67.70.00.2980.6113.770
IJev (exp. average, τw=10\tau_w = 10)68.00.00.2350.5502.741
IJev (exp. average, τw=20\tau_w = 20)68.00.00.2340.5492.718
Best accuracy (post hoc, last, T=19T = 19)69.70.00.2770.5914.139
Qwen3-8B, Newsgroups, 300
Listwise (raw)64.320.70.3310.6805.725
AnyJev-L066.312.70.3060.6304.268
IJev (last round)66.00.00.3100.6504.325
IJev (uniform average)66.00.00.2560.5812.919
IJev (log-average)65.30.00.3160.6354.093
IJev (exp. average, τw=10\tau_w = 10)65.70.00.2630.5842.938
IJev (exp. average, τw=20\tau_w = 20)65.70.00.2610.5832.923
Best accuracy (post hoc, uniform, T=9T = 9)66.70.00.2460.5833.103

Table 7. Newsgroups, 300 items. Fixed rules use T=20T = 20; Acc and Flip are percentages. Iterative flip is 0 by Proposition 1; the iterative runs were executed on the original order only. The uniform average’s Brier/NLL improvements over AnyJev-L0 are significant on 1.7B/4B under the paired bootstrap; on 8B the NLL gain is significant and the Brier gain is borderline. Accuracy gains under the fixed rule (1–2 items) are not significant. The best-accuracy rows select the horizon on the same test set and are descriptive.

ModelTTUniform avg Acc / ECE / NLLLog-average Acc / ECE / NLLs/dec
1.7B169.5 / 0.262 / 3.12869.5 / 0.262 / 3.1280.211
270.5 / 0.205 / 2.42770.8 / 0.249 / 2.7790.363
570.6 / 0.193 / 2.10770.7 / 0.249 / 2.6900.819
1070.8 / 0.189 / 1.95170.8 / 0.249 / 2.6781.578
1570.2 / 0.194 / 1.91170.1 / 0.257 / 2.6682.337
2070.3 / 0.190 / 1.83970.5 / 0.254 / 2.6193.096
4B176.0 / 0.223 / 3.61176.0 / 0.223 / 3.6110.483
275.8 / 0.179 / 2.70576.2 / 0.218 / 3.0980.825
576.8 / 0.168 / 2.42677.0 / 0.211 / 2.9941.850
1076.2 / 0.165 / 2.20676.6 / 0.212 / 2.8873.557
1577.3 / 0.153 / 2.14177.2 / 0.203 / 2.8735.265
2077.1 / 0.156 / 2.06477.2 / 0.203 / 2.8806.972
8B180.4 / 0.181 / 2.90880.4 / 0.181 / 2.9080.821
281.4 / 0.149 / 2.27681.6 / 0.170 / 2.5881.413
582.0 / 0.135 / 2.01481.7 / 0.168 / 2.5123.193
1081.4 / 0.132 / 1.79581.5 / 0.168 / 2.4276.146
1581.5 / 0.131 / 1.75181.5 / 0.168 / 2.4329.096
2081.7 / 0.135 / 1.68581.9 / 0.166 / 2.42512.045

Table 8. Round-count ablation for Banking20: uniform average versus log-average at T=1,2,5,10,15,20T = 1, 2, 5, 10, 15, 20. T=1T = 1 is a single cyclic average from the pointwise order. Accuracy in percent; ECE and NLL lower-is-better; s/dec is seconds per decision including the pointwise pass. Bold marks the best accuracy for each rule and model among these horizons; bold NLL marks the best NLL. The full T=1,…,20T = 1, \ldots, 20 tables are in Appendix § 11.

ModelTTUniform avg Acc / ECE / NLLLog-average Acc / ECE / NLLs/dec
1.7B165.0 / 0.317 / 4.79565.0 / 0.317 / 4.7950.445
264.3 / 0.257 / 3.63565.3 / 0.312 / 4.1080.697
563.0 / 0.267 / 3.25463.0 / 0.328 / 4.1051.454
1061.3 / 0.278 / 3.08362.7 / 0.323 / 4.0832.715
1561.7 / 0.277 / 3.04562.0 / 0.332 / 4.1183.978
2062.0 / 0.275 / 2.98363.0 / 0.324 / 4.1245.239
4B165.7 / 0.297 / 4.54065.7 / 0.297 / 4.5401.003
268.0 / 0.233 / 3.43467.7 / 0.291 / 3.8671.571
567.0 / 0.242 / 2.97467.7 / 0.293 / 3.8493.273
1066.3 / 0.245 / 2.82366.7 / 0.306 / 3.8256.110
1567.3 / 0.238 / 2.70267.3 / 0.302 / 3.7938.947
2067.3 / 0.238 / 2.70267.7 / 0.298 / 3.77011.785
8B164.7 / 0.318 / 4.55364.7 / 0.318 / 4.5531.617
265.0 / 0.261 / 3.62665.3 / 0.311 / 4.1502.534
565.7 / 0.254 / 3.28666.3 / 0.303 / 4.1295.284
1066.3 / 0.251 / 3.08466.0 / 0.308 / 4.1169.869
1565.7 / 0.259 / 3.01465.7 / 0.312 / 4.11814.455
2066.0 / 0.256 / 2.91965.3 / 0.316 / 4.09319.041

Table 9. Newsgroups, 300 items: uniform average versus log-average at T=1,2,5,10,15,20T = 1, 2, 5, 10, 15, 20. Bold marks the best accuracy for each rule and model among these horizons and the best NLL.

TTUniform Acc / ECE / NLLExp. τw=10\tau_w = 10 Acc / ECE / NLLBest at TTs/dec
169.5 / 0.262 / 3.12869.5 / 0.262 / 3.128uniform 69.5 / 3.1280.211
270.5 / 0.205 / 2.42770.1 / 0.209 / 2.426logavg 70.8 / 2.7790.363
369.8 / 0.213 / 2.27469.8 / 0.214 / 2.275logavg 70.7 / 2.7780.515
471.1 / 0.187 / 2.14371.2 / 0.185 / 2.143exp10 71.2 / 2.1430.667
570.6 / 0.193 / 2.10770.3 / 0.195 / 2.110logavg 70.7 / 2.6900.819
670.8 / 0.188 / 2.06470.8 / 0.188 / 2.068avg_last3 71.1 / 2.3240.971
770.5 / 0.193 / 2.01170.3 / 0.195 / 2.016logavg 71.0 / 2.6751.122
870.8 / 0.188 / 1.99471.0 / 0.188 / 2.003exp20 71.0 / 1.9971.274
971.0 / 0.186 / 1.96070.3 / 0.193 / 1.972uniform 71.0 / 1.9601.426
1070.8 / 0.189 / 1.95170.7 / 0.191 / 1.964uniform 70.8 / 1.9511.578
1170.6 / 0.191 / 1.95870.3 / 0.195 / 1.978uniform 70.6 / 1.9581.729
1270.5 / 0.191 / 1.96370.2 / 0.194 / 1.990uniform 70.5 / 1.9631.881
1370.5 / 0.191 / 1.94470.2 / 0.196 / 1.968avg_last3 70.7 / 2.5062.033
1470.3 / 0.192 / 1.92670.5 / 0.192 / 1.948exp10 70.5 / 1.9482.185
1570.2 / 0.194 / 1.91170.5 / 0.194 / 1.937avg_last3 71.2 / 2.3442.337
1670.2 / 0.194 / 1.90370.6 / 0.192 / 1.930avg_last5 71.3 / 2.2212.489
1770.3 / 0.193 / 1.89170.5 / 0.195 / 1.921avg_last5 71.2 / 2.1962.640
1870.6 / 0.189 / 1.87270.6 / 0.190 / 1.901avg_last3 72.0 / 2.2762.792
1970.3 / 0.191 / 1.86170.8 / 0.188 / 1.888avg_last3 71.2 / 2.2852.944
2070.3 / 0.190 / 1.83971.0 / 0.186 / 1.864avg_last3 71.1 / 2.2233.096

Table 10. Qwen3-1.7B, Banking20, full 799, every horizon.

TTUniform Acc / ECE / NLLExp. τw=10\tau_w = 10 Acc / ECE / NLLBest at TTs/dec
176.0 / 0.223 / 3.61176.0 / 0.223 / 3.611uniform 76.0 / 3.6110.483
275.8 / 0.179 / 2.70574.7 / 0.190 / 2.700logavg 76.2 / 3.0980.825
376.0 / 0.182 / 2.52476.1 / 0.181 / 2.521logavg 76.8 / 3.0431.166
476.3 / 0.170 / 2.43776.0 / 0.173 / 2.436logavg 76.8 / 2.9721.508
576.8 / 0.168 / 2.42677.1 / 0.174 / 2.430exp10 77.1 / 2.4301.850
676.8 / 0.166 / 2.34576.6 / 0.163 / 2.346logavg 77.1 / 2.9312.191
777.2 / 0.164 / 2.31977.0 / 0.162 / 2.322uniform 77.2 / 2.3192.533
876.6 / 0.162 / 2.26276.7 / 0.161 / 2.264exp20 76.8 / 2.2622.874
977.0 / 0.167 / 2.25777.1 / 0.163 / 2.263exp10 77.1 / 2.2633.216
1076.2 / 0.165 / 2.20676.2 / 0.166 / 2.210logavg 76.6 / 2.8873.557
1176.6 / 0.161 / 2.20276.7 / 0.162 / 2.207exp20 76.7 / 2.2033.899
1277.0 / 0.157 / 2.18676.8 / 0.159 / 2.194logavg 77.1 / 2.8784.240
1377.5 / 0.157 / 2.17377.0 / 0.158 / 2.181exp20 77.6 / 2.1754.582
1477.6 / 0.157 / 2.15577.0 / 0.159 / 2.163uniform 77.6 / 2.1554.924
1577.3 / 0.153 / 2.14176.6 / 0.163 / 2.148uniform 77.3 / 2.1415.265
1677.0 / 0.157 / 2.12776.7 / 0.164 / 2.136uniform 77.0 / 2.1275.607
1776.6 / 0.161 / 2.12076.6 / 0.165 / 2.133logavg 77.0 / 2.8825.948
1876.8 / 0.158 / 2.10676.8 / 0.163 / 2.121uniform 76.8 / 2.1066.290
1977.0 / 0.157 / 2.09277.0 / 0.163 / 2.108exp20 77.1 / 2.0966.631
2077.1 / 0.156 / 2.06476.6 / 0.164 / 2.074logavg 77.2 / 2.8806.972

Table 11. Qwen3-4B, Banking20, full 799, every horizon.

TTUniform Acc / ECE / NLLExp. τw=10\tau_w = 10 Acc / ECE / NLLBest at TTs/dec
180.4 / 0.181 / 2.90880.4 / 0.181 / 2.908uniform 80.4 / 2.9080.821
281.4 / 0.149 / 2.27681.4 / 0.149 / 2.273logavg 81.6 / 2.5881.413
381.2 / 0.146 / 2.17081.1 / 0.147 / 2.167logavg 81.7 / 2.5792.005
481.6 / 0.137 / 2.05681.9 / 0.135 / 2.051exp10 81.9 / 2.0512.598
582.0 / 0.135 / 2.01481.6 / 0.136 / 2.012uniform 82.0 / 2.0143.193
682.0 / 0.134 / 1.93781.5 / 0.135 / 1.934uniform 82.0 / 1.9373.786
781.6 / 0.133 / 1.91981.2 / 0.137 / 1.919logavg 82.0 / 2.4754.377
881.6 / 0.135 / 1.85781.4 / 0.135 / 1.858exp20 81.7 / 1.8574.967
981.6 / 0.132 / 1.82681.1 / 0.136 / 1.829logavg 81.9 / 2.4345.556
1081.4 / 0.132 / 1.79580.9 / 0.137 / 1.796logavg 81.5 / 2.4276.146
1181.5 / 0.131 / 1.80281.1 / 0.138 / 1.810uniform 81.5 / 1.8026.736
1281.5 / 0.135 / 1.78381.5 / 0.133 / 1.791uniform 81.5 / 1.7837.326
1381.7 / 0.133 / 1.77780.9 / 0.139 / 1.787uniform 81.7 / 1.7777.916
1481.4 / 0.132 / 1.75481.0 / 0.137 / 1.763logavg 81.5 / 2.4298.506
1581.5 / 0.131 / 1.75181.0 / 0.137 / 1.761uniform 81.5 / 1.7519.096
1681.5 / 0.133 / 1.74481.2 / 0.134 / 1.758uniform 81.5 / 1.7449.685
1781.9 / 0.132 / 1.73081.5 / 0.135 / 1.748uniform 81.9 / 1.73010.275
1882.0 / 0.133 / 1.69781.4 / 0.133 / 1.716uniform 82.0 / 1.69710.865
1981.7 / 0.130 / 1.69681.2 / 0.137 / 1.718uniform 81.7 / 1.69611.455
2081.7 / 0.135 / 1.68581.6 / 0.135 / 1.707logavg 81.9 / 2.42512.045

Table 12. Qwen3-8B, Banking20, full 799, every horizon.

TTUniform Acc / ECE / NLLExp. τw=10\tau_w = 10 Acc / ECE / NLLBest at TTs/dec
165.0 / 0.317 / 4.79565.0 / 0.317 / 4.795uniform 65.0 / 4.7950.445
264.3 / 0.257 / 3.63562.3 / 0.277 / 3.632logavg 65.3 / 4.1080.697
362.3 / 0.277 / 3.41862.3 / 0.278 / 3.415logavg 64.0 / 4.0670.949
462.7 / 0.268 / 3.31962.3 / 0.272 / 3.319logavg 63.3 / 4.0321.202
563.0 / 0.267 / 3.25463.0 / 0.269 / 3.258uniform 63.0 / 3.2541.454
662.7 / 0.265 / 3.19862.3 / 0.269 / 3.205avg_last5 63.3 / 3.2941.706
763.0 / 0.263 / 3.18363.0 / 0.265 / 3.196exp20 63.3 / 3.1891.958
862.3 / 0.267 / 3.14761.7 / 0.276 / 3.160uniform 62.3 / 3.1472.211
962.3 / 0.266 / 3.10462.0 / 0.270 / 3.119logavg 63.0 / 4.0732.463
1061.3 / 0.278 / 3.08361.3 / 0.283 / 3.098logavg 62.7 / 4.0832.715
1162.0 / 0.273 / 3.06462.0 / 0.278 / 3.077logavg 62.3 / 4.0882.968
1261.0 / 0.283 / 3.04961.0 / 0.288 / 3.065avg_last5 61.7 / 3.4543.220
1361.7 / 0.276 / 3.04362.0 / 0.277 / 3.065logavg 62.3 / 4.1053.473
1461.3 / 0.279 / 3.03861.0 / 0.288 / 3.065avg_last3 61.7 / 3.8433.725
1561.7 / 0.277 / 3.04562.0 / 0.280 / 3.079exp10 62.0 / 3.0793.978
1661.3 / 0.281 / 3.02561.7 / 0.284 / 3.057avg_last5 62.3 / 3.6594.230
1762.0 / 0.274 / 3.03262.7 / 0.275 / 3.074exp10 62.7 / 3.0744.482
1862.0 / 0.275 / 3.01562.7 / 0.275 / 3.057avg_last3 63.3 / 3.7784.734
1961.7 / 0.278 / 2.99263.0 / 0.272 / 3.033exp10 63.0 / 3.0334.987
2062.0 / 0.275 / 2.98362.3 / 0.279 / 3.021avg_last3 63.0 / 3.7305.239

Table 13. Qwen3-1.7B, Newsgroups, 300, every horizon.

TTUniform Acc / ECE / NLLExp. τw=10\tau_w = 10 Acc / ECE / NLLBest at TTs/dec
165.7 / 0.297 / 4.54065.7 / 0.297 / 4.540uniform 65.7 / 4.5401.003
268.0 / 0.233 / 3.43469.0 / 0.224 / 3.431exp10 69.0 / 3.4311.571
367.0 / 0.247 / 3.16267.0 / 0.249 / 3.158logavg 68.0 / 3.8542.138
468.7 / 0.222 / 3.06668.7 / 0.222 / 3.064exp10 68.7 / 3.0642.706
567.0 / 0.242 / 2.97466.7 / 0.246 / 2.973logavg 67.7 / 3.8493.273
667.3 / 0.232 / 2.89067.3 / 0.232 / 2.891avg_last5 68.0 / 2.9203.840
766.7 / 0.241 / 2.86866.3 / 0.245 / 2.873logavg 68.0 / 3.8094.408
867.0 / 0.238 / 2.82967.0 / 0.240 / 2.842avg_last5 67.3 / 3.0444.975
966.3 / 0.247 / 2.83366.3 / 0.250 / 2.852logavg 66.7 / 3.8355.543
1066.3 / 0.245 / 2.82366.7 / 0.245 / 2.841exp20 66.7 / 2.8306.110
1166.3 / 0.248 / 2.81066.7 / 0.248 / 2.827logavg 67.0 / 3.8346.677
1267.0 / 0.241 / 2.73067.3 / 0.241 / 2.743exp20 67.3 / 2.7347.244
1366.7 / 0.245 / 2.72667.0 / 0.246 / 2.742last 67.7 / 4.0467.812
1467.3 / 0.238 / 2.71667.7 / 0.239 / 2.737avg_last5 68.0 / 2.9788.379
1567.3 / 0.238 / 2.70267.3 / 0.239 / 2.721avg_last5 68.0 / 3.2198.947
1667.3 / 0.239 / 2.70867.7 / 0.240 / 2.737avg_last5 68.3 / 3.0199.514
1767.3 / 0.240 / 2.70968.3 / 0.236 / 2.743exp10 68.3 / 2.74310.082
1867.3 / 0.239 / 2.71168.0 / 0.237 / 2.748avg_last5 68.7 / 3.08610.650
1967.3 / 0.239 / 2.70968.3 / 0.234 / 2.746last 69.7 / 4.13911.217
2067.3 / 0.238 / 2.70268.0 / 0.235 / 2.741avg_last3 68.3 / 3.30011.785

Table 14. Qwen3-4B, Newsgroups, 300, every horizon.

TTUniform Acc / ECE / NLLExp. τw=10\tau_w = 10 Acc / ECE / NLLBest at TTs/dec
164.7 / 0.318 / 4.55364.7 / 0.318 / 4.553uniform 64.7 / 4.5531.617
265.0 / 0.261 / 3.62664.0 / 0.271 / 3.623logavg 65.3 / 4.1502.534
365.0 / 0.270 / 3.55765.0 / 0.271 / 3.561logavg 65.3 / 4.2423.450
465.0 / 0.262 / 3.32964.7 / 0.266 / 3.328logavg 65.3 / 4.1234.367
565.7 / 0.254 / 3.28665.7 / 0.255 / 3.287avg_last3 66.3 / 3.5095.284
665.7 / 0.250 / 3.25066.0 / 0.247 / 3.254exp10 66.0 / 3.2546.201
766.0 / 0.252 / 3.19766.3 / 0.252 / 3.199last 66.7 / 4.3547.118
866.3 / 0.247 / 3.10466.0 / 0.253 / 3.103exp20 66.3 / 3.1028.035
966.7 / 0.246 / 3.10366.7 / 0.249 / 3.111uniform 66.7 / 3.1038.952
1066.3 / 0.251 / 3.08466.3 / 0.255 / 3.096uniform 66.3 / 3.0849.869
1166.3 / 0.253 / 3.04966.0 / 0.261 / 3.057uniform 66.3 / 3.04910.786
1266.3 / 0.252 / 3.02365.7 / 0.261 / 3.031uniform 66.3 / 3.02311.703
1366.3 / 0.252 / 3.02465.7 / 0.262 / 3.038uniform 66.3 / 3.02412.620
1466.0 / 0.255 / 3.01865.3 / 0.265 / 3.039avg_last3 66.7 / 3.60613.537
1565.7 / 0.259 / 3.01465.3 / 0.266 / 3.035last 66.0 / 4.42214.455
1666.0 / 0.255 / 3.00565.7 / 0.260 / 3.033avg_last3 66.3 / 3.62915.372
1766.0 / 0.255 / 2.98166.3 / 0.254 / 3.005exp10 66.3 / 3.00516.289
1866.0 / 0.254 / 2.96565.3 / 0.262 / 2.990uniform 66.0 / 2.96517.206
1965.7 / 0.259 / 2.94365.7 / 0.262 / 2.967avg_last5 66.7 / 3.35518.123
2066.0 / 0.256 / 2.91965.7 / 0.263 / 2.938uniform 66.0 / 2.91919.041

Table 15. Qwen3-8B, Newsgroups, 300, every horizon.

References

  1. [1]Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.arxiv.org/abs/2404.14219
  2. [2]Diogo Almeida. 2026. Introducing System One models & Jev. TypeSafe AI blog, https://typesafe.ai/blog/introducing-system-one-models-and-jev. Published 15 September 2026.
  3. [3]Convai Innovations. 2026. Laya: Multilingual, non-autoregressive System 1 decision engine. https://github.com/NandhaKishorM/laya. GitHub repository.
  4. [4]Blessed Guda, Lawrence Francis, Gabrial Zencha Ashungafac, Carlee Joe-Wong, and Moise Busogi. 2025. Quantifying and mitigating selection bias in llms: A transferable LoRA fine-tuning and efficient majority voting approach. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 2019–2038.
  5. [5]Jianguo Huang, Yao Long Teng, Yanchen Deng, and Bo An. 2026. Consider, then choose: Order-invariant decisions in system one models. arXiv preprint arXiv:2610.18233.arxiv.org/abs/2610.18233
  6. [7]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.arxiv.org/abs/2310.06825
  7. [8]Theodore Lee. 2026. SemIf (formerly Open-Jev). https://github.com/TheoLeeCJ/SemIf-OpenJev. GitHub repository. Accessed 2026-10-29.
  8. [9]LocalLLaMA. 2026. Typed Decisions. https://huggingface.co/datasets/LocalLLaMA/typed-decisions. Hugging Face dataset, revision f7a2487. Accessed 2026-10-29.
  9. [10]Jared Palmer. 2026. Kev: Small Jev-like decision models you can train and run yourself. https://github.com/jaredpalmer/kev. Accessed 2026-10-29.
  10. [11]Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017.
  11. [12]Florian Standhartinger. 2026. JevBench. https://github.com/fstandhartinger/jevbench. Public split. Accessed 2026-10-29.
  12. [13]Yu Sun, Junhao Xu, Jiajia Shi, and Zijin Yang. 2026. Type-safe is not error-free: Typed decision models follow the option name, not the definition bound to it. arXiv preprint arXiv:2609.26758.arxiv.org/abs/2609.26758
  13. [14]TianyuCodings. 2026. NanoJev: A nano replica of Jev. https://github.com/TianyuCodings/NanoJev. GitHub repository. Accessed 2026-10-29.
  14. [15]Tic-top. 2026. llm2jev. https://github.com/tic-top/llm2jev. GitHub repository. Accessed 2026-10-29.
  15. [16]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems, volume 37, pages 95266–95290.arxiv.org/abs/2406.01574
  16. [17]Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. Unveiling selection biases: Exploring order and token sensitivity in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5598–5621, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2406.03009
  17. [18]Yinsong Xu. 2026. LLM2Jev: Turn LLMs into Jev-style decision models. https://github.com/Yinsongxu/LLM2Jev. GitHub repository. Accessed 2026-10-29.
  18. [19]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenggen Huang, Chengxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.arxiv.org/abs/2505.09388

Paper details

vlnc:W_j6u9k3zs
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents