betaFeedback

WorksComputer Science

UltraBERT-JEV: Encoder models for System One Applications

Abstract

We study encoder models that pick one of several options in a single forward pass, without generating text. This technical report compares recipes for fine-tuning UltraBERT-380M with one mask per option. The four recipes differ in how an option is scored and how the scores become probabilities; one follows the expert gate of DeepSeek-V3. Trained on the same data as Qwen3-0.6B with the LLM-as-Jev recipe (67.6% test accuracy), the encoder reaches 69.7% to 72.1%. Scoring an option by "yes" minus "no" instead of "yes" alone gives the larger gain, from 69.7% to 71.3% on the test set and from 57.8% to 59.6% on external benchmarks (56.1% for the decoder). Sigmoid normalisation adds 0.8 test points; a square-root softplus adds none. One request at a time on an NVIDIA A100, the encoder takes 29 ms per decision against 68 ms, but uses more peak memory (2.8 against 2.2 GiB). An auxiliary head, trained on human-marked evidence and on labels selected with a reward adapted from SelfCite, predicts in the same pass the sentences annotated as support for the answer.

Introduction

Software often has to choose among fixed options: route a support ticket, classify an intent, or check a claim against evidence. A chat model can do this, but its answer must be generated and parsed. Jev-style decision models [2] instead return a probability distribution over the options, so a new task needs only a prompt and a list of options. LLM-as-Jev [27] shows that an unmodified decoder language model works as such a model: it reads numbered options and answers with a number. Fine-tuning it with KL anchors helps most for small models and for tasks with many options.

A decision needs almost no generation, while decoders are built for it. Multi-query and grouped-query attention [1, 38] shrink the KV cache for decoding, and the parameters they save can go into larger MLP layers. Du et al. Du et al. [16] show that these layers cause memory spikes in workloads that only read their input. An encoder keeps no KV cache, which is one reason to try it; at our scale, § 4.5 finds no peak-memory advantage.

An encoder also reads the input in both directions. A classifier head would fix the label set, but a masked language model can score any option by whether it fills a mask next to the option with “yes” or “no”. Recent encoders such as ModernBERT [42] and UltraBERT [32] read the long inputs that decisions over documents need.

Reading in both directions also helps with evidence. A decoder serving many questions about one document usually reads the document first, so that its KV cache can be reused, as with prefix caching in vLLM [25] and RadixAttention in SGLang [50]. The cached document states then do not depend on the question. Our encoder reads the question and the document together.

Our contributions are:

  • An encoder decision model. UltraBERT-380M with one mask per option, scoring every option in one forward pass with no new head, for any number of options. It is fine-tuned on the LLM-as-Jev training mixture with Nimble added (§ 3.1).

  • Training recipes. Four ways to score and normalise the mask predictions, one following the DeepSeek-V3

expert gate [14]. Scoring by “yes” minus “no” gives the larger gain; sigmoid normalisation adds 0.8 test points, mostly on MuSiQue.

  • A comparison with decoders under matched data. Against Qwen3-0.6B [34] fully fine-tuned on the same examples, every encoder recipe is more accurate (69.7% to 72.1% against 67.6% on a held-out test split) and faster (29 against 68 ms per decision), but not lighter in peak memory. Two reproduced LoRA models [21] serve as references.

  • Predicting supporting evidence. An auxiliary evidence head in the same forward pass, trained on human-marked labels and labels selected with a reward adapted from SelfCite [7]. On the softmax encoder it ranks annotated sentences higher than ContextCite and AT2 [10, 11].

Method

Input format

Figure 1 shows one row in both formats. The options are shuffled each time a row is used, so the position of the correct option carries no information.

Comparison of decoder and encoder approaches for multiple-choice scoring

Figure 1. One row, two ways. The decoder reads the numbered options and writes the number of its answer; the loss is taken over the tokens of that number. The encoder gets a mask in front of each option and scores each option directly. Two of the four recipes are shown: the plain softmax over the options, which scores an option by the “yes” logit alone, and the sigmoid-normalised loss, which scores it by “yes” minus “no”. Each option’s score goes through a sigmoid, and the sum in the denominator keeps the options competing. The other two normalise “yes” minus “no” with a softmax and with a square-root softplus.

Decoder. The prompt follows LLM-as-Jev: the state, the question, the options numbered [1] to [KK], and an instruction to answer only with the number. The reply is filled in up to “Best answer: [”, with thinking turned off, so each option kk is the token sequence “kk]” and the options form a trie. The decoder chooses the option whose whole sequence has the highest log-probability.

Encoder. The state and question are the same, the instruction asks to mark the correct option with “yes” and the others with “no”, and each option is listed as “[kk] [M] description”, with a mask token in front. There is no chat template. One forward pass gives the logits at every mask, and option kk is scored from the logits of “yes” and “no”, each with a leading space, at its mask.

Loss functions

Decoder: trie loss with KL anchors. The tree loss is the negative log-probability of the gold option’s path through the trie, each step normalised over the tokens legal there,

Ltree=−∑tlog⁡exp⁡zt(gt)∑v∈Vtexp⁡zt(v)(1)\mathcal{L}_{\mathrm{tree}}=-\sum_t\log\frac{\exp z_t(g_t)}{\sum_{v\in\mathcal{V}_t}\exp z_t(v)} \tag*{(1)}

where ztz_t are the logits at step tt, gtg_t the gold token and Vt\mathcal{V}_t the legal tokens. Three anchor terms keep the model close to its starting version p0p_0 [27]: Lmass\mathcal{L}_{\mathrm{mass}} keeps the total probability on the legal tokens, Lout\mathcal{L}_{\mathrm{out}} keeps how the remaining probability is spread over the rest of the vocabulary, and Lpos\mathcal{L}_{\mathrm{pos}} keeps the next-token distribution at the start of the reply and after the closing bracket. They are KL divergences to p0p_0, computed over its 64 most likely tokens and one bucket for the rest. The loss is L=Ltree+λ(Lmass+Lout+Lpos)\mathcal{L}=\mathcal{L}_{\mathrm{tree}}+\lambda(\mathcal{L}_{\mathrm{mass}}+\mathcal{L}_{\mathrm{out}}+\mathcal{L}_{\mathrm{pos}}) with λ=1\lambda=1.

Encoder recipes. Let ℓk(v)\ell_k(v) be the logit of token vv at mask kk, and let gg be the gold option. A score sks_k says how strongly the model accepts option kk; the decision is the option with the highest score. Training raises the gold option’s score relative to the other options, but no mask gets its own yes-or-no target. Thus, a chosen option can still prefer “no”: for example, scores of −3 and −8 select the first option. We train the following four recipes:

  • Yes-logit softmax. The score is the “yes” logit alone, and the options compete through one softmax:

sk=ℓk(yes),pk=esk∑j=1Kesj,Lsoftmax=−log⁡pg.(2)s_k=\ell_k(\mathrm{yes}),\qquad p_k=\frac{e^{s_k}}{\sum_{j=1}^{K}e^{s_j}},\qquad\mathcal{L}_{\mathrm{softmax}}=-\log p_g. \tag*{(2)}
  • Yes-minus-no softmax. The score measures the preference for “yes” over “no”, followed by the same softmax:

sk=ℓk(yes)−ℓk(no),pk=esk∑j=1Kesj,Lsoftmax=−log⁡pg.(3)s_k=\ell_k(\mathrm{yes})-\ell_k(\mathrm{no}),\qquad p_k=\frac{e^{s_k}}{\sum_{j=1}^{K}e^{s_j}},\qquad\mathcal{L}_{\mathrm{softmax}}=-\log p_g. \tag*{(3)}
  • Yes-minus-no sigmoid normalisation. Each score first goes through a sigmoid, and the resulting values are normalised by their sum:

sk=ℓk(yes)−ℓk(no),pk=σ(sk)∑j=1Kσ(sj),Lsig-norm=−log⁡pg.(4)s_k = \ell_k(\mathit{yes}) - \ell_k(\mathit{no}), \qquad p_k = \frac{\sigma(s_k)}{\sum_{j=1}^{K} \sigma(s_j)}, \qquad\mathcal{L}_{\mathrm{sig\text{-}norm}} = -\log p_g. \tag*{(4)}

This is the form of the DeepSeek-V3 expert gate [14], with options in place of experts, normalised over all KK options and with no selection or balancing bias. Each σ(sk)\sigma(s_k) lies between 0 and 1, but only the ratios enter pkp_k, so the options still compete; the decision is still the highest score. The gradient with respect to sks_k is

(1−σ(sk))(pk−1[k=g]).(5)(1-\sigma(s_k))(p_k-\mathbf{1}[k=g]). \tag*{(5)}

When an option’s sigmoid approaches 1, the direct gradient of the loss with respect to its score becomes small. We exclude the sigmoid on the “yes” logit alone: vocabulary logits are around 200 (§ 5.3), so every sigmoid would saturate near 1 and the gradient would be negligible.

  • Yes-minus-no square-root softplus. As an unbounded alternative to the sigmoid, we use h(s)=log⁡(1+es)h(s)=\sqrt{\log(1+e^s)}:

sk=ℓk(yes)−ℓk(no),pk=h(sk)∑j=1Kh(sj),Lsqrt-sp=−log⁡pg.(6)s_k = \ell_k(\mathit{yes}) - \ell_k(\mathit{no}), \qquad p_k = \frac{h(s_k)}{\sum_{j=1}^{K} h(s_j)}, \qquad\mathcal{L}_{\mathrm{sqrt\text{-}sp}} = -\log p_g. \tag*{(6)}

For a clearly rejected option (s≪0s \ll0), h(s)≈es/2h(s) \approx e^{s/2}, so the options behave as in a softmax at twice the temperature. For a clearly accepted option (s≫0s \gg0), h(s)≈sh(s) \approx\sqrt{s} keeps growing, whereas the sigmoid stops at 1. The direct gradient with respect to its score therefore shrinks slowly, where the sigmoid’s nearly vanishes.

Experimental setup

Training data

We train on the Reason+Long mixture of LLM-as-Jev [27], with Nimble added: 13 datasets covering intent routing (CLINC150 [26], MASSIVE [17], Bitext [6]), short reasoning (ReClor [45], LogiQA 2.0 [28], CosmosQA [22], PIQA [5], ANLI [29], ProofWriter [39]), long input (ContractNLI [23], MuSiQue [40], QuALITY [30]) and Nimble [4]. Each row is a state (the passages), a question and a list of options. For example, a MuSiQue row gives about twenty paragraphs and a multi-hop question, and asks whether the paragraphs contain enough information to answer it (“Yes” or “No”). We keep 1,000 training rows per dataset, 13,000 in total, after these filters:

  1. Duplicates. Rows with the same input are collapsed, and removed if their answers differ.

  2. Leakage. A training row is removed if its input or source document appears in the validation split.

  3. Benchmark overlap. A training row is removed if it shares a 13-word n-gram, or a sentence of six or more words, with an evaluation benchmark.

  4. Length. Prompts over 10,240 tokens (Qwen3.5-4B tokenizer) are dropped, never truncated.

The held-out rows come from the validation split, with the same duplicate and length filters, up to 500 per dataset and in whole source documents (6,315 rows). A random 100 rows per dataset form the dev set (1,300 rows), used to choose learning rates and the evidence threshold; the other 5,015 rows form the test set, excluded from model selection. The split is by row, so a dev and a test row can share a source document, such as two statements about one contract.

Models

UltraBERT-380M [32] is fully fine-tuned with each of the four recipes of § 2.2. Qwen3-0.6B [34] is fully fine-tuned with the decoder loss, with a frozen copy as p0p_0; it is the matched comparison. As references, we also report two LoRA models [21], Qwen3-0.6B and Qwen3-4B, reproduced from LLM-as-Jev on the same training set and loss.

Implementation details

Every model is trained for one epoch with a batch of 32 rows. We sweep the learning rate on dev, one run per value: {0.8,1.5,3,5}×10−5\{0.8, 1.5, 3, 5\} \times10^{-5} for the encoder with the softmax over “yes”, {0.8,1.5,2,3}×10−5\{0.8, 1.5, 2, 3\} \times10^{-5} with the sigmoid-normalised loss, and {0.5,1,2}×10−5\{0.5, 1, 2\} \times10^{-5} for the decoder. Both encoder sweeps choose 1.5×10−51.5 \times10^{-5}, which we reuse for the other two recipes; for Qwen the two best rates differ by 0.1 points, and we use 1×10−51 \times10^{-5}. At the chosen rate we train each model several times, changing only the row order and option shuffling, and report the average.

The metric is accuracy, and ALL is its mean over the 13 datasets. On the test set we average each row’s correctness over a model’s runs before computing ALL. For a difference between two models we give a paired 95% bootstrap interval: 2,000 resamples of the test rows within each dataset, with both models scored on the same rows. It covers test-row sampling only, not retraining, and treats rows from one document as independent, so it can be too narrow.

The encoder is scored with fp32 weights under bf16 autocast, since bf16 weights change its long-document results (§ 5.3); the Qwen checkpoints use bf16 weights.

External benchmarks and cost

To see how training changes behaviour outside the mixture, we score six benchmarks as decision rows, never used for any choice: BoolQ [8], MMLU [20], MMLU-Pro [41], ARC-Challenge [9], Winogrande [36] and SciQ [43]. We draw fixed random samples of 500, 1200, 800, 600, 500 and 400 rows, the sizes of LLM-as-Jev, whose samples are not released. External is the mean accuracy over the six.

Time per decision is the mean latency of one request over the dev rows, scored one at a time on one A100, as when a pipeline sends each item as it arrives. It covers the forward pass and option scoring but not tokenisation, after 20 warm-up rows, with the GPU synchronised around every call. The dev rows average 1,100 to 1,200 tokens. The encoder is timed on the softmax checkpoint and the decoders on the reproduced LoRA checkpoints with merged weights; the other checkpoints of each model share their architecture. Section A gives batched throughput.

Results

Encoder against decoder

Table 1 gives the main comparison. Every encoder recipe is above fully fine-tuned Qwen3-0.6B, both on the test set (69.7% to 72.1% against 67.6%) and on the external benchmarks (57.8% to 59.6% against 56.1%), with fewer parameters (380M against 0.6B). These results concern these two models under our setup; the reproduced Qwen3-4B LoRA model, at 77.3%, remains more accurate.

ModelParamsDev ALLTest ALLExternal
UltraBERT-380M, softmax, yes380M71.269.757.8
UltraBERT-380M, softmax, yes − no380M71.771.359.6
UltraBERT-380M, sqrt-softplus, yes − no380M71.671.259.3
UltraBERT-380M, sigmoid-norm, yes − no380M72.772.159.5
Qwen3-0.6B, full FT0.6B69.067.656.1
Qwen3-0.6B, LoRA (reproduced)0.6B64.563.056.9
Qwen3-4B, LoRA (reproduced)4B79.377.373.5

Table 1. Matched comparison: average accuracy (%) over training runs. Test ALL is on the test set; Dev ALL and External are on the dev set and the six external benchmarks. Learning rates were chosen on dev. Bold: best below 1B parameters.

Full fine-tuning against LoRA

The reproduced Qwen3-0.6B LoRA model reaches 63.0% on the test set, 4.6 points below full fine-tuning, so full fine-tuning is the stronger baseline. Its external accuracy (56.9%) is not below that of the full fine-tune (56.1%).

Recipes

Changing the score from “yes” to “yes” minus “no”, both with the softmax, adds 1.6 test points and raises external accuracy from 57.8% to 59.6%. Sigmoid normalisation adds a further 0.8 test points (95% interval [+0.2, +1.3], § 3.3) and nothing on the external benchmarks.

This gain is concentrated in MuSiQue (69.8% against 63.0%), which contributes about 0.5 of the 0.8 points. On the other twelve datasets it averages 0.3 points and goes both ways, from +3.0 on Nimble to −2.4 on PIQA (Table 7).

Table 7. Test accuracy (%) per dataset, averaged over training runs. UB is UltraBERT-380M: UB yes is the softmax over each option’s “yes” logit; UB yes−no is the softmax over “yes” minus “no”; UB sqrt-sp and UB sig-norm normalise “yes” minus “no” with the square-root softplus and the sigmoid, each divided by the sum over options (§ 2.2). Bold: best below 1B parameters.

DatasetRowsUB yesUB yes−noUB sqrt-spUB sig-normQwen fullQwen LoRAQwen-4B LoRA
CLINC15040091.891.591.791.886.379.889.8
MASSIVE40083.784.583.884.976.968.883.5
Bitext40099.399.399.399.396.094.897.2
ReClor40058.960.761.859.957.751.272.2
LogiQA 2.040052.954.855.454.857.056.073.5
CosmosQA40074.874.275.175.259.955.076.0
PIQA40074.877.575.975.170.268.883.8
ANLI40073.172.673.973.663.761.074.0
ProofWriter40060.761.561.461.663.363.070.5
ContractNLI39373.877.474.778.676.165.679.1
MuSiQue40058.963.062.269.862.954.871.5
QuALITY39856.962.160.861.351.550.068.8
Nimble22447.047.849.150.757.350.964.3

Table 7.

The square-root softplus, which has no upper bound, does not reproduce the gain (71.2% against 71.3% for the softmax). The comparison does not show why: the two functions also differ in their gradients, and for rejected options the square-root softplus acts as a softmax at twice the temperature.

Sensitivity to option order and removal

On the dev rows we score each row in four random option orders and once with one wrong option removed, and count how often the answer changes (Table 2). With the raw scores fixed, no normalisation could change the winner, so a change means the encoder recomputed its scores after reading the new list: the test measures the trained model. The two softmax recipes isolate the score; the softmax and the sigmoid over “yes” minus “no” isolate the normalisation used in training.

Softmax, “yes”Softmax, “yes” − “no”Sqrt-softplus, “yes” − “no”Sigmoid-norm, “yes” − “no”
Dev accuracy (%)71.471.571.172.9
External accuracy (%)57.859.659.359.5
Answer changes, options reordered (%)15.614.616.613.7
Answer changes, one wrong option removed (%)5.94.85.44.9
Largest gradient norm in training842139104190
Label frequencies vs. gold (TV)0.1670.2130.1540.139
Position frequencies vs. gold (TV)0.0500.0450.0490.041
Calibration error, top option0.0380.0270.0320.023
Mean ∣sk∣|s_k|60.53.012.37.4

Table 2. Sensitivity to option order and removal, on the dev set. Each row is scored in four random option orders; an answer change under reordering means the chosen option is not the same in all four. One wrong option, never the chosen one, is removed at random from rows with at least three options, and the answer is compared over the remaining options. Dev accuracy is on the first random order, so it differs slightly from Table 1, which keeps the original order. TV: total variation between the predicted and the gold frequencies, over the three datasets with a fixed label set (label) and over option positions (position). Calibration error: expected calibration error of the top option’s probability. ∣sk∣|s_k|: size of the raw option scores. Each metric is averaged over datasets, then over training runs. Bold: best in the row, higher for accuracy and lower for answer changes, TV and calibration error; the gradient norm, the largest over all training steps and so a single batch’s value, and ∣sk∣|s_k| are not ranked.

The score does most of the work. “Yes” minus “no” raises external accuracy from 57.8% to 59.6%, lowers the answer changes on removal from 5.9% to 4.8%, and lowers the largest gradient norm from 842 to 139. Under reordering, the score and the sigmoid each remove about one point of answer changes (15.6%, 14.6%, 13.7%).

On top of the score, the sigmoid raises dev accuracy from 71.5% to 72.9% and lowers the calibration error from 0.027 to 0.023. It also reduces label bias. On the three datasets whose options are the same labels in every row (ContractNLI, MuSiQue and ProofWriter), we compare how often the model predicts each label with how often that label is correct. The total variation (TV) between the two is the share of predictions that would have to change label for them to match, so 0 means no bias towards any label. It is 0.167 for the softmax over “yes”, 0.213 for the softmax over “yes” minus “no” and 0.139 for the sigmoid; for example, on MuSiQue the softmax models predict “Yes” far more often than it is correct (§ 5.2). TV measures bias, not accuracy: a model could predict each label exactly as often as it is correct and still assign it to the wrong rows. The gold option’s sigmoid averages 0.049; for small sigmoid values σ(s)≈es\sigma(s) \approx e^s, so the normalised sigmoid then approximates a softmax.

The square-root softplus has the lowest maximum gradient norm (104) but changes its answer most often under reordering (16.6%).

Time and memory

A decoder must read the whole trie: every option’s score includes the closing bracket and, from ten options on, the later digits. For a fair comparison we time the decoders with a faster scorer that reads the input once and then all trie nodes in one more forward pass on the shared KV cache, under a mask that lets each node see the input and its own earlier tokens. In fp32 its scores match those of the evaluation scorer within 10−410^{-4} (40 dev rows with ten or more options); in bf16, as timed, both choose the same option on 1,297 of the 1,300 dev rows. The evaluation scorer copies the cache for each node and takes 92 ms and up to 8.6 GiB on Qwen3-0.6B.

One row at a time, the encoder takes 29 ms per decision, against 68 ms for Qwen3-0.6B and 124 ms for Qwen3-4B (Table 3); reading the input alone takes the decoders 36 and 85 ms. Peak memory tells a different story. Qwen3-0.6B peaks at 2.2 GiB, against 2.8 GiB for the encoder with the fp32 weights it needs for its accuracy, or 1.9 GiB with bf16 weights, which lose accuracy (§ 5.3). At this scale the encoder is faster but has no peak-memory advantage; we measured peaks, not memory over time.

Modelms per decisionWeightsPeak, rows >4,096 tok.Peak, max
UltraBERT-380M, fp32 weights291.62.62.8
UltraBERT-380M, bf16 weights (lower accuracy)260.81.51.9
Qwen3-0.6B681.12.02.2
Qwen3-4B1247.68.99.3

Table 3. Time and peak GPU memory per decision, scoring the dev rows one at a time on one A100. The decoders use bf16 weights and read the input, then the whole trie of option numbers in one more pass on the shared KV cache. Memory is what PyTorch allocates, in GiB, weights included; rows >4,096 tok.: mean peak over rows longer than 4,096 tokens; max: the largest peak over all rows.

Analysis

The numbers in this section are from the dev set, with one training run per setting, unless noted.

One change at a time

Table 4 changes one part of the recipe at a time. On these single runs, “yes” minus “no” adds 0.4 dev points and 2.0 external points, and the sigmoid a further 1.3 dev points, most of them on MuSiQue (64% to 76%). Single runs on 100 rows per dataset are noisy: here Nimble falls from 55% to 43% with the sigmoid, while on the test set the sigmoid-normalised model is 3.7 points above the softmax over “yes” on Nimble. The words “correct” and “incorrect” in place of “yes” and “no” did not help (70.6% against 71.1% for the single word, 70.1% against 71.5% for the difference).

VariantDev ALLExternalMuSiQueContractNLINimble
Softmax, “yes”71.157.157.075.055.0
Softmax, “yes” − “no”71.559.164.078.048.0
Sigmoid-norm, “yes” − “no”72.859.476.081.043.0
Softmax, “correct”70.656.848.077.052.0
Softmax, “correct” − “incorrect”70.158.054.076.046.0

Table 4. Encoder variants on the dev rows, one training run each (accuracy, %). The first row is the softmax with the “yes” logit, the setting used for the softmax results elsewhere. Bold: best in each column.

Predicted labels

The recipes predict different labels. On MuSiQue, “Yes” is correct for 48% of the dev rows; the sigmoid-normalised models predict it for 47 to 60%, and the two softmax recipes for 64% and 69%. On ProofWriter, “Unknown” is correct for a third of the rows; the softmax never predicts it and the sigmoid-normalised models predict it for 2 to 12%, so it stays largely unlearned.

Numerical precision

The encoder’s masked-token logits are around 200, where bf16 steps by 1 or 2 and options would tie or swap, so we compute the masked-token head, the scores and the normalisation in fp32. Even then, bf16 weights changed MuSiQue accuracy by up to 8 points, presumably through rounding elsewhere in the network; we did not trace where. We therefore keep fp32 weights under bf16 autocast. Re-scoring a Qwen checkpoint in another precision moved its dev accuracy by at most 0.8 points.

Predicting supporting evidence

We add a small auxiliary head on the frozen encoder that highlights, in the same forward pass, the document sentences annotated as supporting the chosen option. For each document token it reads the token’s hidden state and that of the chosen option’s mask, and outputs an evidence score between 0 and 1; a sentence takes the highest score among its tokens. The score is not a calibrated probability, because positive tokens are up-weighted in training, and the head is not evaluated as an explanation of how the encoder decides.

The head is trained on human-marked evidence where it exists, in ContractNLI, MuSiQue, Nimble and ProofWriter. For the other rows, a labelling model proposes sets of sentences for the gold answer, and we keep the set with the best reward adapted from SelfCite [7] (Figure 2, Section B).

Evidence-label generation flowchart

Figure 2. How evidence labels are generated for rows without human evidence. The labelling model, Qwen3.6-27B, proposes up to ten sets of sentences as evidence for the gold answer. The same model then answers the decision prompt three ways for each set SS: with the full document, without SS, and with only SS: a good set is enough to keep the answer (sufficiency) and its removal loses the answer (necessity). The set with the highest reward becomes the label, which trains the evidence head. Unlike SelfCite, the answer scored is the gold answer and not the model’s own response.

Comparison. We compare with the encoder’s average attention, AT2 [11] and ContextCite [10] on the 889 test rows with gold evidence; the word-highlighting threshold is chosen on 267 dev rows. The head is trained on evidence labels, while AT2 and ContextCite are fitted to the model’s response to ablations.

We report sentence average precision (AP) against the gold sentences, how often the top sentence is gold, and the time per row. For the baselines we also report LDS [10], the rank correlation between predicted and actual changes in the answer’s probability when sentences are removed; we did not evaluate it for the heads.

AT2 and ContextCite ran on the softmax encoder, so the like-for-like comparison is the head on that encoder; the head on the sigmoid-normalised encoder also changes the encoder. The heads see the option the encoder chose, and a wrong choice is scored against the gold option’s evidence, so Table 5 measures the decision and the evidence together. The times are on these rows, so they cannot be subtracted from the decision time of § 4.5.

MethodAllCNLIMuSiQueNimblePWTop-1 (%)LDSms per row
Average attention0.250.070.200.340.38140.0969
AT20.330.300.280.300.42230.1571
ContextCite0.410.520.330.330.44440.641,860
Evidence head, softmax encoder0.550.280.320.960.6754–68
Evidence head, sigmoid-norm encoder0.680.550.520.960.6874–72

Table 5. Sentence evidence on the test rows with gold evidence. Sentence AP: average precision against the gold sentences, over all rows and per dataset (CNLI: ContractNLI, PW: ProofWriter). Top-1: share of rows where the top sentence is gold. LDS: how closely the scores follow the model’s response to removed sentences; not evaluated for the heads. The first three methods and the first head use the encoder trained with the softmax. Each head is one training run. Bold: best in each column (lowest for time).

Results. On the softmax encoder, the head has a higher sentence AP than ContextCite and AT2 (0.55 against 0.41 and 0.33), in 68 ms per row against 1,860 ms for ContextCite. The lead comes from Nimble and ProofWriter; ContextCite is better on ContractNLI and level on MuSiQue. The head on the sigmoid-normalised encoder reaches 0.68 and is level with or above ContextCite on those two datasets.

The words this head marks have a precision of 0.33 and a recall of 0.35, a rough guide rather than an exact span. Most of the accuracy comes from the human-marked labels: on the softmax encoder, a head trained on them alone reaches 0.53, and the generated labels raise it to 0.55 (Section B). Of the methods with an LDS, ContextCite follows the model most closely (0.64).

Related work

Jev-style decision models. Most community decision models compared in LLM-as-Jev [27] read the logits of an option letter or label from a decoder, frozen or tuned with LoRA; LLM-as-Jev studies this readout and when fine-tuning helps. AnyJev reads a typed decision from one prefill and corrects the label prior and option-order effects without training [46]. The Contrastive Language Model [24] embeds the state and each candidate separately with a frozen Qwen3-8B and scores them by cosine similarity, so candidates can be cached but never attend to the context. All of these start from a decoder; we start from an encoder that reads the context, the question and all options together.

Encoders that score options. Prompt-based classifiers use the masked-token head of an encoder with a verbalizer for a fixed set of labels [18, 37]. Two encoder decision models are listed on public Jev leaderboards [3]: Laya [12] and Von [31], both built on ModernBERT-large with one mask per option. Both add new layers, described in model cards: Laya trains its head with reinforcement learning on proper scoring rules, and Von uses a listwise softmax loss with a Brier term. We reuse the pretrained masked-token head, compare four recipes against decoders trained on the same data, and report a held-out test split.

Option letters, position and label bias. Reading the letter of the chosen option requires a model to bind the letter to the option text [35]. Models favour certain option labels, and their accuracy changes when the options are reordered [33, 49]. Corrections are usually applied after training, by calibrating with a content-free input [48] or by dividing out a label prior and averaging over rotations of the options [46]. We shuffle the options during training and measure how the recipes differ in the labels they predict.

Encoders and decoders. Controlled studies pretrain encoders and decoders with the same recipe, and find that encoders do well on classification and retrieval [44] and that masked language modelling suits tasks that need text representations [19]. A much larger decoder can still beat an encoder on classification [47]. Decoder designs such as multi-query and grouped-query attention target long generation [1, 38], and Du et al. [16] study what they cost when nothing is generated (§ 1). Modern encoders keep long inputs and fast inference [32, 42].

Attribution and evidence. ContextCite learns a surrogate model from many ablations of the context [10]. AT2 learns to weight attention heads, and needs a decoder pass [11]. SelfCite fine-tunes a model to cite, using the change in the answer probability when the cited text is removed or kept alone as the reward [7]. Rationale benchmarks such as ERASER compare selected evidence with human rationales [15]. Our evidence head is trained on human-marked evidence and SelfCite-style labels, and gives the evidence in the decision’s forward pass.

Conclusion

We fine-tuned an encoder, UltraBERT-380M, as a decision model with one mask per option, and compared four training recipes. On the same training examples, every recipe is more accurate than Qwen3-0.6B trained with LLM-as-Jev (69.7% to 72.1% against 67.6% on the test set). The encoder is faster (29 against 68 ms per decision) but uses more peak memory (2.8 against 2.2 GiB).

Among the recipes, scoring by “yes” minus “no” gives the larger test gain and all of the external gain. Sigmoid normalisation adds 0.8 test points, mostly on MuSiQue, and brings label frequencies closer to the gold ones; the square-root softplus does not, and we cannot say which property of the sigmoid is responsible. An auxiliary head on the frozen encoder predicts annotated evidence in the same pass, and on the softmax encoder it ranks the annotated sentences higher than AT2 and ContextCite, except on ContractNLI.

The results have limits. They cover one encoder, one decoder family and one English training mixture, below one billion parameters; a 4B decoder is still more accurate except on the intent datasets. We did not evaluate whether the evidence head reflects what the model used, and we measured only peak memory.

Two directions follow: larger encoders, to see whether the advantage holds above one billion parameters, and training the decision and the evidence together, so that the evidence head can shape the encoder.

Implementation details

Table 6 lists the training settings. Both models train for one epoch, which is 406 steps at a batch of 32.

UltraBERT-380MQwen3-0.6B (full)
OptimiserAdamW, β=(0.9,0.98)\beta= (0.9, 0.98), ϵ=10−6\epsilon= 10^{-6}AdamW
Weight decay0.10.01
Schedule5% warmup, linear decay to 0100 warmup steps, linear decay to 10%
Gradient clipping1.01.0
Precisionfp32 weights, bf16 autocastfp32 weights, bf16 autocast
AttentionPyTorch SDPAPyTorch SDPA
Gradient checkpointingyesyes
Token limit32,76816,384

Table 6. Training settings. Learning rates are in § 3.3.

The encoder settings are those of the UltraBERT model card, and we change only the learning rate. No training row exceeds either token limit, because the data are cut at 10,240 tokens (§ 3.1).

Batched throughput. These numbers are specific to our implementations, not a comparison of the fastest paths. When the dev rows are scored in batches, the encoder processes 75.5 rows per second, against 16.4 for Qwen3-0.6B and 6.5 for Qwen3-4B. The encoder uses packed variable-length Flash Attention 2 [13] and the decoders use length-sorted padded batches with the cache-copying trie scorer, so part of this gap comes from the padding and the copies.

Evidence head

We train one head on each fine-tuned encoder (sigmoid-normalised and softmax, both at a learning rate of 1.5⋅10−51.5 \cdot10^{-5}); the softmax one matches the encoder used by AT2 and ContextCite.

Model. For document token tt and option kk, the head reads [ht;hk;ht⊙hk][h_t; h_k; h_t \odot h_k], the final hidden states of the token and of option kk’s mask. A two-layer perceptron (512 hidden units, GELU, dropout 0.1) maps it to one logit, whose sigmoid is the token’s evidence score for option kk.

Labels. A token is positive if it overlaps an evidence span. We use gold evidence where it exists (Nimble, ContractNLI, MuSiQue, ProofWriter) and generated spans elsewhere (Figure 2): ANLI, QuALITY, LogiQA 2.0, CosmosQA and the remaining rows of ContractNLI, MuSiQue and ProofWriter. There are 7,778 labelled training rows (7,390 for training, 388 for validation), with 5.2% positive tokens.

Generated spans. Qwen3.6-27B with thinking gets the numbered sentences, the question and the gold answer, and is asked for the smallest set of sentences that supports the answer and without which it is unsupported. We sample ten replies (temperature 1.2, top-pp 0.9), merge identical sets, and drop sets over 384 tokens unless they are one sentence.

Reward. The same model scores the decision prompt with each set SS removed and with only SS kept; q(gold∣⋅)q(\text{gold} \mid\cdot) is the softmax over the options’ summed token log-probabilities. The label is the set with the highest reward log⁡q(gold∣only S)−log⁡q(gold∣without S)\log q(\text{gold} \mid\text{only } S)-\log q(\text{gold} \mid\text{without } S), which is sufficiency plus necessity, with the full-context term cancelling. The best set is always kept, so a row whose candidates are all poor still gets a label, and nothing beyond the token cap makes the set the smallest sufficient one. Unlike SelfCite, we score the gold answer, so the reward is informative only where the labelling model believes it. We did not check the generated labels against human evidence directly; the last paragraph of this appendix is an indirect check.

Training. Token-level binary cross-entropy, with positives weighted by the negative-to-positive ratio (18.1), on the frozen encoder. The head is conditioned on the gold option in training and on the chosen option at test time. AdamW (learning rate 10−310^{-3}, weight decay 0.01), 16 rows per step, three epochs; we keep the epoch with the best validation token average precision.

Evaluation. All methods are scored on the test rows with gold evidence and at least two sentences. A word is highlighted when its evidence score reaches a threshold tuned for word F1 on the 267 dev rows with gold evidence: 0.79 for the sigmoid-normalised encoder and 0.74 for the softmax encoder. On the sigmoid-normalised encoder, the head’s token average precision is 0.46 (0.17 for a random ranking), and 0.63 of its ten highest tokens are gold. On the softmax encoder, adding the generated labels to the gold ones raises sentence average precision from 0.53 to 0.55 and token average precision from 0.38 to 0.39.

Additional results

Table 7 gives the test accuracy for each dataset.

Examples

Each example shows the sigmoid-normalised encoder’s decision and its evidence head’s token scores, on the first test row of the evidence study with at most 1,400 tokens in each dataset (MuSiQue has none). Tokens at or above the dev-tuned threshold of 0.79 are highlighted, darker for a higher score, and gold evidence is underlined. The encoder is correct on all three rows, which were picked by this rule, not by outcome.

ContractNLI. 1,338 tokens. The model chooses option 1, which is correct. Question: Statement: "Receiving Party shall not reverse engineer any objects which embody Disclosing Party’s Confidential Information." Does the contract entail the statement, contradict it, or not mention it?

kkProbabilityOption
185%The contract entails the statement.gold, chosen
20%The contract contradicts the statement.
315%The contract does not mention it.

Contract MUTUAL NONDISCLOSURE AGREEMENT Effective Date: 12/10/98 This Agreement governs the disclosure of information by and between Yahoo! Inc., a California corporation, and Restrac, Inc. ("Participant"). 1. The "Confidential Information" is that confidential, proprietary, and trade secret information being disclosed by the disclosing party described as (please be specific): (a) Yahoo Confidential Information (owned by Yahoo and any of its affiliates): Resume and Employment Related. (b) Participant Confidential Information: Resume and Employment Related. 2. Except as set forth in this Section 2, all Confidential Information shall be in tangible form and shall be marked as Confidential or proprietary information of the disclosing party. If the Confidential Information is disclosed orally or visually, it shall be identified as such at the time of disclosure and confirmed in a writing to the receiving party within thirty (30) days of such disclosure. 3. Each of the parties agrees that it will not make use of, disseminate, or in any way disclose any Confidential Information of the other party to any person, firm or business, except to the extent necessary for negotiations, discussions and consultations with personnel or authorized representatives of the other party and any purpose the other party may hereafter authorize in writing. Each of the parties agrees that it shall disclose Confidential Information of the other party only to those of its employees who need to know such information and who have previously agreed, either as a condition to employment or in order to obtain the Confidential Information, to be bound by terms and conditions substantially similar to those of this Agreement. 4. There shall be no liability for disclosure or use of Confidential Information which is (a) in the public domain through no fault of the receiving party (b) rightfully received from a third party without any obligation of confidentiality, (c) rightfully known to the receiving party without any limitation on use or disclosure prior to its receipt from the disclosing party, (d) independently developed by the receiving party, (e) generally made available to third parties without any restriction on disclosure, or (f) communicated in response to a valid order by a court or other governmental body, as otherwise required by law, or as necessary to establish the rights of either party under this Agreement (provided that the party so disclosing has provided the other party with a reasonable opportunity to seek protective legal treatment for such Confidential Information). 5. "Residual Information" shall mean any Confidential Information of the disclosing party which may be retained in intangible form in the minds of those individuals of the receiving party who have had proper access to such Confidential Information. Notwithstanding anything else in this Agreement, the receiving party shall be free to use any Residual Information for any purpose whatsoever, including, without limitation, the development of its own products, provided that such party shall not be entitled to disclose Residual Information to any third parties unless such disclosure is in the scope of, or as part of, any disclosure of its own products or their development. 6. Each of the parties agrees that it shall treat all Confidential Information of the other party with the same degree of care as it accords to its own Confidential Information and each of the parties represents that it exercises reasonable care to protect its own Confidential Information. 7. Each of the parties agrees that it will not modify, reverse engineer, decompile, create other works from, or disassemble any software programs contained in the Confidential Information of the other party unless otherwise specified in writing by the disclosing party. 8. All materials (including, without limitation, documents, drawings, models, apparatus, sketches, designs and lists) furnished to one party by the other, and which are designated in writing to be the property of such party, shall remain the property of such party and shall be returned to it promptly at its request, together with any copies thereof. 9. This Agreement shall govern all communications between the parties that are made during the period from the effective date of this Agreement to the date on which either party receives from the other written notice that subsequent communications shall not be so governed, provided, however that each party’s obligations under Sections 2 and 3 with respect to Confidential Information of the other party which it has previously received shall continue unless and until such Confidential Information falls within Sections 4 or 5. 10. Neither party shall communicate any information to the other in violation of the proprietary rights of any third party. Neither party acquires any licenses under any intellectual property rights of the other party under this Agreement. This Agreement shall be governed in all respects by the laws of the United States of America and by the laws of the State of California as such laws are applied to agreements entered into and to be performed entirely within California between California residents. This Agreement may only be changed by mutual agreement of authorized representatives of the parties in writing. All notices or reports permitted or required under this Agreement shall be in writing and shall be by personal delivery, telegram, telex, telecopier, facsimile transmission or by certified or registered mail, return receipt requested, and shall be deemed given upon personal delivery, five (5) days after deposit in the mail, or upon acknowledgment of receipt of electronic transmission. Notices shall be sent to the addresses set forth at the end of this Agreement or such other address as either party may specify in writing. IN WITNESS WHEREOF, the parties have executed this Agreement in duplicate as of the date first written above. <TABLE> <S> <C> YAHOO! INC. ——————————————– By: /s/ Matt Rowlen By: /s/ M. J. Fahey ————————————- ————————————— Printed Name: Matt Rowlen Printed Name: M. J. Fahey ————————— —————————– Title: Manager of Business Development Title: President ——————————– ———————————— Address: Yahoo! Address: Lexington, MA ——————————– ———————————- ——————————– ———————————- ——————————– ———————————- </TABLE>

Nimble. 558 tokens. The model chooses option 3, which is correct. Question: Apply this rubric: HIGH—route is correct, custody is intact, and the claimant supplies both a unique identifier and one undisclosed feature without a material conflict; release is permitted. MEDIUM—route is correct and the claim is plausible, but either required identifying element is missing or custody has a gap; hold for more evidence. LOW—route is wrong or a material item, date, location, or identifier conflicts; do not release. Weather, staff clothing, and similar incidental observations are distractors, not material conflicts.

kkProbabilityOption
16%Do not release: LOW confidence because the route is incorrect or material facts conflict with the intake record.
235%Send to the Transit Depot Vault but hold the backpack: MEDIUM confidence because an identifying requirement is missing or custody is incomplete.

| 3 | 60% | Send to the Transit Depot Vault and release to Mara: HIGH confidence because custody is intact and both required identifying elements match without material conflict. | gold, chosen

Table 8 (PDF p. 16).

Context A navy canvas backpack was found on Route 18 bus 604 at 18:20 on 12 May. Transit operator Imani Cole sealed it as T-884 and delivered it to the civic hall receptionist, who transferred the unbroken seal to lost-property custodian Arun Vale without any gap in custody.

Facts The backpack sealed as T-884 was found in the transit vehicle Route 18 bus 604. The custody chain for backpack T-884 has no gap between Imani Cole sealing it and Arun Vale receiving it from the civic hall receptionist. The serial patch on backpack T-884 remained unbroken through Arun Vale’s receipt of it. The serial patch recorded for backpack T-884 is a unique identifier for that backpack. The serial-patch value recorded for backpack T-884 in the intake log is SP-7742. Mara Venn supplied the serial-patch value SP-7742 when she filed her claim. The hidden-feature detail recorded for backpack T-884, a brass key inside a torn inner lining pocket, was not publicly disclosed before Mara Venn supplied it. Mara Venn supplied a matching brass-key hidden-feature detail. All other material items, dates, times, locations, and identifiers Mara Venn supplied agree with the intake record.

Policy Policy routes property found in transit vehicles to the Transit Depot Vault, regardless of where it was handed in.

ProofWriter. 208 tokens. The model chooses option 1, which is correct. Question: Statement: "Gary is cold." Based only on the theory, is the statement true, false, or unknown?

kkProbabilityOption

| 1 | 83% | True: the theory implies the statement. | gold, chosen

20%False: the theory implies that the statement is false.
317%Unknown: the theory implies neither the statement nor its negation.

Table 9 (PDF p. 16).

Theory Charlie is blue. Charlie is quiet. Dave is blue. Dave is cold. Dave is young. Erin is blue. Erin is quiet. Erin is white. Erin is young. Gary is cold. Gary is quiet. Gary is young. All quiet things are furry. If Charlie is cold then Charlie is young. White things are cold. All young things are nice. If Gary is nice then Gary is young. If something is young then it is nice. If something is furry and blue then it is white. If something is young and furry then it is nice.

References

  1. [1]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. arXiv:2305.131245.
  2. [2]Diogo Almeida. Introducing system one models & Jev. TypeSafe AI blog, https://typesafe.ai/blog/introducing-system-one-models-and-jev, 2026.
  3. [3]Benchmark Heaven. JevBench: Jev-class model benchmark (v1.4.2.2). https://benchmarkheaven.com/jev-models/, 2026.
  4. [4]Bespoke Labs. Nimble. https://github.com/bespokelabsai/nimble, 2025.
  5. [5]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of AAAI, 2020.arxiv.org/abs/1911.11641
  6. [6]Bitext. Bitext customer support LLM chatbot training dataset. https://huggingface.co/datasets/bitext/Bitext-customer-support-llm-chatbot-training-dataset, 2023.
  7. [7]Yung-Sung Chuang, Benjamin Cohen-Wang, Shannon Zejiang Shen, Zhaofeng Wu, Hu Xu, Xi Victoria Lin, James Glass, Shang-Wen Li, and Wen-tau Yih. SelfCite: Self-supervised alignment for context attribution in large language models. In International Conference on Machine Learning, 2025. arXiv:2502.09604.arxiv.org/abs/2502.09604
  8. [8]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL, 2019.arxiv.org/abs/1905.10044
  9. [9]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  10. [10]Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander Madry. ContextCite: Attributing model generation to context. arXiv preprint arXiv:2409.00729, 2024.
  11. [11]Benjamin Cohen-Wang, Yung-Sung Chuang, and Aleksander Madry. Learning to attribute with attention. arXiv preprint arXiv:2504.13752, 2025.
  12. [12]Convai Innovations. Laya: an open non-autoregressive decision model. https://huggingface.co/convaiinnovations/laya, 2026.
  13. [13]Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2307.08691
  14. [14]DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024.arxiv.org/abs/2412.19437
  15. [15]Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of ACL, 2020. arXiv:1911.03429.arxiv.org/abs/1911.03429
  16. [16]Kuntai Du, Bowen Wang, Chen Zhang, Yiming Cheng, Qing Lan, Hejian Sang, Yihua Cheng, Jiayi Yao, Xiaoxuan Liu, Yifan Qiao, Ion Stoica, and Junchen Jiang. PrefillOnly: An inference engine for prefill-only workloads in large language model applications. arXiv preprint arXiv:2505.07203, 2025.arxiv.org/abs/2505.07203
  17. [17]Jack FitzGerald et al. MASSIVE: A 1M-example multilingual natural language understanding dataset with 51 typologically-diverse languages. In Proceedings of ACL, 2023.arxiv.org/abs/2204.08582
  18. [18]Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of ACL, 2021. arXiv:2012.15723.arxiv.org/abs/2012.15723
  19. [19]Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Manuel Faysse, Duarte M. Alves, Emmanuel Malherbe, André F. T. Martins, Céline Hudelot, and Pierre Colombo. Should we still pretrain encoders with masked language modeling? arXiv preprint arXiv:2507.00994, 2025.
  20. [20]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021.
  21. [21]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.arxiv.org/abs/2106.09685
  22. [22]Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of EMNLP-IJCNLP, 2019.arxiv.org/abs/1909.00277
  23. [23]Yuta Koreeda and Christopher D. Manning. ContractNLI: A dataset for document-level natural language inference for contracts. In Findings of EMNLP, 2021.
  24. [24]Jacky Kwok, Hangoo Kang, Tarun Suresh, Jon Saad-Falcon, Marco Pavone, Christopher Ré, and Azalia Mirhoseini. Contrastive language models: A system one model for fast and generalizable decision-making. Blog post, Stanford University and NVIDIA Research, 23 September 2026. https://contrastive-lm.notion.site, 2026.
  25. [25]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023.arxiv.org/abs/2309.06180
  26. [26]Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of EMNLP-IJCNLP, 2019.arxiv.org/abs/1909.02027
  27. [27]Yinheng Li and Justin Wagle. LLM-as-Jev: LLMs are already Jev-style decision models—when and how to fine-tune them. arXiv preprint arXiv:2610.02076, 2026.arxiv.org/abs/2610.02076
  28. [28]Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. LogiQA 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.DOI
  29. [29]Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of ACL, 2020.arxiv.org/abs/1910.14599
  30. [30]Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R. Bowman. QuALITY: Question answering with long input texts, yes! In Proceedings of NAACL, 2022.arxiv.org/abs/2112.08608
  31. [31]Victor Hugo Panisa. Von: a non-autoregressive decision model. https://huggingface.co/wfzyx/von, 2026.
  32. [32]Jungwoo Park, Taewhoo Lee, and Jaewoo Kang. UltraBERT: Beyond the limit of BERT pre-training. arXiv preprint arXiv:2610.15028, 2026.arxiv.org/abs/2610.15028
  33. [33]Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, 2024. arXiv:2308.11483.arxiv.org/abs/2308.11483
  34. [34]Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
  35. [35]Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations, 2023. arXiv:2210.12353.arxiv.org/abs/2210.12353
  36. [36]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. In Proceedings of AAAI, 2020.arxiv.org/abs/1907.10641
  37. [37]Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. In Proceedings of EACL, 2021. arXiv:2001.07676.arxiv.org/abs/2001.07676
  38. [38]Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019.arxiv.org/abs/1911.02150
  39. [39]Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Findings of ACL-IJCNLP, 2021.
  40. [40]Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 2022.arxiv.org/abs/2108.00573
  41. [41]Yubo Wang et al. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2024.arxiv.org/abs/2406.01574
  42. [42]Benjamin Warner, Antoine Chaffin, Benjamin Clavié, et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.13663, 2024.arxiv.org/abs/2412.13663
  43. [43]Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, 2017.arxiv.org/abs/1707.06209
  44. [44]Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, and Benjamin Van Durme. Seq vs seq: An open suite of paired encoders and decoders. In International Conference on Learning Representations (ICLR), 2026. arXiv:2507.11412.arxiv.org/abs/2507.11412
  45. [45]Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. ReClor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/2002.04326
  46. [46]Jiamu Zhang, Tianze Yang, Yucheng Shi, Evan Chen, Zixiang Nie, Kelly Wan, Liangjie Hong, Ninghao Liu, and Liang Wu. AnyJev technical report. arXiv preprint arXiv:2610.00831, 2026.arxiv.org/abs/2610.00831
  47. [47]Hang Zhao, Qile P. Chen, Yijing Barry Zhang, and Gang Yang. Advancing single and multi-task text classification through large language model fine-tuning. arXiv preprint arXiv:2412.08587, 2024.arxiv.org/abs/2412.08587
  48. [48]Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, 2021. arXiv:2102.09690.arxiv.org/abs/2102.09690
  49. [49]Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations, 2024. arXiv:2309.03882.arxiv.org/abs/2309.03882
  50. [50]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2312.07104

Paper details

vlnc:W_k5j6y22m
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents