WorksComputer Science
LLMs Are Already Confidence-Aware Zero-Shot Classifiers
Abstract
Zero-shot large language models (LLMs) are increasingly used as text classifiers, but their labels are only useful when we know to what extent we can trust them. We compare black-box methods (verbalised confidence and sampling) with white-box methods that read token probabilities (P(True), and multiple logit-scoring methods using constrained generation). We evaluate eleven open-weight LLMs (1B-32B parameters) on five classification datasets with 3 to 151 labels, measuring classification performance (accuracy, F1-macro), discrimination (AUROC), and calibration (ECE and Brier score). The choice of confidence method rarely impacts accuracy significantly, but logit-based scoring methods lead to much better discrimination (AUROC 0.74-0.77) than black-box methods (AUROC 0.57-0.66). Replacing labels with single-token identifiers preserves discrimination but costs 3-6 accuracy points on average. We also introduce decision-only scoring, which only scores positions where labels diverge in the prefix tree (trie). It reduces decode steps by 88-96% on the three largest label sets and performs comparably to full-label scoring. The best open-weight model-method configurations (Gemma 4 31B and Qwen 3.5 9B) are competitive with the commercial Jev 1.13.0 classifier.
Introduction
Large language models (LLMs), now most commonly used as chatbots or coding agents, have a range of other emergent abilities, including classification ((Wei et al., 2022)). As their performance as zero-shot classifiers across a range of domains has improved, LLMs have been increasingly considered for adoption across a number of high-stakes industries such as politics ([9, 6]), healthcare ([2, 43]), and business ([31]). With this, it is increasingly important to understand when we can trust their outputs, and to what extent, using the model’s confidence as an indicator.
While there is evidence that smaller fine-tuned language models with a classification head often outperform higher-parameter zero-shot LLMs ([5]), fine-tuning such a model is not always possible. Although zero-shot classification with LLMs is not necessarily an optimal solution, it provides more flexibility and can be much more cost-effective, and the gap between zero-shot and fine-tuned classifiers is decreasing. Zero-shot classification can be particularly beneficial in data-scarce environments ((Zhang et al., 2026)), where data annotation costs are high, and when label sets may change often, meaning new classifiers do not need to be trained after each change ([35]). Semi-automated data labelling with LLMs is also growing in popularity ([7, 19]); ensuring the quality of annotations, and good confidence calibration is particularly important in this setting, enabling efficient use of human annotators, without negatively impacting dataset quality ((Wang et al., 2024b)).
Confidence can be extracted from LLMs in multiple ways. Common strategies include asking the model to verbalise its confidence ([47]), sampling multiple responses with a given temperature ((Wang et al., 2022; Xiong et al., 2024)), asking whether its answer is correct ((Kadavath et al., 2022)), and scoring labels using token probabilities ([13, 27] (Zhao et al. 2021)). These confidence extraction methods are rarely compared under the same conditions. The majority of studies investigating confidence extraction focus on question answering (QA) tasks, where knowledge is the key cause of uncertainty. In classification tasks, this is not the case; here uncertainty is much more likely to arise from task ambiguity rather than knowledge issues. In classification problems, labels are often many tokens long and they share decoding paths, so aggregations must represent the full label. To aggregate label probabilities, outputs must be constrained but the model’s first token may be a preamble to the label rather than the answer itself (Wang et al., 2024a).
This work provides empirical evidence for the most effective ways of using decoder-only LLMs as zero-shot classifiers, placing particular importance on each method’s discriminative abilities and confidence calibration as well as pure classification performance (accuracy, F1-macro). We test using popular classification datasets with labels containing overlapping decoding paths to better test the full-label aggregation methods and ensure they are robust to edge cases; an example of label overlap in SST-5 with the Gemma 4 tokeniser would be: “very” “positive” and “very” “negative”. For this, we investigate a number of logit-score aggregation methods.
With the relatively recent release and popularity of Jev-style models, transformer-based classification models that provide a distribution over the whole label set, we use TypeSafe AI’s Jev (1.13.0)1 as a reference point. Zero-shot model-method combinations in this study are competitive with Jev 1.13.0, performing comparably in terms of accuracy and AUROC, but generally worse in terms of raw calibration.
Contributions:
We provide a systematic overview of zero-shot classification with LLMs, focusing on the discrimination and calibration across a range of models for classification tasks.
We introduce a simple scoring rule that only scores and length normalises over decision points in the decode trie, reducing the number of decode steps by 96% on Banking77, 92% on CLINC150, and 88% on 20NG (with the Gemma 4 tokeniser); these reductions in decode steps will not match exactly with the time saved due to overheads.
Experimental code is publicly released on GitHub: https://github.com/minieggz/llm-as-classifier.
Key Findings:
Zero-shot LLMs are natural classifiers that can discriminate relatively well, achieving a mean AUROC of up to 0.802 (Qwen 3.5 9B, constrained with Google length normalisation).
Reducing labels to single tokens often causes lower classification performance but retains discriminative abilities.
Verbalised confidence, or constrained decoding and the choice of scoring methods rarely statistically significantly impact the accuracy of zero-shot LLM classifiers.
Verbalised confidence (on a Likert-scale) achieves the highest calibration overall but is significantly worse in terms of discrimination; methods should not be chosen on calibration alone.
Related Work
Constrained Decoding
As modern LLMs are generally successful at following output constraints, it is common to obtain labels through regex or simple string matching ([3]). Frameworks such as Outlines ((Willard and Louf, 2023)), Guidance ([34]), and XGrammar ([12]) restrict output by programmatically conforming to a regular expression or grammar, masking disallowed tokens at each decoding step. While constrained decoding can ensure we get an answer that fits a specific format, it can distort the LLM’s output distribution ([38]).
Extracting Confidence from LLMs
Confidence can be extracted from LLMs with black-box methods, which have no access to model internals, or white-box methods which have total access.
Unconstrained approaches. Instruction tuning has been reported to degrade token probability calibration relative to base variants ((Kadavath et al., 2022)), motivating the use of verbal approaches ([47]). Confidence can be verbalised either on a Likert-scale or numerically ([47, 14]). Using an LLM-as-a-judge approach, where the model verbalises its confidence separately to the classification label is also reported to be effective ([33]).
Another way to calculate confidence is to run inference with a set temperature a number of times; the more times inference is run, the closer the output distribution should be to the model’s true answer distribution ((Wang et al., 2022; Xiong et al., 2024). The majority decision is chosen as the label, with the confidence equal to the proportion of decisions that were the majority label.
The P(True) method (Kadavath et al., 2022) uses the answer generated as input to asking the model whether it believes that answer is true or false. The log likelihoods of the first tokens are then softmaxed to obtain the probabilities.
Scoring multi-token labels. Many white-box methods focus on token-level probabilities [17]. Confidence is often simply represented by the likelihood of the whole sequence (calculated by summing the log probabilities of each token) [13, 27] or a length-normalised likelihood [21] (Zhao et al. 2021). These methods are mostly applied to QA datasets. This length normalisation is often simple token length. Wu et al. (2016) apply an alternative length normalisation penalty that has not yet been applied to multi-token label scoring for classification.
For classification tasks specifically, some approaches also normalise the likelihood based on a prior for each label, found by inspecting token probabilities after a blank input [44] (Zhao et al. 2021). Kumar et al. [28] flip this approach with Gen-Z by asking what the likelihood of the input is given each possible output label, removing the requirement of length normalisation due to a constant output size.
A common way to avoid multi-token scoring is to replace full labels with single-token identifiers (often letters), but LLMs can show systematic preferences for particular option identifiers (Zheng et al., 2024; Pezeshkpour and Hruschka, 2024). Despite this, some open-source Jev-style methods [18, 32] take this kind of approach.
TypeSafe AI’s Jev API model, trained using Reinforcement Learning for Calibrated Decisions (RLCD) (TypeSafe AI, 2026), supports 255 output classes. It returns a top class label along with a full probability distribution across all labels.
Methodology
In this study, we consider classification tasks with a static label set. Although each method can be used in cases where the label set is constantly changing, using a static label set simplifies the evaluation and enables us to analyse the performance differences at the class level (F1-macro).
Much of the prior work on short-sequence model confidence calibration focuses on the question answering (QA) task [47] (Xiong et al. 2024). QA relies heavily on recalling knowledge; the classification datasets we use do not rely on recall, so the model’s uncertainty reflects the difficulty of assigning a label to a given input, rather than how well the model “knows” a fact.
Datasets
We evaluate on text-classification datasets spanning a number of tasks, including topic classification, intent classification, sentiment detection, and fact-checking. We mainly focus on datasets that have labels of varying lengths and that contain overlapping token paths to investigate the issues of length normalisation and the difficulty of extracting full distributions across the label space. These datasets are: Banking77 [8], CLINC150 [30], 20 Newsgroups [29], and SST-5 [45]. FEVER [46], a fact-checking dataset, is used to evaluate cases where all labels diverge at the first token with labels “SUPPORTS”, “REFUTES”, and “NOT ENOUGH INFO”.
Banking77 and CLINC150 are both intent datasets and have the largest number of classes at 77 and 151 (including out-of-scope) respectively, with a number of overlapping tokens within the labels. 20 Newsgroups, a topic classification dataset, contains overlapping labels that are organised into topics and subtopics; labels with the same parent topic will therefore share decoding paths. SST-5 provides a case where the two labels that share token paths are total opposites (“very positive” and “very negative”).
Large Language Models
In this study, we use only open-weight language models available on HuggingFace, using the Transformers library (Wolf et al., 2020). We cover a range of model families and sizes (1B to 32B parameters). These models include Gemma 4 E2B, Gemma 4 E4B, and Gemma 4 31B [16]; Llama 3.2 1B and 3B (Grattafiori et al., 2024); Qwen 2.5 7B [42]; Qwen 3 8B, Qwen 3 14B, and Qwen 3 32B [41]; Qwen 3.5 4B, and Qwen 3.5 9B [40]. For all models, we are only testing their abilities without chain-of-thought; in most practical classification scenarios, chain-of-thought would significantly increase computational cost.
Decoding Methods
Unconstrained decoding represents a simpler chatbot setup where answers are extracted after generation using only the output text. With constrained methods, we access the individual token probabilities at each decode step.
Unconstrained Generation
Simple inference methods. For the unconstrained generation, we test a number of prompting strategies to extract the answer from the LLM in different formats. These formats include: 1) plaintext label only (“Unconstrained (no conf)”), 2) JSON format with only the label (“Unconstrained JSON (no conf)”), 3) JSON format with confidence before the label (“Unconstrained JSON verbal (pre)”), and 4) JSON format with confidence after the label (“Unconstrained JSON verbal (post)”). We also test single-token substitutions for labels – “ABC” in Table 1 – and Likert-scaled confidence instead of percentage (“Likert”). This Likert scale is “Not at all confident”, “Slightly confident”, “Moderately confident”, “Very confident”, and “Completely confident”, equating to 20%, 40%, 60%, 80%, and 100%.
| Method | Accuracy | F1-macro | AUROC | ECE | Brier |
| Unconstrained (no conf) | 0.724* | 0.715* | 0.500 | 0.276 | 0.276 |
| Unconstrained (no conf ABC) | 0.671 | 0.663 | 0.500 | 0.329 | 0.329 |
| Unconstrained (10 samples) | 0.724* | 0.715* | 0.569 | 0.254 | 0.255 |
| Unconstrained + P(True) | 0.724* | 0.715* | 0.741* | 0.227 | 0.235 |
| Unconstrained JSON (no conf) | 0.723* | 0.718* | 0.500 | 0.277 | 0.277 |
| Unconstrained JSON (no conf ABC) | 0.686 | 0.680 | 0.500 | 0.314 | 0.314 |
| Unconstrained JSON verbal (pre) | 0.719* | 0.718* | 0.618 | 0.217 | 0.241* |
| Unconstrained JSON verbal (post) | 0.722* | 0.719* | 0.658 | 0.212 | 0.231* |
| Unconstrained JSON verbal (pre, Likert) | 0.721* | 0.716* | 0.606 | 0.167* | 0.222* |
| Unconstrained JSON verbal (post, Likert) | 0.723* | 0.720* | 0.636 | 0.150* | 0.202* |
| Unconstrained JSON verbal (pre, ABC) | 0.679 | 0.676 | 0.592 | 0.262 | 0.280 |
| Unconstrained JSON verbal (post, ABC) | 0.681 | 0.675 | 0.624 | 0.248 | 0.263 |
| Constrained (greedy) | 0.723 | 0.714* | 0.500 | 0.277 | 0.277 |
| Constrained (total likelihood) | 0.723* | 0.715* | 0.764* | 0.223 | 0.232 |
| Constrained (total likelihood, decision-only) | 0.723* | 0.715* | 0.764* | 0.223 | 0.232 |
| Constrained (length norm) | 0.723* | 0.715* | 0.759* | 0.213* | 0.227* |
| Constrained (length norm, decision-only) | 0.723* | 0.715* | 0.754 | 0.185* | 0.211* |
| Constrained (Google length norm) | 0.723* | 0.715* | 0.764* | 0.208 | 0.222 |
| Constrained (Google length norm, decision-only) | 0.723* | 0.715* | 0.763* | 0.220 | 0.229 |
| Constrained (probability tree) | 0.723* | 0.714* | 0.763* | 0.223 | 0.232 |
| Constrained (ABC) | 0.667 | 0.662 | 0.758* | 0.256 | 0.266 |
| Constrained JSON (greedy) | 0.715 | 0.707 | 0.500 | 0.285 | 0.285 |
| Constrained JSON (total likelihood) | 0.713 | 0.706 | 0.760* | 0.222 | 0.234 |
| Constrained JSON (total likelihood, decision-only) | 0.716 | 0.707 | 0.763* | 0.222 | 0.234 |
| Constrained JSON (length norm) | 0.681 | 0.679 | 0.738 | 0.249* | 0.264* |
| Constrained JSON (length norm, decision-only) | 0.715 | 0.708 | 0.746* | 0.185* | 0.213* |
| Constrained JSON (Google length norm) | 0.708 | 0.702 | 0.760* | 0.203 | 0.225 |
| Constrained JSON (Google length norm, decision-only) | 0.716 | 0.708 | 0.762* | 0.218 | 0.231 |
| Constrained JSON (probability tree) | 0.715 | 0.707 | 0.762* | 0.222 | 0.234 |
| Constrained JSON (ABC) | 0.683 | 0.680 | 0.766* | 0.247 | 0.254 |
Table 1. Mean score over all model–dataset pairs (excluding Llama 3.2 1B, Llama 3.2 3B). Bold indicates the best mean. *indicates a method is not significantly different from the method with the best mean rank (two-sided Wilcoxon signed-rank test paired over model-dataset pairs, Holm-adjusted ).
By using the JSON format for confidence extraction, we are using a structure that is easy to obtain both the answer and confidence from and that language models are often post-trained on for tool calling ([41] (Grattafiori et al. 2024)). For the plaintext label only, we use regular expressions to extract the largest label first (to prevent cases where “positive” may be matched before “very positive”). We again use regular expressions to extract the JSON object, greedily extracting everything from the first “{” to the final “}”. If there is more than one JSON object, this will be picked up as a generation error once parsed. We use the json module to parse JSON output.
Sampling with temperature. As an extension to simple plaintext labels, we obtain a confidence estimate by sampling a number of language model outputs using a given temperature score ((Wang et al., 2022; Xiong et al., 2024)). To test this method, we use a temperature of 0.5, which Kuhn et al. [27] found to be optimal for sampling-based uncertainty estimation (from a range of 0.25 to 1.5); we sample 10 times per data point. Any output errors here will
be removed completely, so if there are 2 errors, we will divide the number of occurrences of each label by 8 rather than 10. We do note that a resolution of 10% in confidence may impact the results. In all results, this method is referenced as “Unconstrained (10 samples)”.
Constrained Decoding
With constrained decoding, it is possible to obtain a probability distribution over the full label set. We constrain the decoding process using a trie data structure ([11, 32]).
Tree Construction. We represent the constrained decoding process using the trie data structure, using a depth-first approach in its generation. By using the trie, we can optimise the use of the KV-cache for efficient computation. At each decode step, we softmax over the whole vocabulary and then mask the output to only contain the valid next tokens.
A trie is constructed for each prompt and each prompt corresponds to an individual data point. At the end of each label, we calculate the joint likelihood of “end-of-answer” tokens. This may be only the EOS token for some models; in other instruction-tuned models, there are specialised end-of-turn tokens (such as <turn|> for Gemma 4). For the JSON output specifically, the tokens ” and ” are also added. In the trie, we represent the combined “end-of-answer” tokens as token ID “−1”. An example of the trie structure can be seen in Figure 1. In our experiments, we run trie construction for each model-dataset combination and label aggregation is performed on the saved tries afterwards. For JSON responses, we prefill the beginning of the JSON output (for example “{"label": "”). The exact prefill token sequence was obtained through three example outputs by each model.

Figure 1. Example trie from SST-5 (Gemma 4 31B). Each node shows the token with log-probability / probability, where the log-probability is over the whole vocabulary and probability is between siblings. Shaded tokens are decision tokens; dashed tokens are forced and are skipped by the decision-only and probability methods. For other datasets, non-decision tokens can be in the middle of labels.
Tree aggregation. For a label with tokens , let , given input . Most methods assign a score , from which we obtain a distribution over all labels and confidence:
Some methods redefine directly.
Greedy constrained decoding. To directly compare constrained generation against unconstrained generation, we simply use the greedy approach, assigning 100% confidence to the greedy-decoded output. In this instance if the greedy path chooses label , otherwise .
Total log likelihood and length normalisation. The simplest scoring method is a total log likelihood sum for each label () ([13]). A simple extension to this is to normalise the label likelihoods by label length in tokens () ([21] (Zhao et al. 2021)). As well as simple token-length normalisation, we also test the length normalisation penalty proposed by Google for machine translation () (Wu et al., 2016). These scoring rules (Equation (2)) are used in Equation (1) to obtain the distribution and confidence.
where is the label sequence length in tokens and in our experiments, following Vaswani et al. (2017).
Decision-only methods. Decoding each full label, particularly for datasets with large label sets, can be computationally expensive. We propose performing the same calculations introduced above, but applying them to only decision tokens. A decision token is one with siblings, meaning it was not the only choice available (highlighted in Figure 1). Let be the set of positions of the decision tokens on the path to label .
tokens on the path to label . The decision-only scoring functions are
where is the number of decision tokens on the path to , and these scores are used in Equation (1) as before. By removing forced tokens, as well as improving computational efficiency, we may also reduce the bias introduced by length normalisation.
Probability tree. We could not find any direct examples of probability trees being used to represent full-label decoding for classification. For the probability tree method, rather than softmaxing after all labels have been decoded, we softmax all sibling nodes at each step and multiply the probabilities of each token at the end:
where is the set of valid next tokens (siblings of ) after the previous tokens . Figure 1 visualises the probability tree. When all labels diverge at the first token, the probability tree method reduces to a probability distribution over single-token identifiers. In constrained methods where we replace labels with single tokens (“Constrained ABC)” and “Constrained JSON (ABC)”), we use the probability tree.
The P(True) method also makes use of the probability tree to get the probability of “True” vs. “False” on already classified data points.
Evaluation
For every method and model-dataset pair we compute accuracy and F1-macro (classification performance), AUROC (discrimination) ([24]), and ECE ([36, 22]) and top-label Brier score (calibration) ([4, 23]) (Brier score applied to the confidence in the predicted label). We also investigate the number of label generation errors per model.
From these results, we report means over all model-dataset pairs for each method. We use the method with the highest mean rank for each metric as the reference and we compare every other method against it using a two-sided Wilcoxon signed-rank test (Wilcoxon, 1945) paired over model-dataset pairs, with Holm correction [25] within each metric (). Individual model results (Appendix § 11) are reported without tests, as cannot reach .
We then compare the highest performing model-method combinations against Jev 1.13.0 using the same metrics.
Unconstrained evaluation edge cases. When evaluating unconstrained outputs, unparsable errors are treated as incorrect answers. In our F1-macro calculations, we assign invalid predictions to incorrect classes as evenly as possible (the gold label is excluded from this assignment) using a round-robin approach. In each round, we prioritise adding incorrect predictions to the largest class to cause a lower impact on the result. When a method does not provide a confidence score, or confidence is unparsable, we assume 100% certainty. For constrained outputs, this does not apply as it is impossible to obtain a label outside the label set.
Experimental Setup
Key Comparisons
In our experiments, we test a number of decoding methods on 5 datasets (Banking77, CLINC150, FEVER, SST-5, 20NG) using 11 models (see § 3.2). Our main comparisons are constrained vs. unconstrained output, single token output vs. full-label output, JSON format vs. plaintext output, verbal confidence vs. sampling confidence vs. logit-based confidence vs. post-hoc confidence.
The different decoding methods we compare are described in § 4.
Prompting Methodology
When prompting the models, we follow the COSTAR-A [37] prompting style. This format includes “Context”, “Objective”, “Style”, “Tone”, “Audience”, “Response”, and “Answer”.
For our prompts, we adapt COSTAR-A as follows: style and tone are merged into a single section; audience is omitted; and response is replaced by a “Format” section that describes both the structure of the input and the permitted output labels. We also add an “Input” section containing the instance to be classified. The “Context”, “Style & Tone”, and “Format” sections are included in the system prompt, and the “Objective”, “Input”, and “Answer” sections form the user prompt which varies per data point.
With modern LLMs proficient in markdown [10] (Freeburg 2026) and XML [1], all prompts use the markdown format for the above sections and inputs are provided inside XML tags.
Example prompts can be found in Appendix § 12.
Model Settings
For all models, we use 16-bit precision and we use greedy decoding in all instances (other than for the sampling inference method). We use greedy decoding to provide near-deterministic results that may only differ due to hardware-based floating point calculation differences; we generate with a batch size of 1 to mitigate this further and eliminate padding differences.
Results & Discussion
Smaller models generally produce more generation errors. As shown in Figure 2, the number of generation errors generally increases as the number of model parameters decreases. While the general trend is clear from looking at all models, it is most visible within the Gemma 4 family of models where the error rate decreases from 0.22% with Gemma 4 E2B to 0.01% for the 31 billion parameter variant. The mid-sized Qwen 3 and Qwen 3.5 models, however, show that this is not always the case.

Figure 2. Rate of errors made by each model (all unconstrained inference methods: plaintext, JSON with no confidence, verbal pre and post, Likert and percentage) plotted against number of model parameters (in billions) and grouped by model family.
Despite constrained decoding preventing generation errors, it can harm model performance. It is well-investigated that constrained decoding can often distort an LLM’s output distribution, in that it is forced to choose between a set of tokens where all of them have extremely low likelihoods (Wang et al., 2024a). The alternative formats the unconstrained models used include bullet pointing, prepending with “Answer: ”, and using markdown formatting. For the full results, see Table 3 and Table 4.
| Banking77 | CLINC150 | FEVER | SST-5 | 20NG | |
| Llama 3.2 1B | +1.66% / -11.59% | +3.00% / -6.07% | +1.13% / -0.19% | +0.00% / -0.00% | +0.10% / -0.13% |
| Llama 3.2 3B | +0.55% / -0.39% | +2.56% / -1.82% | +0.59% / -0.15% | +0.09% / -0.18% | +0.21% / -0.13% |
| Gemma 4 E2B | +0.16% / -0.03% | +0.31% / -2.78% | +0.00% / -0.10% | +0.00% / -0.00% | +0.11% / -0.83% |
| Gemma 4 E4B | +0.03% / -0.03% | +0.04% / -0.02% | +0.00% / -0.00% | +0.00% / -0.00% | +0.00% / -0.00% |
| Gemma 4 31B | +0.00% / -0.00% | +0.04% / -0.00% | +0.01% / -0.01% | +0.00% / -0.05% | +0.03% / -0.00% |
| Qwen2.5 7B | +0.26% / -0.36% | +0.05% / -0.05% | +0.15% / -0.06% | +0.05% / -0.00% | +0.07% / -0.00% |
| Qwen3 8B | +0.03% / -0.00% | +0.02% / -0.05% | +0.00% / -0.00% | +0.00% / -0.05% | +0.06% / -0.01% |
| Qwen3 14B | +0.00% / -0.03% | +0.15% / -0.09% | +0.00% / -0.00% | +0.00% / -0.00% | +0.10% / -0.00% |
| Qwen3 32B | +0.10% / -0.00% | +0.05% / -0.09% | +0.00% / -0.00% | +0.05% / -0.00% | +0.07% / -0.01% |
| Qwen3.5 4B | +0.13% / -0.00% | +0.04% / -0.02% | +0.00% / -0.00% | +0.05% / -0.00% | +0.04% / -0.04% |
| Qwen3.5 9B | +0.03% / -0.00% | +0.00% / -0.02% | +0.01% / -0.01% | +0.05% / -0.00% | +0.00% / -0.06% |
Table 3. The percentage of data points on which constrained plaintext generation helps (+) (constrained output is correct, unconstrained output is either incorrect or unparsable) and hinders (−) (unconstrained output is correct, constrained output is incorrect) classification performance by model and dataset. Both constrained and unconstrained generation use greedy decoding.
| Banking77 | CLINC150 | FEVER | SST-5 | 20NG | |
| Llama 3.2 1B | +4.22% / -20.13% | +18.20% / -8.62% | +32.25% / -35.92% | +11.22% / -24.75% | +3.93% / -3.13% |
| Llama 3.2 3B | +2.63% / -1.40% | +3.11% / -3.56% | +4.45% / -1.48% | +2.44% / -3.57% | +2.09% / -5.25% |
| Gemma 4 E2B | +0.75% / -0.91% | +0.58% / -0.91% | +1.01% / -7.62% | +3.35% / -5.29% | +1.24% / -2.27% |
| Gemma 4 E4B | +0.29% / -0.49% | +0.62% / -3.98% | +0.33% / -1.00% | +1.49% / -1.86% | +0.59% / -1.28% |
| Gemma 4 31B | +0.29% / -0.19% | +0.09% / -0.13% | +0.11% / -0.06% | +1.49% / -0.81% | +0.10% / -0.15% |
| Qwen2.5 7B | +1.92% / -2.79% | +3.56% / -3.80% | +0.93% / -0.86% | +1.76% / -2.81% | +1.80% / -1.57% |
| Qwen3 8B | +2.11% / -1.17% | +0.91% / -3.15% | +0.33% / -0.78% | +4.66% / -5.02% | +1.22% / -2.36% |
| Qwen3 14B | +0.81% / -1.07% | +0.89% / -3.27% | +0.11% / -1.24% | +7.47% / -8.91% | +1.33% / -2.37% |
| Qwen3 32B | +1.30% / -1.33% | +1.95% / -2.35% | +0.83% / -0.76% | +3.26% / -2.13% | +1.57% / -1.90% |
| Qwen3.5 4B | +1.17% / -1.66% | +1.55% / -2.07% | +0.16% / -1.04% | +2.08% / -3.35% | +1.67% / -1.85% |
| Qwen3.5 9B | +0.78% / -0.94% | +2.16% / -1.27% | +0.05% / -2.28% | +3.98% / -6.56% | +1.24% / -1.29% |
Table 4. The percentage of data points on which constrained JSON generation helps (+) (constrained output is correct, unconstrained output is either incorrect or unparsable) and hinders (−) (unconstrained output is correct, constrained output is incorrect) classification performance by model and dataset. Both constrained and unconstrained generation use greedy decoding.
The confidence extraction method rarely significantly impacts classification performance. Post-hoc confidence methods like P(True) do not change the accuracy at all, as they are extensions of greedy inference with no confidence.
Other greedy generation methods all use JSON-formatted outputs, and the accuracy is only statistically significantly worse than the best performing method (unconstrained output with no confidence) when labels are reduced to single-token replacements. The averaged constrained JSON methods perform statistically worse than the best performing method, but some of the highest performing model-method combinations do use constrained JSON output, indicating that the preferred format is model-dependent (see Table 2).
| Model | Method | Accuracy | F1-macro | AUROC | ECE | Brier |
| Gemma 4 31B | Unconstrained JSON verbal (pre, Likert) | 0.803 | 0.796 | 0.642 | 0.120 | 0.145 |
| Gemma 4 31B | Unconstrained JSON verbal (post, Likert) | 0.802 | 0.796 | 0.673 | 0.107 | 0.140 |
| Qwen 3.5 9B | Constrained (Google length norm) | 0.750 | 0.741 | 0.802 | 0.107 | 0.153 |
| Qwen 3.5 9B | Constrained (length norm, decision-only) | 0.751 | 0.744 | 0.793 | 0.053 | 0.142 |
| Gemma 4 31B | Constrained JSON (Google length norm, decision-only) | 0.802 | 0.795 | 0.775 | 0.190 | 0.190 |
| Gemma 4 31B | Constrained JSON (ABC) | 0.798 | 0.790 | 0.772 | 0.196 | 0.196 |
| Jev 1.13.0 | Jev | 0.788 | 0.780 | 0.801 | 0.086 | 0.134 |
Table 2. Best model–method combinations for each metric, best constrained method (accuracy), best ABC-logit method (accuracy), compared against Jev 1.13.0. Scores are means across all datasets.
In terms of raw calibration, verbal confidence performs well. Likert-scale confidence is better calibrated but leads to worse discriminative abilities. For verbal confidence, asking models for JSON output with answer first and confidence second performed the best in terms of ECE and top-label Brier score (shown in Table 1). While asking for confidence in a Likert scale provides better raw calibration, it leads to a poorer AUROC, most likely due to the significantly lower resolution of the Likert scale compared to percentage output.
Methods that reduce the labels to single tokens often lead to reductions in accuracy and calibration, but perform well discriminatively and are more computationally efficient. Overall (see Table 1), all methods that reduce labels to single tokens perform significantly worse than the highest ranking method on average in terms of accuracy, F1-macro, ECE and top-label Brier score. While the accuracy is lower, they perform well on discrimination (AUROC) with “Constrained JSON (ABC)” achieving the highest average score. The constrained single-token methods also require only one decode step per data point, making these methods quick and resource-efficient when compared to methods requiring the full decoding of each label.
Tree-based decoding and logit scoring lead to significantly better classifiers in terms of discrimination. As shown in Table 1, the constrained logit-based methods perform well, with only two trie-based methods (excluding “greedy”) performing statistically significantly worse on average than the best performing method “Constrained JSON (ABC)” in terms of AUROC. While calibration scores in logit-based methods are not quite
as low as some of the verbalised confidence methods overall, the best performing model and method combination for ECE is Qwen 3.5 9B constrained with length normalisation (decision tokens only). Calibration scores could be improved using a small validation set and temperature scaling [22].
The P(True) method, which is also a logit-based method, is not statistically significantly worse than other logit-based methods in terms of discrimination and maintains the benefits of unconstrained generation for classification performance. Using P(True), however, does not produce a confidence distribution over all labels as other logit-scoring methods do.
Open-weight instruction-tuned models are competitive with Jev 1.13.0. Of the best performing methods in Table 2, Jev 1.13.0 only wins on the top-label Brier score but no model-method combination beats Jev on classification performance and discrimination combined. The model-method combination with the highest accuracy and F1-macro is Gemma 4 31B using unconstrained JSON generation with Likert-scale confidence; this does come at a large cost in discrimination. Gemma 4 31B constrained with JSON output and Google length normalisation on decision tokens only, as well as Gemma 4 31B with constrained single-token JSON output, achieves an accuracy and F1-macro higher than Jev, but performs worse in terms of AUROC, ECE, and top-label Brier. The best performing Qwen 3.5 9B method combinations perform similarly or better in terms of AUROC but do lose on raw classification performance. While the Jev API may be a viable option for those without extensive privacy concerns, open-weight alternatives with no extra training are competitive.
Conclusion
Open-weight zero-shot LLMs are effective classifiers that are competitive with current commercial classification models in terms of accuracy and discrimination. While the raw calibration is not always as strong, it could likely be improved with temperature scaling (Guo et al. [22]). Unconstrained outputs using verbalised confidence have been shown to perform well in terms of accuracy, but methods that use logit scores are shown to be significantly better at providing confidence scores that discriminate between correct and incorrect answers.
Open-weight models can perform very well in terms of classification score, discrimination, and calibration using logit-based methods. Of the models tested, Gemma 4 31B is the most accurate (constrained and unconstrained), but the relatively small Qwen 3.5 9B achieved the best AUROC and ECE scores, with a slightly lower, but still competitive, accuracy and F1-macro of and .
The decision for which model and method to use likely depends on hardware, time or throughput, and privacy requirements. If there are no privacy requirements, commercial models like Jev can be a good option for quick, high-quality classification results; there is also no need for a GPU. If time and throughput are not a large concern, decoding full labels can be beneficial; depending on the label set, this can be more costly. By reducing computations to decision-only steps (see § 4.2), it is possible to save computation, achieving similar results to full decoding. If time is limited and/or there is a very high throughput required, reducing the label set to single tokens can save a lot of time, but it does come at the expense of accuracy. Depending on the model, it may be better to constrain the output to JSON or natural text; a small validation set may be used to test this and which trie-based method to use.
Limitations
Jev’s architecture and training data are not publicly documented, so we cannot be sure whether or not it was trained on the public datasets we have used. The same applies to the open-weight models.
In this study, we do not investigate the impact of temperature scaling on the calibration of our logit-based methods. Temperature scaling (Guo et al. [22]), fit on a small labelled validation set, is likely to improve calibration (ECE and top-label Brier score), while maintaining the same discriminative abilities.
All experiments are zero-shot only. Few-shot prompting may improve classification results further but come at some computational cost. Investigation of this trade-off could be interesting.
Our experiments use a single prompt per method-dataset pair, meaning we do not test for prompt variability. All datasets and prompts are also in English, meaning we cannot confidently draw conclusions on zero-shot LLM classification in other languages, although we expect the same conclusions to hold.
For the “Google length norm” methods, we do not tune the parameter. This could potentially affect classification accuracy, discrimination, and calibration. Sampling also only uses a single setting: 10 samples at temperature 0.5. Testing sampling with a higher resolution and different temperatures could improve calibration for this method.
Experiments use 16-bit precision across four GPU types. Despite greedy decoding and a batch size of 1, floating-point differences could cause small variations in results.
Acknowledgments
We acknowledge the IT Services at The University of Sheffield, utilising their High Performance Computing clusters in our experiments.
Additional Experimental Details
Dataset Usage
In all cases other than for Banking77, all dataset labels are left untouched. For Banking77, we changed “Refund_not_showing_up” to “refund_not_showing_up” and “reverted_card_payment?” to “reverted_card_payment” to be more consistent with the rest of the formatting of the labels, as these are known mistakes within the label set. We expect label outputs to exactly match those of the dataset to be considered correct. For all other datasets, we left label names untouched. For example, all FEVER labels remain in upper case and underscores or dots are left between categories and words for other datasets.
To ensure that out-of-memory errors never occur, we only use examples with an input text of 2048 tokens or under. Only FEVER and 20 Newsgroups are affected, with FEVER reduced from 9,999 examples to 9,975 and 20 Newsgroups reduced from 7,317 to 7,121.
Hardware
Experiments were run across 4 different GPU models: NVIDIA A100 with 40GB of VRAM, NVIDIA A100 with 80GB of VRAM, NVIDIA H100 with 80GB of VRAM, and the NVIDIA GH200 with 96GB HBM3.
Constrained vs. Unconstrained
Results By Model
| Method | Llama 3.2 1B | Llama 3.2 3B | Gemma 4 E2B | Gemma 4 E4B | Gemma 4 31B | Qwen2.5 7B | Qwen3 8B | Qwen3 14B | Qwen3 32B | Qwen3.5 4B | Qwen3.5 9B |
| Unconstrained (no conf) | 0.245 | 0.487 | 0.645 | 0.728 | 0.801 | 0.681 | 0.706 | 0.731 | 0.749 | 0.720 | 0.751 |
| Unconstrained (no conf ABC) | 0.162 | 0.341 | 0.552 | 0.677 | 0.795 | 0.607 | 0.619 | 0.702 | 0.716 | 0.658 | 0.710 |
| Unconstrained (10 samples) | 0.245 | 0.490 | 0.646 | 0.729 | 0.801 | 0.680 | 0.707 | 0.731 | 0.749 | 0.720 | 0.751 |
| Unconstrained + P(True) | 0.245 | 0.487 | 0.645 | 0.728 | 0.801 | 0.681 | 0.706 | 0.731 | 0.749 | 0.720 | 0.751 |
| Unconstrained JSON (no conf) | 0.273 | 0.515 | 0.656 | 0.725 | 0.800 | 0.669 | 0.709 | 0.733 | 0.741 | 0.718 | 0.752 |
| Unconstrained JSON (no conf ABC) | 0.160 | 0.421 | 0.604 | 0.694 | 0.798 | 0.604 | 0.658 | 0.715 | 0.720 | 0.669 | 0.713 |
| Unconstrained JSON verbal (pre) | 0.295 | 0.506 | 0.657 | 0.716 | 0.800 | 0.665 | 0.707 | 0.711 | 0.741 | 0.724 | 0.753 |
| Unconstrained JSON verbal (post) | 0.270 | 0.493 | 0.653 | 0.720 | 0.802 | 0.671 | 0.716 | 0.729 | 0.743 | 0.720 | 0.747 |
| Unconstrained JSON verbal (pre, Likert) | 0.185 | 0.514 | 0.661 | 0.724 | 0.803 | 0.674 | 0.701 | 0.725 | 0.737 | 0.714 | 0.750 |
| Unconstrained JSON verbal (post, Likert) | 0.258 | 0.494 | 0.657 | 0.719 | 0.802 | 0.674 | 0.712 | 0.730 | 0.747 | 0.718 | 0.746 |
| Unconstrained JSON verbal (pre, ABC) | 0.101 | 0.412 | 0.603 | 0.689 | 0.796 | 0.596 | 0.637 | 0.701 | 0.712 | 0.668 | 0.706 |
| Unconstrained JSON verbal (post, ABC) | 0.142 | 0.387 | 0.606 | 0.690 | 0.797 | 0.596 | 0.641 | 0.714 | 0.714 | 0.668 | 0.705 |
| Constrained (greedy) | 0.220 | 0.490 | 0.639 | 0.728 | 0.801 | 0.681 | 0.706 | 0.731 | 0.749 | 0.721 | 0.751 |
| Constrained (total likelihood) | 0.220 | 0.494 | 0.639 | 0.729 | 0.801 | 0.682 | 0.706 | 0.731 | 0.750 | 0.722 | 0.751 |
| Constrained (total likelihood, decision-only) | 0.220 | 0.494 | 0.639 | 0.728 | 0.801 | 0.682 | 0.706 | 0.731 | 0.750 | 0.721 | 0.751 |
| Constrained (length norm) | 0.182 | 0.484 | 0.636 | 0.727 | 0.801 | 0.682 | 0.707 | 0.731 | 0.749 | 0.721 | 0.752 |
| Constrained (length norm, decision-only) | 0.218 | 0.490 | 0.636 | 0.727 | 0.801 | 0.681 | 0.706 | 0.731 | 0.749 | 0.721 | 0.751 |
| Constrained (Google length norm) | 0.210 | 0.493 | 0.639 | 0.728 | 0.801 | 0.682 | 0.706 | 0.731 | 0.750 | 0.722 | 0.750 |
| Constrained (Google length norm, decision-only) | 0.220 | 0.494 | 0.639 | 0.729 | 0.801 | 0.682 | 0.706 | 0.731 | 0.750 | 0.722 | 0.751 |
| Constrained (probability tree) | 0.220 | 0.493 | 0.639 | 0.728 | 0.801 | 0.681 | 0.706 | 0.731 | 0.750 | 0.721 | 0.751 |
| Constrained (ABC) | 0.191 | 0.341 | 0.545 | 0.671 | 0.795 | 0.597 | 0.616 | 0.702 | 0.718 | 0.653 | 0.710 |
| Constrained JSON (greedy) | 0.228 | 0.514 | 0.636 | 0.715 | 0.802 | 0.665 | 0.703 | 0.721 | 0.742 | 0.711 | 0.744 |
| Constrained JSON (total likelihood) | 0.110 | 0.500 | 0.636 | 0.696 | 0.802 | 0.665 | 0.703 | 0.720 | 0.742 | 0.713 | 0.744 |
| Constrained JSON (total likelihood, decision-only) | 0.230 | 0.517 | 0.636 | 0.715 | 0.802 | 0.666 | 0.704 | 0.722 | 0.742 | 0.712 | 0.744 |
| Constrained JSON (length norm) | 0.115 | 0.462 | 0.636 | 0.508 | 0.802 | 0.664 | 0.640 | 0.681 | 0.743 | 0.712 | 0.744 |
| Constrained JSON (length norm, decision-only) | 0.209 | 0.506 | 0.636 | 0.713 | 0.802 | 0.664 | 0.704 | 0.721 | 0.740 | 0.711 | 0.744 |
| Constrained JSON (Google length norm) | 0.116 | 0.495 | 0.636 | 0.655 | 0.802 | 0.665 | 0.703 | 0.715 | 0.743 | 0.712 | 0.744 |
| Constrained JSON (Google length norm, decision-only) | 0.238 | 0.516 | 0.636 | 0.715 | 0.802 | 0.665 | 0.703 | 0.723 | 0.742 | 0.712 | 0.744 |
| Constrained JSON (probability tree) | 0.227 | 0.518 | 0.636 | 0.715 | 0.802 | 0.665 | 0.703 | 0.721 | 0.742 | 0.712 | 0.744 |
| Constrained JSON (ABC) | 0.114 | 0.422 | 0.595 | 0.683 | 0.798 | 0.603 | 0.656 | 0.715 | 0.718 | 0.668 | 0.714 |
Table 5. Accuracy (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.
| Method | Llama 3.2 1B | Llama 3.2 3B | Gemma 4 E2B | Gemma 4 E4B | Gemma 4 31B | Qwen2.5 7B | Qwen3 8B | Qwen3 14B | Qwen3 32B | Qwen3.5 4B | Qwen3.5 9B |
| Unconstrained (no conf) | 0.175 | 0.470 | 0.637 | 0.723 | 0.795 | 0.665 | 0.693 | 0.732 | 0.746 | 0.701 | 0.741 |
| Unconstrained (no conf ABC) | 0.134 | 0.315 | 0.546 | 0.681 | 0.789 | 0.588 | 0.603 | 0.695 | 0.710 | 0.648 | 0.703 |
| Unconstrained (10 samples) | 0.189 | 0.478 | 0.637 | 0.723 | 0.795 | 0.665 | 0.694 | 0.732 | 0.746 | 0.700 | 0.741 |
| Unconstrained + P(True) | 0.175 | 0.470 | 0.637 | 0.723 | 0.795 | 0.665 | 0.693 | 0.732 | 0.746 | 0.701 | 0.741 |
| Unconstrained JSON (no conf) | 0.235 | 0.503 | 0.661 | 0.721 | 0.793 | 0.663 | 0.701 | 0.738 | 0.739 | 0.697 | 0.748 |
| Unconstrained JSON (no conf ABC) | 0.098 | 0.389 | 0.609 | 0.698 | 0.790 | 0.590 | 0.650 | 0.710 | 0.715 | 0.657 | 0.703 |
| Unconstrained JSON verbal (pre) | 0.277 | 0.481 | 0.661 | 0.720 | 0.795 | 0.662 | 0.702 | 0.724 | 0.743 | 0.711 | 0.748 |
| Unconstrained JSON verbal (post) | 0.253 | 0.476 | 0.658 | 0.717 | 0.796 | 0.666 | 0.706 | 0.735 | 0.742 | 0.706 | 0.743 |
| Unconstrained JSON verbal (pre, Likert) | 0.135 | 0.495 | 0.664 | 0.722 | 0.796 | 0.664 | 0.693 | 0.730 | 0.739 | 0.691 | 0.746 |
| Unconstrained JSON verbal (post, Likert) | 0.230 | 0.479 | 0.663 | 0.717 | 0.796 | 0.672 | 0.705 | 0.734 | 0.747 | 0.707 | 0.742 |
| Unconstrained JSON verbal (pre, ABC) | 0.059 | 0.373 | 0.609 | 0.698 | 0.790 | 0.581 | 0.633 | 0.702 | 0.709 | 0.656 | 0.701 |
| Unconstrained JSON verbal (post, ABC) | 0.103 | 0.358 | 0.612 | 0.695 | 0.791 | 0.579 | 0.632 | 0.708 | 0.709 | 0.655 | 0.697 |
| Constrained (greedy) | 0.150 | 0.476 | 0.632 | 0.723 | 0.795 | 0.665 | 0.693 | 0.732 | 0.747 | 0.701 | 0.741 |
| Constrained (total likelihood) | 0.144 | 0.481 | 0.632 | 0.723 | 0.795 | 0.667 | 0.694 | 0.732 | 0.747 | 0.702 | 0.741 |
| Constrained (total likelihood, decision-only) | 0.148 | 0.481 | 0.632 | 0.723 | 0.795 | 0.666 | 0.694 | 0.732 | 0.747 | 0.702 | 0.741 |
| Constrained (length norm) | 0.116 | 0.469 | 0.629 | 0.722 | 0.795 | 0.667 | 0.695 | 0.733 | 0.747 | 0.703 | 0.743 |
| Constrained (length norm, decision-only) | 0.153 | 0.479 | 0.629 | 0.722 | 0.795 | 0.666 | 0.694 | 0.733 | 0.747 | 0.705 | 0.744 |
| Constrained (Google length norm) | 0.137 | 0.480 | 0.631 | 0.723 | 0.795 | 0.667 | 0.694 | 0.732 | 0.747 | 0.703 | 0.741 |
| Constrained (Google length norm, decision-only) | 0.150 | 0.482 | 0.632 | 0.723 | 0.795 | 0.666 | 0.694 | 0.732 | 0.747 | 0.702 | 0.741 |
| Constrained (probability tree) | 0.149 | 0.480 | 0.632 | 0.723 | 0.795 | 0.665 | 0.694 | 0.732 | 0.747 | 0.702 | 0.741 |
| Constrained (ABC) | 0.151 | 0.316 | 0.543 | 0.679 | 0.789 | 0.587 | 0.601 | 0.695 | 0.713 | 0.646 | 0.703 |
| Constrained JSON (greedy) | 0.150 | 0.507 | 0.640 | 0.706 | 0.795 | 0.654 | 0.687 | 0.725 | 0.738 | 0.687 | 0.731 |
| Constrained JSON (total likelihood) | 0.068 | 0.488 | 0.640 | 0.692 | 0.795 | 0.655 | 0.689 | 0.729 | 0.738 | 0.689 | 0.731 |
| Constrained JSON (total likelihood, decision-only) | 0.148 | 0.510 | 0.640 | 0.707 | 0.795 | 0.655 | 0.688 | 0.725 | 0.738 | 0.689 | 0.731 |
| Constrained JSON (length norm) | 0.059 | 0.435 | 0.640 | 0.510 | 0.795 | 0.653 | 0.646 | 0.702 | 0.739 | 0.690 | 0.732 |
| Constrained JSON (length norm, decision-only) | 0.161 | 0.497 | 0.640 | 0.706 | 0.795 | 0.654 | 0.689 | 0.725 | 0.738 | 0.691 | 0.734 |
| Constrained JSON (Google length norm) | 0.095 | 0.480 | 0.640 | 0.654 | 0.795 | 0.654 | 0.690 | 0.729 | 0.739 | 0.689 | 0.731 |
| Constrained JSON (Google length norm, decision-only) | 0.161 | 0.509 | 0.640 | 0.707 | 0.795 | 0.655 | 0.688 | 0.725 | 0.738 | 0.689 | 0.732 |
| Constrained JSON (probability tree) | 0.146 | 0.512 | 0.640 | 0.706 | 0.795 | 0.655 | 0.687 | 0.725 | 0.738 | 0.688 | 0.731 |
| Constrained JSON (ABC) | 0.056 | 0.391 | 0.602 | 0.694 | 0.790 | 0.592 | 0.650 | 0.710 | 0.714 | 0.659 | 0.704 |
Table 6. F1-macro (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.
| Method | Llama 3.2 1B | Llama 3.2 3B | Gemma 4 E2B | Gemma 4 E4B | Gemma 4 31B | Qwen2.5 7B | Qwen3 8B | Qwen3 14B | Qwen3 32B | Qwen3.5 4B | Qwen3.5 9B |
| Unconstrained (no conf) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 |
| Unconstrained (no conf ABC) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 |
| Unconstrained (10 samples) | 0.581 | 0.654 | 0.525 | 0.559 | 0.515 | 0.539 | 0.532 | 0.535 | 0.581 | 0.673 | 0.659 |
| Unconstrained + P(True) | 0.473 | 0.754 | 0.685 | 0.719 | 0.767 | 0.724 | 0.726 | 0.751 | 0.786 | 0.749 | 0.759 |
| Unconstrained JSON (no conf) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 |
| Unconstrained JSON (no conf ABC) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 |
| Unconstrained JSON verbal (pre) | 0.484 | 0.555 | 0.630 | 0.706 | 0.684 | 0.545 | 0.586 | 0.644 | 0.670 | 0.511 | 0.590 |
| Unconstrained JSON verbal (post) | 0.524 | 0.568 | 0.641 | 0.748 | 0.688 | 0.681 | 0.636 | 0.658 | 0.703 | 0.531 | 0.636 |
| Unconstrained JSON verbal (pre, Likert) | 0.531 | 0.550 | 0.581 | 0.657 | 0.642 | 0.594 | 0.555 | 0.641 | 0.618 | 0.576 | 0.587 |
| Unconstrained JSON verbal (post, Likert) | 0.522 | 0.579 | 0.603 | 0.686 | 0.673 | 0.620 | 0.582 | 0.653 | 0.682 | 0.559 | 0.668 |
| Unconstrained JSON verbal (pre, ABC) | 0.379 | 0.552 | 0.608 | 0.697 | 0.668 | 0.538 | 0.595 | 0.612 | 0.596 | 0.478 | 0.541 |
| Unconstrained JSON verbal (post, ABC) | 0.374 | 0.523 | 0.625 | 0.723 | 0.684 | 0.633 | 0.616 | 0.612 | 0.659 | 0.507 | 0.554 |
| Constrained (greedy) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 |
| Constrained (total likelihood) | 0.606 | 0.734 | 0.719 | 0.787 | 0.763 | 0.736 | 0.739 | 0.774 | 0.772 | 0.786 | 0.798 |
| Constrained (total likelihood, decision-only) | 0.605 | 0.733 | 0.720 | 0.787 | 0.764 | 0.735 | 0.738 | 0.774 | 0.771 | 0.786 | 0.797 |
| Constrained (length norm) | 0.693 | 0.748 | 0.739 | 0.776 | 0.706 | 0.738 | 0.744 | 0.787 | 0.772 | 0.781 | 0.792 |
| Constrained (length norm, decision-only) | 0.683 | 0.743 | 0.715 | 0.779 | 0.736 | 0.731 | 0.723 | 0.765 | 0.762 | 0.785 | 0.793 |
| Constrained (Google length norm) | 0.637 | 0.739 | 0.727 | 0.787 | 0.745 | 0.738 | 0.739 | 0.777 | 0.773 | 0.789 | 0.802 |
| Constrained (Google length norm, decision-only) | 0.607 | 0.734 | 0.719 | 0.786 | 0.763 | 0.735 | 0.736 | 0.775 | 0.771 | 0.786 | 0.798 |
| Constrained (probability tree) | 0.605 | 0.731 | 0.720 | 0.786 | 0.764 | 0.734 | 0.737 | 0.774 | 0.772 | 0.786 | 0.797 |
| Constrained (ABC) | 0.656 | 0.735 | 0.698 | 0.791 | 0.762 | 0.756 | 0.745 | 0.758 | 0.766 | 0.764 | 0.779 |
| Constrained JSON (greedy) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 |
| Constrained JSON (total likelihood) | 0.461 | 0.705 | 0.744 | 0.765 | 0.776 | 0.728 | 0.756 | 0.730 | 0.780 | 0.776 | 0.788 |
| Constrained JSON (total likelihood, decision-only) | 0.545 | 0.721 | 0.744 | 0.789 | 0.776 | 0.772 | 0.755 | 0.732 | 0.781 | 0.777 | 0.786 |
| Constrained JSON (length norm) | 0.496 | 0.743 | 0.736 | 0.762 | 0.724 | 0.735 | 0.770 | 0.696 | 0.709 | 0.745 | 0.766 |
| Constrained JSON (length norm, decision-only) | 0.612 | 0.729 | 0.733 | 0.778 | 0.746 | 0.721 | 0.736 | 0.706 | 0.753 | 0.761 | 0.779 |
| Constrained JSON (Google length norm) | 0.487 | 0.716 | 0.744 | 0.760 | 0.766 | 0.731 | 0.757 | 0.740 | 0.779 | 0.777 | 0.788 |
| Constrained JSON (Google length norm, decision-only) | 0.555 | 0.722 | 0.743 | 0.789 | 0.775 | 0.727 | 0.753 | 0.731 | 0.779 | 0.776 | 0.787 |
| Constrained JSON (probability tree) | 0.537 | 0.720 | 0.744 | 0.784 | 0.776 | 0.726 | 0.755 | 0.727 | 0.781 | 0.776 | 0.786 |
| Constrained JSON (ABC) | 0.620 | 0.729 | 0.733 | 0.782 | 0.772 | 0.744 | 0.762 | 0.776 | 0.786 | 0.764 | 0.776 |
Table 7. AUROC (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.
| Method | Llama 3.2 1B | Llama 3.2 3B | Gemma 4 E2B | Gemma 4 E4B | Gemma 4 31B | Qwen2.5 7B | Qwen3 8B | Qwen3 14B | Qwen3 32B | Qwen3.5 4B | Qwen3.5 9B |
| Unconstrained (no conf) | 0.755 | 0.513 | 0.355 | 0.272 | 0.199 | 0.319 | 0.294 | 0.269 | 0.251 | 0.280 | 0.249 |
| Unconstrained (no conf ABC) | 0.838 | 0.659 | 0.448 | 0.323 | 0.205 | 0.393 | 0.381 | 0.298 | 0.284 | 0.342 | 0.290 |
| Unconstrained (10 samples) | 0.495 | 0.386 | 0.344 | 0.253 | 0.196 | 0.302 | 0.281 | 0.257 | 0.231 | 0.221 | 0.201 |
| Unconstrained + P(True) | 0.331 | 0.323 | 0.333 | 0.225 | 0.187 | 0.273 | 0.266 | 0.229 | 0.190 | 0.179 | 0.158 |
| Unconstrained JSON (no conf) | 0.727 | 0.485 | 0.344 | 0.275 | 0.200 | 0.331 | 0.291 | 0.267 | 0.259 | 0.282 | 0.248 |
| Unconstrained JSON (no conf ABC) | 0.840 | 0.579 | 0.396 | 0.306 | 0.202 | 0.396 | 0.342 | 0.285 | 0.280 | 0.331 | 0.287 |
| Unconstrained JSON verbal (pre) | 0.538 | 0.394 | 0.249 | 0.216 | 0.152 | 0.254 | 0.215 | 0.236 | 0.188 | 0.238 | 0.209 |
| Unconstrained JSON verbal (post) | 0.470 | 0.434 | 0.301 | 0.210 | 0.146 | 0.207 | 0.214 | 0.230 | 0.174 | 0.227 | 0.203 |
| Unconstrained JSON verbal (pre, Likert) | 0.494 | 0.270 | 0.180 | 0.194 | 0.120 | 0.162 | 0.174 | 0.188 | 0.146 | 0.180 | 0.160 |
| Unconstrained JSON verbal (post, Likert) | 0.386 | 0.281 | 0.191 | 0.188 | 0.107 | 0.141 | 0.138 | 0.192 | 0.121 | 0.152 | 0.120 |
| Unconstrained JSON verbal (pre, ABC) | 0.682 | 0.479 | 0.277 | 0.237 | 0.156 | 0.316 | 0.295 | 0.246 | 0.266 | 0.320 | 0.247 |
| Unconstrained JSON verbal (post, ABC) | 0.764 | 0.485 | 0.314 | 0.241 | 0.151 | 0.274 | 0.286 | 0.239 | 0.202 | 0.283 | 0.245 |
| Constrained (greedy) | 0.780 | 0.510 | 0.361 | 0.272 | 0.199 | 0.319 | 0.294 | 0.269 | 0.251 | 0.279 | 0.249 |
| Constrained (total likelihood) | 0.356 | 0.238 | 0.329 | 0.228 | 0.192 | 0.265 | 0.267 | 0.242 | 0.204 | 0.143 | 0.138 |
| Constrained (total likelihood, decision-only) | 0.346 | 0.237 | 0.329 | 0.228 | 0.192 | 0.265 | 0.267 | 0.242 | 0.204 | 0.143 | 0.138 |
| Constrained (length norm) | 0.088 | 0.196 | 0.217 | 0.231 | 0.196 | 0.160 | 0.150 | 0.118 | 0.167 | 0.346 | 0.328 |
| Constrained (length norm, decision-only) | 0.140 | 0.094 | 0.312 | 0.198 | 0.187 | 0.217 | 0.243 | 0.223 | 0.164 | 0.066 | 0.053 |
| Constrained (Google length norm) | 0.268 | 0.175 | 0.320 | 0.214 | 0.190 | 0.250 | 0.260 | 0.236 | 0.191 | 0.103 | 0.107 |
| Constrained (Google length norm, decision-only) | 0.314 | 0.220 | 0.328 | 0.226 | 0.191 | 0.262 | 0.265 | 0.241 | 0.201 | 0.132 | 0.131 |
| Constrained (probability tree) | 0.335 | 0.236 | 0.329 | 0.228 | 0.192 | 0.266 | 0.267 | 0.242 | 0.204 | 0.142 | 0.138 |
| Constrained (ABC) | 0.357 | 0.189 | 0.414 | 0.270 | 0.199 | 0.316 | 0.345 | 0.270 | 0.217 | 0.125 | 0.148 |
| Constrained JSON (greedy) | 0.772 | 0.486 | 0.364 | 0.285 | 0.198 | 0.335 | 0.297 | 0.279 | 0.258 | 0.289 | 0.256 |
| Constrained JSON (total likelihood) | 0.352 | 0.216 | 0.340 | 0.232 | 0.190 | 0.278 | 0.260 | 0.233 | 0.170 | 0.151 | 0.142 |
| Constrained JSON (total likelihood, decision-only) | 0.335 | 0.208 | 0.340 | 0.236 | 0.190 | 0.278 | 0.260 | 0.232 | 0.170 | 0.150 | 0.142 |
| Constrained JSON (length norm) | 0.198 | 0.228 | 0.220 | 0.157 | 0.197 | 0.191 | 0.113 | 0.184 | 0.379 | 0.387 | 0.411 |
| Constrained JSON (length norm, decision-only) | 0.188 | 0.098 | 0.322 | 0.198 | 0.184 | 0.221 | 0.225 | 0.195 | 0.391 | 0.123 | 0.110 |
| Constrained JSON (Google length norm) | 0.273 | 0.145 | 0.332 | 0.216 | 0.187 | 0.262 | 0.248 | 0.225 | 0.138 | 0.112 | 0.106 |
| Constrained JSON (Google length norm, decision-only) | 0.304 | 0.193 | 0.338 | 0.233 | 0.190 | 0.274 | 0.257 | 0.230 | 0.163 | 0.140 | 0.134 |
| Constrained JSON (probability tree) | 0.332 | 0.206 | 0.340 | 0.237 | 0.190 | 0.278 | 0.260 | 0.234 | 0.170 | 0.149 | 0.141 |
| Constrained JSON (ABC) | 0.198 | 0.190 | 0.374 | 0.261 | 0.196 | 0.312 | 0.309 | 0.256 | 0.207 | 0.155 | 0.155 |
Table 8. ECE (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.
| Method | Llama 3.2 1B | Llama 3.2 3B | Gemma 4 E2B | Gemma 4 E4B | Gemma 4 31B | Qwen2.5 7B | Qwen3 8B | Qwen3 14B | Qwen3 32B | Qwen3.5 4B | Qwen3.5 9B |
| Unconstrained (no conf) | 0.755 | 0.513 | 0.355 | 0.272 | 0.199 | 0.319 | 0.294 | 0.269 | 0.251 | 0.280 | 0.249 |
| Unconstrained (no conf ABC) | 0.838 | 0.659 | 0.448 | 0.323 | 0.205 | 0.393 | 0.381 | 0.298 | 0.284 | 0.342 | 0.290 |
| Unconstrained (10 samples) | 0.456 | 0.372 | 0.344 | 0.254 | 0.196 | 0.302 | 0.282 | 0.258 | 0.231 | 0.226 | 0.206 |
| Unconstrained + P(True) | 0.308 | 0.305 | 0.335 | 0.232 | 0.187 | 0.277 | 0.267 | 0.231 | 0.198 | 0.206 | 0.182 |
| Unconstrained JSON (no conf) | 0.727 | 0.485 | 0.344 | 0.275 | 0.200 | 0.331 | 0.291 | 0.267 | 0.259 | 0.282 | 0.248 |
| Unconstrained JSON (no conf ABC) | 0.840 | 0.579 | 0.396 | 0.306 | 0.202 | 0.396 | 0.342 | 0.285 | 0.280 | 0.331 | 0.287 |
| Unconstrained JSON verbal (pre) | 0.505 | 0.398 | 0.270 | 0.229 | 0.162 | 0.306 | 0.248 | 0.268 | 0.214 | 0.256 | 0.221 |
| Unconstrained JSON verbal (post) | 0.439 | 0.441 | 0.307 | 0.222 | 0.158 | 0.238 | 0.233 | 0.263 | 0.196 | 0.244 | 0.213 |
| Unconstrained JSON verbal (pre, Likert) | 0.489 | 0.313 | 0.253 | 0.235 | 0.145 | 0.250 | 0.237 | 0.227 | 0.215 | 0.243 | 0.190 |
| Unconstrained JSON verbal (post, Likert) | 0.355 | 0.323 | 0.251 | 0.218 | 0.140 | 0.214 | 0.204 | 0.214 | 0.178 | 0.228 | 0.173 |
| Unconstrained JSON verbal (pre, ABC) | 0.594 | 0.469 | 0.293 | 0.246 | 0.167 | 0.352 | 0.305 | 0.277 | 0.287 | 0.331 | 0.258 |
| Unconstrained JSON verbal (post, ABC) | 0.738 | 0.491 | 0.316 | 0.252 | 0.162 | 0.298 | 0.297 | 0.268 | 0.225 | 0.295 | 0.256 |
| Constrained (greedy) | 0.780 | 0.510 | 0.361 | 0.272 | 0.199 | 0.319 | 0.294 | 0.269 | 0.251 | 0.279 | 0.249 |
| Constrained (total likelihood) | 0.297 | 0.258 | 0.329 | 0.231 | 0.192 | 0.270 | 0.268 | 0.243 | 0.208 | 0.179 | 0.167 |
| Constrained (total likelihood, decision-only) | 0.288 | 0.258 | 0.329 | 0.231 | 0.192 | 0.271 | 0.268 | 0.243 | 0.208 | 0.178 | 0.167 |
| Constrained (length norm) | 0.128 | 0.253 | 0.250 | 0.233 | 0.185 | 0.215 | 0.194 | 0.177 | 0.177 | 0.322 | 0.294 |
| Constrained (length norm, decision-only) | 0.151 | 0.199 | 0.311 | 0.208 | 0.188 | 0.239 | 0.247 | 0.226 | 0.183 | 0.155 | 0.142 |
| Constrained (Google length norm) | 0.216 | 0.227 | 0.320 | 0.219 | 0.190 | 0.259 | 0.261 | 0.237 | 0.198 | 0.164 | 0.153 |
| Constrained (Google length norm, decision-only) | 0.257 | 0.248 | 0.327 | 0.229 | 0.192 | 0.268 | 0.266 | 0.242 | 0.205 | 0.174 | 0.163 |
| Constrained (probability tree) | 0.278 | 0.258 | 0.329 | 0.231 | 0.192 | 0.271 | 0.268 | 0.243 | 0.208 | 0.178 | 0.167 |
| Constrained (ABC) | 0.274 | 0.186 | 0.410 | 0.270 | 0.199 | 0.309 | 0.339 | 0.269 | 0.226 | 0.189 | 0.183 |
| Constrained JSON (greedy) | 0.772 | 0.486 | 0.364 | 0.285 | 0.198 | 0.335 | 0.297 | 0.279 | 0.258 | 0.289 | 0.256 |
| Constrained JSON (total likelihood) | 0.237 | 0.262 | 0.338 | 0.243 | 0.191 | 0.283 | 0.261 | 0.240 | 0.189 | 0.190 | 0.177 |
| Constrained JSON (total likelihood, decision-only) | 0.270 | 0.252 | 0.338 | 0.239 | 0.191 | 0.283 | 0.261 | 0.238 | 0.188 | 0.189 | 0.177 |
| Constrained JSON (length norm) | 0.144 | 0.269 | 0.253 | 0.204 | 0.186 | 0.231 | 0.185 | 0.218 | 0.352 | 0.372 | 0.370 |
| Constrained JSON (length norm, decision-only) | 0.165 | 0.208 | 0.320 | 0.212 | 0.185 | 0.245 | 0.234 | 0.214 | 0.165 | 0.178 | 0.174 |
| Constrained JSON (Google length norm) | 0.176 | 0.232 | 0.330 | 0.240 | 0.188 | 0.270 | 0.250 | 0.231 | 0.174 | 0.174 | 0.164 |
| Constrained JSON (Google length norm, decision-only) | 0.250 | 0.244 | 0.336 | 0.237 | 0.190 | 0.279 | 0.258 | 0.236 | 0.185 | 0.185 | 0.174 |
| Constrained JSON (probability tree) | 0.264 | 0.252 | 0.338 | 0.240 | 0.191 | 0.282 | 0.261 | 0.240 | 0.188 | 0.189 | 0.177 |
| Constrained JSON (ABC) | 0.128 | 0.237 | 0.366 | 0.261 | 0.196 | 0.305 | 0.306 | 0.254 | 0.216 | 0.196 | 0.186 |
Table 9. Top-label Brier (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.
Prompts
Fully rendered prompt (FEVER, plaintext label, Gemma 4 31B chat template)
<bos><|turn>system
# CONTEXT
You are an expert fact-checker. You specialize in
analyzing claims with provided evidence and
deciding whether the evidence supports the
claim, refutes the claim, or there is not
enough evidence to make a clear decision.
# STYLE & TONE
Favor complete brevity. You need not respond with
any reasoning, just the answer.
# FORMAT
* You will be provided with the claim inside
<claim></claim> tags.
* Evidence will be provided within the
<evidence></evidence> tags.
* You will be provided with each set of evidence
inside <evidence_set></evidence_set> tags.
* Within the evidence tags, you will be provided
each piece of evidence inside <evidence_piece
title=[TITLE]></evidence_piece> tags, where
the title of the wiki pages article is
provided in the first tag.
* Respond with ONLY the answer from one of these
options:
- SUPPORTS
- REFUTES
- NOT ENOUGH INFO<turn|>
<|turn>user
# OBJECTIVE
Classify whether the provided evidence SUPPORTS or
REFUTES the given claim, or whether there is
NOT ENOUGH INFO. Rely ONLY on the evidence
provided and use any combination of the pieces
of evidence provided.
# INPUT
<claim>
Aruba is the only ABC Island.
</claim>
<evidence>
<evidence_set>
<evidence_piece title=ABC islands (Lesser
Antilles)>
The ABC islands are the three western-most
islands of the Leeward Antilles in the
Caribbean Sea that lie north of Falcón State,
Venezuela.
</evidence_piece>
</evidence_set>
<evidence_set>
<evidence_piece title=ABC islands (Lesser
Antilles)>
From west to east they are, in order Aruba,
Curaçao, and Bonaire.
</evidence_piece>
</evidence_set>
</evidence>
# ANSWER
Answer (choose between SUPPORTS, REFUTES, NOT ENOUGH
INFO):<turn|>
<|turn>model
<|channel>thought
<channel|>
Fully rendered prompt (Banking77, JSON verbal (post), Gemma 4 31B chat template)
<bos><|turn>system
# CONTEXT
You are an expert customer support analyst for
online banking. You specialize in analyzing
customer queries and classifying them into one of 77 intent classes.
STYLE & TONE
Favor complete brevity. You need not respond with reasoning, just the answer.
FORMAT
The customer query will be provided inside <query></query> tags.
Your response MUST be in JSON format, with the intent label FIRST and confidence SECOND.
The "label" key in your answer must be only ONE of the following intents:
activate_my_card
age_limit
...
The "confidence" key in your answer MUST be provided as a percentage to reflect your confidence in the label you are about to provide.
Return ONLY the JSON object. NO OTHER TEXT.
Example format: { "label": "[INTENT]", "confidence": "X%" } <turn|> <|turn>user
OBJECTIVE
Provide confidence about your ability to classify the intent of the customer query and classify the customer's intent with their query. Choose the most specific intent that best matches their query.
INPUT
<query> How do I locate my card? </query>
ANSWER
JSON output with the intent label FIRST and confidence SECOND ({"label": "[choose between activate_my_card, age_limit, ...]", "confidence": "[PERCENTAGE]%"}):<turn|> <|turn>model <|channel>thought <channel|>
**Fully rendered prompt (CLINC150, JSON verbal (pre, Likert), Gemma 4 31B chat template)**
<bos><|turn>system
CONTEXT
You are an expert virtual assistant analyst. You specialize in analyzing user queries and classifying them into one of 150 intent classes.
STYLE & TONE
Favor complete brevity. You need not respond with reasoning, just the answer.
FORMAT
The user query will be provided inside <query></query> tags.
Your response MUST be in JSON format, with confidence FIRST and the intent label SECOND.
The "confidence" key in your answer MUST be one of the following labels to reflect your confidence in the label you are about to provide:
Not at all confident
Slightly confident
Moderately confident
Very confident
Completely confident
The "label" key in your answer must be only ONE of the following intents:
accept_reservations
account_blocked
alarm
...
Example format: { "confidence": "Not at all confident" | "Slightly confident" | "Moderately confident" | "Very confident" | "Completely confident", "label": "[INTENT]" }<turn|> <|turn>user
OBJECTIVE
Classify the user's intent with their query. Choose the most specific intent that best matches their query. If the query does not belong to any listed intent, answer oos.
INPUT
<query> how would you say fly in italian </query>
ANSWER
JSON output with confidence FIRST and intent label SECOND ({"confidence": "[choose between Not at all confident, Slightly confident, Moderately confident, Very confident, Completely confident]", "label": "[choose between restaurant_reviews, nutrition_info, ...]"}):<turn|> <|turn>model <|channel>thought <channel|>
Fully rendered prompt (SST-5, plaintext label (P(True)), Gemma 4 31B chat template)
<bos><|turn>system
CONTEXT
You are an expert sentiment analysis analyst for movie reviews. You specialize in analyzing review excerpts and classifying them into one of 5 sentiment classes.
STYLE & TONE
Favor complete brevity. You need not respond with any reasoning, just the answer.
FORMAT
You will be provided with the instructions given to the annotator and the annotator's response.
You must respond with whether the annotator is correct or not ("True" or "False").
You must ONLY respond with True or False, no other text.<turn|> <|turn>user
OBJECTIVE
Judge whether the response provided by an annotator is correct (True or False). They were tasked to classify the sentiment of the review excerpt, choosing the sentiment that best matches the overall tone from the list of 5 valid sentiments.
INPUT & ANNOTATOR RESPONSE
"""
INPUT
CONTEXT
You are an expert sentiment analysis analyst for movie reviews. You specialize in analyzing review excerpts and classifying them into one of 5 sentiment classes.
STYLE & TONE
Favor complete brevity. You need not respond with any reasoning, just the answer.
FORMAT
The review excerpt will be provided inside <review></review> tags.
Respond with ONLY the answer from the list of valid sentiments provided:
very negative
negative
neutral
positive
very positive
OBJECTIVE
Classify the sentiment of the review excerpt. Choose the sentiment that best matches the overall tone.
INPUT
<review> no movement , no yuks , not much of anything . </review>
ANSWER
Answer (choose from: very negative, negative, neutral, positive, very positive):
RESPONSE
negative """
ANSWER
Correct? (True | False):<turn|> <|turn>model <|channel>thought <channel|>
Fully rendered prompt (20NG, JSON output (ABC, with prefili for logit method), Gemma 4 31B chat template)
<bos><|turn>system
CONTEXT
You are an expert text classification analyst for online discussion forums. You specialize in analyzing newsgroup posts and classifying them into one of 20 topic categories.
STYLE & TONE
Favor complete brevity. You need not respond with reasoning, just the answer.
FORMAT
The newsgroup post will be provided inside <post></post> tags.
Your response MUST be in JSON format, containing only the category label.
Respond with ONLY the answer from the list of valid categories provided:
A (alt.atheism)
B (comp.graphics)
C (comp.os.ms-windows.misc)
D (comp.sys.ibm.pc.hardware)
E (comp.sys.mac.hardware)
F (comp.windows.x)
G (misc.forsale)
H (rec.autos)
I (rec.motorcycles)
J (rec.sport.baseball)
K (rec.sport.hockey)
L (sci.crypt)
M (sci.electronics)
N (sci.med)
O (sci.space)
P (soc.religion.christian)
Q (talk.politics.guns)
R (talk.politics.mideast)
S (talk.politics.misc)
T (talk.religion.misc)
Example format: { "label": "[CATEGORY_LETTER]" }<turn|> <|turn>user
OBJECTIVE
Classify the newsgroup post into its topic category.
Choose the LETTER that represents the most
specific category that best matches the post's
content.
# INPUT
<post>
I am a little confused on all of the models of the
88-89 bonnevilles.
I have heard of the LE SE LSE SSE SSEI. Could
someone tell me the
differences are far as features or performance. I am
also curious to
know what the book value is for prefereably the 89
model. And how much
less than book value can you usually get them for.
In other words how
much are they in demand this time of year. I have
heard that the mid-spring
early summer is the best time to buy.
</post>
# ANSWER
JSON output containing only the category label
({"label": "choose between A, B, C, D, E, F,
G, H, I, J, K, L, M, N, O, P, Q, R, S,
T"}):<turn|>
<|turn>model
<|channel>thought
<|channel|>```json
{"label": "References
- [1]Anthropic. 2026. Prompting best practices. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags. Claude documentation, accessed 9 November 2026.
- [2]Yaara Artsi, Vera Sorin, Benjamin S Glicksberg, Panagiotis Korfiatis, Girish N Nadkarni, and Eyal Klang. 2025. Large language models in real-world clinical workflows: a systematic review of applications and implementation. Frontiers in Digital Health, 7:1659134.
- [3]Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, and 1 others. 2024. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782.arxiv.org/abs/2405.14782
- [4]Glenn W. Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3.doi.org/10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
- [5]Martin Juan José Bucher and Marco Martini. 2024. Fine-tuned’small’llms (still) significantly outperform zero-shot generative ai models in text classification. arXiv preprint arXiv:2406.08660.arxiv.org/abs/2406.08660
- [6]Michael Burnham, Kayla Kahn, Ryan Yang Wang, and Rachel X Peng. 2026. Political debate: Efficient zero-shot and few-shot classifiers for political text. Political Analysis, 34(3):329–343.DOI
- [7]Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The alternative annotator test for llm-as-a-judge: How to statistically justify replacing human annotators with llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 16051–16081.
- [8]Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd workshop on natural language processing for conversational AI, pages 38–45.arxiv.org/abs/2003.04807
- [9]Veronica Chatrath, Marcelo Lotif, and Shaina Raza. 2024. Fact or fiction? can llms be reliable annotators for political truths? arXiv preprint arXiv:2411.05775.arxiv.org/abs/2411.05775
- [10]Zhongpu Chen, Yinfeng Liu, Long Shi, Zhi-Jie Wang, Xingyan Chen, Yu Zhao, and Fuji Ren. 2025. Mde-val: Evaluating and enhancing markdown awareness in large language models. In Proceedings of the ACM on Web Conference 2025, pages 2981–2991.DOI
- [11]Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00904.arxiv.org/abs/2010.00904
- [12]Yixin Dong, Charlie F Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen. 2025. Xgrammar: Flexible and efficient structured generation engine for large language models. Proceedings of Machine Learning and Systems, 7.arxiv.org/abs/2411.15100
- [13]Ekaterina Fadeeva, Roman Vashurin, Akim Tsygvin, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. LM-polygraph: Uncertainty estimation for language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 446–461, Singapore. Association for Computational Linguistics.arxiv.org/abs/2311.07383
- [14]David Farr, Iain Cruickshank, Nico Manzonelli, Nicholas Clark, Kate Starbird, and Jevin West. 2024. Llm confidence evaluation measures in zero-shot css classification. arXiv preprint arXiv:2410.13047.arxiv.org/abs/2410.13047
- [16]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, and 1 others. 2026. Gemma 4 technical report. arXiv preprint arXiv:2607.02770.
- [17]Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 conference of the north American chapter of the association for computational linguistics: human language technologies (volume 1: long papers), pages 6577–6595.arxiv.org/abs/2311.08298
- [18]Jonathan Geuter, Andrew Cheng, and David Alvarez-Melis. 2026. Releasing the inner jev: A simple recipe for open decision models. arXiv preprint arXiv:2610.17205.arxiv.org/abs/2610.17205
- [19]Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120.DOI
- [21]Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannah Hajishirzi. 2025. Olmes: A standard for language model evaluations. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5020–5048.
- [22]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR.arxiv.org/abs/1706.04599
- [23]Chirag Gupta and Aaditya Ramdas. 2021. Top-label calibration and multiclass-to-binary reductions. arXiv preprint arXiv:2107.08353.arxiv.org/abs/2107.08353
- [24]Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136.
- [25]Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65–70.DOI
- [27]Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664.arxiv.org/abs/2302.09664
- [28]Sachin Kumar, Chan Young Park, and Yulia Tsvetkov. 2024. Gen-Z: Generative zero-shot text classification with contextualized label descriptions. In International Conference on Learning Representations, volume 2024, pages 40169–40192.arxiv.org/abs/2311.07115
- [29]Ken Lang. 1995. Newsweeder: Learning to filter net-news. In Machine learning proceedings 1995, pages 331–339. Elsevier.
- [30]Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.arxiv.org/abs/1909.02027
- [31]Xianzhi Li, Samuel Chan, Xiaodan Zhu, Yulong Pei, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. 2023. Are chatgpt and gpt-4 general-purpose solvers for financial text analytics? a study on several typical tasks. In Proceedings of the 2023 conference on empirical methods in natural language processing: industry track, pages 408–422.arxiv.org/abs/2305.05862
- [32]Yinheng Li and Justin Wagle. 2026. Llm-as-jev: Llms are already jev-style decision models – when and how to fine-tune them. Preprint, arXiv:2610.02076.arxiv.org/abs/2610.02076
- [33]Xiaou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. 2025. Uncertainty quantification and confidence calibration in large language models: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6107–6117.arxiv.org/abs/2503.15850
- [34]Scott Lundberg, Harsha Nori, and Marco Tulio Ribeiro. 2022. Guidance. https://github.com/guidance-ai/guidance.
- [35]Bar Meyuhas, Anat Bremler-Barr, and Tal Shapira. 2024. Iot device labeling using large language models. arXiv preprint arXiv:2403.01586.arxiv.org/abs/2403.01586
- [36]Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29.
- [37]Nzubechukwu C Ohalete, Kevin B Gittner, and Lauren M Matheny. 2025. Costar-a: A prompting framework for enhancing large language model performance on point-of-view questions. arXiv preprint arXiv:2510.12637.arxiv.org/abs/2510.12637
- [38]Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick, Nadia Polikarpova, and Loris D’Antoni. 2024. Grammar-aligned decoding. Advances in Neural Information Processing Systems, 37:24547–24568.arxiv.org/abs/2405.21047
- [40]Qwen Team. 2026. Qwen3.5. https://huggingface.co/collections/Qwen/qwen35. Hugging Face model collection. Models used: Qwen/Qwen3.5-4B and Qwen/Qwen3.5-9B. Accessed 2 November 2026.
- [41]Qwen Team, An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengeng Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
- [42]Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025b. Qwen2.5 technical report. Preprint, arXiv:2412.15115.
- [43]Hajar Sakai and Sarah S Lam. 2026. Large language models for health care text classification: systematic review. Jmir Ai, 5(1):e79202.DOI
- [44]Mario Sanz-Guerrero and Katharina von der Wense. 2025. Mitigating label length bias in large language models. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 1404–1420.
- [45]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.DOI
- [46]James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819.arxiv.org/abs/1803.05355
- [47]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442.