betaFeedback

WorksComputer Science

LLMs Are Already Confidence-Aware Zero-Shot Classifiers

Abstract

Zero-shot large language models (LLMs) are increasingly used as text classifiers, but their labels are only useful when we know to what extent we can trust them. We compare black-box methods (verbalised confidence and sampling) with white-box methods that read token probabilities (P(True), and multiple logit-scoring methods using constrained generation). We evaluate eleven open-weight LLMs (1B-32B parameters) on five classification datasets with 3 to 151 labels, measuring classification performance (accuracy, F1-macro), discrimination (AUROC), and calibration (ECE and Brier score). The choice of confidence method rarely impacts accuracy significantly, but logit-based scoring methods lead to much better discrimination (AUROC 0.74-0.77) than black-box methods (AUROC 0.57-0.66). Replacing labels with single-token identifiers preserves discrimination but costs 3-6 accuracy points on average. We also introduce decision-only scoring, which only scores positions where labels diverge in the prefix tree (trie). It reduces decode steps by 88-96% on the three largest label sets and performs comparably to full-label scoring. The best open-weight model-method configurations (Gemma 4 31B and Qwen 3.5 9B) are competitive with the commercial Jev 1.13.0 classifier.

Introduction

Large language models (LLMs), now most commonly used as chatbots or coding agents, have a range of other emergent abilities, including classification ((Wei et al., 2022)). As their performance as zero-shot classifiers across a range of domains has improved, LLMs have been increasingly considered for adoption across a number of high-stakes industries such as politics ([9, 6]), healthcare ([2, 43]), and business ([31]). With this, it is increasingly important to understand when we can trust their outputs, and to what extent, using the model’s confidence as an indicator.

While there is evidence that smaller fine-tuned language models with a classification head often outperform higher-parameter zero-shot LLMs ([5]), fine-tuning such a model is not always possible. Although zero-shot classification with LLMs is not necessarily an optimal solution, it provides more flexibility and can be much more cost-effective, and the gap between zero-shot and fine-tuned classifiers is decreasing. Zero-shot classification can be particularly beneficial in data-scarce environments ((Zhang et al., 2026)), where data annotation costs are high, and when label sets may change often, meaning new classifiers do not need to be trained after each change ([35]). Semi-automated data labelling with LLMs is also growing in popularity ([7, 19]); ensuring the quality of annotations, and good confidence calibration is particularly important in this setting, enabling efficient use of human annotators, without negatively impacting dataset quality ((Wang et al., 2024b)).

Confidence can be extracted from LLMs in multiple ways. Common strategies include asking the model to verbalise its confidence ([47]), sampling multiple responses with a given temperature ((Wang et al., 2022; Xiong et al., 2024)), asking whether its answer is correct ((Kadavath et al., 2022)), and scoring labels using token probabilities ([13, 27] (Zhao et al. 2021)). These confidence extraction methods are rarely compared under the same conditions. The majority of studies investigating confidence extraction focus on question answering (QA) tasks, where knowledge is the key cause of uncertainty. In classification tasks, this is not the case; here uncertainty is much more likely to arise from task ambiguity rather than knowledge issues. In classification problems, labels are often many tokens long and they share decoding paths, so aggregations must represent the full label. To aggregate label probabilities, outputs must be constrained but the model’s first token may be a preamble to the label rather than the answer itself (Wang et al., 2024a).

This work provides empirical evidence for the most effective ways of using decoder-only LLMs as zero-shot classifiers, placing particular importance on each method’s discriminative abilities and confidence calibration as well as pure classification performance (accuracy, F1-macro). We test using popular classification datasets with labels containing overlapping decoding paths to better test the full-label aggregation methods and ensure they are robust to edge cases; an example of label overlap in SST-5 with the Gemma 4 tokeniser would be: “very” “positive” and “very” “negative”. For this, we investigate a number of logit-score aggregation methods.

With the relatively recent release and popularity of Jev-style models, transformer-based classification models that provide a distribution over the whole label set, we use TypeSafe AI’s Jev (1.13.0)1 as a reference point. Zero-shot model-method combinations in this study are competitive with Jev 1.13.0, performing comparably in terms of accuracy and AUROC, but generally worse in terms of raw calibration.

Contributions:

  • We provide a systematic overview of zero-shot classification with LLMs, focusing on the discrimination and calibration across a range of models for classification tasks.

  • We introduce a simple scoring rule that only scores and length normalises over decision points in the decode trie, reducing the number of decode steps by 96% on Banking77, 92% on CLINC150, and 88% on 20NG (with the Gemma 4 tokeniser); these reductions in decode steps will not match exactly with the time saved due to overheads.

  • Experimental code is publicly released on GitHub: https://github.com/minieggz/llm-as-classifier.

Key Findings:

  • Zero-shot LLMs are natural classifiers that can discriminate relatively well, achieving a mean AUROC of up to 0.802 (Qwen 3.5 9B, constrained with Google length normalisation).

  • Reducing labels to single tokens often causes lower classification performance but retains discriminative abilities.

  • Verbalised confidence, or constrained decoding and the choice of scoring methods rarely statistically significantly impact the accuracy of zero-shot LLM classifiers.

  • Verbalised confidence (on a Likert-scale) achieves the highest calibration overall but is significantly worse in terms of discrimination; methods should not be chosen on calibration alone.

Related Work

Constrained Decoding

As modern LLMs are generally successful at following output constraints, it is common to obtain labels through regex or simple string matching ([3]). Frameworks such as Outlines ((Willard and Louf, 2023)), Guidance ([34]), and XGrammar ([12]) restrict output by programmatically conforming to a regular expression or grammar, masking disallowed tokens at each decoding step. While constrained decoding can ensure we get an answer that fits a specific format, it can distort the LLM’s output distribution ([38]).

Extracting Confidence from LLMs

Confidence can be extracted from LLMs with black-box methods, which have no access to model internals, or white-box methods which have total access.

Unconstrained approaches. Instruction tuning has been reported to degrade token probability calibration relative to base variants ((Kadavath et al., 2022)), motivating the use of verbal approaches ([47]). Confidence can be verbalised either on a Likert-scale or numerically ([47, 14]). Using an LLM-as-a-judge approach, where the model verbalises its confidence separately to the classification label is also reported to be effective ([33]).

Another way to calculate confidence is to run inference with a set temperature a number of times; the more times inference is run, the closer the output distribution should be to the model’s true answer distribution ((Wang et al., 2022; Xiong et al., 2024). The majority decision is chosen as the label, with the confidence equal to the proportion of decisions that were the majority label.

The P(True) method (Kadavath et al., 2022) uses the answer generated as input to asking the model whether it believes that answer is true or false. The log likelihoods of the first tokens are then softmaxed to obtain the probabilities.

Scoring multi-token labels. Many white-box methods focus on token-level probabilities [17]. Confidence is often simply represented by the likelihood of the whole sequence (calculated by summing the log probabilities of each token) [13, 27] or a length-normalised likelihood [21] (Zhao et al. 2021). These methods are mostly applied to QA datasets. This length normalisation is often simple token length. Wu et al. (2016) apply an alternative length normalisation penalty that has not yet been applied to multi-token label scoring for classification.

For classification tasks specifically, some approaches also normalise the likelihood based on a prior for each label, found by inspecting token probabilities after a blank input [44] (Zhao et al. 2021). Kumar et al. [28] flip this approach with Gen-Z by asking what the likelihood of the input is given each possible output label, removing the requirement of length normalisation due to a constant output size.

A common way to avoid multi-token scoring is to replace full labels with single-token identifiers (often letters), but LLMs can show systematic preferences for particular option identifiers (Zheng et al., 2024; Pezeshkpour and Hruschka, 2024). Despite this, some open-source Jev-style methods [18, 32] take this kind of approach.

TypeSafe AI’s Jev API model, trained using Reinforcement Learning for Calibrated Decisions (RLCD) (TypeSafe AI, 2026), supports 255 output classes. It returns a top class label along with a full probability distribution across all labels.

Methodology

In this study, we consider classification tasks with a static label set. Although each method can be used in cases where the label set is constantly changing, using a static label set simplifies the evaluation and enables us to analyse the performance differences at the class level (F1-macro).

Much of the prior work on short-sequence model confidence calibration focuses on the question answering (QA) task [47] (Xiong et al. 2024). QA relies heavily on recalling knowledge; the classification datasets we use do not rely on recall, so the model’s uncertainty reflects the difficulty of assigning a label to a given input, rather than how well the model “knows” a fact.

Datasets

We evaluate on text-classification datasets spanning a number of tasks, including topic classification, intent classification, sentiment detection, and fact-checking. We mainly focus on datasets that have labels of varying lengths and that contain overlapping token paths to investigate the issues of length normalisation and the difficulty of extracting full distributions across the label space. These datasets are: Banking77 [8], CLINC150 [30], 20 Newsgroups [29], and SST-5 [45]. FEVER [46], a fact-checking dataset, is used to evaluate cases where all labels diverge at the first token with labels “SUPPORTS”, “REFUTES”, and “NOT ENOUGH INFO”.

Banking77 and CLINC150 are both intent datasets and have the largest number of classes at 77 and 151 (including out-of-scope) respectively, with a number of overlapping tokens within the labels. 20 Newsgroups, a topic classification dataset, contains overlapping labels that are organised into topics and subtopics; labels with the same parent topic will therefore share decoding paths. SST-5 provides a case where the two labels that share token paths are total opposites (“very positive” and “very negative”).

Large Language Models

In this study, we use only open-weight language models available on HuggingFace, using the Transformers library (Wolf et al., 2020). We cover a range of model families and sizes (1B to 32B parameters). These models include Gemma 4 E2B, Gemma 4 E4B, and Gemma 4 31B [16]; Llama 3.2 1B and 3B (Grattafiori et al., 2024); Qwen 2.5 7B [42]; Qwen 3 8B, Qwen 3 14B, and Qwen 3 32B [41]; Qwen 3.5 4B, and Qwen 3.5 9B [40]. For all models, we are only testing their abilities without chain-of-thought; in most practical classification scenarios, chain-of-thought would significantly increase computational cost.

Decoding Methods

Unconstrained decoding represents a simpler chatbot setup where answers are extracted after generation using only the output text. With constrained methods, we access the individual token probabilities at each decode step.

Unconstrained Generation

Simple inference methods. For the unconstrained generation, we test a number of prompting strategies to extract the answer from the LLM in different formats. These formats include: 1) plaintext label only (“Unconstrained (no conf)”), 2) JSON format with only the label (“Unconstrained JSON (no conf)”), 3) JSON format with confidence before the label (“Unconstrained JSON verbal (pre)”), and 4) JSON format with confidence after the label (“Unconstrained JSON verbal (post)”). We also test single-token substitutions for labels – “ABC” in Table 1 – and Likert-scaled confidence instead of percentage (“Likert”). This Likert scale is “Not at all confident”, “Slightly confident”, “Moderately confident”, “Very confident”, and “Completely confident”, equating to 20%, 40%, 60%, 80%, and 100%.

MethodAccuracyF1-macroAUROCECEBrier
Unconstrained (no conf)0.724*0.715*0.5000.2760.276
Unconstrained (no conf ABC)0.6710.6630.5000.3290.329
Unconstrained (10 samples)0.724*0.715*0.5690.2540.255
Unconstrained + P(True)0.724*0.715*0.741*0.2270.235
Unconstrained JSON (no conf)0.723*0.718*0.5000.2770.277
Unconstrained JSON (no conf ABC)0.6860.6800.5000.3140.314
Unconstrained JSON verbal (pre)0.719*0.718*0.6180.2170.241*
Unconstrained JSON verbal (post)0.722*0.719*0.6580.2120.231*
Unconstrained JSON verbal (pre, Likert)0.721*0.716*0.6060.167*0.222*
Unconstrained JSON verbal (post, Likert)0.723*0.720*0.6360.150*0.202*
Unconstrained JSON verbal (pre, ABC)0.6790.6760.5920.2620.280
Unconstrained JSON verbal (post, ABC)0.6810.6750.6240.2480.263
Constrained (greedy)0.7230.714*0.5000.2770.277
Constrained (total likelihood)0.723*0.715*0.764*0.2230.232
Constrained (total likelihood, decision-only)0.723*0.715*0.764*0.2230.232
Constrained (length norm)0.723*0.715*0.759*0.213*0.227*
Constrained (length norm, decision-only)0.723*0.715*0.7540.185*0.211*
Constrained (Google length norm)0.723*0.715*0.764*0.2080.222
Constrained (Google length norm, decision-only)0.723*0.715*0.763*0.2200.229
Constrained (probability tree)0.723*0.714*0.763*0.2230.232
Constrained (ABC)0.6670.6620.758*0.2560.266
Constrained JSON (greedy)0.7150.7070.5000.2850.285
Constrained JSON (total likelihood)0.7130.7060.760*0.2220.234
Constrained JSON (total likelihood, decision-only)0.7160.7070.763*0.2220.234
Constrained JSON (length norm)0.6810.6790.7380.249*0.264*
Constrained JSON (length norm, decision-only)0.7150.7080.746*0.185*0.213*
Constrained JSON (Google length norm)0.7080.7020.760*0.2030.225
Constrained JSON (Google length norm, decision-only)0.7160.7080.762*0.2180.231
Constrained JSON (probability tree)0.7150.7070.762*0.2220.234
Constrained JSON (ABC)0.6830.6800.766*0.2470.254

Table 1. Mean score over all model–dataset pairs (excluding Llama 3.2 1B, Llama 3.2 3B). Bold indicates the best mean. *indicates a method is not significantly different from the method with the best mean rank (two-sided Wilcoxon signed-rank test paired over model-dataset pairs, Holm-adjusted p>0.05p > 0.05).

By using the JSON format for confidence extraction, we are using a structure that is easy to obtain both the answer and confidence from and that language models are often post-trained on for tool calling ([41] (Grattafiori et al. 2024)). For the plaintext label only, we use regular expressions to extract the largest label first (to prevent cases where “positive” may be matched before “very positive”). We again use regular expressions to extract the JSON object, greedily extracting everything from the first “{” to the final “}”. If there is more than one JSON object, this will be picked up as a generation error once parsed. We use the json module to parse JSON output.

Sampling with temperature. As an extension to simple plaintext labels, we obtain a confidence estimate by sampling a number of language model outputs using a given temperature score ((Wang et al., 2022; Xiong et al., 2024)). To test this method, we use a temperature of 0.5, which Kuhn et al. [27] found to be optimal for sampling-based uncertainty estimation (from a range of 0.25 to 1.5); we sample 10 times per data point. Any output errors here will

be removed completely, so if there are 2 errors, we will divide the number of occurrences of each label by 8 rather than 10. We do note that a resolution of 10% in confidence may impact the results. In all results, this method is referenced as “Unconstrained (10 samples)”.

Constrained Decoding

With constrained decoding, it is possible to obtain a probability distribution over the full label set. We constrain the decoding process using a trie data structure ([11, 32]).

Tree Construction. We represent the constrained decoding process using the trie data structure, using a depth-first approach in its generation. By using the trie, we can optimise the use of the KV-cache for efficient computation. At each decode step, we softmax over the whole vocabulary and then mask the output to only contain the valid next tokens.

A trie is constructed for each prompt and each prompt corresponds to an individual data point. At the end of each label, we calculate the joint likelihood of “end-of-answer” tokens. This may be only the EOS token for some models; in other instruction-tuned models, there are specialised end-of-turn tokens (such as <turn|> for Gemma 4). For the JSON output specifically, the tokens ” and ” are also added. In the trie, we represent the combined “end-of-answer” tokens as token ID “−1”. An example of the trie structure can be seen in Figure 1. In our experiments, we run trie construction for each model-dataset combination and label aggregation is performed on the saved tries afterwards. For JSON responses, we prefill the beginning of the JSON output (for example “{"label": "”). The exact prefill token sequence was obtained through three example outputs by each model.

Example trie from SST-5 (Gemma 4 31B)

Figure 1. Example trie from SST-5 (Gemma 4 31B). Each node shows the token with log-probability / probability, where the log-probability is over the whole vocabulary and probability is between siblings. Shaded tokens are decision tokens; dashed tokens are forced and are skipped by the decision-only and probability methods. For other datasets, non-decision tokens can be in the middle of labels.

Tree aggregation. For a label yy with tokens t1,…,tnt_1,\ldots,t_n, let ℓi=log⁡pθ(ti∣x,t<i)\ell_i = \log p_\theta(t_i \mid x,t_{<i}), given input xx. Most methods assign a score s(y)s(y), from which we obtain a distribution over all labels and confidence:

p^(y∣x)=exp⁡s(y)∑y′∈Yexp⁡s(y′),(1)\hat{p}(y \mid x) = \frac{\exp s(y)}{\sum_{y' \in\mathcal{Y}} \exp s(y')}, \tag*{(1)}
conf⁡(x)=max⁡y∈Yp^(y∣x).\operatorname{conf}(x) = \max_{y \in\mathcal{Y}} \hat{p}(y \mid x).

Some methods redefine p^(y∣x)\hat{p}(y \mid x) directly.

Greedy constrained decoding. To directly compare constrained generation against unconstrained generation, we simply use the greedy approach, assigning 100% confidence to the greedy-decoded output. In this instance p^(y∣x)=1\hat{p}(y \mid x) = 1 if the greedy path chooses label yy, otherwise p^(y∣x)=0\hat{p}(y \mid x) = 0.

Total log likelihood and length normalisation. The simplest scoring method is a total log likelihood sum for each label (sTL(y)s_{\mathrm{TL}}(y)) ([13]). A simple extension to this is to normalise the label likelihoods by label length in tokens (sLN(y)s_{\mathrm{LN}}(y)) ([21] (Zhao et al. 2021)). As well as simple token-length normalisation, we also test the length normalisation penalty proposed by Google for machine translation (sGLN(y)s_{\mathrm{GLN}}(y)) (Wu et al., 2016). These scoring rules (Equation (2)) are used in Equation (1) to obtain the distribution and confidence.

sTL(y)=∑i=1nℓi,sLN(y)=sTL(y)/n,sGLN(y)=sTL(y)/(5+n6)α,(2)\begin{aligned} s_{\mathrm{TL}}(y) &= \sum_{i=1}^{n} \ell_i,\\ s_{\mathrm{LN}}(y) &= s_{\mathrm{TL}}(y) / n,\\ s_{\mathrm{GLN}}(y) &= s_{\mathrm{TL}}(y) / \left(\frac{5+n}{6}\right)^\alpha, \tag*{(2)} \end{aligned}

where nn is the label sequence length in tokens and in our experiments, α=0.6\alpha= 0.6 following Vaswani et al. (2017).

Decision-only methods. Decoding each full label, particularly for datasets with large label sets, can be computationally expensive. We propose performing the same calculations introduced above, but applying them to only decision tokens. A decision token is one with siblings, meaning it was not the only choice available (highlighted in Figure 1). Let D(y)D(y) be the set of positions of the decision tokens on the path to label yy.

tokens on the path to label yy. The decision-only scoring functions are

sTLD(y)=∑i∈D(y)ℓi,sLND(y)=sTLD(y)/m,sGLND(y)=sTLD(y)/(5+m6)α,(3)\begin{aligned} s_{\mathrm{TL}}^{D}(y) &= \sum_{i \in D(y)} \ell_i,\\ s_{\mathrm{LN}}^{D}(y) &= s_{\mathrm{TL}}^{D}(y) / m,\\ s_{\mathrm{GLN}}^{D}(y) &= s_{\mathrm{TL}}^{D}(y) / \left(\frac{5+m}{6}\right)^\alpha, \tag*{(3)} \end{aligned}

where m=∣D(y)∣m = |D(y)| is the number of decision tokens on the path to yy, and these scores are used in Equation (1) as before. By removing forced tokens, as well as improving computational efficiency, we may also reduce the bias introduced by length normalisation.

Probability tree. We could not find any direct examples of probability trees being used to represent full-label decoding for classification. For the probability tree method, rather than softmaxing after all labels have been decoded, we softmax all sibling nodes at each step and multiply the probabilities of each token at the end:

p^(y∣x)=∏i=1npθ(ti∣x,t<i)∑c∈C(t<i)pθ(c∣x,t<i).(4)\hat{p}(y \mid x) = \prod_{i=1}^{n} \frac{p_\theta(t_i \mid x,t_{<i})}{\sum_{c \in\mathcal{C}(t_{<i})} p_\theta(c \mid x,t_{<i})}. \tag*{(4)}

where C(t<i)\mathcal{C}(t_{<i}) is the set of valid next tokens (siblings of tit_i) after the previous tokens t<it_{<i}. Figure 1 visualises the probability tree. When all labels diverge at the first token, the probability tree method reduces to a probability distribution over single-token identifiers. In constrained methods where we replace labels with single tokens (“Constrained ABC)” and “Constrained JSON (ABC)”), we use the probability tree.

The P(True) method also makes use of the probability tree to get the probability of “True” vs. “False” on already classified data points.

Evaluation

For every method and model-dataset pair we compute accuracy and F1-macro (classification performance), AUROC (discrimination) ([24]), and ECE ([36, 22]) and top-label Brier score (calibration) ([4, 23]) (Brier score applied to the confidence in the predicted label). We also investigate the number of label generation errors per model.

From these results, we report means over all model-dataset pairs for each method. We use the method with the highest mean rank for each metric as the reference and we compare every other method against it using a two-sided Wilcoxon signed-rank test (Wilcoxon, 1945) paired over model-dataset pairs, with Holm correction [25] within each metric (α=0.05\alpha= 0.05). Individual model results (Appendix § 11) are reported without tests, as n=5n = 5 cannot reach p<0.05p < 0.05.

We then compare the highest performing model-method combinations against Jev 1.13.0 using the same metrics.

Unconstrained evaluation edge cases. When evaluating unconstrained outputs, unparsable errors are treated as incorrect answers. In our F1-macro calculations, we assign invalid predictions to incorrect classes as evenly as possible (the gold label is excluded from this assignment) using a round-robin approach. In each round, we prioritise adding incorrect predictions to the largest class to cause a lower impact on the result. When a method does not provide a confidence score, or confidence is unparsable, we assume 100% certainty. For constrained outputs, this does not apply as it is impossible to obtain a label outside the label set.

Experimental Setup

Key Comparisons

In our experiments, we test a number of decoding methods on 5 datasets (Banking77, CLINC150, FEVER, SST-5, 20NG) using 11 models (see § 3.2). Our main comparisons are constrained vs. unconstrained output, single token output vs. full-label output, JSON format vs. plaintext output, verbal confidence vs. sampling confidence vs. logit-based confidence vs. post-hoc P(True)P(\mathrm{True}) confidence.

The different decoding methods we compare are described in § 4.

Prompting Methodology

When prompting the models, we follow the COSTAR-A [37] prompting style. This format includes “Context”, “Objective”, “Style”, “Tone”, “Audience”, “Response”, and “Answer”.

For our prompts, we adapt COSTAR-A as follows: style and tone are merged into a single section; audience is omitted; and response is replaced by a “Format” section that describes both the structure of the input and the permitted output labels. We also add an “Input” section containing the instance to be classified. The “Context”, “Style & Tone”, and “Format” sections are included in the system prompt, and the “Objective”, “Input”, and “Answer” sections form the user prompt which varies per data point.

With modern LLMs proficient in markdown [10] (Freeburg 2026) and XML [1], all prompts use the markdown format for the above sections and inputs are provided inside XML tags.

Example prompts can be found in Appendix § 12.

Model Settings

For all models, we use 16-bit precision and we use greedy decoding in all instances (other than for the sampling inference method). We use greedy decoding to provide near-deterministic results that may only differ due to hardware-based floating point calculation differences; we generate with a batch size of 1 to mitigate this further and eliminate padding differences.

Results & Discussion

Smaller models generally produce more generation errors. As shown in Figure 2, the number of generation errors generally increases as the number of model parameters decreases. While the general trend is clear from looking at all models, it is most visible within the Gemma 4 family of models where the error rate decreases from 0.22% with Gemma 4 E2B to 0.01% for the 31 billion parameter variant. The mid-sized Qwen 3 and Qwen 3.5 models, however, show that this is not always the case.

Rate of errors made by each model

Figure 2. Rate of errors made by each model (all unconstrained inference methods: plaintext, JSON with no confidence, verbal pre and post, Likert and percentage) plotted against number of model parameters (in billions) and grouped by model family.

Despite constrained decoding preventing generation errors, it can harm model performance. It is well-investigated that constrained decoding can often distort an LLM’s output distribution, in that it is forced to choose between a set of tokens where all of them have extremely low likelihoods (Wang et al., 2024a). The alternative formats the unconstrained models used include bullet pointing, prepending with “Answer: ”, and using markdown formatting. For the full results, see Table 3 and Table 4.

Banking77CLINC150FEVERSST-520NG
Llama 3.2 1B+1.66% / -11.59%+3.00% / -6.07%+1.13% / -0.19%+0.00% / -0.00%+0.10% / -0.13%
Llama 3.2 3B+0.55% / -0.39%+2.56% / -1.82%+0.59% / -0.15%+0.09% / -0.18%+0.21% / -0.13%
Gemma 4 E2B+0.16% / -0.03%+0.31% / -2.78%+0.00% / -0.10%+0.00% / -0.00%+0.11% / -0.83%
Gemma 4 E4B+0.03% / -0.03%+0.04% / -0.02%+0.00% / -0.00%+0.00% / -0.00%+0.00% / -0.00%
Gemma 4 31B+0.00% / -0.00%+0.04% / -0.00%+0.01% / -0.01%+0.00% / -0.05%+0.03% / -0.00%
Qwen2.5 7B+0.26% / -0.36%+0.05% / -0.05%+0.15% / -0.06%+0.05% / -0.00%+0.07% / -0.00%
Qwen3 8B+0.03% / -0.00%+0.02% / -0.05%+0.00% / -0.00%+0.00% / -0.05%+0.06% / -0.01%
Qwen3 14B+0.00% / -0.03%+0.15% / -0.09%+0.00% / -0.00%+0.00% / -0.00%+0.10% / -0.00%
Qwen3 32B+0.10% / -0.00%+0.05% / -0.09%+0.00% / -0.00%+0.05% / -0.00%+0.07% / -0.01%
Qwen3.5 4B+0.13% / -0.00%+0.04% / -0.02%+0.00% / -0.00%+0.05% / -0.00%+0.04% / -0.04%
Qwen3.5 9B+0.03% / -0.00%+0.00% / -0.02%+0.01% / -0.01%+0.05% / -0.00%+0.00% / -0.06%

Table 3. The percentage of data points on which constrained plaintext generation helps (+) (constrained output is correct, unconstrained output is either incorrect or unparsable) and hinders (−) (unconstrained output is correct, constrained output is incorrect) classification performance by model and dataset. Both constrained and unconstrained generation use greedy decoding.

Banking77CLINC150FEVERSST-520NG
Llama 3.2 1B+4.22% / -20.13%+18.20% / -8.62%+32.25% / -35.92%+11.22% / -24.75%+3.93% / -3.13%
Llama 3.2 3B+2.63% / -1.40%+3.11% / -3.56%+4.45% / -1.48%+2.44% / -3.57%+2.09% / -5.25%
Gemma 4 E2B+0.75% / -0.91%+0.58% / -0.91%+1.01% / -7.62%+3.35% / -5.29%+1.24% / -2.27%
Gemma 4 E4B+0.29% / -0.49%+0.62% / -3.98%+0.33% / -1.00%+1.49% / -1.86%+0.59% / -1.28%
Gemma 4 31B+0.29% / -0.19%+0.09% / -0.13%+0.11% / -0.06%+1.49% / -0.81%+0.10% / -0.15%
Qwen2.5 7B+1.92% / -2.79%+3.56% / -3.80%+0.93% / -0.86%+1.76% / -2.81%+1.80% / -1.57%
Qwen3 8B+2.11% / -1.17%+0.91% / -3.15%+0.33% / -0.78%+4.66% / -5.02%+1.22% / -2.36%
Qwen3 14B+0.81% / -1.07%+0.89% / -3.27%+0.11% / -1.24%+7.47% / -8.91%+1.33% / -2.37%
Qwen3 32B+1.30% / -1.33%+1.95% / -2.35%+0.83% / -0.76%+3.26% / -2.13%+1.57% / -1.90%
Qwen3.5 4B+1.17% / -1.66%+1.55% / -2.07%+0.16% / -1.04%+2.08% / -3.35%+1.67% / -1.85%
Qwen3.5 9B+0.78% / -0.94%+2.16% / -1.27%+0.05% / -2.28%+3.98% / -6.56%+1.24% / -1.29%

Table 4. The percentage of data points on which constrained JSON generation helps (+) (constrained output is correct, unconstrained output is either incorrect or unparsable) and hinders (−) (unconstrained output is correct, constrained output is incorrect) classification performance by model and dataset. Both constrained and unconstrained generation use greedy decoding.

The confidence extraction method rarely significantly impacts classification performance. Post-hoc confidence methods like P(True) do not change the accuracy at all, as they are extensions of greedy inference with no confidence.

Other greedy generation methods all use JSON-formatted outputs, and the accuracy is only statistically significantly worse than the best performing method (unconstrained output with no confidence) when labels are reduced to single-token replacements. The averaged constrained JSON methods perform statistically worse than the best performing method, but some of the highest performing model-method combinations do use constrained JSON output, indicating that the preferred format is model-dependent (see Table 2).

ModelMethodAccuracyF1-macroAUROCECEBrier
Gemma 4 31BUnconstrained JSON verbal (pre, Likert)0.8030.7960.6420.1200.145
Gemma 4 31BUnconstrained JSON verbal (post, Likert)0.8020.7960.6730.1070.140
Qwen 3.5 9BConstrained (Google length norm)0.7500.7410.8020.1070.153
Qwen 3.5 9BConstrained (length norm, decision-only)0.7510.7440.7930.0530.142
Gemma 4 31BConstrained JSON (Google length norm, decision-only)0.8020.7950.7750.1900.190
Gemma 4 31BConstrained JSON (ABC)0.7980.7900.7720.1960.196
Jev 1.13.0Jev0.7880.7800.8010.0860.134

Table 2. Best model–method combinations for each metric, best constrained method (accuracy), best ABC-logit method (accuracy), compared against Jev 1.13.0. Scores are means across all datasets.

In terms of raw calibration, verbal confidence performs well. Likert-scale confidence is better calibrated but leads to worse discriminative abilities. For verbal confidence, asking models for JSON output with answer first and confidence second performed the best in terms of ECE and top-label Brier score (shown in Table 1). While asking for confidence in a Likert scale provides better raw calibration, it leads to a poorer AUROC, most likely due to the significantly lower resolution of the Likert scale compared to percentage output.

Methods that reduce the labels to single tokens often lead to reductions in accuracy and calibration, but perform well discriminatively and are more computationally efficient. Overall (see Table 1), all methods that reduce labels to single tokens perform significantly worse than the highest ranking method on average in terms of accuracy, F1-macro, ECE and top-label Brier score. While the accuracy is lower, they perform well on discrimination (AUROC) with “Constrained JSON (ABC)” achieving the highest average score. The constrained single-token methods also require only one decode step per data point, making these methods quick and resource-efficient when compared to methods requiring the full decoding of each label.

Tree-based decoding and logit scoring lead to significantly better classifiers in terms of discrimination. As shown in Table 1, the constrained logit-based methods perform well, with only two trie-based methods (excluding “greedy”) performing statistically significantly worse on average than the best performing method “Constrained JSON (ABC)” in terms of AUROC. While calibration scores in logit-based methods are not quite

as low as some of the verbalised confidence methods overall, the best performing model and method combination for ECE is Qwen 3.5 9B constrained with length normalisation (decision tokens only). Calibration scores could be improved using a small validation set and temperature scaling [22].

The P(True) method, which is also a logit-based method, is not statistically significantly worse than other logit-based methods in terms of discrimination and maintains the benefits of unconstrained generation for classification performance. Using P(True), however, does not produce a confidence distribution over all labels as other logit-scoring methods do.

Open-weight instruction-tuned models are competitive with Jev 1.13.0. Of the best performing methods in Table 2, Jev 1.13.0 only wins on the top-label Brier score but no model-method combination beats Jev on classification performance and discrimination combined. The model-method combination with the highest accuracy and F1-macro is Gemma 4 31B using unconstrained JSON generation with Likert-scale confidence; this does come at a large cost in discrimination. Gemma 4 31B constrained with JSON output and Google length normalisation on decision tokens only, as well as Gemma 4 31B with constrained single-token JSON output, achieves an accuracy and F1-macro higher than Jev, but performs worse in terms of AUROC, ECE, and top-label Brier. The best performing Qwen 3.5 9B method combinations perform similarly or better in terms of AUROC but do lose on raw classification performance. While the Jev API may be a viable option for those without extensive privacy concerns, open-weight alternatives with no extra training are competitive.

Conclusion

Open-weight zero-shot LLMs are effective classifiers that are competitive with current commercial classification models in terms of accuracy and discrimination. While the raw calibration is not always as strong, it could likely be improved with temperature scaling (Guo et al. [22]). Unconstrained outputs using verbalised confidence have been shown to perform well in terms of accuracy, but methods that use logit scores are shown to be significantly better at providing confidence scores that discriminate between correct and incorrect answers.

Open-weight models can perform very well in terms of classification score, discrimination, and calibration using logit-based methods. Of the models tested, Gemma 4 31B is the most accurate (constrained and unconstrained), but the relatively small Qwen 3.5 9B achieved the best AUROC and ECE scores, with a slightly lower, but still competitive, accuracy and F1-macro of ≈0.75\approx0.75 and ≈0.74\approx0.74.

The decision for which model and method to use likely depends on hardware, time or throughput, and privacy requirements. If there are no privacy requirements, commercial models like Jev can be a good option for quick, high-quality classification results; there is also no need for a GPU. If time and throughput are not a large concern, decoding full labels can be beneficial; depending on the label set, this can be more costly. By reducing computations to decision-only steps (see § 4.2), it is possible to save computation, achieving similar results to full decoding. If time is limited and/or there is a very high throughput required, reducing the label set to single tokens can save a lot of time, but it does come at the expense of accuracy. Depending on the model, it may be better to constrain the output to JSON or natural text; a small validation set may be used to test this and which trie-based method to use.

Limitations

Jev’s architecture and training data are not publicly documented, so we cannot be sure whether or not it was trained on the public datasets we have used. The same applies to the open-weight models.

In this study, we do not investigate the impact of temperature scaling on the calibration of our logit-based methods. Temperature scaling (Guo et al. [22]), fit on a small labelled validation set, is likely to improve calibration (ECE and top-label Brier score), while maintaining the same discriminative abilities.

All experiments are zero-shot only. Few-shot prompting may improve classification results further but come at some computational cost. Investigation of this trade-off could be interesting.

Our experiments use a single prompt per method-dataset pair, meaning we do not test for prompt variability. All datasets and prompts are also in English, meaning we cannot confidently draw conclusions on zero-shot LLM classification in other languages, although we expect the same conclusions to hold.

For the “Google length norm” methods, we do not tune the α\alpha parameter. This could potentially affect classification accuracy, discrimination, and calibration. Sampling also only uses a single setting: 10 samples at temperature 0.5. Testing sampling with a higher resolution and different temperatures could improve calibration for this method.

Experiments use 16-bit precision across four GPU types. Despite greedy decoding and a batch size of 1, floating-point differences could cause small variations in results.

Acknowledgments

We acknowledge the IT Services at The University of Sheffield, utilising their High Performance Computing clusters in our experiments.

Additional Experimental Details

Dataset Usage

In all cases other than for Banking77, all dataset labels are left untouched. For Banking77, we changed “Refund_not_showing_up” to “refund_not_showing_up” and “reverted_card_payment?” to “reverted_card_payment” to be more consistent with the rest of the formatting of the labels, as these are known mistakes within the label set. We expect label outputs to exactly match those of the dataset to be considered correct. For all other datasets, we left label names untouched. For example, all FEVER labels remain in upper case and underscores or dots are left between categories and words for other datasets.

To ensure that out-of-memory errors never occur, we only use examples with an input text of 2048 tokens or under. Only FEVER and 20 Newsgroups are affected, with FEVER reduced from 9,999 examples to 9,975 and 20 Newsgroups reduced from 7,317 to 7,121.

Hardware

Experiments were run across 4 different GPU models: NVIDIA A100 with 40GB of VRAM, NVIDIA A100 with 80GB of VRAM, NVIDIA H100 with 80GB of VRAM, and the NVIDIA GH200 with ∼\sim96GB HBM3.

Constrained vs. Unconstrained

Results By Model

MethodLlama 3.2 1BLlama 3.2 3BGemma 4 E2BGemma 4 E4BGemma 4 31BQwen2.5 7BQwen3 8BQwen3 14BQwen3 32BQwen3.5 4BQwen3.5 9B
Unconstrained (no conf)0.2450.4870.6450.7280.8010.6810.7060.7310.7490.7200.751
Unconstrained (no conf ABC)0.1620.3410.5520.6770.7950.6070.6190.7020.7160.6580.710
Unconstrained (10 samples)0.2450.4900.6460.7290.8010.6800.7070.7310.7490.7200.751
Unconstrained + P(True)0.2450.4870.6450.7280.8010.6810.7060.7310.7490.7200.751
Unconstrained JSON (no conf)0.2730.5150.6560.7250.8000.6690.7090.7330.7410.7180.752
Unconstrained JSON (no conf ABC)0.1600.4210.6040.6940.7980.6040.6580.7150.7200.6690.713
Unconstrained JSON verbal (pre)0.2950.5060.6570.7160.8000.6650.7070.7110.7410.7240.753
Unconstrained JSON verbal (post)0.2700.4930.6530.7200.8020.6710.7160.7290.7430.7200.747
Unconstrained JSON verbal (pre, Likert)0.1850.5140.6610.7240.8030.6740.7010.7250.7370.7140.750
Unconstrained JSON verbal (post, Likert)0.2580.4940.6570.7190.8020.6740.7120.7300.7470.7180.746
Unconstrained JSON verbal (pre, ABC)0.1010.4120.6030.6890.7960.5960.6370.7010.7120.6680.706
Unconstrained JSON verbal (post, ABC)0.1420.3870.6060.6900.7970.5960.6410.7140.7140.6680.705
Constrained (greedy)0.2200.4900.6390.7280.8010.6810.7060.7310.7490.7210.751
Constrained (total likelihood)0.2200.4940.6390.7290.8010.6820.7060.7310.7500.7220.751
Constrained (total likelihood, decision-only)0.2200.4940.6390.7280.8010.6820.7060.7310.7500.7210.751
Constrained (length norm)0.1820.4840.6360.7270.8010.6820.7070.7310.7490.7210.752
Constrained (length norm, decision-only)0.2180.4900.6360.7270.8010.6810.7060.7310.7490.7210.751
Constrained (Google length norm)0.2100.4930.6390.7280.8010.6820.7060.7310.7500.7220.750
Constrained (Google length norm, decision-only)0.2200.4940.6390.7290.8010.6820.7060.7310.7500.7220.751
Constrained (probability tree)0.2200.4930.6390.7280.8010.6810.7060.7310.7500.7210.751
Constrained (ABC)0.1910.3410.5450.6710.7950.5970.6160.7020.7180.6530.710
Constrained JSON (greedy)0.2280.5140.6360.7150.8020.6650.7030.7210.7420.7110.744
Constrained JSON (total likelihood)0.1100.5000.6360.6960.8020.6650.7030.7200.7420.7130.744
Constrained JSON (total likelihood, decision-only)0.2300.5170.6360.7150.8020.6660.7040.7220.7420.7120.744
Constrained JSON (length norm)0.1150.4620.6360.5080.8020.6640.6400.6810.7430.7120.744
Constrained JSON (length norm, decision-only)0.2090.5060.6360.7130.8020.6640.7040.7210.7400.7110.744
Constrained JSON (Google length norm)0.1160.4950.6360.6550.8020.6650.7030.7150.7430.7120.744
Constrained JSON (Google length norm, decision-only)0.2380.5160.6360.7150.8020.6650.7030.7230.7420.7120.744
Constrained JSON (probability tree)0.2270.5180.6360.7150.8020.6650.7030.7210.7420.7120.744
Constrained JSON (ABC)0.1140.4220.5950.6830.7980.6030.6560.7150.7180.6680.714

Table 5. Accuracy (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.

MethodLlama 3.2 1BLlama 3.2 3BGemma 4 E2BGemma 4 E4BGemma 4 31BQwen2.5 7BQwen3 8BQwen3 14BQwen3 32BQwen3.5 4BQwen3.5 9B
Unconstrained (no conf)0.1750.4700.6370.7230.7950.6650.6930.7320.7460.7010.741
Unconstrained (no conf ABC)0.1340.3150.5460.6810.7890.5880.6030.6950.7100.6480.703
Unconstrained (10 samples)0.1890.4780.6370.7230.7950.6650.6940.7320.7460.7000.741
Unconstrained + P(True)0.1750.4700.6370.7230.7950.6650.6930.7320.7460.7010.741
Unconstrained JSON (no conf)0.2350.5030.6610.7210.7930.6630.7010.7380.7390.6970.748
Unconstrained JSON (no conf ABC)0.0980.3890.6090.6980.7900.5900.6500.7100.7150.6570.703
Unconstrained JSON verbal (pre)0.2770.4810.6610.7200.7950.6620.7020.7240.7430.7110.748
Unconstrained JSON verbal (post)0.2530.4760.6580.7170.7960.6660.7060.7350.7420.7060.743
Unconstrained JSON verbal (pre, Likert)0.1350.4950.6640.7220.7960.6640.6930.7300.7390.6910.746
Unconstrained JSON verbal (post, Likert)0.2300.4790.6630.7170.7960.6720.7050.7340.7470.7070.742
Unconstrained JSON verbal (pre, ABC)0.0590.3730.6090.6980.7900.5810.6330.7020.7090.6560.701
Unconstrained JSON verbal (post, ABC)0.1030.3580.6120.6950.7910.5790.6320.7080.7090.6550.697
Constrained (greedy)0.1500.4760.6320.7230.7950.6650.6930.7320.7470.7010.741
Constrained (total likelihood)0.1440.4810.6320.7230.7950.6670.6940.7320.7470.7020.741
Constrained (total likelihood, decision-only)0.1480.4810.6320.7230.7950.6660.6940.7320.7470.7020.741
Constrained (length norm)0.1160.4690.6290.7220.7950.6670.6950.7330.7470.7030.743
Constrained (length norm, decision-only)0.1530.4790.6290.7220.7950.6660.6940.7330.7470.7050.744
Constrained (Google length norm)0.1370.4800.6310.7230.7950.6670.6940.7320.7470.7030.741
Constrained (Google length norm, decision-only)0.1500.4820.6320.7230.7950.6660.6940.7320.7470.7020.741
Constrained (probability tree)0.1490.4800.6320.7230.7950.6650.6940.7320.7470.7020.741
Constrained (ABC)0.1510.3160.5430.6790.7890.5870.6010.6950.7130.6460.703
Constrained JSON (greedy)0.1500.5070.6400.7060.7950.6540.6870.7250.7380.6870.731
Constrained JSON (total likelihood)0.0680.4880.6400.6920.7950.6550.6890.7290.7380.6890.731
Constrained JSON (total likelihood, decision-only)0.1480.5100.6400.7070.7950.6550.6880.7250.7380.6890.731
Constrained JSON (length norm)0.0590.4350.6400.5100.7950.6530.6460.7020.7390.6900.732
Constrained JSON (length norm, decision-only)0.1610.4970.6400.7060.7950.6540.6890.7250.7380.6910.734
Constrained JSON (Google length norm)0.0950.4800.6400.6540.7950.6540.6900.7290.7390.6890.731
Constrained JSON (Google length norm, decision-only)0.1610.5090.6400.7070.7950.6550.6880.7250.7380.6890.732
Constrained JSON (probability tree)0.1460.5120.6400.7060.7950.6550.6870.7250.7380.6880.731
Constrained JSON (ABC)0.0560.3910.6020.6940.7900.5920.6500.7100.7140.6590.704

Table 6. F1-macro (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.

MethodLlama 3.2 1BLlama 3.2 3BGemma 4 E2BGemma 4 E4BGemma 4 31BQwen2.5 7BQwen3 8BQwen3 14BQwen3 32BQwen3.5 4BQwen3.5 9B
Unconstrained (no conf)0.5000.5000.5000.5000.5000.5000.5000.5000.5000.5000.500
Unconstrained (no conf ABC)0.5000.5000.5000.5000.5000.5000.5000.5000.5000.5000.500
Unconstrained (10 samples)0.5810.6540.5250.5590.5150.5390.5320.5350.5810.6730.659
Unconstrained + P(True)0.4730.7540.6850.7190.7670.7240.7260.7510.7860.7490.759
Unconstrained JSON (no conf)0.5000.5000.5000.5000.5000.5000.5000.5000.5000.5000.500
Unconstrained JSON (no conf ABC)0.5000.5000.5000.5000.5000.5000.5000.5000.5000.5000.500
Unconstrained JSON verbal (pre)0.4840.5550.6300.7060.6840.5450.5860.6440.6700.5110.590
Unconstrained JSON verbal (post)0.5240.5680.6410.7480.6880.6810.6360.6580.7030.5310.636
Unconstrained JSON verbal (pre, Likert)0.5310.5500.5810.6570.6420.5940.5550.6410.6180.5760.587
Unconstrained JSON verbal (post, Likert)0.5220.5790.6030.6860.6730.6200.5820.6530.6820.5590.668
Unconstrained JSON verbal (pre, ABC)0.3790.5520.6080.6970.6680.5380.5950.6120.5960.4780.541
Unconstrained JSON verbal (post, ABC)0.3740.5230.6250.7230.6840.6330.6160.6120.6590.5070.554
Constrained (greedy)0.5000.5000.5000.5000.5000.5000.5000.5000.5000.5000.500
Constrained (total likelihood)0.6060.7340.7190.7870.7630.7360.7390.7740.7720.7860.798
Constrained (total likelihood, decision-only)0.6050.7330.7200.7870.7640.7350.7380.7740.7710.7860.797
Constrained (length norm)0.6930.7480.7390.7760.7060.7380.7440.7870.7720.7810.792
Constrained (length norm, decision-only)0.6830.7430.7150.7790.7360.7310.7230.7650.7620.7850.793
Constrained (Google length norm)0.6370.7390.7270.7870.7450.7380.7390.7770.7730.7890.802
Constrained (Google length norm, decision-only)0.6070.7340.7190.7860.7630.7350.7360.7750.7710.7860.798
Constrained (probability tree)0.6050.7310.7200.7860.7640.7340.7370.7740.7720.7860.797
Constrained (ABC)0.6560.7350.6980.7910.7620.7560.7450.7580.7660.7640.779
Constrained JSON (greedy)0.5000.5000.5000.5000.5000.5000.5000.5000.5000.5000.500
Constrained JSON (total likelihood)0.4610.7050.7440.7650.7760.7280.7560.7300.7800.7760.788
Constrained JSON (total likelihood, decision-only)0.5450.7210.7440.7890.7760.7720.7550.7320.7810.7770.786
Constrained JSON (length norm)0.4960.7430.7360.7620.7240.7350.7700.6960.7090.7450.766
Constrained JSON (length norm, decision-only)0.6120.7290.7330.7780.7460.7210.7360.7060.7530.7610.779
Constrained JSON (Google length norm)0.4870.7160.7440.7600.7660.7310.7570.7400.7790.7770.788
Constrained JSON (Google length norm, decision-only)0.5550.7220.7430.7890.7750.7270.7530.7310.7790.7760.787
Constrained JSON (probability tree)0.5370.7200.7440.7840.7760.7260.7550.7270.7810.7760.786
Constrained JSON (ABC)0.6200.7290.7330.7820.7720.7440.7620.7760.7860.7640.776

Table 7. AUROC (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.

MethodLlama 3.2 1BLlama 3.2 3BGemma 4 E2BGemma 4 E4BGemma 4 31BQwen2.5 7BQwen3 8BQwen3 14BQwen3 32BQwen3.5 4BQwen3.5 9B
Unconstrained (no conf)0.7550.5130.3550.2720.1990.3190.2940.2690.2510.2800.249
Unconstrained (no conf ABC)0.8380.6590.4480.3230.2050.3930.3810.2980.2840.3420.290
Unconstrained (10 samples)0.4950.3860.3440.2530.1960.3020.2810.2570.2310.2210.201
Unconstrained + P(True)0.3310.3230.3330.2250.1870.2730.2660.2290.1900.1790.158
Unconstrained JSON (no conf)0.7270.4850.3440.2750.2000.3310.2910.2670.2590.2820.248
Unconstrained JSON (no conf ABC)0.8400.5790.3960.3060.2020.3960.3420.2850.2800.3310.287
Unconstrained JSON verbal (pre)0.5380.3940.2490.2160.1520.2540.2150.2360.1880.2380.209
Unconstrained JSON verbal (post)0.4700.4340.3010.2100.1460.2070.2140.2300.1740.2270.203
Unconstrained JSON verbal (pre, Likert)0.4940.2700.1800.1940.1200.1620.1740.1880.1460.1800.160
Unconstrained JSON verbal (post, Likert)0.3860.2810.1910.1880.1070.1410.1380.1920.1210.1520.120
Unconstrained JSON verbal (pre, ABC)0.6820.4790.2770.2370.1560.3160.2950.2460.2660.3200.247
Unconstrained JSON verbal (post, ABC)0.7640.4850.3140.2410.1510.2740.2860.2390.2020.2830.245
Constrained (greedy)0.7800.5100.3610.2720.1990.3190.2940.2690.2510.2790.249
Constrained (total likelihood)0.3560.2380.3290.2280.1920.2650.2670.2420.2040.1430.138
Constrained (total likelihood, decision-only)0.3460.2370.3290.2280.1920.2650.2670.2420.2040.1430.138
Constrained (length norm)0.0880.1960.2170.2310.1960.1600.1500.1180.1670.3460.328
Constrained (length norm, decision-only)0.1400.0940.3120.1980.1870.2170.2430.2230.1640.0660.053
Constrained (Google length norm)0.2680.1750.3200.2140.1900.2500.2600.2360.1910.1030.107
Constrained (Google length norm, decision-only)0.3140.2200.3280.2260.1910.2620.2650.2410.2010.1320.131
Constrained (probability tree)0.3350.2360.3290.2280.1920.2660.2670.2420.2040.1420.138
Constrained (ABC)0.3570.1890.4140.2700.1990.3160.3450.2700.2170.1250.148
Constrained JSON (greedy)0.7720.4860.3640.2850.1980.3350.2970.2790.2580.2890.256
Constrained JSON (total likelihood)0.3520.2160.3400.2320.1900.2780.2600.2330.1700.1510.142
Constrained JSON (total likelihood, decision-only)0.3350.2080.3400.2360.1900.2780.2600.2320.1700.1500.142
Constrained JSON (length norm)0.1980.2280.2200.1570.1970.1910.1130.1840.3790.3870.411
Constrained JSON (length norm, decision-only)0.1880.0980.3220.1980.1840.2210.2250.1950.3910.1230.110
Constrained JSON (Google length norm)0.2730.1450.3320.2160.1870.2620.2480.2250.1380.1120.106
Constrained JSON (Google length norm, decision-only)0.3040.1930.3380.2330.1900.2740.2570.2300.1630.1400.134
Constrained JSON (probability tree)0.3320.2060.3400.2370.1900.2780.2600.2340.1700.1490.141
Constrained JSON (ABC)0.1980.1900.3740.2610.1960.3120.3090.2560.2070.1550.155

Table 8. ECE (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.

MethodLlama 3.2 1BLlama 3.2 3BGemma 4 E2BGemma 4 E4BGemma 4 31BQwen2.5 7BQwen3 8BQwen3 14BQwen3 32BQwen3.5 4BQwen3.5 9B
Unconstrained (no conf)0.7550.5130.3550.2720.1990.3190.2940.2690.2510.2800.249
Unconstrained (no conf ABC)0.8380.6590.4480.3230.2050.3930.3810.2980.2840.3420.290
Unconstrained (10 samples)0.4560.3720.3440.2540.1960.3020.2820.2580.2310.2260.206
Unconstrained + P(True)0.3080.3050.3350.2320.1870.2770.2670.2310.1980.2060.182
Unconstrained JSON (no conf)0.7270.4850.3440.2750.2000.3310.2910.2670.2590.2820.248
Unconstrained JSON (no conf ABC)0.8400.5790.3960.3060.2020.3960.3420.2850.2800.3310.287
Unconstrained JSON verbal (pre)0.5050.3980.2700.2290.1620.3060.2480.2680.2140.2560.221
Unconstrained JSON verbal (post)0.4390.4410.3070.2220.1580.2380.2330.2630.1960.2440.213
Unconstrained JSON verbal (pre, Likert)0.4890.3130.2530.2350.1450.2500.2370.2270.2150.2430.190
Unconstrained JSON verbal (post, Likert)0.3550.3230.2510.2180.1400.2140.2040.2140.1780.2280.173
Unconstrained JSON verbal (pre, ABC)0.5940.4690.2930.2460.1670.3520.3050.2770.2870.3310.258
Unconstrained JSON verbal (post, ABC)0.7380.4910.3160.2520.1620.2980.2970.2680.2250.2950.256
Constrained (greedy)0.7800.5100.3610.2720.1990.3190.2940.2690.2510.2790.249
Constrained (total likelihood)0.2970.2580.3290.2310.1920.2700.2680.2430.2080.1790.167
Constrained (total likelihood, decision-only)0.2880.2580.3290.2310.1920.2710.2680.2430.2080.1780.167
Constrained (length norm)0.1280.2530.2500.2330.1850.2150.1940.1770.1770.3220.294
Constrained (length norm, decision-only)0.1510.1990.3110.2080.1880.2390.2470.2260.1830.1550.142
Constrained (Google length norm)0.2160.2270.3200.2190.1900.2590.2610.2370.1980.1640.153
Constrained (Google length norm, decision-only)0.2570.2480.3270.2290.1920.2680.2660.2420.2050.1740.163
Constrained (probability tree)0.2780.2580.3290.2310.1920.2710.2680.2430.2080.1780.167
Constrained (ABC)0.2740.1860.4100.2700.1990.3090.3390.2690.2260.1890.183
Constrained JSON (greedy)0.7720.4860.3640.2850.1980.3350.2970.2790.2580.2890.256
Constrained JSON (total likelihood)0.2370.2620.3380.2430.1910.2830.2610.2400.1890.1900.177
Constrained JSON (total likelihood, decision-only)0.2700.2520.3380.2390.1910.2830.2610.2380.1880.1890.177
Constrained JSON (length norm)0.1440.2690.2530.2040.1860.2310.1850.2180.3520.3720.370
Constrained JSON (length norm, decision-only)0.1650.2080.3200.2120.1850.2450.2340.2140.1650.1780.174
Constrained JSON (Google length norm)0.1760.2320.3300.2400.1880.2700.2500.2310.1740.1740.164
Constrained JSON (Google length norm, decision-only)0.2500.2440.3360.2370.1900.2790.2580.2360.1850.1850.174
Constrained JSON (probability tree)0.2640.2520.3380.2400.1910.2820.2610.2400.1880.1890.177
Constrained JSON (ABC)0.1280.2370.3660.2610.1960.3050.3060.2540.2160.1960.186

Table 9. Top-label Brier (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.

Prompts

Fully rendered prompt (FEVER, plaintext label, Gemma 4 31B chat template)

<bos><|turn>system
# CONTEXT
You are an expert fact-checker. You specialize in
    analyzing claims with provided evidence and
    deciding whether the evidence supports the
    claim, refutes the claim, or there is not
    enough evidence to make a clear decision.

# STYLE & TONE
Favor complete brevity. You need not respond with
    any reasoning, just the answer.

# FORMAT
* You will be provided with the claim inside
    <claim></claim> tags.

* Evidence will be provided within the
    <evidence></evidence> tags.

* You will be provided with each set of evidence
    inside <evidence_set></evidence_set> tags.

* Within the evidence tags, you will be provided
    each piece of evidence inside <evidence_piece
    title=[TITLE]></evidence_piece> tags, where
    the title of the wiki pages article is
    provided in the first tag.

* Respond with ONLY the answer from one of these
    options:
    - SUPPORTS
    - REFUTES
    - NOT ENOUGH INFO<turn|>
<|turn>user
# OBJECTIVE
Classify whether the provided evidence SUPPORTS or
    REFUTES the given claim, or whether there is
    NOT ENOUGH INFO. Rely ONLY on the evidence
    provided and use any combination of the pieces
    of evidence provided.

# INPUT
<claim>
    Aruba is the only ABC Island.
</claim>

<evidence>
    <evidence_set>
        <evidence_piece title=ABC islands (Lesser
        Antilles)>
            The ABC islands are the three western-most
            islands of the Leeward Antilles in the
            Caribbean Sea that lie north of Falcón State,
            Venezuela.
        </evidence_piece>
    </evidence_set>
    <evidence_set>
        <evidence_piece title=ABC islands (Lesser
        Antilles)>
            From west to east they are, in order Aruba,
            Curaçao, and Bonaire.
        </evidence_piece>
    </evidence_set>
</evidence>

# ANSWER
Answer (choose between SUPPORTS, REFUTES, NOT ENOUGH
    INFO):<turn|>
<|turn>model
<|channel>thought
<channel|>

Fully rendered prompt (Banking77, JSON verbal (post), Gemma 4 31B chat template)

<bos><|turn>system
# CONTEXT
You are an expert customer support analyst for
    online banking. You specialize in analyzing

customer queries and classifying them into one of 77 intent classes.

STYLE & TONE

Favor complete brevity. You need not respond with reasoning, just the answer.

FORMAT

  • The customer query will be provided inside <query></query> tags.

  • Your response MUST be in JSON format, with the intent label FIRST and confidence SECOND.

  • The "label" key in your answer must be only ONE of the following intents:

    • activate_my_card

    • age_limit

    • ...

  • The "confidence" key in your answer MUST be provided as a percentage to reflect your confidence in the label you are about to provide.

  • Return ONLY the JSON object. NO OTHER TEXT.

  • Example format: { "label": "[INTENT]", "confidence": "X%" } <turn|> <|turn>user

OBJECTIVE

Provide confidence about your ability to classify the intent of the customer query and classify the customer's intent with their query. Choose the most specific intent that best matches their query.

INPUT

<query> How do I locate my card? </query>

ANSWER

JSON output with the intent label FIRST and confidence SECOND ({"label": "[choose between activate_my_card, age_limit, ...]", "confidence": "[PERCENTAGE]%"}):<turn|> <|turn>model <|channel>thought <channel|>


**Fully rendered prompt (CLINC150, JSON verbal (pre, Likert), Gemma 4 31B chat template)**

<bos><|turn>system

CONTEXT

You are an expert virtual assistant analyst. You specialize in analyzing user queries and classifying them into one of 150 intent classes.

STYLE & TONE

Favor complete brevity. You need not respond with reasoning, just the answer.

FORMAT

  • The user query will be provided inside <query></query> tags.

  • Your response MUST be in JSON format, with confidence FIRST and the intent label SECOND.

  • The "confidence" key in your answer MUST be one of the following labels to reflect your confidence in the label you are about to provide:

    • Not at all confident

    • Slightly confident

    • Moderately confident

    • Very confident

    • Completely confident

  • The "label" key in your answer must be only ONE of the following intents:

    • accept_reservations

    • account_blocked

  • alarm

  • ...

  • Example format: { "confidence": "Not at all confident" | "Slightly confident" | "Moderately confident" | "Very confident" | "Completely confident", "label": "[INTENT]" }<turn|> <|turn>user

OBJECTIVE

Classify the user's intent with their query. Choose the most specific intent that best matches their query. If the query does not belong to any listed intent, answer oos.

INPUT

<query> how would you say fly in italian </query>

ANSWER

JSON output with confidence FIRST and intent label SECOND ({"confidence": "[choose between Not at all confident, Slightly confident, Moderately confident, Very confident, Completely confident]", "label": "[choose between restaurant_reviews, nutrition_info, ...]"}):<turn|> <|turn>model <|channel>thought <channel|>

Fully rendered prompt (SST-5, plaintext label (P(True)), Gemma 4 31B chat template)

<bos><|turn>system

CONTEXT

You are an expert sentiment analysis analyst for movie reviews. You specialize in analyzing review excerpts and classifying them into one of 5 sentiment classes.

STYLE & TONE

Favor complete brevity. You need not respond with any reasoning, just the answer.

FORMAT

  • You will be provided with the instructions given to the annotator and the annotator's response.

  • You must respond with whether the annotator is correct or not ("True" or "False").

  • You must ONLY respond with True or False, no other text.<turn|> <|turn>user

OBJECTIVE

Judge whether the response provided by an annotator is correct (True or False). They were tasked to classify the sentiment of the review excerpt, choosing the sentiment that best matches the overall tone from the list of 5 valid sentiments.

INPUT & ANNOTATOR RESPONSE

"""

INPUT

CONTEXT

You are an expert sentiment analysis analyst for movie reviews. You specialize in analyzing review excerpts and classifying them into one of 5 sentiment classes.

STYLE & TONE

Favor complete brevity. You need not respond with any reasoning, just the answer.

FORMAT

  • The review excerpt will be provided inside <review></review> tags.

  • Respond with ONLY the answer from the list of valid sentiments provided:

    • very negative

    • negative

    • neutral

    • positive

    • very positive

OBJECTIVE

Classify the sentiment of the review excerpt. Choose the sentiment that best matches the overall tone.

INPUT

<review> no movement , no yuks , not much of anything . </review>

ANSWER

Answer (choose from: very negative, negative, neutral, positive, very positive):

RESPONSE

negative """

ANSWER

Correct? (True | False):<turn|> <|turn>model <|channel>thought <channel|>

Fully rendered prompt (20NG, JSON output (ABC, with prefili for logit method), Gemma 4 31B chat template)

<bos><|turn>system

CONTEXT

You are an expert text classification analyst for online discussion forums. You specialize in analyzing newsgroup posts and classifying them into one of 20 topic categories.

STYLE & TONE

Favor complete brevity. You need not respond with reasoning, just the answer.

FORMAT

  • The newsgroup post will be provided inside <post></post> tags.

  • Your response MUST be in JSON format, containing only the category label.

  • Respond with ONLY the answer from the list of valid categories provided:

    • A (alt.atheism)

    • B (comp.graphics)

    • C (comp.os.ms-windows.misc)

    • D (comp.sys.ibm.pc.hardware)

    • E (comp.sys.mac.hardware)

    • F (comp.windows.x)

    • G (misc.forsale)

    • H (rec.autos)

    • I (rec.motorcycles)

    • J (rec.sport.baseball)

    • K (rec.sport.hockey)

    • L (sci.crypt)

    • M (sci.electronics)

    • N (sci.med)

    • O (sci.space)

    • P (soc.religion.christian)

    • Q (talk.politics.guns)

    • R (talk.politics.mideast)

    • S (talk.politics.misc)

    • T (talk.religion.misc)

  • Example format: { "label": "[CATEGORY_LETTER]" }<turn|> <|turn>user

OBJECTIVE

Classify the newsgroup post into its topic category.
Choose the LETTER that represents the most
specific category that best matches the post's
content.

# INPUT
<post>
    I am a little confused on all of the models of the
        88-89 bonnevilles.
    I have heard of the LE SE LSE SSE SSEI. Could
        someone tell me the
    differences are far as features or performance. I am
        also curious to
    know what the book value is for prefereably the 89
        model. And how much
    less than book value can you usually get them for.
        In other words how
    much are they in demand this time of year. I have
        heard that the mid-spring
    early summer is the best time to buy.
</post>

# ANSWER
JSON output containing only the category label
    ({"label": "choose between A, B, C, D, E, F,
        G, H, I, J, K, L, M, N, O, P, Q, R, S,
        T"}):<turn|>
<|turn>model
<|channel>thought
<|channel|>```json
{"label": "

References

  1. [1]Anthropic. 2026. Prompting best practices. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags. Claude documentation, accessed 9 November 2026.
  2. [2]Yaara Artsi, Vera Sorin, Benjamin S Glicksberg, Panagiotis Korfiatis, Girish N Nadkarni, and Eyal Klang. 2025. Large language models in real-world clinical workflows: a systematic review of applications and implementation. Frontiers in Digital Health, 7:1659134.
  3. [3]Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, and 1 others. 2024. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782.arxiv.org/abs/2405.14782
  4. [4]Glenn W. Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3.doi.org/10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
  5. [5]Martin Juan José Bucher and Marco Martini. 2024. Fine-tuned’small’llms (still) significantly outperform zero-shot generative ai models in text classification. arXiv preprint arXiv:2406.08660.arxiv.org/abs/2406.08660
  6. [6]Michael Burnham, Kayla Kahn, Ryan Yang Wang, and Rachel X Peng. 2026. Political debate: Efficient zero-shot and few-shot classifiers for political text. Political Analysis, 34(3):329–343.DOI
  7. [7]Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The alternative annotator test for llm-as-a-judge: How to statistically justify replacing human annotators with llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 16051–16081.
  8. [8]Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd workshop on natural language processing for conversational AI, pages 38–45.arxiv.org/abs/2003.04807
  9. [9]Veronica Chatrath, Marcelo Lotif, and Shaina Raza. 2024. Fact or fiction? can llms be reliable annotators for political truths? arXiv preprint arXiv:2411.05775.arxiv.org/abs/2411.05775
  10. [10]Zhongpu Chen, Yinfeng Liu, Long Shi, Zhi-Jie Wang, Xingyan Chen, Yu Zhao, and Fuji Ren. 2025. Mde-val: Evaluating and enhancing markdown awareness in large language models. In Proceedings of the ACM on Web Conference 2025, pages 2981–2991.DOI
  11. [11]Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00904.arxiv.org/abs/2010.00904
  12. [12]Yixin Dong, Charlie F Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen. 2025. Xgrammar: Flexible and efficient structured generation engine for large language models. Proceedings of Machine Learning and Systems, 7.arxiv.org/abs/2411.15100
  13. [13]Ekaterina Fadeeva, Roman Vashurin, Akim Tsygvin, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. LM-polygraph: Uncertainty estimation for language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 446–461, Singapore. Association for Computational Linguistics.arxiv.org/abs/2311.07383
  14. [14]David Farr, Iain Cruickshank, Nico Manzonelli, Nicholas Clark, Kate Starbird, and Jevin West. 2024. Llm confidence evaluation measures in zero-shot css classification. arXiv preprint arXiv:2410.13047.arxiv.org/abs/2410.13047
  15. [16]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, and 1 others. 2026. Gemma 4 technical report. arXiv preprint arXiv:2607.02770.
  16. [17]Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 conference of the north American chapter of the association for computational linguistics: human language technologies (volume 1: long papers), pages 6577–6595.arxiv.org/abs/2311.08298
  17. [18]Jonathan Geuter, Andrew Cheng, and David Alvarez-Melis. 2026. Releasing the inner jev: A simple recipe for open decision models. arXiv preprint arXiv:2610.17205.arxiv.org/abs/2610.17205
  18. [19]Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120.DOI
  19. [21]Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannah Hajishirzi. 2025. Olmes: A standard for language model evaluations. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5020–5048.
  20. [22]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR.arxiv.org/abs/1706.04599
  21. [23]Chirag Gupta and Aaditya Ramdas. 2021. Top-label calibration and multiclass-to-binary reductions. arXiv preprint arXiv:2107.08353.arxiv.org/abs/2107.08353
  22. [24]Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136.
  23. [25]Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65–70.DOI
  24. [27]Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664.arxiv.org/abs/2302.09664
  25. [28]Sachin Kumar, Chan Young Park, and Yulia Tsvetkov. 2024. Gen-Z: Generative zero-shot text classification with contextualized label descriptions. In International Conference on Learning Representations, volume 2024, pages 40169–40192.arxiv.org/abs/2311.07115
  26. [29]Ken Lang. 1995. Newsweeder: Learning to filter net-news. In Machine learning proceedings 1995, pages 331–339. Elsevier.
  27. [30]Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.arxiv.org/abs/1909.02027
  28. [31]Xianzhi Li, Samuel Chan, Xiaodan Zhu, Yulong Pei, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. 2023. Are chatgpt and gpt-4 general-purpose solvers for financial text analytics? a study on several typical tasks. In Proceedings of the 2023 conference on empirical methods in natural language processing: industry track, pages 408–422.arxiv.org/abs/2305.05862
  29. [32]Yinheng Li and Justin Wagle. 2026. Llm-as-jev: Llms are already jev-style decision models – when and how to fine-tune them. Preprint, arXiv:2610.02076.arxiv.org/abs/2610.02076
  30. [33]Xiaou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. 2025. Uncertainty quantification and confidence calibration in large language models: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6107–6117.arxiv.org/abs/2503.15850
  31. [34]Scott Lundberg, Harsha Nori, and Marco Tulio Ribeiro. 2022. Guidance. https://github.com/guidance-ai/guidance.
  32. [35]Bar Meyuhas, Anat Bremler-Barr, and Tal Shapira. 2024. Iot device labeling using large language models. arXiv preprint arXiv:2403.01586.arxiv.org/abs/2403.01586
  33. [36]Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29.
  34. [37]Nzubechukwu C Ohalete, Kevin B Gittner, and Lauren M Matheny. 2025. Costar-a: A prompting framework for enhancing large language model performance on point-of-view questions. arXiv preprint arXiv:2510.12637.arxiv.org/abs/2510.12637
  35. [38]Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick, Nadia Polikarpova, and Loris D’Antoni. 2024. Grammar-aligned decoding. Advances in Neural Information Processing Systems, 37:24547–24568.arxiv.org/abs/2405.21047
  36. [40]Qwen Team. 2026. Qwen3.5. https://huggingface.co/collections/Qwen/qwen35. Hugging Face model collection. Models used: Qwen/Qwen3.5-4B and Qwen/Qwen3.5-9B. Accessed 2 November 2026.
  37. [41]Qwen Team, An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengeng Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  38. [42]Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025b. Qwen2.5 technical report. Preprint, arXiv:2412.15115.
  39. [43]Hajar Sakai and Sarah S Lam. 2026. Large language models for health care text classification: systematic review. Jmir Ai, 5(1):e79202.DOI
  40. [44]Mario Sanz-Guerrero and Katharina von der Wense. 2025. Mitigating label length bias in large language models. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 1404–1420.
  41. [45]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.DOI
  42. [46]James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819.arxiv.org/abs/1803.05355
  43. [47]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442.

Paper details

vlnc:W_h4kbfjad
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents