betaFeedback

WorksComputer Science

Can Jev Improve Demonstration Retrieval for In-Context Learning?

Abstract

Demonstration selection plays a crucial role in the effectiveness of in-context learning (ICL) for large language models (LLMs). Extensive research has explored various retrieval strategies to identify informative demonstrations and improve downstream performance. Recently, the emergence of Jev, a decision-oriented language model capable of producing probabilistic judgments, offers a new perspective on evaluating demonstration utility. This motivates us to investigate a fundamental question: Can Jev improve demonstration retrieval for in-context learning? In this paper, we demonstrate that Jev can effectively enhance demonstration retrieval by directly assessing the utility of candidate demonstrations for a given query. However, applying Jev to large-scale retrieval introduces substantial computational overhead, as each candidate requires an additional decision-making inference. To address this limitation, we propose a simple yet effective knowledge distillation approach that transfers Jev's utility estimation capability to a lightweight multilayer perceptron (MLP). By learning to approximate Jev's probabilistic judgments, the distilled model enables efficient demonstration ranking while maintaining competitive retrieval performance. Our study highlights the potential of decision-oriented language models for demonstration selection and provides an efficient framework for integrating their decision-making capabilities into ICL.

Introduction

In-context learning (ICL) has emerged as a remarkable capability of large language models (LLMs), enabling them to adapt to diverse downstream tasks without requiring parameter updates ([18]). Demonstrations serve as the foundation of ICL, allowing LLMs to infer task-relevant patterns and input–output relationships from a few labeled examples ([1, 16]). Consequently, extensive research has explored various demonstration selection strategies to identify informative examples and improve ICL performance.

Existing demonstration selection methods can be broadly categorized into heuristic-based and training-based approaches. Heuristic-based methods typically quantify the relevance between candidate demonstrations and a given query using lexical matching, such as BM25, semantic similarity in embedding spaces or perplexity, and select the highest/lowest-scoring candidates as demonstrations ([10, 13, 22]). To further improve selection quality, some approaches adopt a multi-stage retrieval paradigm, where an initial retrieval stage narrows the candidate pool based on static relevance scores, followed by further refinement that incorporates demonstration diversity, redundancy reduction, or feedback from LLMs ([5, 12, 17]). In contrast, training-based approaches learn task-aware retrievers by leveraging supervision signals derived from LLM predictions, likelihoods, or performance feedback to identify more informative demonstrations ([25, 20]).

Despite their effectiveness, heuristic-based methods often struggle to capture the actual utility of demonstrations, as predefined relevance metrics do not necessarily reflect their contribution to downstream predictions. Although training-based methods can better estimate demonstration utility through LLM feedback, they typically incur substantial computational costs and require retraining for different tasks or target models, limiting their scalability and cross-task generalizability.

Motivated by the recent emergence of Jev ([19]), a decision model capable of making probabilistic judgments, we investigate whether its decision making capabilities can improve demonstration retrieval for ICL. Specifically, we first employ heuristic retrieval to narrow the candidate pool and then leverage Jev to rerank demonstrations according to their estimated utility for a given query. Our experiments show that this simple strategy yields substantial performance improvements. However, repeatedly evaluating candidate demonstrations with Jev introduces considerable computational overhead and inference latency. To address this limitation, we propose a task agnostic distillation framework that transfers Jev’s utility estimation capability to a lightweight multilayer perceptron (MLP). By learning to approximate Jev’s judgments of demonstration utility, the MLP efficiently refines retrieved candidates without requiring repeated Jev inference. Experimental results demonstrate that our approach achieves a favorable balance between effectiveness and efficiency, while exhibiting strong generalizability across diverse tasks with a single distillation process. Our main contributions are summarized as follows:

  • We investigate the potential of Jev for ICL demonstration retrieval and demonstrate that its probabilistic judgments can effectively identify informative demonstrations.

  • We propose a simple and task-agnostic distillation framework that transfers Jev’s demonstration utility estimation capability to a lightweight MLP, reducing computational overhead while improving generalizability across tasks.

  • Experiments across diverse tasks and LLMs demonstrate the effectiveness of our approach, achieving strong retrieval performance, improved efficiency, and robust cross-task generalization.

Related Work

Demonstration Retrieval in ICL. Learning from demonstrations has emerged as a fundamental capability of LLMs, prompting extensive research on improving their performance through more effective demonstration retrieval strategies. Early studies primarily relied on heuristic-based retrieval methods, such as lexical matching and semantic similarity, to identify demonstrations relevant to a given query [13, 10]. Furthermore, several studies extend efficient embedding-based retrieval by incorporating additional signals, such as perplexity, prediction confidence, and demonstration diversity, to refine candidate selection [6, 12, 24, 22]. But these methods often rely on predefined heuristics that may not reflect actual demonstration utility, limiting their effectiveness and adaptability across LLMs.

Alternatively, training-based methods partially address these limitations by learning demonstration retrievers from LLM feedback. EPR [14] trains a dense retriever using demonstration utility estimated from output probabilities. UDR [7] further introduces multi-task list-wise ranking to learn a unified retriever across tasks. Wang et al. [21] employ reward modeling and knowledge distillation to iteratively optimize retrieval. GenICL [25] leverages generative preference learning to directly optimize demonstration selection based on LLM feedback. More recently, DiSP [20] trains lightweight judges to predict the success of candidate demonstration sets, enabling efficient selection without exhaustive search. Despite their effectiveness, these methods typically require substantial LLM feedback collection and additional training, increasing the cost of adapting to new tasks or models.

Recent Research in Jev. The recent emergence of Jev, a specialized decision model, has attracted increasing attention by reducing the computational cost of LLM-based decision making. Recent studies have investigated its effectiveness and reliability across diverse tasks. Several benchmarking studies have demonstrated its competitive performance in text classification, natural language inference, and other decision-oriented tasks [3, 8]. Meanwhile, Jev-as-a-Judge [9] explores confidence-aware decision making by selectively invoking more powerful LLMs for uncertain predictions, while other studies examine the calibration and reliability of Jev’s probabilistic outputs [11]. More recently, General Decision Models [4] investigates the broader capabilities of decision models and explores efficient decision making through knowledge distillation. Despite these advances, whether Jev’s probabilistic judgments can effectively guide demonstration retrieval, particularly through lightweight and task-agnostic reranking, remains underexplored.

Preliminary Experiment

Problem Formulation

In-Context Learning. Given a labeled dataset D={(xi,yi)}i=1N\mathcal{D} = \{(x_i, y_i)\}_{i=1}^{N} and a target query xqx_q, ICL enables an LLM fθf_\theta to predict the target label without updating its parameters. Specifically, a set of KK labeled demonstrations Cq={(xi,yi)}i=1K⊆DC_q = \{(x_i, y_i)\}_{i=1}^{K} \subseteq D is incorporated into the input context, and the prediction is y^q=arg⁡max⁡y∈YPθ(y∣Cq,xq)\hat{y}_q = \arg\max_{y\in\mathcal{Y}} P_\theta(y \mid C_q, x_q), where Y\mathcal{Y} denotes the label space and PθP_\theta represents the conditional predictive distribution of the LLM.

Demonstration Retrieval. Based on the above definitions, a retrieval method selects KK demonstrations according to a scoring function ss:

Cqs=TopK⁡(xi,yi)∈Ds(xq,xi).(1)C_q^s = \operatorname{TopK}_{(x_i,y_i)\in D} s(x_q, x_i). \tag*{(1)}

Existing methods typically define ss based on lexical or semantic similarity, which may not accurately reflect the utility of demonstrations for the target prediction.

Jev-guided Reranking

Jev is a specialized decision model that estimates the probabilities of candidate decisions given an input context. This capability enables us to evaluate whether a labeled demonstration (xi,yi)(x_i, y_i) is potentially beneficial for predicting a target query xqx_q. Specifically, we formulate demonstration utility estimation as a binary decision problem, where Jev assesses the usefulness of each candidate through the following instruction:

Jev Prompt

Classification Task: [Task description]

Query to Solve: [Target query]

Candidate Demonstration: [Candidate text]

Candidate Label: [Candidate label]

Would this labeled candidate be a useful and relevant in-context demonstration for correctly classifying the query?

[options] false true

[decision]:

Given the instruction II, we define the Jev-based utility score as

sJev(xq,xi,yi)=PJev(Yes∣I,xq,xi,yi).(2)s_{\mathrm{Jev}}(x_q, x_i, y_i) = P_{\mathrm{Jev}}(\mathrm{Yes} \mid I, x_q, x_i, y_i). \tag*{(2)}

A higher score indicates that Jev considers the candidate more likely to be helpful for the prediction. Although Jev enables efficient decision making, evaluating every candidate in the entire pool DD remains computationally expensive. Inspired by existing demonstration retrieval methods, we therefore adopt a two-stage retrieval framework [12, 22]. First, we employ

a pretrained Sentence-Transformer encoder gg to retrieve the top-MM semantically similar candidates:

Rq=TopM⁡(xi,yi)∈Dsim⁡(g(xq),g(xi)),(3)R_q = \operatorname{TopM}_{(x_i,y_i)\in D} \operatorname{sim}(g(x_q), g(x_i)), \tag*{(3)}

where sim⁡(⋅,⋅)\operatorname{sim}(\cdot,\cdot) denotes cosine similarity. In the second stage, Jev evaluates the utility of each retrieved candidate and reranks them according to Eq. (2). The final KK demonstrations are selected as

CqJev=TopK⁡(xi,yi)∈RqsJev(xq,xi,yi),(4)C_q^{\mathrm{Jev}} = \operatorname{TopK}_{(x_i,y_i)\in R_q} s_{\mathrm{Jev}}(x_q, x_i, y_i), \tag*{(4)}

where K≤MK \le M. This two-stage framework combines the efficiency of semantic retrieval with the decision-making capability of Jev, enabling utility-aware demonstration selection without evaluating the entire candidate pool.

Preliminary Experimental Results

To investigate the effectiveness of Jev-guided reranking, we set the candidate pool size to M=20M = 20 and conduct experiments using two backbone LLMs, Llama3.2-1B and Qwen3-1.7B. We employ two open-source decision models, NeoJev (4B)1 and JEV (9B)2, and report the average accuracy across five datasets: TREC for question classification, SST-2 and SST-5 for binary and fine-grained sentiment classification, respectively, AG News for topic classification, and MNLI for natural language inference as used in Wang et al., [20]. Further details on the datasets and experiment are provided in App. § 7. The results are presented in Figure 1.

Figure 1. Average results over five datasets with the increasing of KK. More detailed results are in App. § 8.

JEV-guided retrieval generally improves ICL performance over conventional kNN retrieval across different numbers of demonstrations [10]. As the number of demonstrations increases, both JEV-9B and NeoJEV-4B tend to achieve larger improvements. At 8-shot, JEV-9B improves the average accuracy by 3.34 and 4.45 percentage points on Llama3.2-1B and Qwen3-1.7B, respectively, while NeoJEV-4B achieves gains of 3.30 and 6.29 percentage points. Notably, despite having fewer parameters, NeoJEV-4B achieves comparable performance on Llama3.2-1B and substantially outperforms JEV-9B on Qwen3-1.7B. This observation is consistent with the stronger discriminative capabilities reported for NeoJEV-4B in its official evaluation, suggesting that decision-making quality, rather than model size alone, is critical for identifying informative demonstrations. Although the improvements are not strictly monotonic across shots, these findings demonstrate the potential of decision-oriented models to guide effective demonstration selection beyond semantic similarity.

Quantitative Analysis of Jev Retrieval

Although end-to-end ICL accuracy demonstrates the overall effectiveness of JEV-guided retrieval, it provides limited insight into how JEV modifies the demonstrations retrieved by kNN and whether the newly selected demonstrations are more useful than those they replace. To better understand this process, we analyze JEV-guided retrieval from two complementary perspectives: (1) the extent to which JEV alters the retrieved demonstration set, and (2) whether these changes improve the utility of the selected demonstrations.

Demonstration replacement ratio. Figure 2 illustrates the distribution of the number of demonstrations replaced by different JEV models across varying shot settings. Both JEV-9B and NeoJEV consistently replace a substantial proportion of the demonstrations initially retrieved by kNN. As KK increases, the number of replaced demonstrations generally increases, indicating that JEV-guided reranking substantially modifies the original retrieval results. These findings reveal a notable discrepancy between semantic retrieval and JEV’s assessment of demonstration usefulness, suggesting that semantically similar examples are not necessarily the most informative for ICL.

Figure 2. Distribution of demonstration replacements.

Replacement utility gain. To assess whether JEV selects more useful demonstrations than those discarded from kNN, we introduce Replacement Utility Gain (RUG). Let u(xi,xq)∈{0,1}u(x_i, x_q) \in\{0,1\} indicate whether the LLM correctly predicts query xqx_q using demonstration xix_i alone. Given the previously defined CqsC_q^s and CqJevC_q^{\mathrm{Jev}}, we define RUG as

RUG⁡(xq)=∑xi∈CqJev∖Cqsu(xi,xq)∣CqJev∖Cqs∣−∑xi∈Cqs∖CqJevu(xi,xq)∣Cqs∖CqJev∣.(5)\operatorname{RUG}(x_q)=\frac{\sum_{x_i\in C_q^{\mathrm{Jev}}\setminus C_q^s}u(x_i,x_q)}{\lvert C_q^{\mathrm{Jev}}\setminus C_q^s\rvert}-\frac{\sum_{x_i\in C_q^s\setminus C_q^{\mathrm{Jev}}}u(x_i,x_q)}{\lvert C_q^s\setminus C_q^{\mathrm{Jev}}\rvert}. \tag*{(5)}

Here ∣Cqs∖CqJev∣\lvert C_q^s\setminus C_q^{\mathrm{Jev}}\rvert denotes the number of demonstrations discarded from the original kNN retrieval results, which equals the number of demonstrations introduced by JEV.

Figure 3 shows that JEV-selected demonstrations generally become more useful relative to discarded kNN demonstrations as KK increases. The gains are more consistent on Qwen3-1.7B, whereas Llama3.2-1B exhibits task-dependent improvements, with positive effects on TREC, SST-5, and AG News but negative effects on SST-2 and MNLI. Notably, the slight decline at K=8K=8 may be attributed to the fixed candidate pool size (M=20M=20), which limits the availability of additional informative demonstrations as KK increases. Overall, these results suggest that JEV-guided replacement generally improves demonstration utility across most datasets, particularly under larger demonstration budgets, although its effectiveness remains dependent on the LLMs and task characteristics.

Figure 3. RUG scores across datasets and the average RUG results.

Taken together, these experiments demonstrate that Jev-guided reranking can identify more informative demonstrations than conventional semantic similarity-based retrieval, thereby improving overall ICL performance. Although its effectiveness varies across downstream models and tasks, these findings highlight the potential of Jev as a utility-aware demonstration evaluator.

Method: Multi-objective Distillation

Although JEV-guided retrieval improves the quality of in-context demonstrations, its inference over head remains substantial. Given a candidate pool DqD_q containing MM demonstrations, JEV reranking requires MM forward evaluations. Motivated by knowledge distillation (Xu et al., [23]), we propose a multi-objective distillation framework as shown in Figure 4 that transfers the utility estimation and ranking capabilities of JEV into a lightweight MLP student, enabling efficient demonstration selection without invoking JEV during inference.

Figure 4. Overview of the proposed framework.

Distillation Data Construction

To faithfully distill the teacher’s behavior, we construct a JEV-labeled candidate set for each query xqx_q from validation set. In practice, the validation set is split from the training set and is kept strictly disjoint from the test set. Specifically, given MM retrieved candidates in Rq\mathcal{R}_q, we evaluate each candidate xix_i by Eq. (2). These scores provide supervision for distilling the teacher’s scoring and ranking behavior into a lightweight student model.

Lightweight Utility Estimation

We employ a frozen semantic encoder g(⋅)g(\cdot) to obtain the representations of the query, candidate demonstrations, and their labels. Let q=g(xq)\mathbf{q}=g(x_q), hi=g(xi)\mathbf{h}_i=g(x_i), and li=g(yi)\mathbf{l}_i=g(y_i), where yiy_i denotes the natural language description of the candidate label. We construct a query-aware feature vector:

zi=[q;hi;∣q−hi∣;q⊙hi;simi;li],(6)\mathbf{z}_i = [\mathbf{q};\mathbf{h}_i;|\mathbf{q}-\mathbf{h}_i|;\mathbf{q}\odot\mathbf{h}_i;\mathit{sim}_i;\mathbf{l}_i], \tag*{(6)}

where simi\mathit{sim}_i denotes the cosine similarity between the query and candidate, ⊙\odot represents elementwise multiplication, and [⋅;⋅][\cdot;\cdot] denotes concatenation. Then, a lightweight MLP estimates the individual utility score as

s^(xq,xi,yi)=σ(MLP(zi)),(7)\hat{s}(x_q,x_i,y_i)=\sigma(MLP(\mathbf{z}_i)), \tag*{(7)}

where σ(⋅)\sigma(\cdot) is the sigmoid function.

Multi-objective Distillation

We optimize the student through two complementary objectives: score distillation and ranking distillation. The former preserves the teacher’s utility estimates, while the latter transfers its relative preferences among candidate demonstrations.

Score Distillation. Given teacher scores sJEV(xq,xi,yi)s_{\mathrm{JEV}}(x_q,x_i,y_i) and student predictions s^(xq,xi,yi)\hat{s}(x_q,x_i,y_i), we minimize a soft-label binary cross-entropy:

Lscore=−1M∑i=1M[sJEVlog⁡s^+(1−sJEV)log⁡(1−s^)](8)\mathcal{L}_{\mathrm{score}}=-\frac{1}{M}\sum_{i=1}^{M}\left[s_{\mathrm{JEV}}\log\hat{s}+(1-s_{\mathrm{JEV}})\log(1-\hat{s})\right] \tag*{(8)}

where sJEVs_{\mathrm{JEV}} and s^\hat{s} represent the teacher’s utility score and the student’s predicted score for each candidate, respectively.

Ranking Distillation. Predicting individual demonstration scores may overlook the relative differences between candidates. We therefore introduce a pairwise ranking objective based on RankNet (Burges et al., [2]) to explicitly capture the teacher’s preferences. For any two candidate demonstrations xix_i and xjx_j, we define Δij=sJEV,i−sJEV,j\Delta_{ij}=s_{\mathrm{JEV},i}-s_{\mathrm{JEV},j} and Δ^ij=MLP(zi)−MLP(zj)\hat{\Delta}_{ij}=MLP(\mathbf{z}_i)-MLP(\mathbf{z}_j) as the teacher score difference and student logit difference, respectively. The pairwise ranking loss is defined as:

ℓiq=log⁡(1+exp⁡[−sign⁡(Δij)Δ^ij]).(9)\ell_{iq}=\log\left(1+\exp\left[-\operatorname{sign}(\Delta_{ij})\hat{\Delta}_{ij}\right]\right). \tag*{(9)}

Here, sign⁡(Δij)\operatorname{sign}(\Delta_{ij}) indicates the teacher’s preference direction, taking +1+1 if candidate xix_i is preferred over xjx_j and −1-1 otherwise. To emphasize challenging comparisons, we introduce a difficulty-aware reweighting strategy that dynamically assigns greater importance to candidate pairs with larger ranking errors. The ranking objective is:

Lrank=∑(i,j)∈Vqωijℓij∑(i,j)∈Vqωij(10)\mathcal{L}_{\mathrm{rank}} = \frac{\sum_{(i,j)\in\mathcal{V}_q}\omega_{ij}\ell_{ij}}{\sum_{(i,j)\in\mathcal{V}_q}\omega_{ij}} \tag*{(10)}

where Vq\mathcal{V}_q denotes the set of valid candidate pairs constructed from MM retrieved demonstrations, excluding pairs with identical teacher scores, and ωij\omega_{ij} represents the difficulty-aware weight, defined as:

ωij=exp⁡(ℓij)∑(u,v)∈Vqexp⁡(ℓuv).(11)\omega_{ij} = \frac{\exp(\ell_{ij})}{\sum_{(u,v)\in\mathcal{V}_q}\exp(\ell_{uv})}. \tag*{(11)}

This strategy emphasizes challenging pairs while retaining supervision from all valid pairs.

Joint Optimization. To jointly preserve the teacher’s utility estimation and ranking capabilities, we combine the score distillation and pairwise ranking objectives L=Lscore+Lrank\mathcal{L} = \mathcal{L}_{\mathrm{score}} + \mathcal{L}_{\mathrm{rank}}.

Efficient Demonstration Selection

During inference, given a test query xqx_q3, we first retrieve MM candidate demonstrations using semantic similarity and compute their representations with the frozen encoder. The distilled student then estimates individual utilities in a single batched forward pass by Eq. (7), allowing candidates to be ranked without invoking JEV. Here, the candidate set reranked by the student is denoted as RqJevR_q^{\mathrm{Jev}}.

To further account for the interactions among demonstrations, we introduce a beam search strategy with a beam width of 4 [15]. Starting from an empty context, we iteratively expand each beam by adding candidate demonstrations from RqJevR_q^{\mathrm{Jev}} and evaluate the resulting combinations using the distilled student. To accommodate multiple demonstrations within the MLP input, we construct a position-weighted representation for each ordered combination Com⁡=(x1,…,xk)\operatorname{Com} = (x_1,\ldots,x_k), assigning greater importance to demonstrations closer to the query:

zS=[q;hˉS;∣q−hˉS∣;q⊙hˉS;cos⁡(q,hˉS);lˉS],(12)z_S = [q;\bar{h}_S;|q-\bar{h}_S|;q\odot\bar{h}_S;\cos(q,\bar{h}_S);\bar{l}_S], \tag*{(12)}

where hˉS=Normalize⁡(∑t=1kwtht)\bar{h}_S = \operatorname{Normalize}(\sum_{t=1}^{k}w_t h_t) and lˉS=Normalize⁡(∑t=1kwtlt)\bar{l}_S = \operatorname{Normalize}(\sum_{t=1}^{k}w_t l_t) are the aggregated demonstration and label representations, respectively, with wt=t/∑j=1kjw_t = t/\sum_{j=1}^{k}j denoting the position-dependent weight. The resulting representation is evaluated by the same distilled student.

At each step, only the four combinations with the highest score are retained for subsequent expansion. Finally, the combination with the highest predicted utility is selected as the in-context demonstrations, rather than simply concatenating individually top-ranked candidates. The selected demonstrations are then incorporated into the prompt for in-context learning. Since demonstration selection is independent of the downstream LLM, our framework can be seamlessly applied to different LLMs without requiring model-specific training or modifications.

Experiment

Experimental Setup

Details. We conduct experiments on five text classification datasets: TREC, SST-2, SST-5, AG News, and MNLI. For each dataset, we sample 15% of the original training set (up to 2,000 examples) as queries for distillation. Each query is paired with 32 candidates retrieved using all-MiniLM-L6-v2 and scored by JEV-9B. A shared MLP student with two hidden layers (768 and 256 dimensions) is trained across all datasets using score and ranking distillation. We optimize the student for 50 epochs using AdamW with a learning rate of 3×10−43 \times10^{-4} and batch size 64. During inference top 20 samples from train set are used for 4-shot demonstration selection via beam search (beam width 4). We evaluate on Llama-3.2-1B, Qwen3-1.7B, Llama-3.1-8B, and Qwen3-8B, using up to 2,000 test examples per dataset. Classification accuracy is reported as the evaluation metric. All experiments are conducted on a single NVIDIA A100 80GB GPU. Further details are provided in App. § 7.

Baselines. In addition to semantic kNN retrieval [10], we compare our method with ConE [12] and DiSP [20]. ConE adopts a select-then-rerank strategy, retrieving semantically similar demonstrations and reranking candidate contexts based on the target LLM’s conditional entropy. Lower-entropy contexts are preferred. For fair comparison, we use a candidate pool of 20 and select four demonstrations. DiSP trains lightweight difficulty-aware routers and judges to predict ICL success, enabling budget-constrained demonstration selection. We train DiSP using up to 2,000 training queries per dataset and restrict its candidate pool to 20. All methods use four demonstrations and share the same evaluation settings.

Main Results

As shown in Table 1, both DisJEV and DisNeoJEV consistently outperform conventional retrieval baselines across all four open-source LLMs. Compared with kkNN, DisJEV improves average accuracy by 2.99, 3.68, 0.43, and 1.68 percentage points, respectively. In contrast, ConE and DiSP exhibit inconsistent improvements and even degrade performance on several LLMs. DisNeoJEV further improves upon DisJEV, achieving average accuracies of 63.28%, 77.68%, 79.58%, and 82.39%, highlighting the importance of teacher discrimination quality. These results suggest that distilling demonstration utility knowledge provides a more effective and robust retrieval strategy than conventional alternatives. Moreover, the lightweight and label-agnostic student enables efficient beam search over demonstration combinations and joint training across diverse tasks. We further investigate

MethodTRECSST-2SST-5AG NewsMNLIAvg.
Llama3.2-1B
kkNN72.6058.8338.1086.6542.0559.65
ConE75.2060.5534.1585.9040.6559.29
DiSP69.4055.7739.1087.6445.4259.47
DisNeoJEV80.6057.9141.8591.0545.0063.28
DisJEV80.0058.1441.9088.8844.2962.64
Qwen3-1.7B
kkNN72.4090.2544.3090.3069.9573.44
ConE78.4087.8947.1083.8071.5573.75
DiSP78.4089.6147.1985.8272.3674.68
DisNeoJEV88.2090.1444.8091.7073.5577.68
DisJEV85.2089.9145.4890.1674.8577.12
Llama3.1-8B
kkNN89.4093.9257.6590.6460.7478.47
ConE86.0094.6153.7590.6054.8075.95
DiSP83.2092.0150.4389.1265.5276.06
DisNeoJEV93.2094.6156.8091.6061.7079.58
DisJEV91.2094.7257.1590.3661.0978.90
Qwen3-8B
kkNN88.4093.2349.8092.2078.4080.41
ConE91.0088.9450.4088.8079.2579.68
DiSP88.7091.6750.6290.7879.0680.17
DisNeoJEV93.8091.5152.0092.1582.5082.39
DisJEV92.8091.7952.5891.1682.1282.09
DeepSeek-V4-Flash
kkNN94.4095.0058.0090.6082.0084.00
DisNeoJEV96.2096.0060.0092.6084.4085.84
DisJEV94.4095.4058.0093.4083.6084.96
Qwen3.8-Flash
kkNN96.0096.0056.4091.6088.2085.64
DisNeoJEV96.2096.0060.0092.6088.2086.60
DisJEV95.6096.4060.4089.8089.0086.24

Table 1. Classification accuracy (%) of different demonstration retrieval methods at K=4K = 4. The best result for each model and dataset is highlighted in bold.

computational efficiency in § 5.3 and cross-task generalization in § 5.4.

Generalization to Larger LLMs. We further evaluate both methods on DeepSeek-V4-Flash and Qwen3.8-Flash via APIs on 500 samples, excluding ConE and DiSP due to the difficulty and cost of repeated confidence estimation. DisJEV improves average accuracy over kkNN by 0.96 and 0.60 percentage points, respectively, while DisNeoJEV achieves larger gains of 1.84 and 0.96 points. Although the improvements are relatively modest, these results confirm that utility-aware retrieval remains beneficial for stronger LLMs and that better teacher discrimination can further enhance student performance.

Time Cost

We further compare the computational efficiency of DisJEV and DiSP using Qwen3-1.7B. Since JEV-9B has a larger number of parameters, DisJEV is expected to incur a higher computational cost. As shown in Figure 5, DisJEV incurs substantially lower training costs than DiSP (137.2 vs. 497.7 seconds per 100 samples on average), primarily because our method employs a lightweight MLP student, whereas DiSP relies on a BERT-based selector. During ICL inference, DisJEV also achieves slightly lower average latency (68.6 vs. 69.8 seconds per 100 samples), outperforming DiSP in efficiency on four of the five datasets, with MNLI being the only exception. These results demonstrate that DisJEV achieves superior demonstration selection performance with significantly lower training costs and comparable or slightly better inference efficiency, offering a favorable trade-off between effectiveness and computational overhead.

Figure 5. Time cost comparison between DisJEV and DiSP during training and inference.

Ablation Study

To investigate the effectiveness and generalization capability of DisJEV, we consider three variants: (1) DisJEV-score, which removes the ranking distillation loss and retains only score distillation; (2) DisJEV-rank, which removes the score distillation loss and retains only ranking distillation; and (3) DisJEV-trg, which adopts a leave-one-dataset-out strategy, training the student on four datasets and evaluating it on the held-out dataset without any target-task distillation.

As shown in Table 2, removing either distillation objective leads to a decline in average accuracy, indicating that score estimation and preference ranking provide complementary supervision. The complete DisJEV achieves the best average performance on both Llama3.2-1B and Qwen3-1.7B. More importantly, DisJEV-trg achieves average accuracies of 61.21% and 75.42%, respectively, outperforming kkNN by 1.56 and 1.98 percentage points despite having no access to the target dataset during distillation. Compared with the fully trained DisJEV, the performance decreases by only 1.43 and 1.70 percentage points, respectively. These results demonstrate that the distilled utility estimator can transfer its learned knowledge to unseen tasks, supporting the cross-task generalization capability.

MethodTRECSST-2SST-5AG NewsMNLIAvg.
Llama3.2-1B
kNN72.6058.8338.1086.6542.0559.65
DisJEV-trg76.6057.8042.4086.5542.7061.21
DisJEV-score76.0558.9440.8688.8042.6961.47
DisJEV-rank75.1758.1942.1387.4343.4461.27
DisJEV80.0058.1441.9088.8844.2962.64
Qwen3-1.7B
kNN72.4090.2544.3090.3069.9573.44
DisJEV-trg78.8090.4844.9590.6072.2575.42
DisJEV-score81.4089.6844.7089.8572.9075.71
DisJEV-rank79.6089.4543.8589.6074.1175.32
DisJEV85.2089.9145.4890.1674.8577.12

Table 2. Ablation study of DisJEV at K=4K = 4. The best results are highlighted in bold.

Ranking Consistency

We further evaluate the ranking consistency between the student and teacher models on the test sets using pairwise agreement and Spearman’s rank correlation under M=20M = 20. As shown in Figure 6, both DisJEV and DisNeoJEV achieve approximately 70% pairwise agreement with their respective teachers, indicating that the distilled students effectively preserve the teachers’ relative preferences among candidate demonstrations. Notably, DisNeoJEV generally achieves higher ranking consistency than DisJEV, particularly on SST-2 and MNLI, suggesting more effective preference transfer from NeoJEV-4B. Although Spearman’s correlations are relatively moderate, the consistently positive values across all datasets further support the effectiveness of our multi-objective distillation in transferring demonstration utility knowledge.

Figure 6.

Beam Width

To investigate the impact of beam search width, we evaluate DisJEV and DisNeoJEV under different beam sizes. As shown in Figure 7, the two methods exhibit distinct sensitivities to beam width. DisNeoJEV consistently benefits from larger beam sizes, with performance continuing to improve at a beam width of 6. In contrast, DisJEV achieves its best performance at a beam width of 4, beyond which further expansion leads to performance degradation. Considering the trade-off between retrieval effectiveness and computational cost, we uniformly set the beam width to 4 in all experiments.

Figure 7. Effect of beam size on the average ICL accuracy of DisJEV and DisNeoJEV.

Conclusion

In this paper, we investigate the potential of Jev for improving demonstration retrieval in ICL. Our findings show that Jev can identify informative demonstrations beyond semantic similarity, but its computational overhead limits practical deployment. To address this, we propose DisJEV, a lightweight and task-agnostic framework that distills Jev’s utility estimation and ranking capabilities into a compact MLP. Extensive experiments demonstrate that DisJEV improves retrieval effectiveness and efficiency across diverse tasks and LLMs, while exhibiting promising cross-task generalization. Our study highlights the potential of decision-oriented model distillation for efficient and generalizable demonstration retrieval.

Details of Datasets and Settings

Datasets. Following the experimental benchmarks adopted by Wang et al. (Wang et al., [20], we evaluate our framework on five widely used text classification datasets, covering question classification, sentiment analysis, topic classification, and natural language inference.

TREC. TREC is a question classification benchmark that requires models to identify the semantic type of a given question. We adopt the six coarse-grained categories: abbreviation, entity, description, human, location, and numeric.

SST-2 and SST-5. The Stanford Sentiment Treebank provides two sentiment classification benchmarks. SST-2 formulates sentiment analysis as binary classification with positive and negative labels, whereas SST-5 introduces five fine-grained sentiment categories ranging from very negative to very positive. The latter requires distinguishing subtle differences in sentiment intensity.

AGNews. AGNews is a news topic classification dataset consisting of news articles categorized into four topics: world, sports, business, and science/technology. The task requires models to identify the primary topic of each input text.

MNLI. MNLI is a natural language inference benchmark containing premise–hypothesis pairs from diverse genres. Given a premise and a hypothesis, the model predicts one of three semantic relationships: entailment, neutral, or contradiction.

Evaluation Protocol. For each dataset, we construct a labeled demonstration pool from the training split and evaluate in-context learning on held-out examples. Demonstration selection is performed without accessing the ground-truth labels of evaluation queries. We use dataset-specific label verbalizers and prompt templates, and report classification accuracy as the primary evaluation metric. The specific prompt templates used for each dataset are provided below.

TREC ICL Input

Task: Classify the question type. Labels: abbreviation, description, entity, human, location, numeric. Instruction: Use the examples and output exactly one label. Example ii, i=1,…,Ki = 1, \ldots, K: Question: [Demonstration question] Label: [Demonstration label] Query: Question: [Target question] Label:

SST-2 ICL Input

Task: Classify the sentiment of the sentence. Labels: negative, positive. Instruction: Use the examples and output exactly one label. Example ii, i=1,…,Ki = 1, \ldots, K: Sentence: [Demonstration sentence] Label: [Demonstration label] Query: Sentence: [Target sentence] Label:

SST-5 ICL Input

Task: Classify the sentiment of the sentence. Labels: very negative, negative, neutral, positive, very positive. Instruction: Use the examples and output exactly one label. Example ii, i=1,…,Ki = 1, \ldots, K: Sentence: [Demonstration sentence] Label: [Demonstration label] Query: Sentence: [Target sentence] Label:

AG News ICL Input

Task: Classify the news topic. Labels: World, Sports, Business, Science and Technology. Instruction: Use the examples and output exactly one label. Example ii, i=1,…,Ki = 1, \ldots, K: Article: [Demonstration article] Label: [Demonstration label] Query: Article: [Target article] Label:

MNLI ICL Input

Task: Determine the relation between the premise and hypothesis. Labels: entailment, neutral, contradiction. Instruction: Use the examples and output exactly one label. Example ii, i=1,…,Ki = 1, \ldots, K: Premise: [Demonstration premise] Hypothesis: [Demonstration hypothesis] Label: [Demonstration label] Query: Premise: [Target premise] Hypothesis: [Target hypothesis] Label:

More Results of Preliminary Experiment

Table 3 reports the classification accuracy of different demonstration retrieval methods across five datasets, four downstream LLMs, and varying numbers of demonstrations. Overall, JEv-guided reranking generally outperforms conventional kkNN retrieval, suggesting that semantic similarity alone is insufficient to identify the most informative demonstrations. The improvements become more pronounced as the number of demonstrations increases. At K=8K = 8, JevICL improves the average accuracy over kkNN by 3.34, 4.45, 2.77, and 1.23 percentage points on Llama3.2-1B, Qwen3-1.7B, Llama3.1-8B, and Qwen3-8B, respectively. These results demonstrate the potential of decision-oriented models to improve demonstration selection beyond semantic similarity.

Llama ModelsQwen Models
MethodKKTRECSST-2SST-5AG NewsMNLIAvg.MethodKKTRECSST-2SST-5AG NewsMNLIAvg.
Llama3.2-1BQwen3-1.7B
Zero-shot040.0089.6834.0574.1541.8055.94Zero-shot053.6081.5431.0033.9550.4550.11
kNN178.4072.1339.2574.0039.3560.63kNN165.0087.9640.4083.9567.0568.87
NeoJevICL180.6065.9438.5074.7037.5559.46NeoJevICL177.6085.7843.8584.8070.3072.47
JevICL178.2064.9136.3074.7534.5057.73JevICL178.2088.5343.4584.0069.8572.81
kNN264.2056.3133.8082.9534.4054.33kNN267.0087.9642.0090.3570.5071.56
NeoJevICL267.8051.8337.1584.4536.6555.58NeoJevICL286.0087.0446.2091.4573.7076.88
JevICL265.6053.6736.3083.8033.4054.55JevICL283.8087.9644.4590.1574.1576.10
kNN472.6058.8338.1086.6542.0559.65kNN472.4090.2544.3090.3069.9573.44
NeoJevICL485.0053.5643.6090.9043.9063.39NeoJevICL492.0091.6350.7592.0573.5079.99
JevICL482.0054.2444.6588.7041.1562.15JevICL489.6091.1749.0590.8074.0578.93
kNN677.6061.0138.9087.9546.3062.35kNN675.6090.0243.8590.8568.7573.81
NeoJevICL688.4057.2245.9592.0548.5566.43NeoJevICL690.2092.7850.8592.1074.4080.07
JevICL688.6056.6545.5090.4048.3065.89JevICL689.0091.4048.7091.3074.4578.97
kNN879.0065.4839.1090.6046.3564.11kNN876.8088.7644.4591.2567.9573.84
NeoJevICL889.2061.1244.8092.0049.9567.41NeoJevICL890.2092.4351.7092.1574.1580.13
JevICL887.0061.5845.6590.5552.4567.45JevICL887.4091.5148.8591.3572.3578.29
Llama3.1-8BQwen3-8B
Zero-shot076.8097.1351.0078.2047.8070.19Zero-shot069.8094.2744.2082.7070.8072.35
kNN181.2096.3355.5085.8552.7574.33kNN176.8094.3850.5590.9575.7577.69
NeoJevICL186.4095.3057.5586.3061.2077.35NeoJevICL190.4094.5052.9091.5577.1581.30
JevICL187.4095.8755.2085.3557.4576.25JevICL190.2093.3551.3090.8078.6080.85
kNN279.8093.5852.5588.8050.8573.12kNN283.0093.2350.8591.7578.1579.40
NeoJevICL292.4093.2356.3590.4556.0577.70NeoJevICL294.6094.7255.2092.0080.5083.40
JevICL292.6093.4655.8088.7555.4077.20JevICL292.8092.8952.5590.9581.4082.12
kNN478.2094.7254.2589.8057.0574.80kNN488.4093.2349.8092.2078.4080.41
NeoJevICL495.0094.1558.1091.7060.2579.84NeoJevICL495.8093.8154.8092.4582.7583.92
JevICL493.6094.3857.3090.6062.5079.68JevICL493.2092.6653.0591.3581.7082.39
kNN683.2094.9556.6590.4060.6077.16kNN691.0093.6949.7091.9079.9081.24
NeoJevICL694.4094.8460.3592.4562.8580.98NeoJevICL695.8094.1553.6092.4581.4583.49
JevICL694.2094.6160.2091.2564.4080.93JevICL693.6093.6952.8591.6082.9082.93
kNN884.8095.1856.4090.3563.3077.99kNN892.6093.5850.4092.0080.0581.73
NeoJevICL895.2094.0460.2592.4565.3081.45NeoJevICL894.2093.6953.5592.4582.4583.27
JevICL893.4093.4660.1591.0065.8080.76JevICL893.2093.8153.2591.9082.6582.96

Table 3. Classification accuracy (%) of kkNN, NeoJevICL, and JevICL across different numbers of demonstrations (KK). The best result for each model, shot setting, and dataset is highlighted in bold.

Interestingly, NeoJevICL achieves comparable or even better performance than JevICL despite using a smaller 4B-parameter evaluator. At K=8K = 8, NeoJevICL achieves average accuracies of 67.41%, 80.13%, 81.45%, and 83.27% across the four downstream LLMs, outperforming JevICL on three of them. This observation is consistent with the stronger discriminative capabilities reported for NeoJEV-4B in its official evaluation, suggesting that the quality of decision-making supervision, rather than model size alone, is crucial for effective demonstration reranking.

We also observe that the effectiveness of JEV-guided retrieval varies across downstream models and demonstration budgets. For Llama3.2-1B, improvements are less stable in low-shot settings, whereas both JevICL and NeoJevICL generally provide more substantial gains when K≥4K \geq4. In contrast, the Qwen models exhibit more consistent improvements, particularly with NeoJevICL. Moreover, the relative gains tend to diminish for larger downstream LLMs, suggesting that stronger models may be less sensitive to demonstration quality. Nevertheless, the overall results highlight the effectiveness of decision-guided retrieval and motivate distilling such capabilities into a lightweight demonstration selector.

Beam Size Details

To investigate the detailed impact of beam width on demonstration selection, we evaluate DisJEV and DisNeoJEV across different datasets and LLMs. As shown in Figure 8, although the optimal beam width varies across datasets and teacher models, moderately increasing the beam width generally improves retrieval performance. In particular, Dis-NeoJEV often benefits from larger beam widths, whereas DisJEV tends to achieve its best performance at smaller widths. These results suggest that exploring more demonstration combinations is generally beneficial, although the gains are not strictly monotonic. Considering the overall performance and computational cost, we uniformly set the beam width to 4 in all experiments.

Figure 8. Detailed results across different beam sizes.

References

  1. [1]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.arxiv.org/abs/2005.14165
  2. [2]Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89–96.DOI
  3. [3]Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa. 2026. Evaluating and benchmarking the system one model jev. arXiv preprint arXiv:2609.37647.
  4. [4]Feiyu Duan, Jiayu Lin, Jia Wang, Jun Xiang, Jialiang Wu, Xinnong Zhang, Hanqi Yan, Siyuan Wang, and Zhongyu Wei. 2026. General decision models: Benchmarking and insights beyond jev. arXiv preprint arXiv:2610.03935.arxiv.org/abs/2610.03935
  5. [5]SU Hongjin, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. 2022. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2209.01975
  6. [6]Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, Yichong Xu, and Chenguang Zhu. 2023. In-context demonstration selection with cross entropy difference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1150–1162.
  7. [7]Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. Unified demonstration retriever for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4644–4668.arxiv.org/abs/2305.04320
  8. [8]Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu, Shenlong Zhang, Yijia Zhang, Ling Luo, and Hongfei Lin. 2026a. Specialized decision models vs. general-purpose llms: Benchmarking jev across knowledge, reasoning, and multilingual tasks. arXiv preprint arXiv:2610.11978.arxiv.org/abs/2610.11978
  9. [9]Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. 2026b. Jev-as-a-judge: Accept when confident, escalate when unsure. arXiv preprint arXiv:2609.26550.arxiv.org/abs/2609.26550
  10. [10]Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd workshop on knowledge extraction and integration for deep learning architectures, pages 100–114.
  11. [11]Lingsen Meng. 2026. How far can a commercial decision model’s probabilities be trusted? a calibration audit of jev against open and general-purpose classifiers.
  12. [12]Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2024. Revisiting demonstration selection strategies in in-context learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9090–9101.
  13. [13]Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and trends® in information retrieval, 4(1-2):1–174.
  14. [14]Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 2655–2671.
  15. [15]Rui Song, Fausto Guinchiglia, Yingji Li, Lida Shi, and Hao Xu. 2025. Counterfactual contrastive learning for robust text classification based on word group search. Information Sciences, page 122579.DOI
  16. [16]Rui Song, Yingji Li, Fausto Guinchiglia, and Hao Xu. 2026a. Retrieval augmentation for out-of-distribution robustness in non-knowledge intensive in-context learning. Science China Information Sciences, 69(2):122108.DOI
  17. [17]Rui Song, Yingji Li, Jian Li, Fausto Guinchiglia, and Hao Xu. 2026b. From fake to real: Mitigating out-of-distribution bias in in-context learning via feedback supervision from large language models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 15197–15213. Association for Computational Linguistics.DOI
  18. [18]Rui Song, Yingji Li, Lida Shi, Fausto Guinchiglia, and Hao Xu. 2024. Shortcut learning in in-context learning: A survey. arXiv preprint arXiv:2411.02018.arxiv.org/abs/2411.02018
  19. [19]TypeSafe AI. 2026. Introducing system one models and jev. TypeSafe AI Blog.
  20. [20]Haochun Wang, Chaofen Yang, Jiatong Liu, Jingbo Wang, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2026. Easier to judge than to find: Predicting in-context learning success for demonstration selection. arXiv preprint arXiv:2605.18512.arxiv.org/abs/2605.18512
  21. [21]Liang Wang, Nan Yang, and Furu Wei. 2024. Learning to retrieve in-context examples for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1752–1767.arxiv.org/abs/2307.07164
  22. [22]Hao Xu, Rite Bo, Fausto Giunchiglia, Yingji Li, and Rui Song. 2026. Toward robust in-context learning: Leveraging out-of-distribution proxies for target inaccessible demonstration retrieval. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 20540–20556.
  23. [23]Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116.arxiv.org/abs/2402.13116
  24. [24]Zhao Yang, Yuanzhe Zhang, Dianbo Sui, Cao Liu, Jun Zhao, and Kang Liu. 2023. Representative demonstration selection for in-context learning with two-stage determinantal point process. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5443–5456.
  25. [25]Zheng Zhang, Shaocheng Lan, Lei Song, Jiang Bian, Yexin Li, and Kan Ren. 2025. Learning to select in-context demonstration preferred by large language model. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11345–11360.arxiv.org/abs/2505.19966

Paper details

vlnc:W_fs7845bs
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents