WorksComputer Science
Jev-as-a-Judge for Brazilian Portuguese Information Retrieval: Relevance Assessment and Reranking
Abstract
This study investigates Jev, a System One model designed for fast, structured decision-making, in information retrieval. We use Jev-as-a-judge to generate relevance judgments for four Brazilian Portuguese IR collections covering different domains and compare them with the official relevance judgments of each collection. We also evaluate Jev as a reranker. For relevance assessment, exact agreement ranged from 52.9% to 64.9%, while quadratic weighted Cohen's kappa ranged from 0.52 to 0.82. Most disagreements involved adjacent relevance levels, with at least 94.5% of annotations matching exactly or differing by only one relevance level. Despite differences at the individual query–document level, Jev-generated qrels produced similar retrieval system rankings, with Kendall's tau ranging from 0.90 to 0.93 for NormasTCU and Ulysses-RFCorpus across the two evaluation metrics. For reranking, Jev reordered the top 1,000 BM25 candidates for each of the 50 Quati test queries, increasing nDCG@10 from 0.3329 to 0.6331 (+90.2%) and MRR from 0.6852 to 0.9033 (+31.8%). These findings support Jev as a relevance assessor and show improved ranking effectiveness over BM25 in the evaluated reranking setting.
Introduction
Information Retrieval (IR) systems are designed to identify relevant documents from a collection in response to a user query [1]. A typical search system receives a query, retrieves a set of candidate documents, ranks them according to their estimated relevance, and presents the results to the user. Different retrieval methods can be used throughout this pipeline, and in recent years, large language models (LLMs) have been incorporated into several of its stages.
LLMs can be used to support different retrieval tasks, including query expansion, document expansion, and dense retrieval [2, 3, 4]. They can also be used to make decisions about relevance, either to rerank candidate documents within a retrieval pipeline or to produce the relevance judgments used to build test collections. This paper focuses on these two applications: reranking candidate documents within a retrieval pipeline and producing relevance judgments for test collections.
Among these decision-based applications, relevance assessment has received particular attention. The evaluation of retrieval systems, for example, depends on relevance judgments indicating how well retrieved documents satisfy a user’s information need. These judgments are organized as query relevance judgments (qrels) and are used to compute retrieval effectiveness measures and compare retrieval systems [5]. Producing large numbers of relevance judgments manually can be costly and time-consuming [6], especially for large collections or specialized domains. Consequently, recent work has investigated the use of LLMs to automatically generate relevance judgments or to assist human assessors in the annotation process [6, 7], with some studies showing that LLMs can produce system rankings similar to those obtained from human judgments [6].
This ability to assess the relevance of documents with respect to a query has also been applied to reranking. In a typical two-stage architecture, an efficient first-stage retriever generates a smaller candidate set, which is then processed by a more computationally expensive reranking model. The reranker examines the query and each candidate document and produces a more accurate ranking [8, 9].
Generative language models are powerful and flexible, but using them for large-scale relevance assessment or reranking can require substantial computational resources and incur significant monetary costs. This is particularly relevant when thousands or millions of query–document pairs must be evaluated, or when a reranker must process many candidate documents for each query. Consequently, reducing the cost of model-based decisions while retaining useful language understanding is an important practical problem.
A recent alternative is the use of “System One models”, a term inspired by the System 1/System 2 terminology popularized by Kahneman [10]. In psychology, System 1 refers to fast, automatic, and intuitive reasoning, while System 2 refers to slower and more deliberate reasoning. In September 2026, TypeSafe AI introduced the term “System One models” for models designed for fast, structured decision-making and presented Jev as an example [11].
Unlike conventional LLMs, Jev can interpret natural-language inputs but does not generate natural-language responses. It receives a state and typed questions and returns structured decisions, including probabilities and confidence information. Queries and documents can therefore be provided as part of the state used for decision-making. Jev is designed for tasks such as classification, scoring, routing, filtering, and other bounded decision-making tasks rather than open-ended text generation. It also has substantially lower computational costs and is reported to be up to two orders of magnitude less expensive to operate than frontier LLMs [11].
This design is relevant to information retrieval. Although some IR tasks depend on text generation, such as query and document expansion, others can be reformulated according to the structured decision-making paradigm. For example, relevance assessment is a decision task: given a query and a document, an assessor determines the document’s relevance using a predefined relevance scale. Similarly, reranking requires a model to estimate the relative relevance of candidate documents to a query. Both tasks can be formulated as structured decision problems.
Our contribution is to investigate two applications of Jev in IR using collections in Brazilian Portuguese. We focus on Brazilian Portuguese because most research on information retrieval has focused on English, while other languages receive less attention and have fewer resources for training and evaluation. This makes Brazilian Portuguese a useful setting for testing whether Jev can support IR tasks beyond the language settings most commonly studied. First, we evaluate Jev as a relevance assessor by comparing its judgments with existing relevance assessments (reference qrels) and measuring how well they preserve retrieval system rankings. Second, we evaluate pointwise Jev reranking of BM25 candidates on Quati.
The study is guided by the following research questions (RQ):
RQ1: To what extent do Jev-generated relevance judgments agree with reference relevance judgments across Brazilian Portuguese IR datasets from different domains?
RQ2: To what extent do Jev-generated qrels preserve the relative ranking of retrieval systems obtained using the reference qrels?
RQ3: To what extent does Jev improve the effectiveness of BM25 on Quati when used as a reranker?
Related work
LLM-based relevance assessment
Recent studies have investigated the use of generative LLMs for automatic relevance assessment. Faggioli et al. discussed the benefits and risks of using LLMs for this task, including potential biases, sensitivity to assessment conditions, and differences from human judgments [12]. Thomas et al. showed that LLMs can label documents for relevance, with performance comparable to TREC judges and substantially better than crowd judges [13]. Upadhyay et al. reproduced and extended this line of work by introducing UMBRELA, an open toolkit for generating relevance judgments with LLMs [7].
Upadhyay et al. later conducted a large-scale comparison of manual, LLM-based, and human-in-the-loop relevance assessment strategies using UMBRELA [6]. They found high correlations between system rankings based on LLM-generated and reference judgments for nDCG@20, nDCG@100, and Recall@100. However, the correlations were lower when systems were compared for individual topics. These results show that LLM-generated relevance judgments can produce system-level evaluation results similar to those obtained from reference judgments. Differences can still occur at the query–document level.
LLM-based relevance assessment has also been studied on Brazilian Portuguese collections, with similar results. Despite differences in query–document-level judgments, LLM-generated qrels produced similar system rankings for nDCG@10, nDCG@30, and MRR [14, 15], but not for P@10 and R@10 [15].
LLM-based reranking
Language models have been widely used to rerank candidate documents retrieved by first-stage retrieval systems. Early approaches used pretrained models such as BERT to estimate the relevance of query–document pairs [8]. Later work explored sequence-to-sequence models such as monoT5 and RankT5, showing that pretrained text-to-text models can also be effectively adapted for passage reranking [16, 17]. Multilingual variants have also been studied. Jeronymo et al., for example, evaluated an mT5-XXL reranker in the TREC 2022 NeuCLIR track and found that it was effective for cross-lingual retrieval across multiple languages [18].
More recently, researchers have studied generative LLMs as zero-shot rerankers. In this approach, the model directly ranks candidate documents based on their relevance to a query. RankGPT introduced this approach using GPT models, and subsequent work explored pairwise, setwise, and listwise ranking strategies [19, 20, 21, 22]. These approaches show a trade-off between ranking effectiveness and computational cost: pairwise and listwise strategies can improve ranking quality, but they require multiple model inferences or large prompts, which makes LLM-based reranking expensive for large candidate sets [20, 21].
System One Models and Jev
System One models are a recently introduced class of models designed for fast, structured decision-making. TypeSafe AI introduced the category in 2026 and presented Jev as a model that receives a state and typed questions and returns structured decisions rather than open-ended natural-language responses [11].
Initial studies have investigated Jev as a general-purpose decision model and evaluator in tasks such as rubric-based evaluation [23], evidence-grounded factuality assessment [24], scientific decision-making [25], security evaluation of agent traces [26], and recommendation reranking [27]. Jev has also been evaluated on a wide range of classification, routing, reasoning, and rubric-scoring tasks [28].
Recent technical reports have also explored Jev for search reranking. Coates evaluates product-search reranking using relevance-class probabilities, auxiliary decision signals, and application-defined scoring policies [29]. Bartholomew describes conversational-memory reranking using a single choice over candidate memories, with a separate optional filtering step [30]. Tateno introduces a library supporting both pointwise and listwise relevance scoring and reports listwise reranking and filtering results on NanoHotpotQA [31]. In personalized recommendation, Lyu and Xia use interaction histories as decision context and rank candidate items by their structured-choice probabilities [27].
Our reranking formulation uses the estimated probability that a document answers the query, with multiple questions grouped into each request. It does not ask candidates to compete within a single choice or combine multiple domain-specific decision signals. The contribution of this study lies in its evaluation in Brazilian Portuguese, combining relevance-judgment generation and system-ranking preservation with reranking of BM25 candidates on Quati.
Method
Jev-as-a-judge for relevance assessment
To evaluate Jev-as-a-Judge for relevance assessment, we used Jev to generate relevance judgments for four recently released Brazilian Portuguese IR collections and compared the resulting qrels with the reference qrels provided with each collection.
The collections were selected to cover different domains, document types, query formats, and relevance-judgment approaches. Quati is a general-domain retrieval collection composed of short passages extracted from web pages [32]. JurisTCU, NormasTCU, and Ulysses-RFCorpus contain full-length documents. JurisTCU is a collection of jurisprudential documents from the Brazilian Federal Court of Accounts [33], NormasTCU contains normative acts issued by the same institution [15], and Ulysses-RFCorpus contains legislative proposals from the Brazilian Chamber of Deputies [34].
The collections also differ in how their query–document relevance judgments were generated. Quati and JurisTCU use four relevance levels and use LLM-based approaches to generate the reference labels. JurisTCU also includes a review step by a domain expert. NormasTCU and Ulysses-RFCorpus use three relevance levels annotated by domain experts. In NormasTCU, up to four experts annotated each query–document pair. In Ulysses-RFCorpus, one expert annotated each pair.
Table 1 summarizes the collections used in the analysis. For each collection, we submitted to Jev the query–document pairs with available reference relevance judgments. To generate the Jev qrels, each pair was presented as a state containing the query and document text, followed by a single typed choice question1. We asked Jev version jev-1.13 to select the relevance level using the same labels and wording as those used in the original collections.
| Collection | Domain | Queries | Pairs | Relevance levels | Reference-label source |
| Quati (test set) | Web passages | 50 | 1,933 | 0-3: Irrelevant; Slightly Relevant; Relevant; Highly relevant | LLM-generated |
| JurisTCU | Jurisprudence | 150 | 2,250 | 0-3: Irrelevant; Related; Relevant; Highly relevant | LLM-generated; reviewed by one domain expert |
| NormasTCU | Normative acts | 46 | 812 | 0-2: Irrelevant; Partially Relevant; Relevant | Four expert judgments |
| Ulysses-RFCorpus | Legislative | 692 | 8,205 | 0-2: Irrelevant; Partially Relevant; Relevant | Expert relevance feedback |
Table 1. query–document pairs used in the Jev annotation analysis.
For each test collection, we compared Jev’s selected label with the reference label for each query–document pair using a confusion matrix. We also calculated quadratic weighted Cohen’s across all pairs in each collection using the original ordered labels. The quadratic weighting assigns greater penalties to disagreements between more distant relevance levels.
However, agreement at the individual query–document level does not necessarily show whether differences in relevance judgments affect the evaluation of retrieval systems. To assess the practical impact of these differences, we evaluated whether Jev-generated relevance judgments led to system rankings similar to those obtained using the reference qrels. We performed this analysis for NormasTCU and Ulysses-RFCorpus because human relevance judgments are available for these collections and previous work has already evaluated multiple retrieval systems on them.
For NormasTCU, we considered the 15 retrieval systems evaluated in [15]. For Ulysses-RFCorpus, we considered the 16 systems evaluated in [35]. Table 2 and Table 3 summarize the systems considered for each collection. Further details on the systems evaluated in each collection are provided in the respective original studies.
| Name | Description | Indexed fields |
| BM25 with | SUMMARY+TEXT | |
| BM25 with | TEXT | |
| BM25 with | SUMMARY+TEXT | |
| BM25 with | TEXT | |
| BM25 with | SUMMARY+TEXT | |
| BM25 with | TEXT | |
| BM25 with | SUMMARY+TEXT | |
| BM25 with | TEXT | |
| Semantic search. Model: text-embedding-3-large | TEXT | |
| Semantic search. Model: text-embedding-3-small | TEXT | |
| Semantic search. Model: gemini-embedding-001 | TEXT | |
| Semantic search. Model: Qwen/Qwen3-Embedding-4B | SUMMARY+TEXT | |
| Semantic search. Model: Qwen/Qwen3-Embedding-8B | SUMMARY+TEXT | |
| Top-50 results from reranked with text-embedding-3-small | TEXT | |
| Top-50 results from reranked with gemini-embedding-001 | TEXT |
Table 2. Retrieval systems used in the system-ranking analysis for NormasTCU, as reported by [15], varying in retrieval method and indexed fields.
| Name | Retriever | Document representation |
| HC (bm25) | BM25 | Hierarchical Chunks |
| HC (small) | text-embedding-3-small | Hierarchical Chunks |
| HC (jua) | jua-4B-mixed | Hierarchical Chunks |
| HC (qwen) | Qwen3-Embedding-4B | Hierarchical Chunks |
| CC + FC (bm25) | BM25 | Core Chunks + Full Context |
| CC + FC (small) | text-embedding-3-small | Core Chunks + Full Context |
| CC + FC (jua) | jua-4B-mixed | Core Chunks + Full Context |
| CC + FC (qwen) | Qwen3-Embedding-4B | Core Chunks + Full Context |
| CC + MT (bm25) | BM25 | Core Chunks + Metadata |
| CC + MT (small) | text-embedding-3-small | Core Chunks + Metadata |
| CC + MT (jua) | jua-4B-mixed | Core Chunks + Metadata |
| CC + MT (qwen) | Qwen3-Embedding-4B | Core Chunks + Metadata |
| CC + NC (bm25) | BM25 | Core Chunks + No Context |
| CC + NC (small) | text-embedding-3-small | Core Chunks + No Context |
| CC + NC (jua) | jua-4B-mixed | Core Chunks + No Context |
| CC + NC (qwen) | Qwen3-Embedding-4B | Core Chunks + No Context |
Table 3. Retrieval systems used in the system-ranking analysis for Ulysses-RFCorpus, as reported by [35], varying in retrieval model and document representation.
For each system in both collections, we recomputed nDCG@10 and MRR scores using the Jev-generated qrels and compared the resulting system ranking with the ranking obtained using the reference qrels. Finally, we calculated Kendall’s between the two rankings as a measure of their agreement.
Jev-as-a-judge for reranking
We formulate reranking as a pointwise relevance estimation task within a two-stage retrieval pipeline. Given a query , the first-stage retriever provides a ranked list of candidate documents. Jev assigns a relevance score to each document in the selected candidate set , consisting of the top- retrieved documents available in the corpus. The candidates are then sorted in descending order of their Jev scores.
Unlike the discrete relevance labels used in § 3.1, reranking uses a probability estimate for each candidate document. Jev receives the query and a question asking, in Portuguese, whether the candidate document directly answers it. The instruction also asks Jev to use the probability of relevance as the reranking score and to treat the document as search content rather than as an instruction. When available, the document title is included before its text. Thus, the score is
where the returned value estimates the probability that the document directly answers the query. We use this value directly, without combining it with the first-stage retrieval score or fitting a score calibration model.
The implementation groups multiple document-specific questions for the same query into a single API request. Each question concerns one query–document pair; the model is not asked to generate a permutation or to make explicit pairwise comparisons. Scores from all batches are collected before sorting the candidate set. Document-text whitespace is normalized, and the implementation supports truncation to a configurable character limit before submission; the title is supplied separately and is not included in this limit. The reranking depth, request batch size, and text limit are configurable experimental parameters. When the original ranking tail is retained, documents outside the reranked candidate set are appended below it in their original input order.
We used Jev version jev-1.13.0 to rerank the top 1,000 BM25 candidates for each of the 50 Quati test queries on the 1M-passage corpus, using 100 requests per query with 10 document-specific questions per request. The scored documents corresponded exactly to the top 1,000 candidates in the BM25 run.
To assess reranking effectiveness, we evaluated the original BM25 ranking and its Jev-reranked counterpart on the same queries using the collection’s reference qrels. We computed nDCG using linear relevance gains; for MRR, documents with a reference relevance label greater than zero were considered relevant. All reported scores are macro-averages across queries. This separates retrieval evaluation from the use of Jev-generated judgments studied in § 3.1.
Results and Discussion
Jev-as-a-judge for relevance assessment
Tables 4, 5, 6, and 7 present the confusion matrices obtained by comparing Jev-generated qrels with the reference qrels for the four collections.
Exact agreement between Jev and the reference qrels varied across collections, with accuracy ranging from 53% to 65%. Ulysses-RFCorpus had the highest accuracy (64.9%), followed by JurisTCU (64.3%), NormasTCU (54.8%), and Quati (52.9%).
Most disagreements were limited to one relevance level. The proportion of query–document pairs with a difference of one level ranged from 29.7% in Ulysses-RFCorpus to 44.5% in Quati. Differences of two levels were less frequent, ranging from 2.5% in Quati to 5.5% in NormasTCU. For the collections with four relevance levels, differences of three levels were absent in Quati and occurred in only 0.1% of the pairs in JurisTCU. When Jev disagreed with the reference qrels, the disagreement was usually between adjacent relevance levels rather than between the extremes of the relevance scale.
To further assess the agreement between Jev and the reference qrels, we computed Cohen’s quadratic weighted . While accuracy measures the proportion of exact matches, accounts for agreement expected by chance and uses quadratic weighting to assign larger penalties to disagreements between more distant relevance levels. The values ranged from 0.52 to 0.82 across the four
| Reference | Jev 0 | Jev 1 | Jev 2 | Jev 3 |
| 0 | 473 (24.5%) | 518 (26.8%) | 48 (2.5%) | 0 (0.0%) |
| 1 | 8 (0.4%) | 273 (14.1%) | 107 (5.5%) | 0 (0.0%) |
| 2 | 0 (0.0%) | 45 (2.3%) | 268 (13.9%) | 5 (0.3%) |
| 3 | 0 (0.0%) | 1 (0.1%) | 178 (9.2%) | 9 (0.5%) |
| Accuracy | 52.9% | |||
| Difference of 1 level | 44.5% | |||
| Difference of 2 levels | 2.5% | |||
| Difference of 3 levels | 0.0% | |||
| 0.68 |
Table 4. Confusion matrix comparing Jev-generated and reference relevance judgments for Quati.
| Reference | Jev 0 | Jev 1 | Jev 2 | Jev 3 |
| 0 | 400 (17.8%) | 22 (1.0%) | 6 (0.3%) | 2 (0.1%) |
| 1 | 194 (8.6%) | 224 (10.0%) | 86 (3.8%) | 24 (1.1%) |
| 2 | 1 (0.0%) | 122 (5.4%) | 195 (8.7%) | 195 (8.7%) |
| 3 | 0 (0.0%) | 72 (3.2%) | 80 (3.6%) | 627 (27.9%) |
| Accuracy | 64.3% | |||
| Difference of 1 level | 31.1% | |||
| Difference of 2 levels | 4.6% | |||
| Difference of 3 levels | 0.1% | |||
| 0.82 |
Table 5. Confusion matrix comparing Jev-generated and reference relevance judgments for JurisTCU.
| Reference | Jev 0 | Jev 1 | Jev 2 |
| 0 | 282 (34.7%) | 197 (24.3%) | 34 (4.2%) |
| 1 | 22 (2.7%) | 71 (8.7%) | 52 (6.4%) |
| 2 | 11 (1.4%) | 51 (6.3%) | 92 (11.3%) |
| Accuracy | 54.8% | ||
| Difference of 1 level | 39.7% | ||
| Difference of 2 levels | 5.5% | ||
| 0.52 |
Table 6. Confusion matrix comparing Jev-generated and reference relevance judgments for NormasTCU.
| Reference | Jev 0 | Jev 1 | Jev 2 |
| 0 | 2991 (36.5%) | 873 (10.6%) | 179 (2.2%) |
| 1 | 504 (6.1%) | 696 (8.5%) | 213 (2.6%) |
| 2 | 266 (3.2%) | 848 (10.3%) | 1635 (19.9%) |
| Accuracy | 64.9% | ||
| Difference of 1 level | 29.7% | ||
| Difference of 2 levels | 5.4% | ||
| 0.65 |
Table 7. Confusion matrix comparing Jev-generated and reference relevance judgments for Ulysses-RFCorpus.
collections. JurisTCU had the highest value (0.82), followed by Quati (0.68), Ulysses-RFCorpus (0.65), and NormasTCU (0.52).
Disagreement among relevance annotators is well documented, as relevance assessment is inherently subjective [36]. For example, McDonnell et al. evaluated different annotation configurations and reported values ranging from 0.34 to 0.85 when re-annotating an existing dataset [37]. A similar range was reported in [15], where the pairwise values among human annotators ranged from 0.55 to 0.73. Jev’s values overlap the ranges reported in these human-annotation studies. However, differences in collections, annotation protocols, and reference judgments limit direct comparisons.
The analysis above shows how closely Jev’s individual relevance judgments match the reference qrels. However, the practical importance of these differences depends on whether they change the conclusions of retrieval system evaluations. We therefore also examined the agreement at the system-ranking level, comparing the rankings obtained with the reference qrels and Jev-generated qrels.
For each collection, we ranked the systems listed in Table 2 and Table 3 by nDCG@10 and MRR, first using the reference qrels and then the Jev-generated qrels. We compared the resulting rankings for each metric.
Figure 1 and Figure 2 visualize these comparisons as heatmaps. The first row shows the reference ranking and the second the ranking based on Jev-generated qrels. Each system retains the color and numerical label of its reference position, making changes in order visible across the rows.

Figure 1. Heatmaps showing the rankings of the retrieval systems for NormasTCU using the reference and Jev-generated qrels. Each cell shows the rank of a system, both numerically and by color, with lower values indicating better rankings (rank 1 = best). The columns represent the fifteen IR systems and follow the reference ranking. The left panel shows nDCG@10 and the right panel shows MRR. Differences from the reference ranking are reflected in changes in the color pattern and system order.

Figure 2. Heatmaps showing the rankings of the retrieval systems for Ulysses-RFCorpus using the reference and Jev-generated qrels. Each cell shows the rank of a system, both numerically and by color, with lower values indicating better rankings (rank 1 = best). The columns represent the sixteen IR systems and follow the reference ranking. The left panel shows nDCG@10 and the right panel shows MRR. Differences from the reference ranking are reflected in changes in the color pattern and system order.
For nDCG@10, there was little variation in the rankings for both collections. In NormasTCU, the system initially ranked eighth moved to eleventh position, while systems , , and each moved up by one position. In Ulysses-RFCorpus, only a small number of systems changed positions. The MRR rankings showed similar overall stability. However, more systems changed positions in both collections, including some of the top-ranked systems.
To measure the similarity between the rankings obtained with the reference qrels and Jev-generated qrels, Table 8 shows Kendall’s between the two rankings. The values ranged from 0.90 to 0.93, showing high agreement between the rankings. For NormasTCU, was 0.92 for nDCG@10 and 0.90 for MRR. For Ulysses-RFCorpus, the values were 0.93 and 0.92, respectively.
| Collection | nDCG@10 | MRR |
| NormasTCU | 0.92 | 0.90 |
| Ulysses-RFCorpus | 0.93 | 0.92 |
Table 8. Kendall’s between system rankings obtained with the reference qrels and Jev-generated qrels.
Voorhees notes that correlations between rankings produced using different relevance assessors are generally greater than 0.9. She therefore argues that evaluation schemes with correlations of at least 0.9 should be considered equivalent, since it is not possible to make a more precise comparison than this [38]. Under this criterion, the rankings produced with Jev-generated qrels were equivalent to those obtained with the reference qrels for both collections and both evaluation metrics.
These results allow us to answer the first two research questions:
RQ1: To what extent do Jev-generated relevance judgments agree with reference relevance judgments across Brazilian Portuguese IR datasets from different domains?
Answer: Exact agreement ranged from 52.9% to 64.9%, with the highest value observed for Ulysses-RFCorpus and the lowest for Quati. Most disagreements were between adjacent relevance levels. Across all collections, at least 94.5% of the annotations showed either exact agreement or a difference of no more than one relevance level. Quadratic weighted ranged from 0.52 to 0.82. These values overlap ranges reported in previous human-annotation studies, although differences in evaluation settings limit direct comparisons. Agreement varied across collections, and most disagreements concerned adjacent relevance levels.
RQ2: To what extent do Jev-generated qrels preserve the relative ranking of retrieval systems obtained using the reference qrels?
Answer: Jev-generated qrels preserved the relative ranking of retrieval systems to a high degree. Kendall’s ranged from 0.90 to 0.93 across NormasTCU and Ulysses-RFCorpus for nDCG@10 and MRR. All values met the threshold that Voorhees considers sufficient to indicate functional equivalence between rankings [38]. Thus, although Jev-generated qrels differed from the reference qrels at the query–document level, these differences had limited impact on the relative ranking of retrieval systems in this analysis.
Jev-as-a-judge for reranking
Table 9 compares our BM25 and Jev runs with the published mT5-XL baseline for the Quati 1M corpus. The published BM25 + mT5-XL pipeline uses the multilingual text-to-text model mT5 [39] as a reranker. Its nDCG@10 score is reproduced from the original Quati study [32]; MRR was not reported there.
Table 9. Effectiveness on the 50 Quati test queries using the 1M-passage corpus. The published mT5-XL baseline is reproduced from [32]; dashes indicate unreported values. Jev reranks the top 1,000 candidates from our BM25 run. Published and current runs use different experimental configurations.
Within our experiment, Jev increased nDCG@10 from 0.3329 to 0.6331, an absolute gain of 0.3001 and a relative improvement of 90.2%. MRR increased from 0.6852 to 0.9033 (+31.8%), indicating that the first relevant document generally appeared earlier in the reranked lists. These gains reflect changes in the ordering of the retrieved candidates; relevant documents outside the top 1,000 cannot be recovered by this reranking step.
Our BM25 + Jev result of 0.6331 is numerically below the published BM25 + mT5-XL result of 0.6593. Both evaluations use the 1M-passage corpus, but their BM25 scores also differ (0.3329 versus 0.3991). Without aligning the retrieval configuration, candidate sets, qrels, and metric implementation, these cross-study values provide context but do not establish a controlled comparison between Jev and mT5-XL.
These results support the effectiveness of probability-based pointwise reranking on Quati. However, this experiment covers a single collection of web passages and does not establish that the same gains hold for full-length legal documents. We report descriptive differences without claiming statistical significance. We did not evaluate latency or monetary cost, so effectiveness gains alone do not establish an efficiency advantage over other rerankers.
RQ3: To what extent does Jev improve the effectiveness of BM25 on Quati when used as a reranker?
Answer: On the 50 Quati test queries, reranking the top 1,000 BM25 candidates with Jev increased nDCG@10 from 0.3329 to 0.6331 (+90.2%) and MRR from 0.6852 to 0.9033 (+31.8%). These findings demonstrate improved ranking effectiveness in this setting; further experiments are needed to assess generalization to other collections and retrieval systems.
Conclusions
This paper presented an empirical evaluation of Jev, a System One model designed for fast, structured decision-making, in two information retrieval tasks: relevance assessment and reranking. We conducted experiments on Brazilian Portuguese IR collections covering different domains, document types, and relevance-judgment approaches.
For relevance assessment, agreement between Jev-generated and reference qrels varied across collections. Exact agreement ranged from 52.9% to 64.9%, with most disagreements limited to adjacent relevance levels. Quadratic weighted Cohen’s ranged from 0.52 to 0.82, overlapping ranges reported in human-annotation studies, although differences in evaluation settings limit direct comparisons. More importantly, system rankings based on Jev-generated qrels were highly similar to those obtained with the reference qrels, with Kendall’s ranging from 0.90 to 0.93 for nDCG@10 and MRR. These results show that Jev can generate relevance judgments that preserve the relative ranking of retrieval systems, despite differences at the individual query–document level.
For reranking, Jev improved the ordering of the top 1,000 BM25 candidates on the 50 Quati test queries. nDCG@10 increased from 0.3329 to 0.6331 (+90.2%), and MRR increased from 0.6852 to 0.9033 (+31.8%). These results demonstrate improved effectiveness over BM25 on this collection of web passages; they do not establish superiority over other rerankers, an efficiency advantage, or generalization to legal-document reranking.
Future work could assess generalizability across additional collections, domains, and languages, with reranking experiments covering full-length legal documents, additional first-stage retrievers, comparisons with dedicated rerankers, and paired significance tests. Comparing pointwise and listwise formulations, varying candidate-set sizes, and measuring latency and cost would help characterize the trade-offs of Jev-based reranking. Human-in-the-loop workflows could examine how Jev can assist human assessors in relevance assessment.
References
- [1]Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge, UK (2008). http://nlp.stanford.edu/IR-book/information-retrieval-book.html
- [2]Lin, J., Nogueira, R., Yates, A.: Pretrained Transformers for Text Ranking: BERT and Beyond (2021). https://doi.org/10.48550/arXiv.2010.06467
- [3]Wang, L., Yang, N., Wei, F.: Query2doc: Query expansion with large language models. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9414–9423. Association for Computational Linguistics, Singapore (2023). https://doi.org/10.18653/v1/2023.emnlp-main.585
- [4]Nogueira, R., Yang, W., Lin, J., Cho, K.: Document Expansion by Query Prediction (2019). https://doi.org/10.48550/arXiv.1904.08375
- [5]Voorhees, E.M.: The philosophy of information retrieval evaluation. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) Evaluation of Cross-Language Information Retrieval Systems, pp. 355–370. Springer, Berlin, Heidelberg (2002). https://doi.org/10.1007/3-540-45691-0_34
- [6]Upadhyay, S., Pradeep, R., Thakur, N., Campos, D., Craswell, N., Soboroff, I., Dang, H.T., Lin, J.: A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look (2024). https://doi.org/10.48550/arXiv.2411.08275
- [7]Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., Lin, J.: UMBRELA: UMBrela is the (Open-Source Reproduction of the) Bing RElevance Assessor (2024). https://doi.org/10.48550/arXiv.2406.06519
- [8]Nogueira, R., Cho, K.: Passage Re-ranking with BERT (2020). https://doi.org/10.48550/arXiv.1901.04085
- [9]Qin, Z., Jagerman, R., Hui, K., Zhuang, H., Wu, J., Yan, L., Shen, J., Liu, T., Liu, J., Metzler, D., Wang, X., Bendersky, M.: Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2024). https://doi.org/10.48550/arXiv.2306.17563
- [10]Kahneman, D.: Thinking, Fast and Slow. Farrar, Straus and Giroux, New York (2011)
- [11]Almeida, D.: Introducing System One Models & Jev. TypeSafe AI. https://typesafe.ai/blog/introducing-system-one-models-and-jev
- [12]Faggioli, G., Dietz, L., Clarke, C.L.A., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., Stein, B., Wachsmuth, H.: Perspectives on large language models for relevance judgment. In: Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval. ICTIR ’23, pp. 39–50. Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3578337.3605136
- [13]Thomas, P., Spielman, S., Craswell, N., Mitra, B.: Large language models can accurately predict searcher preferences (2024). https://doi.org/10.48550/arXiv.2309.10621
- [14]Bencke, L., Paula, F., Santos, B., Moreira, V.P.: Can we trust LLMs as relevance judges? In: Anais do XXXIX Simpósio Brasileiro de Bancos de Dados, pp. 600–612. SBC, Porto Alegre, RS, Brasil (2024). https://doi.org/10.5753/sbbd.2024.2430
- [15]Fernandes, L.C., Castro, M.V.B., Santos Ribeiro, L., Silva Pacheco, L.A., Oliveira Sandes, E.F.: NormasTCU — a Brazilian Portuguese IR dataset and an evaluation of LLM-as-a-judge for relevance assessment. Information Processing & Management 64(2, Part A) (2027) https://doi.org/10.1016/j.ipm.2026.105151
- [16]Nogueira, R., Jiang, Z., Pradeep, R., Lin, J.: Document ranking with a pretrained sequence-to-sequence model. In: Cohn, T., He, Y., Liu, Y. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 708–718. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.63
- [17]Zhuang, H., Qin, Z., Jagerman, R., Hui, K., Ma, J., Lu, J., Ni, J., Wang, X., Bendersky, M.: RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses (2022). https://doi.org/10.48550/arXiv.2210.10634
- [18]Jeronymo, V., Lotufo, R., Nogueira, R.: NeuralMind-UNICAMP at 2022 TREC NeuCLIR: Large Boring Rerankers for Cross-lingual Retrieval (2023). https://doi.org/10.48550/arXiv.2303.16145
- [19]Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., Ren, Z.: Is ChatGPT good at search? investigating large language models as re-ranking agents. In: Buomaw, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14918–14937. Association for Computational Linguistics, Singapore (2023). https://doi.org/10.18653/v1/2023.emnlp-main.923
- [20]Qin, Z., Jagerman, R., Hui, K., Zhuang, H., Wu, J., Yan, L., Shen, J., Liu, T., Liu, J., Metzler, D., Wang, X., Bendersky, M.: Large language models are effective text rankers with pairwise ranking prompting. In: Duh, K., Gomez, H., Bethard, S. (eds.) Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1504–1518. Association for Computational Linguistics, Mexico City, Mexico (2024). https://doi.org/10.18653/v1/2024.findings-naacl.97
- [21]Zhuang, S., Zhuang, H., Koopman, B., Zuccon, G.: A setwise approach for effective and highly efficient zero-shot ranking with large language models. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ’24, pp. 38–47. Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3626772.3657813
- [22]Yoon, S., Choi, E., Kim, J., Yun, H., Kim, Y., Hwang, S.-w.: ListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrieval (2024). https://doi.org/10.48550/arXiv.2402.15838
- [23]Rao, D., Callison-Burch, C.: JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places (2026). https://doi.org/10.48550/arXiv.2609.29769
- [24]Li, Y., Miao, Y., Krishnan, R., Padman, R.: JEV-as-a-Judge: Accept When Confident, Escalate When Unsure (2026). https://doi.org/10.48550/arXiv.2609.26550
- [25]Deng, B., Fan, S., Zhang, H., Xie, X.: Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences (2026). https://doi.org/10.48550/arXiv.2609.24965
- [26]Wang, Z., Gao, Y.: JEV as a Judge for Agent Trace Security: An Empirical Comparison with Generative LLM Judges (2026). https://doi.org/10.48550/arXiv.2609.34862
- [27]Lyu, H., Xia, Y.: Decision-Oriented Recommendation Reranking: An Empirical Study of Jev (2026). https://doi.org/10.48550/arXiv.2609.40241
- [28]Deuber, T., Sparrenberg, L., Sifa, R.: Evaluating and Benchmarking the System One Model Jev (2026). https://doi.org/10.48550/arXiv.2609.37647
- [29]Coates, D.: Using Jev as a Search Reranker: Benchmarks and How to Implement. Elasticsearch Labs. Published September 30, 2026; accessed October 15, 2026 (2026). https://www.elastic.co/search-labs/blog/ecommerce-search-reranking-llm-alternative-jev
- [30]Bartholomew, B.: We Added Jev as a Reranker. Here’s What We Learned. Hindsight. Published September 24, 2026; accessed October 15, 2026 (2026). https://hindsight.vectorize.io/blog/2026/09/24/adding-jev-reranker-what-we-learned
- [31]Tateno, Y.: Introducing jev-reranker: Reranking and Relevance Filtering for RAG. Hugging Face Community Article. Published September 19, 2026; accessed October 15, 2026 (2026). https://huggingface.co/blog/hotchpotch/introducing-jev-reranker
- [32]Bueno, M., Oliveira, E.S., Nogueira, R., Lotufo, R., Pereira, J.: Quati: A Brazilian Portuguese information retrieval dataset from native speakers. In: Anais do XV Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, pp. 236–246. SBC, Porto Alegre, RS, Brasil (2024). https://doi.org/10.5753/stil.2024.245426
- [33]Fernandes, L.C., Santos Ribeiro, L., Castro, M.V.B., Silva Pacheco, L.A., Oliveira Sandes, E.F.: JurisTCU: a Brazilian Portuguese information retrieval dataset with query relevance judgments. Language Resources and Evaluation 60(1) (2026) https://doi.org/10.1007/s10579-025-09881-w
- [34]Vitório, D., Souza, E., Martins, L., Silva, N.F.F., Carvalho, A.C.P.d.L., Oliveira, A.L.I., Andrade, F.E.: Building a relevance feedback corpus for legal information retrieval in the real-case scenario of the Brazilian Chamber of Deputies. Language Resources and Evaluation 59(2), 1257–1277 (2025) https://doi.org/10.1007/s10579-024-09767-3
- [35]Fernandes, L.C., Oliveira Lima, J.A., Martim, H., Oliveira, E.S., Lotufo, R., Pereira, J.: Structural Segmentation for Brazilian Legal Document Retrieval (2026). https://doi.org/10.48550/arXiv.2610.18622
- [36]Voorhees, E.M.: Variations in relevance judgments and the measurement of retrieval effectiveness. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ’98, pp. 315–323. Association for Computing Machinery, New York, NY, USA (1998). https://doi.org/10.1145/290941.291017
- [37]McDonnell, T., Lease, M., Kutlu, M., Elsayed, T.: Why is that relevant? collecting annotator rationales for relevance judgments. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 4, pp. 139–148 (2016). https://doi.org/10.1609/hcomp.v4i1.13287
- [38]Voorhees, E.M.: Evaluation by highly relevant documents. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ’01, pp. 74–82. Association for Computing Machinery, New York, NY, USA (2001). https://doi.org/10.1145/383952.383963
- [39]Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., Raffel, C.: mt5: A massively multilingual pre-trained text-to-text transformer. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483–498. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.41