WorksComputer Science
Agent-as-a-Judge
Abstract
Artificial intelligence has progressed from task-specific systems toward general models spanning perception, generation, reasoning, and autonomous interaction, making reliable evaluation essential for comparison, deployment, and post-training feedback. At scale, modern evaluation pipelines increasingly rely on LLM-as-a-Judge to produce scores, preferences, and critiques; in its fixed-context setting, the judge relies on the supplied evaluand and evaluation materials. Yet a reliable verdict may require evidence beyond the supplied evaluand and context—either because critical environmental states remain unrecorded in the static execution log, or because controversial factual claims require external verification. Passive judges cannot obtain that missing evidence before deciding, especially when evaluating agent trajectories, executable artifacts, or domain-specific reports. Recent work therefore develops active evaluators that seek missing evidence and update their grounds for judgment before deciding. We bring these advances together through Agent-as-a-Judge, focusing on how evaluators construct and control checking processes within the broader LLM-based judging landscape. We define the scope through evaluation-directed checks and the use of their results, covering both structured workflows and observation-driven control. We organize the design space by multi-agent collaboration, tool integration, memory, and planning, and show how these mechanisms supply the checks and evidence on which a verdict rests. We consolidate published scores and reproduce missing entries for key agentic judges; the comparison reveals uneven agentic gains and motivates dedicated agentic evaluation. We conclude by characterizing application demands and open challenges in cost, reliability, safety, and governance that condition trusted deployment of agentic judges at scale.
🔗 Project Page: https://github.com/ModalityDance/Awesome-Agent-as-a-Judge
Figure 1. Conceptual landscape of Agent-as-a-Judge. Unlike fixed-context LLM judges, an agentic judge resolves missing evidence through evaluation-directed checking—identifying targets, verifying via tools, and integrating findings into grounded verdicts—driven by four mechanisms (multi-agent collaboration, tool integration, memory, planning) across evaluation paradigms, open challenges, and target domains.
Introduction
Artificial intelligence has progressed from task-specific systems toward general models spanning perception, generation, reasoning, and autonomous interaction, making reliable evaluation essential for comparison, deployment, and post-training feedback [1, 2, 3]. Human judgment has long served as the reference standard for these evaluations [1, 2, 4, 5], yet manual annotation is costly and slow, and inter-annotator disagreement limits consistency and reliability [2, 6, 4, 7]—limits that become especially acute when evaluation must run continuously during post-training or across large-scale public leaderboards [8, 3]. To improve scalability, the field adopted automatic metrics that score outputs against predefined rules or references [9, 10, 11]. These approaches are efficient and reproducible, but they presuppose tasks with formalizable targets and struggle with open-ended generation, fine-grained quality, and judgments that depend on broader context [4, 5, 7].
Rapid advances in large language models (LLMs) have opened a third path [7]. Modern LLMs combine strong semantic understanding with instruction-following capabilities, allowing them to follow evaluation prompts and provide human-like quality judgments [6, 12, 13, 14, 7]. In the fixed-context setting shown in Figure 2a, a judge reads a supplied evaluand—the object under evaluation, such as a response, summary, or trajectory—together with its context and returns a score, preference, or critique [12, 13]. A judge LLM can evaluate large volumes of outputs via repeated inference at machine speed and cost, while maintaining near-human judgment quality on many open-ended tasks [12, 13, 15]. Collectively, these properties make LLM judges a scalable proxy for human preference [7], whose judgments now underpin model comparison and scoring, reward signals for reinforcement learning [16], and large-scale synthetic data curation [17]—placing judge fidelity at the center of modern evaluation and optimization pipelines [3].

Figure 2. Comparison between (a) LLM-as-a-Judge and (b) Agent-as-a-Judge. The former acts as a passive evaluator relying on the given context and model-internal knowledge, while the latter acts as an active evaluator that plans, gathers evidence, uses tools, and iteratively verifies its judgment.
However, this passive design proves insufficient once a reliable verdict depends on evidence beyond the materials provided with the case [18, 19, 20]. What is missing may be an unobserved environment status, such as code execution outcomes, workspace state, and live web pages; or facts that fall outside the judge’s knowledge and must be recovered from external sources [18, 21, 19, 22]. Obtaining them typically requires retrieval, re-execution, or environment interaction before a reliable score, preference, or critique can be issued [19, 23, 20]. The gap is especially visible when the evaluand itself is an agent or a trajectory from an agent, where success depends on external state and procedure as well as the textual response to user [18, 20, 24, 25, 26]. Under passive LLM-as-a-Judge, however, the judge scores only from that fixed packet and cannot gather the missing but decisive evidence before issuing a verdict [18, 20, 27, 19]. Reliable judgement accordingly requires an active judge that can gather and update evidence beyond what was initially provided [19, 20].
The emerging literature answers with Agent-as-a-Judge (Figure 2b): an evaluator that maintains an evaluation state, carries out checks, and incorporates their results before issuing a verdict [18, 19, 20]. Mechanically, tool integration lets judges obtain external evidence [19, 29] [28]; planning guides how claims are inspected and verified as that evidence accumulates [20, 30]; memory retains findings, case artifacts, and curated evaluation experience across steps [31, 32, 27, 33]. Multi-agent collaboration further mitigates single-evaluator bias by distributing criteria, surfacing objections, and aggregating independent assessments [34, 35, 36]. Together, these mechanisms extend what a judge can inspect, ground verdicts in traceable evidence, and sustain evaluation across steps—positioning agentic judges as an increasingly central approach to reliable evaluation.
Existing surveys provide complementary foundations: Li et al. [7] organize LLM-based judgment around what to judge, how to judge, and how to benchmark, while Yehudai et al. [37] map agent capabilities, benchmarks, and evaluation frameworks. Our organizing object is the evaluator’s own checking process: how it obtains findings, connects intermediate results, and controls verification before forming a verdict. We bring work across training pipelines, verification tasks, evaluator architectures, and domains into a survey of Agent-as-a-Judge centered on these mechanisms and their meta-evaluation. Building on this view, our contributions are as follows:
An operational definition and scope for Agent-as-a-Judge: we characterize evaluation-directed checks and the use of their results, distinguishing these processes from fixed-context scoring and replicated judgments, and describe control from prescribed workflows to observation-driven adaptation (Section 2).
A mechanism taxonomy over multi-agent collaboration, tool integration, memory, and planning: we organize the literature under this taxonomy and show how these mechanisms supply the checks and evidence on which a verdict rests (Section 3).
A comparative meta evaluation of agentic judges: we consolidate published scores and reproduce missing entries for five key agentic judges across three unified benchmarks; the comparison shows uneven agentic gains under preference-oriented benchmarks and motivates dedicated evaluation of evidence-seeking reliability (Section 4).
An account of application demands together with open challenges in cost, reliability, safety, and governance that condition trusted deployment of agentic judges at scale (Sections 5–6).
Background
Within the broader LLM-as-a-Judge landscape, we focus on evaluators that construct judgments through explicit checking processes. We first formalize fixed-context judging as a reference setting, then examine the need for additional evidence and introduce the operational scope of Agent-as-a-Judge.
LLM-as-a-Judge
LLM-as-a-Judge emerged as a pragmatic response to a long-standing dilemma in evaluation. From the outset, human judges have assessed quality through pairwise preferences, Likert-scale ratings, and expert rubrics, serving as the reference standard for semantic nuance and contextual appropriateness [1, 2, 8, 12, 4, 5]. Yet manual annotation is costly and slow, and inter-annotator disagreement limits reproducibility when evaluation must run continuously during post-training or across large public leaderboards [2, 6, 8, 7]. Automatic metrics were introduced to improve scalability: they score outputs against predefined rules or references, such as BLEU [9] and ROUGE [10] for translation and summarization, and exact-match or F1 scores in question answering [11]. These approaches are efficient and reproducible, but they presuppose tasks with formalizable targets and struggle
with open-ended generation, fine-grained quality, and judgments that depend on broader context [4, 5, 7]. Human evaluation and automatic metrics thus trade semantic judgment quality against deployment scale.
Large language models opened a third path [7]. Because they follow natural-language instructions and rubrics, they can return scores, preferences, or critiques on open-ended outputs rather than matching fixed surface patterns [6, 12, 13, 14], while repeated inference applies the same evaluation prompt to arbitrarily many evaluands at machine speed and cost [12, 15]. LLM-as-a-Judge therefore targets the semantic flexibility missing from automatic metrics without reintroducing the annotation burden of human evaluation [12, 13, 14].
Early work on passive LLM-as-a-Judge mainly followed two lines [7]. One concerns judgment format: pairwise preference compares two candidates—automated in MT-Bench and Chatbot Arena [12, 8] and reused as alignment rewards [6, 16]—while pointwise scoring assigns an absolute grade to each response [12, 15]. The other enriches the supplied context , typically with explicit rubrics that structure criteria when gold references are absent or insufficient [75], as in G-Eval [13] and Prometheus [14]. Across both lines, the judge still receives a fixed evaluand and context and returns a verdict without actively expanding the available evidence (Figure 2).
Definition (LLM-as-a-Judge). In the fixed-context setting considered here, an LLM-as-a-Judge maps an evaluand and a supplied evaluation context to a verdict:
where is the evaluand, is the supplied evaluation context, and is the verdict.
This passive formulation treats the supplied evaluand and context —rubrics, references, demonstrations, partial traces—as the complete evidence available at verdict time, as in Figure 2a. Once passive LLM judges proved deployable at scale with sufficient alignment to human judgments on many open-ended tasks [12, 13, 7], their verdicts were adopted for pairwise model comparison, reinforcement-learning reward modeling [16], synthetic-data filtering [17], and broader post-training pipelines [3]. This passive contract remains well matched when the evaluand is an open-ended model response, but it is strained once the evaluand evolves further—to agent trajectories and tool-mediated workflows, or to domain reports whose claims must be checked against external state or references [26, 18, 19, 23].
From LLM-as-a-Judge to Agent-as-a-Judge
Under such evaluands, a reliable verdict may depend on evidence that the closed pair does not contain. The failure takes two related forms: decisive evidence about external state that is absent from the supplied text, and an epistemic gap when needed facts fall outside the judge’s parametric knowledge.
First, decisive evidence may be absent from the text of itself—for example when is an agent trajectory, a code patch that must be verified by execution, or a computer-use trace whose effects on files, runtime state, or the GUI are not fully captured in the supplied text [26, 18, 21]. Workspace files, database records, GUI state, execution results, and live web content sit outside what a passive judge can read from the supplied context [18, 21, 20]. An evaluand that looks complete in text may still be inconsistent with that external state, so verification requires obtaining observations—rereading files, querying state, re-executing critical steps—before the verdict [18, 21, 19]. A passive judge confined to cannot produce those external observations [18, 19].
Second, critical facts may be absent from the context and the judge’s parametric knowledge. Whether a claimed dosage matches current prescribing information, whether a clinical recommendation accords with updated guidelines, or whether a scientific statement is supported by the cited primary literature depends on external references that the supplied materials may not contain [19, 21, 23]. When those facts cannot be recovered reliably through internal reasoning alone, external evidence-seeking is necessary for the judge [22]; otherwise a closed forces the verdict onto the judge’s priors and may penalize novel yet valid approaches that the judge simply does not know.
Both failures are instances of the same closed-world limit: lacks evidence the verdict needs, whether that evidence is an environment observation or external fact beyond the judge’s knowledge. Reliable judgment therefore requires an evaluator that can seek the missing evidence before deciding—querying environment state when success is not visible in the supplied text, and recovering documentation or other external facts when the needed knowledge cannot be settled internally [19, 20, 21, 22]. This motivates Agent-as-a-Judge: evaluation processes that construct their grounds for judgment through evidence acquisition and explicit checking.
Agent-as-a-Judge
Figure 2b illustrates the contrast with passive LLM-as-a-Judge. An agentic judge organizes judgment as an explicit evaluation process: it carries out checks, accumulates findings, and grounds its verdict in their results. Tool integration, planning, memory, and multi-agent collaboration provide the principal mechanisms for building this process.
Definition (Agent-as-a-Judge). An Agent-as-a-Judge constructs a verdict about an evaluand through explicit checking operations and the use of their results. At least one operation obtains evidence, performs a targeted verification, or reassesses a judgment using another check’s findings or rationale, beyond replicating an overall score. The resulting findings inform subsequent assessment or verdict formation. We represent this process as:
Here encodes the evaluand and any initial context; summarizes with that initialization; is an evaluation action executed under policy , with as its return; and verdict is produced when evaluation ends at .
Concretely, may retrieve a document, probe workspace or environment state, delegate a sub-check, exchange critiques with another judge, or retrieve a stored evaluation artifact; is the return of that call—a retrieved passage, file contents, an environment status, or a sub-check result—and incorporates it into for subsequent checks and the final verdict. These evaluation processes range from structured workflows that coordinate predefined checks to adaptive policies that revise what to inspect, how to verify it, and when to stop as evidence accumulates. The policy can therefore prescribe a checking sequence or use intermediate observations to select subsequent actions; likewise, can be fixed or determined during evaluation. We distinguish prescribed execution, input-conditioned planning before execution, and observation-driven revision during execution. These control choices apply across all four mechanisms and can coexist within a system.
Survey Scope
We include work whose main contribution is the evaluator and whose reported procedure satisfies the checking and result-use conditions above. The operational boundary rests on what is checked, what result is produced, and where that result is used; the state–action formulation describes the process within this scope. For tool-mediated evaluation, retrieved support or execution outcomes ground the verdict; in collaborative evaluation, targeted sub-checks or reassessment of peer rationales provide the corresponding findings. Role names, additional calls, and voting alone do not establish this process. Fixed retrieval or testing procedures can qualify, while generic task-execution agents do not qualify merely because they maintain state and act sequentially. Appendix A specifies the screening conditions and contrasting protocol examples.
Methodology
Building the evaluation process formalized in Section 2 involves organizing evaluative work, obtaining observations, retaining useful findings, and specifying the checks to perform. We examine these design choices through four complementary mechanisms. Multi-agent collaboration distributes checking tasks, information exchange, and verdict authority across evaluators (§3.1); tool integration provides interfaces for obtaining external observations and execution results (§3.2); memory governs which findings are stored, accessed, and updated (§3.3); and planning specifies checking targets, decompositions, and verification routes (§3.4). These are composable design dimensions: a judge can coordinate multiple evaluators, equip them with tools, retain their findings in memory, and plan the checks within the same process. Across these dimensions, we distinguish checks prescribed in advance, procedures tailored to the initial input, and checks revised from intermediate observations (Section 2). Their interaction determines how intermediate results enter the evaluation state, support a verdict, and guide further checking; the synthesis returns to these connections through evidence integration and stopping decisions. Figure 3 places this methodological axis alongside Evaluation and Applications in the overall survey map.

Figure 3. A taxonomy of Agent-as-a-Judge organizing Methodology (§3), Evaluation (§4), and Applications (§5).
Multi-Agent Collaboration
Multi-agent collaboration distributes checking tasks and verdict authority across evaluators, making perspective, challenge, and resolution explicit parts of the judging protocol [34, 60, 35]. Its central design choices concern who performs each check, which outputs are shared, and who can revise or resolve partial judgments. We first identify those roles and their implementations, then trace how their outputs are exchanged and combined before the final verdict. The inclusion criterion concerns the checking task and information flow: evaluators perform distinct verification tasks or use another check’s findings or rationale in subsequent assessment. Changing persona names while independently repeating the same overall scoring task does not qualify on its own.
What Are the Subagents in a MAD Judge?
Functionally Reviewed subagent roles cluster into adversarial, perspective, analytical, and verdict-control categories (Figure 4). Adversarial roles take a tentative answer or judgment and emit objections or cross-examinations that later stages must address. AgentsCourt [45] stages plaintiff and defendant arguments as advocacy transcripts that later feed legal decision-making. Perspective roles read the evaluand through different identities or expertise and return alternative rationales for the same case. AgentsBench [36] pairs professional and lay judge agents whose independent sentencing recommendations feed later deliberation. Analytical roles transform the evaluand or an upstream answer into inspectable intermediate evidence. GEMA-Score [47] extracts clinical entities that downstream scoring must cite, while MATH-VF [76] formalizes informal reasoning into SimpleMath statements that critics and tool agents verify step by step. Verdict-control roles take partial judgments, discussion state, or protocol signals as input to aggregate, stop, reroute, or authorize a verdict. MATEval [60] routes discussion through feedback and summarization before the final verdict is issued, while CollabEval [59] applies consensus and unchanged-result checks to decide whether deliberation should continue or close.

Figure 4. Role lexicon of subagents in multi-agent judges. The four panels group cleaned subagent names into adversarial, perspective, analytical, and verdict-control roles.
Technically The same role labels map to different implementation classes (Figure 5). A named subagent may be a persona prompt, a structured workflow, a deterministic controller, or a tool-integrated component, and the label alone does not specify which control mode applies. Persona prompts specify an evaluator’s perspective or checking remit within a protocol. Their contribution depends on the surrounding process: debate systems expose peer rationales for further assessment [34, 59], whereas Multi-Agent Verification [62] assigns distinct verification aspects and aggregates their approvals without peer deliberation. Structured workflows chain artifact-producing steps: an extractor, formalizer, critic, or final judge consumes a specific upstream output and emits a constrained downstream artifact. Deterministic controllers apply agreement ratios, unchanged-result checks, or stability tests to decide whether deliberation should stop, continue, or escalate [59, 77]. Tool-integrated components operationalize verification through predefined interfaces, as when AgentV-RL [78] embeds Python calls inside forward and backward verifier trajectories. Adversarial and perspective names mostly resolve to persona prompts, while verdict-control names more often split across deterministic controllers and structured workflows. The implementation class identifies how a role is realized; checking responsibilities, information dependencies, and continuation rules identify its place in the evaluation process.
Figure 5. From functional role names to implementation classes. The Sankey maps subagent and component instances from functional roles to implementation classes.
How Do Subagents Collaborate Within a MAD Judge?
Subagent roles identify who participates in judgment production, but the final verdict depends on how those roles connect their partial outputs before closure. Reviewed MAD judges coordinate those connections through sequential staging, reciprocal deliberation, and parallel aggregation.
Sequential Sequential collaboration makes later stages depend on artifacts produced upstream. In evidence-to-judgment pipelines, advocacy transcripts or collected software evidence become inputs for later judgment: AgentsCourt [45] feeds plaintiff and defendant arguments into legal decision-making, while CodeVisionary [70] routes software requirements through runtime, static-analysis, UI, and web evidence before scoring. In rubric and decomposition pipelines, the upstream artifact constrains what downstream judges can score. Upstream artifacts include refined rubrics, extracted clinical entities, formalized statements, and proposed robot-event timelines that downstream agents score or verify [40, 47, 76, 79]. The benefit is traceability: the verdict can be followed through intermediate artifacts. The risk is upstream lock-in, because a flawed rubric, extraction, formalization, or evidence localization constrains every later judgment.
Reciprocal Reciprocal collaboration turns another subagent’s statement into new input for revision. The relevant operation is state change after peer exposure: a subagent reads another rationale, updates its own position, and the protocol tests whether disagreement has narrowed enough for closure. AgentsBench [36] begins with independent sentencing recommendations, then lets professional and lay judge agents deliberate until a presiding judge detects convergence or resolves residual disagreement. CollabEval [59] runs initial evaluation, discussion, and final-judge phases, with consensus and unchanged-result checks deciding whether deliberation has stabilized. HiMATE [73] uses tiered agree/disagree turns to resolve translation-evaluation ambiguity, while Efficient LLM Safety Evaluation [80] uses critic–defender exchanges to surface safety risks before a judge assigns the risk score. This form is useful when objection and revision can expose hidden assumptions, but its closure rule must guard against agreement that suppresses useful minority evidence.
Parallel Parallel collaboration preserves independent assessments until a resolver combines them. The resolver determines what independence contributes to the verdict. Formulaic merge rules make aggregation reproducible, as when AgentV-RL [78] combines forward and backward verifier confidence or Multi-Agent Verification [62] sums independent aspect-level approvals. Weighted or threshold-based resolution governs whether partial assessments authorize a verdict, including meta-judgments and supporting/refuting arguments whose confidence gap controls further evidence gathering [81, 82]. No-human-in-the-loop [83] synthesizes labels from many LLM configurations through majority vote and a severity hierarchy for persistent conflicts. Parallel systems differ in what the resolver makes visible: a reproducible formula, an authority point, or a fallback path for unresolved disagreement.
Tool Integration
Tool integration allows an Agent-as-a-Judge to move beyond fixed-context inference by actively obtaining information or signals outside the initial prompt. A prompt-only judge must infer quality, correctness, or
risk from the evaluand and the context already placed in the prompt. A tool-augmented judge can instead retrieve external facts, execute programmatic checks, inspect multimodal or interactive artifacts, and use the resulting observations to ground subsequent judgment. We therefore analyze tool integration along three axes: interfaces, selection, and function—which observations are accessible, when an interface is invoked, and how its results support evaluation.
What Interfaces Are Used?
Figure 6 shows which tool interfaces appear most often in the manually coded subset. Execution and search interfaces dominate. Code execution supports programmatic checks [78] [28], while web search [84, 85] and local search [86, 87] bring external or stored evidence into the judging context. Vision models [88] and image operations [32, 50] support multimodal inspection. Solvers [76] and shell access [70] serve more specialized verification or runtime-inspection needs. Translation [23, 63] and computer-use interfaces [70, 89] remain less common, but they mark cases where judges translate queries or evidence across languages or act in interactive environments.
Figure 6. Distribution of 60 tool-interface assignments in a manually coded subset of 34 tool-augmented judges. Each bar shows one interface’s share of the total; papers using multiple interfaces contribute one count per interface type.
How Are Tools Selected?
Tool selection determines when a judge should call a tool, which tool it should call, and whether the returned result is enough for judgment. Across existing systems, this control becomes progressively more adaptive: the judge may follow fixed rules, call tools when current evidence is weak, use observations from one tool call to guide later calls, or actively create new observations. 1) Fixed-rule selection uses predefined signals: the modality, domain tag, or answer type decides whether the case goes to search, code execution, or a solver. VerifiAgent [29] routes checks by reasoning type, while Auto-Eval Judge [86] dispatches cases according to verification type. 2) Evidence-triggered selection asks whether the current evidence is enough. The judge keeps prompt-only reasoning as the default, but escalates to retrieval when the current trace lacks support, as in OpenReward [84] and TALE [85]; UrduFactCheck [23] applies the same idea in multilingual fact checking. 3) Observation-guided selection lets the result of one tool call shape the next tool call or checking step. This pattern is stronger than one-shot escalation because tool outputs become part of the control signal for later checks. Accumulated findings guide further perception steps, cross-channel inspections, and targeted state, trace, or temporal-window queries [90, 70, 91, 92, 79]. 4) Active probing runs a new test or acts on the environment to surface observations that were not available in the initial context. Environment interaction and executable tests expose additional evidence for GUI-task completion, code correctness, and application behavior [89, 93, 94].
What Functions Do Tools Serve?
Tool use gives the judge capacities that are unavailable from the prompt and the candidate answer alone (Figure 7). After a tool is called, the observations it returns can support judgment in several ways. They may add missing knowledge, expose inspectable evidence, test correctness operationally, or make the final score depend on an interaction trace. These uses form four recurring roles. 1) Knowledge expansion links the verdict to information outside parametric memory so judgments can depend on up-to-date or domain-specific facts. OpenReward [84] and TALE [85] retrieve or escalate to outside facts when parametric knowledge is insufficient, while DREAM [95] derives case-specific checks for deep research evaluation. Domain stores and structured pipelines query narrative graphs [24], legal document collections [45, 46], or medical evidence bases [96] before scoring. FACT-AUDIT [27] extends the same grounding pattern to factual audit, while UrduFactCheck [23] applies it in multilingual fact checking. 2) Evidence access and transformation converts opaque evaluands into localized, inspectable artifacts that the judge can cite rather than summarize. Auto-Eval Judge [86] exposes support through log search, while TaTToo [87] does so through table querying. In multimodal settings, OCR, visual differencing, cropping, and specialized perception tools localize textual, visual, or spatial evidence for inspection [97, 50, 32, 98]. The verdict therefore rests on citable support rather than on an overall impression. 3) Operational verification uses executable interpreters, solvers, and programmatic checkers to test intermediate steps, format constraints, or derived consequences, turning plausibility into a failed check, a counterexample, or a satisfied constraint. Existing systems automate code-oriented checking [28], reasoning-type verification [29], and math-oriented formal checking [76]. The same pattern also supports training-time scoring [78], tool-integrated answer checking [99], and format-oriented checking [65]. 4) Trace-based judgment applies when the initial context cannot settle the verdict and the score depends on the trace produced by tool-mediated interaction. Some judges question visual reasoning steps [88] or test the evaluand directly [93]; others act on GUI history [90] or runtime state [89]. A few chain repository inspection with runtime, UI, and web probes in one evaluation [70]. Auxiliary LLM modules or sub-agents route cases [19, 86], extract evidence, or aggregate observations within the loops above [70].
Figure 7. Functional view of tool use in Agent-as-a-Judge. Four overlapping regions show that judges can chain multiple tool roles within one evaluation.
Memory
Memory allows an Agent-as-a-Judge to carry evaluation state across steps, cases, or interactions rather than treating each judgment as an isolated inference. A memory mechanism makes the lifecycle of reusable findings explicit through storage, access, and update operations. We therefore analyze memory along a write–manage–read loop (Figure 8) through three questions: what evaluation state is preserved, how selectively it is read back, and what event authorizes an update. The same system may combine several patterns across these axes.
Figure 8. Memory in Agent-as-a-Judge can be analyzed as a write–manage–read loop over evaluation state. The stored state may take the form of process memory, artifact memory, or experience memory. Later judgments either receive this state directly, access a filtered or summarized form, or target specific artifacts, failures, or templates through retrieval.
What Memory Is Stored?
Three storage structures recur across agentic judges, although a single system may combine them. 1) Process memory accumulates evaluation state step by step within a run rather than as a pre-indexed collection of objects. Later steps read this buffer back to change an ongoing evaluation, most often in one of two ways. For verification, the buffer holds proof state from search or checking—queries, evidence summaries, reflections, and support or contradiction signals—that later steps consume to refine evidence and determine a verdict. Fact-checking judges such as TALE [85] exemplify this pattern through iterative query–evidence–reflection buffers. For interaction scoring, the buffer keeps enough of a long interaction or dialogue to score the current step without forwarding the entire transcript unchanged. Interactive and dialogue judges maintain bounded history through relevance filtering, progressive summarization, or evolving internal state rather than unbounded transcript growth [90, 54]. 2) Artifact memory retains case-specific artifacts or reports that later checking steps re-open by index or retrieval rather than as undifferentiated context. Multimodal and code-inspection judges index crops, document pages, or stage-wise analysis reports for later tool calls and evidence-grounded scoring [32, 70]. Agent-as-a-Judge [18] exposes similar modules, but ablation evidence suggests limited benefit and possible error propagation, making it a cautionary rather than an anchor example. 3) Experience memory curates prior evaluations into structured knowledge—failure pools, scenario taxonomies, or critique templates—that later cases draw on rather than re-deriving from scratch [27, 33].
How Is Memory Read?
Read access ranges from direct carry-forward to condensed readout to selective retrieval. 1) Direct carry-forward places stored state into the next judging context without an explicit choice among memory items. Verification-oriented process memory [85] and interaction-scoring buffers [54] follow this pattern cleanly. Multi-agent deliberation systems also inherit full discussion records [60, 35, 36], but transcript availability alone does not establish selective access, compression, or curation of memory. 2) Condensed readout first constructs a smaller usable view through filtering or summarization, then supplies that view to the judge. Long-horizon GUI evaluation depends on this pattern when current-step reward scoring must use a compressed interaction history rather than the full buffer [90]. 3) Selective retrieval targets a particular stored object when the current step needs it—by index lookup, pool search, or template routing. Strongest cases index specific multimodal artifacts or document pages [32], draw failure cases from audit pools [27], or route critique generation through an error-type template tree [33]. CodeVisionary [70] is a weaker fit because later stages clearly reuse distilled reports, but the available evidence is less explicit about retrieval among report items.
How Is Memory Updated?
The same system may combine several update triggers within one evaluation run. 1) Step-triggered changes memory after each evaluation step, tool observation, reflection, report, or discussion round—the write cadence follows the evaluation process itself. Iterative fact checking extends process memory on each query–evidence cycle [85]; deliberation judges append round-level discussion state [60, 35, 36]; and multimodal inspection adds tool-produced artifacts to artifact memory as checking proceeds [32, 70]. 2) Budget-triggered changes memory because the working state has become too long, too noisy, or too stale to use directly. GUI interaction scoring filters recent history and summarizes older steps once interaction length grows [90], while dialogue judging updates a compact running state each turn rather than accumulating an unbounded transcript [54]. 3) Outcome-triggered changes memory only after an evaluation outcome identifies something worth preserving as future guidance—a low-score audit case, failure pattern, successful critique, or new error type. Within-target factual audit accumulates weak cases and revises scenario taxonomies after poor outcomes [27], while table critique expands its template tree after successful refinements across instances [33]. Deliberation systems that only record round-by-round discussion without budget-triggered or outcome-triggered updates remain at the weaker end of this axis, closer to deliberation carryover than to the managed patterns above.
Planning
Planning makes the checking procedure explicit by specifying what should be inspected and how proof should be gathered. A plan can organize a case-specific workflow before execution or guide successive checks as observations accumulate. The planning object is the checking procedure: its targets, decomposition, and route through the available evidence sources. We therefore distinguish two forms: what-to-check planning, which defines the target of evaluation, and how-to-check planning, which defines the route of evidence gathering (Figure 9).
Figure 9. Planning in Agent-as-a-Judge systems can externalize either what to check or how to gather proof before the verdict.
What to Check?
What-to-check planning defines the target that subsequent requirement checks must address, potentially after an initial inspection of the available context. The planned target can take three forms. 1) Case-specific protocols turn the current query or evaluand into checks tailored to that case. DREAM [95], for example, derives report-specific fact checks and validation questions before executing the evaluation protocol. 2) Rubric planning searches or constructs the criteria that will guide later judgment. Criteria can be selected through rubric search or constructed from the task and inspected environment context [100, 91]. 3) Sub-aspect planning chooses which aspect of an open-ended evaluand should be inspected next. Evaluation Agent [20] uses this pattern in visual evaluation by moving across proposed sub-aspects over successive rounds.
How to Check?
How-to-check planning defines the route by which evidence is gathered once the evaluation target is fixed. This route can specify the order of checks, the granularity of intermediate steps, the evidence source for each step, or the condition for stopping. 1) Route planning writes an evaluation plan before scoring and then follows that route to the verdict, as in EvalPlanner [101]. 2) Claim-level planning decomposes a candidate solution into smaller claims and verifies them along an explicit path. AgentV-RL [78], for example, verifies atomic claims through forward or backward verifier trajectories. 3) Step-level routing decides what kind of evidence each step requires before invoking a tool. TIM-PERM [88] routes steps toward visual, knowledge-based, or logical checks, while VerifiAgent [29] checks completeness and consistency before selecting the matching tool. 4) Evidence-channel dispatch sends a case to the evidence source or evaluation handler required by the planned procedure. CodeVisionary [70] dispatches requirements across runtime, static-analysis, UI, and web evidence channels, while Auto-Eval Judge [86] routes cases according to verification type and expands retrieval when proof is thin.
Synthesis
The four mechanisms offer complementary views of how evaluation processes are built and how their intermediate results support judgment. We first arrange representative systems on a calendar timeline stratified by mechanism (Figure 10). Multi-agent debate that diversifies and challenges single-model judgment is a comparatively early—and still active—direction for judges [34, 60]. Tool augmentation appears on a similar early timeline, yet the toolkit has kept shifting: calculator-style and mixed external APIs opened the loop [63], while recent designs concentrate on code execution and search as the dominant evidence stack [84, 85, 86] [28]. More recently, as rubric-based evaluation has become a focal theme for LLM-based judgment [75], planning attention has moved toward constructing atomic, case-specific, and independently verifiable criteria before scoring [100, 95, 102]. Memory has long been a dedicated research focus in general LLM-based agents for long-horizon interaction and self-evolution [103, 104], where reusable experience across sessions and tasks is widely treated as central to sustained performance. In Agent-as-a-Judge, by contrast, most systems keep only a short within-run buffer of evaluation state [85, 90], and few curate reusable experience across cases [27, 33].

Figure 10. Representative adoption timeline of the four methodological mechanisms in Agent-as-a-Judge. A method may appear in more than one lane when it instantiates multiple mechanisms. Stars mark two sparse patterns discussed in the text: rubric construction on Planning and experience memory on Memory.
From findings to verdicts Across these designs, evidence integration connects checking operations to final judgment. Tools and sub-judges produce observations or partial assessments, memory retains them for reuse, and the checking plan organizes which claims they address. The update incorporates these results into evaluation state, while resolves that state into a verdict. Formulaic aggregation combines partial scores or approvals, as in forward/backward verifier combination and independent aspect verification [78, 62]. Deliberative resolution instead lets evaluators examine competing rationales before a presiding or final judge closes the case [36, 59]. Resolution rules can handle conflicting labels through severity hierarchies or require source-grounded support before accepting a criterion verdict [83, 92, 91]. These choices determine how supporting, conflicting, or differently scoped findings contribute to the verdict.
| System | Planned element | Decision point | Mechanism |
| What-to-check planning | |||
| DREAM [95] | Query-specific evaluation protocol | Before protocol execution | Turns the query into report-specific fact checks and validation questions. |
| APRES [100] | Review rubric | During rubric search | Searches over candidate criteria and keeps the rubric used for revision. |
| Evaluation Agent [20] | Next sub-aspect | Across proposal rounds | Moves from one aspect to the next and stops when the accumulated evidence is sufficient. |
| How-to-check planning | |||
| EvalPlanner [101] | Unconstrained evaluation plan | Before scoring | Writes the checking route first and then executes it. |
| AgentV-RL [78] | Verification path over atomic claims | Per candidate solution | Splits a solution into atomic claims and verifies them forward or backward. |
| TIM-PRM [88] | Per-step evidence route | Before each tool call | Decides whether the next check needs vision, outside knowledge, or logic. |
| VerifiAgent [29] | Two-layer verification policy | Before tool invocation | Checks completeness and consistency before selecting the matching tool. |
| CodeVisionary [70] | Requirement-level evaluation plan | Before context collection | Gathers runtime, static-analysis, UI, and web evidence in the planned order. |
| Auto-Eval Judge [86] | Verification-type dispatch policy | After checklist generation | Routes cases by verification type and expands retrieval when proof is thin. |
Table 1. Planning mechanisms in Agent-as-a-Judge. What-to-check planning decides the evaluation target, while how-to-check planning decides the evidence-gathering route.
From findings to further checks The same intermediate results can determine whether evaluation should continue. Prescribed workflows reach the verdict after their planned checks or debate rounds [101, 34]; observation-driven policies use returned findings to revise checks or determine the stopping point . Consensus, unchanged-result checks, and adaptive stability detection provide explicit closure signals in multi-agent deliberation [59, 77]. Evidence-seeking judges connect continuation to unresolved support through retrieval escalation, sub-aspect inspection, or renewed exploration of supporting and refuting evidence [84, 85, 86, 20, 82]. Here planning specifies the next verification target or route, tool selection supplies the relevant observation interface, and coordination assigns the check and its resolution authority. Agreement, stability, and evidence sufficiency thus serve distinct control signals; their use does not by itself establish calibrated confidence. Together, evidence integration and continuation policies explain how the four mechanisms turn accumulated findings into a verdict or a reason to investigate further.
Evaluation
Evaluation protocols have not kept pace with method development (§ 3): most reported evidence still concentrates on shared judge-quality and task benchmarks (Figure 11). We organize this evidence by the signal each system is validated to produce. Reward modeling turns responses, trajectories, or reasoning traces into preference, reliability, or correctness signals [1, 2, 6, 105, 106]. NLG evaluation instead scores summaries, conversations, stories, and other generated texts along subjective or multidimensional quality axes [12, 13, 7].
Figure 11. Evaluation–benchmark coverage map. Links indicate that a method reports results on, or is explicitly evaluated on, a reported benchmark. Right-node fill and pattern encode benchmark artifact type: ••• fixed response or judgment sets packaged for direct comparison; /// task problems whose reported results require candidate generation before judging; and green nodes, NLG evaluation datasets.
Reward Modeling
Reward-model signals support candidate ranking, reward assignment, data filtering, and reasoning verification. Recent agentic variants change the judging procedure through tool-integrated reasoning [63] [28], planner–coder decomposition [19], multi-verifier aggregation [62], or verification-oriented scoring [29].
Reward models are evaluated in three settings. The first is test-time selection: benchmarks such as MATH [107] and AIME [108] supply problems and references, and the judge selects, accepts, or ranks generated candidates. For example, AgentV-RL [78] distills an agentic verifier and measures Best-of- selection and verifier-guided actor revision. Results therefore depend on the candidate-generation pool as well as the judge. The second is downstream training, which measures the reward signal’s effect on policy learning, data construction, or actor revision. Reported uses include PPO/RLTAF, DPO-data construction, interaction-grounded rewards for application generation, and selection of robot action chunks for policy learning [63, 19, 94, 79]. The third is a benchmark setting using fixed
judgment artifacts for cross-model comparison. Pairwise suites such as RewardBench [109], RM-Bench [110], and JudgeBench [111] dominate this surface; VerifyBench [112], ProcessBench [113], and IFBench [19] extend it to reference-grounded verification, step-level error localization, and instruction-following constraints, respectively. Table 2 catalogs evaluand form, domain coverage, and judgment signal across these benchmarks. These suites measure answer selection, reward reliability, or reference agreement [109, 111] [110], but do not isolate the contributions of planning, tool use, evidence acquisition, aggregation, or stopping. High scores on these suites also may fail to predict gains in test-time selection or downstream training [114] [110]. Environment-aware evaluation complements fixed-artifact suites by testing trajectory verification through interaction or inspection of post-execution state [21, 91]; its results should distinguish evidence-access conditions and report decision coverage alongside accuracy when abstention is allowed.
| Benchmark / dataset | Evaluand form | Evaluand | Domain | Judgment signal | Scale | URL |
| RewardBench [109] | pair-wise | response pair | chat, safety, reasoning, code/math | chosen–rejected preference | 2,985 | § õ |
| RM-Bench [110] | pair-wise | response pair | chat, code, math, safety | subtle correctness under style variation | 1,237 | § õ |
| JudgeBench [111] | pair-wise | response pair | knowledge, reasoning, math, code | objective correctness label | 620 | § õ |
| IFBench [19] | pair-wise | response pair | instruction following | constraint satisfaction | 444 | § õ |
| VerifyBench [112] | point-wise | completion | reference-based answer verification | binary correctness against reference | 3,000 | õ |
| ProcessBench [113] | step-wise | reasoning trace | mathematical reasoning | earliest-error step label | 3,400 | § õ |
Table 2. Reward-model and verifier benchmarks.
Table 3 presents side-by-side reward-model scores on RewardBench [109], RM-Bench [110], and JudgeBench [111] for the agentic methods above [19, 62, 29, 63] [28] and several non-agentic reasoning reward models [115, 116, 117] included as baselines. We retain published scores where available and reproduce missing benchmark entries from publicly released source code, since most original papers do not evaluate all three benchmarks.
| Benchmark | Method | Model | Chat | Chat-Hard | Safety | Reason | Overall |
| RewardBench | TIR-Judge [28]† | TIR-Judge-8B | 94.7 | 77.4 | 88.8 | 95.7 | 89.1 |
| Agentic Reward Model [19] | ArmoRM-Llama3-8B* | 90.8 | 70.0 | 86.5 | 93.2 | 87.6 | |
| Multi-Agent Verification [62] | Qwen3-8B | 78.5 | 80.2 | 86.1 | 86.7 | 82.9 | |
| VerifiAgent [29] | Qwen3-8B | 69.8 | 65.8 | 78.6 | 61.3 | 68.9 | |
| Themis [63] | Themis-7B | 82.7 | 42.3 | 45.7 | 37.1 | 52.0 | |
| RRM [115]† | RRM-7B | 87.7 | 70.4 | 80.7 | 90.0 | 82.2 | |
| RM-R1 [116]† | RM-R1-Distill-7B | 88.9 | 66.2 | 78.4 | 87.0 | 80.1 | |
| Think-RM [117]† | Think-RM-7B | 94.4 | 77.9 | 85.2 | 86.4 | 86.0 | |
| RM-Bench | TIR-Judge [28]† | TIR-Judge-8B | 80.1 | 91.9 | 69.0 | 93.9 | 83.7 |
| Agentic Reward Model [19] | ArmoRM-Llama3-8B* | 60.0 | 58.3 | 54.5 | 86.8 | 64.9 | |
| Multi-Agent Verification [62] | Qwen3-8B | 62.6 | 67.8 | 52.4 | 84.9 | 66.9 | |
| VerifiAgent [29] | Qwen3-8B | 41.3 | 80.0 | 45.9 | 74.5 | 60.4 | |
| Themis [63] | Themis-7B | 48.4 | 49.9 | 53.9 | 48.4 | 50.2 | |
| RRM [115]† | RRM-7B | 58.4 | 81.8 | 56.7 | 84.9 | 70.4 | |
| RM-R1 [116]† | RM-R1-Distill-7B | 64.0 | 83.9 | 56.2 | 85.3 | 72.4 | |
| Think-RM [117]† | Think-RM-7B | 69.3 | 76.0 | 56.5 | 93.7 | 73.9 | |
| JudgeBench | TIR-Judge [28]† | TIR-Judge-8B | 71.8 | 70.7 | 60.8 | 71.7 | 66.7 |
| Agentic Reward Model [19] | ArmoRM-Llama3-8B* | 63.4 | 49.2 | 47.0 | 56.7 | 51.9 | |
| Multi-Agent Verification [62] | Qwen3-8B | 54.3 | 42.5 | 45.7 | 39.0 | 44.8 | |
| VerifiAgent [29] | Qwen3-8B | 50.0 | 39.6 | 41.6 | 53.9 | 45.5 | |
| Themis [63] | Themis-7B | 42.9 | 48.0 | 54.8 | 51.6 | 51.1 | |
| RRM [115]† | RRM-7B | 83.2 | 61.9 | 64.3 | 64.2 | 67.0 | |
| RM-R1 [116]† | RM-R1-Distill-7B | 82.1 | 71.4 | 64.9 | 62.2 | 67.7 | |
| Think-RM [117]† | Think-RM-7B | 67.9 | 42.9 | 67.5 | 67.3 | 64.6 |
Table 3. Local reward-model results on shared judge-quality benchmarks.
Table 3 shows no uniform agentic advantage. On RM-Bench, only TIR-Judge clearly exceeds the non-agentic reasoning reward models; the remaining agentic systems Note: Evaluand-form, evaluand, domain, and judgment-signal cells use controlled labels. Pair-wise rows provide candidate responses or completions with labels, whereas point-wise and step-wise rows test reference-based verification or process-error localization. In the URL column, § marks a GitHub artifact, õ marks a Hugging Face dataset artifact, and marks a project page.
fall below them. On JudgeBench, strong non-agentic models match or slightly surpass the best agentic entry, while most agentic judges perform less well, particularly outside their strongest categories. These patterns are consistent with the effectiveness of preference- and reasoning-oriented reward training on fixed response pairs. They also limit the conclusions that can be drawn about active evidence-seeking, which these comparisons only partly capture.
NLG Evaluation
NLG evaluators assess generated outputs together with task inputs and any source documents, dialogue context, or references. Validation measures agreement with human quality, factuality, error, or facet annotations on those outputs rather than downstream task gain. BLEU, ROUGE, and BERTScore remain common baselines, but only partially align with these judgments in summarization [4], dialogue [118, 119], and story generation [120, 121].
Table 4 catalogs datasets by evaluand, domain, and human signal. Although no single suite spans these settings, they share the goal of recovering human judgments of free-form language quality [13, 34].
NLG quality is multidimensional and often under-specified by a single rubric or reference. Agentic evaluators make that ambiguity inspectable: criteria are separated across agents, borderline cases are debated, rationales are compared, and preliminary scores are revised before aggregation or arbitration. ChatEval [34] is a representative multi-agent debate instance; MAJ-EVAL [35] and CollabEval [59] likewise evaluate free-form text against human quality annotations.
Specialized directions within NLG evaluation focus on particular quality dimensions or generation tasks. Summary factuality systems, including MADISSE [124] and NarrativeFactScore [24], assess whether generated summaries are supported by the source, focusing on factual consistency rather than overall summary quality. Machine-translation evaluators such as HiMATE [73] and M-MAD [72] assess translation quality through MQM and MT-metric targets, with translation errors as the central evaluation signal.
TALE [85] and Multi-Agent LLM Judge [125] move toward answer correctness, reference consistency, or retrieval support, so QA task datasets used only as answer sources should not be treated as NLG evaluator benchmarks.
Judge-reliability and meta-evaluation work instead examines judgments or judging procedures. JudgeBench [111], LLMBar, MLLM-Judge, and JudgeAnything test judgment correctness using response pairs, preference labels, or multimodal cases. They belong closer to the reward-model and judge-quality axis, including when used to study debate, aggregation, or stopping [81, 77].
Reward modeling offers the densest shared comparisons, while NLG reporting remains dataset- and lineage-local. Both primarily validate preference or annotation agreement; dedicated agentic benchmarks are needed to establish when evidence-seeking improves verdict reliability.
Application
Applications extend judgment beyond final text to reasoning, evidence, visual details, interaction histories, and professional constraints. Figure 12 maps general and professional domains; Table 5 summarizes their judgment targets, fixed-context evaluation gaps, and corresponding agentic capabilities.
| Domain | Judgment target | Evaluation gap | Agentic capability |
| General domains | |||
| Math and code | Reasoning steps and executable outputs | Correctness may depend on execution, equivalence, or hidden step errors. | Execution, solver checks, and multi-verifier agreement |
| Fact-checking | Claims, verdicts, and justifications | A label is weak without retrieved support and justification checks. | Retrieval, evidence grounding, and justification verification |
| Conversation | Multi-turn behavior and responses | Quality depends on goals, reactions, and state across turns. | User simulation, memory, and multi-perspective review |
| Multimodal and vision | Images and multimodal outputs | Evidence is often localized in regions, pages, or cross-modal details. | Visual inspection, targeted questioning, and multimodal tools |
| Professional domains | |||
| Medicine | Clinical reports and medical dialogue | Errors are safety-sensitive and require expertise and uncertainty handling. | Role-specialized evaluation, tool-assisted scoring, and simulation |
| Law | Legal arguments and verdicts | Plausible verdicts may miss opposing arguments or disagreement. | Adversarial debate, deliberation, and consensus control |
| Finance | Reports, agents, and risk trajectories | Judgment must account for report logic, time, and trajectory failures. | Planning, logic extraction, and trajectory audit |
| Education | Answers, rubrics, and rationales | Scores need consistent rubrics and inspectable grading evidence. | Rubric planning, evidence recognition, cross-review, and refinement |
Table 5. Application-specific requirements for Agent-as-a-Judge. Each domain is summarized by its judgment target, the limitation of fixed-context judging, and the agentic capability used to address it.
Figure 12. Application map of Agent-as-a-Judge. General domains mainly stress verifiability, evidence gathering, interaction, and multimodal inspection, while professional domains add stronger requirements on expertise, traceability, safety, and consistency.
General Domains
General domains emphasize correctness checking, evidence retrieval, interaction simulation, and multimodal inspection.
Math and Code 1) Step checking inspects intermediate reasoning rather than relying on the final answer alone. HERMES [71] anchors mathematical reasoning to intermediate formal proof steps, while VerifiAgent [29] separates high-level reasoning assessment from tool-based correctness verification. 2) Equivalence verificaNote: *Agentic Reward Model uses ArmoRM-Llama3-8B as the base reward model, with Llama3-8B Instruct and Qwen2.5-Coder-7B used inside the verification modules. †Results are extracted from original-paper comparison tables. RewardBench and RM-Bench entries are converted to percentages for consistency with JudgeBench.
| Benchmark / dataset | Evaluand form | Evaluand | Domain | Human signal | Scale | URL |
| SummEval [4] | point-wise | summary | summarization | quality rating | 1,600 | § õ |
| TopicalChat-USR [118, 119] | point-wise | response | dialogue | quality rating | 360 | § |
| QAGS [122] | point-wise | summary | summarization | factuality label | 474 | § |
| ROCStories [120, 60] | point-wise | story | story generation | error score | 200 | õ |
| WritingPrompts [121, 60] | point-wise | story | story generation | error score | 200 | õ |
| StorySparkQA [123, 35] | point-wise | QA pair | story QA | expert rating | 70 | § õ |
| MSLR-COCHRANE [5, 35] | point-wise | summary | biomedical summarization | facet rating | 102 | § õ |
Table 4. NLG evaluation benchmarks.
tion checks answers with different surface forms. CompassVerifier [56] and xVerify [57] target this setting by checking mathematical and logical outputs beyond literal string match. 3) Signal aggregation combines multiple checks. Multi-Agent Verification [62] distributes assessment across aspect-specific judges, Agentic Reward Modeling [19] combines preference supervision with verifiable correctness signals, and Popper [58] treats judgment as controlled falsification through statistical tests.
Fact-Checking
Evidence grounding retrieves external support before assigning a verdict. UrduFactCheck [23] uses multilingual retrieval and evidence boosting to improve robustness in low-resource fact checking. 2) Justification audit evaluates both the verdict and the explanation that supports it. Adaptive audit and dialectical checking assess verdicts against justifications or supporting and refuting evidence before further investigation or closure [27, 82]. 3) Context modeling tracks long-range factual relations. Narrative-FactScore [24] builds character-level knowledge representations to detect state and relation errors in long narratives.
Conversation and Interaction
User simulation creates controllable interactions for evaluating behavior across turns. IntellAgent [52] uses interactive user simulations to synthesize conversational benchmarks, while the framework of Kazi et al. [126] introduces controllable user goals and automatic measures. 2) State tracking evaluates whether the system handles affective or social dynamics over time. ESC-Judge [53] evaluates emotional-support behavior through standardized coun-
seling skills, Sentient Agent [54] tracks emotion trajectories, and PSYCHE [55] builds psychiatric patient profiles for ethical assessment validation. 3) Perspective review combines reviewer positions on subjective and objective quality. Role-player summarization evaluation [127] frames evaluation as multi-perspective role play with diverse reviewer personas.
Multimodal and Vision 1) Visual generation evaluation checks prompt adherence, controls, and user-defined aspects. Tool orchestration and multi-round inspection assess control adherence, subject consistency, user-tailored aspects, and spatial constraints [50, 20, 98]. 2) Multimodal verification grounds claims, reasoning steps, and trajectory events in observable evidence. Visual questions, targeted image operations, and temporal inspection support image-text fact checking, reasoning rewards, and progress estimation from offline robot trajectories [51, 88, 32, 79].
Professional Domains
Professional domains add requirements for expertise, accountability, safety, and traceability to these evaluation needs.
Medicine 1) Clinical scoring evaluates medical responses or reports through specialized criteria. MAJ-Eval [35] constructs multiple evaluator personas to debate and cross-verify responses, while GEMA-Score [47] uses agent collaboration to compute granular, tool-assisted scores covering disease severity and uncertainty. 2) Clinical simulation evaluates medical behavior through interactive clinical roles. Chat-Coach [48] pairs patient and coach agents to critique trainee-doctor dialogues, while AI Hospital [49] evaluates LLM doctors in multi-agent clinical simulations, though final scoring may still rely on conventional metrics.
Law 1) Adversarial review exposes the judge to conflicting legal arguments. AgentsCourt [45] simulates prosecutors, defense attorneys, and judges so that the final judgment is conditioned on opposing positions. 2) Bench deliberation models how legal decision makers negotiate agreement or preserve dissent. SAMVAD [46] and AgentsBench [36] simulate judicial deliberation, making concurring and dissenting opinions part of the evaluation process.
Finance 1) Report logic checks whether a long-form financial analysis follows a coherent research structure. FinResearchBench [41] extracts logic trees from analyst reports as intermediate structures for assessment, while FinDeepResearch [42] synthesizes hierarchical rubrics for deep financial research evaluation. 2) Risk trajectory audits how financial agents behave over time or across teams. SAEA [43] audits agent trajectories to detect hallucination and temporal misalignment, while From Tasks to Teams [44] traces multi-agent failures such as cross-agent divergence and error propagation.
Education 1) Structured grading makes scoring depend on explicit criteria, evidence recognition, and cross-review. Grade-Like-a-Human [40] and AutoSCORE [38] decompose grading into rubric construction, evidence recognition, and score review. 2) Grading calibration diagnoses disagreement and refines the standards used for later grading. MAJ-Eval [35] uses multi-persona debate to approximate multidimensional human evaluation, while GradeOpt [39] introduces agents that diagnose discrepancies and iteratively refine grading guidelines.
Discussion
The mechanisms that expand what agentic judges can inspect also introduce deployment and governance challenges. We examine these challenges and directions for making evaluation selective, reliable, and controllable.
Challenges
Efficiency Additional evaluation actions increase cost and latency. Training tool use, planning, or adaptive routing may require trajectory data or reinforcement learning rather than simple supervised labels. At inference time, reasoning, tool calls, memory access, and multi-agent coordination incur costs that compound when later checks depend on earlier observations. The resulting latency limits online evaluation, moderation, and reinforcement-learning feedback loops. Efficiency therefore depends on balancing the expected value of further checking against its computational and interaction costs.
Process Reliability Errors can arise from intermediate actions as well as judge-model bias: planning may target the wrong claim, retrieval may supply weak evidence, tool results may be misinterpreted, and deliberation may converge on an incorrect consensus. Memory can preserve these mistakes across steps or cases. A verdict may consequently appear well supported despite failures in its evidence trace. Reliability assessment must therefore examine plans, observations, memory updates, and resolution rules alongside final verdicts.
Safety and Control Access to search engines, executors, databases, and interactive environments exposes judges to prompt injection, tool misuse, and unintended side effects. Multi-agent protocols can propagate unsafe rationales or adversarial instructions through shared outputs. When verdicts become reward signals, downstream optimization may reinforce systematic judging errors. Deployment therefore requires bounded tool permissions and external actions, explicit stopping rules, and escalation paths when automated evaluation cannot safely resolve a case.
Privacy and Governance Persistent evaluation state may contain user information, evidence, prior verdicts, or personalized criteria. Reuse can leak sensitive records, transfer user-specific standards across contexts, or perpetuate outdated judgments, particularly in professional domains involving confidential or identifiable information. Governance must specify what may be stored, retention periods, access rights, deletion requirements, and permitted reuse across cases.
Future Directions
These challenges motivate advances in evaluation control, memory management, evidence creation, and learned judging policies.
Adaptive Control Judges should allocate evaluation effort according to the case: use lightweight assessment when sufficient, decompose difficult cases, and escalate weak evidence to additional tools, agents, or human review. This requires adapting criteria, granularity, and evidence depth while accounting for both the cost and error risk of further actions. Generalization thus includes transferring control strategies across cases rather than applying a fixed rubric and evaluation budget.
Memory Management Memory management should become an explicit part of the judging policy, with decisions about storage, summarization, retrieval, revision, and forgetting. The aim is to support personalization, consistency, and cross-case learning while keeping evidence current and reuse appropriate to the target case. In professional domains, these decisions must also respect confidentiality and distinguish reusable standards from case-specific information.
Autonomous Probing Current probing through tool calls, visual questions, simulations, and environment interaction is often local, predefined, or tied to narrow interfaces. Future judges could generate targeted tests, counterexamples, alternative prompts, environment states, or stress cases when the initial context is insufficient. The challenge is to select probes that reduce judgment uncertainty or expose specific failure boundaries, making evidence creation purposeful rather than merely extending the evaluation trace.
Trainable Judgment Policies Learning could replace hand-designed prompts, workflows, and routing rules with policies for decomposition, tool selection, evidence sufficiency, deliberation, and stopping. Reinforcement learning and process-level supervision offer training signals that can account for verdict quality together with trace cost and reliability. The central difficulty is stability: improvements in planning, tool use, and coordination must preserve controllability and auditability.
Progress should be measured by selective, evidence-grounded, and accountable evaluation: whether the judge seeks appropriate evidence, preserves useful state, and stops or escalates at the right point.
Conclusion
This survey organizes Agent-as-a-Judge around the construction and control of evaluation processes. We established operational criteria for evaluation-directed checks and the use of their results, and connected multi-agent collaboration, tool integration, memory, and planning to the findings on which a verdict rests. We further mapped validation settings, benchmark coverage, and application requirements across general and professional domains. While promising, this evolution presents challenges in computational cost, latency, safety, and privacy. Future progress should prioritize personalization, generalization, and optimization, ultimately realizing truly autonomous evaluators that continuously adapt to the evolving AI landscape.
Supplementary Material
This appendix collects material that supports the survey beyond the main text.
Paper Inclusion Criteria
Section 2 fixes the conceptual boundary of Agent-as-a-Judge; this appendix makes the corresponding screening standard explicit. Screening uses the reported checking procedure, its intermediate results, and their role in verdict formation. The mechanism-specific descriptions below follow the organization of Section 3.
Working definition An agentic judge constructs a verdict about an evaluand through explicit checking operations and the use of their results. Qualifying operations obtain evidence, perform targeted verification, or reassess a judgment using another check’s findings or rationale, beyond replicating an overall score. They may follow a prescribed workflow or adapt to intermediate observations.
Minimal inclusion rule A paper is included when all three conditions hold: (1) Evaluation purpose. Its main contribution is a judge, verifier, reward evaluator, or evaluation procedure that forms a verdict about a specified evaluand. (2) Substantive checking. At least one explicit operation obtains evidence, tests a claim or constraint, or reassesses a judgment using another check’s findings or rationale. Replicated overall scoring, including independent persona variants, does not satisfy this condition on its own. (3) Result use. The reported procedure specifies how the check result is consumed by subsequent assessment or verdict formation. Qualifying results include retrieved passages, execution outcomes, claim-specific verification decisions, extracted discrepancies, and critiques used for reassessment. Counting independent overall scores or labels alone does not satisfy conditions (2)–(3).
Screening record The unit of assessment is a reported system configuration or protocol. For each qualifying process, the screening record identifies the checked object, the checking operation, its output, the downstream consumer, and the source passage or algorithm establishing that dependency. For delegated verification, the checked proposition, constraint, or comparison must be identifiable; changing a persona name or requesting another generic dimension score is insufficient. Inclusion is based on the reported use of results, without requiring that the verdict change on every input. Configurations with insufficient procedural detail are marked unresolved rather than inferred from an agent or tool label.
Control mode For an included configuration, control is characterized separately as prescribed execution, input-conditioned planning, or observation-driven revision. A fixed rule can implement observation-driven control when a returned finding triggers a branch, further check, escalation, or stopping. Conversely, input-specific plans or changing discussion content do not by themselves establish revision of the checking procedure.
Table 6 contrasts common protocols; the following descriptions show how the four mechanisms realize the inclusion conditions.
| Protocol pattern | In scope? | Decisive procedural evidence |
| One-shot scoring or CoT alone | No | No separate checking operation and result-use dependency. |
| Independent overall scores followed by voting or averaging | No | Replicated verdicts are counted without further verification or reassessment. |
| Different personas independently score the same object | No, alone | Role descriptions change, but no distinct verification task or use of peer findings is established. |
| Fixed-round critique and reassessment | Yes | A later assessment consumes another evaluator’s findings or rationale before aggregation. |
| Independent targeted verifiers followed by aggregation | Yes | Each verifier checks an identifiable proposition or constraint; the resolver uses those verification results. |
| Fixed retrieval or execution before judgment | Yes | Retrieved evidence or test outcomes are used to assess the evaluand. |
| Plan generation without executed checks | No, alone | The plan is produced but no corresponding checking results are used. |
| Findings trigger a new check, a revised query, or stopping | Yes | Both a qualifying check and observation-driven control are explicit. |
Table 6. Operational boundary examples. These are protocol patterns; a paper may report more than one configuration.
Planning The judge constructs or selects an explicit plan, rubric, or decomposition that organizes subsequent checks, whose results inform the verdict. The plan may be executed as a prescribed workflow or revised in response to critiques, tests, counterexamples, or other intermediate findings. A plan expressed only as reasoning text, without corresponding checking operations, does not qualify on its own.
Tool integration 1) Tool use. The judge invokes external tools to gather evidence—search or retrieval, compilers or executors, provers, browsers or GUIs, databases—and tool results can change the verdict. 2) Interaction with external state. The judge probes or acts on files, a GUI, a simulator, the web, or executable artifacts to obtain observations used in evaluation. Tool calls may follow a prescribed sequence or be selected from intermediate findings; the external state itself need not change during evaluation.
Multi-agent collaboration The judge delegates identifiable verification tasks or coordinates the use of peer findings and rationales in subsequent assessment. The decisive evidence is the checking responsibility or information dependency, rather than role diversity or the final aggregation operator. Prescribed critique exchanges can satisfy this condition even when the speaking order and number of rounds are fixed; parallel verifiers can satisfy it when they test distinct propositions or constraints rather than replicate overall scoring. For example, peer-informed reassessment appears in debate and collaboration protocols [34, 59], while structured clinical comparison [47] and aspect verification [62] organize distinct checking tasks.
Memory The system maintains an explicit write–manage–read loop over a memory store that affects later actions or the verdict, rather than relying only on a longer context window, a fixed rubric in the prompt, or a chain-of-thought scratchpad. The reported procedure must specify how stored findings or artifacts are accessed and used in evaluation.
Non-examples The following patterns do not qualify on their own: 1) multi-step reasoning in a single forward pass; 2) a generated plan or rubric that is not used to organize subsequent checking operations; 3) replicated overall scoring, including independent persona variants, or self-consistency sampling followed only by voting; 4) appending full history into context without a memory loop; 5) intermediate artifacts that are displayed but not used in subsequent assessment or verdict formation.
Common mis-inclusion cases 1) Standalone formal verification. A prover alone does not constitute an LLM-based judge; an evaluator that invokes a prover and uses its results is assessed by the same checking and result-use conditions. 2) RL training methods or infrastructure. Work whose main contribution is reinforcement-learning training or tooling rather than the judge architecture. 3) Judge as a sub-component. Systems where judging is ancillary to a larger pipeline.
Construction of MAD Subagent Figures
Scope and record construction This appendix documents the manual coding behind Figure 4 and Figure 5. The main text states how to read those figures; Table 7 lists every plotted record for spot-checking inclusion and axis assignments. The figures draw on the MAD judge papers analyzed in § 3: 24 papers and 102 role-level records. Each record names one active judging component in a paper’s protocol; one paper may contribute several records when it names several such positions. During manual review, paper-specific evidence sentences guide whether a named entity is retained and how it is coded on both axes. The functional and technical axes follow the definitions used with those figures in the multi-agent section. Table 7 reports them in abbreviated form in the Func. and Tech. columns; the first table caption defines the abbreviations. We retain only components that produce, check, discuss, aggregate, route, or authorize a judgment within the evaluation run.
| Source paper | Subagent/component | Func. | Tech. |
| Table-Critic [33] | Judge Agent | V-Ctrl. | Tool/env. |
| Critic Agent | Adv. | Tool/env. | |
| Refiner Agent | Analyt. | Workflow | |
| Curator Agent | Analyt. | Workflow | |
| Meta-Judges [81] | Meta-judge agents | Analyt. | Workflow |
| Expert role | Persp. | Prompt | |
| Critic role | Adv. | Prompt | |
| General Public role | Persp. | Prompt | |
| Mediator or summarization role | V-Ctrl. | Workflow | |
| Judging with Many Minds [128] | General Public agent | Persp. | Prompt |
| Critic agent | Adv. | Tool/env. | |
| Judge pool agents | Persp. | Ctrl. | |
| Meta-Judge | V-Ctrl. | Prompt | |
| PINE bias-free agent | Analyt. | Prompt | |
| Adaptive Stability Detection [77] | Judge agents | Persp. | Prompt |
| Adaptive stability detector | V-Ctrl. | Ctrl. | |
| Majority-vote fallback | V-Ctrl. | Ctrl. | |
| Efficient Safety MAD [80] | Critic Agent | Adv. | Prompt |
| Defender Agent | Adv. | Prompt | |
| Judge Agent | V-Ctrl. | Tool/env. | |
| CollabEval [59] | Evaluator agents | Analyt. | Workflow |
| Discussion participants | Persp. | Prompt | |
| Consensus checker/control logic | V-Ctrl. | Ctrl. | |
| Final judge | V-Ctrl. | Workflow | |
| AgentV-RL [78] | Forward agent | Analyt. | Tool/env. |
| Backward agent | V-Ctrl. | Prompt | |
| Multi-Agent LLM Judge [125] | Sample Selection agent | Persp. | Prompt |
| Evaluation agent | V-Ctrl. | Prompt | |
| ReWrite agent | V-Ctrl. | Workflow |
Table 7. Manually identified MAD subagent/component records used for figure construction, part 1 of 3. Func. abbreviates functional role: Adv. = adversarial, Persp. = perspective, Analyt. = analytical, and V-Ctrl. = verdict-control. Tech. abbreviates implementation status: Prompt = persona prompts, Workflow = structured workflows, Ctrl. = deterministic controllers, and Tool/env. = tool-integrated components.
Role lexicon figure Figure 4 assigns each record one functional role and one manually chosen display term kept close to the source paper’s role vocabulary. Each plotted record contributes one count to its functional group; term size within a panel reflects how often that display term appears among records assigned to that group. At render time, each display term is tokenized on whitespace, and the generic tokens judge, role, and agent are removed so panel emphasis falls on distinctive role wording rather than shared MAD naming conventions.
Role-to-implementation Sankey Figure 5 uses the same 102 records but counts functional-role/implementation-status pairs rather than display-term frequencies. Implementation status is coded from how the component is realized in the protocol, not from its role name alone. Persona prompts mark a fixed persona or evaluation dimension without adaptive procedure selection. Structured workflows mark LLM steps coupled to specific upstream and downstream artifacts. Deterministic controllers mark rule-based logic that routes, stops, or aggregates the run. Tool-integrated components mark positions whose judging action invokes tools or external evidence through a predefined interface. Table abbreviations for these classes appear in the first table caption.
Construction of the Tool-Interface Count Figure
Scope and counting unit This appendix documents the manual coding behind Figure 6. The main text states how to read that figure; Table 10 lists every counted paper–interface assignment for spot-checking inclusion and category labels.
| Interface | Count | Share | Papers |
| Code Exec | 17 | 28.3% | [19, 18, 70, 29, 65, 78, 129, 130, 99, 131, 86, 87, 132, 102, 89, 93] [28] |
| Web Search | 13 | 21.7% | [27, 23, 19, 18, 29, 45, 63, 84, 85, 131, 86, 95, 70] |
| Local Search | 8 | 13.3% | [24, 18, 45, 46, 96, 99, 86, 87] |
| Vision Model | 7 | 11.7% | [50, 20, 90, 70, 97, 88, 89] |
| Solver | 5 | 8.3% | [29, 63, 76, 132, 86] |
| Image Ops | 3 | 5.0% | [50, 32, 70] |
| Shell | 3 | 5.0% | [70, 102, 89] |
| Computer Use | 2 | 3.3% | [70, 89] |
| Translation | 2 | 3.3% | [23, 63] |
| Total | 60 | 100% | 34 distinct papers |
Table 10. Counted tool-interface assignments for Figure 6. Share is each category count divided by the 60 counted interface uses. Interface abbreviations match the figure legend.
The figure summarizes a manually coded subset of the tool-augmented judges discussed in the survey: 34 papers and 60 counted interface assignments. It is not an exhaustive count of all cited methods. Each assignment links one paper to one interface category; a paper may receive several assignments when its judging protocol uses several external interfaces, but at most one assignment per category. The figure denominator is therefore 60 counted interface uses, not 34 papers. The Share column reports each category count divided by 60; it is not the share of papers using that interface.
During manual review, paper-specific descriptions of the judging protocol guide whether an external interface is retained and which category it receives. We count only interfaces that the judge can call during judgment production, evidence gathering, checking, or interaction. We exclude tools used only by the evaluand or candidate generator, offline preprocessing outside the judging loop, and papers without tool-augmented judging interfaces.
Interface categories Each row in Table 10 uses one of nine interface categories below. Longer label text appears in the figure legend; the coding rules state what qualifies for each abbreviated label. Code Exec covers general programmatic execution through an interpreter or sandboxed runtime during judging. Web Search covers open-web retrieval or live search APIs. Local Search covers fixed corpora, structured stores, knowledge graphs, indexed artifacts, or lookup in input tables rather than open-web search. Vision Model covers vision-language, OCR, segmentation, or parsing models invoked as perception tools. Solver covers dedicated symbolic solvers, theorem provers, or calculator interfaces counted separately from general code execution. Image Ops covers deterministic image operations such as crop, zoom, diff, or overlay that produce inspectable visual evidence. Shell covers terminal, bash, or shell-level command execution. Computer Use covers GUI or environment interaction beyond a single static capture, including scripted UI actions or computer-use agents. Translation covers machine-translation interfaces used while judging cross-lingual inputs.
Complete Evaluation–Benchmark Coverage Map
Scope and counting unit This appendix documents the manual coding behind Figure 11 and Figure 13. The main text states how to read the simplified map; Figure 13 lists every coded method–benchmark link for spot-checking coverage and benchmark labels.
Figure 13. Complete evaluation–benchmark coverage map. Unlike Figure 11, this version retains all 100 coded links and singleton benchmark nodes. Links indicate that a method reports results on, or is explicitly evaluated on, a reported benchmark. Right-node fill and pattern follow Figure 11; see that caption for the artifact-type legend.
The maps draw on the evaluation methods analyzed in § 6: 28 methods, 100 method–benchmark links, and 68 normalized benchmark nodes in the complete appendix figure. Each link records that a method reports results on, or is explicitly evaluated on, a named benchmark in its source paper. A method may contribute several links when it reports several benchmarks, but at most one link per method–benchmark pair after light name normalization. During manual review, paper-specific source notes guide whether a benchmark name is retained and which evaluation family it supports.
| Source paper | Subagent/component | Func. | Tech. |
| ChatEval [34] | Debater/referee agents | Adv. | Prompt |
| General Public | Persp. | Prompt | |
| Critic | Adv. | Prompt | |
| News Author | Persp. | Prompt | |
| Psychologist | Persp. | Prompt | |
| Scientist | Persp. | Prompt | |
| Summarizer | V-Ctrl. | Workflow | |
| Multi-Agent-as-Judge [35] | Stakeholder agents | Persp. | Workflow |
| In-group moderator | Persp. | Prompt | |
| Out-group/impartial aggregator | V-Ctrl. | Workflow | |
| Persona creation LLM | Persp. | Workflow | |
| SAMVAD [46] | Judge Agent | V-Ctrl. | Tool/env. |
| Prosecution Counsel Agent | Adv. | Tool/env. | |
| Defense Counsel Agent | Adv. | Tool/env. | |
| Adjudicator Agents | Analyt. | Prompt | |
| Orchestrator | V-Ctrl. | Ctrl. | |
| M-MAD [72] | Accuracy expert | Persp. | Prompt |
| Fluency expert | Persp. | Prompt | |
| Terminology expert | Persp. | Prompt | |
| Style expert | Persp. | Prompt | |
| Debater A | Adv. | Prompt | |
| Debater B | Adv. | Prompt | |
| Consensus checker | V-Ctrl. | Ctrl. | |
| Final judge | V-Ctrl. | Prompt | |
| AgentsCourt [45] | Judge agent | V-Ctrl. | Tool/env. |
| Plaintiff agent | Adv. | Prompt | |
| Defendant agent | Adv. | Prompt | |
| Judge Assistant agent | Analyt. | Tool/env. | |
| AgentsBench [36] | Presiding professional judge | V-Ctrl. | Ctrl. |
| Professional judge agents | Persp. | Ctrl. | |
| Lay judge or juror agents | Persp. | Ctrl. | |
| MATEval [60] | Evaluator Agent | Analyt. | Prompt |
| Feedback Agent | V-Ctrl. | Prompt | |
| Summarizer Agent | V-Ctrl. | Workflow | |
| Multi-agent verification [62] | Aspect Verifiers | Analyt. | Prompt |
| Verification aggregator/selector | V-Ctrl. | Ctrl. | |
| Verifier engineer or validation selection procedure | Analyt. | Prompt |
Table 8. Manually identified MAD subagent/component records used for figure construction, part 2 of 3.
The main-text figure drops links whose benchmark appears for only one method, leaving 47 links, 15 shared benchmark nodes, and 21 methods with at least one shared benchmark. It therefore emphasizes comparable empirical surfaces rather than the full coverage set. Figure 13 restores singleton benchmarks and is intended as a completeness check, not a separate taxonomy.
We code benchmark nodes at the benchmark-series level. Name normalization groups related releases under a shared series label. We exclude links supported only by local reruns or other evidence outside a method’s originally reported benchmark set.
Benchmark artifact types Each benchmark node is assigned one artifact type for visual encoding in both figures. The three classes follow the legend in Figure 11. Fixed response or judgment sets package candidate responses, rankings, or step labels for direct judge comparison. Task benchmarks requiring generated candidates supply tasks, references, or checks whose reported results require generating candidates before judging. NLG evaluation datasets provide open-text outputs with human quality, factuality, or facet labels for evaluator meta-evaluation. Candidate extensions in the plotting data are rendered with muted styling when they appear in the simplified main-text figure. For boundary cases excluded from the NLG-evaluation axis, see § 6.2.
| Source paper | Subagent/component | Func. | Tech. |
| FACT-AUDIT [27] | Appraiser agent | V-Ctrl. | Prompt |
| Inquirer agent | Analyt. | Tool/env. | |
| Quality Inspector agent | Analyt. | Workflow | |
| Evaluator agent | Analyt. | Prompt | |
| Prober agent | Analyt. | Workflow | |
| MATH-VF [76] | Formalizer | Analyt. | Workflow |
| Critic | Adv. | Workflow | |
| SymPy-oriented tool agent | Analyt. | Tool/env. | |
| Z3-solver-oriented tool agent | Analyt. | Tool/env. | |
| No-human-in-the-loop [83] | CI Pattern Audit Agent | Analyt. | Prompt |
| Recommendation Issue Audit Agent | Analyt. | Prompt | |
| Report Generation Agent | Analyt. | Ctrl. | |
| Multiple LLM judges | V-Ctrl. | Ctrl. | |
| Consensus synthesizer/conflict policy | V-Ctrl. | Ctrl. | |
| CodeVisionary [70] | Central-control LLM agent | Analyt. | Tool/env. |
| Context distillation agent | Analyt. | Tool/env. | |
| Multiple judge agents | Persp. | Ctrl. | |
| Report generator/summarizer | V-Ctrl. | Workflow | |
| Diverse role-players [127] | General Public Judger | Persp. | Workflow |
| Critic Judger | Persp. | Prompt | |
| News Author Judger | Persp. | Prompt | |
| Dynamic subjective users | Persp. | Prompt | |
| Aggregator | Persp. | Ctrl. | |
| Grade like a human [40] | Rubric generation agent/stage | Analyt. | Ctrl. |
| Grading agent/stage | V-Ctrl. | Workflow | |
| Post-grading review agent/stage | V-Ctrl. | Ctrl. | |
| Sampling component | Analyt. | Ctrl. | |
| HiMATE [73] | Tier-1 MQM category agents | Analyt. | Prompt |
| Subtype evaluator role | Analyt. | Workflow | |
| Self-reflection/correction role | Analyt. | Prompt | |
| Collaborative discussion role | Analyt. | Prompt | |
| Weighted scoring component | V-Ctrl. | Ctrl. | |
| GEMA-Score [47] | Entity Extraction Agent | Analyt. | Workflow |
| Objective Clinical Accuracy Agent | Persp. | Prompt | |
| Subjective Expressiveness Evaluation Agent | Persp. | Prompt | |
| Score Evaluation Agent | V-Ctrl. | Workflow | |
| Total | 102 records |
Table 9. Manually identified MAD subagent/component records used for figure construction, part 3 of 3.
References
- [1]Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.arxiv.org/abs/1706.03741
- [2]Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2009.01325
- [3]Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, and Jie Tang. A survey of post-training scaling in large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2771–2791, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.140. URL https://aclanthology.org/2025.acl-long.140/.DOI
- [4]Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409, 2021.arxiv.org/abs/2007.12626
- [5]Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey Kuehl, Erin Bransom, and Byron C. Wallace. Automated metrics for medical multi-document summarization disagree with human evaluations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9871–9889, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.549. URL https://aclanthology.org/2023.acl-long.549/.DOI
- [6]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2203.02155
- [7]Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshua Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2757–2791, 2025.arxiv.org/abs/2411.16594
- [8]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastassios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 8359–8388. PMLR, 2024. URL https://proceedings.mlr.press/v235/chiang24b.html.arxiv.org/abs/2403.04132
- [9]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, 2002.
- [10]Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, 2004.
- [11]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016.arxiv.org/abs/1606.05250
- [12]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Process- ing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc.
- [13]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruocheng Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.153. URL https://aclanthology.org/2023.emnlp-main.153/.
- [14]Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2023.arxiv.org/abs/2310.08491
- [15]Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=xsELpEpPn4A.
- [16]Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif vs. rlhf: scaling reinforcement learning from human feedback with ai feedback. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2309.00267
- [17]Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On LLMs-driven synthetic data generation, curation, and evaluation: A survey. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 11065–11082, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.658. URL https://aclanthology.org/2024.findings-acl.658/.DOI
- [18]Mingchen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang, Dmitri Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. Agent-as-a-judge: Evaluate agents with agents. In Forty-second International Conference on Machine Learning, 2025.
- [19]Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou, and Juanzi Li. Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15934–15949, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.775. URL https://aclanthology.org/2025.acl-long.775/.DOI
- [20]Fan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao, and Ziwei Liu. Evaluation agent: Efficient and promptable evaluation framework for visual generative models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7561–7582, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.374. URL https://aclanthology.org/2025.acl-long.374/.DOI
- [21]Wentao Shi, Yu Wang, Yuyang Zhao, Yuxin Chen, Fuli Feng, Xueyuan Hao, Xi Su, Qi Gu, Hui Su, Xunliang Cai, and Xiangnan He. AJ-Bench: Benchmarking agent-as-a-judge for environment-aware evaluation, 2026. URL https://arxiv.org/abs/2604.18240. Accepted to ACL 2026 Findings.
- [22]Hongru Wang, Cheng Qian, Manling Li, Jiahao Qiu, Boyang Xue, Mengdi Wang, Heng Ji, Amos Storkey, and Kam-Fai Wong. Position: Agent should invoke external tools ONLY when epistemically necessary, 2026. URL https://arxiv.org/abs/2506.00886.
- [23]Sarfraz Ahmad, Hasan Iqbal, Momina Ahsan, Numaan Naeem, Muhammad Ahsan Riaz Khan, Arham Riaz, Muhammad Arslan Manzoor, Yuxia Wang, and Preslav Nakov. UrduFactCheck: An agentic fact-checking framework for Urdu with evidence boosting and benchmarking. In Christos Christodoulopoulos, Tannoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, pages 22788–22802, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.1240. URL https://aclanthology.org/2025.findings-emnlp.1240/.DOI
- [24]Yeonseok Jeong, Minsoo Kim, Seung-won Hwang, and Byung-Hak Kim. Agent-as-judge for factual summarization of long narratives. In Christos Christodoulopoulos, Tannoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 23602–23619, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.1204. URL https://aclanthology.org/2025.emnlp-main.1204/.DOI
- [25]Reshmi Ghosh, Tianyi Yao, Lizzy Chen, Sadid Hasan, Tianwei Chen, Dario Bernal, Huitian Jiao, and HM Hossain. Sageval: The frontiers of satisfactory agent based nlg evaluation for reference-free open-ended text. arXiv preprint arXiv:2411.16077, 2024.arxiv.org/abs/2411.16077
- [26]Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101, 2025.arxiv.org/abs/2309.07864
- [27]Hongzhan Lin, Yang Deng, Yunxuan Gu, Wenxuan Zhang, Jing Ma, See-Kiong Ng, and Tat-Seng Chua. FACT-AUDIT: An adaptive multi-agent framework for Ran Xu, Jingjing Chen, Jiayu Ye, Yu Wu, Jun Yan, Carl Yang, and Hongkun Yu. Incentivizing agentic reasoning in llm judges via tool-integrated reinforcement learning, 2025.
- [29]Jiuzhou Han, Wray Buntine, and Ehsan Shareghi. VerifAgent: a unified verification agent in language model reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2025, 2025.arxiv.org/abs/2504.00406
- [30]Manya Wadhwa, Zayne Rea Sprague, Chaitanya Malaviya, Philippe Laban, Junyi Jessy Li, and Greg Durrett. Evalagents: Discovering implicit evaluation criteria from the web. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=erGpkHCybv.arxiv.org/abs/2504.15219
- [31]Weixiang Zhao, Xingyu Sui, Yulin Hu, Jiahe Guo, Haixiao Liu, Biye Li, Yanyan Zhao, Bing Qin, and Ting Liu. Teaching language models to evolve with users: Dynamic profile modeling for personalized alignment. In Proceedings of the Thirty-Ninth Conference on Neural Information Processing Systems (NeurIPS 2025), 2025.arxiv.org/abs/2505.15456
- [32]Shengyuan Ding, Xinyu Fang, Ziyu Liu, Yuhang Zhang, Yuhang Cao, Xiangyu Zhao, Haodong Duan, Xiaoyi Dong, Jianze Liang, Bin Wang, Conghui He, Dahua Lin, and Jiaq i Wang. Arm-thinker: Reinforcing multimodal generative reward models with agentic tool use and visual reasoning, 2025.arxiv.org/abs/2512.05111
- [33]Peiying Yu, Guoxin Chen, and Jingjing Wang. Table-critic: A multi-agent framework for collaborative criticism and refinement in table reasoning, 2025. URL https://arxiv.org/abs/2502.11799.
- [34]Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chatal: Towards better LLM-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=FQepisCUWu.arxiv.org/abs/2308.07201
- [35]Jiaj u Chen, Xueyan Lu, Jiaowei Wang, Humin Zeng, Jing Huang, Jiri Gesi, Ying Xu, and Dakuo Wang. Multi-agent-as-judge: Aligning LLM-agent-based automated evaluation with multi-dimensional human evaluation. In First Workshop on Multi-Turn Interactions in Large Language Models, 2025. URL https://openreview.net/forum?id=7AetgL7eVL.
- [36]Cong Jiang and Xiaolei Yang. Agentsbench: A multi-agent llm simulation framework for legal judgment prediction. Systems, 13(8):641, 2025.DOI
- [37]Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. A survey on evaluation of LLM-based agents. In Findings of the Association for Computational Linguistics: ACL 2026, pages 26690–26714, 2026. doi: 10.18653/v1/2026.findings-acl.1330. URL https://aclanthology.org/2026.findings-acl-1330/.DOI
- [38]Yun Wang, Zhaojun Ding, Xuansheng Wu, Siyue Sun, Ninghao Liu, and Xiaoming Zhai. Autoscore: Enhancing automated scoring with multi-agent large language models via structured component recognition. arXiv preprint arXiv:2509.21910, 2025.arxiv.org/abs/2509.21910
- [39]Yucheng Chu, Hang Li, Kaiqi Yang, Harry Shomer, Hui Liu, Yasemin Copur-Gencturk, and Jiliang Tang. A llm-powered automatic grading framework with human-level guidelines optimization. arXiv preprint arXiv:2410.02165, 2024.arxiv.org/abs/2410.02165
- [40]Wenjing Xie, Juxin Niu, Chun Jason Xue, and Nan Guan. Grade like a human: Rethinking automated assessment with large language models. arXiv preprint arXiv:2405.19694, 2024.arxiv.org/abs/2405.19694
- [41]Rui Sun, Zuo Bai, Wentao Zhang, Yuxiang Zhang, Li Zhao, Shan Sun, and Zhengwen Qiu. Finresearch-bench: A logic tree based agent-as-a-judge evaluation framework for financial research agents. In Proceedings of the 6th ACM International Conference on AI in Finance, pages 656–664, 2025.
- [42]Fengbin Zhu, Xiang Yao Ng, Ziyang Liu, Chang Liu, Xianwei Zeng, Chao Wang, Tianhui Tan, Xuan Yao, Pengyang Shao, Min Xu, et al. Findeeepresearch: Evaluating deep research agents in rigorous financial analysis. arXiv preprint arXiv:2510.13936, 2025.arxiv.org/abs/2510.13936
- [43]Zichen Chen, Jiaao Chen, Jianda Chen, and Misha Sra. Standard benchmarks fail – auditing llm agents in finance must prioritize risk, 2025. URL [https://arxiv.org/abs/2502.15865] (https://arxiv.org/abs/2502.15865).
- [44]Zichen Chen, Jianda Chen, Jiaao Chen, and Misha Sra. From tasks to teams: A risk-first evaluation framework for multi-agent llm systems in finance. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025.DOI
- [45]Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huajjun Li, Kang Liu, and Jun Zhao. Agentscourt: Building judicial decision-making agents with court debate simulation and legal knowledge augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9399–9416, 2024.arxiv.org/abs/2403.02959
- [46]Prathamesh Devadiga, Omkaar Jayadev Shetty, and Pooja Agarwal. Samvad: A multi-agent system for simulating judicial deliberation dynamics in india. arXiv preprint arXiv:2509.03793, 2025.arxiv.org/abs/2509.03793
- [47]Zhenxuan Zhang, Kinhei Lee, Weihang Deng, Huichi Zhou, Zihao Jin, Jiahao Huang, Zhifan Gao, Dominic C. Marshall, Yingying Fang, and Guang Yang. Gema-score: Granular explainable multi-agent score for radiology report evaluation. CoRR, abs/2503.05347, March 2025. URL https://doi.org/10.48550/arXiv.2503.05347.
- [48]Hengguan Huang, Songtao Wang, Hongfu Liu, Hao Wang, and Ye Wang. Benchmarking large language models on communicative medical coaching: a dataset and a novel system. In Findings of the Association for Computational Linguistics ACL 2024, pages 1624–1637, 2024.
- [49]Zhihao Fan, Lai Wei, Jialong Tang, Wei Chen, Wang Siyuan, Zhongyu Wei, and Fei Huang. AI hospital: Benchmarking large language models in a multi-agent medical interaction simulator. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10183–10213, 2025.arxiv.org/abs/2402.09742
- [50]Jifang Wang, Xue Yang, Longyue Wang, Zhenran Xu, Yiyu Wang, Yaowei Wang, Weihua Luo, Kaifu Zhang, Baotian Hu, and Min Zhang. A unified agentic framework for evaluating conditional image generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12626–12646, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.620. URL https://aclanthology.org/2025.acl-long.620/.DOI
- [51]Alimohammad Beigi, Bohan Jiang, Dawei Li, Tharindu Kumarage, Zhen Tan, Pouya Shaeri, and Huan Liu. Lrq-fact: Llm-generated relevant questions for multimodal fact-checking. arXiv preprint arXiv:2410.04616, 2024. URL https://arxiv.org/abs/2410.04616.
- [52]Elad Levi and Ilan Kadar. Intelligent: A multi-agent framework for evaluating conversational ai systems. arXiv preprint arXiv:2501.11067, 2025.arxiv.org/abs/2501.11067
- [53]Navid Madani and Rohini Srihari. ESC-judge: A framework for comparing emotional support conversational agents. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16059–16076, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.811. URL https://aclanthology.org/2025.emnlp-main.811/.
- [54]Bang Zhang, Ruotian Ma, Qingxuan Jiang, Peisong Wang, Jiaqi Chen, Zheng Xie, Xingyu Chen, Yue Wang, Fanghua Ye, Jian Li, et al. Sentient agent as a judge: Evaluating higher-order social cognition in large language models. arXiv preprint arXiv:2505.02847, 2025.arxiv.org/abs/2505.02847
- [55]Jingoo Lee, Kyungjo Lim, Young-Chul Jung, and Byung-Hoon Kim. Psyche: A multi-faceted patient simulation framework for evaluation of psychiatric assessment conversational agents. arXiv preprint arXiv:2501.01594, 2025.arxiv.org/abs/2501.01594
- [56]Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, Chengqi Lyu, Yuzhe Gu, Wenwei Zhang, Derek F. Wong, Songyang Zhang, and Kai Chen. CompassVerifier: A unified and robust verifier for LLMs evaluation and outcome reward. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33454–33482, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.1698. URL https://aclanthology.org/2025.emnlp-main.1698/.DOI
- [57]Ding Chen, Qingchen Yu, Pengyuan Wang, Wentao Zhang, Bo Tang, Feiyu Xiong, Xinchi Li, Minchuan Yang, and Zhiyu Li. xverify: Efficient answer verifier for reasoning model evaluations. arXiv preprint arXiv:2504.10481, 2025.arxiv.org/abs/2504.10481
- [58]Kevin Huang, Ying Jin, Ryan Li, Michael Y Li, Emmanuel Candes, and Jure Leskovec. Automated hypothesis validation with agentic sequential falsifications. In Proceedings of the 42nd International Conference on Machine Learning, 2025.arxiv.org/abs/2502.09858
- [59]Vivek Gupta, Anup Shirgaonkar, Tao Zhang, Zhixu Du, Justin Chih-Yao Chen, Xintong Li, Zhao Meng, Yujie Cheng, Congying Xia, and Yunyao Li. CollabEval: Enhancing LLM-as-a-judge via multi-agent collaboration, 2026. URL https://arxiv.org/abs/2603.00993.
- [60]Yu Li, Shenyu Zhang, Rui Wu, Xiutian Huang, Yongrui Chen, Wenhao Xu, Guilin Qi, and Dehai Min. MatEval: A multi-agent discussion framework for advancing open-ended text evaluation. In Database Systems for Advanced Applications: 29th International Conference, DASFAA 2024, Gifu, Japan, July 2-5, 2024, Proceedings, Part VII, pages 415–426, Berlin, Heidelberg, 2024. Springer-Verlag. ISBN 978-981-97-5574-5. doi: 10.1007/978-981-97-5575-2_31. URL https://doi.org/10.1007/978-981-97-5575-2_31.
- [62]Shalev Lifshitz, Sheila A McIlraith, and Yilun Du. Multi-agent verification: Scaling test-time compute with multiple verifiers. arXiv preprint arXiv:2502.20379, 2025.arxiv.org/abs/2502.20379
- [63]Lei Li, Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Ningyu Zhang, and Hua Wu. Tool-augmented reward modeling. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=d94x0gWTUX.arxiv.org/abs/2310.01045
- [65]Kaishuai Xu, Tiez heng Yu, Yi Cheng, Wenjun Hou, Liangyou Li, Xin Jiang, Lifeng Shang, Qun Liu, and Wenjie Li. Learning to align multi-faceted evaluation: A unified and robust framework. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 9488–9502, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256- 5. doi: 10.18653/v1/2025.findings-acl.494. URL https://aclanthology.org/2025.findings-acl.494/
- [70]Xinchen Wang, Pengfei Gao, Chao Peng, Ruida Hu, and Cuiyun Gao. Codevisionary: An agent-based framework for evaluating large language models in code generation, 2025.arxiv.org/abs/2504.13472
- [71]Azim Ospanov, Zijin Feng, Jiacheng Sun, Haoli Bai, Xin Shen, and Farzan Farnia. Hermes: Towards efficient and verifiable mathematical reasoning in llms, 2025. URL https://arxiv.org/abs/2511.18760.
- [72]Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, and Zuozhu Liu. M-MAD: Multidimensional multi-agent debate for advanced machine translation evaluation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7084–7107, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.351. URL https://aclanthology.org/2025.acl-long.351/.DOI
- [73]Shijie Zhang, Renhao Li, Songsheng Wang, Philipp Koehn, Min Yang, and Derek F. Wong. HiMATE: A hierarchical multi-agent framework for machine translation evaluation. In Christos Christodoulopoulos, Tammoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11121–11145, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.593. URL https://aclanthology.org/2025.findings-emnlp.593/.DOI
- [75]Hao Chen, Ziyu Han, Yukun Yan, Qingfu Zhu, Maosong Sun, and Wanxiang Che. From holistic evaluation to structured criteria: Rubrics across the evolving LLM landscape. arXiv preprint arXiv:2606.08625, 2026.arxiv.org/abs/2606.08625
- [76]Kuo Zhou and Lu Zhang. Step-wise formal verification for llm-based mathematical problem solving, 2025. URL https://arxiv.org/abs/2505.20869.
- [77]Yujie Hu, Quan Zou, Shixiang Gu, Tong Xiao, and Jingbo Zhu. Multi-agent debate for LLM judges with adaptive stability detection, 2025. URL https://arxiv.org/abs/2510.12697.
- [78]Jiazheng Zhang, Ziche Fu, Zhiheng Xi, Wenqing Jing, Mingxu Chai, Wei He, Guoqiang Zhang, Chenghao Fan, Chenxin An, Wenxiang Chen, Zhicheng Liu, Haojie Pan, Dingwei Zhu, Tao Gui, Qi Zhang, and Xuanjing Huang. Agentvr-l: Scaling reward modeling with agentic verifier. In Proceedings of the 64rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, USA, July 2026. Association for Computational Linguistics.
- [79]Sheng Hu, Weiyi Lu, Lingbing Zeng, Gan Weng, Weiwei Zhang, Kai Xie, Xiaofeng Mou, and Yi Xu. ARS: Agentic Reward System for Robot Learning, 2026. URL https://arxiv.org/abs/2609.34484.
- [80]Yichao Wu, Gen Li, Bo Lan, Jinhao Duan, Zhijie Wang, and Jianye Hao. Efficient LLM safety evaluation through multi-agent debate, 2025. URL https://arxiv.org/abs/2511.06396.
- [81]Curtis Braun, Subodh Kumar, and Prasad Calyam. Leveraging LLMs as meta-judges: A multi-agent framework for evaluating LLM judgments, 2025. URL https://arxiv.org/abs/2504.17087.
- [82]Yifei Li, Xiaohan Zheng, Wentao Qian, and Liansheng Zhuang. DARE: Dialectical Agentic Reasoning for Structured Knowledge Fact Checking. 2026. URL https://arxiv.org/abs/2609.13808.
- [83]Tao Zhang, Kehui Yao, Luyi Ma, Jiao Chen, Reza Yousefi Maragheh, Kai Zhao, Jianpeng Xu, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. No-human in the loop: Agentic evaluation at scale for recommendation, 2025. URL https://arxiv.org/abs/2511.03051.
- [84]Ziyou Hu, Zhengliang Shi, Minhang Zhu, Haitao Li, Teng Sun, Pengjie Ren, Suzan Verberne, and Zhaochun Ren. Openreward: Learning to reward long-form agentic tasks via reinforcement learning, 2025. URL https://arxiv.org/abs/2510.24636.
- [85]Sher Badshah, Ali Emami, and Hassan Sajjad. Tale: A tool-augmented framework for reference-free evaluation of large language models, 2025. URL https://arxiv.org/abs/2504.07385.
- [86]Roshita Bhonsle, Rishav Dutta, Sneha Vavilapalli, Harsh Seth, Abubakarr Jaye, Yapei Chang, Mukund Rungta, Emmanuel Aboah Boateng, Sadid Hasan, Ehi Nosakhare, et al. Auto-eval judge: Towards a general agentic framework for task completion evaluation, 2025. URL https://arxiv.org/abs/2508.05508.
- [87]Jiaru Zou, Soumya Roy, Vinay Kumar Verma, Ziyi Wang, David Wipf, Pan Lu, Sumit Negi, James Zou, and Jingrui He. TaTT00: Tool-grounded thinking PRM for test-time scaling in tabular reasoning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=zc1ezBrr5m.
- [88]Peng Kuang, Xiangxiang Wang, Wentao Liu, Jian Dong, and Kaidi Xu. Tim-prm: Verifying multimodal reasoning with tool-integrated prm, 2025. URL https://arxiv.org/abs/2511.22998.
- [89]Chaoqun Cui, Jing Huang, Shijing Wang, Liming Zheng, Qingchao Kong, and Zhixiong Zeng. Agentic reward modeling: Verifying gui agent via online proactive interaction, 2026. URL https://arxiv.org/abs/2602.00575.
- [90]Tao Xiong, Xavier Hu, Yurun Chen, Yuhang Liu, Changqiao Wu, Pengzhi Gao, Wei Liu, Jian Luan, and Shengyu Zhang. Gui-pra: Process reward agent for gui tasks, 2025. URL https://arxiv.org/abs/2509.23263.
- [91]Qiushi Sun, Da Yu, Alexander Knop, Amer Sinha, Lingpeng Kong, Ben Kao, Ravi Kumar, and Pritish Kamath. VeriCUA: Verification as Agentic Exploration for Computer-Using Agents, 2026. URL https://arxiv.org/abs/2610.40578.
- [92]Reza Sanayee, Razvan-Gabriel Dumitru, Sunjie Hou, MohammadHossein Rezaei, Haris Riaz, Robert Vacareanu, MingJai Yang, Ying Liu, Mihai Surdeanu, and Daniel Yue Zhang. Evidence-First Agentic Verification, 2026. URL https://arxiv.org/abs/2610.17724.
- [93]Zeyao Ma, Jing Zhang, Xiaokang Zhang, Jiaxi Yang, Zongmeng Zhang, Jiajun Zhang, Yuheng Jing, Lei Zhang, Hao Zheng, Wenting Zhao, Junyang Lin, and Binyuan Hui. Scaling agentic verifier for competitive coding, 2026. URL https://arxiv.org/abs/2602.04254.
- [94]Wayne Chi, I-Hung Hsu, Li Chen, Wei-Lin Chiang, Ion Stoica, Anastasios Nikolaos Angelopoulos, Ameet Talwalkar, Chris Donahue, and Cho-Jui Hsieh. Interaction-as-a-Judge, 2026. URL https://arxiv.org/abs/2610.17895.
- [95]Elad Ben Avraham, Changhao Li, Ron Dorfman, Roy Ganz, Oren Nuriel, Amir Dudai, Aviad Aberdam, Noah Flynn, Elman Mansimov, Adi Kalyanpur, and Ron Litman. Dream: Deep research evaluation with agentic metrics, 2026. URL https://arxiv.org/abs/2602.18940.
- [96]Hang Zhang, Ruheg Wang, Yuelu Ji, Mingu Kwak, Xizhi Wu, Chenyu Li, Li Zhang, Wenqi Shi, Yifan Peng, and Yanshan Wang. Scaling medical reasoning verification via tool-integrated reinforcement learning. arXiv preprint arXiv:2601.20221, 2026.arxiv.org/abs/2601.20221
- [97]Reuben Tan, Baolin Peng, Zhengyuan Yang, Hao Cheng, Oier Mees, Theodore Zhao, Andrea Tupini, Isar Meijier, Qianhui Wu, Yuncong Yang, Lars Liden, Yu Gu, Sheng Zhang, Xiaodong Liu, Lijuan Wang, Marc Pollefeys, Yong Jae Lee, and Jianfeng Gao. Multimodal reinforcement learning with adaptive verifier for ai agents, 2026. URL https://arxiv.org/abs/2512.03438.
- [98]Xiaofeng Tan, Jiaxin Lin, Haobo Yuan, Tiancheng Shen, Zihan Su, Rui Chen, Hongsong Wang, and Ming-Hsuan Yang. Getting Space Right: Agentic Rewards for Image Generation, 2026. URL https://arxiv.org/abs/2610.14852.
- [99]Minki Kang, Jongwon Jeong, and Jaewoong Cho. T1: Tool-integrated verification for test-time compute scaling in small language models. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=tBkLWfmuG I.arxiv.org/abs/2504.04718
- [100]Bingchen Zhao, Jenny Zhang, Chenxi Whitehouse, Minqi Jiang, Michael Shvartsman, Abhishek Charnalia, Despoina Magka, Tatiana Shavrina, Derek Dunfield, Qisin Mac Aodha, and Yoram Bachrach. APRES: An agentic paper revision and evaluation system, 2026. URL https://arxiv.org/abs/2603.03142.
- [101]Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking-LLM-as-a-Judge, 2025. URL https://arxiv.org/abs/2501.18099.
- [102]Mohit Raghavendra, Anisha Gunjal, Bing Liu, and Yun-zhong He. Agentic rubrics as contextual verifiers for swe agents, 2026. URL https://arxiv.org/abs/2601.04171.
- [103]Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model-based agents. ACM Transactions on Information Systems, 43(6):1–47, 2025. doi: 10.1145/3748302.DOI
- [104]Pengfei Dui. Memory for autonomous LLM agents: Mechanisms, evaluation, and emerging frontiers, 2026. URL https://arxiv.org/abs/2603.07670.
- [105]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.
- [106]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. URL https://arxiv.org/abs/2305.20050.
- [107]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
- [108]Mathematical Association of America. American invitational mathematics examination (AIME). https://maa.org/maa-invitational-competitions/, 2026. Accessed 2026-06-10.
- [109]Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James V. Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanneah Hajishirzi. RewardBench: Evaluating reward models for language modeling, 2024. URL https://arxiv.org/ Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-Bench: Benchmarking reward models of language models with subtlety and style, 2024. URL https://arxiv.org/abs/2410.16184.
- [111]Sijun Tan, Sivuan Zhang, Kyle Montgomery, and William Y. Yang. JudgeBench: A benchmark for evaluating LLM-based judges, 2024. URL https://arxiv.org/abs/2410.12784.
- [112]Yuchen Yan, Jin Jiang, Zhenbang Ren, Yiyun Li, Xudong Cai, Yang Liu, Xin Xu, Mengdi Zhang, Jian Shao, Yongliang Shen, Jun Xiao, and Yueting Zhuang. VerifyBench: Benchmarking reference-based reward systems for large language models, 2025. URL https://arxiv.org/abs/2505.15801.
- [113]Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. ProcessBench: Identifying process errors in mathematical reasoning, 2024. URL https://arxiv.org/abs/2412.06559.
- [114]Vijay Viswanathan, Yanchao Sun, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. Checklists are better than reward models for aligning language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=RPRqKqhr6.
- [115]Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Reward reasoning model. arXiv preprint arXiv:2505.14674, 2025. URL https://arxiv.org/abs/2505.14674.
- [116]Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, and Heng Ji. RM-R1: Reward modeling as reasoning. arXiv preprint arXiv:2505.02387, 2025. URL https://arxiv.org/abs/2505.02387.
- [117]Ilgee Hong, Changlong Yu, Liang Qiu, Weixiang Yan, Zhenghao Xu, Haoming Jiang, Qingru Zhang, Qin Lu, Xin Liu, Chao Zhang, and Tuo Zhao. Think-RM: Enabling long-horizon reasoning in generative reward models. arXiv preprint arXiv:2505.16265, 2025. URL https://arxiv.org/abs/2505.16265.
- [118]Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anushree Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. Topical-Chat: Towards knowledge-grounded open-domain conversations. In Proceedings of Interspeech, 2019.arxiv.org/abs/2308.11995
- [119]Shikib Mehri and Maxine Eskanazi. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.64. URL https://aclanthology.org/2020.acl-main.64/.
- [120]Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, 2016.
- [121]Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 889–898, 2018.arxiv.org/abs/1805.04833
- [122]Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, 2020.
- [123]Jiaju Chen, Yuxuan Lu, Shao Zhang, Bingsheng Yao, Yuanzhe Dong, Ying Xu, Yunyao Li, Qianwen Wang, Dakuo Wang, and Yuling Sun. StorySparkQA: Expert-annotated QA pairs with real-world knowledge for children’s story-based learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17351–17370, Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.961. URL https://aclanthology.org/2024.emnlp-main.961/.
- [124]Mahnaz Koupaee, Jake W. Vincent, Saab Mansour, Igor Shalyminov, Han He, Hwanjun Song, Raphael Shu, Jianfeng He, Yi Nian, Amy Wing-mei Wong, Kyu J. Han, and Hang Su. Faithful, unfaithful or ambiguous? multi-agent debate with initial stance for summary evaluation. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 12209–12246, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.609. URL https://aclanthology.org/2025.naacl-long.609/.
- [125]Hongliu Cao, Ilias Drouich, Robin Singh, and Eoin Thomas. Multi-agent llm judge: automatic personalized llm judge design for evaluating natural language generation applications, 2025.arxiv.org/abs/2504.02867
- [126]Taaha Kazi, Ruiliang Lyu, Sizhe Zhou, Dilek Hakkani-Tür, and Gokhan Tur. Large language models as user-agents for evaluating task-oriented-dialogue systems. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 913–920. IEEE, 2024.
- [127]Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang. Large language models are diverse role-players for summarization evaluation. In CCF international conference on natural language processing and Chinese computing, pages 695–707. Springer, 2023.
- [128]Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yaning Jia, Peijun Qing, Lin Shi, Arman Cohan, Yujun Yan, and Soroush Vosoughi. Judging with many minds: Do more perspectives mean less prejudice? on bias amplification and resistance in multi-agent based LLM-as- judge. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 17356–17392, Suzhou, China, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-emnlp.941. URL https://aclanthology.org/2025.findings-emnlp.941/.DOI
- [129]Ruixiang Feng, Zhenwei An, Yuntao Wen, Ran Le, Yiming Jia, Chen Yang, Zongchao Chen, Lisi Chen, Shen Gao, Shuo Shang, Yang Song, and Tao Zhang. Cosineverifier: Tool-augmented answer verification for computation-oriented scientific questions, 2025. URL https://arxiv.org/abs/2512.01224.
- [130]Aditya Pathak, Rachit Gandhi, Vaibhav Uttam, Arnav Ramamoorthy, Pratyush Ghosh, Aaryan Raj Jindal, Shreyash Verma, Aditya Mittal, Aashna Ased, Chirag Khatri, Yashwanth Nakka, Devansh, Jagat Sesh Challa, and Dhruv Kumar. Rubric is all you need: Improving llm-based code evaluation with question-specific rubrics. In Proceedings of the 2025 ACM Conference on International Computing Education Research V.1, ICER ’25, page 181–195, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400713408. doi: 10.1145/3702652.3744220. URL https://doi.org/10.1145/3702652.3744220.
- [131]Arduin Findeis, Floris Weers, Guoli Yin, Ke Ye, Ruoming Pang, and Tom Gunter. Can external validation tools improve annotation quality for LLM-as-a-judge? In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15997–16020, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.779. URL https://aclanthology.org/2025.acl-long.779/.DOI
- [132]Jiaqi Liu, Kai Xiong, Peng Xia, Yiyang Zhou, Haonian Ji, Lu Feng, Siwei Han, Mingyu Ding, and Huaxiu Yao. Agent0-vl: Exploring self-evolving agent for tool-integrated vision-language reasoning, 2025. URL https://arxiv.org/abs/2511.19900.








