betaFeedback

WorksComputer Science

Estimation of Propagated Risk in LLM-based Multi-Agent Systems

Abstract

Large Language Model (LLM) based multi-agent systems introduce a distinctive reliability challenge in which actions taken by upstream agents can affect the correctness of downstream agents. While prior work has investigated error propagation and developed methods for estimating uncertainty inherited from upstream agents, existing approaches do not explicitly model how realized sender-receiver interactions contribute to downstream error risk under different sender correctness states. In this work, we study agent-level error-risk estimation from multi-agent execution traces. We propose Interaction-Effect Propagation (IE-Prop), an interaction-conditioned model that learns separate effects for correct and incorrect upstream agents. For each sender-receiver interaction, the model combines these conditional effects using the sender's estimated error risk, and recursively propagates the resulting risk estimates through the execution graph. We evaluate our approach across three multi-agent architectures and three benchmarks. Across all evaluated architectures, our approach raises the final-node AUROC over the established baseline by approximately 0.05 points on average. Furthermore, tracking risk across multiple agent steps contributes a distinct +0.03 AUROC gain in sequential systems. These results show that realized inter-agent interactions provide predictive information beyond receiver-local evidence, while the value of multi-step risk tracking depends on the system's information-flow structure.

Introduction

LLM-based multi-agent systems decompose complex tasks across interacting agents, making intermediate outputs part of the context for subsequent generations. This interaction structure creates a reliability problem that does not arise in isolated generation: an upstream error can influence downstream predictions, but its effect depends on how

Figure 1. Error dynamics in LLM-based multi-agent systems: propagation of an upstream error (top), correction of an upstream error (middle), and erroneous override of a correct upstream answer (bottom).

the receiving agent uses the information it receives. Empirical analyses of multi-agent failures have documented error transmission through inter-agent communication [3] (Zhang et al. 2025), and controlled studies show that erroneous and corrective peer information can affect downstream behavior differently [31, 14]. Together, these findings suggest that reliability in such systems depends not only on whether an upstream agent is likely to be wrong, but also on the realized interaction through which its output is consumed.

Existing propagation-aware approaches already account for the possibility that upstream errors are transmitted. PropUQ-MAS [24], for example, combines local uncertainty with inherited parent risk and uses an acceptance/critique mechanism to attenuate propagation when erroneous information is rejected or corrected. Its recurrence nevertheless treats inherited risk as a predefined contamination contribution: for fixed local uncertainty and edge acceptance, increasing parent risk cannot lower the receiver’s estimated risk. This leaves open a more general question, whether the association between an observed sender-receiver interaction and receiver error should depend on the sender’s correctness.

We introduce Interaction-Effect Propagation (IE-Prop), post-hoc agent-level error-risk estimation from completed multi-agent execution traces. Such risk estimates can support selective prediction and deferral and targeted re-runs [8, 11]. For post-hoc risk estimation, we distinguish two increasingly rich information sets available in a completed trace. First, receiver-local uncertainty signals such as entropy, verbalized uncertainty, provide a conventional estimate of the receiver’s reliability. Second, a multi-agent trace exposes relational information unavailable in isolated generation: the messages received from upstream agents and the receiver’s response to them. For each such interaction, we refer to the agent providing information as the sender and the agent consuming it as the receiver. Our primary empirical question is whether upstream interaction information provides predictive signal beyond the receiver text.

We address this question with an interaction-conditioned error-risk model. For each realized interaction, the model learns two effects on the receiver’s error score, corresponding to a correct and an incorrect sender. Because sender correctness is unknown at prediction time, the two effects are combined using the sender’s estimated error risk. The resulting interaction adjustment is added to receiver-local evidence, and node risks are computed recursively through the execution graph. This formulation allows the same observed interaction to have different, and potentially opposite, associations with receiver error under the two sender states, while retaining a single risk estimate that can be propagated downstream. The contributions of our work are:

  1. Interaction-conditioned, sender-state-aware error modeling (§ 3). We introduce an agent-level error-risk model (IE-Prop) in which each realized sender-receiver interaction has separate signed effects under correct and incorrect sender states. This allows the downstream association of an interaction to depend on both what occurred between the agents and the sender’s correctness.

  1. Recursive risk propagation and learning (§ 3, § 4). We develop a tractable recursive procedure that combines the two interaction effects using estimated sender risk and propagates the resulting scalar risk through the execution graph. The estimator is trained end to end from node-level correctness labels, allowing a downstream node to incorporate information accumulated through earlier interactions without explicitly maintaining a joint distribution over upstream error states.

  1. Empirical characterization of interaction and propagation effects (§ 7). Through controlled comparisons across three multi-agent architectures, we examine whether realized upstream interactions provide predictive information beyond receiver-local evidence and distinguish the contributions of interaction modeling and recursive risk propagation. We find that interaction information generally improves downstream error-risk discrimination, al-

MethodMASProp.Estimator
P(True) ([17])––Self-eval.
SelfCheckGPT ([26])––Sample consistency
Semantic Entropy ([10])––Semantic entropy
BTProp ((Hout et al., 2025))–✓Belief tree
SAUP ((Zhao et al., 2025))–✓Situation weights
UProp ([7])–✓Information theory
RUPA ([25])–✓Relational graph
MATU ([5])✓–Tensor factor
Score Deviation ([20])✓–Conformal calibration
HalluProp ([22])✓✓Noisy-OR
PropUQ-MAS ([24])✓✓Weighted Noisy-OR
Interaction-Effect Propagation (ours)✓✓Signed logit mix

Table 1. Comparison of LLM error-risk estimation methods by multi-agent system (MAS) support, risk propagation (Prop.), and underlying estimation mechanism.

Figure 2. Overview of Interaction-Effect Propagation. (A) Downstream error risk depends on realized sender–receiver interactions rather than monotone error propagation. (B) Interaction effects conditioned on sender correctness are combined using estimated sender risk and receiver-local evidence. (C) Three evaluated multi-agent topologies: sequential, fully connected, and shallow hierarchy.

though the magnitude varies across architectures and datasets.

Our results indicate downstream error risk is a property of realized inter-agent interactions, while showing that the size and the source of the benefit depend on the graph, the data and the strength of the receiver’s own evidence.

Related Work

LLM confidence and error-risk estimation. Local signals include verbalized confidence ([23, 34] (Xiong et al. 2024)), self-evaluation ([17]), semantic entropy ([21, 10]), sampling consistency ([26]), and trained correctness predictors ([27, 1, 18]); benchmarks such as LM-Polygraph compare them ([9, 35]). BT-Prop estimates claim factuality by propagating beliefs over a tree of logically related statements ([12]). These methods estimate the uncertainty or confidence in LLM response for an isolated generation.

Uncertainty and reliability in agent systems. SAUP propagates uncertainty across steps in a single-agent trajectory using situational weights ((Zhao et al., 2025)), UProp adds a mutual-information term with earlier decisions ([7]), RUPA aggregates over a relation-typed trajectory graph ([25]), and MATU derives a system-level score from re-sampled multi-agent runs ([5]); surveys argue that agent UQ needs formulations beyond single outputs ([19, 29]). These works study uncertainty over sequential trajectories or whole-system executions. Information preservation during inter-agent communication has also been investigated in decentralized settings ([32]).

Error propagation in multi-agent systems. PropUQ-MAS is the closest method to ours ([24]). It treats an upstream error as a contamination event: each agent’s local uncertainty is combined with a self-reported acceptance probability for every incoming message in a noisy-OR product, so transmission is monotone and a parent error can only increase receiver risk. We do not assign a single transmission probability to a parent error. Rather we learn separate effects of the realized sender-receiver interaction under correct and erroneous sender states, which allows both harmful propagation and correction. [22] introduce HalluProp, which estimates agent-level hallucination risk before execution using role–query alignment and topology-aware Noisy-OR propagation. Descriptive studies measure how hallucinations survive handoffs [33, 15], and failure-attribution methods locate the responsible agent after a failure [13] (Zhang et al. 2025).

Graphical-model connection. Our formulation is related to logistic graphical models in which node-local and relational terms combine additively in logit space [28, 2]. We adopt this structure as an inductive bias, allowing each incoming interaction to contribute a separate, correctness-conditioned adjustment to the receiver’s error log-odds. This provides a compositional formulation for propagating risk without explicitly maintaining a joint distribution over agents’ error states (Appendix C).

Interaction-Effect Error-Risk Estimation

Problem setup. We represent an executed multi-agent system as a directed acyclic graph G=(V,E)G = (V, \mathcal{E}), where VV is the set of agents in the execution and E⊆V×V\mathcal{E} \subseteq V \times V is the set of directed edges representing realized information flow between agents. An edge (p,v)∈E(p,v) \in\mathcal{E} indicates that agent vv receives the visible output of agent pp. Each agent vv has a role ρv\rho_v, a visible output mvm_v and, after execution, an error indicator Ev∈{0,1}E_v \in\{0,1\}, and its error risk is rv=P(Ev=1)r_v = P(E_v = 1). EvE_v is observed for training and evaluation traces but is unknown at inference. Let qq be the task. Each agent has a local risk score

bv=gϕ(wv),(1)b_v = g_{\phi}(w_v), \tag*{(1)}

where wvw_v holds agent-local risk signal and gϕg_{\phi} is a small network. For an interaction p→vp \to v, we refer to pp as the sender (or parent) and vv as the receiver, and their visible outputs are mpm_p and mvm_v, respectively. Each realized interaction is embedded by a frozen text encoder

xpv=Enc⁡(q,ρp,mp,ρv,mv)(2)x_{pv} = \operatorname{Enc}(q, \rho_p, m_p, \rho_v, m_v) \tag*{(2)}

The embedding xpvx_{pv} represents the realized exchange rather than only the parent message. It can therefore capture textual evidence of how the receiver used the upstream information - for example, whether it followed, rejected, revised, or amplified it. Because mvm_v is required to construct xpvx_{pv}, the estimator is necessarily post-hoc and can only score an interaction after receiver has produced its output.

State-conditioned interaction model. The association between a realized sender–receiver interaction and receiver error may differ depending on whether the sender is correct or incorrect. To capture this distinction, an edge head (fθ)(f_{\theta}) maps (xpv)(x_{pv}) to two scalars

fθ(xpv)=(δpv(0),δpv(1)).(3)f_{\theta}(x_{pv}) = \left(\delta_{pv}^{(0)}, \delta_{pv}^{(1)}\right). \tag*{(3)}

δpv(0)\delta_{pv}^{(0)} is the effect of the realized interaction on the receiver’s error log-odds if the sender is correct (Ep=0)(E_p = 0), and δpv(1)\delta_{pv}^{(1)} its effect if the sender is wrong (Ep=1)(E_p = 1). Positive values raise the receiver’s error log-odds and negative values lower them. The pair can therefore represent both propagation and correction: δpv(1)>0\delta_{pv}^{(1)} > 0 describes a receiver that adopts an erroneous parent message, and δpv(1)<0\delta_{pv}^{(1)} < 0 a receiver that detects and corrects it.

We combine receiver-local and interaction-specific evidence additively in log-odds space. The local score bvb_v provides the receiver’s baseline error log-odds, while each incoming interaction contributes an adjustment selected by the corresponding sender state. For an error configuration e={ep:p∈Pa⁡(v)}e = \{e_p : p \in\operatorname{Pa}(v)\}, where ep∈{0,1}e_p \in\{0,1\} indicates whether sender pp is correct (0) or incorrect (1), we define

ηv(e)=bv+∑p∈Pa⁡(v)δpv(ep).(4)\eta_v(e) = b_v + \sum_{p \in\operatorname{Pa}(v)} \delta_{pv}^{(e_p)}. \tag*{(4)}

and model the receiver’s conditional error probability as

P(Ev=1∣EPa⁡(v)=e)=σ(ηv(e)).(5)P(E_v = 1 \mid E_{\operatorname{Pa}(v)} = e) = \sigma\left(\eta_v(e)\right). \tag*{(5)}

This additive logistic form is a modeling choice: it provides a compositional inductive bias in which incoming interactions contribute separately to the receiver’s log-odds. Appendix § 12 gives the corresponding graphical-model interpretation.

Inference with uncertain sender states. Eq. (5) specifies the receiver risk if the correctness state of every sender were known. At inference these states are unknown, so for each incoming edge we use the estimated sender error probability r^p\hat{r}_p as a soft belief over the two sender states and average the corresponding interaction contributions in logit space:

δˉpv=(1−r^p)δpv(0)+r^pδpv(1).(6)\bar{\delta}_{pv} = (1-\hat{r}_p)\delta_{pv}^{(0)} + \hat{r}_p\delta_{pv}^{(1)}. \tag*{(6)}

The receiver’s predicted risk is obtained by adding these expected interaction contributions to its local score,

r^v=σ(bv+∑p∈Pa(v)δˉpv).(7)\hat{r}_v = \sigma\left(b_v + \sum_{p\in\mathrm{Pa}(v)} \bar{\delta}_{pv}\right). \tag*{(7)}

This averaging can be expressed in terms of expectations. Specifically, Eq. (6) computes δˉpv=EEp[δpv(Ep)]\bar{\delta}_{pv} = \mathbb{E}_{E_p}[\delta_{pv}^{(E_p)}], where the expectation is taken over the sender’s error state using its estimated error probability r^p\hat{r}_p. By linearity of expectation, Eq. (7) then applies the sigmoid to the expected receiver logit, yielding r^v=σ(EEPa(v)[ηv(EPa(v))])\hat{r}_v = \sigma(\mathbb{E}_{E_{\mathrm{Pa}(v)}}[\eta_v(E_{\mathrm{Pa}(v)})]).

Risk estimates are computed recursively in topological order over the execution graph. Roots receive r^v=σ(bv)\hat{r}_v = \sigma(b_v), while each non-root node uses its parents’ recursively estimated risks in Eqs. (6)–(7). So in a sequential graph a node’s risk depends on interactions further upstream through its parent’s risk. Appendix § 11.§ 1 shows that this update also admits a reverse-KL barycenter interpretation.

Approximation to marginal inference. Eq. (7) is a compositional inductive bias, not an exact identity: with Zv=ηv(EPa(v))Z_v = \eta_v(E_{\mathrm{Pa}(v)}), exact marginalization of Eq. (5) over the unknown parent states would compute E[σ(Zv)]\mathbb{E}[\sigma(Z_v)], whereas Eq. (7) computes σ(E[Zv])\sigma(\mathbb{E}[Z_v]) from one scalar per parent, at cost O(k)O(k) instead of 2k2^k for kk parents, and is exact when the two effects of every uncertain edge coincide. Appendix § 10 derives a variance-based bound on the local discrepancy between exact marginalization and Eq. (7). Appendix § 12 describes the induced probabilistic model. Either way, r^p\hat{r}_p enters only as the weight that mixes the two state-conditional effects, so how it is used is fixed by the model rather than learned.

Training

We train models fθ(xpv)f_\theta(x_{pv}) and gϕ(wv)g_\phi(w_v), in an end-to-end setting, on execution traces generated from training questions. Each trace contains the observed interaction within agents. Predictions are computed recursively using Eqs. (6)–(7), and the model parameters ϕ\phi and θ\theta are optimized jointly by minimizing binary cross-entropy between the predicted risk r^v\hat{r}_v and the correctness label EvE_v, summed over labeled nodes:

L=∑v∈VBCE⁡(r^v,Ev).(8)\mathcal{L} = \sum_{v\in V} \operatorname{BCE}(\hat{r}_v,E_v). \tag*{(8)}

Gradients therefore propagate through the recursive risk computation, allowing both the receiver-local model gϕg_\phi and the interaction model fθf_\theta to be learned jointly. Evaluation is performed on execution traces from held-out questions (§ 6). Algorithm 1 summarizes the complete procedure for recursive risk estimation, end-to-end training, and K-fold prediction.

Evaluation Design

We evaluate four estimators under common training and evaluation protocol discussed in details in § 6.

Receiver-Only. The local estimator σ(bv)\sigma(b_v) (Eq. (1)) predicts the receiver’s error risk using three local uncertainty signals: verbalized uncertainty (verb.), Maximum Sequence Probability (MSP), prediction entropy.

Receiver-Text. Building on the Receiver-Only model, this baseline adds context about the receiver’s specific task, role, and final text output, but strictly ignores any incoming messages from the sender. We use this as our main benchmark to determine whether analyzing the actual upstream conversation gives us better error predictions than just looking at the receiver in isolation.

Free learner. An unconstrained edge model that directly estimates the interaction adjustment δˉpv\bar{\delta}_{pv} from the interaction representation xpvx_{pv}. Unlike our proposed model, it does not decompose the adjustment into sender-state-conditioned effects or prescribe how the parent risk is used.

IE-Prop. Our model from Eq. (7), which uses r^p\hat{r}_p to mix the interaction effects for correct and incorrect sender states.

Recursion Control. To isolate the contribution of recursively propagated risk, we train variants of IE-Prop in which the propagated parent risk r^p\hat{r}_p is replaced by the parent’s local risk σ(bp)\sigma(b_p). This comparison isolates whether recursively accumulated upstream information improves downstream risk estimation beyond the immediate parent’s local evidence and observed sender–receiver interaction.

As an external reference we also report PropUQ-MAS, computed from each agent’s verbalized uncertainty, MSP, and entropy with our implementation of its propagation rule. We report AUROC for the final node. The description of the considered local risk estimates and metric is provided in Appendix § 16 and Appendix § 17 respectively.

These comparisons address whether upstream interaction information adds predictive signal beyond what is available from the receiver’s text alone, and how our interaction-conditioned approach performs against existing propagation-based uncertainty methods.

Experimental Setup

Systems. We evaluate three distinct four-agent topologies instantiated with Qwen3 models (Yang et al., 2025) (0.6B, 1.7B, and 4B variants): a sequential pipeline where each agent observes only its immediate predecessor; a fully connected architecture where agents attend to all preceding outputs; and a specialist hierarchy where a final summarizer aggregates independent parallel responses. Full architectural details, including specific agent roles and prompt-level mitigations for degenerate copying behavior in the fully connected system, are deferred to Appendix D.

Tasks and labels. We evaluate across three question-answering benchmarks: GSM8K (Cobbe et al., [6]), MedQA (Jin et al., [16]), and the text-only subset of WebQA (Chang et al., [4]). Treating each dataset, graph topology, and model-size combination as an independent setting yields 27 distinct experimental evaluations for each methods. To train and evaluate the risk estimators, each agent is assigned a binary correctness label by extracting the final answer from its visible output and comparing it against the ground truth. Full details regarding train-test data splits, automated answer extraction protocols (e.g., deterministic vs. LLM-based read-outs), and task-specific matching criteria are provided in Appendix D.

In-Domain. Estimators are trained on the execution traces from a dataset’s training split and evaluated on its corresponding held-out test split. Detailed in-domain results are provided in § 7.

Transfer. To evaluate whether the learned risk signals generalize to unseen distributions, we employ a leave-one-dataset-out transfer protocol. For each architecture and model size, estimators are trained on execution traces from two datasets and evaluated on the held-out third dataset. Detailed cross-dataset transfer results are in Appendix E.

Results

We evaluate whether realized inter-agent exchanges improve downstream error-risk estimation, whether sender-state conditioning contributes beyond an unconstrained interaction model. We then examine downstream error discrimination with correct upstream context and isolate the contribution of recursive risk propagation. Table 2 reports seed-averaged final-node AUROC for Qwen3-0.6B, 1.7B, and 4B across datasets.

EstimatorGSM8KMedQA
Verb.MSPEntropyVerb.MSPEntropy
0.6B1.7B4B0.6B1.7B4B0.6B1.7B4B0.6B1.7B4B0.6B1.7B4B0.6B1.7B4B
Sequential
Receiver-Only0.6220.5310.5180.5410.5870.6780.5700.5880.7200.4950.5160.5310.5080.4860.6860.5120.4860.688
Receiver-Text0.6680.6200.6740.6220.6030.7100.6480.6220.7310.5260.5680.6100.5360.5650.7050.5240.5610.706
Free Learner0.6870.6820.6710.6330.5840.6880.6570.5600.6910.4980.5970.6570.5110.5720.6960.5010.5710.704
PropUQ-MAS0.6200.6410.5540.3820.4940.7560.3950.5140.7830.5000.5480.6800.4730.4820.7370.4710.4950.750
IE-Prop (Ours)0.7110.7080.6840.6600.6220.7160.6710.6100.7230.5210.6130.6900.5330.5840.7280.5180.5840.743
Fully connected
Receiver-Only0.6160.5930.5070.5910.4770.5340.6030.4880.4570.4930.5090.5910.5500.4570.5260.5620.4590.557
Receiver-Text0.6960.7160.6520.6510.6820.6650.6550.6700.6760.4880.5710.5940.5290.5600.5520.5340.5540.574
Free Learner0.7180.7800.7190.7040.7450.7650.7230.7600.7820.5120.5570.6650.4850.5770.6690.4950.5830.707
PropUQ-MAS0.6670.7540.6130.5360.6360.7840.5470.6770.8190.5000.5470.6990.5670.5440.6950.5990.5640.742
IE-Prop (Ours)0.7390.7990.7330.7270.7630.7990.7430.7790.8120.5290.5730.6920.5090.5950.6900.5190.6020.727
Hierarchy
Receiver-Only0.5600.7120.5640.6360.5680.7490.6520.5270.7830.4980.5910.6720.5100.5990.6820.5110.6000.725
Receiver-Text0.7090.7760.7620.7310.6640.7740.7400.6590.7850.5020.6280.6940.4780.6090.6830.4910.6150.715
Free Learner0.7230.7800.7710.7360.7790.8550.7520.8150.8720.4800.6260.7000.4710.6820.7520.5010.6770.769
PropUQ-MAS0.6710.7100.5630.6390.7250.8650.6530.7480.8910.4710.5740.7040.5140.5870.7500.4840.5750.780
IE-Prop (Ours)0.7480.8020.7840.7590.7940.8710.7690.8350.8860.5010.6450.7200.4900.7000.7720.5230.6960.788

Table 2. Final-node AUROC on GSM8K and MedQA by base-model size (Qwen3 0.6B, 1.7B and 4B): in-domain settings, seed mean per model size. Columns are grouped by dataset, then uncertainty signal, then model size. Bold: best value in each column within each topology. Underline: second best value in each column within each topology.

Interaction information beyond receiver text

The receiver’s visible output provides useful evidence for error-risk estimation, but does not exhaust the predictive information available in the execution trace. In Table 2, IE-Prop exceeds Receiver-Text in 46 of the 54 topology–dataset–uncertainty–model-size combinations. Averaged over the datasets and uncertainty signals, the gains for the 0.6B, 1.7B, and 4B models are respectively +0.015, +0.030, and +0.025 in sequential systems; +0.036, +0.060, and +0.123 in fully connected systems; and +0.023, +0.087, and +0.068 in the hierarchy. Interaction information therefore improves mean discrimination at each evaluated size within all three topologies, although magnitude of the gain depends on setting.

Figure 3 shows that IE-Prop also improves over Receiver-Text on WebQA in all 12 evaluated settings, achieving the highest AUROC in nine. It leads in most hierarchical settings and at the two smaller model sizes in sequential systems. However, PropUQ-MAS performs best at 4B in sequential systems for both MSP and entropy, indicating that the relative advantage depends on architecture and model size.

Figure 3. Final-node AUROC on WebQA for sequential and hierarchical systems using MSP and entropy across Qwen3 model sizes, averaged over seeds.

State-conditioned effects

We next compare IE with the Free Learner to examine whether the structured sender-state-conditioned update improves on an unconstrained interaction adjustment. In Table 2, IE achieves higher seed-averaged AUROC in all 54 displayed combinations, covering every model size, topology, dataset, and uncertainty signal. Averaged over datasets, uncertainty signals, and model sizes, the gains are approximately +0.026 in the sequential system, +0.021 in the fully connected system, and +0.019 in the hierarchy.

The consistency of this comparison across model sizes supports the state-conditioned formulation as a useful inductive bias for the evaluated GSM8K and MedQA settings. This predictive advantage does not, by itself, establish that the two learned heads identify distinct error-propagation and correction mechanisms; it shows that the structured formulation improves error-risk discrimination relative to the Free Learner. Additional experiments with oracle-supervised sender-state conditioning are presented in Appendix § 15.

Downstream risk discrimination with correct upstream context

A correct upstream response does not guarantee that the receiver will remain correct. Figure 4 examines sequential MedQA interactions restricted to cases where the sender’s answer is correct. Within this subset, IE assigns higher predicted risk to incorrect receivers than to correct receivers at each evaluated model size.

Figure 4. Predicted receiver risk on sequential MedQA traces with correct upstream answers. Top: Risk distributions by receiver correctness. Bottom: Median risk and AUROC across Qwen3 model sizes.

Discrimination improves across the three sizes in this diagnostic, with AUROC values of 0.68, 0.72, and 0.88 for the 0.6B, 1.7B, and 4B models, respectively. These results show that the estimator can distinguish downstream errors even when the upstream answer is correct. They do not establish independence from estimated sender risk, since the model still uses that estimate when combining its interaction effects. Rather, they demonstrate that its predictive utility extends to interactions in which downstream failure cannot be attributed simply to an incorrect upstream answer.

Contribution of recursive risk propagation

Finally, we isolate the contribution of recursive propagation by comparing IE with a matched variant that replaces recursively estimated parent risk with local parent risk. In the sequential ablation shown in Figure 5, recursion improves final-node AUROC in all nine displayed dataset–model-size combinations. The overall mean increases from 0.631 to 0.655 (+0.024).

Heatmap of final-node AUROC gains from recursive risk propagation across datasets and model sizes

Figure 5. Final-node AUROC gains from recursive risk propagation in the sequential system using Qwen3 models. Cells show gains and corresponding scores (without → with recursion), averaged over three seeds.

Averaged over datasets, the gains are +0.024, +0.025, and +0.024 for the 0.6B, 1.7B, and 4B models, respectively. Thus, the average benefit is similar across model sizes, although individual improvements vary. The largest gain occurs on MedQA with the 4B model (+0.040), while gains averaged over model sizes are +0.026 on GSM8K, +0.026 on MedQA, and +0.019 on WebQA. These results indicate that recursively accumulated parent risk provides predictive information beyond local parent uncertainty and immediate sender–receiver interaction in the evaluated sequential systems.

Discussion

Our results suggest that downstream error risk depends not only on upstream correctness but also on how receivers respond to upstream information. As shown in § 7.3, IE-Prop distinguishes downstream errors even when upstream answers are correct, illustrating why sender correctness alone is insufficient to characterize downstream risk. Moreover, its improvement over the Free Learner (§ 7.2) supports sender-state-conditioned interaction modeling as a useful predictive inductive bias. The benefit of recursive propagation depends on the system’s information-flow structure. In sequential systems, earlier interactions reach the final agent only through intermediate agents, making recursively accumulated risk informative beyond the immediate interaction (§ 7.4). In contrast, in shallow hierarchies, the final agent receives information directly from independent specialists, allowing interaction modeling to improve risk discrimination without multi-hop recursive accumulation. These findings suggest interaction modeling and recursive risk propagation provide complementary predictive benefits, whose relative importance depends on system topology.

Conclusion

In this work, we investigated post-hoc agent-level error-risk estimation using execution traces from LLM-based multi-agent systems. We introduced IE-Prop, an approach that models the contribution of realized sender–receiver interactions to downstream error risk by learning separate effects conditioned on whether the sender is correct or incorrect. By combining these effects using estimated sender risk and recursively propagating the resulting estimates through the execution graph, our method captures interaction-dependent error dynamics beyond monotone propagation assumptions. Across three multi-agent architectures and three question-answering benchmarks, our approach generally improves downstream error-risk discrimination over receiver-local and existing propagation-based baselines. Our analyses further demonstrate that interaction information remains predictive even when upstream answers are correct, while recursive propagation provides additional benefits in sequential systems. Together, these findings underscore the value of modeling realized inter-agent interactions for downstream error-risk estimation.

Limitations

Our evaluation focuses on three four-agent architectures, three question-answering benchmarks, and one model family, leaving generalization to larger and more diverse multi-agent systems for future work. Differences in agent roles and prompting protocols also limit direct comparisons across architectures. Although interaction modeling generally improves error-risk discrimination, the gains vary across settings, and the distinct contributions of sender-state conditioning and recursive propagation are not fully established in every topology. Furthermore, our estimator operates post-hoc and relies partly on automated correctness labels. Finally, the additive interaction model approximates marginal inference, and its learned effects represent predictive associations rather than causal relationships. Future work could explore broader agentic settings, alternative aggregation mechanisms, and more direct supervision of sender-state effects.

Ethical Considerations

This work investigates error-risk estimation in LLM-based multi-agent systems, with the goal of improving their reliability. However, inaccurate risk estimates may create a false sense of confidence in incorrect outputs, particularly in high-stakes domains such as medical question answering. Our method is evaluated on established benchmarks and does not guarantee correctness or safety in real-world deployments. Additionally, the learned risk estimates may not generalize reliably to unseen domains, model families, or multi-agent architectures. Finally, the use of multiple LLM agents and additional risk-estimation models incurs computational costs and associated environmental impacts.

Computational Resources

All experiments were conducted using a computing infrastructure equipped with four NVIDIA A100 GPUs. These GPUs supported the complete experimental workload, including LLM hosting, multi-agent inference, execution trace generation, and training of the risk-estimation networks.

Approximation Bound

Let Zv=bv+∑pδpv(Ep)Z_v = b_v + \sum_p \delta_{pv}^{(E_p)}, where the parent states EPa(v)E_{\mathrm{Pa}(v)} follow some distribution π\pi, and let μ=Eπ[Zv]\mu= \mathbb{E}_{\pi}[Z_v]. The exact marginal under the conditional model of Eq. (5) is Eπ[σ(Zv)]\mathbb{E}_{\pi}[\sigma(Z_v)], and Eq. (7) computes σ(μ)\sigma(\mu) when given the exact parent marginals.

Bound. Taylor’s theorem with the Lagrange remainder gives σ(Z)=σ(μ)+σ′(μ)(Z−μ)+12σ′′(ξ)(Z−μ)2\sigma(Z) = \sigma(\mu) + \sigma'(\mu)(Z-\mu) + \frac{1}{2}\sigma''(\xi)(Z-\mu)^2 for some ξ\xi between μ\mu and ZZ. Taking expectations, the linear term vanishes, so ∣E[σ(Z)]−σ(μ)∣≤12sup⁡z∣σ′′(z)∣Var⁡(Z)|\mathbb{E}[\sigma(Z)]-\sigma(\mu)| \le\frac{1}{2}\sup_z|\sigma''(z)|\operatorname{Var}(Z). With σ′′=σ(1−σ)(1−2σ)\sigma'' = \sigma(1-\sigma)(1-2\sigma), the supremum is attained at σ=12±123\sigma= \frac{1}{2} \pm\frac{1}{2\sqrt{3}} and equals 1/(63)1/(6\sqrt{3}), so

∣E[σ(Zv)]−σ(E[Zv])∣≤1123Var⁡(Zv).(9)|\mathbb{E}[\sigma(Z_v)]-\sigma(\mathbb{E}[Z_v])| \le\frac{1}{12\sqrt{3}}\operatorname{Var}(Z_v). \tag*{(9)}

Assumptions and interpretation. The bound uses only the first two moments of ZvZ_v, so it holds for any parent-state distribution, including correlated ones. Writing Δpv=δpv(1)−δpv(0)\Delta_{pv} = \delta_{pv}^{(1)}-\delta_{pv}^{(0)}, we have Zv=bv+∑pδpv(0)+∑pEpΔpvZ_v = b_v + \sum_p \delta_{pv}^{(0)} + \sum_p E_p\Delta_{pv} and

Var⁡(Zv)=∑pΔpv2rp(1−rp)+2∑p<qΔpvΔqvCov⁡(Ep,Eq).\operatorname{Var}(Z_v) = \sum_p \Delta_{pv}^{2}r_p(1-r_p) + 2\sum_{p<q}\Delta_{pv}\Delta_{qv}\operatorname{Cov}(E_p,E_q).

The error therefore vanishes when the two effects coincide, and grows with strong, uncertain and positively co-varying parent effects; correlations do not enter the update itself but change the variance that controls its error. Exactness holds whenever Var⁡(Zv)=0\operatorname{Var}(Z_v)=0: when the parent states are known, when δpv(0)=δpv(1)\delta_{pv}^{(0)}=\delta_{pv}^{(1)} on every uncertain edge, or when every parent configuration with positive probability gives the same logit. The bound is local: it compares the update and exact marginalization under the same parent distribution and does not account for errors in propagated parent marginals.

Single parent. For k=1k=1, as in our sequential system, Var⁡(Zv)=Δpv2rp(1−rp)≤Δpv2/4\operatorname{Var}(Z_v)=\Delta_{pv}^{2}r_p(1-r_p)\le\Delta_{pv}^{2}/4, so the error is at most Δpv2/(483)\Delta_{pv}^{2}/(48\sqrt{3}) and is largest for parents of maximal uncertainty. The final agent of the fully connected system has k=3k=3 parents that share the first agent as an ancestor, so the covariance term applies there.

Natural-Parameter Interpretation and Training Analysis

Reverse-KL barycenter

Each parent configuration ee induces a receiver distribution pe=Bernoulli⁡(σ(ηe))p_e=\operatorname{Bernoulli}(\sigma(\eta_e)) with ηe=bv+∑pδpv(ep)\eta_e=b_v+\sum_p\delta_{pv}^{(e_p)}. Consider summarizing the collection {pe}\{p_e\} by the single Bernoulli q=Bernoulli⁡(r)q=\operatorname{Bernoulli}(r) that minimizes Ee∼π[DKL(q∥pe)]\mathbb{E}_{e\sim\pi}[D_{\mathrm{KL}}(q\mathbin\Vert p_e)]. Since ∂rDKL(q∥pe)=logit⁡(r)−ηe\partial_rD_{\mathrm{KL}}(q\mathbin\Vert p_e)=\operatorname{logit}(r)-\eta_e and the objective is strictly convex, the minimizer satisfies logit⁡(r)=Eπ[ηe]\operatorname{logit}(r)=\mathbb{E}_{\pi}[\eta_e], i.e. r=σ(E[Zv])r=\sigma(\mathbb{E}[Z_v]). With the additive natural parameter, Eπ[ηe]=bv+∑p[(1−rp)δpv(0)+rpδpv(1)]\mathbb{E}_{\pi}[\eta_e]=b_v+\sum_p[(1-r_p)\delta_{pv}^{(0)}+r_p\delta_{pv}^{(1)}], which depends on π\pi only through the individual parent marginals rpr_p, for any dependence between parents. This is an interpretation of the scalar composition rule, not a justification that it yields the exact risk, and not a claim that it is empirically preferable to other operators. The exact mixture over a binary variable is itself Bernoulli with mean Eπ[σ(ηe)]\mathbb{E}_{\pi}[\sigma(\eta_e)]; the update instead aggregates the state-conditioned distributions in reverse-KL (natural-parameter) geometry. Eq. (7) further replaces the exact parent marginals by propagated ones, so its error combines the local term of Eq. (9) with errors accumulated upstream.

Induced Probabilistic Model

Conditioned on a completed trace XX, the node-wise conditionals of Eq. (5) can be combined into the DAG factorization P(E∣X)=∏vP(Ev∣EPa(v),X)P(E\mid X)=\prod_v P(E_v\mid E_{\mathrm{Pa}(v)},X), a Bayesian network over error indicators whose structure is the communication graph. This is a modeling assumption about dependencies among correctness variables given the trace, not a claim about the causal structure of failures, and it does not model how messages were generated. The deployed estimator does not maintain this joint model; it propagates only the scalar risks of Eq. (7).

Relation to sigmoid belief networks and log-linear models. Each conditional is a logistic regression on parent-state indicators, i.e. a sigmoid belief network [28] whose weights Δpv\Delta_{pv} and intercept bv+∑pδpv(0)b_v+\sum_p\delta_{pv}^{(0)} are computed from the trace. Equivalently, the parent-state indicators enter a log-linear model of the receiver’s error with first-order interaction terms only [2]. Noisy-OR [30], the contamination rule, replaces the logistic link by a product of per-parent failure terms, so a wrong parent can only increase the receiver’s risk.

Figure 6. The fully connected system. Every agent reads the visible outputs of all earlier agents (six edges). The Alternative-Method agent and the Checker first write a private draft without the earlier outputs; the draft is never passed on.

Exact marginalization and parent dependence. The exact marginal of a receiver is ∑eπv(e) σ(ηv(e))\sum_e \pi_v(e)\,\sigma(\eta_v(e)), where πv\pi_v is the joint distribution of its parents’ states. Parents that share ancestors are dependent under the induced network even when the conditionals are fixed, so πv\pi_v is not a product of marginals in general. Exact inference must sum over the joint states of the latent ancestors that a receiver depends on. Variable elimination reduces this cost to exponential in the induced width of the elimination order, but a kk-parent logistic receiver still requires the 2k2^k-entry joint of its parents, and overlapping ancestry forces joint tables over shared ancestors. In the controlled sequential system every receiver has one parent, and in the fully connected system the final agent’s three dependent parents have only eight joint states, so exact marginalization is cheap in both; the update matters for wider graphs.

Controlled QA: Details and Full Results

Labels. Each agent’s label is derived from the answer stated in its visible output, after hidden reasoning and uncertainty tags are removed, and the same procedure labels training and test traces; training uses every node’s label as a target, not only the final node’s. On GSM8K the answer is the last boxed number, or otherwise the last number in the text, and it is compared with the gold number. On MedQA and WebQA the answer is extracted by a read-out with the same Qwen3 model that generated the trace: given the question, the agent’s role and its visible response, but not the gold answer, the model returns a JSON object naming the option letter (MedQA) or the complete-sentence answer (WebQA) that the response supports, or UNKNOWN, and it is instructed not to solve the question itself. Decoding is greedy with thinking disabled and a short output budget, and an invalid JSON response is retried once. Correctness is decided in code: exact option match for MedQA, and full recall of normalized gold-answer tokens against any accepted answer for WebQA. A node whose response supports no answer is unlabeled, and training traces with any unlabeled node are dropped. In the hierarchy, the Summarizer’s label is the system evaluator’s verdict; in the fully connected system every agent is labeled as in the sequential system. Test items whose final node is unlabeled are dropped, and oracle evaluations additionally drop items with an unlabeled parent; oracle predictions were not computed for the sequential MedQA 4B setting.

Hyperparameters are shared by every cell: hidden width 256, one hidden layer, dropout 0.1, AdamW with learning rate 10−310^{-3} and weight decay 10−410^{-4}, cosine schedule, batch 256, at most 30 epochs with patience 5, and five question-level folds; each fold model is selected by validation final-node AUROC and the five fold predictions are averaged at test time. The oracle numbers should not be read as attainable targets, since parent and final-node labels largely coincide.

Cross-dataset transfer

Figure 7 evaluates leave-one-dataset-out transfer with entropy-based uncertainty: learned estimators are trained on traces from two datasets and evaluated on the held-out third dataset. It reports IE’s final-node AUROC and its difference from Receiver-Only and the training-free PropUQ-MAS. The held-out IE AUROCs are 0.608, 0.561, and 0.575 in the sequential system, 0.605, 0.622, and 0.546 in the fully connected system; and 0.755, 0.624, and 0.556 in the hierarchy, for GSM8K, MedQA, and WebQA, respectively.

Figure 7. Cross-dataset transfer performance across three multi-agent topologies, showing final-node AUROC and differences relative to Receiver-Only and PropUQ-MAS baselines.

IE improves over Receiver-Only most clearly on held-out MedQA across all three graphs. Its advantage over PropUQ-MAS is less consistent: the paired intervals overlap zero in seven of the nine graph–held-out-dataset comparisons, while sequential GSM8K favors IE and fully connected GSM8K favors PropUQ-MAS. Transfer therefore offers evidence that learned interaction information can remain useful outside its training datasets, but not that IE uniformly dominates a propagation-based uncertainty reference. Because Figure 7 does not include Receiver-Text or matched recursion controls, it does not establish that the transfer improvements arise from state-conditioned recursion or exceed a receiver-text predictor.

Oracle-Supervised Sender-State Conditioning

The original interaction-effect (IE) objective supervises the sender-state heads δpv(0)\delta_{pv}^{(0)} and δpv(1)\delta_{pv}^{(1)} only through their risk-weighted mixture. Consequently, good predictions need not imply that the heads are aligned with sender correctness. To test this, we add an oracle-conditioned training loss that selects the head corresponding to the observed sender label:

E^ivoracle=σ(biv+∑p∈Pa⁡(v)δipv(Eip)),\widehat{E}_{iv}^{\mathrm{oracle}}=\sigma\left(b_{iv}+\sum_{p\in\operatorname{Pa}(v)}\delta_{ipv}^{(E_{ip})}\right),
Loracle=14N∑i=1N∑v∈VieligibleBCE⁡(E^ivoracle,Eiv),\mathcal{L}_{\mathrm{oracle}}=\frac{1}{4N}\sum_{i=1}^{N}\sum_{v\in\mathcal{V}_{i}^{\mathrm{eligible}}}\operatorname{BCE}\left(\widehat{E}_{iv}^{\mathrm{oracle}},E_{iv}\right),
Ltotal=Lpred+λLoracle.(10)\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{pred}}+\lambda\mathcal{L}_{\mathrm{oracle}}. \tag*{(10)}

where Vieligible\mathcal{V}_{i}^{\mathrm{eligible}} contains non-root nodes with known receiver and parent labels. The original Lpred\mathcal{L}_{\mathrm{pred}} is unchanged and uses the same 1/(4N)1/(4N) normalization. At test time, inference uses the original soft recursive mixture without sender labels.

We evaluate all three architectures on GSM8K, MedQA, and WebQA, across three Qwen3 sizes and three uncertainty signals (three seeds; five question-level folds per seed). We select λ=0.1\lambda=0.1 from {0,0.1,0.3,1,3}\{0, 0.1, 0.3, 1, 3\} by mean out-of-fold final-node AUROC on training data. Relative to the original IE model, the mean change in deployed AUROC is +0.004+0.004 (sequential), approximately 00 (fully connected), and −0.001-0.001 (hierarchy). Thus, oracle supervision leaves predictive performance largely unchanged.

To assess head alignment, we compute the oracle-state swap gap: Gswap=AUROC⁡(Ep)−AUROC⁡(1−Ep)G_{\mathrm{swap}}=\operatorname{AUROC}(E_p)-\operatorname{AUROC}(1-E_p), where each receiver uses either the observed or flipped sender labels to select its incoming heads. With λ=0\lambda=0, the mean gaps are +0.271+0.271, −0.050-0.050, and −0.283-0.283 for sequential, fully connected, and hierarchical systems, respectively; at λ=0.1\lambda=0.1, they increase to +0.570+0.570, +0.740+0.740, and +0.679+0.679. These oracle diagnostics use cross-fitted training traces that also served in fold-level validation; independent test traces corroborate the trend in the sequential system. Swapping estimated sender risks in the deployed soft rule likewise produces a larger AUROC gap after oracle supervision. These results support improved sender-state alignment, but do not identify causal error-transmission or correction effects: strong sender–receiver correctness correlations can also drive large swap gaps.

Local Uncertainty Signals

We consider three agent-local uncertainty signals: verbalized uncertainty, Maximum Sequence Probability (MSP), and prediction entropy. Each signal is evaluated independently as an input to the local risk estimator gϕg_{\phi} in Eq. (1). Let hvh_v denote the generation context of agent vv, and let yv=(yv,1,…,yv,Tv)y_v=(y_{v,1},\ldots,y_{v,T_v}) denote the generated token sequence used for uncertainty estimation.

Verbalized uncertainty. Verbalized uncertainty is obtained from the model’s self-reported confidence in its generated answer [23, 34] (Xiong et al. 2024). Let cv∈[0,1]c_v\in[0,1] denote the confidence expressed by agent vv, where larger values indicate greater confidence. We convert this to an uncertainty score as

uvverb=1−cv.(11)u_v^{\mathrm{verb}} = 1 - c_v. \tag*{(11)}

Thus, larger values indicate greater self-reported uncertainty.

Maximum Sequence Probability (MSP). MSP uses the probability assigned by the language model to the generated token sequence. Let pLM(yv,t∣hv,yv,<t)p_{\mathrm{LM}}(y_{v,t} \mid h_v, y_{v,<t}) denote the conditional probability of the generated token yv,ty_{v,t}. We compute the length-normalized sequence probability as

svMSP=exp⁡(1Tv∑t=1Tvlog⁡pLM(yv,t∣hv,yv,<t)).(12)s_v^{\mathrm{MSP}} = \exp\left(\frac{1}{T_v}\sum_{t=1}^{T_v}\log p_{\mathrm{LM}}(y_{v,t} \mid h_v, y_{v,<t})\right). \tag*{(12)}

The corresponding uncertainty score is

uvMSP=1−svMSP.(13)u_v^{\mathrm{MSP}} = 1 - s_v^{\mathrm{MSP}}. \tag*{(13)}

Length normalization reduces the dependence of the score on response length, while lower sequence probability indicates greater uncertainty.

Prediction entropy. Prediction entropy measures uncertainty in the model’s next-token distributions. Let pv,t(k)p_{v,t}(k) denote the predicted probability of token kk at generation step tt, and let V\mathcal{V} be the vocabulary. The token-level predictive entropy is

Hv,t=−∑k∈Vpv,t(k)log⁡pv,t(k).(14)H_{v,t} = -\sum_{k \in\mathcal{V}} p_{v,t}(k)\log p_{v,t}(k). \tag*{(14)}

We aggregate token-level entropies over the generated sequence:

uvent=1Tv∑t=1TvHv,t.(15)u_v^{\mathrm{ent}} = \frac{1}{T_v}\sum_{t=1}^{T_v} H_{v,t}. \tag*{(15)}

Higher entropy indicates a more dispersed predictive distribution and therefore greater uncertainty. Unlike semantic entropy [21, 10], this signal operates directly on token-level distributions without grouping sampled responses by semantic equivalence.

Integration with local risk estimation. For each uncertainty signal s∈{verb,MSP,ent}s \in\{\mathrm{verb}, \mathrm{MSP}, \mathrm{ent}\}, we use the corresponding scalar uv(s)u_v^{(s)} as the agent-local input wvw_v to the learned local estimator:

bv=gϕ(uv(s)).(16)b_v = g_{\phi}(u_v^{(s)}). \tag*{(16)}

The local error-risk estimate is σ(bv)\sigma(b_v), which serves as the Receiver-Only prediction and provides the local component of the interaction-effect propagation model. All three signals are evaluated separately under the same experimental protocol.

Evaluation Metric

We evaluate error-risk estimation using the Area Under the Receiver Operating Characteristic Curve (AUROC). For each evaluated final-node response ii, let Ei∈{0,1}E_i \in\{0,1\} denote its correctness label, where Ei=1E_i = 1 indicates an incorrect answer and Ei=0E_i = 0 indicates a correct answer. Let r^i∈[0,1]\hat{r}_i \in[0,1] denote the corresponding predicted error risk.

The receiver operating characteristic (ROC) curve plots the true positive rate (TPR) against the false positive rate (FPR) as the decision threshold τ\tau varies:

TPR⁡(τ)=P(r^i≥τ∣Ei=1),FPR⁡(τ)=P(r^i≥τ∣Ei=0).(17)\begin{aligned} \operatorname{TPR}(\tau) &= P(\hat{r}_i \ge\tau\mid E_i = 1),\\ \operatorname{FPR}(\tau) &= P(\hat{r}_i \ge\tau\mid E_i = 0). \tag*{(17)} \end{aligned}

AUROC summarizes the area under this curve and can equivalently be interpreted as

AUROC⁡=P(r^+>r^−)+12P(r^+=r^−),(18)\operatorname{AUROC} = P(\hat{r}^{+} > \hat{r}^{-}) + \frac{1}{2}P(\hat{r}^{+} = \hat{r}^{-}), \tag*{(18)}

where r^+\hat{r}^{+} and r^−\hat{r}^{-} are the predicted risks of randomly selected incorrect and correct responses, respectively.

An AUROC of 0.5 corresponds to random discrimination, while 1.0 indicates perfect separation between incorrect and correct responses. Higher AUROC therefore indicates better discrimination of downstream errors. Unlike threshold-dependent classification metrics, AUROC evaluates the ranking of predicted risks across all decision thresholds, without requiring a fixed operating threshold. It measures discrimination rather than probability calibration. We compute AUROC on held-out final-node responses and report seed-averaged results where applicable.

Algorithm 1 Interaction-effect propagation: inference, training and KK-fold prediction.

Trace G=(V,E)G=(V,\mathcal{E}) for task qq; agent vv has role ρv\rho_v, visible output mvm_v, local signal wvw_v

1: procedure PROPAGATE(G,θ,ϕ,T=1)(G,\theta,\phi,T=1) 2: for (p,v)∈E(p,v)\in\mathcal{E} do 3: xpv←Enc⁡(q,ρp,mp,ρv,mv)x_{pv}\leftarrow\operatorname{Enc}(q,\rho_p,m_p,\rho_v,m_v) ▷ Eq. (2); frozen 4: (δpv(0),δpv(1))←fθ(xpv)(\delta_{pv}^{(0)},\delta_{pv}^{(1)})\leftarrow f_\theta(x_{pv}) 5: end for 6: for v∈Vv\in V in topological order do 7: bv←gϕ(wv)b_v\leftarrow g_\phi(w_v) ▷ Eq. (1) 8: for p∈Pa⁡(v)p\in\operatorname{Pa}(v) do 9: δˉpv←(1−r^p)δpv(0)+r^pδpv(1)\bar{\delta}_{pv}\leftarrow(1-\hat{r}_p)\delta_{pv}^{(0)}+\hat{r}_p\delta_{pv}^{(1)} ▷ Eq. (6) 10: end for 11: r^v←σ(bv+∑p∈Pa⁡(v)δˉpv)\hat{r}_v\leftarrow\sigma\left(b_v+\sum_{p\in\operatorname{Pa}(v)}\bar{\delta}_{pv}\right) ▷ Eq. (7) 12: end for 13: return {r^v}v∈V\{\hat{r}_v\}_{v\in V} ▷ roots: r^v=σ(bv)\hat{r}_v=\sigma(b_v) 14: end procedure

15: procedure TRAIN(D)(D) ▷ labeled traces; § 4 16: split DD into KK question-level folds D1,…,DKD_1,\ldots,D_K 17: for k=1,…,Kk=1,\ldots,K do 18: initialize θk,ϕk\theta_k,\phi_k 19: repeat ▷ epochs; minibatches B⊂D∖DkB\subset D\setminus D_k 20: {r^v}←\{\hat{r}_v\}\leftarrow PROPAGATE(G;θk,ϕk)(G;\theta_k,\phi_k), G∈BG\in B 21: L←∑G∈B∑v∈VBCE⁡(r^v,Ev)\mathcal{L}\leftarrow\sum_{G\in B}\sum_{v\in V}\operatorname{BCE}(\hat{r}_v,E_v) ▷ Eq. (8) 22: AdamW step on ∇θk,ϕkL\nabla_{\theta_k,\phi_k}\mathcal{L} ▷ through all r^p\hat{r}_p 23: until final-node AUROC on DkD_k stops improving 24: keep the best epoch; record its node logits on DkD_k 25: end for 26: return {(θk,ϕk)}k=1K\{(\theta_k,\phi_k)\}_{k=1}^{K} 27: end procedure

28: function PREDICT(G)(G) 29: return 1K∑k=1K\frac{1}{K}\sum_{k=1}^{K} PROPAGATE(G;θk,ϕk,T)(G;\theta_k,\phi_k,T) 30: end function

References

  1. [1]Amos Azaria and Tom Mitchell. 2023. The Internal State of an LLM Knows When It’s Lying. Preprint, arXiv:2304.13734.arxiv.org/abs/2304.13734
  2. [2]Julian Besag. 1974. Spatial Interaction and the Statistical Analysis of Lattice Systems. Journal of the Royal Statistical Society Series B: Statistical Methodology, 36(2):192–225.DOI
  3. [3]Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail? Preprint, arXiv:2503.13657.arxiv.org/abs/2503.13657
  4. [4]Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. 2021. WebQA: Multihop and Multimodal QA. Preprint, arXiv:2109.00590.arxiv.org/abs/2109.00590
  5. [5]Tiejin Chen, Huaiyuan Yao, Jia Chen, Evangelos E. Papalexakis, and Hua Wei. 2026. Every Response Counts: Quantifying Uncertainty of LLM-based Multi-Agent Systems through Tensor Decomposition. In Proceedings of ACL 2026.
  6. [6]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. Preprint, arXiv:2110.14168.arxiv.org/abs/2110.14168
  7. [7]Jinhao Duan, James Diffenderfer, Sandeep Madireddy, Tianlong Chen, Bhavya Kailkhura, and Kaidi Xu. 2025. UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making. Preprint, arXiv:2506.17419.arxiv.org/abs/2506.17419
  8. [8]Ran El-Yaniv and Yair Wiener. 2010. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(53):1605–1641.DOI
  9. [9]Ekaterina Fadeeva, Roman Vashurin, Akim Tsyvgun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. LM-Polygraph: Uncertainty Estimation for Language Models. In Proceedings of EMNLP 2023.arxiv.org/abs/2311.07383
  10. [10]Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630.DOI
  11. [11]Yonatan Geifman and Ran El-Yaniv. 2017. Selective Classification for Deep Neural Networks. Preprint, arXiv:1705.08500.arxiv.org/abs/1705.08500
  12. [12]Bairu Hou, Yang Zhang, Jacob Andreas, and Shiyu Chang. 2025. A probabilistic framework for LLM hallucination detection via belief tree propagation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3076–3099, Albuquerque, New Mexico. Association for Computational Linguistics.
  13. [13]Jun Hou, Priya Pitre, Yi Fang, and Xuan Wang. 2026. EDGE: Error Dependency Graph-Guided Multi-Error Attribution in Multi-Agent LLM Systems. In Proceedings of EMNLP 2026.arxiv.org/abs/2609.01360
  14. [14]Yibo Hu. 2026. One Axis, No Brake: Self-Knowledge Limits the Filtering of Harmful Peer Conformity in LLMs. Preprint, arXiv:2609.18998.arxiv.org/abs/2609.18998
  15. [15]Saeid Jamshidi, Arghavan Moradi Dakhel, Kawser Wazed Nafi, and Foutse Khomh. 2026. Hallucination Cascade: Analyzing Error Propagation in Multi-Agent LLM Systems. Preprint, arXiv:2606.07937.arxiv.org/abs/2606.07937
  16. [16]Di Jin, Eileen Pan, Nassim Oufattolle, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. Preprint, arXiv:2009.13081.arxiv.org/abs/2009.13081
  17. [17]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova Dassarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022. Language Models (Mostly) Know What They Know. Preprint, arXiv:2207.05221.arxiv.org/abs/2207.05221
  18. [18]Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. 2024. Large Language Models Must Be Taught to Know What They Don’t Know. In Advances in NeurIPS 2024.arxiv.org/abs/2406.08391
  19. [19]Michael Kirchhof, Gjergji Kasneci, and Enkeljeda Kasneci. 2025. Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents. In Proceedings of ICML 2025.arxiv.org/abs/2505.22655
  20. [20]Adam Kostka and Jaroslaw A. Chudziak. 2026. Controlling uncertainty and hallucination risk in multi-agent fact verification. In Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, volume 337 of Proceedings of Machine Learning Research, pages 3143–3161. PMLR.
  21. [21]Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. In Proceedings of ICLR 2023.arxiv.org/abs/2302.09664
  22. [22]Shi Lin, Chenpei Wang, Peng Qian, Dezhang Kong, Minghao Li, Yufeng Li, and Xun Wang. 2026. Before agents speak: Pre-hoc failure risk inference in multi-agent systems. Preprint, arXiv:2607.26836.arxiv.org/abs/2607.26836
  23. [23]Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. Preprint, arXiv:2205.14334.arxiv.org/abs/2205.14334
  24. [24]Yaokun Liu, Yifan Liu, Daniel Yue Zhang, Ruichen Yao, Zelin Li, and Dong Wang. 2026. PropUQ-MAS: Propagation-Aware Uncertainty Quantification for LLM Multi-Agent Systems. In Proceedings of EMNLP 2026.arxiv.org/abs/2608.22130
  25. [25]Zhengzhao Ma, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, and Le Sun. 2026. From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents. Preprint, arXiv:2608.16002.arxiv.org/abs/2608.16002
  26. [26]Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. In Proceedings of EMNLP 2023.
  27. [27]Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2020. Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics.
  28. [28]Radford M. Neal. 1992. Connectionist learning of belief networks. Artificial Intelligence, 56(1):71–113.DOI
  29. [29]Changdae Oh, Seongheon Park, To Eun Kim, Jiatong Li, Wendi Li, Samuel Yeh, Xuefeng Du, Hamed Hassani, Paul Bogdan, Dawn Song, and Sharon Li. 2026. Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities. In Proceedings of ACL 2026.
  30. [30]Judea Pearl. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Mateo, CA.
  31. [31]Jiaming Qu, Lucheng Fu, and Yibo Hu. 2026. Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity. Preprint, arXiv:2606.01637.arxiv.org/abs/2606.01637
  32. [32]Lucas Nerone Rillo, Zhanhong Jiang, Nastaran Saadati, Aditya Balu, Baskar Ganapathysubramanian, Chinmay Hegde, and Soumik Sarkar. 2026. Adko: Agentic decentralized knowledge optimization. Preprint, arXiv:2605.07863.arxiv.org/abs/2605.07863
  33. [33]Prabhjot Singh and Bhushan Pawar. 2026. The Hallucination Snowball: Modeling Error Propagation as State Transitions in Multi-Agent LLM Pipelines. In ICML 2026 FAGEN Workshop.arxiv.org/abs/2608.14588
  34. [34]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. In Proceedings of EMNLP 2023.
  35. [35]Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Akim Tsvigun, Daniil Vasilev, Rui Xing, Abdelrahman Boda Sadallah, Kirill Grishchenkov, Sergey Petrakov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025. Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph. Transactions of the Association for Computational Linguistics.

Paper details

vlnc:W_4uh5dxdg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents