betaFeedback

WorksComputer Science

Multi-Agent Retrieval-Augmented Causality Assessment (MARCA) of Individual Case Safety Reports in Pharmacovigilance

Abstract

Introduction: Increasing volumes of individual case safety reports (ICSRs) challenge expert-led causality assessment in pharmacovigilance. We developed MARCA (Multi-Agent Retrieval-Augmented Causality Assessment), a four-agent framework for Naranjo-based assessment, and evaluated agreement with human experts and exploratory differences between non-RAG and RAG-labelled scores. Methods: MARCA combined scientific literature retrieval, drug-label assessment, objective-evidence classification, and clinical causality reasoning using Gemini 2.5 Pro and DeepSeek-V3.2. Agent-generated rationales informed human-assisted scoring. A consensus expert reference standard covered 215 ICSRs and 1,562 drug-event combinations (DECs) from FAERS (Q1-Q2 2025). Agreement was measured using percentage agreement and Cohen's kappa. Results: Question-level agreement was 96.8% for Naranjo Q2 (kappa = 0.491; 95% CI 0.265-0.663), 85.4% for Q5 (kappa = 0.269; 95% CI 0.180-0.340), and 96.8% for Q10 (kappa = 0.922; 95% CI 0.885-0.951). Final causality classification agreement was 79.4% (95% CI 71.8-85.3; kappa = 0.587, 95% CI 0.450-0.690). In a retrospective subset of 60 DECs, agreement was higher for RAG-labelled than non-RAG scores for Q5 (83.3% vs 65.0%) and Q10 (100.0% vs 18.3%). Discussion: MARCA-assisted assessment showed substantial variation in agreement across Naranjo questions, with lower concordance for alternative causes. Retrospective RAG comparisons are exploratory and do not establish prospective effectiveness.

Introduction

In the United States (US), the FDA Adverse Event Reporting System (FAERS) received more than two million individual case safety reports (ICSRs) of suspected post-authorization adverse events (AEs) in 2025, nearly three times the volume reported in 2011 [1]. Individual-level causality assessment is one of the most time- and resource-intensive tasks in pharmacovigilance (PV) and has become increasingly challenging as reporting volumes continue to grow [2, 3].

Off-the-shelf, general-purpose large language models (LLMs) have been investigated as potential tools to support individual-level causality assessment. Abate et al. evaluated general-purpose LLMs using the World Health Organization (WHO) causality assessment tool. Their findings revealed difficulties in identifying listed AEs in the Summary of Product Characteristics (SmPC), along with inconsistent clinical and pharmacological reasoning [4]. Subsequently, Heckmann et al. evaluated biomedical LLMs using structured prompt engineering and the Naranjo algorithm, finding the optimal prompt-algorithm combination and achieving 63.3% agreement in final causality classifications (Gwet’s AC1 =0.644= 0.644) [5]. Kefala et al. demonstrated that a state-of-the-art general-purpose LLM could accurately identify listed AEs directly from product labels when given a targeted task, addressing a specific limitation that had constrained the assessment of the first Naranjo question in previous experiments [6]. More recently, Heckmann et al. applied inference-time temperature optimization, increasing agreement in final individual-level causality classifications between human experts and LLMs from a baseline of 45.0% to 72.0%, highlighting the substantial influence of inference settings on LLM performance in causality assessment [7]. Despite these advances, previous experiments have demonstrated persistent limitations in addressing pivotal questions of the Naranjo algorithm that require multistep clinical judgment, particularly those involving temporal plausibility, the identification of alternative causes of AEs, and the assessment of objective evidence.

Multi-agent architecture offers a potentially suitable approach to addressing these limitations by decomposing complex causality assessments tasks into specialized subtasks, allowing individual agents to focus on distinct components of the assessment process in the subtask [8]. Furthermore, previous experiments did not explicitly incorporate expert-derived knowledge into LLM-based systems. Retrieval-augmented generation (RAG) was therefore hypothesized to support specialized tasks requiring clinical judgment by providing agents with access to external, expert-curated information rather than relying solely on knowledge acquired during model pretraining [9].

To our knowledge, no published study has combined a multi-agent architecture with selective RAG for individual-level causality assessment in PV. Therefore, this study aimed to (1) develop a Multi-Agent Retrieval-Augmented Causality Assessment (MARCA) framework for Naranjo-based causality assessment and (2) quantify agreement between MARCA-assisted assessments and a human expert reference standard and explore the question-specific differences between stored non-RAG and RAG-labelled assessments.

Materials and Methods

Study Design

An overview of the study design is provided in Figure 1. MARCA assessed preprocessed FAERS ICSRs through four specialized LLM agents responsible for scientific evidence retrieval, drug label assessment, objective evidence assessment, and clinical causality assessment. A CrewAI orchestrator coordinated agent execution and information exchange. Selective RAG, supported by an expert-curated knowledge base, was evaluated for Naranjo questions requiring complex clinical judgment, specifically temporal plausibility (Question 2, Q2), alternative causes (Q5), and objective evidence (Q10). The framework generated question-level rationales and causality assessments, which were evaluated against human expert assessments and benchmarked descriptively against previously published single-agent LLM approaches.

Study design

Figure 1. Study design. A FAERS individual case safety report is processed by four specialized agents that assess scientific evidence, drug label listedness, objective evidence, and clinical causality. A knowledge base containing expert-assessed drug–event combinations supports selective retrieval-augmented generation for Naranjo Questions 2, 5, and 10. The outputs are integrated to generate question-level rationales, calculate the cumulative Naranjo score, and determine the final causality classification before export for downstream evaluation.

Legend: AE = adverse event; DEC = drug–event combination; FAERS = FDA Adverse Event Reporting System; FDA = U.S. Food and Drug Administration; ICSR = individual case safety report; KB = knowledge base; LLM = large language model; MARCA = Multi-Agent Retrieval-Augmented Causality Assessment; MOA = mechanism of action; Q = Naranjo Algorithm question; RAG = retrieval-augmented generation.

Disclaimer: The figure is AI-generated.

Data Sources, Pre-Processing, and Sampling

A total of 2,577,782 records were extracted from FAERS quarterly data files covering the first two quarters (Q1–Q2) of 2025. Duplicate records were handled according to the predefined data-cleaning procedure, retaining the most recent or most complete record for each PRIMARYID [10]. ICSRs were integrated using their unique report identifiers (PRIMARYID) to reconstruct case-level records containing patient demographics, drug exposures, reported AEs, clinical outcomes, therapeutic indications, and reporter characteristics (Supplementary Figure S1 and Supplementary Table 1) [11, 12]. Patient ages were standardized to years using the corresponding FAERS age-unit codes, and implausible values were excluded. Drug names were normalized to RxNorm concepts (RxCUIs) using the Unified Medical Language System (UMLS) 2025AA release, while AE terms were coded to MedDRA version 27.1 Preferred Terms (PTs) were mapped to their corresponding High-Level Group Terms (HLGTs) and System Organ Classes (SOCs). To obtain a computationally feasible dataset, a sampling procedure was implemented (Supplementary Figure 2). Drug-event combinations (DECs) were constructed by pairing reported drug entries with adverse event terms within each ICSR. Drug role codes recorded in FAERS were retained to distinguish primary suspect (PS), secondary suspect (SS), and concomitant (C) medications. Drug active ingredients were subsequently mapped to Anatomical Therapeutic Chemical (ATC) classification level 4, while AE terms were mapped to MedDRA HLGTs. ATC level 4-HLGT combinations were then randomly sampled using the slice_sample() function from the dplyr package (version 1.2.1) in R (version 4.5.1), with a fixed random seed (1234) to ensure reproducibility. The sampling procedure preserved coverage across all 1,061 ATC level 4 classes and 337 HLGT categories represented in the source dataset, which comprised 55,177 FAERS cases, 4,668 unique drugs, and 9,168 distinct PTs. From the stratified dataset, 215 ICSRs were selected for expert causality assessment based on feasibility. These reports yielded 1,562 DECs, which constituted the evaluation dataset. ICSRs containing the sampled ATC level 4–HLGT combinations were subsequently retrieved, and structured clinical narratives were generated programmatically using a standardized template incorporating patient demographics, drug exposures, therapeutic indications, reported AEs, dosing information, clinical outcomes, and treatment timelines. These structured narratives served as the primary case inputs for both human expert assessment and the MARCA framework (Supplementary Figure S3).

FAERS Variables
Case ID
Case Version
Event Date
Authorization Number
Manufacturer Number
Manufacturer Sender
Literature Reference
Age
Age Code
Age Group
Sex
E Sub
Weight
Weight Code
Occupation Code
Reporter Country
Occurrence Country
Adverse Reaction(s)
Case ID Number
Drug Sequence
Role Code
Drug Name
Product AI
Val VBM
Route
Dose VBM
Cumulative Dose (Char)
Cumulative Dose (Unit)
Dechallenge
Rechallenge
Lot Number
Expiration Date
NDA Number
Dose Amount
Dose Unit
Dose Form
Dose Frequency
Outcome Code
FDA Code
Manufacture Date
Initial FDA Date
FDA Date
Report Source Code
Indication for Use
Therapy Start Date
Therapy End Date
Duration
Duration Code

Supplementary Table 1. FAERS ICSR Structured Template. Legend: NA = Not available, CN = Consumer, YR = Year, FAERS = Food and Drug Administration Adverse Event Reporting System

Relational structure of the FAERS Database, showing interconnected Therapy, Indication, Drug, Demographic, Reaction, Outcome, and Report_Sources tables

Figure S1. Relational Structure of the FAERS Database. The FAERS database comprises interconnected tables capturing patient demographics, drug exposure, therapy, indications, reported reactions, clinical outcomes, and report sources. These tables are linked through shared identifiers to enable integration of case-level adverse event data. Legend: AE = adverse event; CASEID = case identification number; FAERS = FDA Adverse Event Reporting System; MedDRA = Medical Dictionary for Regulatory Activities; PT = Preferred Term; SEQN = sequence number; SOC = System Organ Class.

Workflow for Stratified Down-Sampling of FAERS Q1-Q2 2025 Data

Supplementary Figure 2. Workflow for Stratified Down-Sampling of FAERS Q1-Q2 2025 Data. FAERS reports from Q1–Q2 2025 were standardized using drug and adverse event terminology mapping, stratified by unique ATC4–HLGT combinations, and down-sampled to one representative case per stratum to create the expert-assessed dataset. Legend: AE = adverse event; ATC4 = Anatomical Therapeutic Chemical Classification System, Level 4; FAERS = FDA Adverse Event Reporting System; HLGT = High Level Group Term; MedDRA = Medical Dictionary for Regulatory Activities; PT = Preferred Term; Q1–Q2 2025 = first and second quarters of 2025; RxCUI = RxNorm Concept Unique Identifier. Disclaimer: The figure is AI-generated.

Retrieval of ICSRs and generation of standardized structured clinical narratives for human expert and MARCA-based causality assessment

Figure S3. Retrieval of ICSRs and generation of standardized structured clinical narratives for human expert and MARCA-based causality assessment. ICSR retrieval based on sampled ATC level 4–HLGT combinations and programmatic generation of standardized clinical narratives for human expert and MARCA assessment. Legend: ICSR = individual case safety report, MARCA = Multi-Agent Retrieval-Augmented Causality Assessment. Disclaimer: The figure is AI-generated.

Expert Causality Assessment

Causality was assessed using the Naranjo Adverse Drug Reaction Probability Scale [13]. The junior assessor evaluated 215 ICSRs, encompassing 1,562 DECs. For each Naranjo question, the junior assessor assigned a numerical score and documented the supporting clinical rationale in a structured assessment dataset. All assessments were subsequently reviewed by a senior pharmacovigilance expert, and question-level scores were reconciled to establish the final consensus reference standard. These consensus assessments constituted the human expert gold standard against which MARCA outputs were evaluated. The assessment criteria, Standard Operative Procedures, and review procedures are described in Appendix 1.

MARCA System Architecture

MARCA was developed using the CrewAI multi-agent orchestration framework (version 0.126.0), comprising four specialized agents described in sections 2.4.1-2.4.4 [8, 14]. The Naranjo algorithm and chain-of-thought (CoT) prompting were selected as the causality assessment method and prompting strategy, respectively, based on their superior performance in previous comparative studies [5, 7, 15]. All models were configured with a sampling temperature of zero to minimize output variability and enhance reproducibility [7]. The overall architecture and representative prompts are presented in Figure 1 and Supplementary Table S2.

AgentRepresentative prompt design
Agent 1 – Scientific Evidence Retrieval (Gemini 2.5 Pro)Role: expert medical searcher and literature summarizer. Goal: analyze PubMed results for [drug] and [adverse event] to summarize mechanism of action and clinical evidence. Search string: “[drug name] AND [adverse event term] adverse event” (NCBI Entrez); up to 3 records retrieved per DEC; summary truncated to 800 characters before being passed to Agent 4.
Agent 2 – Drug Label Assessment (DeepSeek-V3.2)System prompt: “You are a medical documentation assistant that evaluates AE-drug relations.” User prompt (not-listed path only): “Determine if the following adverse event is medically related to the drug based on clinical reasoning. Drug: [drug name]. Adverse Event: [adverse event]. Output: Related – <Yes/No/Unclear>; Explanation – <brief medical explanation>.” Events found directly in the FDA label reference file are classified as Listed without invoking this prompt.
Agent 3 – Objective Evidence Assessment (DeepSeek-V3.2)System prompt: “You are a clinical pharmacovigilance expert. Classify events as ‘Sign’ (value 1) or ‘Symptom’ (value 0).” User prompt: “Analyze the medical term: ‘[adverse event]’” returns a JSON object with the sign/symptom classification and a brief supporting rationale.
Agent 4 – Clinical Causality Assessment (Gemini 2.5 Pro)Receives the full structured case context (demographics, drug exposure, temporal data, dechallenge/rechallenge, concomitant medications; truncated to 1,400 characters) plus the Agent 1 literature summary. Produces a clinical rationale and a descriptive plausibility rating (high/medium/low) for Naranjo questions 2 through 9, following Chain-of-Thought (CoT) reasoning.

Table S2. Representative Prompt Designs by Agent.

Agent 1 - Scientific Evidence Retrieval

Agent 1, powered by Gemini 2.5 Pro, was designed to retrieve and synthesize scientific evidence relevant to each DEC. For each DEC, PubMed was queried through the National Center for Biotechnology Information (NCBI) Entrez E-utilities Application Programming Interface (API) using a structured search string combining the drug, AE term, and the expression "adverse event" (i.e., "[drug name] AND [adverse event term] adverse event"). Up to three article records were retrieved, including their PubMed identifiers (PMIDs), titles, and available abstracts. The retrieved information was formatted into a standardized literature corpus and provided to a CrewAI agent instructed to summarize the drug's mechanism of action (MoA) and the available clinical evidence relevant to the reported DEC. The resulting narrative summary was subsequently passed to Agent 4 as supporting evidence for causality assessment. The input structure, processing procedure, prompt template, and expected output are illustrated in Supplementary Figure 4.

Agent 1 scientific evidence retrieval workflow

Supplementary Figure 4. Agent 1 - Scientific Literature Retrieval and Synthesis. PubMed evidence retrieval via the NCBI Entrez E-utilities API, standardized literature corpus generation, and LLM-based synthesis of drug mechanisms of action and clinical evidence to support Agent 4 causality assessment. Disclaimer: The figure is AI-generated.

Agent 2 - Drug Label Assessment

Agent 2, powered by DeepSeek-V3.2, was developed to assess the listedness of reported AEs in FDA-approved drug labeling and provide supporting information for Naranjo Question 1. For each drug, a reference file containing AE terms was generated from the product label using the approach described by Kefala et al. [6]. This procedure involved extracting the relevant safety information from the product label and organizing the identified AEs into a drug-specific reference list. For each DEC, the reported AE term was compared with the entries in the corresponding drug-label reference file. When an exact match was identified after case normalization, the event was classified as "Listed in label." When no matching term was identified, the LLM was prompted to evaluate whether the reported AE was medically related to the AE in the product label based on clinical reasoning. The model returned a categorical assessment (Yes, No, or Unclear) together with a brief clinical explanation. The input structure, decision rules, prompt template, and output categories are illustrated in Supplementary Figure 5.

Agent 2 Drug Label Assessment schematic

Supplementary Figure 5. Agent 2 Drug Label Assessment. Comparison of reported adverse events with drug-specific reference lists extracted from FDA-approved product labeling, followed by LLM-based clinical assessment of unmatched events and assignment of standardized output categories. Disclaimer: The figure is AI-generated.

Agent 3 - Objective Evidence Assessment

Agent 3, powered by DeepSeek-V3.2, was designed to support the assessment of Naranjo Question 10 by distinguishing objectively observable or measurable clinical signs from subjective symptoms. For each reported AE term, the agent applied a structured prompting strategy instructing the model to classify the event as either a "Sign" (value = 1) or a "Symptom" (value = 0), based on whether the event represented an objectively observable clinical finding or a subjective patient-reported experience. Psychiatric disorders were explicitly assigned to the symptom category according to the implemented classification rules. The agent generated a structured JSON output containing the AE term, its classification, the corresponding numerical value, and a brief medical rationale supporting the decision. This classification provided a standardized, term-level proxy for objective evidence relevant to Naranjo Question 10. The input structure, classification rules, prompt template, and expected output are presented in Supplementary fig:6.

Agent 4 - Clinical Causality Assessment

Agent 4, powered by Gemini 2.5 Pro, was designed to integrate case-specific clinical information with the scientific evidence synthesized by Agent 1 to generate a comprehensive causality assessment for each DEC. The input comprised a structured clinical context containing available case demographics, relevant medical history, drug exposure, therapeutic indications, treatment dates, adverse event timing, dosage information, dechallenge and rechallenge findings, clinical outcomes, and concomitant medications. Within the CrewAI workflow, Agents 1 and 4 formed a sequential dependency, whereby the literature summary generated by Agent 1 was made available to Agent 4 as additional context. Agent 4 was instructed to produce a standardized causality report addressing temporal plausibility, biological plausibility, dechallenge and rechallenge, objective evidence, confounding factors, previous similar reactions, and an overall synthesis. The report included clinical reasoning relevant to Naranjo questions and a descriptive plausibility rating (High, Medium, or Low). All reasoning was required to be based on the available case information and supporting literature. In the absence of retrieved scientific evidence, the assessment could rely on the available structured case information, without assuming that missing clinical information was negative evidence. The input structure, prompt template, sequential integration with Agent 1, and expected output format are illustrated in Supplementary fig:7.

Output of MARCA

The four agents generated complementary drug-label, objective sign-symptom, literature, and clinical reasoning outputs at DEC level. The human junior assessor then reviewed the agent-generated rationales and assigned numerical Naranjo scores, which were stored separately from the consensus human expert reference scores. Accordingly, the evaluated workflow incorporated human scoring of agent-generated evidence rather than fully automated question-level scoring. The resulting question-level scores and supporting rationales were recorded alongside the corresponding consensus human expert assessments in a structured dataset. Total Naranjo scores were calculated separately for the MARCA-assisted and human expert assessments by summing the scores across all ten Naranjo questions. Final causality classifications were then derived using the standard Naranjo thresholds (Definite, Probable, Possible, and Doubtful).

Selective Retrieval-Augmented Generation

RAG was selectively applied to Naranjo Questions 2 (temporal relationship), 5 (alternative causes), and 10 (objective evidence), which had been identified as particularly challenging in previous LLM-based causality assessments [5, 7]. For each DEC, an initial assessment was generated without RAG using the available case information. When the RAG module was enabled, it generated both non-RAG and RAG-assisted question-level outputs, without using the human expert score as a prerequisite for retrieval. The RAG-assisted outputs incorporated expert-annotated reference cases, question-specific scoring rules, and relevant clinical rationales retrieved from a knowledge base (KB). To prevent direct self-retrieval, all records belonging to the ICSR under evaluation were excluded from the retrieval pool using the corresponding FAERS case identifier. Drug pharmacological classes were resolved sequentially using the openFDA drug label API, the RxNorm API, and, when neither source provided a suitable classification, through DeepSeek-V3.2. Three term frequency–inverse document frequency (TF-IDF) indices were constructed to support hierarchical retrieval at the individual-drug, pharmacological-class, and case levels. Candidate reference cases were ranked according to cosine similarity between their TF-IDF representations and the corresponding query. Retrieval strategies were tailored to the clinical requirements of each Naranjo question. For Question 2, retrieval prioritized reference cases involving the same drug, expanding to pharmacologically related drugs when fewer than three suitable references were identified, with global retrieval used as a fallback. When both treatment initiation and AE onset dates were unavailable, retrieval was restricted to reference cases with similarly missing temporal information, and a predefined score of zero was incorporated into the prompt. For Question 5, the same hierarchical retrieval strategy was applied, with preference given to reference cases exhibiting comparable information availability regarding concomitant medications and comorbidities. For Question 10, retrieval relied exclusively on AE term similarity, independently of drug identity, reflecting the agent's task of distinguishing objectively observable clinical signs from subjective symptoms. Up to three reference cases were retrieved for Questions 2 and 5, and up to two for Question 10. A minimum cosine similarity threshold of 0.01 (arbitrary) was applied during hierarchical retrieval. The retrieved references were incorporated into question-specific prompts for DeepSeek-V3.2 (the template is provided in the GitHub repository), together with their retrieval tier, cosine similarity score, case identifier, drug, AE term, and corresponding human expert score and clinical rationale. The model was instructed to use these references to inform and calibrate its assessment rather than mechanically reproduce the retrieved expert scores. The resulting RAG-assisted outputs were subsequently evaluated against the human expert reference standard. A retrospective ablation analysis was conducted on a subset of 60 DECs for which paired non-RAG and RAG-labelled question-level assessments were available. The non-RAG assessments in this subset exhibited disagreement with the human expert reference standard on at least one of Naranjo Questions 2, 5, or 10. The subset was used to compare question-level agreement between the two conditions and was not assumed to represent all discordant DECs in the complete evaluation dataset. The provenance of the RAG-labelled scores as original retrieval-run predictions could not be independently verified from the saved evaluation exports. All prompts were finalized before the evaluation was performed and remained unchanged throughout the assessment. Structured reasoning prompts were used across the framework, with standardized output formats specified for the drug-label and objective-evidence assessment agents.

Study Outcomes and Acceptable Agreement Thresholds

The primary outcome was the agreement for each DEC between MARCA and human expert assessments at the individual Naranjo question level, evaluated for Questions 2–10 at the DEC level using percentage agreement and Cohen’s κ\kappa coefficient [16, 17]. Question 1 was excluded from the primary comparative analysis because its assessment relied primarily on FDA drug-label listedness detection through Agent 2, rather than the multistep clinical reasoning targeted by MARCA, and because automated AE listedness detection had previously been evaluated separately [6]. Nevertheless, Question 1 was retained in the calculation of the cumulative Naranjo score and evaluated descriptively. The secondary outcome was agreement in the final causality classification, assessed using percentage agreement and unweighted Cohen’s κ\kappa. Final classifications were derived by summing the scores across all ten Naranjo questions and applying the standard classification thresholds: Doubtful (≤0\leq0), Possible (1–4), Probable (5–8), and Definite (≥9\geq9). Exact agreement in cumulative numerical scores was also examined. For questions with invariant scores across both assessments, Cohen’s κ\kappa was considered non-estimable, and percentage agreement was reported descriptively.

An exploratory outcome evaluated the incremental contribution of selective RAG to question-level agreement for Questions 2, 5, and 10. This retrospective ablation analysis included 60 DECs for which the initial non-RAG assessment disagreed with the human expert reference standard on at least one of the three targeted questions (i.e., Questions 2, 5, and 10). Agreement with the reference standard was compared between non-RAG and RAG-assisted assessments to characterize the effect of retrieval on question-level scoring (Questions 2, 5, and 10).

Predefined question-specific percentage-agreement thresholds, established in consultation with industry collaborators before the MARCA evaluation, were used as performance benchmarks (Supplementary Table 3). Thresholds were set at >95% for Questions 1–4 and 8, reflecting their anticipated reliance on relatively direct evidence from regulatory product information or case-specific clinical data, and >60% for Questions 5, 9, and 10, which were considered more dependent on complex clinical interpretation or incompletely documented information. No predefined thresholds were established for Questions 6 and 7 because placebo-response information and toxic drug concentration measurements are rarely available in FAERS reports.

Naranjo QuestionAcceptable AgreementRationaleConsiderations
1>95%Can be determined from the product label.The term conclusive is subjective and may reduce agreement to 60-70%.
2>95%If the event preceded drug exposure, it would not constitute an ICSR.The key consideration, not addressed by Naranjo, is whether the time to onset is clinically reasonable.
3>95%Can be derived from structured ICSR fields.For multiple drugs or events, dechallenge may not correspond to the evaluated DEC. Antagonists are rarely reported (primarily naloxone for opioid overdose).
4>95%Can be derived from structured ICSR fields.Rechallenge may not correspond to the evaluated DEC; antagonists are seldom reported.
5>60%Requires consideration of concomitant drugs, medical history, and patient-related factors.Co-medications are easier to identify than underlying medical conditions.
6N/APlacebo use does not apply to post-marketing reports.Not applicable to ICSRs; only theoretical non-exposure periods could serve as a comparator.
7N/ADrug concentrations are rarely reported.Therapeutic or toxic concentration thresholds are often unavailable except in overdose.
8>95%Can be derived from structured ICSR fields.Structured fields may not correspond to the same DEC; dose-related severity changes are uncommon.

Supplementary Table 3. Supplementary Table 3. Acceptable Agreement Thresholds for each Naranjo Question. Acceptable agreement thresholds, supporting rationale, and key considerations for each Naranjo Algorithm question. Thresholds were established in consultation with experts from GSK. Legend: DEC = drug-event combination; ICSR = individual case safety report.

Statistical Analysis

Continuous variables were summarized using means, medians, and standard deviations, while categorical variables were presented as frequencies and percentages. The characteristics of the human expert-assessed dataset were descriptively compared with those of the broader stratified FAERS dataset to evaluate similarities in patient demographics, drug exposures, and AE distributions. Agreement between MARCA and human expert assessments was evaluated at the individual DEC level using percentage agreement and Cohen’s κ\kappa coefficient. Question-level agreement was calculated for all ten Naranjo questions, with Question 1 reported descriptively but excluded from the primary comparative analysis, which focused on Questions 2–10. Cumulative scores used for final causality classification were calculated by summing all ten Naranjo questions (Q1–Q10), including Question 1. The resulting scores were classified as Doubtful (≤0\leq0), Possible (1–4), Probable (5–8), or Definite (≥9\geq9). Agreement in final causality classifications was evaluated using percentage agreement and Cohen’s κ\kappa. For questions with invariant scores across both assessments, Cohen’s κ\kappa was considered non-estimable, and percentage agreement was reported descriptively. Agreement patterns were summarized using question-level comparison tables and confusion matrices reporting absolute frequencies and within-expert-category percentages. Directional scoring disagreements were evaluated by calculating the difference between MARCA-generated and human expert-assigned scores for each DEC and Naranjo Question 2–10. Positive differences indicated higher scores assigned by MARCA, whereas negative differences indicated lower scores relative to the human expert reference standard. The frequencies of positive, negative, and concordant assessments were calculated for each question, together with mean signed score differences, to identify systematic directional patterns in causality scoring. MARCA performance was descriptively benchmarked against three previously published LLM-based causality assessment studies [4, 5, 7], using percentage agreement, Cohen’s κ\kappa or Gwet’s AC1 where reported, and available score-error measures. Cross-study comparisons were restricted to outcomes considered sufficiently comparable in terms of causality assessment algorithm, unit of analysis, dataset characteristics, and metric definition. Outcomes that could not be meaningfully compared were designated as not comparable (NC), whereas metrics unavailable in the original publications were designated as not reported (NR). No formal pooled statistical comparisons were performed across studies. The incremental contribution of selective RAG was evaluated using question-level percentage agreement, unweighted Cohen’s κ\kappa, mean signed score differences, and frequencies of overestimation, agreement, and underestimation were compared between non-RAG and RAG-assisted assessments to characterize the effect of retrieval on scoring agreement and directional disagreement patterns.

Implementation and Reporting Standards

Data acquisition, preprocessing, harmonization, and sampling were performed in R version 4.5.1. MARCA, RAG retrieval, and statistical analyses were implemented in Python version 3.11. TF-IDF vectorization and cosine-similarity retrieval were performed using scikit-learn, while CrewAI was used for multi-agent orchestration. External API integrations include openFDA, RxNorm, NCBI Entrez/PubMed, Google Gemini, and DeepSeek. LLM inference was performed through API-based access to Gemini 2.5 Pro and DeepSeek-V3.2, with temperature set to zero. Case and literature context provided to the models was truncated to 1,400 and 800 characters, respectively, to manage computational and API constraints. All evaluation runs were conducted without prompt modification after the assessment protocol was finalized. Analysis code, agent prompts, and model outputs are available through the GitHub repository provided in the Data Availability statement.

Results

ICSR and Drug-Event Combination Characteristics

The expert-assessed dataset comprised 215 unique ICSRs, encompassing 1,562 DECs. Female patients accounted for 131 ICSRs (60.9%), while the United States of America was the most frequently reported country of the reporter (130 ICSRs, 60.5%). Patient age was available for 141 ICSRs (65.6%), with a median age of 59 years. The median number of reported AEs per ICSR was 3 (mean 4.35, standard deviation [SD] 4.34; range 1–53). The median number of distinct drugs represented among the evaluated DECs was 1 (mean 1.23, SD 0.80; range 1–7). The number of DECs per ICSR had a median of 4 (mean 7.27, SD 27.18; range 1–371), with one ICSR contributing 371 DECs (23.8% of the evaluated combinations) (Supplementary Table 4 and Supplementary Table 5).

VariableTotal Dataset (n = 55,177)MeanMedianSDExpert-assessed Dataset (n = 215)MeanMedianSD
Age (years)38,22754.9059.021.1614153.2659.0020.70
Weight (kg)19,14073.0571.037.684265.8767.6225.91
AEs per case55,1339.385.015.202154.353.004.34
Distinct drugs per case55,13315.808.043.672151.231.000.80

Supplementary Table 4. Summary of Continuous Variables for the Total and Expert-assessed Datasets. Summary statistics for continuous demographic and case-level variables in the total FAERS dataset and the expert-assessed dataset used for MARCA agreement analysis. Legend: AE = adverse event; FAERS = FDA Adverse Event Reporting System; SD = standard deviation.

VariableMinMedianMax
DECs per case14371
AEs per case1353
Distinct drugs per case117

Supplementary Table 5. Case-level Characteristics of the Expert-assessed Dataset. Distribution of DECs, AEs, and drugs per case. Legend: AE = adverse event; DEC = drug-event combination.

Question-Level Agreement

A representative example of an ICSR evaluated using MARCA is presented in Figure 2. Agreement between MARCA and human expert assessments for Naranjo Questions 2–10 is summarized in Table 1. Although excluded from the primary question-level analysis, Question 1 showed 89.3% agreement with the expert reference standard (unweighted Cohen’s κ\kappa = 0.787), below the prespecified >95% agreement threshold. Perfect percentage agreement (100%) was observed for Questions 3, 4, and 6–9. For Question 3, both MARCA and human experts assigned a score of zero to 1,546 DECs and +1 to the remaining 16 DECs (Cohen’s κ\kappa = 1.000). In contrast, all DECs received a score of zero for Questions 4 and 6–9, resulting in perfect percentage agreement but non-estimable Cohen’s κ\kappa coefficients due to the absence of score variability. Among questions exhibiting disagreement, Question 10 demonstrated the highest percentage agreement (96.8%) and chance-corrected agreement (Cohen’s κ\kappa = 0.922; 95% CI 0.885-0.951), with similar mean scores between MARCA and human expert assessments (0.70 vs. 0.72, respectively). Question 2 also demonstrated high percentage agreement (96.8%), although Cohen’s κ\kappa was lower (κ\kappa = 0.491; 95%CI 0.265-0.663), reflecting the predominance of zero scores in both assessments. Question 5 exhibited the lowest percentage agreement (85.4%) and Cohen’s κ\kappa (κ\kappa = 0.269; 95%CI 0.180-0.340). The mean score for Question 5 was lower for MARCA than for human expert assessments (−0.85 vs. −0.50, respectively), indicating a systematic tendency toward lower causality scores when evaluating alternative causes.

Naranjo QuestionDescriptionAgreement (%)Cohen’s κ\kappa (95%CI)Mean MARCA ScoreMean Expert Score
Q2Temporal association96.8%0.491 (0.265-0.663)0.090.04
Q3Improvement following drug discontinuation100.0%1.0000.010.01
Q4Reappearance following rechallenge100.0%N/A0.000.00
Q5Alternative cause excluded85.4%0.269 (0.180-0.340)−0.85−0.50
Q6Reaction following placebo administration100.0%N/A0.000.00
Q7Toxic drug level detected100.0%N/A0.000.00
Q8Dose-response relationship100.0%N/A0.000.00
Q9Similar reaction to previous exposure100.0%N/A0.000.00
Q10Objective evidence confirmation96.8%0.922 (0.885-0.951)0.700.72

Table 1. Agreement Between MARCA and Human Expert Assessors Across Naranjo Questions. Agreement between MARCA and human expert assessors across Naranjo Q2–Q10, based on the expert-assessed dataset. Mean MARCA score and mean human score represent the mean scores assigned by each rater for each question. Imbalanced score distributions resulted in reduced or undefined κ\kappa values despite high percentage agreement.

Representative example of an ICSR evaluated using MARCA

Figure 2. Representative example of an ICSR evaluated using MARCA. Example of a structured individual case safety report (ICSR) processed by MARCA, illustrating the integration of patient demographics, suspected drug information, and reported adverse events to assess data completeness, identify missing clinical information, and highlight medication errors and relevant clinical considerations. Legend: ICSR = individual case safety report, MARCA = Multi-Agent Retrieval-Augmented Causality Assessment. Disclaimer: The figure is AI-generated.

Direction of Scoring Disagreements

The direction and magnitude of scoring disagreements between MARCA and human expert assessments are summarized in Table 2. Score differences were calculated as the MARCA-assigned score minus the corresponding human expert score, with positive and negative values indicating overestimation and underestimation, respectively. Question 5 exhibited the largest directional disagreement, with MARCA assigning lower scores than human experts in 205 of 1,562 DECs and higher scores in 23 DECs, resulting in a mean signed difference of −0.350. For Question 2, MARCA assigned higher scores in 44 DECs and lower scores in 6 DECs, yielding a small positive mean difference (+0.049). For Question 10, MARCA assigned lower scores in 42 DECs and higher scores in 8 DECs, resulting in a mean signed difference of −0.022. Overall, the largest systematic difference was observed for Question 5, indicating that MARCA tended to assign lower causality scores than human experts when evaluating alternative causes. This finding does not independently establish whether more alternative causes were correctly identified.

Naranjo QMARCA > Expert (n)MARCA = Expert (n)MARCA < Expert (n)Mean difference
Q2441,5126+0.049
Q3, Q4, Q6–Q901,56200.000
Q5231,334205−0.350
Q1081,51242−0.022

Table 2. Direction of scoring disagreements between MARCA and human expert assessors across Naranjo questions 2-10. Positive mean differences indicate higher scores assigned by MARCA than by experts, whereas negative values indicate lower scores. A mean difference of zero indicates no directional difference.

Classification-Level Agreement

At the final Naranjo classification level, based on cumulative scores across all ten questions (Q1–Q10), MARCA and human expert assessments agreed for 79.4% (95% Confidence Interval, 95%CI 71.8–85.3) of the 1,562 DECs (unweighted Cohen’s κ\kappa = 0.587, 95%CI 0.450–0.690). Exact agreement in cumulative numerical scores was 71.6%. The mean total scores were 0.48 for MARCA and 0.74 for human experts, corresponding to a mean signed difference of −0.262. The distribution of MARCA classifications according to the expert-assigned category is presented in Table 3. Agreement was highest for Doubtful: 765 of 867 expert-assessed Doubtful DECs were also classified as Doubtful by MARCA (88.2%); 100 were classified as Possible and two as Probable. For the Possible category, 470 of 685 DECs were classified concordantly (68.6%), while 195 were classified as Doubtful and 20 as Probable by MARCA. For the Probable category, six of ten DECs were classified concordantly (60.0%), while four were classified as Possible. No DECs were classified as Definite by either approach.

Naranjo categoryExpert-assigned (n)Correctly classified by MARCA (n)Agreement (%)Most common misclassification
Doubtful86776588.2%Possible (n = 100, 11.5%)
Possible68547068.6%Doubtful (n = 195, 28.5%)
Probable10660.0%Possible (n = 4, 40.0%)
Definite0N/AN/AN/A

Table 3. Table 3. Naranjo Classification Agreement Between MARCA and Human Expert Assessors by Classification Category. Agreement between MARCA and human expert assessors across Naranjo classification categories based on the complete Q1–Q10 Naranjo scoring framework. Agreement was calculated as the proportion of expert-assigned DECs within each category that were correctly classified by MARCA. The most common direction of misclassification is reported for each category. Legend: N/A = not applicable.

Effect of Selective Retrieval-Augmented Generation

Table 4 compares the stored non-RAG and RAG-labelled scores in the retrospective subset of 60 DECs for which at least one initial non-RAG score disagreed with the human expert reference standard on Questions 2, 5, or 10. For Question 10, percentage agreement increased from 18.3% in the non-RAG condition to 100.0% in the RAG-labelled condition, while Cohen’s κ\kappa increased from 0.007 to 1.000. Under non-RAG conditions, MARCA assigned a lower score than the human expert assessment in 49 of 60 DECs, with a mean signed score difference of −0.817. No disagreements remained in the RAG-labelled scores, and the mean signed difference was 0.000. For Question 5, percentage agreement increased from 65.0% to 83.3%, while Cohen’s κ\kappa increased from 0.016 to 0.571. The number of lower MARCA scores decreased from 20 to ten of 60 DECs, and the mean signed difference changed from −0.967 to −0.500. For Question 2, percentage agreement increased from 98.3% to 100.0%, with the initial disagreement resolved. Cohen’s κ\kappa was 0.000 for non-RAG scores and was non-estimable for the RAG-labelled scores because both sets of ratings contained only an invariant category. Because the ablation cohort was retrospectively selected on the basis of disagreement with the expert reference standard, these results describe changes in saved score agreement within this subgroup, rather than overall performance or prospectively validated RAG effectiveness. In addition, RAG-labelled predictions were exported from the comparison workbook rather than matched to independently verified original retrieval-run logs.

QuestionNon-RAG (%)RAG (%)κ\kappa (Non-RAG)κ\kappa (RAG)
Q2 - Temporal association98.33%100.00%0.000N/E
Q5 - Alternative causes65.00%83.33%0.0160.571
Q10 - Objective evidence18.33%100.00%0.0071.000

Table 4. Percentage Agreement and Cohen's κ\kappa between MARCA and Human expert assessors Under Non-RAG and RAG Conditions. Question-level evaluation of MARCA against human expert consensus under non-RAG and RAG conditions. Legend: κ\kappa = Cohen's Kappa; RAG =Retrieval-Augmented Generation.

Comparison with Previous Studies

Table 5 compares MARCA with three previous studies evaluating LLM-based causality assessment: Abate et al. [4], Heckmann et al. [5] [5], and Heckmann et al. [7] [7]. Differences in causality algorithms, units of analysis, and dataset composition limit direct comparison across studies; consequently, the comparisons are descriptive and do not establish superiority in a shared test population. Accordingly, comparisons were restricted to reported metrics considered sufficiently comparable. Abate et al. used the WHO causality assessment algorithm, so the question 2 and 5 values shown for that study are approximate conceptual analogues (WHO questions 2.5 and 1) rather than like-for-like Naranjo items, and no WHO analogue exists for question 10.

Among the studies applying the Naranjo algorithm, MARCA showed 96.8%, 85.4%, and 96.8% agreement for Questions 2, 5, and 10, respectively. The corresponding values reported by Heckmann et al. [7] were 63.6%, 74.1%, and 65.4%, yielding descriptive differences of 33.2, 11.3, and 31.4 percentage points. Findings from Abate et al., who used a different causality assessment framework, were presented as contextual evidence rather than directly comparable Naranjo question-level estimates.

At the final classification level, MARCA showed 79.4% agreement with expert assessment (Cohen’s κ=0.587\kappa= 0.587). The 71% agreement for GPT-3.5-turbo and 53% for Gemini 1.5 Pro reported by Abate et al. are median question-level values, not final-classification results, and are therefore not directly comparable. The best-performing biomedical LLM configuration evaluated by Heckmann et al. [5] achieved 63.3% agreement; the baseline configuration achieved 45.0% and the optimized configuration achieved 72.0% in Heckmann et al. [7]. MARCA was evaluated without inference-time temperature optimization. The mean MARCA–expert difference in the total Naranjo score was −0.262, indicating a lower average MARCA score than the expert assessment. In contrast, Heckmann et al. [7] reported a mean difference of +0.37.

Reporting checklists

TRIPOD-LLM [18] and CHART [19] reporting checklists are provided in Supplementary tables 6 and Supplementary Table 7.

SectionItemCHART checklist itemManuscript location / reporting assessment
Title and AbstractTitle and AbstractTitle and Abstract
Title1aState that the study is assessing one or more generative AI-driven chatbots for clinical evidence or health advice.pp. 1–2 (title; abstract)<br>R - Clinical causality assessment and agents
Abstract/ Summary1bApply a structured format, if applicable.p. 2 (abstract)<br>R - Structured abstract
IntroductionIntroductionIntroduction
Background2aState the scientific background, rationale, and healthcare context for evaluating the generative AI-driven chatbot(s), referencing relevant literature when applicable.p. 3 (§1)<br>R - Prior work and healthcare rationale
Background2bState the aims and research questions including the target audience, intervention, comparator(s), and outcome(s).pp. 2–3 (abstract; §1)<br>R - Aims, evaluation and comparator
MethodsMethodsMethods
Model Identifiers3aState the name and version identifier(s) of the generative AI model(s) and chatbot(s) under evaluation, as well as their date of release or last update.pp. 5–6, 9 (§§2.4, 2.8)<br>P - Model IDs given; release/last-update dates not stated
Model Identifiers3bState whether the generative AI model(s) and chatbot(s) are open-source or closed-source/proprietary.pp. 5–6 (§2.4)<br>P - Named commercial models; licensing/openness not explicit
Model Details4aState whether the generative AI model was a base model or a novel base model, tuned model, or fine-tuned model.pp. 5–6 (§2.4)<br>P - Uses named base models, but tuning status not explicit
Model Details4bIf a base model is used, cite its development in sufficient detail to identify the model.pp. 5–6 (§2.4)<br>P - Model names given; model-development sources not fully cited
Model Details4cIf a novel base model, tuned model, or fine-tuned model is used, describe the pre- and/or post-implementation/deployment data and parameters.NA - No novel base-model training or fine-tuning documented
Prompt Engineering5aDescribe the evolution of study prompt development.pp. 5, 7 (§§2.4, 2.5)<br>P - Prompt strategy/freeze, not evolution
Prompt Engineering5aiDescribe the sources of prompts.pp. 5–7 (§§2.4–2.5); Supp. Table 2<br>P - Task templates and input sources, not prompt-design provenance
Prompt Engineering5aiiState the number and characteristics of the individual(s) involved in prompt engineering.NR - Prompt engineers not identified
Prompt Engineering5aiiiProvide details of any patient and public involvement during prompt engineering.NR - Public involvement in prompt design not reported
Prompt Engineering5bProvide study prompts.pp. 5–7 (§§2.4–2.5); Supp. Table 2<br>P - Representative prompts only, not all prompts
Query Strategy6aState route of access to generative AI model.p. 9 (§2.8)<br>R - API access documented
Query Strategy6bState the date(s) and location(s) of queries for the generative AI-driven chatbot(s) including the day, month, and year as well as city and country.NR - Dates and locations of API queries not reported
Query Strategy6cDescribe whether prompts were input into separate chat session(s).NR - Separate-session design not documented
Query Strategy6dProvide all generative AI-driven chatbot output/responsespp. 9, 14 (§§2.8, 8)<br>P - Outputs mentioned in repository, full response set not established
Prompting7aDefine the ground truth or reference standard used to define successful generative AI-driven chatbot performance.p. 5 (§2.3)<br>R - Senior-reviewed consensus expert reference standard
Prompting7bDescribe the process undertaken for generative AI-driven chatbot performance evaluation.pp. 8–11 (§§2.6–2.7, 3.2–3.5) R - Methods and agreement results
Prompting7biState the number and characteristics of team members involved in performance evaluation.pp. 5–6 (§§2.3, 2.4.5) P - Assessor roles, limited personnel characteristics
Prompting7biiProvide details of any patients and public involvement during the evaluation process.NR - Public involvement in evaluation not reported
Prompting7biiiState whether evaluators were blinded to the identity of the generative AI-driven chatbot(s) under assessment.NR - Blinding of evaluators not reported
Sample Size8Report how the sample size was determined.p. 4 (§2.2) P - Selection by feasibility; formal sample size rationale not provided
Data Analysis9aDescribe statistical analysis methods, including any evaluation of reproducibility of generative AI-driven chatbot responses.pp. 8–9 (§2.7) P - Statistics described; response reproducibility not tested
Data Analysis9aiReport the measures used for performance evaluation.pp. 8–9 (§§2.6–2.7) R - Agreement and error metrics
ResultsResultsResults
10aReport the performance evaluation undertaken including the alignment between generative AI-driven chatbot output and ground truth or reference standard using quantitative or mixed methods approaches as applicable.pp. 10–11 (§§3.2–3.5); Tables 1–4 R - Agreement to gold standard
10bFor responses deviating from the ground truth or reference standard, state the nature of the difference(s).pp. 10–12 (§§3.3, 4.3); Table 2 R - Direction, distribution and category of errors
10cIndicate where the study protocol can be accessed or state that a protocol was not prepared.NR - Study protocol availability not stated
DiscussionDiscussionDiscussion
11aInterpret study findings in the context of relevant evidence.pp. 11–12 (§§4.1–4.3) R - Interpretation in prior evidence context
11bDescribe the strengths and limitations of the study.p. 13 (§4.4) R - Strengths and limitations
11cDescribe the potential implications for practice, education, policy, regulation, and research.p. 13 (§§4.5–4.6) R - Implications and future work
Open ScienceOpen ScienceOpen Science
Disclosures12aReport any relevant conflicts of interest for all authors.p. 14 (§5) R - Conflicts statement
Funding12bReport sources of funding and their role in the conduct and reporting of the study.p. 14 (§7) R - Funding statement
Ethics12cDescribe the process undertaken for ethical approval.NR - No ethics approval/waiver statement
Ethics12ciDescribe the measures taken to safeguard data privacy of patient health information, as applicable.NR - Data privacy safeguards not described
Ethics12ciiState whether permission/licensing was obtained for the use of original, copyrighted data.NR - Licensing/permission statement absent
Protocol12dProvide a study protocol.NR - No study protocol linked or declared absent
Data availability12eState where study data, code repository, and model parameters can be accessed.pp. 9, 14 (§§2.8, 8) P - Data/code repository cited; model parameter access not covered

Supplementary Table 7. CHART Checklist

Discussion

MARCA met the predefined percentage-agreement thresholds for the primary Naranjo questions evaluated, but the interpretation of high agreement requires consideration of imbalanced score distributions. In particular, perfect agreement for Questions 4 and 6–9 reflected invariant zero scores rather than discrimination between different clinical assessments. The modular architecture provided separately reviewable evidence summaries and rationales, potentially supporting a tiered expert-review workflow. However, the recorded MARCA scores were compared retrospectively with expert consensus, and the results do not establish that the system can autonomously replace expert causality assessment.

Comparison with Previous Studies

MARCA’s high agreement on question 2 may partly reflect a shared default score, since therapy dates are almost never documented in FAERS narratives; the substantially lower Cohen’s κ\kappa (0.491) indicates that the high percentage agreement is influenced by the predominance of zero scores and should not be interpreted as strong discrimination across the less frequent nonzero categories. Heckmann et al. [7] showed the opposite pattern: their model inferred temporal plausibility from narrative context instead of defaulting conservatively, which produced lower percentage agreement (63.6%) and a near-chance κ\kappa (0.011) once base-rate imbalance is accounted for [5, 7]. On question 10, MARCA’s near-perfect κ\kappa substantially exceeds the κ\kappa reported by Heckmann et al. [5, 7]. On question 5, the limited chance-corrected agreement shared by MARCA and Heckmann et al. [7], despite the two systems using entirely different architectures, suggests this question is a structural difficulty for LLM-based Naranjo assessment of spontaneous reports in general, driven by the question’s high base-rate skew rather than a weakness specific to either system.

Interpretation of Remaining Disagreement

The remaining disagreements were concentrated in Question 5, which exhibited the lowest agreement between MARCA and human expert assessments. MARCA assigned lower scores more frequently for this question (205 DECs versus 23 with higher scores), resulting in a mean signed difference of −0.350. This pattern indicates a more conservative tendency in scoring alternative causes but does not establish whether this tendency is clinically more accurate. Using the complete Naranjo Q1–Q10 score, discordance in final classifications was concentrated largely at the Doubtful–Possible boundary: 195 expert-assessed Possible DECs were classified as Doubtful by MARCA, whereas 100 expert-assessed Doubtful DECs were classified as Possible. In the 60-DEC exploratory ablation subset, agreement was higher for the saved RAG-labelled scores on Questions 2, 5, and 10. Nevertheless, these analyses were performed in a disagreement-enriched subset, and the RAG-labelled scores could not be independently linked to original retrieval-inference logs. Their differences should not be interpreted as prospective proof of the effect of retrieval.

An additional consideration is the construction of the RAG knowledge base. The KB comprised the expert-assessed DECs used in the evaluation dataset, although the ICSR under evaluation was excluded from its own retrieval pool. Thus, other expert-assessed cases from the evaluation dataset remained available as references. The RAG findings should therefore be interpreted as retrieval-assisted reasoning within the study cohort rather than as performance evaluated against a completely independent knowledge base. Future studies using a strictly held out or externally developed retrieval corpus would provide a stronger test of the generalizability of the RAG component.

Strengths and Limitations

A principal strength of MARCA is its modular architecture, which provides question-specific, reviewable rationales and structured comparisons with human expert assessments. The observed agreement was high on the temporal association and sign-versus-symptom tasks, while evaluation of alternative causes remained more challenging. The improvements in the exploratory comparison of saved non-RAG and RAG-labelled scores require confirmation using independently documented retrieval runs. Several limitations qualify these findings. Spontaneous reports often lack treatment dates, de-challenge and rechallenge outcomes, drug concentrations, and prior reaction histories, limiting the information available for several Naranjo questions. The evaluation comprised 215 ICSRs and 1,562 DECs from one reporting system; individual ICSRs contributed multiple DECs, including one case contributing a substantial proportion of the records, which limits the independence of DEC-level observations. The retrospective RAG ablation involved 60 DECs selected because the non-RAG scores disagreed with expert assessments, and the retrieval knowledge base overlapped with the evaluation cohort, although the assessed ICSR was excluded from its own reference pool. The provenance of the saved RAG-labelled scores was not independently verified against original inference logs. Finally, the framework used one combination of foundation models (Gemini 2.5 Pro and DeepSeek-V3.2), so the contribution of the architecture cannot be separated from model choice. An additional limitation is the clustering of multiple drug–event combinations within individual ICSRs. Because each DEC was treated as a separate observation in the primary agreement analyses, individual reports contributing numerous combinations may have disproportionately influenced the overall estimates.

Future Directions

A continuously growing, human-in-the-loop KB is a natural next step, with newly emerging safety signals vetted by an expert before being added to the retrieval corpus, while maintaining expert review and clear separation between retrieval and evaluation data. Benchmarking against a larger panel of independent expert assessors, using majority scoring as the reference standard, would also reduce reliance on any single assessor’s judgement. Evaluating MARCA using data from other spontaneous reporting systems, including the UK's Yellow Card Scheme, the European Union's EudraVigilance, and the Japanese Adverse Drug Event Report (JADER) database, would help assess its generalizability across different pharmacovigilance settings. Furthermore, evaluating alternative foundation models within the same multi-agent architecture could help distinguish the contribution of task specialization and agent orchestration from that of the underlying model selection.

Conclusion

MARCA demonstrates the feasibility of organizing individual-level causality assessment into specialized, reviewable components supported by retrieval of scientific and expert-derived evidence. Agreement with expert assessments was high for temporal association and objective sign-versus-symptom classification, while the evaluation of alternative causes remained the principal source of disagreement. The retrospective ablation suggests potential benefits of retrieval, but independent confirmation of retrieval-run provenance and prospective validation are needed. MARCA should therefore be considered a decision-support framework that complements, rather than replaces, expert clinical judgment.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Author Contributions

DRL: Writing - original draft, Formal analysis, Validation and Visualization. AB: Writing - review and editing, Data curation, Validation, and Visualization. MS: Conceptualization, Formal analysis, Project administration, and Writing – review and editing. All authors contributed to the work, critically reviewed the manuscript, approved the final version, and agree to be accountable for the content of the work.

Funding & ethics

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. This study analysed publicly available, de-identified data from the FDA Adverse Event Reporting System. No patients were contacted, and ethical approval and informed consent were not required.

Acknowledgements

Generative AI was used to support the conceptual design, generation, and refinement of schematic figures included in this manuscript. Figures were developed using ChatGPT (OpenAI, GPT-6 for text-based assistance; accessed 23rd October 2026) and its integrated image-generation functionality. The specific version of the underlying image-generation model was not available in the generation records.

AI assistance included the development of visual layouts, graphical elements, and scientific annotations. All AI-generated figures were critically reviewed, revised, and validated by the authors against the study methodology, source data, and computational framework. The authors take full responsibility for the accuracy, scientific integrity, and final content of all figures.

Data Availability Statement

The datasets generated for this study can be found in the MARCA: Multi-Agent Retrieval-Augmented for Individual-level Causality Assessment in Pharmacovigilance https://github.com/daisyraelopez/Multi-Agent-Retrieval-Augmented-for-Individual-level-Causality-Assessment-MARCA-in-Pharmacovigilance/tree/main.

References

Tables

MetricAbate et al. [4] [4]Heckmann et al. [5] [5]Heckmann et al. [7] [7]MARCA (present study)
ModelGPT-3.5-turbo / Gemini 1.5 ProMedicine LLaMA-3 8BGPT-5.2Gemini 2.5 Pro + DeepSeek-V3.2
AlgorithmWHONaranjoNaranjoNaranjo
Sample150 VAERS cases150 cases (140 FAERS + 10 VAERS)723 FAERS cases215 FAERS cases (1,562 DECs)
Q2 agreement66.0% / 68.0%*54.0%63.6%96.8%
Q5 agreement54.7% / 30.0%*52.7%74.1%85.4%
Q10 agreementND63.3%65.4%96.8%
κ\kappa (Q2 / Q5 / Q10)NRNR0.011 / 0.180 / 0.2200.491 / 0.269 / 0.922
Final classification agreementNC — median question-level agreement: 71% (GPT-3.5) / 53% (Gemini)63.3% (Gwet’s AC1 = 0.644)45.0% baseline → 72.0% optimized79.4% (κ\kappa = 0.587)
Mean score error (LLM − expert)NRNR+0.37−0.262

Table 5. Cross-Study Comparison of Agreement Metrics for Automated and Human Causality Assessment. Comparison of key agreement metrics across studies evaluating causality assessment. The Closest conceptual WHO analogues are Q2 (WHO Question 2.5) and Q5 (WHO Question 1); no direct analogue exists for Q10. Legend: ND = not directly comparable; NR = not reported by the original authors. AC1 = Gwet’s first-order agreement coefficient, not directly comparable to Cohen’s κ\kappa. Abate et al. final-classification cells are not comparable because the reported percentages represent median question-level agreement.

Supplementary Material

† Equal contribution to first authorship

  • Correspondence:

Angela Boccia

Department of Drug Design and Pharmacology

University of Copenhagen

Universitetsparken 2, 2100 Copenhagen, Denmark

Email: [email protected]

9>60%Previous exposure is rarely documented and may only appear in free text.ICSRs are not linked across individuals; information depends on reporter history, and similar is drug-class dependent.
10>60%Objective confirmation may only be documented in free text.The definition of confirmation may vary between assessors.

Table 6.

SectionItemReporting guideline itemDesignLLM taskManuscript location / reporting assessment
Title1Identify the study as developing, fine-tuning and/or evaluating the performance of an LLM, specifying the task, the target population and the outcome to be predicted.AllAllp. 1 (title); p. 2 (abstract) R - Study identification and target task
Abstract2See TRIPOD-LLM for abstracts.AllAllp. 2 (abstract) R - Structured abstract
IntroductionIntroductionIntroductionIntroductionIntroduction
Background3aExplain the healthcare context/use case (for example, administrative, diagnostic, therapeutic and clinical workflow) and rationale for developing or evaluating the LLM, including references to existing approaches and models.AllAllp. 3 (§1) R - Clinical context and rationale
Background3bDescribe the target population and the intended use of the LLM in the context of the care pathway, including its intended users in current gold standard practices (for example, healthcare professionals, patients, public or administrators).E HAllpp. 3, 6, 13 (§§1, 2.4.5, 4.6) P - Intended users not explicitly characterized
Objectives4Specify the study objectives, including whether the study describes the initial development, fine-tuning or validation of an LLM (or multiple stages).AllAllpp. 2–3 (abstract; §1) R - Study aims
MethodsMethodsMethodsMethodsMethods
Data5aDescribe the sources of data separately for the training, tuning and/or evaluation datasets and the rationale for using these data (for example, web corpora, clinical research/trial data, EHR data or unknown).AllAllpp. 4, 7 (§§2.2, 2.5) P - Evaluation and knowledge-base data described; development split not established
Data5bDescribe the relevant data points and provide a quantitative and qualitative description of their distribution and other relevant descriptors of the dataset (for example, source, languages and countries of origin).AllAllp. 4, 9 (§§2.2, 3.1); Supp. Tables 4–5 R - Sample characteristics and variables
Data5cSpecifically state the date of the oldest and newest item of text used in the development process (training, fine-tuning and reward modeling) and the evaluation datasets.AllAllp. 4 (§2.2) P - Reporting period given; earliest/latest individual record dates not stated
Data5dDescribe any data preprocessing and quality checking, including whether this was similar across text corpora, institutions and relevant sociodemographic groups.AllAllp. 4 (§2.2) R - Preprocessing and terminology mapping
Data5eDescribe how missing and imbalanced data were handled and provide reasons for omitting any data.AllAllpp. 4, 7, 12–13 (§§2.2, 2.5, 4.4) P - Missingness discussed, but complete missing-data handling not defined
Analytical Methods6aReport the LLM name, version and last date of training.AllAllpp. 5–6, 9 (§§2.4, 2.8) P - Model versions named; last training/release dates absent
Analytical Methods6bReport details of the LLM development process, such as LLM architecture, training, fine-tuning procedures and alignment strategy (for example, reinforcement learning and direct preference optimization) and alignment goals (for example, helpfulness, honesty and harmlessness).M DAllpp. 5–6 (§2.4) P - Agent architecture described; no novel LLM training or fine-tuning documented
Analytical Methods6cReport details of how the text was generated using the LLM, including any prompt engineering (including consistency of outputs), and inference settings (for example, seed, temperature, max token length and penalties), as relevant.M D EAllpp. 5–9 (§§2.4–2.5, 2.8); Supp. Table 2 P - Prompts/settings summarized, not full inference metadata
Analytical Methods6dSpecify the initial and postprocessed output of the LLM (for example, probabilities, classification and unstructured text).AllAllpp. 5–7 (§§2.4.1–2.5) R - Agent outputs, human scoring and RAG scores
Analytical Methods6eProvide details and rationale for any classification and, if applicable, how the probabilities were determined and thresholds identified.AllC OFpp. 6, 8 (§§2.4.5, 2.6) R - R anjo rules and classification thresholds
LLM Output7aInclude metrics that capture the quality of generative outputs, such as consistency, relevance, accuracy and presence/type of errors compared to gold standards.AllQA IR DG SS MTpp. 10–11 (§§3.2–3.5); Tables 1–4 P - Agreement evaluated; output rationale quality not independently assessed
LLM Output7bReport the outcome metrics’ relevance to the downstream task at deployment time and, where applicable, the correlation of metric to human evaluation of the text for the intended use.E HAllpp. 8, 13 (§§2.6, 4.6) P - Clinical relevance addressed; no deployment outcomes
LLM Output7cClearly define the outcome, how the LLM predictions were calculated (for example, formula, code, object and API), the date of inference for closed-source LLMs and evaluation metrics.E HAllpp. 6–9 (§§2.4.5–2.7) P - Metrics/score calculation described; closed-model inference date not reported
LLM Output7dIf outcome assessment requires subjective interpretation, describe the qualifications of the assessors, any instructions provided, relevant information on demographics of the assessors and inter-assessor agreement.AllAllp. 5 (§2.3); p. 6 (§2.4.5) P - Assessor roles reported; experience and initial inter-rater agreement absent
LLM Output7eSpecify how performance was compared to other LLMs, humans and other benchmarks or standards.AllAllpp. 8–9, 11 (§§2.7, 3.6); Table 5 R - Cross-study benchmarking approach
Annotation8aIf annotation was done, report how the text was labeled, including providing specific annotation guidelines with examples.AllAllp. 5 (§2.3); Appendix 1 P - Structured scoring described; item-specific annotation details in Appendix
Annotation8bIf annotation was done, report how many annotators labeled the dataset(s), including the proportion of data in each dataset that was annotated by more than one annotator, and the inter-annotator agreement.AllAllp. 5 (§2.3) P - Junior/senior review described; independent dual-annotation proportion absent
Annotation8cIf annotation was done, provide information on the background and experience of the annotators or the characteristics of any models involved in labeling.AllAllp. 5 (§2.3) P - Assessor positions given, not detailed experience
Prompting9aIf research involved prompting LLMs, provide details on the processes used during prompt design, curation and selection.AllAllpp. 5–7 (§§2.4–2.5) P - Prompt choice and freeze reported; development/evolution limited
9bIf research involved prompting LLMs, report what data were used to develop the prompts.AllAllpp. 5–7 (§§2.4–2.5); Supp. Table S2 P - Prompt inputs given; prompt-development dataset not separately described
Summariza-tion10Describe any preprocessing of the data before summarization.AllSSpp. 5, 9 (§§2.4.1, 2.8) R - Abstract retrieval and text truncation
Instruction tuning/alignment11If instruction tuning/alignment strategies were used, what were the instructions, data and interface used for evaluation, and what were the characteristics of the populations doing the evaluation?M DAllNA - No instruction fine-tuning reported
Compute12Report compute, or proxies thereof (for example, time on what and how many machines, cost on what and how many machines, inference time, floating-point operations per second), required to carry out methods.M D EAllp. 9 (§2.8) P - Constraints mentioned; no compute time, hardware or cost
Ethical approval13Name the institutional research board or ethics committee that approved the study and describe the participant-informed consent or the ethics committee waiver of informed consent.AllAllNR - No ethics committee approval or waiver statement
Open science14aGive the source of funding and the role of the funders for the present study.AllAllp. 14 (§7) R - No specific funding stated
Open science14bDeclare any conflicts of interest and financial disclosures for all authors.AllAllp. 14 (§5) R - Conflict-of-interest statement
Open science14cIndicate where the study protocol can be accessed or state that a protocol was not prepared.HAllNR - No protocol availability or protocol declaration
Open science14dProvide registration information for the study, including register name and registration number, or state that the study was not registered.HAllNR - No registration status statement
Open science14eProvide details of the availability of the study data.AllAllp. 14 (§8) R - Repository given for study datasets
Open science14fProvide details of the availability of the code to reproduce the study results.AllAllpp. 9, 14 (§§2.8, 8) R - Repository given for analysis code
Public involvement15Provide details of any patient and public involvement during the design, conduct, reporting, interpretation or dissemination of the study or state no involvement.HAllNR - Patient/public involvement not addressed

Table Supplementary Table 6. Supplementary Table 6. TRIPOD-LLM Checklist. Legend: M = LLM methods; D = de novo LLM development; E = LLM evaluation; H = LLM evaluation in healthcare settings; C = classification; OF = outcome forecasting; QA = long-form question answering; IR = information retrieval; DG = document generation; SS = summarization and simplification; MT = machine translation; API = application programming interface.

ResultsResultsResultsResultsResults
Participants16aWhen using patient/EHR data, describe the flow of text/EHR/patient data through the study, including the number of documents/questions/participants with and without the outcome/label and follow-up time as applicable.E HAllpp. 4, 9 (§§2.2, 3.1); Supp. Fig. 2 P - Main counts reported; complete selection flow not fully described
Participants16bWhen using patient/EHR data, report the characteristics overall and for each data source or setting and development/evaluation splits, including the key dates, key characteristics and sample size.E HAllpp. 4, 9 (§§2.2, 3.1); Supp. Tables 4–5 R - Sample and case-level characteristics
Participants16cFor LLM evaluation that includes clinical outcomes, show a comparison of the distribution of important clinical variables that may be associated with the outcome between development and evaluation data, if available.E HAllp. 9 (§3.1); Supp. Table 4 P - Descriptive comparison, no development/evaluation split
Participants16dWhen using patient/EHR data, specify the number of participants and outcome events in each analysis (for example, for LLM development, hyperparameter tuning and LLM evaluation).E HAllpp. 9–11 (§§3.1, 3.5) R - 215 ICSRs; 1,562 DECs; 60-DEC subset
Performance17Report LLM performance according to prespecified metrics (see item 7a) and/or human evaluation (see item 7d).AllAllpp. 10–11 (§§3.2–3.5); Tables 1–4 R - Primary, secondary, exploratory outcomes
LLM updating18If applicable, report the results from any LLM updating, including the updated LLM and subsequent performance.AllAllNA - No post-training model update evaluated
DiscussionDiscussionDiscussionDiscussionDiscussion
Interpretation19aGive an overall interpretation of the main results, including issues of fairness in the context of the objectives and previous studies.AllAllpp. 11–12 (§§4.1–4.3) P - Results interpreted; fairness not formally analyzed
Limitations19bDiscuss any limitations of the study and their effects on any biases, statistical uncertainty and generalizability.AllAllp. 13 (§4.4) R - Limitations described
Usability of the LLM in context19cDescribe any known challenges in using data for the specified task and domain context with reference to representation, missingness, harmonization and bias.E HAllpp. 12–13 (§§4.2–4.4) R - Missing data, data-source limitations and clustering
Usability of the LLM in context19dDefine the intended use for the implementation under evaluation, including the intended input, end-user and level of autonomy/human oversight.E HAllpp. 6, 13 (§§2.4.5, 4.6) P - Human oversight described; intended users not fully specified
Usability of the LLM in context19eIf applicable, describe how poor quality or unavailable input data should be assessed and handled when implementing the LLM; that is, what is the usability of the LLM in the context of current clinical care.E HAllpp. 12–13 (§§4.2, 4.4) P - Incomplete reporting discussed, not an operational deployment procedure
Usability of the LLM in context19fIf applicable, specify whether users will be required to interact in the handling of the input data or use of the LLM, and what level of expertise is required of users.E HAllpp. 6, 13 (§§2.4.5, 4.5–4.6) P - Human intervention described; required expertise not specified
Usability of the LLM in context19gDiscuss any next steps for future research, with a specific view of the applicability and generalizability of the LLM.AllAllp. 13 (§4.5) R - Next steps and external evaluation

Table 8.

AGENT 2 | Drug Label Assessment

Label listedness (Q1) + unlisted-event plausibility

01 INPUT

  • DEC: suspected drug + reported MedDRA AE term

  • Drug-label reference: product-specific CSV listing AEs extracted from a labeling PDF.

  • Example input: ICATIBANT / Epistaxis

  • If no CSV is available, output No label list available.

02 PROCESSING

  • Offline: extract label PDF text, identify safety section, extract unique AE terms into CSV.

  • Step 1: lowercase exact string match of AE against list.

  • Step 2: only if unmatched, ask LLM about clinical relatedness (Yes/No/Unclear).

  • No unlisted AE is relabeled as formally listed based on plausibility alone.

03 OUTPUT

  • Dictionary fields: drug, ae, status, explanation.

  • Statuses: Listed in label; Not listed (Yes/No/Unclear); No label list available; or error.

  • Clinical explanation populated for the unlisted LLM branch.

  • Label listedness and clinical plausibility remain distinct outputs.

CODE-LEVEL PROMPT / OUTPUT TEMPLATE

Adapted code excerpts; whitespace and some wording compressed for layout.

Determine if the following adverse event is medically related to the drug based on clinical
  reasoning.

Drug: {drug}
Adverse Event: {ae}
Output in this format:
Related - start: <Yes/No/Unclear> end.
Explanation - start: <Brief medical explanation> end.

SCHEMATIC INPUT / OUTPUT EXAMPLE

OUTPUT DICTIONARY (schema only)
drug: "ICATIBANT"
ae: "Epistaxis"
status: one of
  - Listed in label
  - Not listed (Yes/No/Unclear)
  - No label list available
  - Error during LLM call
explanation: [text or empty]
Agent 3 - Objective Evidence Assessment

Figure Supplementary Figure 6. Agent 3 - Objective Evidence Assessment. Structured LLM-based classification of reported adverse events as signs or symptoms, with corresponding numerical values and medical rationales to support Naranjo Question 10. Disclaimer: The figure is AI-generated.

Agent 4 - Clinical Causality Assessment

Figure S7. Agent 4 - Clinical Causality Assessment. Integration of structured clinical case information and Agent 1's literature synthesis to generate standardized causality reports addressing temporal and biological plausibility, dechallenge and rechallenge, objective evidence, confounding factors, and overall clinical plausibility. Disclaimer: The figure is AI-generated.

Standard Operating Procedure

Document IDMARCA-PV-CA-001
Status / versionApproved v0.1

Table 9.

Purpose and applicability

To standardize manual, question-level causality assessments and the creation of an adjudicated human reference standard for the MARCA research study. The unit of assessment is one specific suspected drug-adverse event combination (DEC), not the whole ICSR. The MARCA repository describes 215 FAERS ICSRs yielding 1,562 DECs in the human-reference set. This procedure applies to retrospective evaluation of those structured case narratives; no direct patient contact or intervention is permitted.

Personnel and responsibilities

  • Junior assessor: reviews the structured narrative and permitted supporting sources, completes Q1-Q10 independently, records sources and clinical reasoning, and signs/dates the first-pass assessment.

  • Senior pharmacovigilance expert: reviews every first-pass assessment and the available evidence, documents changes or unresolved ambiguities, and participates in case-by-case reconciliation.

  • Study custodian (Senior pharmacovigilance expert): maintains version-controlled narratives, controls access to model predictions, checks completeness, freezes approved consensus records, and retains the audit trail.

Case materials and evidence standards

Assessors must use the same version of the structured FAERS narrative, including available demographics, suspected and concomitant medications, drug indications, dose and exposure dates, AE terms, onset/treatment timelines, clinical outcomes, and reported tests. The PRIMARYID identifies a specific FAERS report version; if multiple versions of an underlying case are present, the version-selection policy must be documented before review.

Any external FDA label or biomedical publication consulted for Q1 or mechanistic context must be logged with its bibliographic identifier/URL, version or publication date, and access date. External information may support general drug-event plausibility but cannot substitute for missing patient-specific dates, negative rechallenge evidence, laboratory findings, or alternative diagnoses. MARCA predictions must remain hidden until human consensus has been finalized.

Assessment workflow

  1. Verify the ICSR identifier, version, suspected drug name, and MedDRA Preferred Term for the AE. Create a separate entry for each DEC, including multiple AEs for one drug or multiple suspected drugs for one AE.

  2. Read the full narrative once before scoring; extract positive findings, negatives explicitly reported, chronology, concomitant treatments, comorbidities, and objective investigations.

  3. For each of the 10 Naranjo questions, choose Yes, No, or Do not know; assign the scale-defined number; enter one brief clinical rationale and the supporting narrative excerpt or source reference.

  1. Use Do not know when information is absent, contradictory, or insufficient. Do not treat an unreported action (e.g., no documented rechallenge or placebo administration) as an observed negative result.

  1. Sum the 10 numerical scores and derive the final category using the official cutoffs. Record any uncertainty; do not modify scores to fit a desired final category.

  1. Submit the first-pass record to the senior expert, reconcile each disputed response, and freeze the signed consensus assessment before comparing it with model predictions.

Naranjo scoring matrix

Score each item for the specific DEC. “Do not know” always scores 0 but must remain distinct from an observed “No” in the extraction form.

ItemNaranjo question (condensed)YesNoUnknown
Q1Prior conclusive evidence for the drug–AE association+1+10000
Q2AE began after suspected drug administration+2+2−1-100
Q3AE improved after withdrawal or a specific antagonist+1+10000
Q4AE recurred after readministration (rechallenge)+2+2−1-100
Q5Other causes could independently explain the AE−1-1+2+200
Q6AE recurred following placebo administration−1-1+1+100
Q7Toxic drug concentration documented in blood/fluids+1+10000
MARCASOP
Q8AE severity varied with drug dose+100
Q9Similar AE on previous exposure to same/similar drug+100
Q10AE confirmed by objective evidence+100

Table 5 (PDF p. 44).

Item-specific decision rules

Q1 - Prior reports. Use a documented source indicating a recognized drug–AE association, preferably regulatory product information or credible scientific evidence. MARCA uses FDA-label assessment for Q1; specify explicitly whether label listedness is accepted as the study’s operational proxy for “previous conclusive reports.” Not finding an AE in a searched label is not, by itself, proof that no conclusive reports exist.

Q2 - Temporal sequence. Assign Yes (+2) only if the available timeline supports AE onset after exposure; consider plausible latency when interpreting the sequence. Assign No (−1) only if the AE clearly predates first relevant exposure. If drug start/onset timing is unknown, contradictory, or too imprecise to establish order, assign Do not know (0); never infer dates from general pharmacology alone.

Q3 - Dechallenge. Assign Yes (+1) if improvement after documented drug discontinuation or specific antagonist is reported. If the outcome after withdrawal is explicitly not improved, use No (0). If withdrawal or post-withdrawal course is absent or ambiguous, use Do not know (0).

Q4 - Rechallenge. Use Yes (+2) only with documented re-exposure and recurrence; No (−1) requires documented re-exposure without recurrence. If no rechallenge is recorded, use Do not know (0). No rechallenge should be undertaken for research purposes.

Q5 - Alternative causes. Review concomitant medications, underlying diseases, presenting illness, infections, procedures, and other clinically plausible etiologies. Yes (−1): at least one documented alternative could independently cause the AE. No (+2): available case-specific evidence adequately supports the absence or exclusion of credible alternatives. Do not know (0): information is insufficient to determine whether alternatives exist. Absence of reported comorbidities or concomitant drugs is NOT sufficient by itself to assign No (+2). Log the competing explanation and supporting facts.

Q6 - Placebo. Yes (−1) or No (+1) requires a reported placebo challenge and its observed result. Where there is no placebo exposure, use Do not know (0).

Q7 - Toxic concentration. Yes (+1) requires a documented drug concentration in a fluid together with a known toxic range. If testing confirms nontoxic concentrations, use No (0); if no relevant measurement is reported, use Do not know (0).

Q8 - Dose response. Yes (+1) requires documented dose escalation/worsening or reduction/improvement with relevant chronology. If tested dose changes showed no relationship, use No (0); otherwise Do not know (0).

Q9 - Prior similar reaction. Yes (+1) requires a documented similar reaction to the same or pharmacologically related drug in a previous exposure. If explicit prior exposure without such reaction is reported, use No (0); otherwise Do not know (0).

Q10 - Objective confirmation. Yes (+1) requires documented corroboration of the AE by an examination finding, laboratory result, imaging study, pathology, instrument-measured parameter, or similar verifiable observation. A reported MedDRA PT, diagnostic label, or subjective symptom alone does not establish objective confirmation. Use No (0) for explicitly absent objective corroboration after evaluation and Do not know (0) if no objective investigation or finding is described. Name the evidence in the rationale.

Overall classification

Calculate the total as the arithmetic sum of Q1–Q10. Classify each DEC as definite (≥9\ge9), probable (5–8), possible (1–4), or doubtful (≤0\le0). Preserve question-level responses, scores, rationale, and total; the same ICSR may yield different classes for different DECs.

Senior review and reconciliation

  • The senior expert verifies the case/drug/event pairing, evidence provenance, response category, score mapping, and arithmetic for every DEC.

  • Any proposed correction is logged with question number, junior response/score, senior proposed response/score, specific supporting evidence, and final resolution. The junior and senior discuss discrepant items and document the agreed final score and rationale.

  • If disagreements cannot be resolved, refer the DEC to a predefined escalation process or mark it unresolved; do not silently impose consensus. The actual escalation pathway and whether unresolved DECs were included must be confirmed from study records.

  • Retain initial assessments separately from final consensus and record reviewer identity/date. Report initial inter-rater agreement only if genuinely independent dual assessments were collected; post-reconciliation consensus is not the same metric.

Data quality, independence, and audit trail

  • Run validation checks for all 10 responses and allowable numerical scores, nonmissing rationales, correctly summed totals, and class thresholds; validate uniqueness at the DEC identifier level.

  • Lock the input narrative version and document source cutoffs and label retrieval dates. No new information may be added to the human assessment from MARCA outputs.

  • Keep the evaluation gold standard separate from any expert-annotated knowledge base exposed to MARCA during evaluation. Identify and document any overlapping cases, labels, or records before reporting an unbiased held-out comparison.

  • Freeze final consensus records and preserve the audit log and source identifiers according to the study’s approved data-governance plan.

Structured assessment form (per DEC)

A standardized assessment dataset was used to document human expert causality assessments and compare them with MARCA-generated outputs. Each row represented an individual DEC, uniquely identified by a record identifier (id) and linked to the corresponding FAERS case (case_id).

Case identification and clinical information

FieldDescription
idUnique assessment record identifier
case_idIdentifier linking the DEC to the corresponding FAERS case
drug_nameSuspected drug associated with the assessed event
ptAdverse event coded as a MedDRA Preferred Term
case_narrativeStructured clinical narrative containing the available patient, drug exposure, adverse event, and clinical timeline information (see supplementary figure 3)
decSupplementary DEC-related metadata recorded in the dataset

Table A.1.

Question-level causality assessment

For each DEC, all ten Naranjo questions were evaluated individually. The human expert assessment consisted of a numerical score and a supporting clinical rationale for each question. MARCA-generated scores and rationales were stored in corresponding fields, together with question-level agreement indicators.

Naranjo questionHuman expert scoreHuman clinical rationaleMARCA scoreMARCA rationaleAgreement
Q1q1_score_humanq1_reasoning_humanq1_score_aiq1_reasoning_aiq1_agreement
Q2q2_score_humanq2_reasoning_humanq2_score_aiq2_reasoning_aiq2_agreement
Q3q3_score_humanq3_reasoning_humanq3_score_aiq3_reasoning_aiq3_agreement
Q4q4_score_humanq4_reasoning_humanq4_score_aiq4_reasoning_aiq4_agreement
Q5q5_score_humanq5_reasoning_humanq5_score_aiq5_reasoning_aiq5_agreement
Q6q6_score_humanq6_reasoning_humanq6_score_aiq6_reasoning_aiq6_agreement
Q7q7_score_humanq7_reasoning_humanq7_score_aiq7_reasoning_aiq7_agreement
Q8q8_score_humanq8_reasoning_humanq8_score_aiq8_reasoning_aiq8_agreement
Q9q9_score_humanq9_reasoning_humanq9_score_aiq9_reasoning_aiq9_agreement
Q10q10_score_humanq10_reasoning_humanq10_score_aiq10_reasoning_aiq10_agreement

Table A.2.

Each human expert rationale documented the evidence supporting the assigned score, including relevant information from the clinical narrative or external reference sources when applicable.

Final causality scores and agreement

FieldDescription
score_humanTotal Naranjo score assigned by the human expert assessment
score_aiTotal Naranjo score generated by MARCA
score_agreementAgreement indicator between the final human and MARCA numerical scores

Table 10.

Final causality categories were determined from the total Naranjo scores according to the standard classification thresholds: Definite (≥9), Probable (5–8), Possible (1–4), and Doubtful (≤0).

Supplementary MARCA-generated evidence

The dataset also retained additional MARCA-generated outputs to support evaluation of the framework:

FieldDescription
q1_ai_answerDrug label assessment output supporting Q1
literature_ai_answerScientific literature retrieval and summarization output
moa_ai_answerMechanism-of-action and pharmacological evidence summary
questions2_9_ai_answerClinical causality assessment output covering Q2–Q9

Table 11.

These fields provided supplementary information on the evidence retrieved and reasoning generated by the framework.

Human expert assessment and reconciliation

The human expert assessments were conducted before comparison with MARCA-generated results. Following the initial assessment, the assigned Naranjo scores and supporting rationales were reviewed by a senior pharmacovigilance expert and reconciled to establish the final reference standard.

The reconciled human expert scores and clinical rationales were represented by the q1_score_human–q10_score_human and q1_reasoning_human–q10_reasoning_human fields, respectively.

MARCA-generated results were subsequently compared with the finalized human expert assessments at both the individual-question and total-score levels. Agreement was recorded using the corresponding question-level and overall agreement fields.

Pre-finalization checks

  • ☐ One completed assessment per intended DEC, correct pairings and narrative version.

  • ☐ Complete Q1–Q10 responses, permitted scores, and concise source-based rationales.

  • ☐ No unreported rechallenge/placebo or unmeasured drug level incorrectly scored as an observed No.

  • ☐ Chronology, alternative etiologies, and objective evidence explicitly considered.

  • ☐ Arithmetic, thresholds, and classification verified.

  • ☐ All senior corrections and final reconciliations traceable.

  • ☐ Reference standard frozen prior to model evaluation, and overlap with the RAG knowledge base audited.

References and provenance

MARCA public repository (README): https://github.com/daisyraelopez/Multi-Agent-Retrieval-Augmented-for-Individual-level-Causality-Assessment-MARCA-in-Pharmacovigilance.

Naranjo CA et al. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther. 1981;30:239–245.

U.S. National Library of Medicine, LiverTox: Naranjo Adverse Drug Reaction Probability Scale and classification table. https://www.ncbi.nlm.nih.gov/books/NBK548069/

References

  1. [1]Food, U.S. and A. Drug, FDA Adverse Event Monitoring System (AEMS). 2026.
  2. [2]Barbieri, M.A., et al., Artificial intelligence in pharmacovigilance signal management: a review of tools, implementations, research, and regulatory landscape. Expert Opinion on Drug Safety, 2026. 25(2): p. 207–222.
  3. [3]Pilipiec, P., M. Liwicki, and A. Bota, Using machine learning for pharmacovigilance: a systematic review. Pharmaceutics, 2022. 14(2): p. 266.DOI
  4. [4]Abate, A., et al., Off-the-Shelf Large Language Models for Causality Assessment of Individual Case Safety Reports: A Proof-of-Concept with COVID-19 Vaccines: A. Abate et al. Drug Safety, 2025. 48(7): p. 805–820.
  5. [5]Heckmann, N.S., et al., Biomedical Large Language Models and Prompt Engineering for Causality Assessment of Individual Case Safety Reports in Pharmacovigilance. Pharmaceutical Research, 2026: p. 1–14.DOI
  6. [6]Kefala, M., et al., Development of a European Union Time-Indexed Reference Dataset for Assessing the Performance of Signal Detection Methods in Pharmacovigilance using a Large Language Model. arXiv preprint arXiv:2603.26544, 2026.arxiv.org/abs/2603.26544
  7. [7]Heckmann, N.S., et al., Optimizing Large Language Models for Causality Assessment in Pharmacovigilance: Developing a Performance Metric as Objective for Bayesian Hyperparameter Optimization. arXiv preprint arXiv:2607.03704, 2026.arxiv.org/abs/2607.03704
  8. [8]Moura, J., CrewAI: Framework for orchestrating role-playing autonomous AI agents. https://github.com/crewAIInc/crewAI, 2024.
  9. [9]Gao, Y., Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint arXiv:2312.10997, 2023.arxiv.org/abs/2312.10997
  10. [10]Kiguba, R., et al., Navigating duplication in pharmacovigilance databases: a scoping review. BMJ open, 2024. 14(4): p. e081990.DOI
  11. [11]Potter, E., et al., FDA adverse event reporting system (FAERS) essentials: a guide to understanding, applying, and interpreting adverse event data reported to FAERS. Clinical Pharmacology & Therapeutics, 2025. 118(3): p. 567–582.DOI
  12. [12]Khaleel, M.A., et al. A standardized dataset of a spontaneous adverse event reporting system. in Healthcare. 2022. MDPI.
  13. [13]Naranjo, C.A., et al., A method for estimating the probability of adverse drug reactions. Clinical Pharmacology & Therapeutics, 1981. 30(2): p. 239–245.pubmed.ncbi.nlm.nih.gov/7249508
  14. [14]Duan, Z. and J. Wang, Exploration of llm multi-agent application implementation based on langgraph+ crewai. arXiv preprint arXiv:2411.18241, 2024.arxiv.org/abs/2411.18241
  15. [15]Wei, J., et al., Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 2022. 35: p. 24824–24837.arxiv.org/abs/2201.11903
  16. [16]McHugh, M.L., Interrater reliability: the kappa statistic. Biochemia medica, 2012. 22(3): p. 276–282.pubmed.ncbi.nlm.nih.gov/23092060
  17. [17]Roaché, D., Intercoder reliability techniques: Percent agreement. The SAGE encyclopedia of communication research methods, 2017. 4: p. 252.DOI
  18. [18]Gallifant, J., et al., The TRIPOD-LLM reporting guideline for studies using large language models. Nature medicine, 2025. 31(1): p. 60–69.
  19. [19]Collaborative, C., et al., Reporting guideline for chatbot health advice studies: the CHART statement. JAMA network open, 2025. 8(8): p. e2530220.

Paper details

vlnc:W_43px5s68
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

ChatGPT

View statement
Looking for related papers…
Contents