Multi-agent system aids drug side effect causality assessment accuracy
Multi-Agent Retrieval-Augmented Causality Assessment (MARCA) of Individual Case Safety Reports in Pharmacovigilance
Computation and Language
Summary
Assessing whether a drug caused a side effect is complicated and time-consuming. The authors created MARCA, a system that uses four specialized agents to gather evidence from scientific literature, drug labels, and clinical data to help make this judgment. When compared with human experts, MARCA showed strong agreement on most questions but found some difficulty with identifying alternative causes. The system's retrieval features improved agreement in retrospective tests, but more study is needed to confirm future benefits.
What this means in practice
- •For pharmacovigilance teams: Assist with faster and evidence-supported assessment of drug-side effect causality using multi-agent evidence retrieval and scoring.
- •For regulatory safety reviewers: Improve consistency in evaluating individual case safety reports by integrating automated evidence gathering with expert judgment.
Authors
Daisy Rae Lopez, Angela Boccia, Maurizio Sessa
Abstract
Introduction: Increasing volumes of individual case safety reports (ICSRs) challenge expert-led causality assessment in pharmacovigilance. We developed MARCA (Multi-Agent Retrieval-Augmented Causality Assessment), a four-agent framework for Naranjo-based assessment, and evaluated agreement with human experts and exploratory differences between non-RAG and RAG-labelled scores. Methods: MARCA combined scientific literature retrieval, drug-label assessment, objective-evidence classification, and clinical causality reasoning using Gemini 2.5 Pro and DeepSeek-V3.2. Agent-generated rationales informed human-assisted scoring. A consensus expert reference standard covered 215 ICSRs and 1,562 drug-event combinations (DECs) from FAERS (Q1-Q2 2025). Agreement was measured using percentage agreement and Cohen's kappa. Results: Question-level agreement was 96.8% for Naranjo Q2 (kappa = 0.491; 95% CI 0.265-0.663), 85.4% for Q5 (kappa = 0.269; 95% CI 0.180-0.340), and 96.8% for Q10 (kappa = 0.922; 95% CI 0.885-0.951). Final causality classification agreement was 79.4% (95% CI 71.8-85.3; kappa = 0.587, 95% CI 0.450-0.690). In a retrospective subset of 60 DECs, agreement was higher for RAG-labelled than non-RAG scores for Q5 (83.3% vs 65.0%) and Q10 (100.0% vs 18.3%). Discussion: MARCA-assisted assessment showed substantial variation in agreement across Naranjo questions, with lower concordance for alternative causes. Retrospective RAG comparisons are exploratory and do not establish prospective effectiveness.