EvalRx enables diagnosis and verified repair of foundation models
EvalRx: Agentic Diagnosis and Verified Repair of Foundation Models
Artificial Intelligence
Summary
Foundation AI models often make mistakes, but it's hard to understand why or to fix them reliably. The authors present EvalRx, a tool that not only finds patterns in these mistakes but also tests ideas about their causes through rigorous checks. EvalRx then suggests and verifies improvements that actually enhance the model without causing new problems. This approach was tested on various language, vision-language, and audio-language models, showing consistent improvements in performance.
What this means in practice
- •For machine learning engineers: Diagnose specific failure patterns in large AI models and verify repairs to improve accuracy without causing new errors.
- •For product teams deploying ai: Implement a verified process for improving AI model behavior across language, vision, and audio modalities in deployed applications.
Authors
Jiaqi Liu, Ruinan Jin, Yitong Tang, Beidi Zhao, Gexin Huang, Qiong Zhang, Xiaoxiao Li
Abstract
Model evaluation reveals where models fail, but rarely why or whether a proposed fix genuinely works. We introduce EvalRx, an open-source agentic framework for statistically grounded model diagnosis and repair. EvalRx discovers systematic failure modes, explores candidate mechanisms, formulates falsifiable hypotheses, and verifies them by executing frozen, pre-specified tests before using supported diagnoses to guide tiered repair. Candidate repairs are validated against the unchanged model and promoted only when they improve the target behavior without unacceptable regression. Across language, vision-language, and audio-language settings spanning Gemini, Qwen, Gemma, and Nemotron, EvalRx improves performance in all 12 evaluated model family-modality combinations, with gains ranging from +1.4 to +10.9 percentage points (+5.8 points on average). These results show that model evaluation can serve not only to measure failures, but to support an auditable process for diagnosing and repairing them.