Interactive tool helps fix errors in AI retrieval and generation systems
RECTIFY: An Interactive Workbench for Post-Evaluation RAG Diagnosis, Repair, and Verification
Software Engineering
Summary
AI systems that find and generate information often make mistakes like missing important facts or including wrong details. The authors created RECTIFY, an interactive tool that helps developers see which problems need fixing and suggests focused steps to repair them. RECTIFY lets developers test fixes in a safe space before applying them, making it easier to improve these AI systems. The tool works well across different types of search methods and helps prioritize repairs effectively.
What this means in practice
- •For ai system developers: Use RECTIFY to identify and prioritize specific fixes in retrieval-augmented generation systems by turning evaluation results into actionable repair tasks.
- •For software quality assurance teams: Apply RECTIFY to audit and verify fixes in information retrieval components through interactive testing before deployment, improving system reliability.
Authors
Keerthana Murugaraj, Salima Lamsiyah, Martin Theobald
Abstract
Retrieval-Augmented Generation (RAG) evaluators can identify failures such as weak retrieval, poor grounding, incomplete answers, and unsupported generation, but they rarely help developers decide what to repair next. We present RECTIFY, an interactive Streamlit workbench that turns evaluated RAG cases into auditable repair workflows. RECTIFY filters cases that do not require repair, routes remaining failures into actionable families and finegrained repair slices, and generates editable repair cards that developers can approve, reject, or verify through sandbox reruns. On a controlled RAG benchmark, RECTIFY surfaces interpretable failure profiles across BM25, dense, and hybrid retrieval: BM25 mainly triggers noisy-retrieval repairs, while dense and hybrid retrieval leave smaller sets of multi-part underretrieval and underused-evidence cases. Additional analyses show that pre-filtering reduces unnecessary repair candidates and that slicelevel routing yields more targeted repair cards than broad family-level diagnosis. RECTIFY is publicly available as an open-source Streamlit workbench 1 for helping developers turn evaluation results into inspectable repair decisions.