Calibrated repairs improve accuracy in financial question answering
When Self-Verification Hurts: Calibrated Answer Repair for Financial Question Answering
Computation and Language
Summary
When answering questions about financial data, models sometimes pick plausible numbers but assign them wrongly, like mixing up entities or units. The authors found that simply asking models to check and fix their answers can make mistakes worse or damage good parts. They created a method called Calibrated Answer Repair (CAR) that predicts when a fix is needed, targets specific checks, and limits changes to avoid harming correct content. This approach improves accuracy, recovers many missed details, and costs less effort than repeating revisions.
What this means in practice
- •For financial software developers: Improve accuracy of automated systems answering financial queries by applying calibrated repair to reduce assignment errors.
- •For financial data analysts: Use targeted answer repair techniques to validate and refine machine-generated financial answers before reporting or analysis.
Authors
Celine Guo, Weida Liang, Yujian Fu, Anqi Dang, Yanzheng Jin, Kenji Kawaguchi, Wenya Wang
Abstract
Financial question answering can fail even when a model extracts a plausible number from the supplied evidence: the answer may attach it to the wrong entity, period, unit, or accounting concept. Asking the model to verify and revise its draft can help, but open-ended revision may alter content that was already correct or inspect the wrong relation and reinforce the original error. We formulate repair as a decision that balances the information it may recover against the correct content it may damage. Based on this trade-off, we introduce Calibrated Answer Repair (CAR), which first estimates whether a draft is likely to benefit from repair, then selects a targeted check, localizes the relevant evidence or draft span, and limits the permitted edit. Across FinRank, Fin-RATE, and FinQA with three open-weight models, CAR outperforms generic verification in all dataset-model settings. At the system level, it recovers 14.6% of initially missed answer requirements while damaging 2.1% of initially correct ones. Its post-draft token cost is over 30 percent lower than the iterative revision alternatives. Ablations on two dataset-model settings show that evidence localization and bounded editing are the main contributors to CAR's gains.