Post-training quantization preserves medical ai answers but not explanations
When Quantization Preserves Accuracy but Not Evidence: Explanation-Aware Post-Training Quantization for Medical LLMs
Computation and LanguageArtificial Intelligence
Summary
When medical AI models give answers, they also explain their reasoning. This paper finds that a common technique to make these models smaller and faster keeps the answers accurate but can make the explanations less trustworthy. The authors create a new method that keeps the explanations more faithful to the original full-size model while maintaining answer quality. They test this on medical question-answering tasks and show their method better preserves the link between explanations and answers.
What this means in practice
- •For medical ai developers: Improve deployment of compressed medical AI by preserving explanation quality alongside answer accuracy.
- •For natural language processing engineers: Create quantized language models that retain rationale support for answers in explanation-critical applications.
Authors
Yeji Kim, Mi-Young Kim, Randy Goebel
Abstract
Post-training quantization (PTQ) enables efficient deployment of large language models, and PTQ methods are usually optimized and evaluated with generic reconstruction, perplexity, or answer accuracy. But in explanation-critical domains, preserving only the final answer may be insufficient, since users may also inspect generated rationales to judge whether a prediction is trustworthy. We study this issue in medical multiple-choice question answering, where rationales should provide evidence that supports the selected answer. We propose an explanation-aware objective for transformation-based PTQ. Our method builds an offline faithfulness cache from full-precision teacher rationales and uses it during optimization to preserve answer-supporting evidence tokens and evidence-conditioned answer behavior. We instantiate it on OSTQuant under W4A4KV4 quantization and evaluate four 7B--8B medical and instruction-tuned LLMs on MedExQA, MedExpQA, and ChallengeClinicalQA. While a same-calibration OSTQuant baseline preserves task accuracy, it can substantially weaken answer-supporting rationales. Our objective is to preserve the full-precision model's answer-supporting behavior rather than improve gold-label accuracy, and our method better preserves the full-precision model's answer behavior and rationale-to-answer support. These results suggest that PTQ for explanation-critical settings should evaluate preservation of answer-supporting evidence, not only answer accuracy. Code and evaluation scripts are available at https://github.com/dut0817/EAQuant.