OnPTQ improves multimodal AI decision accuracy by tuning token choices
Calibrate the Decisions That Change the Future: On-Policy Post-Training Quantization for Multimodal Large Language Models
Artificial Intelligence
Summary
Large AI models that handle text and images often need to be made smaller and cheaper to use by converting their knowledge into fewer bits, a process called quantization. The authors found that existing ways to adjust the models after this conversion missed important feedback loops because changing one word early on can change all future words. They created a method named OnPTQ that looks at the model’s own choices step-by-step and carefully adjusts where mistakes have the biggest effect on future outputs. This leads to better performance without changing how the AI runs once it’s deployed.
What this means in practice
- •For machine learning engineers: Reduce errors in quantized multimodal AI models by calibrating token predictions that significantly impact future outputs during deployment.
- •For ai infrastructure teams: Improve efficiency of large vision-language models by post-training adjustments that maintain accuracy without changing inference frameworks.
Authors
Wenxiao Fan, Jingling Fu, Lichen Ma, Yu He, Luohang Liu, Jinbao Xue, Ke Zhang, Junshi Huang, Kan Li
Abstract
Post-training quantization (PTQ) lowers deployment cost for multimodal large language models, but calibration typically reconstructs fixed sequences with local objectives. This overlooks autoregressive feedback: a quantization-induced token change redirects the prefix and changes future states. Yet on-policy coverage alone is insufficient because many decision mismatches barely affect future generation. We propose OnPTQ, an on-policy framework that calibrates on trajectories visited by the current quantized policy. On shared prefixes, OnPTQ identifies quantization-eroded boundaries, evaluates competing tokens through short counterfactual rollouts, and combines current discrepancy with branch consequence into a Decision--Consequence risk. The risk prioritizes critical states, while context anchoring and trajectory refresh preserve multimodal behavior and keep calibration aligned with the updated policy. We further derive a Decision--Consequence bound linking behavioral deviation to current policy discrepancy and action-conditioned future-value span. Across vision--language and omni-modal Qwen models under multiple low-bit settings, OnPTQ improves downstream performance and yields fewer correctness flips against the corresponding Dense/FP16 references, without changing the deployed inference graph.