UniJev improves decision confidence separate from accuracy in multimodal AI
UniJev: Separating Accuracy from Calibration in Trimodal System One Decision Models
Computer Vision and Pattern Recognition
Summary
Many AI systems that answer questions using text, images, and videos give a confidence score with their answer. However, it’s hard to make the confidence trustworthy across all these types of input. The authors created UniJev, a system that learns from all three types together to improve answer accuracy. They found current fine-tuning methods didn’t reliably improve confidence calibration, so they developed Cora, a new way to better align predicted confidence with actual success without hurting accuracy. This means AI can be more trustworthy when it’s unsure or sure about its answers.
What this means in practice
- •For software engineers: Build AI systems that provide accurate confidence scores alongside answers for text, image, and video inputs, improving trustworthiness in automated decisions.
- •For product teams in ai startups: Integrate UniJev-based calibrated confidence models to enhance user trust in AI products handling mixed media content such as social media analysis or content moderation.$Commercial implications: Enables delivery of AI products with reliable confidence measures across multiple data types, appealing to customers who require trustworthy multimodal AI outputs.
Authors
Jianing Wang, Youwen Jian, Derong Wang, Jinyang Yu, Qingyi Si, Chuanyu Qin, Yanghua Xiao, Hongcheng Guo
Abstract
System One models produce probabilistic judgments that software can consume directly, making reliable confidence as important as answer accuracy. Yet joint training across text, images, and video remains underexplored, and the contributions of supervised fine-tuning (SFT) and reinforcement learning for calibrated decisions (RLCD) are difficult to separate. We introduce UniJev, a trimodal framework that scores candidate answers with a shared head under joint supervised training across text, images, and video. Supervised decision models improve accuracy over their backbone readouts across all three modalities, while the existing RLCD algorithms change accuracy little and have inconsistent effects on visual calibration across recipes. These findings motivate a separate supervised objective for confidence calibration. We introduce Cora (Confidence--Outcome Reliability Alignment), which groups current predictions by confidence and optimizes each prediction's confidence toward the empirical accuracy of its bin. At both 2B and 9B, Cora lowers image and video expected calibration error (ECE) while preserving accuracy, without a reinforcement stage. We release our code at https://github.com/wjnwjnwj/UniJev.