AI and human teams make safer decisions with fewer reviews
Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making
Artificial IntelligenceMachine Learning
Summary
When humans and AI work together to make decisions, humans can stop unsafe choices made by AI, but it costs time and effort for humans to always check. The authors designed a system called CARE that learns safely from human feedback to make better AI decisions over time while still keeping human oversight when needed. CARE is flexible and works with many AI types. It significantly reduces how often humans need to step in, while keeping decisions safe and aligned with human values.
What this means in practice
- •For autonomous vehicle operators: Implement CARE to reduce driver interventions while ensuring safe decisions from AI driving systems.
- •For robotics system developers: Use CARE to integrate human feedback safely into robot decision-making, improving automation and safety.
Authors
Chenyu Zhang, Rachel Luo, Boyi Li, Anjali Parashar, Marco Pavone, Apoorva Sharma
Abstract
In human-AI collaborative decision making, human review can prevent unsafe AI decisions, but each human judgment is costly. Treating human intervention after AI abstention as a one-off fallback misses the opportunity to improve future AI decisions for greater automation, yet AI adaptively learning from selectively queried human feedback breaks safety guardrails calibrated for old models. We approach this challenge with CARE---calibrated adaptive rectification and escalation---an end-to-end pipeline that combines AI models and human reviewers to guarantee safe, human-aligned decisions, while continuously learning from human feedback to achieve greater automation with fewer human queries. CARE is principled, general, modular, and works with any black-box AI model. Our novel adaptive calibration module guarantees risk control at every time step for any rectification module. We further show how CARE improves query efficiency when the AI model is well trained and the human-AI misalignment has a clear structure. Experiments on four safety-critical real-world datasets spanning driving, language, and robotics demonstrate that CARE achieves human-aligned decisions while reducing human queries by 25-81% relative to baselines.