AI trust calibration testbed reveals gaps in risk assessment policies
CRiDiT: Instantiating a run-time testbed for trust calibration in AI-infused systems
Human-Computer Interaction
Summary
When people use AI systems, it's important their trust matches how reliable the system really is to avoid mistakes or dangers. The authors built a working testbed called CRiDiT that tracks how human trust and AI trustworthiness interact in real time and checks for mismatches. They tested it in hiring, finance, and legal scenarios, finding that some parts of the system's method to detect and fix trust gaps didn't work as expected. This shows that trust calibration needs to focus on the underlying evidence rather than just summary numbers, and that how the system reacts after detecting over-trust should be improved.
What this means in practice
- •For system engineers: Test and improve trust calibration policies in AI components within complex infrastructures using a real-time interactive testbed.
- •For financial compliance teams: Evaluate how automated trust assessments align with human decisions in financial risk scenarios for safer AI adoption.
Authors
Yuntian Ding, Nicolas Herbaut, Camille Salinesi
Abstract
The integration of AI into larger technical infrastructures has made the alignment of human trust with system trustworthiness, known as trust calibration, a critical engineering concern, since misplaced trust in either direction leads to operational and safety risks. While conceptual frameworks provide a strong foundation for understanding trust calibration, their translation into running systems remains a challenge, because there are few testbeds in which human trust inputs, machine trustworthiness evidence, gap detection and remediation operate together within a closed loop. This paper instantiates CRiDiT (Computational Risk-Sensitive biDirectional Trust) as a run-time testbed, operationalising machine-side trust with Dempster-Shafer Theory and PCR5 redistribution, human-side trust with Subjective Logic, and calibration with a threshold-based trust gap. Following the Design Science Research methodology, we exercise the artifact across three high-stakes scenarios (hiring, financial, legal), producing 144 logged interaction steps across fifteen sessions. The analysis shows that the artifact captures trust calibration dynamics as intended, and reveals three points at which the instantiated policy departs from its design requirements: the machine-side estimate begins from a global benchmark rather than task-relevant evidence; risk-sensitive thresholds do not produce risk-sensitive triggering; and the calibration policy assigns explanatory prompts to over-trust, where corrections narrowed the gap in all 6 observed cases. Since the first two arise from the same design decision, to make the difference of two estimated scalars the calibration criterion, they point toward a common requirement: that the criterion should operate on the evidence rather than on scalars derived from it. The third concerns what follows detection, and shows that the action vocabulary inherited from trust repair does not align with what the interaction logs show to be effective. The work contributes the artifact, a characterisation of its run-time behaviour, and the requirements this characterisation elicits.