Personalized rating models can check if missing answers change predictions

Can the Remaining Answers Change the Prediction? Worst-Case Stability Checks for Personalized Rating Models

Artificial Intelligence

Summary

Sometimes, when people rate things like interviews or counseling sessions, some ratings might be missing, which can affect the final prediction made by a personalized model. The authors studied a mathematical way to check the worst-case impact of these missing ratings without changing the current prediction. They tested this approach on data from clinicians and experts rating simulated interviews and counseling sessions. While the method can confirm that certain predictions stay the same despite missing answers, it does not guarantee the predictions are correct or stable at every step, and its usefulness in real-world clinical settings is still unknown.

What this means in practice

  • For clinical data analysts: Check whether missing ratings in personalized clinical models might change predictions to validate model reliability under incomplete data.
  • For customer feedback system designers: Certify stability of personalized rating predictions despite some missing customer ratings to improve system trustworthiness.

Tested on simulated data.

Authors

Priyanka Siddappa, Muntaqim Ahmed Raju, Shifat Haider Al Amin, Ruizhe Ma

Abstract

Missing ratings can overturn a personalized model's current prediction. For completed models with additive response scores, exact worst case bounds check all missing assignments without changing the current predictor. We evaluate this audit retrospectively in two self-triage cohorts, 108 clinicians rating three simulated psychiatric interviews, and 126 counseling demonstrations annotated by ten experts. Under shared random half histories, Fisher adaptation based on observed ratings reduces log loss relative to response frequency by 0.1071 nats in the clinician audit (95% CI [0.0867,0.1261]) and 0.0223 in counseling (95% CI[0.0121,0.0321]). The counseling test excludes target experts and source videos from training. Certification covers 5.4% and 10.5% of targets; fixed width acquisition raises coverage to 13.7% and 28.6% at the same budgets. With weights and responses held fixed, the observed-rating update also has lower loss than an affine partial update while certifying exactly the same evaluated targets. The certificate establishes agreement with the completed model, not correctness or stability at intermediate updates. These rating tasks test the method; clinical usefulness remains untested.
View PDFOpen arXiv