Oracle evaluation can underestimate error in thermal heart rate estimation
Oracle headroom without signal: null-calibrated evaluation of candidate selection for thermal heart rate estimation
Computer Vision and Pattern Recognition
Summary
When using thermal cameras to estimate heart rate from faces, many possible signals can be chosen. To find the best signal, people often use an "oracle" that picks the closest estimate to the true heart rate, but this can be misleading because it sometimes just picks a lucky guess. The authors show with math and data that having many candidate signals often leads to overly optimistic error estimates when using an oracle. They suggest reporting additional details and using controls to better evaluate signal selection methods.
What this means in practice
- •For biomedical device engineers: Evaluate heart rate estimation systems more reliably by accounting for chance matches when selecting signals without a physiological reference.
- •For computer vision developers: Design better algorithms for selecting facial thermal signals by understanding the limits of oracle-based evaluations in benchmark tests.
Authors
Mohammad Rakibur Rahman, Nhi Nguyen, Sasan Sharifipour, Le Nguyen, Manuel Lage Cañellas, Miguel Bordallo López, Constantino Álvarez Casado
Abstract
Camera-based physiological monitoring can produce multiple estimates from several facial regions, extraction methods, and processing settings. Signal quality indices aim to select reliable estimates without a physiological reference, and their potential is often assessed with an oracle that selects the estimate closest to the reference in each window. This retrospective selection can reward chance agreement. We model the effect with order statistics. For K independent candidates unrelated to the reference, the expected oracle error decreases approximately as 1/K. We analyze thermal heart rate estimation on 96 iBVP recordings with 168 candidates per 10 s window. The oracle achieves a mean absolute error of 0.91 bpm, compared with 10.74 bpm for the best fixed configuration, 18.03 bpm for the best quality index, and 8.61 bpm for a constant predictor. With K = 24, a forehead signal from another recording matches the correct one, with 4.62 against 4.61 bpm. Oracle evaluations should report candidate count, valid coverage, and matched null controls.