Odometry accuracy improves by scoring with uncertainty estimates
You Should Be Properly Scoring Your Odometry
Robotics
Summary
When measuring odometry — the path a robot or vehicle thinks it traveled — people often just check how close the path is to the real one. The authors explain that this misses an important detail: the estimator's own guess about how uncertain it is. By using a better scoring method that considers both the estimate and its uncertainty, they can spot when the system is too confident and likely wrong. They also created a software tool to help test this, and found common systems often trust their measurements more than they should.
What this means in practice
- •For robotics engineers: Use proper scoring to improve odometry estimators by evaluating both their accuracy and self-reported confidence, preventing overconfident tracking failures.
- •For autonomous vehicle teams: Assess the reliability of localization systems by testing reported uncertainties, helping ensure safer navigation through better uncertainty reporting.
Authors
Ola Rønning, Usama Saqib, Andrzej Wąsowski
Abstract
When we evaluate the performance of our odometry, it is common practice to score the estimated track against a ground truth. Unfortunately, scoring uses point metrics, such as the root mean square error, that ignore the covariance matrix which estimators like filters and smoothers already report. Using the covariance matters for two reasons. First, the covariance encodes the estimator's uncertainty, so it tells us whether the estimator trusts its own output. An overconfident estimator will not report itself lost. Second, the covariance weights the error in each direction of the estimate. Without the covariance, an estimator is unduly penalized for a high error in an uncertain direction. Instead of point metrics, we should use strictly proper scoring rules. These rules score the estimate together with its reported uncertainty. Strictly proper scoring rules recover the point metrics when no covariance is reported, and they diagnose covariance inconsistency when covariance is reported. Using a one-sided pairwise test, we show that two estimators can expose overconfidence in at least one of them without a ground truth. Strictly proper scoring rules and our pairwise test are available in our open-source framework smfeval. As a case study, we use smfeval to assess the uncertainty quality of the translational component of ground-based LiDAR-inertial odometry. Across four filters we find overconfidence - the worst case reports centimeter certainty with kilometer error. Knowing the filters are overconfident, we investigate the mechanism. The investigation traces overconfidence to filters crediting LiDAR measurements with more new information than they carry.