Robot escape room benchmark ensures verifiable manipulation skills
HATCH: Hardware-Attested Task Chains for Verifiable Manipulation
Robotics
Summary
Robot manipulation tasks are hard to verify because current tests are often closed or easy to cheat. The authors created HATCH, a physical robot escape room that anyone can build and run, where robots must complete a series of connected tasks confirmed by sensors. This setup is fair because the scoring is automatic, tasks can’t be prepared in advance, and the order of objects changes each time. The entire system is open for anyone to use, helping people fairly compare robot manipulation skills in real physical settings.
What this means in practice
- •For robotics platform developers: Use the HATCH benchmark to test and verify robot manipulation capabilities in realistic physical environments with automated, sensor-based scoring.
- •For industrial automation engineers: Implement HATCH-style chained task evaluations to certify robots’ manipulation reliability on new hardware setups before deployment in production lines.
Authors
Shiza Charania
Abstract
Capability claims in robot manipulation are unverifiable, and existing leaderboards are each closed, simulated, or gameable. We propose HATCH, a robot escape room: an openly specified, physically replicable, sensor-scored, and unrehearsable benchmark. Tasks span five skill categories at three difficulty levels and are chained so that the sensor confirming one task's completion unlocks the next, turning a room into a single long-horizon evaluation. Success is confirmed by instrumentation rather than human judgment, scoring is embodiment-agnostic, and object placements are randomized by a run-day seed. Entrants submit attested self-runs (uncut video, sensor logs, and the seed), with occasional organized events for calibration. Task specifications, build guide, bill of materials, and scoring rubrics are released under an open license.