Specola tool measures robot behavior against detailed rules in simulation
SPECOLA: A Tool for Specification Driven Robot Evaluation
RoboticsFormal Languages and Automata Theory
Summary
Robots are usually tested by whether they complete a task, but that misses important details about how well or safely they do it. The authors created Specola, a tool that checks robot actions against detailed rules written in a formal logic called Signal Temporal Logic (STL). This lets people see exactly how close a robot was to failing or breaking rules during simulated runs. Specola works with many popular simulators and helps give a clearer picture of robot performance.
What this means in practice
- •For robotics developers: Evaluate robot control policies against rich safety and task specifications beyond simple success rates in simulation environments.
- •For robotics simulation platform teams: Integrate Specola's STL monitoring to provide detailed behavioral evaluation and requirement tracking within existing robot simulators.
Authors
Immanuel Ampomah Mensah, Hadas Kress-Gazit
Abstract
Simulation benchmarks are a useful way to measure progress in robotics; typically they provide success rate as the dominant measure of performance. However, success rate reports only whether a rollout completed the task, hiding by what margin it succeeded, how sensitive the success is to goal parameter changes, requirements it may have violated, and whether success persisted. Signal Temporal Logic (STL) provides a formal language for capturing specifications; it can be used to define requirements, and measure how close a robot behavior is to satisfying or violating them. Using STL monitors in a simulator, however, requires exposing simulator state and mapping it to specification-relevant signals. We present Specola, a tool for evaluating robot behaviors in simulation with respect to STL specifications. Specola provides an interface for writing specifications over simulator signals, and creates and manages the relevant monitors while a policy executes. We provide a Python API and command-line interface for integrating monitoring into existing policy execution loops. We provide an adapter interface for connecting new simulators, and provide adapters for nine simulation environments and benchmarks, including LIBERO, RoboCasa, Isaac Lab, and RoboDojo. We provide case studies that illustrate the nuanced evaluation that Specola enables. Code and adapters are available at https://github.com/specola-eval/specola.