Video based robot learning uses temporal logic for better task understanding
Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning
RoboticsArtificial Intelligence
Summary
Teaching robots to perform tasks by watching videos is tricky because it’s hard to figure out exactly what the robot should learn. The authors present a method that turns videos into a special kind of logical description, called Signal Temporal Logic, which clearly shows the order and timing of actions. This helps robots learn more effectively and work across different kinds of robots by focusing on the important parts of the task, not on how the robot looks. Their method performs better than some existing approaches on a variety of robotic tasks.
What this means in practice
- •For robotics engineers: Create robot control policies by converting task videos into formal temporal logic specifications for improved learning and inspection.
- •For automation system integrators: Develop adaptable robot behaviors guided by videos of humans or animals using temporal logic to support cross-platform transfer.
Authors
Merve Atasever, Keyan Azbijari, Cagan Bakirci, Bo-Ruei Huang, Tolga Izdas, Zahra Shahrooei, Richard Yang, Erdem Biyik, Jyotirmoy V. Deshmukh
Abstract
Video-based policy learning is particularly promising, as it illustrates target behaviors without requiring action annotations or embodiment-matched demonstrations. A central challenge is deciding what information should be transferred from the video to the robot. Existing approaches commonly convert visual observations into scalar similarity or value signals, or ask foundation models to directly generate reward code. These approaches can make the temporal structure of a task difficult to inspect, ground, and reuse. We present Video2STL, a framework that converts observation-only videos into parametric Signal Temporal Logic (STL) specifications and uses the resulting formal representation for robot learning. A vision-language model extracts an embodiment-independent semantic event trace and constructs a bank of symbolic temporal specifications. The model determines the task structure, while numerical predicate thresholds and temporal bounds are grounded from successful robot trajectories. For policy learning, we separate short- and long-timescale temporal information: short-horizon specifications provide dense rewards through rolling-window quantitative robustness, while a causal monitor over a retained long-horizon specification provides one-time progress rewards for valid temporal prefixes. The same representation supports cross-embodiment transfer from human or animal videos to robot control. Across four manipulation tasks, Video2STL achieves $85.8\%$ average success-once and $67.0\%$ success-at-end, compared with $81.5\%/59.5\%$ for native dense PPO and $65.0\%/42.3\%$ for Text2Reward; in quadruped locomotion, Qwen-3.8 and GPT-5.6-based Video2STL policies achieve $100\%$ success across velocities from $0.3$ to $2.1\,\mathrm{m/s}$ while remaining competitive in high-speed energy efficiency. Project webpage: \href{https://video2stl.github.io/}{video2stl}.