Robot video evaluation improved with unified task grounded scene graphs
RGraph-Eval: Task-Grounded Scene Graphs for Evaluating Robot Video Generation
Robotics
Summary
It can be hard to tell if videos showing robots doing tasks are actually correct, because the robot might appear to do the wrong action or move the wrong object. The authors created a way to check robot videos by turning what happens in them into detailed diagrams called scene graphs that track objects and actions over time. Their method uses advanced AI to recognize objects, actions, and movements in 3D and compares the video to a real example, producing a clear score that shows where differences happen. They also created a big collection of robot task videos labeled with these diagrams to help test their approach.
What this means in practice
- •For robotics development teams: Check and improve robot-generated task videos by identifying exactly where robot actions and object movements differ from correct demonstrations.
- •For industrial automation engineers: Assess robot task execution accuracy in video data to ensure proper object handling and placement during industrial tasks.
Authors
Aleksandr Lazarev, Sofia Zimina, Roman Kuznetsov, Tatiana Zemskova, Iaroslav Koshelev, Dmitry Yudin
Abstract
Generated robot videos can look convincing while depicting the wrong interac- tion: a different object is moved, a grasp disappears, or a placement ends out- side its destination. We introduce RGraph-Eval, a reference-based framework that makes these discrepancies explicit through task-grounded temporal scene graphs. Unlike approaches that construct separate evidence representations for different evaluation dimensions, we organize task entities, interactions, and motion within a unified, interpretable scene graph representation. A foundation-model pipeline identifies manipulation roles, grounds persistent object masks, infers and verifies state transitions and action intervals, and reconstructs manipulated- object motion in three dimensions. Applying this representation to a generated video and a reference demonstration supports an interpretable RGraph-Eval score with visible-object, relation-change, and 3D path-shape components. We also introduce ShareRobot-SG, a collection of 1,000 manipulation scenes from ten ShareRobot sources annotated with the multimodal representation. Our eval- uation separates graph-generation quality from agreement with human video judgments, and includes comparisons with adapted SVG2 and SG-Ego, a bench- mark of three video generators. The framework connects robot-video assessment to the observable entities and interactions underlying a task, while making the graph annotations reusable beyond evaluation. Code will be publicly available at https://github.com/SashaDance/rgraph-eval.