Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos
2026-08-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors address the problem of planning steps from a start to a goal, where existing methods treat all actions as possible without checking if they make sense for the task. They propose CEFITO, a new method that learns to represent actions based on what has happened so far and uses this to rule out actions that don't fit the task during planning. This helps the system focus on plausible steps, making the planning more accurate. Their method performs better than older approaches on two standard tests for procedure planning.
procedure planningaction sequencelatent representationfeed-forward neural networkdiffusion-based inferencetask constraintsoptimizationaction-conditioned representationinference-time planningbenchmark
Authors
Mohamed Afham, Christoph Reich, Oliver Hahn, Daniel Cremers, Stefan Roth
Abstract
Procedure planning seeks to estimate a sequence of actions to transition from an observed initial state to a given goal state. Current procedure planning approaches directly predict action sequences from latent representations using feed-forward neural networks or diffusion-based inference. These paradigms treat every action as plausible, lacking the ability to enforce task-specific logical constraints that render certain actions irrelevant or not plausible. We propose CEFITO, a procedure planning approach that learns a predictor to express an action-conditioned representation space. Based on this representation space, we formulate procedure planning as a task-constrained optimization problem. Unlike prior methods, CEFITO explicitly reasons over the action space by omitting irrelevant actions during inference-time planning. This reformulation enables effective procedure planning and achieves state-of-the-art accuracy on two established procedure planning benchmarks.