Embodied agents improve task performance by evolving execution strategies
EnactOpt: Diagnosis-Guided Harness Evolution for Embodied Agents
Robotics
Summary
Embodied agents—robots or software that interact with the world—need a framework, called a harness, to organize what they see, decide what actions to take, and learn from what happens. Sometimes failures come not from the agent itself but from how this harness presents information and guides the agent. The authors introduce EnactOpt, a method that improves these harnesses by using limited feedback without changing the main models inside the agent. They show that by editing how the agent’s environment and decisions are described and acted on, the agent’s success rate in tasks improves noticeably across different test environments.
What this means in practice
- •For robotics engineers: Improve robot interaction success by evolving how task context and actions are organized without changing the robot's core AI.
- •For virtual assistant developers: Enhance success in virtual environments by refining how assistant agents interpret observations and execute actions using limited feedback.
Authors
ZhiYuan Feng, Zhuoshi Pan, Zhaolu Kang, Zhongxia Chen, Shuo Yang, Sen Ye, Heng Wang, Han Li
Abstract
Embodied agents built on foundation models rely on an execution harness to organize observations, guide actions, and incorporate execution feedback into subsequent decisions. Failures can arise not only from limited model capabilities, but also from how the harness presents evidence and guides its use. These model-external mechanisms provide an adaptation target without updating model parameters, but task-specific interaction experience can be costly to collect. Task-level outcomes alone do not reveal whether relevant evidence was missing from the model input or was available but inadequately used. Moreover, these two aspects are coupled: effective action guidance may require evidence that the current context omits, while exposing that evidence alone does not ensure an appropriate response. We introduce EnactOpt, which uses limited execution feedback to jointly evolve reusable textual skills and context-construction code while keeping the underlying models and runtime fixed. It diagnoses adjacent decisions by relating action feedback to the actual model input and subsequent action. Structured episode summaries guide joint revisions through prompt-level edit operators, and candidate harnesses undergo compatibility checks and validation-based retention. Across VLABench, ALFWorld, and Embodied Agent Interface (EAI), EnactOpt improves task success by approximately 10-27 percentage points over the initial harnesses and outperforms the evaluated GEPA, TextGrad, and Trace2Skill baselines. These gains extend to task conditions not used for harness evolution or candidate selection. The results support a complementary route to embodied adaptation: turning limited execution experience into reusable behavioral improvements by jointly refining how evidence is presented and acted upon.