Learned policies speed repetitive computer tasks with greater reliability
Neuro-Symbolic Computer Use: Learning Reusable Policies for Reliable and Efficient Execution
Artificial Intelligence
Summary
Many computer tasks happen over and over with slight differences each time, but current computer agents redo all their planning every time, which is slow and buggy. The authors show a way to create reusable policies that remember stable steps in workflows and use smart models only for changes in each run. These policies are improved by testing, fixing errors, and trying new starting points without needing special evaluators. Their method works better and faster than previous ones on complex task sets.
What this means in practice
- •For automation engineers: Create reliable and reusable automation scripts that reduce runtime computation by remembering stable workflow parts and adapting only to new inputs.
- •For business process teams: Deploy pre-learned policies to accelerate repetitive office tasks with fewer failures and lower latency.
Authors
Hyewon Suh, Thanh Minh Nguyen, Chih-Lun Lee, Darrow Hartman, Lizhao Liu, Xin Eric Wang, Ang Li, Jiachen Yang
Abstract
Many computer tasks recur: the same workflow runs many times, with new inputs and from different starting states. Current computer-use agents re-plan every step of every run, which makes them costly and unreliable on such tasks. We introduce neuro-symbolic computer use, in which a recurring workflow is executed by a learned policy rather than re-derived by an agent on each run. The policy fixes the decisions that are stable across runs (ordering, variables, loops, and branches) in executable code, and delegates observation-dependent decisions, such as grounding and state checks, to neural models. We learn these policies with neuro-symbolic policy iteration: starting from one agent trajectory, it executes the policy, diagnoses failures with task-completion and step-level judges, and revises the code with a coding model informed by an agent's continuation from the point of failure, without access to the benchmark evaluator. Iterating on generated parameter and initial-state variants makes the policy reusable, and a pre-action verifier guards each state-mutating step at deployment. On OSWorld-Verified and ScienceBoard, the learned policies achieve the highest Pass^3 of all methods in all four settings, 3.6-15.8 points above the base agent, while cutting per-run cost by 15-217$\times$ and latency by 3.4-5.1$\times$. On OSWorld-Verified, policies built only on variants transfer to the held-out original tasks, exceeding AutoRPA by 8.6-17.5 points in Pass^3.