Pastforward speeds up on-device gui agents by reusing prior computation
PastForward: Faster On-Device GUI Agents via Computational Experience Reuse
Artificial Intelligence
Summary
Using apps on phones and other devices often involves complex computer programs that help automate tasks on the screen. These programs can be slow because they redo lots of computations every time they act. The authors created PastForward, a system that remembers and reuses past computations during normal app use to speed up the process without losing accuracy. This means the device can act faster and keep sensitive information private since everything happens locally.
What this means in practice
- •For mobile app developers: Implement faster on-device automation agents that reduce delay during user interactions by reusing prior GUI computation.
- •For smartphone manufacturers: Integrate optimized GUI agent systems that maintain user privacy by performing more efficient local task inference without cloud reliance.$Commercial implications: Enables enhanced privacy-focused assistant features that run efficiently on device, appealing to privacy-conscious consumer markets.
Authors
Taehwan Park, Changmin Lee, Hayeon Lee, Taesik Gong
Abstract
Running GUI agents on edge devices can keep sensitive screens and interaction histories local, but the computational cost of inference at every action step makes deployment challenging. Existing GUI agent systems either perform full vision-language model (VLM) inference at each action step or reuse coarse-grained knowledge matched to prior tasks. However, dynamic mobile environments and user tasks make it difficult to fully utilize prior task executions without additional fine-tuning or task-specific offline exploration. To address this challenge, we present PastForward, a system that accelerates GUI agents through validated, fine-grained reuse of computational experience accumulated during ordinary task execution. During decoding, PastForward retrieves prior output sequences as device-adaptive multi-token proposals and verifies them in a single VLM forward pass. Across action steps, it uses prior GUI transitions to begin next-step inference while the device executes the current action, retains the early computation only when the predicted screen matches the observed screen, and carries reusable KV states forward. We evaluate PastForward on AndroidWorld workloads derived from real mobile usage patterns using multiple VLM backbones across server and edge platforms. On device, PastForward achieves action-step latency speedups of 1.63-2.36$\times$ while maintaining task success rates.