Camera first AI agents need fast memory and slow reasoning layers
How Should a Camera-First AI Agent Work?
Artificial IntelligenceHuman-Computer InteractionLogic in Computer Science
Summary
Some AI systems using cameras face a problem: objects they need to see can disappear before the AI finishes thinking about them. The authors argue that focusing on visual recognition alone is not enough; timing matters because some visual details are short-lived. They suggest a design with two parts: one part quickly stores important visual information before it vanishes, and another part takes longer to think and reason with that stored information. Their work shows that current single-step AI systems are often too slow to catch fast-changing visual details, so splitting memory and reasoning could help.
What this means in practice
- •For mobile app developers: Create apps that respond reliably to quickly appearing and disappearing objects by separating fast visual memory storing from slower reasoning processes.
- •For robotic system engineers: Design robot vision systems with distinct fast memory and delayed reasoning layers to better interact with fleeting visual events.
A position paper. It proposes an approach and reports no results.
Authors
Xiaofan Wu, Xi Zeng, Miaoxia Chen, Peishan Chen, Shuyan Li, Chen Wang, Jiyun Yao, Hanyong Zhong, Jiahao Zhu
Abstract
Camera-first agents must act on the visual world as it unfolds. We argue that the central obstacle for future camera-first agents is not recognition ability - multimodal models (vision-language and live/streaming alike) generalize well through language and remain the right tool for semantic reasoning - but time. Many visual targets are ephemeral: they appear, are query-relevant, and leave within a short window. Binding memory to the same end-to-end path as reasoning means that by the time a monolithic pipeline completes, the evidence is gone. We formalize this as Ephemeral-Target Interaction (ETI) and a Perceptual Deadline Contract separating a hard memory deadline from a soft reasoning deadline, grounded in the bounded, passive nature of sensory and working memory. We propose a two-deadline architecture: an always-on, bounded, non-semantic fast path; an active remembering mechanism that commits a moment to durable memory before it is overwritten (event-driven in v1); and a deferred deliberation layer (multimodal model + tools). We instantiate the contract on real MOT17 track windows, a consistency check, and calibrate the single-path baseline from measured frontier-model latency: even a short answer takes 1.2-9.3 s, and an agentic identify-and-search longer. Answers can be late; evidence cannot be missed.