Vision language agents control optics experiments using visual feedback
Vision-Language Agents for Active Perception in Optics Laboratories
Artificial IntelligenceComputer Vision and Pattern RecognitionRobotics
Summary
Some scientific experiments are tricky because they don’t give clear instructions or goals for what to do next. This paper shows that vision-language models, which can understand images and language together, can actively control optics experiments by looking at camera images and deciding how to adjust equipment. The authors tested these models on different optics setups and found they can figure out how changes affect what they see and take smart actions to get better feedback. This means such models could help run real experiments by interpreting what’s happening and deciding what to do next without needing explicit numeric goals.
What this means in practice
- •For optical engineers: Use vision-language agents to autonomously adjust and optimize complex optical setups based on camera feedback without predefined numeric targets.
- •For industrial automation teams: Incorporate vision-language models to visually monitor and control equipment with sparse or ambiguous signals during manufacturing or testing processes.
Authors
Ryan Lopez, Sachin Vaidya, Seou Choi, Serena Landers, Marin Soljačić
Abstract
Vision-language models (VLMs) are increasingly being used in scientific workflows, but their ability as agents to directly control laboratory experiments from visual feedback remains underexplored. This capability is important because many laboratory tasks do not naturally provide dense, pre-defined numerical objectives: informative signals can be sparse, intermittent, or visually ambiguous. A more general laboratory agent should instead be able to interpret visual observations, take actions to acquire useful feedback, and adapt its behavior based on the consequences of those actions. We study whether general-purpose VLMs can perform this kind of closed-loop scientific control using experimental optics as a testbed. We evaluate agents on three experimental systems that isolate distinct capabilities: a Michelson interferometer, a two-mirror cavity, and a four-mirror optical relay. The agents observe camera images, directly issue actuator and measurement commands, and retain their interaction history without receiving an engineered scalar objective during control. Across these experiments and matched simulations, we find that, given task-specific natural-language guidance, VLMs can estimate actuator-response relationships, resolve ambiguous observations through intervention, and actively create informative visual feedback when signals are sparse. These results suggest that pretrained multimodal models can serve as important decision-making agents within the experimental loop. Our work also establishes optics as a physically grounded testbed for visual reasoning and active perception in scientific agents.