Summary
It is difficult for AI systems to take user requests, search the internet, and create 3D virtual environments that match what the user wants. The authors created a way to measure how well these systems perform this task by introducing two new scores that check if the retrieved content is relevant and if the 3D world delivered meets the user's request. They also built a system called WorldSearcher which tries to find ready-made 3D worlds or build them from videos, improving success by fixing mistakes along the way. Their findings show that even if a system finds relevant content, it does not always produce a good 3D world. Training parts of the system to recover from errors helps deliver better results more efficiently.
What this means in practice
- •For virtual reality developers: Create virtual environments for users based on live web search results, improving content relevance and delivery through recovery techniques.
- •For web application engineers: Integrate tools that turn visual web search results into immersive 3D experiences linked to user requests with measurable quality control.
Authors
Zixiao Gu, Yabo Chen, Xunzhi Xiang, Yu He, Haibin Huang, Chi Zhang, Yunbo Wang, Xuelong Li
Abstract
Agentic systems can interpret user requests, search the live web, and use external tools, but their ability to transform retrieved web content into a usable 3D world has not been systematically evaluated. No established end-to-end pipeline or benchmark exists for this capability. We introduce Search-to-World, an end-to-end evaluation task covering request understanding, web visual-content retrieval, and 3D-world delivery. We define Observed Retrieval Rate (ORR) and World Delivery Rate (WDR) to distinguish observing relevant content from successfully delivering a request-aligned, perceptually acceptable world. We also present WorldSearcher, a reuse-then-reconstruction harness that connects existing search agents to world delivery: it first retrieves reusable 3D worlds and, when none are available, reconstructs a world from video. A structured recovery controller revises temporal grounding, replaces source videos, or reformulates queries after failure. Using WorldSearcher, we benchmark representative models on Search-to-World and study supervised fine-tuning (SFT) for recovery subagents. Results show that delivery depends on the underlying agentic model, and that relevant-content observation does not ensure world delivery. Jointly training recovery agents improves delivery success and action efficiency. Search-to-World makes agentic 3D-world delivery measurable, while WorldSearcher provides a practical evaluation harness with recovery capabilities.