Agentic model development improves video discovery retrieval at scale
Verify, Don't Trust: Agentic Model Development for Video Discovery Retrieval at Scale
Information RetrievalArtificial IntelligenceMachine Learning
Summary
It can be hard to trust changes made to complicated video search systems because mistakes or hidden problems can lead to wrong conclusions about what works. The authors created EvoPilot, a method where specialized automated agents propose and test improvements while humans verify results to avoid errors. This was tested over a month on a large video retrieval system and found real improvements that earlier automated trials missed due to evaluation bugs. Their approach keeps detailed records and reproducible checks, helping systems recover from interruptions and avoid wasteful compute.
What this means in practice
- •For video platform engineers: Improve video recommendation systems by applying human-verified automated model testing to safely and efficiently find real improvements.
- •For search system developers: Deploy durable, auditable automated research pipelines that manage long experiments and handle bugs in evaluation for large-scale search indexes.
Authors
Hao Fu, Baiting Zhu, Minglei Chen, Yinjie Huang, Shuai Ding
Abstract
Large language model (LLM) agents can propose, implement, and evaluate model changes. Autoresearch loops demonstrate this capability through minutes-scale iterations on a self-contained program. Online autoresearch instead spans asynchronous systems, hours-long variants, and weeks-long campaigns that can influence a product. A completed run can still support an invalid conclusion when a code change is a no-op, data windows leak, evaluator semantics drift, or the two arms traverse different serving funnels. We present EvoPilot, a human-gated method for long-horizon online autoresearch. Role-specific agents execute each round through a versioned domain skill and typed adapter. Durable records preserve experiments and failures; deterministic checks enforce recorded lessons. We study a 37-day campaign for the retrieval system that powers Video Deep Dive (VDD), an online experience for discovering follow-on videos after a user opens a seed video. The campaign covered seven directions and used an hourly refreshed index of hundreds of millions of videos. Earlier manual experiments had not established a benefit from an interaction head. A primitive autoresearch attempt revisited the direction but incorrectly attributed an offline hit-rate decline of 22 percentage points to the head. We then introduced EvoPilot. Its human-gated verification traced the drop to a pre-existing evaluation defect that produced output depths of 3,000 and 600. After repair, a matched comparison measured an offline improvement of 3.20 percentage points. Post-study replay and mutation tests rejected invalid comparisons while admitting valid counterparts. Durable state recovered an interrupted round, and artifact reuse avoided approximately five GPU-hours. Separately, a seven-day randomized online evaluation estimated a 0.66% relative increase in the VDD slice of Good Search Result Rate for Retention (GSRR).