Multilingual test challenges AI agents to find complex web answers
HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents
Artificial IntelligenceMachine Learning
Summary
Finding answers on the internet can be tricky, especially when the information is hidden in videos, maps, or scanned images. The authors created HyperBrowseComp, a tough test with questions in 13 languages that need finding clues spread across different kinds of web content. They made sure the questions were hard enough that AI models without internet access couldn't answer them easily. They also checked how well different AI models and humans could solve these questions to see how challenging the test really is.
What this means in practice
- •For ai developers: Evaluate and improve AI browsing agents’ ability to locate hard-to-find web evidence across languages and media types.
- •For search engine engineers: Test retrieval and reasoning pipelines that combine diverse web sources including videos and images to enhance search accuracy.
Authors
Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri, Seung Hun Eddie Han, Ryandito Diandaru, Qinrong Cui, Jan Christian Blaise Cruz, Badrinath Chandana, Peerawat Chomphooyod, Ahmed Attia, Jonibek Mansurov, Emilio Villa-Cueva, Canh Duong Nguyen, Imran Turganov, Minghao Wu, Peerat Limkonchotiwat, Irina Nikishina
Abstract
We introduce HyperBrowseComp, a multilingual and multimodal browsing benchmark comprising 423 manually authored and human-validated questions across 13 languages, written by native or highly proficient speakers. Questions are designed to be extremely challenging. Each question targets a concise, publicly verifiable answer whose discovery requires locating obscure evidence, following multi-step clue chains, or inspecting heterogeneous sources such as videos, scanned documents, images, or maps. Easier questions are filtered out by evaluating them with models without internet access to reduce the likelihood that they can be answered with parametric knowledge alone. We evaluate several models using provider-native search and a shared external retrieval harness under a common agent protocol. To contextualize model performance and effort, we also conduct a human evaluation on a sample of the questions. HyperBrowseComp provides a challenging testbed for persistent information seeking across languages and evidence modalities, with difficulty arising from discovering and connecting evidence on the open web.