Agent uncovers hidden defects in large Slovak web document collection
Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2
Computation and Language
Summary
Large language models often handle many languages without focusing on unique problems for each one. This paper describes how the authors used an AI agent to carefully examine a huge collection of Slovak web documents to find errors, like machine-made content slipping through filters. The agent created automated tests that a Slovak speaker then checked, confirming most issues. This process is faster and cheaper than running many classification passes with AI and produces reusable tests for future use.
What this means in practice
- •For language technology teams: Improve quality control of multilingual text collections by identifying and testing for language-specific data problems.
- •For web content moderators: Detect large-scale machine-generated document farms that evade filters in web content datasets at low computational cost.
Tested on one dataset.
Authors
Marek Šuppa, Ivan Vykopal, Jaroslav Kopčan
Abstract
Multilingual pipelines such as FineWeb2 apply one recipe across a thousand languages, leaving the failure modes of individual languages unanticipated. We use an LLM once, as a corpus debugger rather than a classifier: a tool-using agent explores the corpus through a read-only, budgeted API and compiles each finding into an executable test, which a native speaker then validates. On the 26.5M-document Slovak portion of FineWeb2, three runs costing \$2.21-\$2.84 each found seven findings over four phenomena. The largest is a machine-generated content farm of 1,408,437 documents (5.32%) that passed the pipeline's filters, 86.6% of it passing even stricter thresholds. Blinded labels support 6 of the 7 findings (defect rates .53-1.00 against .08 in a random control) and do not support one. The compiled suite scans the full corpus in 0.7 s of wall-clock time without a GPU, against \$12k-\$90k per LLM classification pass. The best static sampling baseline validates more selectors than any agent run, at under half the cost. The agent contributes exact prevalence, a precision estimate checked against sampled documents, an account of why the pipeline missed each slice, and a reusable test. We release the framework, Slovak test suite, validation data and agent trajectories.