Compact model improves processing of Czech web pages
Size Matters: Foundation Model for Czech HTML documents
Computation and Language
Summary
Understanding web pages automatically is hard because many existing models are big, ignore the special structure of HTML, or can't process long pages effectively. The authors created HTML-LM, a smaller model made to understand HTML and trained on a huge number of Czech web pages using several learning methods. This model can better classify and analyze web pages in Czech, even beating bigger models. It is already used to process thousands of pages every second and is publicly available.
What this means in practice
- •For web developers: Automatically classify and analyze Czech web pages more efficiently by using a model specialized for HTML and Czech language structure.
- •For content moderation teams: Process large volumes of Czech web documents to identify relevant content or issues rapidly with an efficient, scalable model.
Authors
Martin Dvořák, Vít Tlustoš, Artyom Voronin, Martin Habrovec, Kateřina Podlesná, Barbora Rišová, Josef Vonášek
Abstract
Creating universal, high-quality representations of web documents in high-traffic industrial environments requires models that are both performant and economic. Existing approaches, however, often depend on large models, overlook the structural information inherent in HTML, or are constrained by short context windows, limiting their ability to process real-world web pages. We present HTML-LM, a compact foundation model with 154 million parameters that addresses these limitations through HTML-aware training and a ModernBERT-based architecture. It was trained on 100 million web documents using multiple objectives, including masked language modeling, bag-of-words prediction, and contrastive distillation from large language models. Consequently, HTML-LM sets a new state-of-the-art for classification and regression applications in the Czech Internet domain, surpassing both larger encoders and small-sized LLMs. The model is deployed in production, processing thousands of web documents per second, and released to the community under the CC BY-NC 4.0. https://huggingface.co/Seznam/html-lm.