Terms txt protocol enables web crawler deals with identity and payment
terms.txt: A Consent and Compensation Protocol for Agentic Web Access
Networking and Internet ArchitectureArtificial IntelligenceCryptography and SecurityComputers and Society
Summary
The web traditionally allowed search engines to crawl websites freely while sending users back in return. However, as AI programs now crawl much more and use many pages per user visit, this old deal is breaking down. The authors created terms.txt, a new file format that lets websites specify rules for different web users, including who they are, what they want, and what they should pay. Their system includes security features and can track if these rules are followed, all with minimal delay added to web requests.
What this means in practice
- •For web hosting providers: Implement terms.txt to set clear access rules and payment terms for automated web crawlers and AI agents visiting hosted sites.
- •For cloud service operators: Use the protocol to monitor and manage large-scale AI content fetching, enabling better audit and compensation models for web data access.$Commercial implications: Enables development of paid API access and licensing solutions for AI platform customers using extensive web data.
Authors
Rajarshi Chowdhury
Abstract
The open web ran on an unwritten bargain: sites admitted crawlers, and search engines sent visitors back. Public measurements show that bargain breaking under AI crawlers and agents. Automated clients now make up most requests, training dominates Cloudflare-classified crawling, and the largest AI platforms fetch thousands of pages for each visitor they return. The web's common control, robots.txt, cannot express identity, purpose, terms, or price, can be circumvented, and newer alternatives are largely proprietary CDN features. We specify terms.txt, a robots.txt-style file for per-path, per-purpose machine-access terms, plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. We define what the exchange can enforce, audit, and leave to contract. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.