Papers for

web hosting providers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Terms txt protocol enables web crawler deals with identity and payment

terms.txt: A Consent and Compensation Protocol for Agentic Web Access

Abstract: The open web ran on an unwritten bargain: sites admitted crawlers, and search engines sent visitors back. Public measurements show that bargain breaking under AI crawlers and agents. Automated clients now make up most requests, training dominates Cloudflare-classified crawling, and the largest AI platforms fetch thousands of pages for each visitor they return. The web's common control, robots.txt, cannot express identity, purpose, terms, or price, can be circumvented, and newer alternatives are largely proprietary CDN features. We specify terms.txt, a robots.txt-style file for per-path, per-purpose machine-access terms, plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. We define what the exchange can enforce, audit, and leave to contract. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.

Thu 10 SeptNetworking and Internet ArchitectureArtificial IntelligenceCryptography and Security
The gist
The web traditionally allowed search engines to crawl websites freely while sending users back in return. However, as AI programs now crawl much more and use many pages per user visit, this old deal is breaking down. The authors created terms.txt, a new file format that lets websites specify rules for different web users, including who they are, what they want, and what they should pay. Their system includes security features and can track if these rules are followed, all with minimal delay added to web requests.
Open 2609.11152v1