Engine improves reversible pseudonymisation for uk english and russian documents
Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine
Computation and LanguageCryptography and Security
Summary
Many professionals send documents containing personal data to external AI language tools. To protect privacy, sensitive details are replaced with placeholders before sending and restored afterward. The authors tested several existing tools and created a new engine that better masks and recovers personal data in English and Russian documents. Their engine drastically reduces visible personal data compared to others and restores placeholders accurately, although some issues remain with Russian name forms. The work is based on synthetic data and has yet to be tested on real documents.
What this means in practice
- •For legal document processors: Improve document privacy by automatically replacing and restoring personal data in English and Russian before using external language models.
- •For multilingual data privacy teams: Provide a tested tool and benchmark for masking personal information in bilingual (Russian and UK English) text sent to AI language services.
Tested on simulated data.
Authors
Vladimir Beskorovainyi
Abstract
Professionals send working documents to external large language models. A common safeguard is reversible pseudonymisation: personal data are replaced with placeholders before the request and restored locally in the answer. We present an open benchmark of five existing open tools and a regular-expression baseline on Russian and UK English, and an open engine built on the two strongest tools. The benchmark excludes each tool's own authors' dataset from its verdict, compares the engine with each tool on identical datasets for detection, reports masking precision, and tests on a held-out set of synthetic documents whose files, hashes, decision rules, run plan and stopping rule were published before any run. Two local LLMs test whether placeholders survive. All evaluation data are synthetic, and the benchmark was run by the author of the engine; code, data, predictions and raw answers are released. On external datasets no existing tool meets all predefined thresholds: in UK English the identifier and leak thresholds are not met, in Russian the identifier threshold is borderline. On the last held-out set of 1,000 documents per language, the best existing tools leave a whole value visible in 61% (Russian) and 58% (UK) of documents. On these documents, from the generator it was developed on, the engine does so in 0.3% [0.1; 0.9] and 0.2% [0.1; 0.7], leaves any part of a value visible in 0.8% and 0.2%, and meets all detection thresholds in both languages. In UK English it also restores at least 99% of its placeholders after 760 answers. In Russian it restores only 94.6% [93.6; 95.6] of name placeholders, against a 99% threshold, mainly because the inflection it borrows from pii-guard produces wrong surname forms. The engine was developed over six earlier held-out sets from the same generator; real documents were not tested.