Tenetbench measures ai language models against russian professional ethics codes
TenetBench: A Clause-Anchored Russian Corpus for Evaluating Language Models against Professional Codes of Ethics
Computation and Language
Summary
Working professionals follow codes of ethics that guide how they should act, but AI assistants can sometimes help break these rules without users noticing. The authors created TenetBench, a dataset of thousands of conflicts between user requests and Russian professional ethics codes, paired with harmless requests for comparison. They tested multiple language models to see if the models would produce unethical responses or warn users. Most models failed often, producing conflicting content without warnings. The dataset and tools can help improve AI assistants by testing how well they follow professional rules.
What this means in practice
- •For software developers: Test language models to detect and reduce responses that violate professional ethics using an automated, expandable dataset.
- •For compliance teams: Evaluate AI assistants' compliance with professional codes of ethics to improve monitoring and risk mitigation strategies.
Authors
Yegor Anichkov, Valery Vinogradov, Andrey Furov, Marina Polukoshko, Sergey Bolovtsov
Abstract
Codes of professional ethics guide and restrict specialists, who increasingly work with assistants built on large language models. Such an assistant can breach the user's code, or help the user breach it, without the user noticing. Evaluations reporting broad categories of harm cannot show how often this happens or which provision is breached. TenetBench pairs 3,433 requests conflicting with clauses of fifteen Russian-language professional codes with benign twins that differ only in the legitimacy of what is asked. An automated pipeline generates the tasks from the text of a code, so the corpus grows with every code added and a provision yields new tasks on every run. It also produced tasks and human-validated verdicts from two English codes of other jurisdictions. In human validation of the frozen pipeline, specialists rated 0.995 of the conflicting tasks plausible, and the model judge agreed with the adjudicated label at Gwet's AC1 of at least 0.894 on 1,860 elements. We evaluated fourteen model and reasoning-mode combinations on whether they produce the conflicting text, decline or warn. Eleven of the fourteen produce conflicting content on most such requests, mostly without a warning, and nearly three warnings in four accompany conflicting content. The models that conform more do so by declining while still serving the legitimate twin, and enabling reasoning halved the refusals of two models. We hope the dataset and pipeline will help evaluate how language models observe professional norms and let diagnostic datasets be built for particular professions and organisations.