Benchmark tests show challenges in reasoning over privacy-protected records

CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records

Cryptography and SecurityArtificial Intelligence

Summary

Dealing with private records means some details are hidden or changed, making it hard to answer questions that mix tables and free-text information. This paper presents CIPHER, a set of expert-made questions from finance, health, and law records to test how well systems handle this problem. The authors found that many systems struggle, especially with choosing the right information and understanding conditions, even when given the right records. They also noticed that privacy protections can both hide needed facts and remove distractions, affecting performance in different ways. CIPHER helps others study these challenges and improve reasoning over sensitive, mixed-format records.

What this means in practice

  • For hospital data teams: Evaluate how privacy methods affect clinical record analysis using combined tabular and text data under different protections.
  • For financial compliance teams: Test automated systems that answer questions from privacy-hidden consumer finance records to improve regulatory risk assessments.

Authors

Suparno Roy Chowdhury, Manan Roy Choudhury, Dhruv Madhwal, Vivek Gupta

Abstract

Reasoning over privacy-constrained records requires combining structured attributes with evidence from free-text narratives. We introduce CIPHER (Cross-record Inference over Privacy-Hardened Evidence Records), a benchmark of expert-validated questions from consumer-finance, clinical, and law-enforcement records. The questions cover common tabular operations and include executable SQL supervision. We evaluate retrieval, prompting, table-specialist, and hybrid symbolic-neural systems under native redaction and surrogate-based evidence restoration. All system families exhibit substantial failures even when supporting records are provided. Most errors arise from incorrect record selection and predicate interpretation rather than arithmetic execution. Privacy transformations have non-uniform effects, sometimes obscuring necessary evidence and sometimes reducing distraction. CIPHER provides a reproducible testbed for diagnosing these failures and assessing how transformations of sensitive text affect reasoning over hybrid records.
View PDFOpen arXiv