Benchmark shows challenges linking redacted records with mixed data types
CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records
Cryptography and SecurityArtificial Intelligence
Summary
Working with records that combine structured data and written narratives can be hard when privacy rules hide or alter some information. The authors created CIPHER, a test set of tricky questions using real financial, medical, and law data to see how well computer systems can reason with such protected information. They found many systems struggle to pick the right records and understand the question details, especially when privacy changes obscure key facts. This work helps show where current technology still fails and provides a way to test improvements.
privacyrecord linkagestructured datafree-text narrativesSQLbenchmarkinformation retrievalpredicate interpretationdata redactionhybrid symbolic-neural systems
Authors
Suparno Roy Chowdhury, Manan Roy Choudhury, Dhruv Madhwal, Vivek Gupta
Abstract
Reasoning over privacy-constrained records requires combining structured attributes with evidence from free-text narratives. We introduce CIPHER (Cross-record Inference over Privacy-Hardened Evidence Records), a benchmark of expert-validated questions from consumer-finance, clinical, and law-enforcement records. The questions cover common tabular operations and include executable SQL supervision. We evaluate retrieval, prompting, table-specialist, and hybrid symbolic-neural systems under native redaction and surrogate-based evidence restoration. All system families exhibit substantial failures even when supporting records are provided. Most errors arise from incorrect record selection and predicate interpretation rather than arithmetic execution. Privacy transformations have non-uniform effects, sometimes obscuring necessary evidence and sometimes reducing distraction. CIPHER provides a reproducible testbed for diagnosing these failures and assessing how transformations of sensitive text affect reasoning over hybrid records.