Large metadata gap weakens cipher recognition by large language models
ACTS: A multi-tier benchmark evaluating LLM cipher identification under controlled blind conditions
Cryptography and Security
Summary
Identifying which encryption method was used on a piece of scrambled text is hard for large language models without extra information. The authors created a test called ACTS to measure how well these models recognize ciphers when given more or less background information. They found that these models do much better when they see metadata like filenames than when they only see the encrypted text. Forced reasoning steps, like explaining their thinking or self-correcting, did not help much. Traditional machine learning methods still outperform language models in this task under blind conditions.
What this means in practice
- •For security software developers: Develop more accurate cipher detection tools that perform reliably even without metadata about encrypted files.
- •For data forensics teams: Improve the ability to analyze encrypted data in blind scenarios where file metadata is unavailable or unreliable.
Authors
Youssef Hamdi Zafaan Ibrahim, Mohammed Khalaf Salama
Abstract
We introduce ACTS (Artifacts in Cipher Testing Suite), a reproducible benchmark that isolates cryptanalytic ability through tiered metadata deprivation (Tier-1: full metadata; Tier-2: filename only; Tier-3: completely blind) and tests forced reasoning (Tier-5: chain-of-thought, code-as-reasoning, self-correction) on ciphertext alone. A 10-configuration ablation study on 7,000 files, using a single 70/30 train-test split for feature-removal analysis, provides additional evidence at scale. Live API inference on a v2b corpus with fully randomised padding (PKCS7, ISO 10126, and ANSI X9.23 selected per file), unique CSPRNG keys, and unique plaintexts (127 files per model, 381 Tier-1 records, 380 Tier-3 records across three cloud systems, supplemented by 254 autonomous agentic evaluations in Tier-4) yields a combined Tier-3 accuracy of 30.8%, only modestly above the 14.3% random baseline for seven-way classification. The corresponding combined metadata-dependency gap is 40.9 percentage points (Tier-1: 71.7% vs. Tier-3: 30.8%). Against a classical Random Forest (69.2% on 7,000 files, trained on engineered byte-level features), the observed live gap is 38.4 percentage points. Because this comparison spans different input representations and training paradigms, the gap should be interpreted as an overall capability difference rather than a clean factorial decomposition. Six findings are reported: (1) Metadata dependency remains large; (2) Scaling failure under blind conditions; (3) Forced reasoning is epiphenomenal; (4) The observed live capability gap exceeds earlier heuristic estimates; (5) The signal is primarily structural rather than statistical; (6) Heuristic invariance versus ML fragility reveals different failure modes.