Frontier and open models tackle crypto asset accounting accuracy
Crypto Accounting Bench: Evaluating Frontier and Open-Weight Models on Crypto-Asset Accounting Tasks
Artificial IntelligenceComputation and Language
Summary
Accounting for cryptocurrency transactions can be really complex because it involves tracking things like amounts, currencies, and legal rules. The authors created Crypto Accounting Bench (CAB), a test set to see how well advanced language models can reproduce the exact bookkeeping entries organizations made for their crypto trades. They tested 12 language models and found the best ones got about 77% accuracy overall but only succeeded perfectly on about 57% of the tasks. The biggest challenge is picking the right accounts and making full, correct entries.
What this means in practice
- •For crypto accounting software teams: Improve automated bookkeeping in crypto accounting software by training models to generate precise journal entries from transaction data.
- •For financial compliance teams: Use model outputs evaluated on CAB to support audit and regulatory compliance for crypto asset transactions.
- •For tax preparation service providers: Enhance preparation of crypto tax forms by generating detailed, balanced journal entries reflecting transaction full context.$Commercial implications: This paper enables more accurate digital tools that produce complete crypto bookkeeping entries needed for tax compliance products.
Authors
Kareem Khattab, Omar Khattab, Mohamed Ibrahem
Abstract
We introduce Crypto Accounting Bench (CAB), a benchmark for assessing whether frontier and open-weight language models can reconstruct the complete journal entry that an organization actually posted for a crypto-asset transaction. CAB contains 118 evaluation tasks drawn from 7 pseudonymized organizations. Each task combines transaction mechanics, asset quantities and base-currency values, wallet and legal-entity context, counterparty evidence, related transaction legs, recurrence, tax-lot evidence, and the organization's complete chart of accounts. The target is a balanced structured entry with every required account, side, amount, currency, and full-precision asset quantity. We evaluate 12 models spanning proprietary frontier systems and open-weight releases over 3 independent attempts per task, producing 4,248 trajectories. We report 3 metrics: Mean Score, Best@3, and Pass@3. Pass@3 is the fraction of tasks with at least 1 of 3 attempts that satisfies every rubric criterion and required gate. The leading model reaches 77.43% Mean Score, while the best Pass@3 is 56.78%. Deterministic diagnostics, read from each task's best of 3 attempts and macro-averaged across the 12 models, show higher base-amount agreement (97.8%) than deciding-account accuracy (56.3%). Together with the failure analysis, these results identify account selection and complete-entry composition as the main remaining challenges on CAB.