Summary
Detecting fraud in financial transactions is challenging because fraud patterns can be hidden in complex customer behavior over time. The authors compare three versions of hidden Markov models (HMMs) that track how a customer moves through different behavior types, including one linked to fraud. They improve the model by using a neural network to better represent transaction data and show that the simple output of the model is often misunderstood as a direct fraud score, proposing a correction for this. While their approach doesn't outperform top fraud prediction tools in ranking transactions, it offers better interpretability and calibrated probabilistic outputs to assist fraud detection teams.
What this means in practice
- •For financial fraud analysts: Use calibrated regime-based scores to better interpret and triage suspicious transactions in sequential customer histories.
- •For financial data engineers: Incorporate neural variational HMM embeddings to enhance modeling of transaction sequences for fraud pattern discovery beyond fixed feature sets.
Tested on one dataset.
Abstract
We present a three-tier progression of Hidden Markov Models: maximum-likelihood (Baum-Welch), variational Bayesian (VBEM), and a neural variational extension (Neural VBEM), that model each customer's transaction history as a trajectory through a small number of latent behavioural regimes, one of which is empirically identified as fraud-associated. The Neural VBEM HMM replaces the fixed Gaussian-multinomial emission family with a learned encoder, compressing a 741-dimensional transaction representation into a 64-dimensional latent space in which the VBEM HMM's posterior operates; a UMAP projection of this space reveals that the discovered regimes are not discrete clusters but ordered segments of a single continuous behavioural manifold, with confirmed fraud concentrated at its extreme. We show that the model's natural output, that is, the posterior probability of regime membership, is routinely mistaken for a fraud probability, and quantify the resulting miscalibration (the regime-membership interpretation error, MRIE); a corrected posterior-predictive score, closes most of this gap. We further distinguish batch (smoothed) inference, which uses look-ahead unavailable at deployment time, from filtered (forward-only) inference, and report both. On IEEE-CIS transaction data, the neural tier achieves a 14.4$\times$ fraud enrichment in its identified regime; while its AUPRC trails a discriminative XGBoost baseline, we show this gap is structural and not incidental, and argue the model is best positioned as a calibrated triage and interpretability layer rather than a drop-in ranking replacement.