Financial speech recognition benefits from task complementarity not regularization
Does Financial Speech Need Continual Learning Regularization? A Study of EWC and mini-KFAC on Earnings Call ASR
Computation and LanguageMachine Learning
Summary
Speech recognition systems often struggle to keep what they’ve learned when updated with new data, especially in specialized areas like finance. The authors studied how to improve transcription of financial earnings calls without losing earlier knowledge. Surprisingly, they found that updating the system with new financial speech actually helped it perform better, so typical methods meant to prevent forgetting were not helpful. When such methods are needed, a technique called mini-KFAC worked better than an older one named diagonal EWC.
What this means in practice
- •For speech recognition engineers: Improve updating of financial domain speech models by verifying task complementarity before using continual learning regularization.
- •For machine learning infrastructure teams: Use mini-KFAC EWC instead of diagonal EWC for more stable and accurate incremental training of speech models when guarding against forgetting.
Authors
Thai Quoc On, Homayoon Beigi
Abstract
Earnings call transcription presents a realistic continual learning scenario where Automatic Speech Recognition (ASR) systems are updated with new recordings but risk catastrophic forgetting of previously learned financial speech patterns. Adapting Whisper-small from a general-domain speech corpus to the Earnings22 benchmark reveals the opposite: all evaluated methods exhibit improvement on held-out financial speech from the same domain during adaptation rather than forgetting. This task complementarity makes regularization counterproductive. Nevertheless, when regularization is warranted, a layer-wise mini-Kronecker-Factored Approximate Curvature (mini-KFAC) approximation seems to achieve lower word error rates (WERs) and more stable optimization than standard diagonal Elastic Weight Consolidation (EWC). The results suggest that task complementarity should be assessed before applying continual learning constraints, and prefer mini-KFAC EWC over diagonal EWC when needed.