Numerical problems disrupt calculations between very similar distributions

On numerical issues of divergences between nearly equal distributions

Machine Learning

Summary

When comparing two very similar groups of data using common mathematical tools like KL divergence or Jensen-Shannon divergence, small errors from computer rounding can mess up the results. The authors found these issues affect how both the values and their rate of change (gradients) are calculated. These numerical problems can cause trouble in areas like machine learning where such comparisons guide training, including knowledge distillation and reinforcement learning. The paper offers new, more stable ways to do these calculations and shows code examples that pass thorough tests.

What this means in practice

  • For machine learning engineers: Improve training stability in models using divergences by integrating numerically stable computations for near-equal distributions.
  • For data scientists in finance: Enhance accuracy of probabilistic model comparisons where subtle distribution differences affect risk estimations.

Authors

Brandon Amos

Abstract

Barron et al. (2026) recently identified numerical issues in the softmax and soft cross-entropy loss. We show that similar issues are present in standard divergence operations between categorical distributions. This includes the forward and reverse Kullback-Leibler (KL) divergence and generalized Jensen-Shannon divergence (JSD), as well as Monte Carlo KL estimates and entropy. When the distributions are nearly equal, rounding and cancellation issues affect the forward and gradient computations. These issues can affect settings using divergences in optimization objectives, such as knowledge distillation, reinforcement learning, and proximal methods. We propose numerically stable reformulations and provide example implementations for each setting, validated by property-based tests.