Learning Dynamics of Logits Debiasing for Long-Tailed Semi-Supervised Learning
2026-08-31 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study problems in semi-supervised learning where some classes have many examples and others have few (long-tailed distributions). They find that prediction biases come from how model outputs (logits) change during training and tend to reflect class imbalances. To measure this bias, they use logits from unrelated images and prove these align with class proportions. Using these insights, the authors create DyTrim, a new method that dynamically adjusts training by pruning gradients to reduce bias, which improves learning on imbalanced data. Their experiments show DyTrim works well across different models and datasets.
long-tailed distributionsemi-supervised learninglogitsclass imbalancepseudo-labelinggradient pruninglogit adjustmentreweightingresamplinglearning dynamics
Authors
Yue Cheng, Jiajun Zhang, Xiaohui Gao, Weiwei Xing, Zhanxing Zhu
Abstract
Long-tailed distributions are prevalent in real-world semi-supervised learning (SSL), where pseudo-labels tend to favor majority classes, leading to degraded generalization. While many long-tailed semi-supervised learning (LTSSL) methods have been proposed, the mechanisms by which they implicitly debias logits remain poorly understood. In this work, we revisit LTSSL through the lens of learning dynamics and provide a theoretical characterization of logits debiasing. Specifically, we derive a step-wise decomposition of the logits updates, showing that predictions are dominated by class-imbalance bias that reliably reflects label priors. To expose this effect, we use the logits of a task-irrelevant baseline image as an indicator of accumulated bias and prove that they converge to the class prior. This provides a unified view where LTSSL remedies such as logit adjustment, reweighting, and resampling correspond to reshaping gradient dynamics. Based on this insight, we propose DyTrim, a principle-based dynamic pruning framework that reallocates gradient budget through class-aware pruning on labeled data and confidence-based soft pruning on unlabeled data. We provide theoretical guarantees that DyTrim reduces class bias and improves generalization. Extensive experiments on standard LTSSL benchmarks show consistent gains across architectures and methods. Code available at: https://jiajun0425.github.io/DyTrim