Papers for

data scientists in finance

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Numerical problems disrupt calculations between very similar distributions

On numerical issues of divergences between nearly equal distributions

Abstract: Barron et al. (2026) recently identified numerical issues in the softmax and soft cross-entropy loss. We show that similar issues are present in standard divergence operations between categorical distributions. This includes the forward and reverse Kullback-Leibler (KL) divergence and generalized Jensen-Shannon divergence (JSD), as well as Monte Carlo KL estimates and entropy. When the distributions are nearly equal, rounding and cancellation issues affect the forward and gradient computations. These issues can affect settings using divergences in optimization objectives, such as knowledge distillation, reinforcement learning, and proximal methods. We propose numerically stable reformulations and provide example implementations for each setting, validated by property-based tests.

Fri 13 NovMachine Learning
The gist
When comparing two very similar groups of data using common mathematical tools like KL divergence or Jensen-Shannon divergence, small errors from computer rounding can mess up the results. The authors found these issues affect how both the values and their rate of change (gradients) are calculated. These numerical problems can cause trouble in areas like machine learning where such comparisons guide training, including knowledge distillation and reinforcement learning. The paper offers new, more stable ways to do these calculations and shows code examples that pass thorough tests.
Open → 2611.18511v1

Large-step gradient descent reveals new dynamics in neural network training

Classification with Large-Step Gradient Descent: Unifying Self-Stabilization and Convergence Rates

Abstract: Neural networks are often trained with gradient descent (GD) using step sizes that exceed classical stability thresholds, a regime where standard convergence theory no longer applies. Despite exhibiting non-monotonic loss and oscillatory dynamics, this regime frequently leads to faster convergence and better generalization. Existing theoretical perspectives offer complementary but incomplete insights: one identifies a self-stabilization mechanism that drives GD toward flat regions but does not yield convergence rates; another derives rigorous rates for specific losses but does not explain the underlying stabilization mechanism; and a third unifies these views for overparameterized least squares, but relies on the existence of a minimizer manifold, which is absent in classification settings. In this work, we extend this unified geometric perspective to exponential-tail losses, including logistic and cross-entropy losses in separable linear classification, where the loss infimum is not attained at any finite parameter. We identify a curvature-defined reference subspace that replaces the role of the minimizer manifold, yielding a coupled dynamical system in which orthogonal oscillations are damped by parallel progress along directions of decreasing sharpness. Under regularity conditions, which we verify for logistic and multiclass cross-entropy losses on symmetric data models, we derive explicit bounds for the iterates, not just the loss, covering both the unstable and stable regimes, and revealing a three-phase damping structure in the latter. Our analysis further uncovers a transient implicit bias induced by large step sizes: although GD converges asymptotically to the max-margin direction, large steps can leave persistent orthogonal residuals and, in certain geometries, amplify them, affecting finite-time alignment with the max-margin direction.

Thu 12 NovMachine Learning
The gist
Training neural networks often involves adjusting parameters step by step, but choosing big steps usually breaks normal rules. The authors explain how large steps cause the training to oscillate yet still lead to good results by finding stable flat regions. They discovered a geometric way to understand these behaviors even when classical assumptions don’t hold, especially for common classification losses like logistic regression. Their findings also show that big steps influence the direction neural networks settle on, sometimes creating lasting effects that slow perfect alignment.
Open → 2611.17459v1

Synthetic data reduces fairness gaps caused by imbalanced training data

Synthetic Data Augmentation for Fairness under Data Imbalance: Theory and Adaptive Allocation

Abstract: Data imbalance across sensitive groups and class labels is common in classification problems and can contribute to substantial fairness disparities. We study how such imbalance affects Equal Opportunity and how synthetic data augmentation can mitigate the resulting unfairness. We first identify two mechanisms, prevalence imbalance and group-size imbalance, through which changes in the training distribution shift the learned decision boundary and induce disparities in group true positive rates. Motivated by this analysis, we study exact full-balance augmentation, which equalizes all group-label cells. We establish conditions under which full balancing strictly improves Equal Opportunity together with a prediction-error comparison under the oracle balanced distribution, and show that the Equal Opportunity gap of the synthetic-data classifier converges to that of the oracle balanced classifier as the sample size grows and the generator error vanishes. The finite-sample convergence rate agrees, up to logarithmic factors and generator-approximation error, with the information-theoretic lower-bound rate under the corresponding regularity regimes. Since exact balancing need not provide the best fairness-accuracy trade-off in practice, we further propose Accuracy-Constrained Adaptive Synthetic Augmentation (ACASA), which learns both where and how much to augment by minimizing estimated fairness disparity subject to a constraint on predictive accuracy. We provide a finite-sample guarantee comparing the selected augmentation allocation with the population-optimal accuracy-feasible allocation. Simulation studies and real data application support the theoretical findings and demonstrate that synthetic augmentation can substantially reduce fairness disparities while achieving favorable empirical fairness-accuracy trade-offs.

Wed 11 NovMachine Learning
The gist
When computers learn from data that has imbalances among different groups, their decisions can be unfair. The authors studied how these imbalances affect fairness, especially the chance that the computer correctly identifies positive cases equally for all groups. They showed that adding carefully made synthetic data can help to balance these groups and improve fairness. They also developed a method that smartly decides how much synthetic data to add while keeping accuracy high. Tests on simulated and real data confirmed that their approach reduces fairness gaps effectively.
Open → 2611.14870v1

Knowledge distillation improves interpretable low-cost classifiers on tabular data

Knowledge Distillation from Neural Networks to Differentiable Logic Networks

Abstract: Differentiable logic networks (DLNs) are interpretable, edge-friendly classifiers that learn both their logic operators and their connections through gradient descent, achieving accuracy competitive with multi-layer perceptrons at orders-of-magnitude lower inference cost. Their accuracy, however, is bounded by the problems of vanishing gradients, scalability, and lack of an ecosystem, leaving a gap relative to conventional neural networks on several datasets. We introduce knowledge distillation for DLNs to narrow this gap. We distill from a trained multi-layer perceptron teacher into a DLN student, integrating the distillation objective into a two-phase training loop without altering the architecture or the inference procedure. The teacher is discarded after training. Hence, the distilled model retains the interpretability, sparsity, and inference cost of a conventionally trained DLN. We investigate this approach through four variants: a joint hyperparameter search that optimizes the distillation weight and temperature alongside the architecture, a grid search over these two hyperparameters with the architecture fixed, and two single-phase variants that employ distillation during only the neuron-function or only the connection phase. On 20 public tabular classification tasks, distillation improves balanced-class accuracy at negligible cost to gate-level operation counts and model size, though the benefit varies by dataset. Performance improvements of up to 15% are observed, in some cases surpassing the teacher itself.

Fri 6 NovMachine Learning
The gist
Interpretable classifiers called differentiable logic networks (DLNs) run efficiently but can be less accurate than standard neural networks. The authors show that teaching a DLN by copying knowledge from a trained neural network can boost DLN accuracy without losing its benefits like simplicity and low runtime cost. They try several ways to combine this teaching process during DLN training and test on many tabular datasets. Sometimes, the improved DLN even beats the neural network it learned from, while still being easy to understand and fast to run.
Open → 2611.09024v1

Algorithms learn complex shapes with better efficiency under Gaussian noise

Beyond Multi-Index Models: Learning Intersections of Degree-2 PTFs

Abstract: We give an algorithm for PAC learning intersections of $k$ degree-2 polynomial threshold functions over $R^d$ under a Gaussian distribution. Our algorithm achieves classification error $ε$ using $d^{O(k)}2^{O_k(\log^3(1/ε))}$ time and samples, improving over the $d^{O(k^2/ε^2)}$ running time of the polynomial regression approach. To our knowledge, our result is the first to obtain a dimension exponent independent of the accuracy parameter, even for $k=2$. We complement our algorithmic result with an SQ lower bound showing that the complexity dependence of $d^{Ω(k)}$ is unavoidable. Lifting the input to polynomial features turns the target into an intersection of $k$ halfspaces. This suggests using the first two labeled moments for subspace recovery. In the lifted space, however, these moments can reveal nothing. Our starting point is a Gaussian comparison theorem showing that, for sufficiently regular defining quadratics, the labeled moments needed for subspace recovery are approximately preserved. To handle arbitrary defining quadratics, we establish a new structural result showing that every direction with significant resampling influence can be detected through degree-$O(k)$ moments. These results yield an initial $d^{O(k)}2^{\mathrm{poly}_k(1/ε)}$-time learner using adaptive restrictions. To improve the exponential dependence on $1/ε$, we develop a two-stage dimension-reduction approach that avoids the costly adaptive partitioning. After identifying a subspace containing all directions with large influence, we use generalized regularity and Gaussian comparison to recover the remaining relevant quadratic features through a new derivative-based moment operator. The recovered features support an intersection close to the target. Combining them with robust hinge-loss minimization gives our quasipolynomial-time learner.

Tue 3 NovMachine LearningData Structures and Algorithms
The gist
This work tackles the problem of learning complicated decision shapes made by combining multiple curved boundaries called degree-2 polynomial threshold functions. The authors develop a new algorithm that can learn these shapes more efficiently from data that follows a bell curve distribution. They improve on previous methods by reducing how much the running time depends on the accuracy we want. Their approach uses advanced mathematics to identify key directions within the high-dimensional data where the shape’s boundaries matter most. This allows building a model that closely approximates the original shape using fewer resources.
Open → 2611.05114v1

Constant-factor agnostic learning of affine halfspaces under log-concave data

Proper Agnostic Learning of Affine Halfspaces under Log-Concave Marginals

Abstract: We give a proper constant-factor agnostic learner for affine halfspaces under full-dimensional log-concave marginals. For arbitrary binary labels, the algorithm uses independent labeled samples and returns a single affine halfspace with error at most $C\mathrm{OPT}+\varepsilon$. Its sample and real-arithmetic complexity are polynomial in dimension, inverse accuracy, and the logarithm of the reciprocal failure probability. The mean and covariance of the marginal need not be known, and the sample complexity can be made linear in this logarithmic confidence factor. The main geometric result compares global disagreement with disagreement on a balanced restriction: for a minority halfspace of mass $p$, global disagreement is within universal constant factors of $p$ times conditional disagreement, provided the latter is sufficiently small. To construct the restriction from noisy labels, we estimate the minority cap's mean and covariance robustly, approximately minimize the classical zonoid-depth dual objective, and expand a superlevel set to a marginal quantile. The disagreement comparison cancels the noise amplification caused by conditioning and allows the learned halfspace to be used on the entire space. The analysis combines degree-four sum-of-squares moment bounds for log-concave laws with a translation-based reduction from balanced affine learning to homogeneous halfspace learning.

Tue 3 NovMachine Learning
The gist
This paper addresses the challenge of learning halfspaces — simple dividing lines or planes that separate data — when the data is noisy and follows a specific spread known as a log-concave distribution. The authors provide an algorithm that learns a single halfspace with error close to the best possible, without needing prior knowledge of the data’s mean or spread. They achieve this by carefully analyzing regions where classifiers disagree and robustly estimating the data distribution even under noisy labels. Their approach provides guaranteed accuracy with resources growing reasonably with problem size and desired confidence.
Open → 2611.04344v1

Kolmogorov-Arnold networks match multilayer perceptrons on clinical data

Kolmogorov-Arnold Networks vs. Multi-Layer Perceptrons: A Rigorous Statistical Benchmark on Clinical Tabular Data

Abstract: Kolmogorov-Arnold Networks (KANs) have recently been proposed as a promising alternative to Multi-Layer Perceptrons (MLPs). However, empirical evaluations on structured medical datasets remain scarce, and commonly reported metrics like AUROC can yield overly optimistic performance estimates on imbalanced clinical cohorts. In this study, we present a standardized benchmark comparing eleven architectures - nine KAN variants (FastKAN, WavKAN, ReLUKAN, ChebyKAN, TaylorKAN, GramKAN, JacobiKAN, LegendreKAN, and TabKAN), a tuned StandardMLP, and a Random Forest baseline - across seven heterogeneous medical tabular datasets spanning oncology, cardiology, nephrology, neurology, endocrinology, and obstetrics. Our experimental protocol employs a 3x5 Repeated Stratified K-Fold cross-validation scheme with nested Bayesian hyperparameter optimization and evaluates discrimination via Area Under the Precision-Recall Curve (AUPRC), calibration through the Brier Score, and computational efficiency. Formal equivalence testing is conducted using the Bayesian Correlated t-Test with a Region of Practical Equivalence (ROPE) at +/- 0.01. The results indicate that on datasets with severe class imbalance, dedicated KAN architectures (particularly TabKAN and ReLUKAN) can improve minority-class detection and achieve higher AUPRC than standard MLPs. Nevertheless, across pooled datasets, KAN variants demonstrate comparable overall predictive discrimination rather than a systematic advantage, while requiring 1.20x to 1.48x longer inference latency per sample and up to 12.5x more trainable parameters. Consequently, StandardMLP remains a strong baseline on the efficiency Pareto frontier, while KAN variants provide a viable alternative when precision on rare positive cases is the primary clinical objective. The code for reproducing the results is available at https://github.com/rafalwodzinski/kan_mlp_comparison.

Mon 2 NovMachine LearningArtificial Intelligence
The gist
Clinical datasets often have many more healthy cases than rare positive ones, making detection of rare cases difficult. The authors compared a newer type of neural network called Kolmogorov-Arnold Networks (KANs) with standard neural networks called multi-layer perceptrons (MLPs) on several medical datasets. They found that some KAN variants detect rare positive cases better but take longer to run and require more parameters. Overall, standard MLPs remain a solid choice for general use due to their efficiency. KANs may be better when catching rare, important cases is the main goal.
Open → 2611.02856v1

Robust learning of multi-index models under logconcave distributions

Robust Learning of Multi-Index Models under Logconcave Distributions via Weak Partial Learners

Abstract: We develop a new algorithmic framework for learning Boolean concept classes under logconcave distributions. Our framework applies to broad families of multi-index models and yields significantly faster robust learners than were previously known in these settings. It extends to partial classification and testable learning. Consequently, we obtain improved learners for several fundamental concept classes, including functions of halfspaces and multiclass linear classifiers, under logconcave marginals. Our starting point is a new boosting framework based on weak partial learners. Classical boosting approaches require often-intractable distribution-free weak learners or yield weak approximation guarantees. We show instead that weak learners for partial classification, where labels may be undefined on part of the domain, suffice for strong agnostic learning. The key challenge is that boosting requires a weak learner that succeeds under conditioned distributions that are not guaranteed to be logconcave. To address this, we develop a recursive moment-based partitioning procedure: given a logconcave distribution $X$ and a conditional version $X'$, we partition the space into polynomially many convex regions so that, on most regions, the low-degree moments of $X$ and $X'$ approximately agree. On each good region, a well-behaved multi-index model correlates with a low-degree polynomial, giving a weak hypothesis. Combining these hypotheses yields the required weak learner. A key ingredient of our approach is an efficient method for detecting moment discrepancy. Specifically, we show how to find a low-dimensional subspace that captures any non-trivial discrepancy between $X$ and $X'$, if one exists. Our approach combines spectral analysis of moment tensors with recent progress on isoperimetry for logconcave measures. This discrepancy-finding procedure may be of independent interest.

Fri 30 OctMachine Learning
The gist
Learning complex decision functions from data efficiently is a challenge, especially when the data follows specific mathematical shapes called logconcave distributions. The authors present a new way to combine simple, partial learners that only predict some of the data well, to build stronger, more reliable learners for these settings. Their method breaks the problem into parts where certain mathematical properties hold, making it possible to detect patterns and improve predictions faster than before. This approach enhances the understanding and efficiency of learning broad classes of models, including common types like linear classifiers.
Open → 2610.41961v1

Achieving best possible predictions across many loss measures

Omniprediction for All Mixable Proper Losses and Beyond

Abstract: Omniprediction concerns the task of making predictions that are simultaneously accurate with respect to different losses. Earlier results by [CV09] show that in an online learning setting, $O(\log(|\mathcal{F}||\mathcal{L}|))$ regret is achievable simultaneously against every predictor in a finite predictor class $\mathcal{F}$ and for every loss in a finite class $\mathcal{L}$ of mixable proper losses, while recent advances [OKK25, FP26] show that $\widetilde{O}(\sqrt{T\log(|\mathcal{F}||\mathcal{L}|)})$ regret is achievable simultaneously for every loss in a finite class $\mathcal{L}$ of bounded losses. In this work, we ask whether sublinear regret remains possible when $\mathcal{L}$ is an infinite class of proper losses. Our first result gives a negative answer in general: even when $\mathcal{F}$ contains only two linear predictors, it is impossible to achieve sublinear regret simultaneously for all bounded proper losses. We then show that $\widetilde{O}(T^{1/3}(\log|\mathcal{F}|)^{2/3})$ regret is achievable simultaneously for \textit{all} mixable proper losses and that this bound is optimal up to logarithmic factors. This optimal rate stands in stark contrast to the constant (in $T$) regret achievable for a finite class of mixable proper losses. Finally, we extend our algorithm to obtain a best-of-all-worlds result: in addition to the new regret guarantee for all mixable proper losses, the resulting algorithm simultaneously achieves the existing guarantees for finite loss classes. With appropriate choices of these finite classes, our algorithm simultaneously achieves constant regret for squared loss and log loss, $\widetilde{O}(T^{1/3}(\log|\mathcal{F}|)^{2/3})$ regret for all mixable proper losses, and $\widetilde{O}(\sqrt{T\log|\mathcal{F}|})$ regret for $0$-$1$ loss, all Lipschitz losses, all GLM losses, and other structured families.

Thu 29 OctMachine LearningData Structures and Algorithms
The gist
This work studies how to make predictions that perform well under many different ways of measuring errors (called losses). The authors show that when considering infinitely many possible losses, it’s impossible to be very accurate on all at once. However, they provide a method that achieves the best possible accuracy when focusing on a special class of these losses called mixable proper losses. Their method also combines different guarantees to perform well across many common loss types used in practice.
Open → 2610.41023v1

Deep linear networks reveal how early learning improves predictions

Investigating the Early Signatures of Feature Learning in Deep Linear Networks

Abstract: Deep neural networks learn data-dependent features, a capacity widely credited for their empirical success. However, explicitly characterizing what features emerge -- and when they improve generalization -- is challenging, since training is nonlinear and non-convex and the data is often anisotropic. Here, we give a closed-form account of feature learning for a deep linear network in a mean-field parameterization, trained by one gradient step from initialization on high-dimensional data carrying a low-dimensional (spiked) signal. In the infinite-width mean-field limit, one step updates each layer's feature kernel by a target-aligned rank-one spike, amplified by both the learning rate and the layer depth. We next assess how these features impact prediction through kernel ridge regression (KRR), finding that KRR on the learned feature kernel yields scalar inflation of the KRR estimator at initialization -- an "anti-shrinkage" mechanism that reinflates task-relevant directions suppressed by ridge, while also amplifying noise. To study when this inflation improves generalization, in the overparameterized proportional limit we derive closed-form expressions for the bias, variance, and generalization errors of these estimators in terms of key macroscopic parameters -- notably the spike strength and spike-target alignment. Crucially, we find that a one-step generalization advantage vanishes for isotropic inputs and appears when both the spike strength and an effective signal-to-noise ratio are sufficiently large, demonstrating that a successful first gradient step both requires and benefits from low-dimensional task structure.

Tue 27 OctMachine Learning
The gist
Understanding how deep neural networks learn important features from data is often tricky. This paper studies a simplified kind of network—deep linear networks—right after the first step of training. The authors find exact expressions describing how these networks start learning useful features, especially when the data has clear signals hidden in noise. They show that the first training step can improve the model’s predictions only when the data’s important patterns are strong and clearly aligned with the task. This helps explain when and why deep learning begins to work well early in training.
Open → 2610.37834v1

LinMix improves language model training by better data domain weighting

LinMix: Data Mixing as Linear Domain Attribution

Abstract: Language model performance depends strongly on the mixture of domains in the pretraining data corpus. Popular data mixing methods select these domain weights by training many small proxy models on random mixes and then fitting a regression model to predict performance from the mix. We recognize this practice as a domain attribution problem, where the regression measures each domain's contribution to the target metric. Through this lens, the standard Dirichlet sampling of proxy data mixes is a poor regression design that requires tuning and cannot support a linear fit for hundreds of domains. Building on classical optimal experimental design, we introduce LinMix, which samples mixes at the vertices of the capped simplex so that each domain is at its cap or absent. This hyperparameter-free vertex sampling decorrelates domain effects, enabling an interpretable linear model whose coefficients are the domain attributions. The optimal mix given by LinMix provably keeps only domains with positive attribution values and requires no search over candidate mixes. We further establish the optimality of vertex sampling, and prove that Dirichlet sampling inflates the attribution error when its mixes concentrate around the prior data mix. Pretraining experiments with 1B-parameter OLMo2 and 4B Qwen3.5 language models show that LinMix outperforms Dirichlet-based data mixing across reasoning, code, and math benchmarks, with relative gains of 46% on MBPP and 14% on GSM8K. At 288 fine-grained domains, only vertex sampling supports a linear fit, and LinMix improves benchmark accuracy by 11% on average over the best nonlinear baseline.

Tue 27 OctMachine LearningComputation and Language
The gist
Language models learn from lots of text data that come from different types of sources called domains. How much each domain contributes to training affects the model's performance. The authors show that common ways of mixing these domains rely on poor sampling strategies that make it hard to tell which domains truly help. They introduce LinMix, a method that selects extreme mixes where each domain is either fully included or not, making it easier to measure each domain's impact. Tests with large language models showed LinMix leads to better results in tasks like coding and math.
Open → 2610.37390v1

Time series forecasting improves by learning channel sharing and specialization

Learning What to Share: Expert Specialization for Time-Series Forecasting

Abstract: A central challenge in multivariate forecasting is deciding what to share across channels and where to specialize. Mixture-of-experts models offer flexible sharing, yet it remains unclear how much of their forecasting gains comes from added capacity or expert allocation. Through controlled comparisons across eleven datasets and three backbones, we show that informative channel assignments improve forecasting beyond matched shared capacity and random allocation. On frozen transformer backbones, persistent channel-specific mixtures outperform input-dependent routing on average. To examine when and how much specialization helps, we introduce GalaxyTS, a controlled synthetic dataset with exact shared and channel-specific forecasting references. It reveals how persistent channel mixtures exploit predictive differences concealed by short input windows and measures gains over matched shared corrections. These insights motivate Persistent Closed-Form Routing (PCR), a plug-and-play method that alternates exact conditional estimation of channel-specific mixture weights with gradient-based training of low-rank residual experts around a frozen pretrained forecaster. Averaged over four forecasting horizons, PCR-K4 achieves dataset-balanced mean squared error reductions of 3.86-14.25% relative to the three frozen backbones. In matched comparisons, PCR outperforms competing specialization methods on average while requiring substantially fewer trainable parameters.

Sat 24 OctMachine LearningArtificial Intelligence
The gist
Forecasting many related time series together is hard because it's unclear when to share information across them or treat each one separately. The authors studied models that split the work between expert parts and found that linking specific data channels to fixed experts helps more than random or input-based assignments. They created a synthetic dataset to understand when specializing experts benefits forecasting and introduced a new method that learns the best way to share and specialize with fewer parameters. Their approach improved accuracy across multiple datasets while keeping most of the forecasting model fixed.
Open → 2610.33822v1

Gradient clipping improves optimization under heavy tailed noise

Weak-$L^α$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

Abstract: Gradient clipping is a key tool for obtaining high-probability convergence guarantees for stochastic first-order methods under heavy-tailed noise. Existing analyses typically assume a bounded central moment of order $α\in(1,2]$, which excludes distributions with tails of order $u^{-α}$ at the critical exponent. In this paper, we consider smooth convex optimization under affine heavy-tailed noise, whose scale may grow with the gradient norm. We derive high-probability convergence guarantees for clipped SGD (Clip-SGD) and its accelerated variant Clip-SSTM under a weak-$L^α$ tail condition, which is strictly weaker than the corresponding moment bound. For $1<α<2$, the clipping bounds have the same powers of the clipping level as in the moment case, and both methods attain the additive stochastic rate $K^{-(α-1)/α}$. The gradient-dependent noise contributes a faster-decaying term of order $K^{-2(α-1)/α}$, up to logarithmic factors. At $α=2$, we characterize the additional logarithmic factor on the Lorentz scale and establish its necessity for the additive error at fixed confidence. We also analyze Clip-SGD with prescribed clipping levels and derive finite-horizon guarantees for reaching a neighborhood determined by the clipping bias. For additive noise, matching lower bounds show that this neighborhood is unavoidable, up to constants and the natural error cap. Numerical experiments with Pareto noise illustrate the theoretical results.

Tue 20 OctMachine Learning
The gist
When computers learn from data, random noise can make the learning unpredictable, especially if this noise has extreme values that happen often. The authors studied a method called gradient clipping that trims very large updates during learning. They show that this technique works well even when the noise is unusually heavy-tailed and depends on the learning steps themselves. Their results give guarantees about how quickly the learning process improves, and they confirm these findings with experiments using a type of extreme noise distribution.
Open → 2610.28107v1

Spatial temporal network improves forecasting by separating trend and seasonality

DST-Net: Decomposed Spatio-Temporal Network for Time Series Forecasting

Abstract: Series decomposition has emerged as a widely acknowledged paradigm in deep time series forecasting models. However, existing approaches predominantly employ a homogeneous modeling strategy for all subsequences, thereby overlooking the distinct inductive biases inherent in different components. We identify that the trend component exhibits dominant spatial coupling across channels but carries significant temporal redundancy, whereas the seasonal component is characterized by local temporal evolution but remains susceptible to attention distraction. To address this, we propose a novel disentangled architecture that tailors specific modeling strategies to align with the intrinsic properties of each component. For the trend branch, we design a Latent Graph Block, employing latent graph modeling to capture global topological dependencies and system-wide co-movements. Conversely, for the seasonal branch, we introduce a Gaussian Multi-scale Attention mechanism to mitigate attention distraction. By injecting learnable Gaussian priors, we impose structural constraints that compel the model to focus on multi-granular local contexts, effectively filtering out global noise and accurately capturing local evolutionary phases. Extensive experiments on multiple real-world benchmarks demonstrate the effectiveness of our proposed model.

Thu 15 OctMachine LearningArtificial Intelligence
The gist
Forecasting future data points in time series is tricky because trends and seasonal patterns act differently. The authors propose a new model that treats the trend and seasonal parts separately to handle their unique behaviors better. For trends, the model uses a graph-based method to capture relationships across different channels. For seasonal parts, it uses a focused attention method to avoid distractions from irrelevant data. Their approach shows improved accuracy on multiple real-world datasets.
Open → 2610.21123v1

Clipped stochastic gradient descent converges under heavy-tailed noise

Last-Iterate Convergence of Clipped SGD with Linear Decay

Abstract: In this paper, we study the last-iterate convergence of clipped stochastic gradient descent under heavy-tailed noise. We assume that the stochastic gradients are unbiased and that the noise has a bounded central $p$-th moment, where $1<p\le2$. For smooth convex objectives, a linearly decreasing learning rate gives a high-probability error bound of order $O(LR^2\log(1/δ)/T+σR(\log(1/δ)/T)^{(p-1)/p})$. We further establish last-iterate guarantees for nonsmooth composite objectives and smooth composite objectives with proximal updates. In addition, when a causal adversary replaces at most $q$ gradients, the same clipped update achieves a bound with $\log(1/δ)$ replaced by $\log(1/δ)+q$. The corresponding lower bounds for the clean and corrupted oracle models are also provided.

Tue 13 OctMachine Learning
The gist
This paper studies how a specific method for training machine learning models, called clipped stochastic gradient descent (SGD), performs when the data is noisy in a challenging way. The authors show that by carefully reducing the learning rate, the method can reliably approximate the best solution even when the noise has heavy tails (less predictable extreme values). They also prove that this approach works for different types of optimization problems and remains robust when some of the gradients are corrupted by adversaries. This means the technique can handle difficult data and some intentional interference while still converging well.
Open → 2610.18555v1

Random features achieve optimal learning in robust kernel regression

On the Optimality of Random Features under General Conditions: Misspecified Cases and Robust Learning

Abstract: Random features provide a standard approach to scaling kernel methods to large datasets by approximating the kernel with a finite-dimensional feature map. Existing works of random feature methods primarily focus on well-specified models and light-tailed noise conditions, leaving a gap in understanding their performance under more general conditions, such as misspecified models and heavy-tailed noise. In this paper, we provide a comprehensive and improved analysis of kernel ridge regression with random features (KRR-RF) under general conditions. In the light-tailed noise setting, we improve upon the worst-case uniform bounds of the feature map used in the previous works and lead to the minimax optimal learning rates with $2r+γ>α_0$, where $(r,γ,α_0)$ denote the regularity of the target function, the effective dimension of the kernel, and the embedding index of the RKHS, respectively. In the heavy-tailed noise setting, we leverage a recently developed Hilbert-valued Fuk-Nagaev inequality and extend the same optimal results under finite weak moment conditions. The additional cost to achieve robustness is a mildly stronger explicit sample-size requirement than in the light-tailed case. For both settings, our results recover the conditions random features budgets (up to log factors) required in the previous works. In the most benign case when $α_0=γ$, KRR-RF are minimax optimal for all $r \in (0,1]$. To the best of our knowledge, this is the first work to establish optimal learning rates for KRR-RF under weak moment conditions in the misspecified setting. Numerical experiments and a real data analysis validate the derived learning rates and demonstrate their practical relevance in the presence of mis-specification and heavy-tailed noise.

Tue 13 OctMachine Learning
The gist
Random features help make complex machine learning methods called kernel methods work faster on big data. The authors studied how well these random features work when the assumptions about the data are not perfect or when the data have unusual noise, like big outliers. They found the best possible error rates under these tricky conditions, even when using a limited number of random features. Their results show these methods remain reliable and accurate for a wider range of realistic problems than previously understood.
Open → 2610.18333v1

Statistical methods improve identification and estimation for nonsymmetric point processes

Identication and Estimation of Nonsymmetric Determinantal Point Processes

Abstract: Determinantal point processes (DPPs) provide tractable distributions for nite random sets, but the usual symmetric-kernel specication imposes nonpositive pairwise dependence. Statistical research on nonsymmetric DPPs, which can accommodate positive dependence, remains limited. We study identication and estimation for this class. First, we establish the equivalence of two sucient conditions for a nonsymmetric kernel to dene a valid DPP. Second, we develop the identication theory for nonsymmetric DPPs. Third, we develop cardinality-truncated maximum-likelihood estimation and estimation by averaged stochastic gradient descent. Under suitable identication and regularity conditions, we establish consistency and asymptotic normality for both estimators. Simulations examine identication, the eects of truncation, and the nite-sample performance of the estimators. We illustrate the methods with two cyber applications at markedly dierent sampling frequencies: weekly sector-by-breach data and second-by-second enterprise authentication data with more than ve million set observations.

Sat 10 OctMachine Learning
The gist
Determinantal point processes (DPPs) help model patterns where items occur randomly, but traditional methods only handle situations where items tend to repel each other. The authors study nonsymmetric DPPs, which can capture situations where items might also attract each other, a more flexible approach. They develop conditions for these models to be valid, create ways to estimate their parameters, and show the statistical properties of these estimators. Their methods are tested through simulations and applied to real-world cybersecurity data with very different time scales.
Open → 2610.14520v1

Kernel method tests how well conditional models match real data

Conditional Kernel Stein Discrepancy

Abstract: Kernel Stein discrepancies (KSDs) provide a versatile tool for comparing distributions. One of their main applications is in quantifying the goodness-of-fit (GoF) between a data-generating distribution and a prescribed target distribution. In this work, we study the related problem of conditional GoF quantification: given only a (possibly non-normalized) conditional target model, without information on the distribution of its covariates, and samples from a joint distribution, the goal is to assess how well the conditional distribution of the samples matches the target. To tackle this setting, we present a framework that allows lifting unconditional KSDs to the conditional setting through an operator-valued kernel on the covariate space, going beyond the known Euclidean case. We establish that our suggested statistic vanishes if and only if the conditional model and the true conditional distribution agree for almost all covariates and deploy it to test conditional GoF on smooth manifolds and on discrete spaces. Our experiments on level, power, and runtime demonstrate the viability of testing on these domains using the proposed statistic.

Thu 8 OctMachine Learning
The gist
Sometimes, we want to check if a prediction rule fits how real data behaves when it depends on some other information. The paper's authors created a tool that can check this fit even when they don’t know the exact distribution of that other information. Their method uses a math technique that works on many types of data, like points on curves or categories in sets. They also show their approach works well through experiments comparing accuracy and speed. This helps people test complex models in new kinds of data spaces.
Open → 2610.11863v1

New method improves neural network uncertainty estimates at scale

Scalable AI Uncertainty Quantification via Generalized Laplace Active Subspaces

Abstract: Reliable uncertainty quantification (UQ) is essential for deploying neural networks in scientific and high-stakes applications, but full Bayesian inference over the network parameters is computationally infeasible. We propose a low-rank generalized Laplace approximation for neural-network UQ based on a small number of data-informed curvature directions. Starting from a generalized Bayesian posterior defined through an empirical loss, we construct a local Gaussian approximation around a pretrained set of weights in this active curvature subspace. The posterior variances in the retained subspace are available in closed form, and the prior variance is calibrated by an empirical Bayes procedure. The generalized Bayesian formulation allows us to compare two posterior scalings: the standard Bayesian scaling associated with the summed negative log likelihood, and a mean-loss scaling in which the empirical loss is normalized by the number of data. A central finding is that the standard scaling induces a data-size dependent contraction of the posterior variance in the leading active directions. In regression problems, this can force the low-rank framework to retain additional weak-curvature directions in order to achieve nominal coverage of calibration data. When posterior samples are propagated through the non-linear network, these additional directions can degrade the coherence of the predictive intervals and shift the posterior predictive mean away from the pretrained model. In contrast, the generalized mean-loss scaling yields a more stable, lower dimensional active subspace and produces calibrated, coherent predictive confidence intervals. These results indicate that generalized Laplace active subspaces provide a practical and scalable route to calibrated uncertainty quantification in neural networks.

Thu 8 OctArtificial Intelligence
The gist
Knowing how uncertain a neural network is about its predictions is important but usually requires too much computing power. The authors present a new way to approximate uncertainty using fewer important directions in the network’s parameter space, making it faster and more practical. They show that adjusting how uncertainty is scaled helps keep predictions reliable and clear, avoiding problems that happen with older methods. This technique can help make AI safer and more trustworthy in real-world settings.
Open → 2610.11738v1

Lair Net improves predictions for complex tabular data patterns

LAIR-Net: Leaky Alignment-Impulse Residual Networks for Tabular Regression

Abstract: Deep randomized models fix hidden-layer parameters through random initialization and learn only closed-form readouts, typically adding depth by stacking random trans formations without target-aware control of hidden-state evolution. We propose LAIR Net, the Leaky Alignment-Impulse Residual Network, which mixes a shallow learned anchor into each hidden state through a leaky residual transition. We derive a depth uniform bound on input-perturbation sensitivity and use controlled simulations to attribute gains over a randomized baseline to the anchor rather than recursion or added capacity. Benefits emerge when a nonlinear target structure is learnable at the available noise level and diminish for nearly linear targets or dominant noise. Across 23 benchmark datasets, LAIR-Net achieves the best average rank among eight randomized networks and twelve conventional models, with relative performance associated with the same nonlinear-structure and noise quantities identified in simulation.

Thu 8 OctMachine Learning
The gist
This work looks at how to improve making predictions from tables of data with many columns. The authors introduce LAIR Net, a special kind of network that mixes a learned reference input at each step to better keep track of patterns. They show that this helps when the relationship in the data is complex and not just a simple line, especially when there's reasonable noise in the data. They tested their approach on many benchmark datasets and found it outperforms many other methods on average.
Open → 2610.11538v1

Provable recovery of deep input convex polynomial networks with active learning

Beyond Overparameterization: Provable Learning of Input-Convex Multi-Layer Polynomial Networks with Active Queries

Abstract: The theoretical understanding of multi-layer neural networks is largely confined to overparameterized settings, which obscure parameter identifiability and incur high sample complexity. Neural tangent kernel (NTK) provides a general theory for wide networks, but does not offer efficient sample-complexity guarantees. Recent feature-learning results go beyond kernel methods for single-neuron, multi-index, and hierarchical targets. However, the analysis is often restricted to shallow or specific architectures and to the overparameterized regime. We break this paradigm to achieve parameter-level recovery of deep target networks, albeit by using active data queries. Specifically, we study $L$-layer polynomial networks with even degree-$k$ monomial activations and nonnegative higher-layer weights. This structure makes the target network input-convex, while the optimization landscape remains highly nonconvex with respect to the parameters. Leveraging input convexity and active queries, we propose \textbf{ASPIRE} (\textbf{A}ctive \textbf{S}am\textbf{P}ling for \textbf{I}terative \textbf{R}ecovery via \textbf{E}igendirections), a layerwise sampling-based diagonalization algorithm that recovers all network parameters to $δ$-accuracy with sample complexity $ \widetilde O_{k,L}\left(d^{L^2+O(L)}δ^{-2e}\right) $ in polynomial time. To our knowledge, this is the \emph{first} parameter-recovery guarantee for deep target networks whose exponent grows only polynomially with depth, as well as the \emph{first} justification for the effectiveness of using high-quality data in neural network training, with a remarkably \emph{exponential} separation.

Sun 4 OctMachine Learning
The gist
Training very deep neural networks usually requires many examples, making it hard to figure out the exact network parameters. This paper studies a special kind of deep network where the output is made up of polynomial parts and the network is input-convex, a property that helps in analysis. The authors introduce an algorithm called ASPIRE that actively selects the right inputs to ask about, allowing it to efficiently and accurately recover the network’s parameters. This breaks new ground by providing guarantees on learning deep networks without needing excessive data, especially thanks to using carefully chosen queries.
Open → 2610.04999v1

Slower-growing attention kernels improve learning from heavy-tailed data

Attention Kernels for Learning Maps Between Heavy-Tailed Measures

Abstract: Operator learning on probability measures can be accomplished with transformers. For measures with polynomial tails, the exponential weighting in softmax can make the corresponding measure-level attention integrals diverge. This motivates replacing the exponential with slower-growing functions. We construct two benchmarks for operator learning on measures with closed-form targets. We use these benchmarks to study attention kernel growth and data transformation in post-norm transformers. Without data transformation, the softmax models exhibit ensemble collapse on both heavy-tailed benchmarks, while the three slower-growing kernels avoid collapse. Symlog preprocessing allows softmax to avoid collapse on the matrix inverse task but not on the sheared swap task. On the Gaussian control, all four kernels perform similarly. We also examine how sample size affects the sensitivity of empirical energy and Wasserstein distances to tail differences. These results support slower-growing attention kernels as an effective design choice for post-norm transformers learning from heavy-tailed ensembles.

Wed 30 SeptMachine Learning
The gist
Standard transformer models use a softmax function that can overemphasize large inputs, which causes problems when dealing with data that has heavy tails—rare but extreme values that appear more often than normal. The authors found that replacing the softmax with slower-growing functions helps avoid these issues when learning from such data. They tested these new functions on different tasks, showing they prevent the models from collapsing and work well, especially on difficult cases. This suggests a better way to design transformer-based models that learn from data with extreme variations.
Open → 2610.00564v1

Learning robust models that adapt to shifting data distributions

Learning the Robustness Mechanism with Bilevel Optimization

Abstract: We propose a distributionally robust learning framework where parameters defining the robustness mechanism are learned from held-out data instead of extensively tuned. Using bilevel optimization with both upper and lower level minimax problems, we create two instances of our framework to tackle setups with and without group labels in the training set. Theoretically, we provide sample complexity analysis for our robustness mechanism learning paradigm, showing that it achieves generalization guarantees comparable to exhaustive grid search while being more computationally efficient. Empirically, we evaluate our framework under a challenging setup when both intra-group and inter-group test distribution shifts occur at the same time, thereby demonstrating the efficacy and scalability of our method.

Mon 28 SeptMachine Learning
The gist
Machine learning models often struggle when the data they see changes in unexpected ways. This paper introduces a way to teach models to be more robust by automatically learning how to handle these changes using a special optimization process. Instead of guessing how to make the model tough against tricky data, the method learns the best settings from extra data. The authors show that this approach works well both in theory and practice, especially when different groups of data change in various ways at the same time.
Open → 2609.35541v1

Benign overfitting occurs for many norms beyond Euclidean in regression

Benign Overfitting for General Norms and Distributions

Abstract: Understanding why predictors can generalize despite interpolating noisy training data is a central puzzle in machine learning. Most work on such "benign overfitting" studies minimum-2-norm linear regression, reflecting the inductive bias of gradient descent. However, modern optimizers such as Adam and Muon use non-Euclidean update geometries, favoring solutions associated with other norms. Analyzing regression for non-Euclidean norms is substantially more difficult, with known results essentially limited to Gaussians. In this paper, we develop a method to analyze benign overfitting in linear regression for general norms and general (sub-Gaussian) distributions. As a special case, we prove that minimum-p-norm interpolation with p>1 can benignly overfit even for non-Gaussian distributions, under suitable conditions. Perhaps surprisingly, for the 1-norm, benign overfitting does not hold in general for well-behaved (but non-Gaussian) distributions, showing that existing positive 1-norm results rely crucially on Gaussianity. Our proof analyzes the geometry of the dual optimization problem, using concentration and central limit tools to show it is approximately Euclidean in many high-dimensional cases.

Sun 27 SeptMachine Learning
The gist
Predictors that perfectly fit noisy data can often still perform well on new data, a phenomenon called benign overfitting. Most studies focus on solutions that minimize the usual Euclidean norm, but the authors study general norms that reflect other optimization methods like Adam. They show that for many norms and data distributions, benign overfitting still happens, but for the 1-norm it can fail outside Gaussian settings. Their analysis uses geometric and probabilistic tools to understand why this happens in high-dimensional cases.
Open → 2609.33675v1

Automatic relevance determination explains input selection in gaussian process regression

Sparsity by Default: The Theory and Practice of ARD in Gaussian Process Regression for Variable Selection

Abstract: Automatic relevance determination (ARD) is the standard device for input selection in Gaussian process (GP) regression. By giving the covariance kernel a separate lengthscale for every input and learning those lengthscales by maximizing the marginal likelihood, ARD lets the data decide which coordinates matter: irrelevant inputs receive very large lengthscales and are effectively switched off. We trace this mechanism to the Bayesian Occam's razor embodied in the marginal likelihood, derive the gradient through which it prunes inputs, and emphasize that ARD delivers effective rather than exact sparsity. We review the algorithms used in practice and the rules that turn lengthscales into selections, and we survey the asymptotic theory, distinguishing the fixed-domain identifiability obstruction on the lengthscales from the high-dimensional selection-consistency guarantees recently established for hierarchical GP priors, and noting what remains open for plain ARD. We compare ARD with spike-and-slab priors, sparse axis-aligned and global-local shrinkage priors including the Bayesian lasso and horseshoe, penalized likelihood kriging, sensitivity and projection criteria, and additive kernels. We argue that ARD endures because of its seamless integration with kernel learning, universal software support, and low cost, and we close with its limitations and remedies.

Sun 27 SeptMachine Learning
The gist
Selecting which inputs matter in predictions can be tricky. The authors discuss a common method called automatic relevance determination (ARD) that helps identify and ignore unimportant inputs by adjusting parameters during training. They explain its strengths and weaknesses, compare it to other methods, and explore the theory behind why it works. Their review helps clarify when ARD effectively finds key inputs and what challenges remain.
Open → 2609.33550v1

Domain-adapted diffusion models improve conditional independence testing

Domain-Adapted Diffusion Models for Conditional Independence Testing

Abstract: Conditional independence (CI) is a fundamental concept in statistics and machine learning. Recent advances in conditional generative modeling provide flexible tools for generative-model-based CI tests, which rely on an estimated conditional distribution to generate randomized samples. However, errors in estimating this distribution accumulate in existing Type I error bounds, and consistency of the generative estimator alone does not guarantee asymptotic Type I error control. To address this limitation, we formulate conditional generative modeling as a domain adaptation problem and leverage auxiliary data from multiple source domains to improve estimation in the target CI testing domain. We propose Domain-Adapted Diffusion (DA-Diff), a multi-source domain adaptation framework for conditional diffusion models based on weighted empirical risk minimization over both target and source domains. We establish the convergence rate of DA-Diff and show how transferable source data can improve target-domain estimation through an increased effective sample size while controlling transfer bias. Building on DA-Diff, we further propose Domain-Adapted Conditional Independence Testing (DA-CIT) and show that its Type I error satisfies $P(p \leq α) \leq α+ o(1)$. Experiments demonstrate that DA-Diff improved conditional generation quality compared with transfer-learning diffusion baselines, while DA-CIT provides strong Type I error control and competitive power.

Sun 27 SeptMachine Learning
The gist
Conditional independence testing helps find out if one thing affects another, once you know a third thing. The authors noticed current methods can make mistakes because they don't estimate needed probabilities well enough. They treated this challenge like learning from different but related sources, improving estimates by borrowing knowledge from similar data. This approach leads to better control of errors and stronger test results compared to older methods.
Open → 2609.33490v1

Weighted conformal prediction improves coverage under shifting data

Sharp training-conditional coverage for conformal prediction under covariate shift

Abstract: Weighted split conformal prediction reweights calibration scores by the likelihood ratio between the test and training covariate distributions and guarantees marginal coverage under covariate shift. We study its coverage conditional on the calibration data. An elementary argument, based on a single concentration inequality at a fixed population quantile, gives explicit training-conditional bounds without unspecified constants, and shows that the relevant scale is not the supremum of the likelihood ratio but a variance proxy built from the chi-squared divergence of the shift and from the average of the ratio over the part of the test population, of probability equal to the miscoverage level, where it is largest. A two-point lower bound shows that the root-m rate and the chi-squared contribution are intrinsic to the shift. Run at an explicitly inflated level, the weighted quantile becomes a deterministic PAC prediction set. We compare it with randomized rejection sampling and with importance-weighted learn-then-test and, through a certified choice of a clipping level for the likelihood ratio, map the regime in which each gives the narrower valid set. The analysis extends to estimated likelihood ratios and to tail functionals estimated from an unlabeled source sample, which yields a fully finite-sample certificate.

Sun 27 SeptMachine Learning
The gist
When data changes between training and testing, it can be hard to trust predictions. The authors study a method that adjusts predictions to maintain correct coverage even when the data shifts. They provide clear mathematical guarantees about how well this adjustment works and identify the key factors influencing its accuracy. Their work also compares different techniques and offers practical ways to use estimated data distributions, making predictions more reliable.
Open → 2609.33456v1

Nonasymptotic error bounds for conformalized quantile regression under covariate shift

Conformalized Quantile Regression and Minimax Limits of Fixed-Score Calibration under Known Covariate Shift

Abstract: In this paper, we study nonasymptotic $L^p$ error bounds for interval length and conditional coverage in split conformalized quantile regression (CQR). Our bounds rely on local regularity conditions and accuracy guarantees for the estimated quantiles. We further instantiate our bounds for quantile regression with sparse ReLU neural networks. We also consider covariate shift, where the calibration and test covariates have different distributions, and derive nonasymptotic bounds for this setting. We obtain matching minimax upper and lower bounds in expectation for two constructed fixed-score calibration benchmarks under known covariate shift. The bounds match for every $p\in[1,\infty]$ in the scalar problem and for finite $p$ in the $K$-threshold problem; for the latter, a high-probability minimax lower bound holds for every $p\in[1,\infty]$.

Mon 21 SeptMachine Learning
The gist
Predicting accurate ranges for future events based on data is tricky when conditions change between training and testing. The authors studied how well a method called conformalized quantile regression works when the data used to adjust the method differs from the data where predictions happen. They provided detailed mathematical limits on the errors in these predictions, even when relying on neural networks. Their work helps understand how reliable these prediction intervals are under different conditions.
Open → 2609.24929v1

Low-rank classifiers limit noise influence under adversarial attacks

Tail-Weight Control and Localized Generalization in Nearly Low-Rank Adversarial Classification

Abstract: We study norm-constrained linear classification under Eu clidean adversarial perturbations in a Gaussian model with a low-dimen sional informative subspace and an independent noise tail. For bounded ramp loss, we prove that a principal-space witness with risk below one half forces every near-optimal predictor to have small tail weight. A path-specific density bound yields constants without requiring positive tail variance. Under isotropic principal covariance, we establish a unique population minimizer and joint local growth. Boundary normalization then removes the common attack penalty from centered margins, giving localized finite-sample guarantees governed by principal dimension and total tail energy. Globalized growth removes the entrance condition at weaker constants; a model-aware comparison retains local guarantees. Experiments with twenty paired repetitions show decreasing excess risk and tail use with sample size, and nearly unchanged behavior when tail dimension grows at fixed total energy. Pure-noise controls and optimizer diagnostics clarify the scope and limitations of these conclusions.

Sun 20 SeptMachine LearningArtificial Intelligence
The gist
This paper looks at how machine learning classifiers that use only a few important features handle attacks that try to fool them by changing inputs slightly. The authors show mathematically and with experiments that good classifiers tend to ignore noisy features and focus on important ones. They prove that under specific conditions, these classifiers keep their accuracy even when bad data tries to trick them. Their work helps understand why and when classifiers resist certain attacks and how sample size or number of noisy features affects this.
Open → 2609.23688v1

Aggregated posterior sampling improves data generation realism

Aggregated Posterior Predictive Checks for Generative Modeling

Abstract: Latent variable generative models are commonly fit using simple priors over latent variables, but draws from these priors often fail to produce realistic data. This failure is due to a mismatch between the prior and the aggregated posterior, the distribution of latent variables induced by the fitted model and the data. This mismatch is often viewed as evidence that the prior is misspecified and should be replaced. Alternatively, in modern generative models, a two-stage strategy is increasingly used where first, the model is fit, and second, the aggregated posterior is estimated (van den Oord et al.,2017; Rombach et al., 2022.). Synthetic data are then obtained by sampling from this aggregated posterior instead of the prior. To check such procedures, we introduce the aggregated posterior predictive check (APPC). Theoretically, we establish sufficient conditions under which the APPC is asymptotically calibrated. For probabilistic principal component analysis, we show that the APPC can remain calibrated under a misspecified latent prior when pervasive factors permit recovery of the signal space. Experiments with variational autoencoders show that aggregated posterior sampling improves generation for heavy-tailed and clustered data relative to Gaussian prior sampling while performing comparably to models with more flexible latent priors.

Thu 17 SeptMachine Learning
The gist
Generative models try to create realistic data by learning from examples, but they often use simple assumptions that make the generated data look unrealistic. The authors study a method that samples from a more accurate distribution of hidden factors learned by the model, rather than from the simple initial assumptions. They introduce a way to check if this improved sampling method works well and confirm it behaves as expected under certain conditions. Their tests show this approach makes generated data look better for complex types of data without needing more complicated assumptions beforehand.
Open → 2609.20999v1