Papers for

statistical modelers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Bregman geometry reveals new views on exponential family distributions

Bregman geometry in regular exponential families: dual representations and relative smoothness

Abstract: Regular exponential-family distributions are ubiquitous in statistics due to their appealing and convenient properties. Notably, they can be expressed in terms of Bregman divergences, revealing fundamental connections between statistics, convex analysis, and information geometry. However, this geometric structure has not been systematically exploited to characterize these distributions with respect to their expectation parametrization. Building on a projective characterization, we establish a general construction of equivalent dual Bregman representations, together with normalization and uniqueness results. These representations express log-densities as ordinary Bregman divergences in dual coordinate systems, providing geometric characterizations of the parameter space. So-called $β$- and $α$-scaled dual representations emerge as particular instances of this framework. We further derive dual Bregman representations of conjugate priors and establish that they preserve conjugacy. Applications to classical distributions illustrate the geometric structures underlying seemingly different statistical models. Moreover, we show that these representations provide natural reference geometries for establishing relative smoothness of the primal Bregman representation of the distribution, with explicit bounds that are sharp in the scalar and separable settings. An illustrative application to the Poisson distribution sheds new light on several works in the literature, providing complementary geometric justifications for reference geometries while deriving existing algorithms.

Fri 13 NovMachine Learning
The gist
Exponential family distributions are common tools in statistics with neat mathematical features. The authors study these distributions through the lens of Bregman geometry, which connects geometry with statistics and information theory. They develop new geometric representations that clarify how these distributions behave when described by their average values. These ideas also help understand related statistical models and improve methods for working with them, like in the example of the Poisson distribution.
Open → 2611.18480v1

Clustering variables in causal graphs improves analysis and clarity

Efficient and Practical Clustering of Variables in Causal Graphs

Abstract: Clustering variables in a graph may improve visual clarity and simplify causal inference. Harnessing the full potential of clustering requires an automated method to detect useful clusters in a graph. We introduce a novel class of clusters called conduits, and present a sound, complete and efficient algorithm for detecting all conduits in the graph. In addition to algorithmic detectability, conduits appear naturally in causal graphs stemming from real-world applications, enabling substantial size reduction. Furthermore, the structure of conduits enables the application of known identifiability results regarding clustering. We demonstrate these properties in a simulation study with real-world graphs.

Fri 13 NovMachine Learning
The gist
Figuring out which parts of a complex system work closely together can make understanding cause-and-effect easier and clearer. The authors introduce a new way to automatically find special groups of variables called conduits in these systems. Their method works quickly and completely finds all such groups, which often appear in real-world problems. This helps simplify big cause-and-effect maps and uses established results to analyze them better.
Open → 2611.17971v1

Learning mixtures of exponential family models with structured labels

Learning Mixtures of Exponential Family

Abstract: Motivated by causal inference and physical computing systems, we consider the problem of learning mixture of two bounded-support exponential-family distributions with a shared interaction parameter matrix and mixture component-specific linear parameter vectors. The unknown sequence of component labels belongs to a known structured class. We propose an estimator based on a convex loss that avoids the log-partition function; for each fixed label assignment, estimation is a convex optimization problem. We show that the conditional population loss uniquely identifies the interaction matrix and the linear parameter attached to each sample. For a global empirical minimizer, we establish finite-sample bounds on the interaction error in Frobenius norm and the average squared linear-parameter error. The bounds depend on the logarithm of the number of admissible label assignments, and, for fixed dimension and model bounds, imply consistency when this complexity grows sublinearly with sample size. Additional occupancy and separation conditions yield component and label recovery.

Wed 11 NovMachine Learning
The gist
This paper looks at how to learn from data made up of two combined sets of related groups, each with its own specific traits, but sharing some common interactions. The authors focus on mixtures where the sequence of group labels follows known patterns. They create a method that estimates the shared and individual qualities without using a complex calculation step, making it easier to solve. Their approach mathematically ensures the important parameters can be identified accurately from data, even if the group labels come from complex but known structures.
Open → 2611.15698v1

Improving Markov chain sampling by adding new transitions

Beyond Reweighting: Certified Improvements from New Transitions in Discrete MCMC

Abstract: Optimizing transition probabilities on a fixed grap may leave substantial limitations on the efficiency of reversible Markov chain Monte Carlo (MCMC). We develop a framework for transition-support design that considers both allowed transitions and their stationary flows. For a fixed target distribution, observable, and node-flow budget, we derive a necessary and sufficient dual condition for support augmentation to strictly reduce the optimal value of the asymptotic variance, together with computable primal--dual certificates that establish improvements over every admissible original-support reweighting. We also construct a family in which adding a single edge reduces integrated autocorrelation time by an $Ω(n)$ factor relative to optimal original-support weighting, demonstrating that structural limitations can impose arbitrarily large relative losses in estimation efficiency. To realize these gains, we investigate energy-informed edge selection and constrained stationary-flow optimization. Experiments on molecular and Ising state graphs distinguish the potential value of new transitions from the performance of kernels constructed on them: on identical augmented Ising supports, full weight optimization certifies strict improvements in 24/24 conditions, whereas degree-based Metropolis--Hastings weighting certifies none. Our results provide a principled approach to characterizing, certifying, and exploiting transition-support improvements in explicitly represented finite state spaces.

Wed 11 NovMachine Learning
The gist
Markov chain Monte Carlo (MCMC) methods are used to estimate probabilities by randomly moving through possible states, but their speed depends on how they jump between states. The authors show that simply adjusting jump probabilities on an existing network of states is often not enough to make sampling faster. They develop a way to identify when adding new possible jumps (transitions) can drastically improve efficiency and prove how to certify these improvements. Their tests on models like molecules and magnets demonstrate that adding certain transitions can cut down waiting times, making estimates more accurate.
Open → 2611.15668v1

Entropy and divergence concepts extended to non-invariant probabilities

Thermodynamic Formalism for Non-Invariant Probabilities: Entropy, Divergence, Channels

Abstract: Let $\mathcal{M}$ and $\mathcal{M}_σ$ denote, respectively, the sets of Borel and $σ$-invariant probabilities on the uncountable set $Ω=\{1,2,...,d\}^\mathbb{N}$. Given a Hölder Jacobian $J$, the dual Ruelle operator $μ\in \mathcal{M} \mapsto \mathcal{L}_{J}^* (μ)$ will play the role of a dynamical channel; generalizes the action of a stochastic matrix on probability vectors. Since $\mathcal{L}_{J}^*$ generally breaks $σ$-invariance, we introduce the \emph{ni-entropy} $h_{ni}(μ)$ for non-invariant measures, employing the concept of the expected value function on cylinders of size one. $h_{ni}(μ)$ differs from Shannon entropy but extends the Kolmogorov-Sinai entropy from $\mathcal{M}_σ$ to $\mathcal{M}$; $h_{ni}$ is concave and u.s.c. This leads us to the concept of ni-pressure $\mathcal{P}(A)$ of a potential $A:Ω\to \mathbb{R}$. $μ\in \mathcal{M}$ is not a density with respect to a dominating measure. We define an {\it ni-Kullback-Leibler divergence} $D_{ni}(ρ|| ν)$ not assuming absolute continuity. We investigate concepts of Information Theory such as Minimal Output Entropy, Donsker-Varadhan variational principle, a variational formula for $h_{ni}$, bounds for divergence, information projections, and their connections to the ni-pressure $\mathcal{P}$.

Tue 10 NovInformation Theory
The gist
The paper deals with probability measures that do not follow traditional invariance rules in dynamical systems. The authors introduce new mathematical tools, like ni-entropy and ni-Kullback-Leibler divergence, to describe and measure these non-invariant probabilities. These tools extend existing concepts used in dynamics and information theory, allowing for a broader understanding and analysis of systems where probabilities evolve under more general channels. They also study how these new concepts connect to measures of pressure and information flow.
Open → 2611.13904v1

Clique constraints improve exact causal learning efficiency in some cases

Clique Constraints for Exact Causal Structure Learning

Abstract: Exact score-based causal structure learning supports explicit structural constraints and optimisation certificates, but its branch-and-cut search can be computationally demanding. We augment exact causal structure learning methods with an upper bound on the maximum clique size of the learned undirected skeleton, enforced at integer candidates through lazy cycle and clique constraints. On static linear-Gaussian Erdős-Rényi and scale-free DAGs, we evaluate callback order, alternative clique-cut policies, incumbent repair, fractional separation, and Big-M bounds. We report causal-recovery metrics jointly with optimality gaps, primal and dual bounds, node throughput, MIPSOL events, and callback cost. Two variants receive particular attention: B3 selects the most violated minimal (k+1)-clique inequality at each callback, whereas C3 repairs an infeasible integer candidate by deleting violating edges and refitting affected regression columns using QR-based least squares. The results do not indicate that clique constraints uniformly accelerate optimisation. However, their benefit depends on graph density, sample size, and cut-management policy, while large collections of correlated constraints can reduce node throughput. Tighter valid Big-M bounds reduced the fixed-budget optimality gap in most tested settings, without necessarily changing SHD or F1 at a fixed time limit. These findings characterize the computational trade-offs of clique-constrained exact causal learning, and identify selective separation and incumbent repair as promising directions.

Mon 9 NovMachine Learning
The gist
Causal structure learning is a way to find cause-and-effect relationships from data, but it can be very slow for complex problems. The authors explore adding limits on groups of connected variables (cliques) to speed up this process. They test different ways to include these clique constraints and find the benefits depend on the data and settings. Sometimes it helps make learning faster, but sometimes it adds overhead. Their work clarifies when and how these constraints can be useful for precise causal discovery.
Open → 2611.12747v1

Polynomial density neurons clarify exact outputs for missing inputs

Evidence or Marginal? What Moment-Vector Propagation Computes in Polynomial Density Neurons

Abstract: When a neuron's input is lost, or known through a belief $q$, what should it output? For a neuron storing a joint polynomial density $N$ with input density $D$, two formulas compete. $p_\mathrm{CD}=E_q[N]/E_q[D]$ costs one matrix-vector product; Hierarchical Correlation Reconstruction introduces it as a constant-denominator approximation to $p_\mathrm{EU}=E_q[N/D]$. For a density model both are exact answers to different questions: $p_\mathrm{CD}$ is Bayesian updating with the belief as virtual evidence; $p_\mathrm{EU}$ installs the belief as a new marginal (Jeffrey's rule). With a missing input $p_\mathrm{CD}$ returns the model's conditional; $p_\mathrm{EU}$ generally does not. This post hoc reading is proved, and illustrated outside the pair family up to $k=8$ inputs; it has no pre-registered test, only one exploratory real-data pilot. Post hoc, closed forms fall within a factor 2 of all 112 measured gaps, and negative density begins near input correlation $1/k$. In a pre-registered synthetic study (up to 32 inputs), 5 of 15 hypotheses are supported, 2 inconclusive and 8 not supported, including all three on transfer beyond equicorrelated Gaussians. Post hoc, where $p_\mathrm{EU}$ is defined, truncation usually outweighs the gap for Beta beliefs in pair neurons; triple neurons under flat and data beliefs are exceptions.

Mon 9 NovMachine LearningArtificial Intelligence
The gist
The paper looks at how certain neurons that use polynomial models should respond when some input data is missing or uncertain. The researchers compare two ways to calculate the neuron output: one that conditions on the missing input as evidence, and another that treats the missing input as changing the overall input distribution. They show mathematically which approach corresponds to which interpretation and test this on various data setups. Their findings help clarify how these neurons propagate information when not all inputs are known exactly.
Open → 2611.12339v1

Improve sampling in complex high dimensional models with geometry aware monte carlo

Geometry-Aware Score-Repellent Monte Carlo

Abstract: The recently proposed Score-Repellent Monte Carlo (SRMC) method improves MCMC sampling by tilting the target density $π$ in $\mathbb{R}^d$ with the factor $\exp(-α\,θ_n^\top s(x))$, where the score function $s(x) = \nabla_x \log π(x)$, and $θ_n$ is the score average from past samples. However, a single scalar repellence strength $α\ge0$ cannot handle both the steep and the flat directions in anisotropic energy landscapes at once. We introduce \emph{Geometry-Aware SRMC} (GA-SRMC), which generalizes the Euclidean alignment to $θ^\top M s(x)$ for any symmetric positive-definite matrix $M$. A stochastic-approximation central limit theorem under a trace constraint on $M$ identifies the trace-normalized inverse Fisher $M_{\rm opt}\propto S^{-1}$ as the unique minimizer of the worst-direction covariance bound, elevating the Fisher information $S=\text{Cov}_π(s,s)$ from a proposal-side preconditioner (e.g., FisherMALA, natural gradient) to the bound-optimal repulsion-side matrix, with a single online estimate $\widehat S_n$ serving both roles at no additional computational cost. The full-Fisher online metric $M_n$ converges almost surely to $M_{\rm opt}$ and satisfies a CLT, providing a theoretical guarantee for the online implementation of the GA-SRMC framework. On an $8$-target simulation spanning the condition number $κ\in[1.3,100]$ and dimension $d\in[2,785]$, GA-SRMC dominates SRMC in ESS by $\boldsymbol{4.3\times}$ on anisotropic Gaussian ($κ=100$) and $\boldsymbol{6.8\times}$ on MNIST under the MALA baseline, by $\boldsymbol{16\text{--}32\%}$ on three well-conditioned logistic-regression posteriors, and by up to $\boldsymbol{+48\%}$ under FisherMALA; gains peak on the most anisotropic targets, and posterior-mean MSE is lower than with SRMC in all $16$ full-Fisher implementations.

Fri 6 NovMachine Learning
The gist
Sampling methods help computers explore complicated mathematical landscapes, but can struggle when these landscapes have both steep and flat directions. The authors improve an existing method called Score-Repellent Monte Carlo by using geometric information to adapt how the sampling avoids past locations. This makes the method much better at navigating tricky shapes, especially when directions behave very differently. Their approach updates automatically during sampling and consistently performs better in tests on various real and simulated problems.
Open → 2611.08909v1

Query limits exposed for advanced sampling from complex distributions

Towards the Query Complexity of Sampling under Strong Log-Concavity and Log-Sobolev Inequalities

Abstract: We study the query complexity of sampling from smooth distributions on $\mathbb R^d$ using first-order oracles. For $d$-dimensional strongly log-concave targets with condition number $κ=2$, we prove that any randomized sampler requires $\exp({Ω(\log^{1/3} d)})$ queries even from a constant warm start. This rules out polylogarithmic dimension dependence and strictly separates strongly log-concave sampling from Gaussian sampling. For targets with $L$-smooth potentials satisfying an $α$-log-Sobolev inequality with $κ= L/α$, we prove an $Ω(κ)$ query lower bound for $d\gtrsim\log^2κ$. This shows that log-Sobolev sampling is strictly harder than strongly log-concave sampling. Our proofs construct instance-dependent tests that target samples typically pass but outputs of algorithms making few queries typically fail. For strongly log-concave targets, we embed a sequence of hidden Gaussian into high-degree tensor polynomial with smooth gates, forcing any algorithm to uncover the layers sequentially. The log-Sobolev construction hides a constant fraction of the target mass in angular caps while controlling the log-Sobolev constant via delicate estimates.

Wed 4 NovComputational ComplexityData Structures and Algorithms
The gist
Sampling means picking values from complicated probability shapes, which helps in statistics and machine learning. The authors show that for certain smooth and nicely curved high-dimensional shapes, any sampling method needs a lot more steps than previously thought, especially as dimensions grow. They prove this by designing tricky versions of shapes where sampling becomes surprisingly hard. This means some common assumptions about easy sampling from these shapes are incorrect, and more effort is needed for certain cases.
Open → 2611.05813v1

Estimation risks clarified when nuisance parameters vanish

Exact Global Minimax Risk near Nonidentification

Abstract: We study squared-error estimation when a nuisance parameter is unidentified on a submodel but affects a target that vanishes there. For models with higher-order profile expansions, we give sufficient conditions under which the largest critical-path minimax risk is the exact asymptotic minimax risk over the full parameter space. An efficient observable feature, negligible calibration error and stable recovery yield one estimator attaining the leading constant uniformly, with the profile coordinates unknown. The estimator uses calibration estimates, a local feature mean and an empirical recovery feature. The constant is expressed through a bounded-normal decision problem. We verify the conditions in two models. In centered Gaussian logistic errors-in-variables regression with one surrogate and two unknown variances, we determine the fifth-order global coefficient risk and its exact constant. A sparse extension separates variance-decomposition uncertainty from direction estimation and specifies when the scalar constant is preserved. In a degree-balanced replicated-network model, we determine the sixth-order global edge-contrast risk when block proportions are unknown. Triangles identify a parameter combination; a bowtie supplies the additional information needed for the contrast.

Fri 23 OctMachine Learning
The gist
This paper studies how to make the best possible estimates in statistical models when some related parameters are unclear or vanish, which often complicates analysis. The authors find exact ways to measure the smallest possible errors in such cases and provide concrete estimators that achieve these limits. They demonstrate their approach on two complex models involving noisy data and network structures, showing how to separate uncertainty sources and identify critical parameters for precise estimation.
Open → 2610.32546v1

Stationary infinite memory chains yield adaptive empirical confidence bounds

Empirical concentration for chains with infinite memory

Abstract: Concentration inequalities for time series usually involve unknown quantities measuring dependence. This prevents their direct use as confidence bounds, or as criteria for learning algorithms. Recent works proved empirical concentration inequalities for stationary Markov chains, where the unknown quantities are estimated. We extend this approach beyond the Markov case, by considering stationary chains with infinite memory. We first establish a bounded-differences inequality on bounded Polish metric spaces under a summable Wasserstein contraction condition. We then estimate its dependence coefficients from a single trajectory, using either transition counts when the state space is finite, or derivatives of kernel conditional-density estimates for a continuous state space. We give general error bounds and illustrate their dependence on the memory tail. Finally, a fixed multiplicative safety factor turns these estimators into empirical concentration bounds. While some regularity assumptions are required to make the bounds empirical, our procedure remains fully adaptive in the sense that it does not require the knowledge of the constants involved in these assumptions. The empirical confidence bounds derived from our results are asymptotically as tight as the theoretical ones.

Fri 23 OctMachine Learning
The gist
Predicting time series data is hard because the data points depend on a long history, which is often unknown and complicated. The authors develop a new way to measure how much past data influences the present, even when the history goes infinitely far back. They then show how to estimate this influence from just one observed data sequence, making predictions more confident and reliable. Their method adapts automatically without needing to know hidden constants, improving how we handle complex time-dependent data.
Open → 2610.32265v1

NPMLE achieves optimal rates for estimating Gaussian mixtures

Minimax Optimality of the NPMLE for Gaussian Mixtures in Hellinger Distance

Abstract: We analyze the problem of estimating univariate Gaussian mixture densities using the nonparametric maximum likelihood estimator (NPMLE) based on $n$ independent and identically distributed samples. We show that the unconstrained NPMLE achieves the minimax optimal rate $\log(n)/n$ in squared Hellinger distance, when the true mixing measure is sub-Gaussian. In addition, we show that when the true mixing measure has compact support, the NMPLE constrained to a compact interval also achieves the minimax optimal rate, which is slightly faster and of order $\log(n)/(n \log \log n)$. Our proof consists of sharply characterizing the local Hellinger bracketing entropy of the class of finite Gaussian mixtures. One of our key technical innovations is to sharply bound the $L^2$ bracketing entropy of balls of bounded-degree polynomials, which may be of independent interest. As an application, we establish the regret-optimality of $g$-modeling procedures based on the NPMLE in the Gaussian Empirical Bayes problem with bounded priors.

Wed 21 OctInformation TheoryMachine Learning
The gist
Estimating a mix of bell curves (Gaussian mixtures) from data can be tricky. This paper shows that a certain estimation method, the nonparametric maximum likelihood estimator (NPMLE), performs as well as theoretically possible in measuring how close the estimate is to the true distribution. The authors prove this works well even when the data comes from mixtures with bounded or sub-Gaussian spreads. They use advanced mathematical tools to understand the complexity of these mixture models and also show how this improves beyond statistics, helping in areas like Empirical Bayes techniques.
Open → 2610.30109v1

Binomial distribution has highest uncertainty among certain dependent sums

The binomial distribution maximises entropy among sums of negatively related indicators with fixed mean and support

Abstract: We show that the binomial distribution supported on $\{0,1,\ldots,n\}$ and with mean $np$ for some $0<p<1$ maximises entropy within the class of sums of negatively related indicator random variables with this mean and support. Our results generalise the main result of Yu (2008), who showed that this binomial distribution maximises entropy within the smaller class of ultra-log-concave random variables of order $n$. Our proof makes use of a Markov chain construction of Yu (2008), together with a stochastic ordering argument.

Mon 19 OctInformation Theory
The gist
This paper shows that when you combine multiple yes-or-no outcomes that are somehow negatively related, the most unpredictable total number you can get has a binomial distribution shape. The authors extend earlier work that proved this was true under a narrower set of conditions. They used some clever mathematical tools like Markov chains and probability orderings to prove their result. This helps us understand how randomness works when outcomes influence each other negatively.
Open → 2610.26543v1

Sample size requirements grow rapidly for weak latent-variable signals

Sample complexity of weak-signal latent-variable models

Abstract: We study parametric estimation in latent-variable models in the weak-signal regime, where a small signal-strength parameter $β$ controls the dependence of the observations on the unknown parameter. At $β=0$, this dependence disappears, so all parameter values induce the same observed distribution, which we call the null law. We develop a unified framework for analyzing sample complexity in the weak-signal regime by expanding the likelihood ratio around this shared null law in powers of $β$. We prove that if the unknown parameter is first globally identifiable at order $r$ in this expansion, then estimation to fixed accuracy requires at least on the order of $β^{-2r}$ samples. We further identify a likelihood-separation stability condition that is necessary for any estimator to attain this rate and, under mild regularity conditions, sufficient for both maximum likelihood and method-of-moments estimators to attain it. We also study the regime in which the target accuracy vanishes together with the signal strength, and show that global localization and local refinement may be governed by different likelihood orders, leading to distinct statistical scales. Finally, we apply our framework to broad classes of algebraically structured latent-variable models, including additive-noise channels and exponential-family models.

Sat 17 OctInformation Theory
The gist
This paper looks at how many samples you need to estimate hidden patterns in data when the signal is very faint. The authors find that if the signal appears only at a certain level in a math expansion, you need a lot more data to get accurate estimates, growing quickly as the signal weakens. They also show conditions that let standard estimation methods reach these limits. Their work applies to many statistical models involving hidden variables, like those with noise or exponential family distributions.
Open → 2610.24770v1

ReLU networks approximate smooth functions with optimal error rates

Optimal approximation of smooth functions by ReLU neural networks

Abstract: We study uniform approximation of the Hölder ball $\mathcal H_1^s([0,1]^D)$ by fully connected ReLU networks. Except in the one-dimensional Lipschitz case, existing constructions remain separated from the information-theoretic efficiency limit. This article closes the gap for every $s>0$ and $D\ge1$: networks with width $O(N)$, depth $O(L)$, and parameter magnitude $O(1)$ attain uniform approximation error of order $(N^2L^2\log N)^{-s/D}$. No relation between width and depth is required. The construction uses new encoding and decoding mechanisms that exploit higher-order finite-difference bounds for local Taylor coefficients of the target function. As a statistical consequence, empirical risk minimizers over the corresponding clipped network classes attain the minimax regression rate $n^{-2s/(2s+D)}$ without additional logarithmic factors. The paper also gives a detailed exposition of the key approximation techniques, covering both established and new mechanisms.

Thu 15 OctMachine Learning
The gist
Approximating smooth functions accurately with neural networks is important but often inefficient. The authors show how to construct ReLU neural networks that approximate these smooth functions as well as theoretically possible, improving on previous approaches. Their method uses cleverly designed network architectures and mathematical techniques to capture local details of the functions. This leads to better predictions from data in statistical tasks without extra penalties.
Open → 2610.22693v1

Wide neural networks converge functionally at rate inverse square root of width

Lifting Finite-Sample Gaussian Approximation of Wide Neural Networks to Functional Convergence

Abstract: Randomly initialised neural networks of width $m$ converge to Gaussian processes as $m \to \infty$, with two types of quantitative bounds characterising this convergence. Finite-sample bounds compare the network with its Gaussian limit at a fixed finite sample set, on an arbitrary domain, in the quadratic Wasserstein distance $\mathcal W_2$, achieving a convergence rate of $m^{-1/2}$ with explicit constants. However, these constants grow with the number of samples, so the bound becomes uninformative as the sample set is refined. Functional bounds remove that dependence by comparing the two as random functions, at the price of spherical input, metrics weaker than $\mathcal W_2$, or slower rates. In this paper, we therefore lift a finite-sample bound to a functional one. The lifting refines the sample set until it is dense in the domain, so the bound must first be made independent of the number of samples. We achieve this by measuring the error per sample, which makes the constant uniform. For a polynomial activation and any compact input set, this gives a rate of $m^{-1/2}$ in the supremum norm, with an explicit constant.

Tue 13 OctArtificial IntelligenceMachine Learning
The gist
When neural networks get very wide, their behavior becomes similar to a type of random process called a Gaussian process. Previous studies measured this similarity only on limited sets of examples or with some trade-offs. The authors show how to extend these finite-example measures so that the similarity holds over all inputs, keeping the accuracy of comparison and a clear rate of convergence as the network width grows. This makes the relationship between wide neural networks and their Gaussian limits clearer and more practical for any input.
Open → 2610.18166v1

DisCo algebra reveals limits of summarizing joint probabilities

DisCo Algebra: Why Disintegration and Combination Are Not Inverses

Abstract: We present DisCo algebra---a compositional framework for joint probability built from a symmetric cooperad (disintegration) and a symmetric operad (combination) on the bare family $\mathcal{P}_n=\mathrm{Prob}([0,1]^n)$. No probability integral transform, no external summary functions, no enrichment of objects with conditional kernels. The disintegration side uses block-wise joint CDFs as intrinsic, measure-determined 1D summaries, yielding a lossy compression. The combination side reads the same summaries and reconstructs via regular conditional measures, yielding a lossless embedding. We prove that combination followed by disintegration is the identity on admissible inputs, but disintegration followed by combination is not---the information loss is exactly the conditional KL divergence between the true conditional distribution and the conditionally independent product. This asymmetry reveals a hard limit on how much cross-block dependence can be captured by scalar block summaries.

Sat 10 OctMachine Learning
The gist
This paper studies how to break down and rebuild joint probability information using simple summaries. The authors show that compressing data with these summaries loses some details about how parts depend on each other. Rebuilding from the summaries can perfectly restore original data only when done in one direction, but not the other. This finding highlights a fundamental limit on summarizing complex dependencies with simple numeric summaries.
Open → 2610.15006v1

Kernel method tests how well conditional models match real data

Conditional Kernel Stein Discrepancy

Abstract: Kernel Stein discrepancies (KSDs) provide a versatile tool for comparing distributions. One of their main applications is in quantifying the goodness-of-fit (GoF) between a data-generating distribution and a prescribed target distribution. In this work, we study the related problem of conditional GoF quantification: given only a (possibly non-normalized) conditional target model, without information on the distribution of its covariates, and samples from a joint distribution, the goal is to assess how well the conditional distribution of the samples matches the target. To tackle this setting, we present a framework that allows lifting unconditional KSDs to the conditional setting through an operator-valued kernel on the covariate space, going beyond the known Euclidean case. We establish that our suggested statistic vanishes if and only if the conditional model and the true conditional distribution agree for almost all covariates and deploy it to test conditional GoF on smooth manifolds and on discrete spaces. Our experiments on level, power, and runtime demonstrate the viability of testing on these domains using the proposed statistic.

Thu 8 OctMachine Learning
The gist
Sometimes, we want to check if a prediction rule fits how real data behaves when it depends on some other information. The paper's authors created a tool that can check this fit even when they don’t know the exact distribution of that other information. Their method uses a math technique that works on many types of data, like points on curves or categories in sets. They also show their approach works well through experiments comparing accuracy and speed. This helps people test complex models in new kinds of data spaces.
Open → 2610.11863v1

Symbolic expression learning improves understanding of complex unnormalized distributions

Symbolic Density Estimators for Unnormalized Distributions

Abstract: Estimating the symbolic or analytical form of probability density functions (PDFs) from observed samples is a fundamental challenge in statistical and computational modelling. This process is critical for deriving interpretable and generalizable relationships characterizing the underlying phenomenon. Traditionally, this estimation depends strongly on domain expertise and prior field-specific knowledge, with experts selecting appropriate functional forms or parametric families based on empirical evidence and theoretical understanding. The coefficients of these forms are then typically determined through parameter estimation. In this paper, we develop a framework for estimating symbolic expressions of unnormalized distributions from observed samples using domain-specific prior knowledge, such as the range of interactions and a predefined set of primitive functions. We integrate deep generative models with symbolic regression (SR), incorporating inductive biases, such as factorizing large distributions, to keep the problem tractable. The deep generative models we examine include likelihood-based models, viz., flow models, and score-based models. Experiments show the effectiveness of the proposed framework for estimating density functions for multivariate toy distributions as well as lattices from computational physics, namely, XY model and $φ^4$ theory. When applied to the renormalization problem in $φ^4$ theory, the proposed framework estimates compact symbolic approximations of the hamiltonian function at different scales directly from samples, yielding expressions that may be challenging to derive using traditional perturbative or analytic approaches in nonperturbative settings.

Wed 7 OctMachine Learning
The gist
Finding simple formulas to describe complicated probability patterns is very hard. The authors created a method that combines smart deep learning with symbolic math to discover clear, exact formulas describing these patterns, even when usual assumptions don’t apply. They tested it on examples from physics, where it helped understand complex behaviors without traditional calculations. This approach makes it easier to see the core rules behind complicated systems.
Open → 2610.10807v1

Gaussian process designs reveal what model assumptions can be tested

What Can a Gaussian Process Design Test

Abstract: A Gaussian process (GP) model can agree with the data for two reasons: its assumptions are right, or the chosen inputs could never have shown that they are wrong. The distinction can be checked from the design before any responses are observed. Every model implies relations that its noiseless responses must satisfy at the chosen inputs, such as the middle value lies on the line through its two neighbours. For GPs built from finitely many features, these relations are exactly the null space of the kernel matrix. Gale duality gives them a geometric interpretation, in which each observation has a vector and the smallest groups of observations that can expose an error are the circuits. For other kernels the relations become soft: response patterns may be improbable under the prior rather than algebraically impossible. A standard test then combines two kinds of evidence. Structural evidence comes from a violated relation and grows without limit as the noise falls. Prior-based evidence only says that a departure is improbable under the prior. With all inputs at the two ends of an interval, for example, a GP can reject a straight line against a large curvature, but only because the implied intercept is improbable, never because curvature was seen. In simulations the predicted power matched the observed rejection rates. Choosing the next input by predicted power raised the power against a localised discrepancy from 0.48 to 0.72, against 0.51 when choosing by predictive variance, and a grid in two dimensions contained exact tests of additivity that a Latin hypercube lacked. The test itself is classical. The contribution is the prospective reading of that test: before observing the responses, the design already determines what kind of contradiction it can produce.

Wed 7 OctMachine Learning
The gist
Gaussian process (GP) models fit data either because their assumptions are correct or because the test points can’t expose errors. The paper shows how, before collecting any data, the choice of inputs determines what kinds of model mistakes can be detected. Some patterns in the data must hold true if the model is right, and violations of these patterns provide strong evidence against the model assumptions. The authors also show how to choose test points to increase the chance of spotting discrepancies.
Open → 2610.10122v1

Adaptive method improves causal discovery with few data points

AdaPS-LiNGAM: Adaptive Predecessor Selection for Linear Non-Gaussian Acyclic Models under Small-Sample Settings

Abstract: Causal discovery becomes particularly challenging when the available sample size is small relative to the number of variables. This challenge also arises in the linear non-Gaussian acyclic model (LiNGAM), an identifiable framework for causal discovery from observational data. DirectLiNGAM estimates a causal order, which arranges variables so that causes precede their effects, by sequentially identifying an exogenous variable and removing its linear effect from the remaining variables. We establish a structural limitation of this procedure: when the number of variables exceeds the sample size, repeated residualization necessarily becomes degenerate before the full causal order can be determined. Our analysis further reveals that each residual can be reconstructed using only a graph-determined subset of variables already placed earlier in the causal order, termed the active boundary. This result motivates AdaPS-LiNGAM (Adaptive Predecessor Selection LiNGAM), which reconstructs each residual directly from the original observations using an adaptively chosen sparse subset of those earlier variables. The same subset-selection principle is also applied to the final pruning step for edge estimation. Experiments on synthetic data demonstrate that AdaPS-LiNGAM provides accurate causal-structure recovery in sample-limited settings and degrades more gradually as the sample size decreases.

Wed 7 OctMachine Learning
The gist
Finding cause-and-effect relationships between variables is hard when you have only a small amount of data, especially when the variables are many. The authors studied a popular causal method called DirectLiNGAM and found it breaks down when there are more variables than samples. They discovered a new way to pick a smaller, relevant group of earlier variables to better predict each step, leading to a new method named AdaPS-LiNGAM. Tests on simulated data show this method gives more accurate causal insights when data is limited.
Open → 2610.09782v1

Mean field fluctuations reveal uncertainty in neural network training outputs

Fluctuations of Nonlinear Observables in Mean Field Neural Network Training

Abstract: Mean field limits describe the training dynamics of wide neural networks through the evolution of the empirical distribution of their parameters. Although functional central limit theorems characterize the asymptotic fluctuations of this distribution, quantities of practical interest are typically nonlinear observables of the parameter distribution rather than the distribution itself. In this work, we show how these mean field fluctuations propagate to finite dimensional nonlinear observables for shallow neural networks trained by stochastic gradient descent. Working in the weighted Sobolev space in which the limiting fluctuation process is constructed, we apply a functional Delta method under ordinary Fr{é}chet differentiability, without requiring Lions derivatives with respect to the measure variable. We obtain a central limit theorem for the observables and, under a suitable representation of their differentials, an explicit covariance formula inherited from the underlying mean field fluctuation theory. We also study whether prescribed quantities of interest can be recovered from the selected observations. Under a constant rank assumption, we prove that a quantity of interest factors locally through the observation functional if and only if, throughout a neighborhood, the kernel of the differential of the observation is contained in that of the quantity of interest. Thus, a differential condition expressed directly in the ambient Sobolev space yields an exact nonlinear local factorization. These results provide a framework both for quantifying finite-width uncertainty on observable, statistically or physically meaningful quantities and for assessing whether the chosen observations contain the information required to identify them.

Wed 7 OctMachine Learning
The gist
Training wide neural networks involves many parameters that change over time, and scientists use mathematical models to understand this process in large networks. This paper looks at how small random variations in these parameters affect the specific quantities people actually care about, like predictions or measures derived from the network. The authors provide a way to calculate how these fluctuations influence these quantities and determine when the observations fully capture the information needed. This helps quantify uncertainty when working with networks of finite size and understand if the chosen data reveals everything important.
Open → 2610.09768v1

Reflected anchored Langevin algorithm samples nonsmooth constrained distributions

Reflected Anchored Langevin Algorithms

Abstract: First order Langevin algorithms for constrained sampling in machine learning, such as projected Langevin Monte Carlo which are based on discretizations of reflected Langevin dynamics, require differentiable log densities that limits their applicability. This paper introduces reflected anchored Langevin dynamics (RALD), a reflected diffusion that converges to non-differentiable targets on constrained domains. The method uses a smooth anchored reference potential and multiplies the drift and noise covariance of its reflected Langevin dynamics by the same state dependent scaling factor. Its Euler-Maruyama discretization with projection gives reflected anchored Langevin Monte Carlo (RALMC) algorithm. We prove explicit convergence bounds and iteration complexity for RALMC in the 2-Wasserstein distance to the target distribution. Numerical experiments are provided to illustrate the theoretical predictions and the empirical performance of the method.

Wed 7 OctMachine Learning
The gist
Sampling from complicated data distributions that have hard boundaries and sharp corners is a common problem in machine learning, but existing methods need smooth shapes to work well. The authors introduce a new approach called reflected anchored Langevin dynamics that can handle sampling from nonsmooth and constrained spaces by combining a smooth reference shape with a special scaling in the process's movement and noise. They develop a practical algorithm from this idea and prove it gets closer to the target distribution effectively. Their experiments confirm the theory and show the method can handle cases traditional methods struggle with.
Open → 2610.09522v1

Likelihood approximation using chained normalizing flows for simulation inference

A perspective note on likelihood approximation and inference for complex simulation models using a chain of aggregated normalizing flows

Abstract: We present a new perspective on the problem of likelihood approximation within the framework of simulation-based inference that promotes scalable and controllable simulation routines for large-scale data analysis, allows efficient parameter space exploration or smooth interpolation in high-dimensions and, thus, supports valid statistical treatments of hypothesis testings as well as uncertainty quantification. In particular, we consider a chain of $n$-aggregated normalizing flows for likelihood approximation scheme, where a set of upfront replicated observation datasets from the forward complex simulation model pass through the first set of bijective transformations, and then subsequently pass to the other sets of bijective transformations. Here, we assume that, for any $k \in \{1,\,2, \ldots, n\}$, the parameters corresponding to the first $k$ sets of bijective transformations are estimated sequentially, in some sense of optimality, for constructing flexible probability distributions, regardless of the remaining $(n-k)$ sets of bijective transformations. Moreover, our objects of interest are to highlight two complementary mathematical arguments that leverage an informatics-theoretic formalization, based-on empirical likelihood estimators under moment restrictions, and a sequential decision-making paradigm, with mixing distributions, for updating and aggregating the estimated parameters of the overall normalizing flows. As a by-product, the framework provides a reliable surrogate model, conditioned on the model parameters defining the forward computational simulation, that allows samples generation, with statistical powers, and facilitates computationally tractable scheme in the Bayesian paradigm for inference, hypothesis testings and uncertainty quantification.

Mon 5 OctMachine Learning
The gist
Complex simulations often generate data that is hard to analyze because their likelihood functions are difficult to compute. The authors propose using a series of linked mathematical transformations called normalizing flows to approximate these likelihoods more efficiently. By sequentially estimating parameters of these flows, their approach allows for faster and more flexible exploration of possible model settings. This method also supports statistical tasks like hypothesis testing and measuring uncertainty in predictions.
Open → 2610.07391v1

Guidance estimation improves steering of diffusion model sampling

WEIRDO: WEak resIdual Regularized DOob's h-transform diffusion alignment

Abstract: We study the problem of estimating the guidance that steers the distribution learned by a diffusion generative model toward a tilted target $q_0 \propto w\,p_0$ at inference time. Relying on the stochastic optimal control approach, we observe that the exact drift correction is the gradient of the logarithm of Doob's $h$-function, and we study the problem of estimating it from a sample. In the present paper, we assume that the score of the pretrained model is available, that the tilting weight is bounded and positive, and that the reference distribution has a bounded support, no smoothness of the weight is required. Introducing a penalized least-squares risk in which the penalty is the residual of the space-time harmonicity equation satisfied by the $h$-function, measured in a dual Sobolev norm, we derive high-probability bounds on the squared error of the resulting guidance estimate. Since the penalty vanishes at the target, the estimator is free of regularization bias, and in favourable scenarios its rate of convergence is faster than the minimax rate of estimating first-order derivatives of a smooth regression function. Assuming that $w$ is bounded and positive with $\mathbb{E}_{p_0}[w^{-\mathrm{s}}] < \infty$ for some $\mathrm{s} \in (0,\infty]$, and that the reference data are compactly supported, we prove that the guidance is estimable in squared $L^2$ at rate $\varepsilon_n^{\mathrm{s}/(\mathrm{s}+4)}$, where $\varepsilon_n = n^{-2(β-1)/(2(β-1)+d)}.$ We also transfer the obtained bounds to the total variation distance between the marginals of the estimated and the exactly guided samplers, and illustrate the performance of the suggested approach with numerical experiments.

Wed 30 SeptMachine Learning
The gist
Diffusion generative models create data by gradually adding and removing noise. Sometimes, you want to guide these models to generate data that fits a slightly different target distribution. The authors study how to estimate the best guidance signal at inference time to steer the model properly, even when the exact target is complicated. They propose a new method that uses a penalty based on a mathematical condition called space-time harmonicity, which helps estimate this guidance more accurately and with good theoretical guarantees. Their approach works under minimal smoothness assumptions and is supported by experiments.
Open → 2609.39531v1

Information splits explain probability deviation events geometry

An information identity reveals the geometry of deviation events

Abstract: In a deviation event, the empirical measure of $n$ independent draws lands in a set of distributions. The exponent of its probability splits, at every $n$, into two nonnegative terms. The first is the rate: $n$ times the relative entropy, from the population, of the distribution of a typical draw under the event. The second is the dependence that conditioning induces among the draws, measured by their total correlation. Their relative sizes depend on the geometry of the set. This paper focuses on this exact split and on the geometry that the interplay of its two terms reveals. The dependence vanishes exactly when the event confines every draw to a single set, and the classical rate of that constraint is exact. On a set invariant under a compact group that fixes the population and leaves no invariant set of intermediate probability, the rate vanishes; uniform convergence over a hypothesis class that such a group permutes is one. On a half-space whose threshold sits a fixed number of standard errors above the mean, the fraction of the exponent that is dependence tends to a function of the event's probability alone. On a set split into pieces, the dependence is the pieces' average plus $n$ times the Jensen-Shannon divergence of their marginals, minus the entropy of their weights. The split extends to a general reference law. When the information projection is such a law, the split gives the exact value of the expectation in the exponential change of measure to the projection.

Tue 29 SeptInformation Theory
The gist
When you randomly draw samples, sometimes the results deviate from what you expect. This paper finds that the chance of such deviations can be exactly split into two parts: one measuring how different the most likely outcome is from the usual, and another measuring how the samples become dependent when you know the event happened. The balance between these parts depends on the shape of the event you consider. The authors also explore how this split works under different conditions and sets, revealing hidden structure in how deviations occur.
Open → 2609.38159v1

Inferring dynamic changes from distribution snapshots requires multiple time points

Beyond Gradient Flow: Identifiability and Recovery from Distribution Snapshots

Abstract: Inferring dynamics from snapshots of evolving distributions is fundamentally underdetermined: the Fokker-Planck equation constrains the drift $F$ only through its score-weighted divergence $\nabla\cdot F+F\cdot\nabla\logρ$, leaving a $ρ$-solenoidal gauge invisible to any single-time constraint. Time-indexed transport formulations cannot resolve this ambiguity: every admissible marginal path admits a curl-free explanation, minimum-action reconstruction selects it, and marginal fit alone cannot distinguish dynamically inequivalent explanations. Requiring one autonomous field to explain several marginals instead makes part of the hidden circulation visible as $\nabla\logρ$ changes across marginals. Separating instantaneous Fokker-Planck source constraints from the snapshot experiment, we show that the source constraints identify the field modulo the kernel of a stacked score-weighted divergence operator. For generic Gaussian shape variation, source constraints at $K\ge m$ time points in intrinsic dimension $m$ eliminate every polynomial gauge direction, whereas finitely many density snapshots alone admit aliasing; we give the obstruction explicitly. At a Gaussian anchor, for Sobolev smoothness $s$ and $n$ samples per time point, we derive a conditional lower rate $(nK)^{-2s/(2s+m+1)}$ for the tangent snapshot experiment, with a matching upper rate in a degreewise benchmark. Strong-form fitting is non-orthogonal to score error and cannot be repaired by spectral filtering. Instead, we estimate using smooth test functions while retaining the known diffusion term, and derive a finite-sample bound that separates sampling error from fixed-grid quadrature bias. Planted-circulation experiments confirm the predicted gauge contraction and expose a design tension between cross-slice information and covariance-aware whitening.

Mon 28 SeptMachine Learning
The gist
It is very hard to figure out how something changes over time if you only look at snapshots of where it is at different moments. The authors show that looking at just one snapshot doesn’t provide enough clues because some hidden details remain invisible. By comparing multiple snapshots over time, it becomes possible to uncover parts of the hidden dynamics. They also explain how to estimate these changes reliably using mathematical tools and provide limits on how well you can do this depending on the number of snapshots and samples. Their experiments confirm the theory and reveal trade-offs in the process.
Open → 2609.35060v1

New method finds lower-dimensional shape behind complex data sets

Recovering Lower-Dimensional Semialgebraic Support of a Measure from its Moments

Abstract: Recovering probability measures from their moments has numerous applications, esp. in connection with the method of moments in statistics and optimization. In the setting where measure need not be finitely atomic, but its support is known to be compact and semialgebraic with codimension at least one, the problem is still open. We combine moment-matrix kernel information with the Christoffel--Darboux kernel to provide a discrete approximation of the support. To validate the proposed approach, we test our algorithm on analytically computed moments and pseudo-moments arising from polynomial optimization problems without unique global minimizers. This complements well-known recent work on recovery of measures with algebraic support, where the kernel of a moment matrix can reveal polynomials vanishing on the support, and on recovery of sufficiently regular full-dimensional supports, where estimators constructed by thresholding the Christoffel--Darboux kernel are known to converge asymptotically to the support.

Sun 27 SeptMachine Learning
The gist
Sometimes, people want to understand the shape or boundary where certain invisible things, called measures, live. This problem gets tricky when the shape has lower dimensions, like a flat surface inside a 3D space. The authors came up with a new way to approximate these hidden shapes using math tools called moment matrices and Christoffel--Darboux kernels. They tested their method with examples where the shape isn’t just points but has more complex structures, showing their approach can work beyond previous simpler cases.
Open → 2609.33420v1

Normalizing flows learn complex distributions without example samples

SIMANF: Sample Free Learning of Unnormalized Distributions via Simulated Annealing in Normalizing Flows

Abstract: Efficiently learning and sampling from high dimensional, multimodal unnormalized distributions without target samples remains a challenging problem. Although normalizing flows can generate samples efficiently, training based on the reverse KL divergence using only the unnormalized target density may suffer from mode collapse. We introduce SIMANF, a sample free framework that integrates simulated annealing with normalizing flows. SIMANF progressively transforms the target distribution from a smooth initial form to the original target distribution and trains the flow sequentially across these stages. By transferring the learned representation between stages, the method promotes mode coverage while progressively capturing finer features of the target distribution. Following annealing, a final refinement stage combines the reverse KL divergence with an importance weighted forward KL objective using samples generated by the flow. SIMANF requires no target samples during training and uses only the unnormalized density. We demonstrate its effectiveness on Many-Well distributions and high dimensional Scalar Phi4 lattice field theory distribution.

Sat 26 SeptMachine Learning
The gist
Learning and sampling from complicated probability shapes that don’t have examples is hard. The authors created SIMANF, a method that slowly changes a simple shape into the complicated target shape while training a model called a normalizing flow at each step. This gradual change helps the model cover all parts of the target shape better and avoid ignoring important details. After this process, the model is fine-tuned using samples it generates itself, even though no real example samples are given during training. The method works well on challenging test problems with many modes and high dimensions.
Open → 2609.32279v1

Formula reveals loss patterns in shallow relu networks with bias

Population loss in shallow ReLU networks: Bias & families of critical points

Abstract: The main result presented is a formula for the population loss in the student-teacher kernel model that is applicable to shallow ReLU networks with bias. This extends previous work of Choo and Saul (2009) and Brutzkus and Globerson (2017). The formula makes essential use of Owen's T-function. The necessary theory of the T-function is given and a high precision coding using MPFR for the T-function, based on an algorithm of Komelj (2023), is available on request. It is shown that various families of spurious minima described in past papers of Arjevani and the author extend to biased networks and that the loss is always strictly decreased when bias is added. The change in landscape geometry caused by adding bias appears to be relatively mild. Only the simplest examples are described in this paper where it is assumed that the number of inputs is equal to the number of neurons (this restriction is for reasons of length). A review of relevant previous results on unbiased networks is included. Aside from Gaussian statistics, the main mathematical tools and ideas come from analytic geometry (analytic and subanalytic sets, the Curve Selection Lemma).

Fri 25 SeptMachine Learning
The gist
The paper finds a formula that helps understand errors in simple neural networks that use ReLU and include a bias term. This extends earlier work that looked only at networks without bias. The formula involves a special mathematical function called Owen's T-function, which the authors explain and provide precise code for. They show that adding bias reduces the error and only slightly changes the network’s error landscape. The work focuses on the case where inputs and neurons have the same number, keeping the math simple.
Open → 2609.30661v1

Discrete diffusion model improves simulation of spin systems and sampling

Discrete Diffusion Models via Evolving Variational Autoregressive Networks

Abstract: Conventional score-based diffusion models learn scores without representing normalized densities, whereas tractable normalized models support both sampling and direct likelihood evaluation. A recent tensor-network approach provides such a representation but is largely restricted to low-dimensional lattices. Here we introduce a discrete diffusion model that parameterizes normalized probability distributions using variational autoregressive networks. Explicit Markov jump operators govern the forward noising and reverse denoising dynamics, extending discrete diffusion models with normalized distributions to spin systems on higher-dimensional lattices. We apply this framework to the two- and three-dimensional Ising models across ordered, critical, and disordered regimes, accurately computing thermodynamic quantities including free energy, energy, and magnetization. We further integrate the framework with Monte Carlo sampling, using adaptive diffusion steps to maintain high acceptance rates even at low temperatures while enhancing sample diversity. These results establish a neural-network framework for the discrete diffusion model with normalized probability distributions.

Wed 23 SeptMachine Learning
The gist
Models that describe complex systems often need to generate realistic examples and calculate how likely these examples are, but doing both is hard. The authors created a new method that represents probabilities exactly using special neural networks and simulates changes in systems like magnetic spins in multiple dimensions. Their method calculates important physical properties accurately and can work well with existing sampling techniques to produce diverse examples, even under difficult conditions. This approach helps better understand and simulate physical systems with complex interactions.
Open → 2609.27306v1

Copula operad framework links dependence and entropy additively

Copula Operad and Copula Entropy

Abstract: We construct a symmetric operad $\mathfrak{C}$ on the class of all multivariate copulas, with operadic composition given by Sklar substitution. This places the hierarchical combination of dependence structures into an algebraic framework. Restricting to absolutely continuous copulas whose densities lie in $L\log L$, we prove that copula entropy is additive under composition, making it an additive character on suitable finite-entropy suboperads. We exhibit explicit closed suboperads (bounded, $L^{p}$, boundary-growth) and show by counterexample that componentwise $L\log L$ does not imply closure under composition.

Thu 17 SeptInformation Theory
The gist
Understanding how variables relate to each other can be tricky, especially when combining multiple relationships at once. The authors created a mathematical framework that describes how these relationships, represented by something called copulas, combine together. They showed that a measure of uncertainty called copula entropy adds up neatly when combining copulas this way. However, not all types of copulas behave nicely under this combination, which they demonstrated with examples.
Open → 2609.20512v1

MaxEnt probability distributions on spheres inform models of cognition

Maximum Entropy Probability Distributions on Spheres with Fixed Mean Busemann Function and Holomorphic-Information-Geometric Model of Cognition

Abstract: In the first half of the paper we revisit the question regarding MaxEnt probability distributions on spheres. We derive families of MaxEnt distributions on spheres in real and complex vector spaces with fixed expected Busemann function (energy). As a particular case, we deduce sub-families on canonical energy levels where inverse temperature equals the dimension of the sphere. In the second part we focus on the information manifold of canonical MaxEnt distributions on the sphere in the complex vector space. This manifold is isomorphic to the Bergman ball. We introduce the reproducing kernel on this manifold and use the RKHS theory to elaborate the model of cognition. In particular, we state the principle of minimal cognitive effort in RKHS and demonstrate its dual relationship with the MaxEnt principle for probability distributions on the boundary sphere.

Thu 17 SeptInformation Theory
The gist
The paper studies the best ways to spread probabilities on spheres when we fix certain energy-like values. The authors find special families of these distributions in both real and complex spaces. Then they link these distributions to a geometric space called the Bergman ball, using advanced math tools called reproducing kernels. They propose a new way to think about cognition as minimizing effort in this geometric space, showing how it relates to the idea of maximum entropy for probabilities on the sphere.
Open → 2609.20410v1

Compressed subspaces speed up uncertainty analysis in large models

Compressed Active Subspaces for Scalable Bayesian Inference

Abstract: Active subspace methods provide a framework for quantifying predictive uncertainty in high-dimensional models by identifying and performing inference along parameter directions that have the greatest influence on the model output. However, the construction of active subspaces requires storing many full-dimensional model gradients, which becomes prohibitive as model size increases. We address this limitation by proposing Compressed Active Subspaces (CAS), a scalable approach that first maps the model parameters to a compressed space using a structured isometric embedding and then constructs the active subspace within this reduced parameterization. Our approach substantially reduces the memory required for active subspace construction and enables Bayesian inference for large models where standard active subspace methods become impractical. We demonstrate the scalability of CAS on neural networks of increasing size while maintaining predictive performance and robust uncertainty estimates.

Thu 17 SeptMachine LearningArtificial Intelligence
The gist
High-dimensional models often have many parameters, making it hard to understand which ones really affect the outcome. The authors propose a way to shrink these parameters into a smaller, compressed set without losing important information. This makes it easier and less memory-intensive to estimate uncertainty in predictions, especially for big models like neural networks. Their method still gives good predictions and reliable uncertainty estimates while being much more scalable.
Open → 2609.19539v1

Probability distribution evolution explained through least action principle

Physics of Information Geometry - Part I: Principle of Least Action on the Probability Simplex

Abstract: We develop a least-action framework for describing how a probability distribution can evolve from an equilibrium state to a prescribed nonequilibrium state under constrained incremental changes. Taking a Gibbs distribution as the equilibrium reference, the framework gives a direct physical meaning to the geometry of the probability simplex: distance from equilibrium corresponds to nonequilibrium free energy, while changes between successive distributions carry an informational kinetic cost. The Pythagorean structure of relative entropy then provides the central insight of the work. It shows that intermediate distributions chosen via sequential information projections can reduce the kinetic cost of large transitions and establishes an energy-conservation-like relation between the kinetic expenditure along a path and the free energy accumulated in reaching the target distribution. Motivated by this geometry, we construct a greedy least-action path through successive information projections, obtain a closed-form characterization of each projection through the Lambert W function, and establish a finite-step performance guarantee. We further show that state-dependent costs can be incorporated naturally by reshaping the underlying Gibbs reference, providing a thermodynamic interpretation of path penalties as modifications of the effective energy landscape. Together, these results provide a unified view of distributional evolution through least action, information geometry, and nonequilibrium thermodynamics.

Tue 8 SeptInformation Theory
The gist
The paper looks at how a probability distribution changes from a balanced state to an unbalanced one using a physics-inspired framework. It treats moving through possible probability states like a journey where distance measures how far from balance you are and movement costs information energy. By breaking big changes into smaller steps chosen carefully, less energy is used overall. The authors also connect this to thermodynamics, showing how the shape of the underlying probability landscape affects the costs to move through it.
Open → 2609.08285v1

Markov chains produce work only with non-Gaussian models in Bayesian inference

Thermodynamic Cyclic Processes with Markov Samplers in Bayesian Inference

Abstract: The concept of Markov chain Monte Carlo (MCMC) cycles, an analogy to cyclic processes in heat engines, is presented in order to examine Bayesian inference problems. In this effort, we develop adaptive ensemble schedulers that allow the tuning of external parameters of a Bayesian canonical ensemble during an MCMC run, realising the MCMC cycles in practice. We run these cycles on different statistical models. As a fundamental insight, we find (both theoretically and in practice) that such systems can produce a non-zero net work output if and only if the considered model is non-Gaussian. As such, they may serve as a measure of non-Gaussianity in Bayesian inference, which we test on an example from supernova cosmology.

Mon 7 SeptArtificial Intelligence
The gist
The paper looks at Bayesian inference—the process computers use to make decisions based on data—and draws an analogy to heat engines that do cycles. The authors develop a way to run Markov chain Monte Carlo cycles by tuning parameters during sampling. They find that these cycles can produce a measurable effect, or "work," only when the statistical model is non-Gaussian, meaning it doesn't follow the simple bell curve. This discovery offers a way to tell if data or models are non-Gaussian, tested on a cosmology example with supernova data.
Open → 2609.07660v1