Papers for
data compression engineers
Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.
Less information can sometimes lead to better decision accuracy
When Does Less Information Pay? An Exact Finite Atlas of Representation, Payoff, and Cost
Abstract: A representation that retains one bit about a four-state world can solve a task perfectly, while another that retains 1.5 bits on average reaches only 75% accuracy. The difference lies in which distinctions each representation preserves. We examine this relationship in a uniform four-state world by enumerating all admissible representations and binary action rules. Constructions are compared both by replacing one component while holding the others fixed and by choosing the best action rule separately for each module. We consider three tasks, their mixtures, and four separate hypothetical cost tariffs. In some comparisons, the representation retaining less world information has the higher payoff; in others, the more informative representation does. There are also pairs with different information amounts but equal payoff--that is, equal task accuracy. If a representation preserves all existing distinctions and adds new ones, its best zero-cost payoff cannot decrease. Lower-information advantages therefore depend on which distinctions are retained. Equal counts of opposing advantages under representation replacement follow from coding symmetry. In broader comparisons, the shares depend on how repeated realizations are counted. A different task set or resource price can change which member of a fixed pair performs better. These results are exact, complete censuses of the specified finite sets of constructions and comparisons. The atlas provides a reproducible worked model for examining how action constraints, counting rules, and cost assumptions affect comparisons. It illustrates established distinctions between information quantity and decision value rather than proposing a new general law of their relation. $I(W;R)$ measures retained world information, and payoff measures task accuracy.
Rate distortion functions explained with new construction approach
A Constructive Theory of Closed-Form Rate-Distortion Functions
Abstract: Rate-distortion functions are defined by a variational problem whose analytic solution depends on the source distribution and distortion measure. We develop a posterior-based construction framework using conditional entropy as a common distortion coordinate. Building on the rate-entropy formulation, we establish entropy-level attainability for discrete sources with finite entropy and absolutely continuous sources with integrable log densities. We give a constructive rate-entropy coding proof for finite-alphabet sources and such continuous sources, without a second-moment assumption. Marginal compatibility ensures that a candidate posterior and output distribution recover the source, while a Gibbs representation and a kernel mass bound certify attainment of maximum conditional entropy. The negative log posterior identifies candidate fixed distortions. Excess rate equals a conditional entropy gap and decomposes into budget slack and posterior mismatch. Convolution decompositions and conjugate posteriors provide two construction paths. They recover established results, including the Gamma-source rate-distortion function under Itakura-Saito distortion, and yield a closed-form parametric expression for zeta sources under divisibility-constrained logarithmic multiplicative distortion. Exact expressions are obtained on specified intervals for negative binomial sources under one-sided counting distortion and Gamma sources under one-sided mean error. A residual-count coordinate for Poisson entropy budgets and rate-entropy parametrizations of Gamma and Dirichlet counting channels connect model parameters to remaining uncertainty. The framework unifies distortion selection, compatible-channel construction, and optimality verification.
Algorithms for finding smallest substring compression schemes efficiently
Exact and Approximation Algorithms for Computing Smallest SESs and BMSs
Abstract: Compression schemes represent repetitive strings compactly by using repeated substrings. Computing the minimum size of a representation of a given input string is important for compression and for understanding relationships between compression schemes, although this task is NP-hard for several schemes. In this paper, we introduce exact and approximation algorithms for finding smallest substring equation systems (SESs) and bidirectional macro schemes (BMSs). Specifically, for an input string of length $n$, we give deterministic and randomized algorithms for computing a smallest SES using $O^*(4^n)$ and $O^*(2^n)$ time and space, respectively. We also give a deterministic algorithm for computing a smallest BMS using $O^*(2^n)$ time and space. We further give deterministic and randomized $(1+α)$-approximation algorithms for smallest SESs using $O^*(4^{e^{-α}n})$ and $O^*(2^{e^{-α}n})$ time and space, respectively, for every $α\ge 0$. A conversion from SESs to BMSs gives $4(1+α)$-approximation algorithms for smallest BMSs with the same time and space bounds. All time bounds hold in the worst case, and the randomized algorithms return correct answers with high probability. Our algorithms adapt several approaches to exact and approximation algorithms for combinatorial problems, such as dynamic programming over determinant expansions and a greedy algorithm for monotone submodular functions.
Binary tokenization is hard but allows approximate efficient solutions
Tokenization: Hardness and Bicriteria Approximation Algorithms
Abstract: We study Binary Tokenization: given a corpus of binary strings over $\{0,1\}$ and a vocabulary budget $K$, select at most $K$ substrings (called vocabulary) to minimize the total number of tokens needed to represent the corpus. The primitive symbols $0$ and $1$ are always available, and each document must be segmented separately. Letting $n$ denote the total corpus length and $M$ for the maximum document length, we prove that, for every fixed $\varepsilon\in(0,1)$, Binary Tokenization problem does not have polynomial-time $n^{1-\varepsilon}$-approximation algorithms unless $\mathsf{P}=\mathsf{NP}$. To complement the hardness result, we give a polynomial-time $(γ,γ)$-bicriteria algorithm, with $γ= \sqrt{M}\log n$: The algorithm outputs a vocabulary of size at most $γK$ with number of tokens at most $γ{\rm OPT}_K$, where ${\rm OPT}_K$ is the optimal number of tokens under the original budget $K$. We also study a storage-budget variant, in which the sum of the selected substrings' lengths is bounded by $B$. For $2\le B<n$, we obtain an $(O(\log n\log B),O(\log n\log B))$-bicriteria approximation algorithm.
Improved precision in lossy compression data limits and code sizes
Third-Order Asymptotics for Fixed-Length Lossy Compression of Discrete Memoryless Sources
Abstract: This paper determines the third-order asymptotics of the logarithm of the minimum codebook size for fixed-length lossy compression of discrete memoryless sources. For an excess-distortion probability constraint $\varepsilon\in(0,1)$ and every interior distortion level, we establish a third-order correction of $\frac12\log n+O(1)$, where $n$ is the blocklength. This sharpens the general bounds of Kostina and Verdú [IEEE Trans. Inf. Theory, 2012]. The main technical innovation lies in the converse for positive dispersion. Under a reverse change of measure, we retain the exponential weights associated with tilted information and distortion slack and control them jointly using the multivariate weighted-strip estimate of Scarlett, Martinez, and Guillén i Fàbregas [Proc. IEEE ISIT, 2015]. A pairing argument verifies the required covariance nondegeneracy uniformly over all reconstruction words whose distortion balls meet a typical set, yielding the matching converse.
Tight optimal bounds found for one-shot channel simulation and representation
The Optimal Constant for One-Shot Channel Simulation and Strong Functional Representation Lemma
Abstract: We prove a tight bound for one-shot channel simulation via a stronger version of the strong functional representation lemma: for any random variables $X,Y$, there exists a random variable $Z$ with $I(Z;X)=H(Y|X,Z)=0$ and $H(Y|Z)\le I+\log I+(1-γ)\log e-\log\log e+O(I^{-1}\log I)$, where $I=I(X;Y)$ and $γ$ is the Euler-Mascheroni constant. A more convenient bound is $H(Y|Z)<I+\log(I+2)+1$. The first bound is notable in the sense that the constant term $(1-γ)\log e-\log\log e$ is optimal, and cannot be improved for any bound stated only in terms of $I$. Hence, this is the tightest form of the strong functional representation lemma up to the third order. We also prove that the Poisson functional representation is almost optimal up to a constant gap, and provide a simple finite-blocklength achievability bound that implies the $(1/2)\log n$ redundancy of the simulation of memoryless non-singular channel. These are proved via a generalization of the functional information, in particular, the exponential functional information, which closely approximates the cost of one-shot channel simulation.
Complexity limits and algorithms for efficient string tokenisation
How Hard Is Tokenisation? Algorithms and Complexity Beyond NP-Hardness
Abstract: Given a collection of strings and a vocabulary budget, how well can we minimize its total token count, and how does the answer change with the tokenisation model? For binary direct tokenisation, previous work proved APX-hardness but left constant-factor approximability unresolved \citep{kastreva2026}. We rule out every polynomial-time $M^{1-\varepsilon}$ approximation, for fixed $\varepsilon>0$, unless $\mathrm{P}=\mathrm{NP}$, even on distinct equal-length strings with $K=m-1$. Here $M$ is the total input length and $m$ is the number of input strings. We complement this lower bound with a deterministic polynomial-time $O(M/\log M)$ approximation algorithm and stronger bounds for selected vocabulary budgets. For the different objective of maximizing compression savings, GreedTok was shown to attain a constant fraction of the optimum on unrestricted alphabets \citep{baliga2026}; we establish a deterministic EPTAS for direct savings on every fixed alphabet and analyze greedy selection by true marginal gains. For exact optimization, we give a binary direct algorithm running in $2^{O(n\log\log n/\log n)}\poly(I)$ time, where $n$ is the total length of the distinct input strings and $I$ is the input encoding length. Finally, we resolve two unary merge-model hardness questions left by the bounded-alphabet study \citep{kastreva2026}: standard unary bottom-up tokenisation is APX-hard, and unary optimal pair encoding is strongly NP-complete, strengthening the previously known weak hardness.
Binary prefix coding with unequal letter costs is NP-complete
Binary Huffman Coding with Unequal Letter Costs Is NP-Complete
Abstract: Binary prefix coding assigns prefix-free binary words to symbols to minimize their frequency-weighted cost. Huffman solved the equal-letter-cost case in polynomial time in 1952 [Proc. IRE'52]. For unequal letter costs, Karp gave an integer-programming formulation in 1961 [IRE Trans. Inf. Theory'61]. Exact polynomial-time algorithms for fixed integer costs were given by Golin and Rote [IEEE Trans. Inf. Theory'98] and Bradford, Golin, Larmore, and Rytter [J. Algorithms'02]. For input-dependent costs, Golin, Mathieu, and Young gave a polynomial-time approximation scheme [SICOMP'12]. The exact complexity when letter costs are part of the input had remained unresolved. We resolve this $65$-year-old problem by proving that deciding whether there exists a binary prefix-free code of frequency-weighted cost at most a given threshold is NP-complete, even when the letter costs are consecutive positive integers $(M,M+1)$ supplied as part of the input. All frequencies, letter costs, and thresholds are encoded in binary. Consequently, finding an optimal code is NP-hard. Our proof uses two polynomial-time reductions, starting from Planar Independent Set. The intermediate problem prescribes the number of leaves at each depth of a full ordered binary tree and minimizes a signed linear score on their sorted right-edge counts. The first reduction represents an independent-set verification program by a tree growing level by level. A fixed-profile identity converts narrow coordinate restrictions into linear penalties that enforce the computation. The second reduction adds large dyadic terms to the symbol frequencies to force prescribed depths. These terms contribute a constant to the weighted number of ones, leaving smaller signed perturbations to reproduce the tree score.
Faster space efficient algorithms improve bijective Burrows-Wheeler transform
Revisiting (Bijective) Burrows-Wheeler Transform Space-Time Trade-Off Constructions
Abstract: The bijective Burrows-Wheeler transform (BBWT) replaces the delimiter of the ordinary BWT by a Lyndon factorization and sorts factor conjugates in $ω$-order. It admits quadratic-time constant-workspace algorithms and linear-time linear-space algorithms. We extend the small-space BWT trade-off of Crochemore et al. [JDA,'15] to the BBWT and accelerate both constructions. For $τ\in [3..n]$, let $σ_τ\le\min(σ,τ+1)$ be the maximum construction-batch alphabet size. We construct either transform in $ O\!\left( \frac{n^2}τ \frac{\logτ}{\log\logτ} \right) $ time and $O(τσ_τ)$ additional space, assuming $τσ_τ=O(n)$. This improves the transferred $O((n^2/τ)\logτ)$ bound by a $\log\logτ$ factor. If $σ_τ\in w^{O(1)}$, where $w\inΩ(\log n)$ is the word size, the time improves to $ O\!\left( \frac{n^2}τ + \frac{n\logτ}{\log\logτ} \right). $ For inversion, the same bounds hold with the global alphabet size $σ$ in place of $σ_τ$.
Efficient online string prediction improves update and prediction speeds
Efficient Lempel-Ziv Prediction with the Run-Length BWT
Abstract: Lempel-Ziv Plurality (LZP) is an algorithm of Kosoy that predicts the next symbol of a string by using its Lempel-Ziv factorization, and makes at most $O(z\log n)$ mistakes on every string of length $n$ with $z$ LZ77 phrases. We give efficient online implementations of LZP and of a new variant of it. Our first implementation maintains a run-length encoded Burrows-Wheeler transform (RLBWT) of the reversed history. It implements LZP, uses $O(z\log z\log^2 n)$ bits, and has worst-case update and prediction times $O(\log z+\log\log n)$ and $O(σ(\log z+\log\log n))$, respectively, where $σ$ is the size of the alphabet. This is an exponential improvement over Kosoy's implementation, whose update time is $O(z\log^7 n)$. We then introduce the Lempel-Ziv Majority (LZM) predictor, which has the same mistake bound and update time, and predicts in $O(1)$ time independently of $σ$. These two predictors accept new symbols at any time, without knowing the alphabet in advance. For an alphabet known in advance, a different data structure gives $O(σ(1+\log z/\log\log n))$ time per symbol, which is constant when $σ=O(1)$ and $z=\mathrm{polylog}(n)$.
Conditional smoothing improves prediction from training batches
Batch Universal Prediction via Conditional Additive Smoothing
Abstract: Modern language models motivate renewed interest in universal prediction problems in which a predictor can exploit previously observed data before predicting a fresh sequence. We study an idealized version of this setting by introducing \emph{batch regret}: a conditional analogue of classical average regret in which a predictor observes a batch of $n$ training sequences and is then evaluated on an independent test sequence generated by the same unknown source. Additive smoothing estimators, which add a bias to the observed token frequencies before predicting the next token, played a fundamental role as a simple prediction strategy with almost-optimal average regret for memoryless and Markov sources. Their equivalent under batch regret are the conditional additive smoothing estimators, which estimate the conditional distribution of the test sequence given the training batch. In this work, we study their asymptotical batch regret for binary memoryless and first-order Markov sources. For binary memoryless sources, we show how the optimal bias of the estimator, and consequently the corresponding batch regret, change as a function of the training batch size $n$, by extending the Poisson-based analysis of Krichevsky. For binary Markov sources with positive transitions, we characterize the leading coefficient of batch regret and its dependence on whether the initial law is known, free, or stationary. Finally, we introduce a Rényi analogue of batch regret and identify the corresponding optimal predictors as conditional $α$-NML estimators, thereby giving an operational interpretation to a conditional version of Sibson's $α$-mutual information.
Substring complexity computed efficiently using burrows-wheeler transform runs
Substring Complexity in BWT-Runs Bounded Space
Abstract: Given a string $T[1..n]$, its substring complexity is defined as $δ=\max_k d_k/k$, where $d_k$ is the number of its distinct substrings of length $k$ in $T$. We show how to compute it in $O(n+r\log r)$ time, using only $O(r)$ words of space, where $r$ denotes the number of runs in the Burrows-Wheeler transform.
Fisher information maximized by parity bits for correlated data
A Sharp Fisher-Information Inequality for Boolean Pairs: Proof of the Amari--Kobayashi Conjecture
Abstract: Consider two terminals that observe correlated uniform binary vectors and separately compress their observations to one bit. We prove that, for every blocklength and every correlation parameter in $(-1,1)$, the Fisher information retained by the resulting pair of bits is maximized by matching parity functions of an appropriate degree, thereby resolving a conjecture of Amari and Kobayashi. The proof combines a Fourier-based bound on the derivative of the normalized covariance with a scalar inequality for signed polynomials subject to a global range constraint.
Gaussian source compression rates precisely quantified with new math
Exact Second-Order Asymptotics for the Gaussian Gray--Wyner Problem
Abstract: We revisit the Gray-Wyner problem of lossy compression of bivariate Gaussian sources under the quadratic distortion measure and derive the exact second-order asymptotics. Specifically, we first identify the boundary rate tuples using the explicit first-order asymptotics by Yu (TIT 2023) and then characterize the tradeoff among three second-order coding rates subject to a constraint on the joint excess-distortion probability. Our main contribution is the achievability proof, where we derive a covariance shell covering lemma that is analogous to the type covering lemma for the discrete case. In particular, our covering lemma classifies the source sequences into power shell types via their empirical covariance matrix. For each such power shell type, we derive sufficient conditions on the compression rates so that covering within distortion levels are achievable. Finally, asymptotical analysis via the Berry-Esseen theorem yields the desired result.
How generative compression systems transfer semantic information across tasks
Semantic Transferability of Generative Rate-Distortion-Perception Systems
Abstract: Generative lossy compression combines a rate-limited representation with a stochastic decoder that preserves source-dependent information while generating unresolved detail. We ask what remains reusable when a rate-distortion-perception (RDP) system selected for one operating point is transferred to a prescribed conditional-generation task. We distinguish the single-letter RDP factorization from its block-code implementation and compare Full Redesign with three increasingly restrictive transfer strategies. Renderer-Only Transfer retains only the renderer and redesigns semantic steering. Representation-Induced Transfer additionally retains the semantic latent law induced by the original representation. Full-Model Transfer retains a compatible RDP code, marginalizing over the task conditional law to form its semantic encoder. For finite alphabets, we characterize Renderer-Only Transfer's exact compression-common-randomness rate region, renderer-hull controllability condition, and compression-rate penalty relative to Full Redesign. Under unlimited common randomness, this penalty is the minimum residual semantic information $I(S;U|X)$ forced by the inherited renderer. Representation-Induced Transfer is exact if and only if the semantic condition depends on the source only through the recurrent-class label $C$ of the end-to-end RDP test channel. Every compatible RDP code reused under Full-Model Transfer preserves $C$ and supports every task satisfying $X-C-S$. Any additionally transferable within-class semantic dependence must satisfy $I(S;X|C)\le R_0-H(C)$, where $R_0$ is the inherited compression rate. The bound is exact when $R_0=H(C)$. A scalar Gaussian specialization yields a renderer-noise controllability threshold, a closed-form transfer penalty, and examples of identical Gaussian RDP test channels with different Renderer-Only Transfer capabilities.
Outage concept extended to source coding for non-ergodic data sources
Distortion-Rate Outage Probability for Non-ergodic Classes of Sources
Abstract: The concept of outage has been commonly used in the context of transmission rates over non-ergodic channels. For those channels, one channel state or realization may be encountered for the whole transmission duration of a codeword. In the dual setting of source coding and whenever source statistics are not ergodic, we define in this work the notion of outage; we discuss related properties, consider the special case of compact source classes and provide examples.
Exact algorithms improve smallest grammar problem complexity and hardness
Exact Algorithms and Complexity of the Smallest Grammar Problem
Abstract: The Smallest Grammar Problem (SGP) asks for a smallest straight-line grammar generating a given string, where size is the total length of the right-hand sides. Rules may be nested, so locally optimal choices of repeated fragments need not combine into an optimal grammar. We show that in an optimal grammar one can choose an occurrence of the expansion of each auxiliary nonterminal such that these intervals are pairwise disjoint. Hence the dictionary can be read off from a partition of the input, decoupling the choice of the dictionary from the parsing of rules. Using this property together with a bound on references to rules of length two, we obtain an exact algorithm running in $O^*(1.8723^n)$ time and $O(n)$ words of space, improving the $O^*(3^n)$ bound of Casel et al. for general grammars. For alphabet size $σ$ and $R=\log_σn$, we give a $2^{O((n/R)\log(R+1))}$-time algorithm, which is subexponential for fixed alphabets. Conversely, using an anchor-based projection from encoded grammars back to source grammars, we prove that SGP is NP-complete over a binary alphabet, answering a question of Casel et al., and construct instance families of tunable alphabet size on which no $2^{o(n/R)}n^{O(1)}$-time algorithm exists under ETH. Finally, for every $c>1$ we give a constant-degree integer polynomial formulation of SGP with $O(c^n)$ variables. By the barrier theorem of Belova, Golovnev, Kulikov, Mihajlin, and Sharipov, a proof that SGP is $λ^n$-SETH-hard under their deterministic fine-grained reductions would imply new lower bounds for Boolean series-parallel circuits or for arithmetic circuits.
Encoding trees enable flexible variable-length prefix coding for arbitrary symbol sets
Encoding Trees: A Method for Producing a Variable-Length Prefix Code From Sets of Arbitrary Size
Abstract: In this work we formalize a new class of binary trees called encoding trees. We extend this structure in two ways, through labelling to create the class of labelled encoding trees, and pruning to create the class of pruned encoding trees. We discuss how these two extensions can be combined to create the the class of pruned labelled encoding trees. Using the supplementary results that we prove about these classes of trees, we construct a code that uses pruned labelled encoding trees to convert a string of symbols from a sequence of non-empty sets to a binary string. We prove that this code is a variable-length prefix code that can process any valid string of symbols from the given sequence of non-empty sets. Furthermore, we construct a code from the class of pruned labelled encoding trees that goes in the opposite direction, from binary strings to a string of symbols from a sequence of non-empty sets of arbitrary symbols. We prove that this code is also a variable-length prefix code that can process any binary string of any length, provided that the sequence of non-empty sets is sufficiently long. We provide an equation that establishes the minimum sequence length to encode any binary string of a given length, based on that length.
Universal coding method achieves near optimal data compression rates
Universal coding with the exhaustive peres transform
Abstract: Peres's iteration of von Neumann's procedure is completed into a bijection of $\bits^N$: the recursion runs to exhaustion, nothing is discarded, and every input splits into fair bits and residual bits. The residual strings form a complete prefix code that determines the number of ones of the input, so the fair bits are exactly uniform given the residual for every exchangeable law. For a Bernoulli source the likelihood factorises exactly, in decoding order, into a factor $1/2$ per fair bit and a Bernoulli factor per residual bit, with biases obtained by iterating two explicit maps. The number of fair bits per input symbol tends to the entropy almost surely, also on larger alphabets. Code B stores the fair bits and codes only the residual with an arithmetic coder; its ideal length is that of the two-part code, at most $\log\_2(N+1)+1$ bits above $-\log\_2\Pr\_θ(s)$ for every input $s$ and bias $θ$. Code C uses Rice codes instead. In experiments up to $N=2^{20}$ its mean regret stays below $200$ bits, where a Rice coder on the raw input loses a fixed fraction of the entropy on skewed sources. For sources with memory, the Burrows--Wheeler transform of the reversed string arranges all context orders in one tree of intervals; coding the leaves of a pruned tree as Bernoulli strings adapts to every Markov order up to its depth, and on suffix-closed trees the leaf weights determine the split points.
Adaptive covering complexity links static and self referencing compression
On the Adaptive Covering Complexity of Strings: Unifying Relative and Self-Referential Compression
Abstract: We introduce and study the adaptive covering complexity C adapt S (s), defined as the minimum number of contiguous factor blocks required to partition a string s when each block is either a factor of a fixed external dictionary S or a factor of the target's already decoded prefix. This hybrid admissibility model formalizes the structural interaction between static dictionary matching and dynamic nonoverlapping self-referencing. Our main theoretical result is the exact logarithmic law which is tight and proves that dynamic self-reference can reduce phrase count by at most an exponential factor over static dictionary covering alone. We establish structural equivalences showing that the measure bridges word-segmented Relative Lempel-Ziv and non-overlapping LZ77, and we prove exact preset-history LZ77 sandwich bounds. Furthermore, we establish the optimality of the longest-match greedy parse via an exchange lemma and present an optimal implementation based on dual suffix automata running in O(M + n) time and space for a constant alphabet. These analytical results are complemented by counting bounds, stationary-ergodic asymptotic invariance, a dictionaryselection NP-hardness result, and a concise empirical illustration.
Gaussian distributions maximize information cost for correlation targets
A Self-Coupling Gaussian Extremal Inequality for Information-Constrained Optimal Transport with Applications to Rate-Distortion-Perception
Abstract: How much mutual information is needed to achieve a target correlation between two copies of a prescribed distribution? We prove a sharp Gaussian extremal inequality for this self-coupling problem: among distributions with a fixed mean and covariance, the Gaussian has the greatest minimum information cost for meeting a positive semidefinite cross-covariance target. The inequality is strict for non-Gaussian distributions when the target is positive definite. The proof uses a midpoint gluing construction. Two Ornstein--Uhlenbeck half-bridges, each started from the prescribed endpoint law, meet at a common intermediate distribution and are joined by conditional gluing. Changing the Gaussian reference then turns the resulting relative-entropy bound into the desired extremal inequality. As a first application, scaling one endpoint of the self-coupling establishes Gaussian extremality for rate-distortion-perception with unlimited common randomness, under quadratic distortion and squared Wasserstein-2 perception measures. We also establish an analogous result for finite common randomness, where the problem involves two mutual information constraints associated with an auxiliary random variable. In that setting, we retain an intermediate observation on one half-bridge and balance the two information budgets simultaneously using the Poincaré-Miranda theorem.
Query-dependent reading improves accuracy of quantized inner product estimates
Query-Dependent Reading for Quantized Inner-Product Estimation: A Minimax Separation
Abstract: We study quantization for inner-product estimation when storage and reading have separate budgets. In the static bit-probe model [Yao, 1981], an encoder stores $w$ bits about $x$ before seeing a query $y$, and a reader probes at most $r<w$ bits to estimate $x^\top y$. Both vectors lie in the $d$-dimensional Euclidean unit ball. Protocols may be randomized, and computation is unrestricted. Performance is measured by $d$ times the worst-case mean-squared error over $x,y$, averaged over protocol randomness. First, we show that query-independent reading cannot benefit from extra storage. Any such reader is equivalent to fully reading an $r$-bit message, even when later reads depend on previously returned bits. At fixed positive rates $W=w/d$ and $R=r/d$, the normalized minimax error converges to $1/(4^R-1)$, independent of $W$. The upper bound was already known [Ordentlich and Polyanskiy, 2025]. We prove the matching lower bound for arbitrary scalar decoders, extending the result for unbiased compressors [Safaryan et al., 2020]. Then, we show that query-dependent reading makes extra storage useful. We construct a two-batch protocol using Gaussian projections and random thresholds, with asymptotic normalized error at most $(π+o(1))/[4R\log(W/R)]$ as $W/R\to\infty$. At fixed $R$, this error vanishes as storage grows, giving an unbounded advantage over query-independent reading. We also prove a finite-dimensional lower bound for arbitrary encoders, decoders, and fully branching readers. In the doubly sparse regime $W/R\to\infty$ and $R\log(W/R)\to0$, it gives normalized error at least $(1-o(1))/[2R\log(W/R)]$. Thus, our protocol is optimal within a factor of $π/2$ in the leading constant.
Computing information bottleneck values is computationally hard
The Information Bottleneck is NP-hard
Abstract: Given a joint distribution $P_{XY}$ on a finite alphabet, the information bottleneck method seeks a representation $T$ of $X$ that is as informative as possible about $Y$ while compressing $X$, where $T$ and $Y$ are conditionally independent given $X$. We show that computing the optimal value $\mathrm{IB}(P_{XY},R) = \sup_T \{ I(Y;T) : I(X;T) \le R \}$ of the information bottleneck problem, as well as its Lagrangian relaxation $L_β(P_{XY}) = \sup_T \big[I(Y;T) - βI(X;T)\big]$, is NP-hard. Moreover, unless $\mathrm{P} = \mathrm{NP}$, neither $\mathrm{IB}(P_{XY},R)$ nor $L_β(P_{XY})$ can be approximated in polynomial time within an additive error of order $|\mathcal{X}|^{-3/2}$.
New entropy inequality tightens bounds for log-concave distributions
Sharp Rogers--Shephard Inequalities for Rényi Entropy
Abstract: We establish sharp weighted Rogers--Shephard inequalities for Rényi differential entropy. For independent copies $X,X'$ of a log-concave random vector in $\R^n$, we prove $h_p(aX-bX')\le h_p(X)+n\log(a+b)$ for all $a,b>0$ and $1\le p\le\infty$, where orders $1$ and $\infty$ correspond to Shannon entropy and min-entropy, respectively. The Shannon case settles the entropic Rogers--Shephard conjecture of Madiman and Kontoyiannis and its higher-dimensional formulation. For $1\le p<\infty$, equality holds precisely for invertible affine images of random vectors with independent exponential coordinates. At $p=\infty$, the larger equality class is characterized by possibly extended-valued convex potentials that are positively homogeneous of degree one. For every $0<p<1$, uniform laws on sufficiently high-dimensional simplices violate the unweighted bound. Thus $p=1$ is the critical order for this bound to hold in every dimension. The proof combines weighted convex intersections with an exponential lift to a log-affine probability density. At the Shannon endpoint, this lift yields an exact deficit decomposition with a conditional mutual information term for an additive channel with asymmetric Laplace noise.
New methods clarify compression limits for gaussian mixture data
Rate-Distortion Theory of Gaussian Mixtures via Posterior Label Sampling
Abstract: We study the mean-squared error (MSE) rate-distortion function (RDF) of finite Gaussian mixture models (GMMs) in $\mathbb{R}^d$, for which neither the RDF nor an optimal test channel is known in closed form. It can be sandwiched between the genie-aided conditional RDF $R_{\mathbf{X} | T}(D)$, which can be characterized in closed-form for a GMM, and $R_{\mathbf{X} | T}(D)+I(\mathbf{X};T)$, where $T$ denotes the latent component label. We show that the upper bound can be achieved operationally by the proposed Gibbs-relabelling method (GRM), which conveys a label from the posterior $P_{T|\mathbf{X}}$ through \textit{channel simulation} at rate $I(\mathbf{X};T)$, followed by vector quantization of the resulting exactly Gaussian conditional sources by reverse water-filling. Using the Shannon lower bound (SLB), we prove that GRM attains the SLB, and hence the RDF, of a general finite mixture if and only if every component attains its own SLB. For a GMM this holds throughout the fully active interval $D/d<\min_{k,m}λ_{k,m}$, where $λ_{k,m}$ are the eigenvalues of the component covariances, and for scalar GMMs the condition is also necessary. Since GRM cannot operate below the label cost $I(\mathbf{X};T)$, we propose a complementary stochastic scalar quantization (SSQ) scheme for low rates, which simulates an $M$-ary reconstruction index. We show that at low rates SSQ with $M\geq 2$ matches the Shannon RDF to first order: $R(D_{\max}-δ)=δ/(2τ^2)+o(δ)$, where $τ^2$ is a directional sub-Gaussian proxy of the mixture, and quantify the price of a finite codebook through a strict rate gap at high rates and a distortion floor whose joint scaling in $M$ and number of mixture components is characterized. Numerical results illustrate both regimes and the finite-blocklength cost of channel simulation.
Information bounds keep constant while preserving rounds in communication tasks
Constant-Loss Round-Preserving Direct Sums for Information Complexity
Abstract: Strong direct-sum theorems relate the information needed to solve many independent instances of a communication problem to the information needed for one instance. For Gap-Majority composition, an optimal information bound was previously known, but preserving the number of communication rounds incurred an additional logarithmic loss. We prove a round-preserving Gap-Majority lemma with constant additive information loss, independent of both the number of instances and the number of rounds. More generally, we establish a variance direct-sum theorem for additive square-integrable observables over product distributions. For protocols with a bounded number of terminal answer values, the extracted single-coordinate protocols preserve the round budget, and their average information cost is at most the original information cost divided by the number of coordinates, plus a term depending only on the answer alphabet and the allowed variance loss. The proof uses centered counterfactual conditional means and a global variance amortization argument. Rather than encoding absolute conditional means for every possible terminal answer, we encode the information not already determined by the common view. The total squared innovation is controlled by a telescoping variance potential, yielding an information loss independent of the number of coordinates. We also derive same-round mean-square estimation bounds for arbitrary terminal outputs.
DisCo algebra reveals limits of summarizing joint probabilities
DisCo Algebra: Why Disintegration and Combination Are Not Inverses
Abstract: We present DisCo algebra---a compositional framework for joint probability built from a symmetric cooperad (disintegration) and a symmetric operad (combination) on the bare family $\mathcal{P}_n=\mathrm{Prob}([0,1]^n)$. No probability integral transform, no external summary functions, no enrichment of objects with conditional kernels. The disintegration side uses block-wise joint CDFs as intrinsic, measure-determined 1D summaries, yielding a lossy compression. The combination side reads the same summaries and reconstructs via regular conditional measures, yielding a lossless embedding. We prove that combination followed by disintegration is the identity on admissible inputs, but disintegration followed by combination is not---the information loss is exactly the conditional KL divergence between the true conditional distribution and the conditionally independent product. This asymmetry reveals a hard limit on how much cross-block dependence can be captured by scalar block summaries.
Neural polar codes improve data compression without explicit source models
Neural Polar Codes for Source Coding
Abstract: This paper presents a data-driven approach to lossless and lossy source compression using polar codes. Classical polar source coding achieves fundamental limits with low complexity for binary memoryless sources. However, for unknown q-ary sources with memory, successive-cancellation (SC) decoding requires an explicit source model and can be computationally prohibitive. We present analytical SC recursions and show that exact implementations scale quadratically with alphabet size and cubically with state-space size. To overcome these limitations, we develop neural polar decoders trained directly from data to approximate conditional distributions of synthetic polar coordinates without requiring an explicit source model. These learned distributions enable estimation of entropy rates, mutual information, and rate-distortion functions, and support explicit polar compression schemes. For lossless compression, we identify high-entropy polarized coordinates and reconstruct the remaining coordinates using the learned SC recursion. For lossy compression, we combine neural polar decoders with a neural test channel to estimate and optimize the mutual information-distortion tradeoff and construct an explicit lossy polar code. We also extend the framework to Wyner-Ziv compression with decoder side information by learning conditional polarized distributions and a reconstruction mapping. Experiments on memoryless sources, sources with memory, and real genomic data show that the learned decoders closely match analytical SC decoders when available and remain applicable when conventional methods are computationally infeasible or the source model is unavailable.
Discrete Wyner Ziv problem second order coding rates improved
Exact Second-Order Asymptotics for the Discrete Wyner--Ziv Problem
Abstract: We revisit the discrete Wyner--Ziv problem and establish exact second-order asymptotics. Our main contribution is a second-order converse bound, which is derived using posterior decomposition, concentration inequalities, and a conditional Gaussian approximation for martingales. Furthermore, we extend the previous best known achievability result of Li and Li (arXiv 2025) by allowing the test channel to depend on the type of the observed source sequence. Exact second-order asymptotics are established by combining our achievability and converse bounds. In particular, we show that the achievability bound of Li and Li (arXiv 2025) is not optimal in general. Specifically, we provide two numerical examples to illustrate our results: a binary asymmetric source and a quaternary source. For the first example, the achievability bound of Li and Li (arXiv 2025) achieves the optimal second-order asymptotics. However, in the second example, we show that our achievability result is optimal, which reduces the second-order coding rate of Li and Li (arXiv 2025) by $10.77\%$ at the excess-distortion probability of $0.1$ and by $21.35\%$ at the excess-distortion probability of $0.2$.
Dynamic time warping shows strong limits for fast approximation
Fine-Grained Hardness of Approximating Dynamic Time Warping
Abstract: We prove conditional lower bounds for approximating dynamic time warping (DTW) over the uniform metric on three symbols. Let $N, M$ be the two string lengths and $n, m$ their respective numbers of runs. Under the Gap Block Disjointness (GBD) hypothesis, for every fixed $κ\in (0,1/6)$ and $δ\in (0,1)$, no deterministic algorithm $N^{κδ}$-approximates DTW on two explicit strings of length $N$ in $O(N^{2-δ})$ time. This matches the known tradeoff between approximation and running-time exponents for deterministic algorithms within any fixed factor greater than six in the approximation exponent. For run-length encoded strings, the Orthogonal Vectors Hypothesis (OVH) rules out any constant-factor approximation in $\widetilde{O}((nm)^{1-δ})$ time for every constant $δ>0$. More generally, for every fixed $η\in (0,1)$, it rules out $(N+M)^{1-η}$-approximation in the same running time. These bounds extend to every fixed metric with at least three points, including absolute distance on $\{0,1,2\}$. Both reductions use long runs to force equal-symbol matches in low-cost alignments. For explicit strings, we combine Boolean gadgets with deterministic gap amplification to obtain a polynomial approximation gap. For run-length encoded strings, an encoding of regular-expression membership gives accepting instances an alignment whose cost does not increase as selected runs grow. Rejecting instances have distance at least the chosen run length, while the number of runs stays fixed.
Finite precision impacts accuracy of randomized vector balancing methods
Finite-Precision Gram-Schmidt Walks
Abstract: The Gram-Schmidt Walk is a randomized vector-balancing algorithm whose subgaussian guarantees support applications in discrepancy, experimental design, and data compression; however, these theoretical guarantees are established in exact arithmetic, whereas implementations must approximate least-squares directions, boundary updates, and sampling probabilities in finite precision. This is important because small numerical errors can change which coordinates freeze and thereby alter the subsequent trajectory. We analyze the concentration of the perturbed Gram-Schmidt walk directly under bounded, potentially biased and history-dependent errors. For $n$ input vectors of Euclidean norm at most one, we obtain a modified MGF bound depending on key error sources which recovers the original result as the error goes to zero. We also construct a full-column-rank instance in which bounded update errors produce bias of order $\min\{n^2\varepsilon,n\}$, showing that updates accumulate error unavoidably under this model. Finally, we validate our findings in a variety of settings by ablating on the bit precision and problem size.
Binary word patterns precisely identified by letter counts and smoothness
Shuffle Squares in Differentiable Words
Abstract: Experiments on binary run-length differentiability lead to sharp computer-assisted criteria for shuffle squares. Every $C^3$-word of length greater than $16$ is a shuffle square exactly when both letter multiplicities are even. All $34$ nonempty even-Parikh exceptions are smooth and persist in every higher differentiability class. For $C^2$ the sharp threshold is $48$, with $212$ nonempty exceptions. At level $C^1$ no global parity threshold exists, but every even-Parikh non-shuffle-square of length at least $36$ has proper nonempty shuffle-square prefixes and suffixes. Exactly $230$ nonempty even-Parikh $C^1$-words have no nonempty shuffle-square prefix. The full tree avoiding such prefixes eventually consists of $422$ periodic rays. Consequently, a nonempty Kolakoski prefix is a shuffle square exactly when both multiplicities are even, apart from lengths $4$ and $8$. We also characterize classes of morphisms reflecting shuffle squares. For doubly binary words, we determine the exact deletion distance and largest twins, and prove sharp bounds for single local repairs. Exact recurrences, residual-state checks, and separate Python programs make the finite computations reproducible.
New findings on subsequence occurrence limits in binary word sets
Subsequence Analysis Problems for Binary Parikh Matrices
Abstract: The universality index corresponding to a word is the largest integer k such that every word of length k over the given alphabet occurs in it as a subsequence. Relative to languages this notion can be investigated with respect to both existential and universal quantifiers, with the former corresponding to the existence of a word in the language with universality index at least k, while the latter considers the index across all words that the language contains. In this work we study the existential (exists-universality) and universal (forall-universality) subsequence universality (as introduced in [Adamson et al., ISAAC 2023]) for binary languages consisting of all words with a fixed number of letters a, letters b, and subsequences ab. We prove that both exists- and forall-universality admit exact arithmetic characterizations for these languages. Extending the above notions, we end the paper by initiating the analysis of the probability of a fixed word occurring as subsequence of the words in such a language. To this end, we prove that, for every fixed pattern word w and an error tolerance, given as input the subsequence counts describing a language, the ratio of words in the language having the pattern w as a subsequence admits an additive approximation scheme that is polynomial-time in the binary encoding of the input counts.
Binary sources need fewer symbols for optimal information compression
On the Cardinality of Optimal Representations in the Binary-Source Information Bottleneck
Abstract: The information bottleneck (IB) seeks a representation $U$ of a source $X$ that retains as much information as possible about a target $Y$, subject to a constraint on $I(U;X)$. A classical argument shows that it suffices to consider representations with at most $|\mathcal{X}|+1$ symbols, and this bound is known to be tight whenever $|\mathcal{X}| \geq 3$. We show that the binary case behaves differently: if $X$ is binary and $Y$ is finite, then for every joint distribution of $(X,Y)$ and every rate constraint, the IB optimum is attained by a binary $U$. Hence the bound $|\mathcal{U}| \leq |\mathcal{X}|+1$ sharpens to $|\mathcal{U}| \leq |\mathcal{X}|$ for binary sources. The proof combines a separating hyperplane argument with the observation that, for a binary source, the ratio of the second derivatives of the two entropy functions involved is concave.
Lz77 compression improves by adjusting window size with match length
Improved LZ77 Compression with Match-Length-Dependent Sliding Windows
Abstract: We devise and analyze WLZ, a family of LZ77 encoders whose sliding-window sizes depend on match length: short matches use smaller windows and shorter distance fields, while long matches retain access to distant repetitions. Write $B:=\log_2 W$ for the maximum window $W$. A parsing-transfer bound charges both window restrictions and missed matches, preserving established convergence and finite-input minimax orders, and a sharper block charge proves stationary-ergodic universality of exact greedy WLZ when full-window matching begins at length $o(B)$. A calibrated window schedule never increases minimum token cost relative to a single-window baseline and saves $Ω(BW^{-β})$ in rate on specified iid sources, with $β>2$. The remaining gains come from the token code rather than the windows. On a uniform iid source, a growing phrase cap with a cheap length symbol improves redundancy from the $Ω(\log B/B)$ of the original 1977 fixed-field format, whatever phrase cap it uses, to $O(\log\log B/B)$. Recoding a single-symbol run $(r,1)$ as $(1,r)$, with a logarithmic-cost count $r$ that may exceed the match cap, codes inputs with $O(n^α)$ runs, $0\leα<1$, in $O(n^α\log n)$ bits, versus $Ω(n)$ for that format with phrase cap $Θ(\log W)$ and $W=o(n)$. On globally $p$-periodic inputs, Huffman coding the fields of a capped parse with nearest-distance ties reduces the large-file rate from $Θ(B/W)$ under fixed-width coding to at most $3/W$, including tables and framing. Finally, selection among complete WLZ codes yields an entropy-rate estimator consistent almost surely and in mean, with finite-data error bounds for fixed nonuniform iid sources.
Net occurrences in morphic words precisely counted in new study
Exact Net-Occurrence Counts in Purely Morphic Regular Epistandard Words
Abstract: Finite Fibonacci words have recently been shown to contain exactly three net occurrences---occurrences of repeated factors whose one-letter left and right extensions are unique. This unexpectedly small constant raises a natural question: is it peculiar to the Fibonacci recurrence, or part of a broader morphic phenomenon? We study a family over the alphabet $\{0,1,\ldots,d-1\}$ determined by positive integers $e_0,\ldots,e_{d-1}$. For each letter $a$, let $L_a$ be the morphism that fixes $a$ and maps every other letter $b$ to $ab$; we consider the finite approximants $S_m=μ^m(0)$ generated by $μ=L_0^{e_0}\cdots L_{d-1}^{e_{d-1}}$. These words are the period-aligned finite approximants of the purely morphic regular epistandard family considered here. We prove a sharp dichotomy: for every $m\ge2$, $S_m$ has exactly three net occurrences when $e_{d-1}=1$, and exactly two when $e_{d-1}\ge2$; the initial approximant is also completely classified. The proof combines palindromic prefixes, return-word factorizations, and overlapping net-occurrence covers. As a consequence, when all exponents are equal to one---the standard $d$-bonacci case---every noninitial period-aligned finite approximant has exactly three net occurrences, placing the Fibonacci phenomenon in a wider epistandard framework.
Researchers determine exact optimal factor for universal integer coding
Optimal Universal Coding of Integers
Abstract: Universal coding of integers (UCI) provides binary codewords for positive integers such that, for every nonincreasing source distribution $P$, the average codeword length stays within $K$ times $\max\{1,H(P)\}$. The smallest constant $K$ is called the minimum expansion factor of UCI $\mathcal{C}$, denoted $C_{\mathcal{C}}^{*}$. The optimal minimum expansion factor $C^*=\inf\{C_{\mathcal{C}}^{*}\}$ is the minimum expansion factor corresponding to the optimal UCI. The optimal minimum expansion factor is currently known to lie in the range $2\le C^*\le 2.0386$. In this paper, we construct a family of one-point plus uniform-tail distributions and prove that, for every universal code, the worst-case ratio is attained by a distribution in this family, so that the family is least favorable for the UCI problem. We further establish an inequality, called the \emph{UCI inequality}, which plays the same role for UCI as the Kraft inequality does for prefix codes: for any real number $B$, it decides whether $B$ lies below or above $C^*$. Through the UCI inequality, we obtain an equivalent definition of $C^*$. By numerical computation, we determine $C^*=2.000124757036101\cdots$, the first fifteen decimal digits being certified. Once $C^*$ is known, we can theoretically construct the optimal UCI.
Algorithmic statistics reveals symmetrical patterns in data structure
CAS II: Symmetric Partitions as Kolmogorov Models
Abstract: In algorithmic statistics a string x is explained by a finite set containing it, and Kolmogorov's structure function records the smallest such model at each level of complexity. Vereshchagin's strong models, those computable from the data by a total algorithm, are essentially the cells of simple partitions. We read a partition of binary strings as a hypothesis, with the cell containing x as its model, and develop algorithmic statistics over symmetric partitions: the orbit partitions of groups acting on strings. The Galois connection between subgroups and partitions gives each ambient group a lattice of symmetric partitions, with canonical certificates, canonical costs, and an algebra of hypotheses. The resulting structure function and symmetric sophistication measure which part of the regularity of x is symmetric. For the full symmetric group every partition is symmetric: cells recover all Kolmogorov models, cells of cheap partitions recover exactly the strong models, and normal and strange strings are characterized by symmetry. For GL(n,2) the cells are exactly the linearly homogeneous sets, so linear symmetry is a restricted model class. For nonzero x, the linear-symmetry structure function lies in a band between the sufficiency line and the trivial bound, and both edges are attained: there are stochastic normal strings whose simple structure is invisible to linear symmetry. We also give coordinates on the space of permutation groups: each group is an element of a Burnside ring (its type) together with a permutation (its placement), and restriction moves refine partitions via the Mackey formula. In these coordinates the collapse for the symmetric group is a statement about placement, a linear hypothesis is determined by its type up to n^2 bits, and the maximal gap theorem shows that any space of symmetry hypotheses small enough to search is small enough to miss simple structure.
Information distance cannot mimic lattices or Euclidean spaces exactly
Complete bipartite embeddings, lattice non-embeddability, and non-Euclidean powers of information distance
Abstract: We encode the vertices of every finite complete bipartite graph as binary strings at each scale. The information distances between these strings approximate the scaled graph distances with an additive error bounded by a constant independent of the scale. We also assign one fixed infinite binary sequence to each vertex; suitable prefixes satisfy the same distance estimate with logarithmic error. This extends Hutter's construction for $K_{3,3}$ and resolves his open problem on scale-embeddings of finite complete bipartite graphs. Combined with his complete bipartite obstruction, these embeddings show that no positive power of information distance can be represented exactly by distances in a real Hilbert space, resolving his question about such representations. For conditional prefix complexity, we further prove that at most $2^{δ+C}$ strings lie between any two strings with total distance exceeding their mutual distance by at most $δ$, where $C$ is independent of the endpoints. This bound rules out scale-embeddings of the integer line and every positive-dimensional integer lattice, resolving Hutter's lattice embedding problem for both string and sequence embeddings. Restriction to the integer lattice gives the same conclusion for $(\mathbb R^m,\|\cdot\|_1)$ for every $m\ge1$.
Task scope shapes information needs more than receiver type
On Task Scope and Information Retention in Source Coding
Abstract: We argue that dividing codec design into Coding for Machines (CfM) and Coding for Humans (CfH) is a misleading distinction for deciding what information a codec may discard. Receiver identity does not determine admissible information loss. The required rate depends on task scope, including the predictions to support, their losses and tolerated risks, the encoder observation, and the permitted decoding procedures. Notably, a machine task may have a higher minimum rate than a restricted human decision. Rate savings on selected machine tasks apply only to the stated requirements, not to an intrinsic ordering by receiver type. We extend source and feature coding to finite task families, derive when restricting the encoder observation preserves the minimum rate, and show that equality between source and split-feature coding rates can no longer hold as the task scope expands.
Proof settles three key conjectures about binary communication channels
Three Conjectures on Binary Channels for the Doubly Symmetric Binary Source
Abstract: We settle three conjectures concerning a doubly symmetric binary source $(X,Y)$ with crossover $p$. Consider Markov chains $U - X - Y - V$ with $U,V$ binary, and let $\mathcal{A}$ be the set of rate triples $(I(U;V),I(U;X),I(Y;V))$ attainable with arbitrary binary channels $X\to U$, $Y\to V$, and $\mathcal{B}$ the subset attainable with binary symmetric channels. The averaged BSC conjecture, Conjecture 5.2 of Pichler, Piantanida and Matz (2022), asserts $\operatorname{conv}\mathcal{A}=\operatorname{conv}\mathcal{B}$. We prove this for every $p\in[0,1]$. Two conjectures of Dikshtein, Ordentlich and Shamai (2022) concern the double-sided information bottleneck at $p=0$, where $Y=X$ and the two channels see the same source: their Conjecture 1 identifies the exact maximum of $I(U;V)$ at prescribed rates $I(U;X)$ and $I(Y;V)$, and their Conjecture 2 the exact minimum. We prove both for binary $U,V$: the two extrema are attained by the same pair of Z/S-channels, in opposite orientation for the maximum and in the same orientation for the minimum. The proofs were found with substantial AI assistance, and all three theorems are formalised in Lean 4 with Mathlib, depending only on the standard axioms. The development is available at https://github.com/g-pichler/bsc-averaging . The proof of Conjecture 1 of Dikshtein, Ordentlich and Shamai (2022) contains three certified computations, a polynomial bound, an interval sweep and a polynomial positivity certificate, all of which are checked in Lean.
Exact threshold found for entropy concavity in Bernoulli sums
The Sharp Rényi and Tsallis Threshold in the Shepp--Olkin Concavity Problem
Abstract: Let $B_1,\ldots,B_n$ be independent Bernoulli random variables with parameters $p_1,\ldots,p_n$, and let $S=\sum_i B_i$. Hillion and Johnson proved that the Shannon entropy of $S$ is jointly concave in the parameter vector and proposed corresponding critical-order conjectures for R'enyi and Tsallis entropies, with predicted thresholds $2$ and approximately $3.65986$, respectively. We determine both thresholds exactly. For every $0<q<1$, the power sum $\sum_k \mathbb P(S=k)^q$ is jointly concave in $(p_1,\ldots,p_n)$, and strictly concave on the open parameter cube. Consequently, the R'enyi and Tsallis entropies of order $q$ are jointly concave. At $q=1$ this agrees with the Shannon theorem. For every $q>1$, joint concavity fails already for the sum of two Bernoulli variables: a transverse interpolation in which the two parameters move in opposite directions gives strict local convexity for both entropies. Hence the universal joint-concavity range for both families is exactly $0<q\leq 1$. Below order one, the proof combines the Hillion--Johnson transport inequality with an explicit nonlinear telescoping correction. The corrected local curvature reduces to a two-dimensional quadratic form. An exact Riccati identity, together with a one-sided zero-crossing argument, proves positivity of its determinant throughout the full range $0<q<1$.
New formula links geometric mean quantization to probability masses
Geometric mean quantization via adaptive approximation
Abstract: Let $ν$ be a compactly supported Borel probability measure on $\mathbb R^{d}$ with $ν(B(x,r))\leq Cr^{a}$ for some $a>0$. Refine a dyadic cube exactly when its mass is at least $t$, and let $\mathcal{L}_ν(t)$ be the mean depth at which this refinement stops. We show that the lower and upper geometric-mean quantization dimensions of $ν$ are the lower and upper limits of $\log(1/t)/\mathcal{L}_ν(t)$. The dimension exists precisely when $(q-1)\sum_{Q}ν(Q)^{q}$, summed over all dyadic cubes, converges as $q\downarrow1$, and it is then determined by this limit. The mass-threshold formula yields harmonic integral bounds in terms of the local dimensions and encloses entropy and quantization dimensions in a common spectral interval. Convergence in law of the local information rates is equivalent to convergence of the rescaled spectra in a window of width $1/k$ around $q=1$; the two dimensions are then the arithmetic and the harmonic mean of the limit law, and we quantify their difference by sharp bounds and variance identities. Without any convergence assumption, vanishing threshold variance still forces equality of the corresponding lower and upper dimensions. Bernoulli mixtures realise every local-dimension law with compact support in $(0,1]$, and a regime-switching example separates convergence in law from almost-everywhere convergence.
Entropy bounds tighten for sums of independent discrete variables
Sharp High-Entropy Bounds for Sums of Independent Discrete Random Variables
Abstract: Sharp high-entropy lower bounds for the entropy of a sum were known for identically distributed summands in torsion-free abelian groups and in prime cyclic groups. For arbitrary independent summands, Gavalakis, Goh and Kontoyiannis obtained an additive constant of $1/8$ and conjectured that the sharp constant is $1/2$. We prove that independent discrete random variables $X,Y$ with finite Shannon entropies satisfy $H(X+Y)\ge (H(X)+H(Y))/2+1/2-o(1)$ in every torsion-free abelian group as $\max{H(X),H(Y)}\to\infty$. The same conclusion holds in the prime cyclic group $\mathbb F_p$ when both $\max{H(X),H(Y)}$ and $\log_2 p-\max{H(X),H(Y)}$ tend to infinity. We give explicit error bounds in both settings. The proof extracts a component with paired point probabilities while controlling the entropy of the remainder independently of its support. Discrete rearrangement and uniform perturbation then transfer the continuous entropy power inequality to this component. In prime cyclic groups, an additional estimate controls the entropy lost under modular reduction. Binomial distributions show that the constant $1/2$ is optimal.
Multi-agent communication improves solving complex tasks and compression
Scaling Discovery through Test-Time Communication
Abstract: Science advances not in isolation but through collaboration, yet existing agentic systems capture little of this. Whether communicating agents help remains an open question with mixed prior results. We show that test-time communication can substantially outperform independent parallel attempts on challenging tasks, where sharing a breakthrough can push the whole group forward. We first study the effect of scaling multi-agent test-time communication, where agents have no predefined roles and communicate via a shared directory, on ARC-AGI-3, a benchmark requiring novel problem solving. We find that a team of $k$ communicating agents, team@$k$, matches the success rate of $4k$ independent agents, and this advantage grows with $k$, suggesting gains compound with scale. The effect is not merely efficiency: a task that no single agent can solve, a team of agents can solve reliably. Furthermore, these gains transfer to research-oriented tasks, given sufficient compute. On polyomino packing, communicating agents outperform best@$k$ and exceed the prior best-known score. On MNIST classifier compression, communication surpasses the best-known human solution. A team of four agents produced a 1,957-byte classifier submission achieving 99.4% test accuracy, smaller than both the best-known human solution and the best single-agent result. These gains are not unconditional. Independent agents may outperform communication when compute is limited or when a clear measure of progress is absent. However, under sufficient compute and clear feedback, multi-agent communication consistently yields stronger results.
Copula operad framework links dependence and entropy additively
Copula Operad and Copula Entropy
Abstract: We construct a symmetric operad $\mathfrak{C}$ on the class of all multivariate copulas, with operadic composition given by Sklar substitution. This places the hierarchical combination of dependence structures into an algebraic framework. Restricting to absolutely continuous copulas whose densities lie in $L\log L$, we prove that copula entropy is additive under composition, making it an additive character on suitable finite-entropy suboperads. We exhibit explicit closed suboperads (bounded, $L^{p}$, boundary-growth) and show by counterexample that componentwise $L\log L$ does not imply closure under composition.
Efficient sequence prediction balances speed and pattern complexity
Stringological sequence prediction III: layered ziplines and a tradeoff between efficiency and expressivity
Abstract: In previous papers, we began the study of sequence prediction algorithms adapted to stringological word complexity measures. In particular, we defined a complexity measure called Arithmetic Repetition Complexity (ARC) which admits a polynomial-time prediction algorithm with a mistake bound quasilinear in the complexity. Here, we show a weaker complexity measure related to ARC that admits an especially efficient prediction algorithm: an algorithm that runs in quasilinear time and polylog space for appropriate highly-structured sequences. The complexity measure is defined via a restricted class of "zipline programs" (a variant of straight-line programs), which we call layered. We thus get a less expressive measure with a more efficient algorithm (compared to our results for ARC), demonstrating a possible tradeoff.
Methods to measure compression limits of quantized Gaussian signals
Computing the entropy rate of a quantized stationary Gaussian process
Abstract: We consider a stationary Gaussian process observed after uniform quantization. The entropy rate of the resulting integer sequence is the fundamental limit on lossless compression of the quantized signal, but it has no closed form, and the classical high-resolution approximation breaks down whenever the spectral density is small or vanishing on part of the band, as happens routinely after smoothing or filtering. Here we present two methods for computing the rate. The first is an analytical approximation obtained by combining an exact dithering identity with the Kolmogorov-Szegő formula; the quantization noise power acts as a floor on the spectral density, so the rate remains finite where the classical formula fails. The second is a Monte Carlo estimate of the exact rate. Its minimum-phase spectral factor represents the process as a finite moving average of Gaussian innovations, making the quantized sequence a hidden Markov process whose optimal (fully adapted) particle filter is available in closed form; by the Shannon-McMillan-Breiman theorem the rate follows from the filter's log-likelihood on a single long sequence. In experiments across six process families, the approximation agrees with the estimate to within a few millibits per sample over most of the parameter range for quantization steps up to the signal standard deviation, including strongly filtered processes on which the classical formula fails; at the most strongly filtered points the discrepancy grows to a few percent of the rate, and for much coarser steps the estimator should be used. We also compare lossless coders against this limit. At a step of one quarter of the standard deviation, linear predictive coding followed by an entropy coder comes within 2 to 4 percent of the entropy rate and FLAC within 3 to 32 percent, whereas five general-purpose compressors on the raw samples remain 12 to 89 percent above it.
Fix-free code construction solved for three codeword lengths across alphabets
The 3/4 Conjecture for q-Ary Fix-Free Codes With at Most Three Distinct Codeword Lengths
Abstract: We prove the \(3/4\) conjecture for \(q\)-ary fix-free codes with at most three distinct codeword lengths, for every integer \(q\geq2\). Every prescribed length distribution with Kraft sum at most \(3/4\) is realized by a deterministic construction. We introduce a matrix approach based on an exact identity for the overlap between forbidden prefix extensions and suffix residuals. Natural numerical order fixes the shortest layer, and the remaining selection problem is expressed through row and column counts. A fixed-cardinality interpolation theorem supplies feasible sets of every intermediate cardinality between nested endpoints, provided their differences satisfy one-sided uniqueness and either acyclicity or integral slack. Reverse-order selection handles the uniform cases directly; in the remaining cases, layer completion and aligned groups provide endpoints for interpolation. Together, these methods extend the binary three-length result to arbitrary finite alphabets and give a deterministic procedure for constructing the code.
Algorithm tests binary rank of matrices with fewer queries
Testing the Binary Rank with Polynomial Query Complexity
Abstract: We provide an adaptive two-sided error testing algorithm for the binary rank of a $0,1$ matrix $M$ with query complexity $O(d^3\log(d+1)/ε^2)$, where $d$ is the tested binary rank bound and $ε$ is the distance parameter. This answers an open question posed by Parnas, Ron and Shraibman~\cite{parnas2021property}, who asked whether the binary rank can be tested with query complexity polynomial in $d$ and $1/ε$. Furthermore, our testing algorithm can be used to find an approximate binary decomposition of $M$ with an additional $d(n+m)$ queries. That is, under the promise that the binary rank of $M$ is at most $d$, we show how to find, with probability at least $5/6$, two $0,1$ matrices $A',B'$ such that $M' = A' \cdot B'$ is a $0,1$ matrix which differs from $M$ on at most an $O(ε)$ fraction of its entries.
Leveraged learning measures information gained per answer bit received
Leveraged Learning: entropy destroyed per bit received
Abstract: A learner holds a prior belief over boolean maps that answer a finite set of $Q$ questions, and receives answers one by one. Each answer costs surprisal and destroys uncertainty, not only about the question asked but every question still unasked. We call the ratio the leverage: table entropy destroyed per bit of surprisal received. It is unit for a uniform prior, but with an intelligent prior can be higher (not lower). Averaged over the truth prior and over the question order, the leverage is found exactly, and is generated by one sequence: the mean entropy $G_\ell$ of the answers to $\ell$ questions. Sending the number of input bits to infinity at fixed asked fraction $t = \ell/Q$, the increments of that sequence become a profile $γ(t)$, and initial question entropy $η_0$. The leverage closes to a thermodynamic limit. $L(t) = [η_0 - (1-t)γ(t)]/\int_0^t γ$. Exchangeable priors, by de Finetti, all give a flat $γ(t)$ and hence a hyperbolic $L(t)$, their deduction confined to a boundary layer at $t = 0$. We construct a simplicity prior that escapes this, grading Boolean maps by the degree of their polynomial over $\mathbb{F}_2$ and budgeting weight across degree shells by a CDF $F$. Reed-Muller capacity then gives $γ(t) = 1 - F(t)$ exactly, so any nonincreasing profile, and any leverage curve it generates, is realizable at macroscopic times.