State space models dynamically reshape sequence memory for better conditioning
MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models
Machine LearningArtificial Intelligence
Summary
Sequence models help computers understand and generate data that changes over time, like speech or text. Traditional methods to guide these models use fixed underlying memory, limiting adaptability. The authors present MaRK, a way to adjust the core memory behavior of a pre-trained model using context, allowing it to respond differently at each step. This improves the model's ability to generate sequences iteratively while keeping the system stable and efficient to fine-tune.
What this means in practice
- •For natural language processing engineers: Enable fine-tuning of large sequence models for iterative text generation with fewer trainable parameters and stable memory adaptation.
- •For signal processing developers: Design dynamic filters in time-series software that adjust their memory behavior efficiently for different input contexts.
Authors
Syed Ibrahim Omer, Ginny Y. Wong. Xiangyu Zhao
Abstract
State Space Models (SSMs) offer an efficient alternative to Transformers for sequence modeling, yet conditioning pre-trained SSMs for iterative generation typically operates outside the recurrent operator, through input injection or activation modulation. While such mechanisms expose the model to conditioning information, they leave the underlying temporal dynamics fixed. We introduce MaRK (Markov-adapted Recurrent Kernels), a dynamic operator-conditioning framework that maps context vectors directly into bounded modulations of a frozen SSM's recurrence ($A$), read-in ($B$), read-out ($C$), skip ($D$), and discretization ($Δ$) parameters. Viewed through the lens of LPV-SSM systems, MaRK induces a context-indexed family of Markov parameter sequences, allowing each diffusion timestep to reshape the model's input-output memory kernel. We instantiate MaRK on a frozen 111M-parameter Hydra SSM backbone and study three adapter geometries: Hypernet, Chebyshev polynomial, and Discrete Cosine Transform kernels. Since these adapters modify the Markov parameter sequence through low-rank auxiliary maps on the frozen backbone, parameter-efficient fine-tuning arises as a structural consequence of the adaptation mechanism itself, requiring only 6.3--11M trainable auxiliary parameters to transition from a bidirectional objective to an iterative diffusion regime. The bounded recurrence parameterization further yields an analytic Affine Quadratic Stability certificate for the modulated recurrence. Through synthetic LPV recovery experiments and Markov-operator diagnostics, we show that MaRK recovers coordinate-invariant temporal operators under matched assumptions and produces distinct, stable timestep-conditioned memory profiles. Empirically, the Chebyshev variant yields the strongest performance, achieving an average validation loss of 2.55, followed by the DCT (2.59) and Hypernet (3.77) geometries.