StrTransformer: Source-Wise Structured Transformers for Unsupervised Blind Source Recovery
2026-05-25 • Machine Learning
Machine Learning
AI summaryⓘ
The authors present StrTransformer, a new method for separating mixed signals into their original sources without prior knowledge. Instead of using a typical encoder, their model directly adjusts the hidden source signals along with a mixer and special Transformer branches that focus on each source. Each source is broken into smaller parts, processed with attention to local patterns, and reconstructed to enforce structural consistency. They also introduce a controller that helps each branch focus on different time scales, improving separation and specialization. Their experiments show the method can recover distinct source patterns effectively.
blind source recoveryTransformerlatent variablesmulti-scale representationmasked reconstructionstructural priortemporal scaleattention mechanismsignal separationpermutation symmetry
Authors
Yuan-Hao Wei
Abstract
This paper proposes StrTransformer, a source-wise structured Transformer framework for blind source recovery and branch-wise latent modeling. Instead of using an encoder to infer latent variables, StrTransformer directly optimizes the latent source matrix together with an observation-space mixer and source-wise structural Transformer branches. The mixer enforces reconstruction consistency, while each Transformer branch imposes a differentiable structural constraint on one latent source trajectory. Specifically, each source is converted into multi-scale patch tokens, randomly masked, processed by a locality-biased Transformer, and evaluated through a masked patch reconstruction energy. This energy acts as an implicit source-wise structural prior. To encourage different latent branches to specialize into different temporal regimes, StrTransformer further introduces an ordered multi-scale controller that learns branch-specific patch-scale weights, ordered scale centers, and locality attention slopes. The resulting objective combines observation reconstruction, source-wise structural regularization, and modular auxiliary penalties for separation and scale specialization. We analyze the decoupling and coupling structure of the objective, the regularized exact-reconstruction fiber, and the reduction of permutation symmetry induced by ordered branch descriptors. A controlled case study shows that the learned branches converge to distinct temporal-scale structures and recover source-aligned latent trajectories under post-hoc evaluation.