Bi-lstm detection improved with pre-whitening and bcjr soft posterior sharing
Pre-Whitening and BCJR Posterior Distillation for Bi-LSTM Detection in Faster-than-Nyquist Signaling
Machine Learning
Summary
Detecting faster-than-Nyquist signals, which are densely packed digital signals, is challenging because of interference between symbols. The authors find that changing the recurrent neural network’s structure to use multiple nested windows does not improve detection accuracy. Instead, they improve results by processing the input to remove certain noise correlations ('pre-whitening') and teaching the network using the soft outputs of a more complex BCJR detector. This combined approach reduces errors closer to the BCJR benchmark with only a small increase in network size.
What this means in practice
- •For communication system engineers: Enhance digital receiver designs for faster-than-Nyquist signaling with improved accuracy using pre-whitening and distillation techniques.
- •For hardware signal processing teams: Implement low-complexity recurrent detectors in hardware that approach optimal performance through input preprocessing and soft posterior guidance.
Tested on simulated data.
Authors
Nurettin Safak, Osman Tokluoglu, Enver Cavus
Abstract
Recurrent detectors such as bidirectional long short-term memory (Bi-LSTM) networks are low-complexity alternatives to the optimal Bahl-Cocke-Jelinek-Raviv (BCJR) detector for faster-than-Nyquist (FTN) signaling. Motivated by convolutional detectors that build the intersymbol interference (ISI) structure into their architecture, we ask whether processing nested ISI windows in separate recurrent branches improves the bit error rate (BER) of a Bi-LSTM. Across roughly 260 controlled trainings it does not: at a matched parameter budget and a matched readout, the multi-window architecture never significantly beats a plain Bi-LSTM. Nested windowing is an invertible rearrangement that adds no information, extra branches only add bottlenecks, and a distillation diagnostic shows the network is already near optimal for its window. The limitation is therefore the observation model, not the architecture. Keeping the architecture fixed, we pre-whiten the input, restoring the conditional independence that colored matched-filter noise violates, and distill the BCJR soft posterior into the network. With 3.4% more parameters this reaches 1.05 times the BCJR BER at a compression factor of 0.8 and 1.89 times at 0.7, improving to 1.47 times when the whitened window is widened. The 23.7% BER reduction at 0.8 requires an ill-conditioned ISI matrix but is not monotone in the conditioning, and it holds across five independent noise realizations and a symbol-level McNemar test.