Bi-LSTM signal detection improves using pre-whitening and BCJR teaching
Pre-Whitening and BCJR Posterior Distillation for Bi-LSTM Detection in Faster-than-Nyquist Signaling
Machine Learning
Summary
Detecting signals in faster-than-Nyquist communication is hard because signals can interfere with each other. The authors looked at a special kind of neural network called Bi-LSTM to see if breaking the problem into smaller parts helps. They found it does not improve accuracy much. Instead, they improved the signal detection by first cleaning the input data (pre-whitening) and then teaching the Bi-LSTM network using information from a nearly optimal method called BCJR. This approach reduced errors and worked well across tests.
Bi-LSTMFaster-than-Nyquist signalingIntersymbol interference (ISI)Bit error rate (BER)Pre-whiteningBCJR detectorPosterior distillationRecurrent neural networkMatched filterSignal detection
Authors
Nurettin Safak, Osman Tokluoglu, Enver Cavus
Abstract
Recurrent detectors such as bidirectional long short-term memory (Bi-LSTM) networks are low-complexity alternatives to the optimal Bahl-Cocke-Jelinek-Raviv (BCJR) detector for faster-than-Nyquist (FTN) signaling. Motivated by convolutional detectors that build the intersymbol interference (ISI) structure into their architecture, we ask whether processing nested ISI windows in separate recurrent branches improves the bit error rate (BER) of a Bi-LSTM. Across roughly 260 controlled trainings it does not: at a matched parameter budget and a matched readout, the multi-window architecture never significantly beats a plain Bi-LSTM. Nested windowing is an invertible rearrangement that adds no information, extra branches only add bottlenecks, and a distillation diagnostic shows the network is already near optimal for its window. The limitation is therefore the observation model, not the architecture. Keeping the architecture fixed, we pre-whiten the input, restoring the conditional independence that colored matched-filter noise violates, and distill the BCJR soft posterior into the network. With 3.4% more parameters this reaches 1.05 times the BCJR BER at a compression factor of 0.8 and 1.89 times at 0.7, improving to 1.47 times when the whitened window is widened. The 23.7% BER reduction at 0.8 requires an ill-conditioned ISI matrix but is not monotone in the conditioning, and it holds across five independent noise realizations and a symbol-level McNemar test.