Advanced optimized detection scheme based on a neural network in a bandwidth-limited IM/DD system.

In this Letter, we propose a high-performance optimized detection scheme based on a neural network (NN) in a receiver digital signal processing (DSP) for bandwidth-limited intensity modulation and direct detection (IM/DD) transmission systems. The NN-based optimized detection scheme consists of two components, an NN-based lookup table (NN-LUT) and an NN-based log-maximum estimation with a fixed number of surviving state (NN-MAP) decoder. The NN-LUT provides more accurate and sufficient information (PI) to the decoder than the conventional filter-form PI without increasing computational complexity. The NN-MAP decoder can be optimized for detection using the NN-form PI. The proposed scheme is implemented in a 122-Gbps IM/DD optical 4-level pulse amplitude modulation (PAM-4) system with a 3.7-GHz 3-dB bandwidth. The results demonstrate that the proposed NN-MAP scheme can enhance receiver sensitivity by 1.4 dB under the same complexity compared to conventional optimized detection, achieving a 20% forward error correction threshold. To the best of our knowledge, this represents the inaugural instance of NN being employed for optimized detection.