Optical amplifiers play a critical role in ensuring the efficient transmission of signals in optical fiber communication systems by mitigating attenuation. Our research focused on designing a 400-channel ultra-dense wavelength division multiplexing system with 6.25 GHz narrow channel spacing and 100 Gbps per channel data rate, utilizing an advanced modulation technique, namely quadrature amplitude modulation. To counteract the effects of four-wave mixing (FWM) transmission impairment, we investigated the most suitable cascaded arrangement of optical amplifiers, including erbium-doped fiber amplifier (EDFA), thulium-doped fiber amplifier (TDFA), and praseodymium-doped fiber amplifier. We conducted experiments to measure attenuation and dispersion under the influence of FWM, while taking into account various parameters such as bit error rate (BER), optical signal-to-noise ratio (OSNR), quality factor (Q-factor), eye closure, and received output power with varying distance, number of channels, and input power. Our simulation results indicated that the TDFA-EDFA-TDFA amplifier cascaded arrangement is optimal for signal transmission with minimal attenuation under FWM. We observed a minimum BER value of approximately 10⁻¹⁴, a maximum OSNR value of 40.6 dB, a maximum Q-factor of 13.8 dB, and a minimum eye closure value of 1.39 dB with varying distances. We also recorded a minimum value of received crosstalk of − 10.3 dB and a maximum Q-factor of 17.6 dB with varying numbers of channels. With varying input powers, we noted a maximum output power of − 42 dBm and a minimum BER value of about 10⁻¹⁵.