Optical communication systems represent the backbone of modern long-haul communication networks. Nonlinear impairments such as the Kerr effect pose threat and deteriorate transmission in long-haul optical communication systems. At high launch powers, nonlinear effects become more important and their compensation is entailed to improve the transmission performance. In this paper, one of the major nonlinear effects of four waves mixing (FWM) is addressed using a theoretical model and validated through simulation in Optisystem. Two mechanisms with and without dispersion variation are investigated. The impact of FWM is investigated in terms of various values of launch power. The transmission performances of the proposed model are evaluated on the basis of bit error rate, optical signal to noise ratio, and quality factor using key parameters of effective area, nonlinear refractive index, nonlinear dispersion, and linear dispersion.