Nonlinear effects play a major role in optical fibers and are occurred due to the changes in refractive index of the optical medium with respect to the intensity of light propagation. Four-wave mixing (FWM) and cross-phase modulation (XPM) are the major nonlinear effects that affect the performance of the dense wavelength division multiplexing (DWDM) system. In FWM, three signals are combined to form a fourth spurious mixing component results in waste of power from wanted signals to unwanted spurious signals which lead to cross talk and degrade the system performance. Nonlinear phase changing due to power variations in co-propagating light is called XPM which causes pulse distortion and affects the system transmission capacity. This paper shows the effect of XPM and FWM in return-to-zero (RZ) and non-return-to-zero (NRZ) signals for various ranges of dispersion. The effect has been analyzed based on eye diagrams, Q values, BER values, and FWM power for two, four, and eight channels at 10 Gbps in DWDM systems. © 2018, Springer Nature Singapore Pte Ltd.