Radio over Fiber is the integration of the wireless communication over the Fiber system is the next generation communication region. The effect of the four wave mixing plays an important role while transferring the data in the Fiber systems. The system introduces more adherable results during the data transmissions in Radio over Fiber Systems. Hence we propose the new optimization algorithm so called Cognitive Optimization algorithms which can be integrated to overcome the effects of above mentioned. The cognitive optimization is the new algorithm which deals with the adoption of modulations techniques which can be adaptive as the distance of the implementation increases. We have implemented with the different modulations and compared with the our algorithms to prove that the adoption of our algorithms finds its suitable when the Radio over Fiber is can be extended to the 4G networks and also it has been tested with the LTE-A signals. © 2018, Springer Science+Business Media, LLC, part of Springer Nature.