An adaptive impairments assessment is necessary to evaluate the impact of various linear and nonlinear effects in future generation cognitive optical transport links. In this paper, an Adaptive Fiber Impairment Mitigation (AFIM) algorithm is proposed to identify a suitable mitigation scheme for the cognitive environment. The AFIM algorithm will assess fiber impairments and adaptively select a suitable mitigation scheme with minimum complexity based on the present network conditions and user performance target. The performance of AFIM algorithm is compared with Fixed Fiber Impairment Mitigation approach in terms of outage probability and outage capacity analysis. An Orthogonal Frequency Division Multiplexing based Mode Division Multiplexing system with Few Mode Fiber (FMF) is suggested as a solution to increase the nonlinearity threshold limit of the system. The L 2-by-3 nonlinear transform based Peak to Average Power Ratio reduction technique is implemented to mitigate fiber nonlinear effects in FMF based backbone and backhaul links. The performance analysis of the FMF system has been evaluated and compared with that of Single Mode Fiber system. The proposed analytical model and mitigation schemes are integrated with the AFIM algorithm to realize the cognitive optical network. Further, the result shows that AFIM algorithm enhances the system capacity by more than 6-folds at an outage probability of 0.5 and reduces the outage probability to 0.6 at the capacity range of 20 Gbps. © 2020, Springer Science+Business Media, LLC, part of Springer Nature.