Global market forecasts predicted that by 2020, more than 26 billion internet devices and connections universally interconnected will require nearly 3 times the data traffic generated when compared to the year 2015. The increase in data traffic, demands for enormous bandwidth capacity. The potential to deliver 10 Gbps of huge data to individual businesses and households will be of paramount importance and a challenging issue for the present day service providers. An intensive study is carried out for the Fiber-To-The-Home Passive Optical Network (FTTH PON) for their use in the optical communication, due to their high data rates and more bandwidth. The current evolution of Next Generation-Passive Optical Networks Stage 2 (NG-PON2) network is the primary key technology for the growing demands of higher bandwidth and transmission of the data to the subscribers present in the access network from the service providers. Time Wavelength Division Multiplexing PON (TWDM-PON) architecture is the viable essential solution for NG-PON2 which provides more bandwidth for bidirectional transmission. This article proposes a design for extended reach TWDM-PON based on reflective semiconductor optical amplifier (RSOA). The exclusive feature of the RSOA is the wavelength conversion, which replaces the transmitters in the subscriber end. The Quality of Service (QoS) performance is critically analyzed for different optical modulation formats in proposed extended reach TWDM-PON using RSOA. The TWDM-PON using RSOA is simulated and investigated for different photodetectors. The analysis is also carried for various distance and data rates. The results exhibited that APD receivers have better performance of minimum bit error rate obtained is 10-11 and minimum Q factor is 6.2 when compared with PiN receivers. The comparative analysis of different modulation formats shows that the Carrier Suppressed Return to Zero- Differential Phase Shift Keying (CSRZ-DPSK) gives the best performance for longer distance and large data rates and Return to Zero(RZ) gives the least performance. © Science and Information Organization.