A number of applications are growing day by day and so the traffic. The need for bandwidth is also increasing at a rapid rate. The bandwidth and speed with which data can be transferred was very less when compared to core networks. The access network which was once a bottleneck is no longer so because of use of optic fiber (FTTH networks). A number of variants of Passive Optical Network (PON) have been proposed like the WDM PON and the Hybrid PON. Hybrid PON is a combination of TDM PON and WDM PON and is advantageous over WDMPON. This paper focuses on high capacity networks that can provide high data rate and long reach in the access part of the network. NRZ modulation format is normally used for transmission. We consider the advantages provided by the advanced modulation formats like DPSK. This modulation format is used to here and its benefits are evaluated in Hybrid PON network to increase the capacity and the reach of the network. Parameters like the BER and the Q factors are analysed using Optisystem Software. Distortion and the phenomena of dispersion can limit the performance of such a system. Hence Dispersion compensation mechanisms like the Dispersion Compensation Fiber (DCF) are used in the system to transmit data over large distance.