With primitive approaches, the diagnosis and therapy were operated at the cellular, molecular, or even at the genetic level. As the diagnostic techniques are more concentrated towards molecular level, multi modal imaging becomes specifically essential. Multi-modal imaging has extensive applications in clinical as well as in pre-clinical studies. Positron Emission Tomography (PET) has flourished in the field of nuclear medicine, which has motivated it to fuse with Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) for PET/CT and PET/MRI respectively. However, the challenges in PET/CT are due to the inability of simultaneous acquisition and reduced soft tissue contrast, which has led to the development of PET/MRI. Also, MRI offers the better soft tissue contrast over CT. Hence, fusion of PET and MRI results in combining structural information with functional image from PET. Yet, it has many technical challenges due to the interference between the modalities. Also, it must be resolved with various approaches for addressing the shortcomings of each system and improvise on the image quantification system. This review elaborates on the various challenges in the present PET/MRI system and the future directions of the hybrid modality. Also, the different data acquisition and analysis techniques of PET/MRI system are discussed with enhanced details on the software tools. © 2018, The Author(s).