Fifth and future generation (5G and B5G) wireless networks aim to serve users with higher data rates and lower latency. Data traffic due to the rapid growth in communication has motivated the study of Multiple Input Multiple Output (MIMO) systems. They utilize multiple antennas in both transmitter and receiver sides. It is necessary to improve the existing technology to achieve fast and reliable communication. In this research work, a rectangular array antenna based hybrid beamforming in a massive MIMO model has been proposed to improve the spectral efficiency of the system. Thus channel capacity with small RF chains is used. To achieve the high signal strength in the main lobe, Chebyshev tapering has been used to suppress the side lobes signals. In this manner, the proposed Hybrid Beamforming for Massive Output MIMO has been realized with a small complexity and higher spectral efficiency. In this research work, the spectral efficiency of both proposed Hybrid and fully-digital beamforming with a different number of RF chains for a various number of antennas at the transmitter, the receiver side has been analyzed. From the simulation results, it has been observed that the proposed rectangular array antenna based Hybrid beamforming in a massive MIMO system reduces the computational complexity up to 99% as compared with conventional fully digital beamforming to achieve the same spectral efficiencies, which is a productive model for 5G wireless networks. © 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.