Optical scanning holography is an incoherent optical image processing system. It is a technique, where the complete information of an object or image will be recorded as a hologram and later reconstructed to get back the original image. In the hologram reconstruction process, a virtual image is formed along with the real image, which appears as a twin image noise. To eliminate such noises, a technique of hybrid algorithm is used while recording the hologram itself. The MATLAB R2012b image processing tool is used for simulation and the simulated values are tabulated and compared with the existing method in terms of peak signal to noise ratio, mean square error. In the reconstruction, the proposed method results are having 26% increment in the MSE and PSNR values. To further improve the MSE and PSNR values a case study using different denoising techniques combined with the proposed hybrid algorithm is used and found considerable improvement of 32%. Hence the image quality is increased. © 2020 Inderscience Enterprises Ltd.