In many image and video processing algorithms, pre-processing filters are the first and most important step. These filters reduce the noise present in the images, preserving the required information depending on the action performed. Due to increasing demands for real time processing systems, the traditional software solutions cannot be chosen. In this paper, we are introducing a new hardware architecture for image processing filters, which are optimized to increase the speed of processing, which is vital for real time systems. The decision based algorithm used for optimizing the median filter helps in avoiding corruption of original pixels and the hardware optimization used for the Gaussian filter reduces the required computations, making it faster. The proposed architecture is simulated in ModelSim and the simulation results are presented. © 2018 IEEE.