White blood cells (Leukocytes) are considered to be an essential part of the human body’s immune system. The count of WBCs is considered to be a parameter for the indication of disease. Over time several methods have been proposed to classify these WBCs into their subtypes namely Neutrophils, Eosinophils, Basophils, Lymphocytes, and Monocytes which helps in the estimation of the body’s WBC count. These methods range from various morphological image processing-based methodologies to advanced deep neural systems. Due to the superior ability of neural systems to achieve the state of the art results more research is been carried out in this field. However, most of the such previously proposed methods have concentrated only in establishing and explaining the overall methodology for achieving high accuracy scores and less emphasis has been made in discussing the impact of modular changes in such methodologies like the impact of various activation functions, optimizers and data pre-processing methods very explicitly for this problem. This has led to a deficiency of work to be carried out with very recently developed activation functions and more essentially optimization algorithms other than backpropagation. It is extremely essential to explore and analyse different modules of the methodology to accelerate future research work further which might possibly help the research community in achieving a much better and efficient solution. This paper compares various architectures and discusses the behaviour and impact of different hyperparameters and proposes a novel methodology by incorporating recently developed swish activation to enhance the results. Unlike previously proposed methods of proposing single better neural network model this paper suggests a good choice of modular changes that could be incorporated in future works to enhance their results. © 2020, Springer Science+Business Media, LLC, part of Springer Nature.