Fine Needle Aspiration Cytology (FNAC) is a quick and minimally invasive technique used to diagnose breast cancer, specifically ductal carcinoma. The incidence of breast cancer (ductal carcinoma) is high among Indian women. Consequently, there is a volume burden on laboratories, primarily regional centers, for diagnosis. The pressure on laboratories and doctors to make a timely and correct diagnosis can be resolved by automating the process to an extent, especially where expertise is limited. Recent advances in Artificial Intelligence techniques on large and complex data have enabled better understanding in the domain of Computer-Aided Diagnosis, which helps both automate and digitize diagnosis. In this study, we have leveraged Convolutional Neural Networks (CNNs) to automate the diagnosis of ductal carcinoma in breast from images produced after FNAC is performed on breast tissue. We created a data set of FNAC images of breast lesions and extracted 1020 Region of Interest (RoI) patches from Giemsa-stained lesions and 631 RoI patches from H&E-stained lesions. The performance of various CNNs was tested on these patches. Three networks performed very well and have the potential to assist doctors in diagnosis. One of them was a light network we built—BFCNet (Breast FNAC Classification Network). It produced the highest average accuracies in the binary classification of Giemsa-stained patches (97.53%) and H&E-stained patches (96.59%). This network fits the data properly and performs well in other parameters. © 2021, The Author(s), under exclusive licence to Springer-Verlag London Ltd. part of Springer Nature.