The recent increase in computational power and data available to the general public has given rise to the development of a plethora of highly performant deep neural network architectures. A popular and successful application of these has been in the field of image recognition, in which many yield promising results. Yet, there are differences and caveats in each, and knowing the right one to choose for a particular task can save a lot of money, time, and computational resources. In order to take a closer look at the differences in their performance, we have compared 5 of the many top architectures, namely the VGG16, VGG19, DenseNet121, ResNet50, and ResNet152V2 in terms of their error percentage, class-balanced error percentage, standard deviation of class-balanced accuracy, and training time. To achieve this, we have used 2 datasets, the Kuzushiji MNIST or KMNIST dataset, which is a balanced dataset with 10 classes and the Kuzushiji 49 or K49 dataset, an imbalanced dataset with 49 classes, both of which comprise cursive characters from the Japanese Hiragana script. All of the results were computed and verified using 10 fold cross validation. For the Kuzushiji MNIST dataset, the VGG16 showed the best results in all of the aspects. For the Kuzushiji 49 dataset, the DenseNet121 showed the highest performance in all aspects other than training time, and the latter was least for the VGG16. © 2020 IEEE.