The automatic classification of animal images is an onerous task due to the challenging image conditions, especially when it comes to animal breeds. In this paper, we built a semi-supervised learning based Multi-part Convolutional Neural Network (MP-CNN) that classifies 35,992 animal images from ImageNet into 27 different classes of animals. The proposed model classifies the animals on both generic and fine-grained level. The animal breeds are accurately classified using Multi-part Convolutional Neural Network with a hybrid feature extraction framework of Fisher Vector based Stacked Autoencoder. Furthermore, with Semi-supervised learning based pseudo-labels, the model classifies new classes of unlabeled images too. Modified Hellinger Kernel classifier has been used to re-train the misclassified classes of animals and thereby improve the performance obtained from MP-CNN. The model has experimented with varied tasks to analyze its performance in each of the cases. The experimental results have proved that the coalesced approach of MP-CNN with pseudo-labels can accurately classify animal breeds and we have achieved an accuracy of 99.95% from the proposed model. © 2013 IEEE.