Emotion is subjective which convey rich semantics based on an image that induces different emotion based on each individual. A novel method is proposed for emotion classification by using deep learning network with transfer learning method. Transfer learning techniques are the predictive model that reuses the model trained on related predictive problems. The purpose of the proposed work is to classify the emotion perception from images based on visual features. Image augmentation and segmentation is performed to build powerful classifier. The performance of deep convolution neural network (CNN) is improved with transfer learning techniques in large scale Image-Emotion-dataset effectively. The experiments conducted on this dataset and result shows that proposed method achieve promising significant effect on emotion classification with good accuracy and PDA value, when compared with other state-of-art methods. © 2020, Springer Science+Business Media, LLC, part of Springer Nature.