Nowadays, deep learning technique becomes the most popular fast-growing machine learning method in an Artificial Neural Network. The Convolution Neural Network (CNN) is one of the deep learning architecture that has been applied in the field of image analysis and image classification. In this paper, we proposed a novel emotion learning model with a deep learning network. The aim of the learning model is to reduce the affective gap, that extracts the objects and background features of an image semantically, such as high-level and low-level features. These extracted features accompanied with few others and it is more effective in emotion prediction model based on visual concepts of image, that leads to better emotion recognition performance. For training and testing, the experiment is conducted on IAPS (International Affective Picture System) dataset, the Artistic Photos, and the Emotion-Image dataset. An experimental result shows that the proposed model combines visual-content and low-level features of the image that provides promising results for Affective Emotion Classification task. © The Author(s) 2020.