One of the most important characteristics of human emotion is micro-expression. This micro-expression can be grouped based on several classes such as Contempt, Fear, Happy, Tense, Disgust, Repression, Surprise, and Sad. Even though micro-expression being one of the major research interest areas, there is a lot of scope for improvement in terms of micro-expression to emotion mapping. The primary objective of this paper is to infer the emotions of humans based on their micro-expressions using deep learning techniques. In this paper, videos from standard micro-expression benchmarked databases like CASME I,SMIC-HS, SMIC-NIR, SMIC-VIS, and CASME II were collected and parsed into images and then trained and tested using an Attention embedded Residual Network model. This model brings a novel method of embedding Attention on Residual Convolution Neural Networks, which results in carrying the most significant dominating features till the very end. The proposed work on these micro-expression datasets yields better results with a state-of-the-art accuracy of more than 95% in each dataset. For verification purposes, the videos collected real-time manually from different gender people were tested and an accuracy score of 87.37% and an F1_score of 86.92% was achieved. © 2020, Institute of Advanced Scientific Research, Inc.. All rights reserved.