Face recognition is one of the emerging areas in the field of biometric and computer vision that plays an important role in numerous time-bound applications such as ATM payment, criminal identification, E-Learning, healthcare, and online gaming. It can be compromised by various imposter attacks such as masks, print, or replay attacks. So, there is a requirement of a light-weight powerful classifier that could take significantly less time to minimize those effects by observing the liveness of a current person. In this paper, a lightweight permuted Xceptio-Inception/Reduction Convolutional Neural Network classifier has been proposed using depthwise convolution, permutation, reshape, and residual techniques for texture-based facial liveness recognition. It has been validated with moderately dense ImageNet benchmarked Convolutional Neural Network classifiers with respect to weight size, accuracy, precision, and recall. Here, we have considered some of the variants of most popular convolution neural networks such as AlexNet, Inception, ResNet, and VGGNet and applied these models for textured based facial liveness recognition. Before the training and testing of those classifiers, all the frontal face images from the FRAUD2, NUAA, and CASIA FASD imposter datasets had normalized, and the multi-colored space LBP feature maps extracted from these normalized image frames had supplied as inputs to the classifiers. The results show that the proposed convolutional neural network performs best among the above-standardized network models, whose total weights consumes less memory space, which leads to fast liveness face recognition. In the end, comparison with the previous work shows that it achieves almost the highest success rate and lowest Equal Error Rate as a non-intrusive classifier. © 2020, Springer Science+Business Media, LLC, part of Springer Nature.