The autonomous vehicle is the dream project of most of the majestic companies; however, providing a full-fledged autonomous vehicle is very complicated. In this paper, the pedestrian actions are captured using cameras and fine-tuned within a limited amount of time. Certain features of the captured video and their efficient feature descriptors achieve improved accuracy in pedestrian action recognition. The Skeleton based Spatio-Temporal Interest Points (S-STIP) feature is combined with the new interclass discriminative dictionaries. The sparse descriptor is constructed using sparse coding based on orthogonal matching pursuit algorithm and dictionary learning based on Efficient Block Coordinate Descent (EBCD) algorithm. Finally, the sparse descriptor is given as input to the SVM classifier for recognising pedestrian actions. The human action data sets KTH, Weizmann and JAAD are used for experimentation, and the combination of the S-STIP feature and the enhanced sparse descriptor achieves better performance compared to other existing action recognition methods. © 2021 Inderscience Enterprises Ltd.. All rights reserved.