In this paper, a novel and robust rotation and scale invariant structuring elements based descriptor (RSSED) for pedestrian classification in infrared (IR) images is proposed. In addition, a segmentation method using difference of Gaussian (DoG) and horizontal intensity projection is proposed. The three major steps are moving object segmentation, feature extraction and classification of objects as pedestrian or non-pedestrian. The segmentation result is used to extract the RSSED feature descriptor. To extract features, the segmentation result is encoded using local directional pattern (LDP). This helps in the identification of local textural patterns. The LDP encoded image is further quantized adaptively to four levels. Finally the proposed RSSED is used to formalize the descriptor from the quantized image. Support vector machine is employed for classification of the moving objects in a given IR image into pedestrian and non-pedestrian classes. The segmentation results shows the robustness in extracting the moving objects. The classification results obtained from SVM classifier shows the efficacy of the proposed method. © 2016 Elsevier B.V.