Pedestrian detection is a significant problem in infrared (IR) images that find varieties of applications in defense systems. The performance of the state-of-the-art of pedestrian detection methods in IR images still have abundant space for improvement towards accuracy. In this paper, a three-level filtering-based pedestrian block detection method is proposed. In addition, a rotation and scale invariant structure element descriptor (RSSED) is proposed for pedestrian detection in infrared (IR) images. To extract RSSED features, the pedestrian block detection result is encoded using local binary pattern (LBP). The LBP encoded image is quantised adaptively to four levels. Further, the proposed RSSEDis used to generate the feature descriptor fromthe quantised image. Finally, support vector machine (SVM) is used to classify the objects in given IR image into pedestrian and nonpedestrian. The experimental results demonstrate that the proposed method performs effectively in pedestrian detection than the other methods. Copyright © 2017 Inderscience Enterprises Ltd.