Surface defect inspection plays a vital role in leather manufacturing. Current practice involves an expert to inspect each piece of leather individually and detect defects manually. However, such a manual inspection is highly subjective and varies quite considerably from one assorter to another. Computer vision system for natural material like leather is a challenging research problem. This study describes the application of computer vision system to capture leather surface images and use of a novel Fast Convergence Particle Swarm Optimization (FCPSO) algorithm on a set of handcrafted texture features viz., GLCM and classified using supervised classifiers viz., Multi Layer Perceptron (MLP), Decision Tree (DT), SVM, Naïve Bayes, K-Nearest Neighbors (KNN) and Random Forest (RF). FCPSO using modified fitness function by selective band Shannon entropy is implemented to segment industrial leather images. Segmentation efficiency of the proposed FCPSO algorithm is evaluated and its performance is compared with other optimization algorithms. Efficiency of the segmentation algorithms is evaluated using performance measures such as average difference (AD), Area Error Rate (AER), Edge-based structural similarity index (ESSIM), F-Score, Normalized correlation coefficient (NK), Overlap Error (OE), structural content (SC), Structural similarity index (SSIM) and Zijdenbos similarity index (ZSI). Correlation of the segmented area using FCPSO with the experts’ ground truth is found to be high with R value of 0.84. Feature extraction is carried out using GLCM texture features and the most prominent features were selected using statistical t-test and correlation coefficient. Experimental results showed encouraging results for random forest classifier confirming the potential of the proposed system for automatic leather defect classification. © 2020, Springer Science+Business Media, LLC, part of Springer Nature.