Due to the huge increase in customers for different fabrics in this generation, the texture of the fabrics becomes an important issue thus bringing the requirement for correct and perfect detection of the fabric defects. In the existing semiautomated systems, a quality inspector takes 5 m/min with a defect and 15 m/min without defect to identify and rectify the defects with the resolution of 1 mm/pixel. This process results in the loss of the factory’s overall throughput and efficiency. While manufacturing fabrics, there may be various defects like hole, missing yarn, broken yarn, stain, etc. These defects incur huge losses to the textile industry as they cause customer dissatisfaction. In order to reduce such losses, detection of defects beforehand is very important. Our project uses the concept of deep learning for the detection of colored fabric defects. The working and reliability of the fabric defect detection system presented is evaluated through vigorous experiments of real fabric samples with different defects. © Springer Nature Singapore Pte Ltd. 2020.