A multi-robot-based fault detection system for railway tracks is proposed to eliminate manual human visual inspection. A hardware prototype is designed to implement a master–slave robot mechanism capable of detecting rail surface defects, which include cracks, squats, corrugations, and rust. The system incorporates ultrasonic sensor inputs coupled with image processing using OpenCV and deep learning algorithms to classify the surface faults detected. The proposed Convolutional Neural Network (CNN) model fared better compared to the Artificial Neural Network (ANN), random forest, and Support Vector Machine (SVM) algorithms based on accuracy, R-squared value, F1 score, and Mean-Squared Error (MSE). To eliminate manual inspection, the location and status of the fault can be conveyed to a central location enabling immediate attention by utilizing GSM, GPS, and cloud storage-based technologies. The system is extended to a multi-robot framework designed to optimize energy utilization, increase the lifetime of individual robots, and improve the overall network throughput. Thus, the Low Energy Adaptive Clustering Hierarchy (LEACH) protocol is simulated using 100 robot nodes, and the corresponding performance metrics are obtained. © 2020, Springer-Verlag London Ltd., part of Springer Nature.