In India, a huge country with the largest railway network in the world at almost 1,20,000 kilometers has a huge problem of railway collisions. There have been multiple news stories ranging from the death of 13 children in Uttar Pradesh, India to 5 elephants dying due to railway related accidents. Most of these issues are due to the lack of management at railway crossings. Human monitoring has been a proposed solution, however, due to India's sheer scale, it is very difficult to monitor every crossing, especially in the rural and forested areas. In this paper, we are proposing a system that can automate the monitoring of railway crossings. We plan on finding out the most frequented railway crossings and fixing an overhead camera at the scene, which can monitor the crossing. This feed will be fed into a SSD object detection algorithm that will detect an object in the feed. Once the object has been detected, the object will be monitored and if the object has been on the track or in the vicinity of the tracks, an alert will be sent to the train stations to both the stations closest to the object - saying the trains needs to slow down in this area. The object detected will be transferred to the operator as well who can give the driver an idea what to do - honk for humans and slow down for animals for example. We feel that this will effectively reduce the collisions and solve the problem of railway collisions. © 2019 IEEE.