Deep learning opens up a plethora of opportunities for academia and industry to invent new techniques to come up with modified or enhanced versions of standardized neural networks so that the customized technique is suitable for any specialized situations where the problem is about learning a complex mapping from the input to the output space. One such situation lies in a farm with huge cultivation area, where examining each of the plant for any anomalies is highly complex that it is impractical, if not impossible, for humans. In this chapter, the authors propose an optimized deep learning architectural model, combining various techniques in neural networks for a real-world application of deep learning in computer vision in precision farming. More precisely, thousands of crops are examined automatically and classified as healthy or unhealthy. The highlight of this architecture is the strategic usage of spatial and temporal features selectively so as to reduce the inference time.