The coconut palm plantation industry relies heavily on expert advice to identify and treat infections. Computer vision in deep learning technology opened up an avenue in the agriculture domain to find a solution. This study focuses on the development of an end-to-end framework to detect stem bleeding disease, leaf blight disease, and pest infection by Red palm weevil in coconut trees by applying image processing and deep learning technology. A set of hand-collected images of healthy and unhealthy coconut tree images were segmented by employing popular segmentation algorithms to easily locate the abnormal boundaries. The custom-designed deep 2D-Convolutional Neural Network (CNN) is trained to predict diseases and pest infections. Also, the state of the art Keras pre-trained CNN models VGG16, VGG19, InceptionV3, DenseNet201, MobileNet, Xception, InceptionResNetV2, and NASNetMobile were fine-tuned to classify the images either as infected or as healthy through the inductive transfer learning method. The empirical study ascertains that k-means clustering segmentation was more effective than the Thresholding and Watershed segmentation methods. Furthermore, InceptionResNetV2 and MobileNet obtained a classification accuracy of 81.48% and 82.10%, respectively, and Cohen's Kappa values of 0.77 and 0.74, respectively. The hand-designed CNN model achieved 96.94% validation accuracy with a Kappa value of 0.91. The MobileNet model and customized 2D-CNN model were deployed in the web application through the micro-web framework Flask to automatically detect the coconut tree disease or pest infection. © 2021 Elsevier B.V.