Tongue analysis is an effective indicative strategy for assessing the state of the internal organs and to detect associated diseases. In this paper, we propose a disease detection method with the use of a regular smartphone and to track disease on the go. Colour, texture and geometric features are extracted from the images of the tongue captured and are used to train classification models to detect associated diseases. The primary objective of this paper is to classify images of tongues which are healthy or are diagnosed with either thyroid, heart disease or gastritis. The secondary objective is to perform comparative analysis of machine learning algorithms to find the best performing models and their differences in hope to achieve better performance than those which have been achieved till date. In addition, this paper also takes into consideration the recent breakout, COVID-19, and aims to identify features which potentially help to classify tongues of patients who have been diagnosed with this pandemic. © 2020, World Academy of Research in Science and Engineering. All rights reserved.