The objective of this research is to develop an automatic medical diagnostic system which can be readily available to the common man, especially to those who cannot receive proper medical care. The approach basically includes a combination of soft and hard inputs. Soft inputs include a variety of common symptoms such as fever, headache and cough. Each selected disease corresponds to a variety of general symptoms. Hard inputs include images of the tongue as it is widely used by the doctor to diagnose the various disorders. The analysis of hard inputs was divided into two phases, namely chromatic colour analysis and texture-based statistical analysis. Once the feature vectors are encoded from the hard and soft inputs, they are fed to a neural network for developing a classification model. Neural network is trained with four different algorithms and the performance is analysed. Copyright © 2014 Inderscience Enterprises Ltd.