This paper presents a person identification system which combines recognition of facial features as well as spoken word using visual features alone. It incorporates a face recognition algorithm to identify the person, followed by spoken word recognition of 'lip-read' password. For face recognition, PCA is used for feature extraction, followed by a KNN based classification on the reduced dimensionality features. Spoken word recognition of passwords is performed using a Visual Lip reading (Visual ASR). The visual features corresponding to the spoken word is extracted using DWT, which are then recognized using a HMM based approaches. Since evidences from face recognition and visual lip reading could be complementary in nature, the scores from the two modalities are combined. Based on the combined evidences, decision making is for person identification is carried out. The performance for face identification is 90% while the accuracy for visual speech recognition is 72%. By combining these evidences, an improved accuracy 98% is achieved. © 2015 IEEE.