Tracking faces and identifying expression from a video files is a complex mining application due to the fact that even a delicate change in the emotion of the face could convey a great variation as an expression. This paper presents a robust novel method to track both male and female faces and recognize the facial expression in video files. Face localization in a digital image, which is being grabbed from the video source is performed using the hue-saturation and YCbCr based algorithms. Morphological algorithm over this region signals the eyes and mouth position in that region. Lips, eyebrows and eyelids are then identified. Distance between eyebrow and eyelid and the slope of the curvature of the lips are derived which facilitates in deciding the facial expression of the given digital image. © Medwell Online, 2006.