Sign language is the most communal language used by the deaf people among themselves for any kind of communication. People who are not familiar with Sign Language face difficulty in interacting with the deaf people. Hence, an effective system should be established to attain and discriminate the sign language. In our proposed system, we developed a framework for extracting and recognizing the sign gesture language and which is classified into four stages like noise removal using median filter, Segmentation using Modified Region Growing Algorithm (MRGA), feature extraction and recognition using Adaptive Genetic Fuzzy Classifier (AGFC). We have used Genetic algorithm with Fuzzy classifier to find out the optimal rules generated by Fuzzy classifier.