Over the past two years, gesture recognition has become the powerful communication source to the hearing-impaired society. Furthermore, it is supportive in creating interaction between the hu man and the computer. However, the intricacy against the gesture recognition arises when the environment is relatively complex. In this paper, recognition algorithm with feature selection based on Self-Improved Genetic Algorithm (SIGA) is proposed to promote proficient gesture recognition. Furthermore, the recognition process of this paper includes segmentation, feature extraction and feed- forward neural network classification. Subsequent to the gesture recognition experiment, the performance analysis of the proposed SIGA is compared with the conventional methods as reported in the literature along with standard Genetic Algorithm (GA). In addition, the effect of optimization and the feature sensitivity is also demonstrated. Thus, this method makes aggregate performance against the conventional algorithms. © TAETI.