In this research paper, the problem of vision-based sign language recognition which is used to translate signs to native or foreign language is addressed. This paper aims in designing a framework for segmenting and tracking skin objects from continuous signing videos and developing a fully automatic system to recognize signs that starts with breaking up signs into manageable subunits. A variety of spatiotemporal discriminative descriptors are extracted to form a feature vector for each subunit. A boosting algorithm is applied to the subunits to learn the subset of weak classifiers and combining them to strong classifier for each sign. The results obtained from the system shows that this proposed approach is promising for an effective and scalable system on real-world hand gesture recognition from continuous video sequences using boosted subunits.