The use of computer vision along with machine learning techniques resulted in a breakthrough in medical and social fields. One such intervention is the sign language detection, enabling communication for differently-abled people. Hand gestures are the common method used for this. This paper deals with using image processing techniques like RGB to binary image conversion, skin detection, edge detection to extract the important features of the gesture and provide them as inputs to Convolution Neural Networks (CNN) to improve the accuracy in predicting American Sign Language alphabets (ASL). Using the preprocessed image data set of the hand gesture in our proposed mechanism has predicted the alphabets with higher accuracy up to 99% as compared to the model trained with original data set of images. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021