Breast cancer is one of the significant tumor death in women. Computer-aided diagnosis (CAD) supports the radiologists in recognizing the irregularities in an efficient manner. In this work, a novel CAD system proposed for mammogram image analysis based on grey wolf optimizer (GWO) with rough set theory. Texture, intensity, and shape-based features are extracted from mass segmented mammogram images. To derive the appropriate features from the extracted feature set, a novel dimensionality reduction algorithm is proposed based on GWO with rough set theory. GWO is a novel bio-inspired optimization algorithm, stimulated based on hunting activities and social hierarchy of the grey wolves. In this paper, a hybridization of GWO and Rough Set (GWORS) methods are used to find the significant features from the extracted mammogram images. To evaluate the effectiveness of the proposed GWORS, we compare it with other well-known rough set and bio-inspired feature selection algorithms including particle swarm optimize, genetic algorithm, Quick Reduct and Relative Reduct. From empirical results, it is observed that the proposed GWORS outperforms the other techniques in terms of accuracy, F-Measures and receiver operating characteristic curve. © 2021, The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature.