The complexity in retrieving diverse images with different affine transformations poses a challenging issue to researchers. Hence, this paper offers one such framework targeting the aforesaid concern. Accordingly, a three step retrieval framework is proposed that initially extracts Invariant Zernike Moment Descriptor (IZMD) features from the query database. The attained features are then vector quantized by the Geodesic Self-Organizing Map (GeoSOM) to produce the feature codebook. Finally, a slight variant of the inverted indexing scheme operates on the GeoSOM codebook to produce the closely related images. This enforces a weighting and matching strategy that reduces the search space and time. Simulation analysis of the presented framework is performed on color and medical datasets using the standard evaluation measures. Relative analysis with the state-of-the-art schemes show betterment in terms of Precision-Recall (P-R) and other performance parameters. © 2019, Springer Science+Business Media, LLC, part of Springer Nature.