In this paper the task of recognizing the speaker with the constraint of limited data using the learning Vector Quantization (LVQ) is demonstrated. The present day speaker recognition systems use the Gaussian-Mixture-Model- Universal-Background-Model (GMM-UBM) as a modelling technique when training data is sparse. Since the performance of the LVQ depends on the tuning of the parameters, we have found that the fine tuned LVQ yields better results than the GMM-UBM. Further, the linearly combined LVQ and GMM-UBM modelling techniques significantly improved the performance over the individual modelling techniques. Copyright © 2009 by IICAI.