Wireless mobile communication has become more familiar and popular among other communication system for transmitting data to enable the user to communicate with other third parties at any place at any time. Due to third party interception, transmission of data, connection establishment, connection release, billing processes are in insecure environments. So many researchers had put their effort to develop an innovative authentication scheme for GSM network. The existing authentication scheme provides security with high communication and computational overhead using a symmetric key cryptography. The proposed scheme mainly focuses on providing security to global system for mobile communication system using hyperelliptic curve cryptography and DNA computing technique for mutual authentication, secret key generation and, data encryption and decryption algorithm. Finally, a security analysis shows that authentication algorithm is more secure and efficient. Copyright © 2015 Inderscience Enterprises Ltd.