Despite many online messaging services, short message service (SMS) is still used because of its simplicity. However, such services can be easily cracked, making SMS more vulnerable to send confidential information. Therefore, appropriate cryptographic algorithms must be applied to ensure security. One of the most common cryptographic techniques used for encrypting and decrypting messages is the Vigenere cipher. However, Vigenere cipher can be cracked easily because the previous approach involved 26 × 10 matrixes, and the key comprised of alphabets. If the key length is small, then it becomes easier to crack the plain text by permutation. So here in this work, we have proposed a modified Vigenere cipher technique to enhance its security standards. Existing Vigenere cipher attack is not more secured, and thus we developed a method that improves the security in Vigenere cipher using Rivest-Shamir-Adleman (RSA). RSA is an asymmetric algorithm that has public and private keys. The advantage of using this algorithm is that finding factors for the large composite numbers is difficult. This method is secured but time consuming. In this proposed method, we combined RSA with Vigenere cipher so that the proposed algorithm is secured and consumes less time than RSA algorithm. This approach is widely used to encrypt SMS. © 2020, Operations Research Society of China, Periodicals Agency of Shanghai University, Science Press, and Springer-Verlag GmbH Germany, part of Springer Nature.