Steganography deal with hiding information science, which offers an ultimate security in defence, profitable usages, thus sending the imperceptible information, will not be bare or distinguished by others. The aim of this paper is to propose a novel steganographic method in JPEG images to highly enrich a data security by RSA algorithm and attains higher payload by modified quantisation table. The goals of this paper are to be recognised through: 1) modify the quantisation table of the JPEG-JSTEG tool, hiding secret message with its middle frequency to offer great embedding capacity; 2) for challenge, secure RSA algorithm is used to prevent data from extraction. A broad experimental evaluation compares the performance of our proposed work with existing JSTEG was conducted. This algorithm resulted in greater PSNR values and steganogram histogram is more similar. Experimental results reported that the proposed system is a state-of-the model, contributing abundant payload and beating the statistical revealing. Besides, our method has better in all the parameters than JPEG-JSTEG method. Copyright © 2019 Inderscience Enterprises Ltd.