In 3G, the server platform of IP multimedia subsystem (IMS) offers multimedia service to the multimedia clients through the exchange of authenticated key agreement (AKA) scheme. Since the IMS service is transmitted over public networks, the server and client of IMS shares a common session-key via the execution of AKA scheme to secure the communication service over a public network. Though there are some identical steps involved in the key authentication Scheme of IMS, the communication efficiencies of IMS server and client are still conciliating. Besides, it is having the issues of insecurity and session compatibility dealt with the real time transport protocol (RTP). To resolve the issues of IMS, this paper proposes three novel schemes: (1) The first scheme is multimedia server–client authentication scheme and its purpose is to mitigate the packet congestion on the networks; (2) The second scheme is traffic on background and its purpose is to analyze the signal congestion, bandwidth consumption and network utilization of the system; and (3) The last one is fault detection index and its purpose is to prevent the SIP vulnerabilities, like bye and invite. In addition, we integrate the proposed and existing authentication schemes, such as EPS-AKA, AP-AKA, UMTS-AKA, X-AKA, S-AKA and so on., in the IMS server and client environments to testify the metrics, namely round trip time, call response time, packet generation, flow analysis, RTP session, network utilization, throughput analysis, bandwidth utilization and flooding attack detection realistically. © 2015, Springer Science+Business Media New York.