Enhancing security in instant messaging systems with a hybrid SM2, SM3, and SM4 encryption framework.

With the rapid integration of instant messaging systems (IMS) into critical domains such as finance, public services, and enterprise operations, ensuring the confidentiality, integrity, and availability of communication data has become a pressing concern. Existing IMS security solutions commonly employ traditional public-key cryptography, centralized authentication servers, or single-layer encryption, each of which is susceptible to single-point failures and provides only limited resistance against sophisticated attacks. This study addresses the research gap regarding the complementary advantages of SM2, SM3, and SM4 algorithms, as well as hybrid collaborative security schemes in IMS security. This paper presents a hybrid encryption security framework that combines the SM2, SM3, and SM4 algorithms to address emerging threats in IMS. The proposed framework adopts a decentralized architecture with certificateless authentication and performs all encryption and decryption operations on the client side, eliminating reliance on centralized servers and mitigating single-point failure risks. It further enforces an encrypt-before-store policy to enhance data security at the storage layer. The framework integrates SM2 for key exchange and authentication, SM4 for message encryption, and SM3 for integrity verification, forming a multi-layer defense mechanism capable of countering Man-in-the-Middle (MITM) attacks, credential theft, database intrusions, and other vulnerabilities. Experimental evaluations demonstrate the system's strong security performance and communication efficiency: SM2 achieves up to 642 times faster key generation and 2.2 times faster decryption compared to RSA-3072; SM3 improves hashing performance by up to 11.5% over SHA-256; and SM4 delivers up to 22% higher encryption efficiency than AES-256 for small data blocks. These results verify the proposed framework's practicality and performance advantages in lightweight, real-time IMS applications.