The rapid expansion of multimedia communication necessitates the development of advanced and resilient cryptographic systems. As computational power grows, traditional encryption methods become susceptible to brute-force attacks. Cybersecurity threats are in a constant state of flux, requiring the creation of innovative cryptographic systems to outpace these developments. Chaotic systems offer a very good promise in cryptography as they are sensitive to initial conditions and parameters. In this study, the authors propose a novel hypercryptographic system (referred as NHCS) for color images that integrates a new hyperchaotic system with the RSA algorithm. This approach combines the unpredictable behavior of the hyper-chaotic system with the robust encryption capabilities of the RSA algorithm. This combination provides an additional layer of security. The hyper-chaotic systems offer a broader range of parameter values leading to chaos. Moreover, due to their four-dimensional nature, they can be employed to encrypt four-color channels. The NHCS algorithm has a large key space (of size 2704), 1024-bit RSA encrypted for higher security. The key sensitivity of this algorithm is of the order 10−16. The effectiveness of NHCS is validated through numerous test images from the USC-SIPI database and is bolstered by statistical analysis. The two indicators NPCR (∽100% ) and UACI (>33%) ensure that the system is robust against differential attacks. We perform a correlation analysis of the images and observe that correlation coefficients between adjacent pixels of the encrypted images are close to zero. Further, the information entropy values of the encrypted images are close to the ideal value 8. PSNR, MSE, and other statistical measures are found near their respective ideal values. A comparison with the other methods shows that NHCS is very promising.