Abstract Compared with ordinary chaotic systems, memristor-based chaotic systems have more complex dynamic behaviors and are more suitable for image encryption algorithms. In this paper, a four-dimensional chaotic system is constructed by introducing a cubic nonlinear memristor into a three-dimensional chaotic system. Firstly, the dynamic characteristics of the constructed memristor-based chaotic system are analyzed in detail, and the simulation results show that the system has different attractors with different topological structures at different simulation times. Within a fixed simulation time, the system has 15 attractors with different topological structures under different parameter values, and there is a phenomenon of multiple stability in the system, indicating high complexity. Based on the above discoveries, a color image encryption algorithm including scrambling and diffusion is designed. Experimental results show that this algorithm can perfectly hide the information of the plaintext image, and the decrypted image is consistent with the plaintext image. Finally, the security of the algorithm is analyzed by using key space and so on. The analysis results indicate that the encryption algorithm designed in this paper can effectively resist external attacks and has high security.