As the low complexity of low-dimensional chaotic system and the slow speed of image encryption, this paper proposes a fast encryption algorithm of color image based on four-dimensional chaotic system. Firstly, we propose a new method of designing four-dimensional chaotic system based on the classical equations of three-dimensional chaotic system, to increase the complexity and key space of the encryption algorithm. Secondly, according to the nature of color images’ pixels channel, we design a new pseudo-random sequence generator and reuse the random sequence, to improve the speed of image encryption. Finally, the methods of row-major and column-major are used to diffuse the original image and the cat map with parameter is used to scramble the image pixels, respectively, to achieve the effect of encryption. The results of simulation and security analysis show that the proposed encryption algorithm is of good performance on security, robustness and high encryption speed.