Recently, the color image encryption algorithm based on chaos theory has become the focus of current research. When encrypting color images, the common practice is to treat color images as different gray components and process them severally, which results in more redundancy and low efficiency. The security of chaotic cryptosystems also depends on the performance of the chaotic systems adopted. The structures of many current chaotic systems are fixed, making their behaviors highly predictable. Additionally, the range of chaotic region parameters is limited and discontinuous. To solve the above-mentioned problems, a new color image encryption algorithm (CIEA) using fractal and chaos theory is presented, which fully considers the inherent connection among the RGB components of color images. First, we propose a variable-structure discrete hyperchaotic system (VSDHS) to solve the dilemma encountered by existing chaotic systems. The excellent dynamic properties of VSDHS are verified by rigorous mathematical proof and simulation performance analyses. Then, VSDHS-CIEA is designed using fractal theory and VSDHS. The algorithm makes full use of the inherent connection among the different components of color images and performs cross-plane confusion. Simulations and performance analyses prove that VSDHS-CIEA has higher security and better performance than some representative image encryption algorithms.