This study proposes a color image encryption method combining Fresnel diffraction, multi-phase retrieval, and hyperchaotic mapping in two stages. First, the color image is converted into an amplitude-type computer-generated hologram (CGH) that overlaps in the frequency domain and separates in the spatial domain. The Gerchberg-Saxton (GS) multi-phase retrieval algorithm encrypts the CGH into a meaningless ciphertext in the second stage, scrambling/diffusing it further using four random sequences generated from the Lorenz hyperchaotic system. The experiment on public color images shows that the decrypted color images are highly consistent with plaintext images. This method's key has a large space (approximately 10∧301) and a small volume. Slight changes to the plaintext completely alter the ciphertext, and its pixel values are statistically independent, thus securing the proposed method against chosen-plaintext attacks and chosen-ciphertext attacks.