The double random phase encoding (DRPE) image encryption method has garnered significant attention in color image processing and optical encryption thanks to its R, G, and B parallel encryption. However, DRPE-based color image encryption faces two challenges. Firstly, it disregards the correlation of R, G, and B, compromising the encrypted image’s robustness. Secondly, DRPE schemes relying on Discrete Fourier Transform (DFT) and Discrete Fractional Fourier Transform (DFRFT) are vulnerable to linear attacks, such as Known Plaintext Attack (KPA) and Chosen Plaintext Attack (CPA). Quantum walk is a powerful tool for modern cryptography, offering robust resistance to classical and quantum attacks. Therefore, this study presents an optical color image encryption algorithm that combines two-dimensional quantum walking (TDQW) with 24-bit plane permutation, dubbed OCT. This approach employs pseudo-random numbers generated by TDQW for phase modulation in DRPE and scrambles the encrypted image’s real and imaginary parts using the generalized Arnold transform. The 24-bit plane permutation helps reduce the R, G, and B correlation, while the generalized Arnold transform bolsters DRPE’s resistance to linear attacks. By incorporating TDQW, the key space is significantly expanded. The experimental results validate the effectiveness and security of the proposed method.