In this paper, a new color image encryption method based on confusion and diffusion processes has been introduced. A chaotic system based sub-block image swapping has been proposed in order to permutate the positions of the pixels as a confusion method. Moreover, the pixels in sub-block have been spirally shifted to increase the confusion level. Modulo and XOR operations which are based on a chaotic system have been suggested in order to transform the values of the pixels as a diffusion method. To do that, each pixel value is XORed with the next pixel and then modulo operation has been performed for the current and next pixels throughout the image. Double Random Phase Encoding (DRPE) method is also utilized as an optical setup in this study. In addition, an algorithm is suggested in order to reduce the correlations between R, G, B components of the color image. The security analyses performing histogram analysis, correlation coefficient analysis of two adjacent pixels, information entropy analysis, chi-squared test and differential attack have been carried out in order to show that the proposed algorithm scheme is robust against differential, statistical and exhaustive attacks. Moreover, secret key sensitivity, initial condition sensitivity, cropping attack, noise test and plaintext attacks have been done to evaluate the performance of the introduced algorithm. All analysis results have been carried out in MATLAB environment.