Cryptography is one of the most important security mechanisms for the transmission of digital media on the Internet. Most proposed cryptographic image methods based on chaotic maps are dependent upon image sizes and most of them worked on square images. In this paper for tackling this problem, a scale-invariant color image encryption method in three-dimensional space is presented. At first, the two-dimensional color image is converted into three-dimensional space, in this case, the red, green, and blue color spectrums are divided into a set of gray-level square sub-images. Then, to have confusion and diffusion properties, the 3D substitution and 3D permutation operation are performed on the sub-images. In substitution operations, the pixel values of the sub-images are changed with the help of XOR and circular shift operators with appropriate keys. In permutation operation, the position of the pixels is changed using modular three-dimensional chaos mappings. To have scale-invariant three-dimensional permutation, the sub-images are divided into one or more windows with equal size, and then 3D modular chaotic map operations are performed on each window with separate keys. Depending on the number of sub-images, there may be two last windows that have overlapping. To increase the speed of color image encryption, the steps of a 3D modular chaotic map on the windows can be implemented in parallel. The proposed approach relative to the similar color image encryption algorithms increases the key space and improves standard parameters, such as entropy, sensitivity, adjacent pixel correlations, and histogram uniformity.