We propose a multi-depth three-dimensional (3D) image cryptosystem by employing the phase retrieval algorithm in the Fresnel and fractional Fourier (Fr-FrF) domains. Encryption was realized by applying the phase retrieval algorithm based on the double-random-phase-encoding architecture in which two encryption keys will be incessantly updated in each iteration loop. The phase-only functions (POFs) are generated in two cascaded Fr-FrF transforms (Fr-FrFT), serving as decryption keys to efficiently reduce the speckle noise and crosstalk between encrypted 3D image depths. The use of Fr-FrFT position parameters and fractional order as decryption keys further extended the key space, enhancing the cryptosystem's security level. Numerical simulations demonstrated the feasibility and robustness of our proposed scheme.