In this paper, security of a nonlinear optical cryptosystem based on double random phase encoding (DRPE) is analyzed. In the cryptosystem, a time-varying nonlinear term is combined with the classical DRPE technique to enhance security. The advantages of the nonlinear term are its easy distribution and no effect on the decrypted image. Also, it is confirmed that the method is resistant to conventional plaintext attacks such as known-plaintext attack, chosen-plaintext attack, and ciphertext only attack. However, our security analysis shows that the method is vulnerable to the modified chosen-plaintext, and known-plaintext attacks. The original image can be recovered successfully using the proposed attack algorithms. Numerical simulation results are presented to validate the proposed attack methods and test effectiveness of our analysis.