Abstract Traditional optical encryption systems have security risks due to their linearity and usually encounter problems such as the heavy burden of key transmission and storage. This paper proposes a novel security-enhanced optical image authentication and encryption framework that combines diffractive imaging-based encryption with the vector decomposition algorithm (VDA). Chaotic random phase masks (CRPMs) are used to encrypt data for authentication via VDA, and a pair of complementary binary matrix keys are utilized to extract information from the encrypted data to generate ciphertext. During the authentication and decryption processes, a sparse reference image is reconstructed from the ciphertext for verification. If the authentication is successful, image decryption can be executed using a key-assisted phase retrieval algorithm. The employment of nonlinear VDA, an additional layer of authentication, and the use of CRPMs and binary matrix keys enhance security and address key burden concerns. Simulation results demonstrate the feasibility, effectiveness, and security of the scheme.