A low-noise and fast optical encryption method for three-dimensional (3-D) information using the double-phase method is proposed. First, the 3-D information is encoded into a phase-only hologram (POH) by the angular-spectrum diffraction and the double-phase method. Second, the chaotic random phase mask (CRPM) is generated by the hybrid logical map and the iterative chaotic map with infinite collapses map to modulate the POH and obtain the ciphertext. As the secret key, the CRPM cannot only improve the secret key space of the scheme but also achieve the purpose of scrambling and hiding 3-D information. Third, the background noise of the decrypted image is successfully reduced by the cross-shaped filter, which is designed for the first time based on the POH spectrum distribution. The proposed scheme has successfully improved the encryption speed of the 3-D information and the quality of the reconstructed images. Numerical simulation and optical results show the effectiveness and feasibility of the proposed encryption scheme.