A novel optical encryption based on polarization is proposed and applied to a holographic memory system. Original binary data are described as two orthogonal linear polarization states. These input polarization states can be modulated by use of two polarization-modulation masks located at the input and the Fourier planes. Each modulation mask can convert an input polarization state into a random polarization state. Once encrypted, the polarization state is recorded as a hologram. For the decryption, the hologram can generate a vector phase-conjugate beam. When the same polarization-modulation masks are used, the vector phase-conjugate readout can cancel the polarization modulation at each mask, and the original polarization state can be recovered. The encryption of the proposed method is evaluated numerically. We also present experimental results by demonstrating holographic recording in a bacteriorhodopsin film.