In this paper, we propose an optical high-fidelity decryption technology for Visual Cryptography (VC) based on incoherent optical polarization exclusive NOR (XNOR) operation. The plaintext image is divided into two binary ciphertexts by the Random-Grid-based Visual Secret Sharing (RGVSS) algorithm. These ciphertexts are printed onto the polarizing films with orthogonal polarization. The P- and S-polarizing pixels in the polarizing films represent the “black” and “white” pixels of the ciphertexts, respectively. Due to the optical transmission property of polarizing film, the pixel in the decrypted image appears bright only when the stacked pixels for the two ciphertexts have the same polarizing orientation. High-fidelity image decryption is then achieved. The proposed technology has been successfully demonstrated through simulation and experimentation. Thanks to its advantages of no pixel expansion, incoherent illumination, being free of computer assistance, and flexibility of ciphertext carriers, this method provides a broad prospect for convenient, large-format, high-fidelity information recovery applications for VC.