Exclusive-OR (XOR) operation is a popular technique in optical cryptography. The linear and inverse properties of XOR operation provide a simple way of encryption but in some cases fail to provide strong security. In XOR-based image encryption, each pixel is individually encrypted, therefore secret information can be traced with partial knowledge of the key. In this paper, we describe a simple and practical method to improve the strength and effectiveness of XOR-based encryption. A higher level of security is achieved by implementing a phase retrieval algorithm with XOR operation. Another advantage of the proposed scheme is that the optical XOR operation is demonstrated using incoherent light sources, which make the system compact, speckle free, and cost effective. Also, to overcome the problem of quality degradation while decrypting, quick response code-based ciphertext has been implemented. Both simulation and experimental results are demonstrated as a proof-of-concept.