The key distribution protocol based on ghost imaging has attracted wide attention due to its strong robustness, fast key generation rate and security without the influence of distance. However, the lack of identity authentication leads to potential security risks, while there are still major challenges in how to minimize storage costs, the communication overhead and computational overhead involved in the process of key distribution. In this paper, we propose a key distribution protocol under the ghost imaging framework, which could be achieved mutual authentication between users and key distribution centers on public networks. In the process of information interaction, only the intensity sequences encrypted by ghost imaging need to be transmitted, which reduces the amount of data and ensures the confidentiality and imperceptibility of information. Meanwhile, the illumination patterns used for decryption are only reserved in the key distribution center without transmission, which not only reduces the security risks caused by the theft of the illumination patterns, but also reduces the user’s demand for storage and computing power. Simulation and experiments show the effectiveness of this protocol, which can effectively resist exhaustive attack, tampering attack and noise attack. This protocol provides a new idea for key distribution and a new protection method for optical secure communication.