As a promising candidate material for holographic 3D displays, lithium niobate (LN) is limited by its low photorefractive (PR) response. Recently, it has been reported that bismuth dopants significantly improve the PR properties of LN crystals. However, the mechanism of photorefraction enhancement and whether the performance can be further optimized are not clear. In this paper, we demonstrate that Zn2+, In3+, and Zr4+ co-dopants can enhance the photorefraction of LiNbO3:Bi crystals. In particular, the PR sensitivity of LN:Bi,Zn8.0 crystal reaches 11.7 cm/J at 488 nm, with a diffraction efficiency of 16.67% and a response time of 290 ms. We propose that Bi ions occupy Nb sites, forming BiNb2−/BiNb0 in LN:Bi,Zn crystals, while still occupying Li sites, forming BiLi2+/BiLi4+ in LN:Bi,Zr crystals, when the Zn/Zr concentration exceeds the doping threshold. These occupying models are confirmed by the atomic resolution of scanning transmission electron microscopy. Additionally, we find that the lone-pair electron effect of Bi is pronounced when Bi3+ ions occupy Nb sites, forming the most highly efficient PR centers, which induce an outstanding PR response in LN:Bi,Zn8.0 crystal. Our results clarify the occupation of bismuth ions in Zn, In, or Zr co-doped LiNbO3:Bi and confirm that the PR performance can be further improved by site control.