As a p-type semiconductor, layered SnSe has attracted more and more attention because of its great potential application in the field of optoelectronics. However, the strong phonon scattering caused by abundant intrinsic vacancy defects dramatically reduces the performance of carrier transport. It is significant to effectively compensate for the intrinsic defects and reduce the phonon scattering for photodetection materials. In this letter, a novel and simple method is used to reduce the scattering and thus improve the detector performance. The inhibition effect of doping on phonon scattering is systematically studied by experiments and theoretical calculations. The Bi-doped SnSe photodetector exhibits great responsivities of 2.13 A·W-1 (447 nm), 1.35 A·W-1 (655 nm) and 1.91 A·W-1 (980 nm) at 5 V, which are about 2∼3 folds better than those of the undoped device. Furthermore, for the Bi-doped SnSe photodetector, the Ion/Ioff are about 46.7, 20.3 and 30.3 for 447 nm, 655 nm and 980 nm, respectively, which are much higher than those of the SnSe photodetector. The photoluminescence and absorption are performed to confirm the bandgap and defects energy level. Meanwhile, the temperature-dependent current-voltage curves measurement is utilized to prove that the enhancement in response performance is because of the decrease in intensity of phonon scattering, which is attributed to the reduction of scattering centers and the weakening of the effect of vacancy defects on the structural translational asymmetry. All these results evidently illustrate that adjustment in phonon scattering is an effective way to achieve high-performance SnSe photodetectors.