The quenching effect is a phenomenon in which the scintillation efficiency decreases when energetic particles with high linear energy transfer, such as high-energy ions, deposit energy within the scintillator. From the viewpoint of discriminating between neutrons and gamma rays in the neutron scintillator, evaluating the quenching effect is crucial because the high-energy ions produced by neutron reactions are used to detect neutrons. Using the user-defined subroutine in the Monte Carlo simulation code PHITS, we demonstrated the calculation of the pulse height spectra obtained from Li-containing scintillators, in which the quenching effect is considered based on the Birks’ formula. By comparing the experimental pulse height spectra with simulation results, which consider the experimental broadening, for the neutron peak and Compton edge formed by mono-energetic gamma rays, we determined the quenching coefficient in the Birks’ formula for Li glass, Ce:LiCaAlF6 and Eu:LiCaAlF6 scintillators.