Mid-infrared (MIR) lasers based on nonlinear frequency conversion by nonlinear optical (NLO) crystals are urgently desired for various military and civilian applications. LiInSe2 (LISe) crystal has aroused intense attention due to its remarkable optical properties. However, the presence of point defects in LISe increases absorption loss and degrades the laser damage threshold (LDT), which limits the practical application in high-power MIR lasers. Therefore, it is essential to reveal the defect types and their formation mechanism in LISe, which is beneficial to design a rational strategy to improve the crystal quality. Here, large-sized LISe crystals with a diameter of 26 mm were successfully grown by the optimized Bridgman method using oriented seed crystal. Furthermore, three types of defects, VSe, VLi, and InLi in the as-grown LISe crystals were verified by photoluminescence (PL) spectra. Therefore, we designed the thermal annealing process under Li2Se vapor to compensate for the losses of Li and Se elements and reduce the defect concentration in LISe. Moreover, the evolution of defects in LISe crystals was determined by X-ray photoelectron spectroscopy (XPS), PL and ultrafast transient absorption spectra (TAS). All the experimental results confirm that the defect concentration in LISe greatly decreases and the crystal quality was obviously improved after annealing under Li2Se vapor. More importantly, this work provides a general strategy to reveal the defects in crystals and to improve the crystal quality of MIR chalcogenides for practical applications.