Infrared nonlinear optical crystals are widely used in all-solid-state lasers as pivotal devices for laser frequency conversion. BaGa4Se7 (BGSe) crystal has attracted much attention due to its excellent optical properties. However, the presence of point defects in BGSe increases the absorption loss and reduces the laser-induced damage threshold, thus limiting its practical application in high-power mid- and far-infrared lasers. Therefore, it is crucial to reveal the types of defects and their formation mechanisms in BGSe crystal, which is conducive to the design of rational strategies to improve the crystal quality. In this work, the defect activation energies were calculated by photoluminescence (PL) spectroscopy and the existence of three types of defects, GaBa, VSe and VBa were verified in the as grown BGSe crystal. Therefore, a thermal annealing process under BaSe vapor was rationally designed to compensate for the loss of Ba and Se elements and to reduce the defect concentration in BGSe. In addition, the changes of defects in BGSe crystal were determined by X-ray photoelectron spectroscopy (XPS) and photoluminescence (PL) spectroscopy. All the experimental results confirm that the defect concentration in BGSe decreases greatly and the crystal quality is significantly improved after annealing in BaSe vapor.