In this paper, Ca6BaP4O17:Sm3+ and Li+ co-doped Ca6BaP4O17:Sm3+ phosphors were synthesized in air and argon atmospheres using a solid-state reaction method. The phosphor morphologies and crystal structure were studied using scanning electron microscopy and X-ray diffraction, respectively. The emission and absorption characteristics were investigated using photoluminescence emission spectroscopy and diffuse reflectance spectroscopy. The surface states and composition of phosphor were investigated using X-ray photoelectron spectroscopy. The emission integrated intensities of the phosphors sintered in an argon atmosphere increased 3.5 fold than the ones sintered in air atmosphere, with Li+ ions becoming embedded in the lattice of the Ca6BaP4O17:Sm3+ phosphor. This occurs because there are fewer defect/oxygen vacancies and less of the secondary phase forms, leading to better Sm3+ emission. The results suggest that sintering a mixture of the raw materials of a phosphor in an argon atmosphere is a good approach for synthesizing Ca6BaP4O17:Sm3+ phosphor powders. The color purity and CIE values of an optimized phosphor sample sintered in an argon atmosphere with an Li+ ion compensator were calculated to be ~ 99.6% and (0.612,0.386) in the orange–red region under 405-nm excitation, respectively. Moreover, the solid solubility of Sm3+ ions in the Ca6BaP4O17 host can be enhanced by using an argon atmosphere in the synthesis process.