We demonstrated the fabrication of long-period fiber gratings (LPFGs) in the tapered single-mode fiber by CO2 laser. The phase-matching curves of the LPFGs inscribed in the tapered fiber with different waist diameters have been studied theoretically and experimentally. When the waist diameter of the tapered fiber decreased from 90 μm to 22 μm, the slope of the phase-matching curve changed dramatically. The refractive index sensitivity can be increased by inscribing the LPFGs in the tapered fiber around the dispersion turning point. A maximum refractive index sensitivity of 21702.78 nm/RIU has been achieved in the range of 1.445–1.457 for the LPFG inscribed in the tapered fiber with a waist diameter of 50 μm. This kind of LPFGs inscribing in the tapered fiber may be a good candidate for the development of refractive index based chemical and biological sensors.