TeO2-ZnF2-BaF2-Ta2O5 fluorotellurite glasses have excellent optical properties and stability which makes them promising candidates for mid-infrared (mid-IR) fiber applications. In this work, the properties and structure of fluorotellurite glasses with gradient change of BaF2 content were compared. Under the irradiation of femtosecond (fs) laser, the potential relationship between fs laser-induced damage characteristics, glass composition and structural property were systematically studied. The fs laser-induced damage threshold (FLIDT) of investigated fluorotellurite glass samples increased from 892.45 mJ/cm2 by more than one-third to 1223.84 mJ/cm2 and the attenuation coefficient of multi-pulse FLIDT increases from 7.69 to 25.29 with the increase of [TeO3] content from 0.312 to 0.565 induced by increasing BaF2 content. Further, structural transformation from TeO4 units to TeO3 units in the glass network triggered by fs laser irradiation is confirmed by Raman spectra measurements. This study provides theoretical guidance for tailoring the fs laser damage threshold through glass microstructural modulation by composition optimization, which is of great importance to improve the output power in mid-IR supercontinuum (SC) spectra and rare earth (RE) doped fiber laser systems.