In-plane easily magnetized scandium substituted thulium iron garnet films(Tm3Fe5-xScxO12) with soft magnetic properties have recently attracted attention for fluxgate magnetometer applications. In this paper, thin Tm3Fe5-xScxO12 films were prepared on gadolinium gallium garnet substrates by liquid phase epitaxial(LPE) method. Microstructural properties, composition, magnetic properties and the in-plane anisotropy of the films were discussed. We found that the films turned from tensile stress to compressive stress as the Sc3+ substitution decreased, and the in-plane anisotropy was mainly caused by stress reducing as Sc3+ declined. By analyzing the in-plane magnetic hysteresis loops, the magnetic performance was the best when x = 0.75. Tm3Fe4.25Sc0.75O12 film had the lowest in-plane coercive field(0.006 Oe) and the highest initial magnetic permeability(378), which was more beneficial to realize fluxgate magnetometers.