To analyze the oscillation and evolution characteristics of nanofluid thermocapillary convection instability, a two-phase mixture model was utilized to simulate nanofluid thermocapillary convection in a rectangular cavity and investigate the impact of nanoparticle volume fraction on the oscillatory flow. The results indicate that the two-phase mixture model has a lower critical temperature difference compared to the single-phase model, and the oscillation mode exhibits distinct changes for small nanoparticle volume fractions. At a high Marangoni number, the thermocapillary convection displays periodic oscillatory behavior, with the flow field composed of several dynamic convection vortices. An increase in nanoparticle volume fraction results in a decrease in the critical temperature difference, as well as the amplitude and variation range of temperature oscillation. However, the oscillation period shows an approximate linear increase. Furthermore, there is a non-uniform distribution of nanoparticles at the vicinity of solid wall, especially a significant concentration gradient near the side wall.