A novel two-dimensional model for the thermocapillary instability of a thin liquid film was developed and validated in the paper. The model incorporates a novel solution for the unperturbed velocity components and the thin film thickness (based on the boundary layer approach and balance equation) subject to evaporation under the boundary conditions of either mechanical interaction between the liquid and vapor phases or stationary vapor above the interphase boundary. A novel model for thermocapillary instability in a thin film was developed in frames of the linear perturbation method, i.e. modified Orr–Sommerfeld equation, taking into account the surface and London–van der Waals forces. The critical Reynolds number was computed by considering the two-dimensional disturbances, which according to the Squire’s theorem are more dangerous than the three-dimensional disturbances. For constant surface tension at the interphase interface, the unperturbed velocity profile is parabolic, and maximum increases, while the critical Reynolds number decreases with the decreasing capillary number. If the surface tension at the interphase interface depends on the temperature, the maximum of the undisturbed velocity profile increases with the decreasing capillary number and increasing modified Marangoni number, which entails more rapid decrease in the critical Reynolds number. It was also shown that the flow is destabilized by the increase in the temperature difference between the wall and the vapor, by the decrease in the absolute pressure and by the increase in the thermal conductivity. To confirm existence of different flow regimes in the nanofluid film, numerical simulations of capillary flow was performed, which exhibit qualitative consistence with the stability theory.