Thermal error of the spindle is one of the primary contributors of the inaccuracy of the precision machine tools. In order to study the thermal characteristics of the spindle, finite element analysis has been widely used. However, the accuracy of numerical simulation is highly dependent on the boundary conditions especially the coefficients of convection heat transfer. In this paper, the inverse heat conduction theory is introduced and used for optimizing the coefficients of convection heat transfer. Then, the temperature field and thermal error of the spindle are simulated in ANSYS. Based on the simulation results, a new method called mean impact value is proposed to select the thermal key points in the spindle system. Finally, the verification experiments are conducted on a precision horizontal machining center. By comparing the simulation results with the experimental data, the correctness and effectiveness of the heat convection coefficient optimization are verified. In addition, the results of thermal error modeling based on multiple variables including the temperatures of the thermal key points show that the result of the thermal key point selection is satisfying.