Polyurethane (PU) coatings for automotive interior parts are frequently exposed to high temperatures, particularly in extreme environments such as deserts and equators, which cause thermal aging. Although both physical and chemical processes are involved in aging, most previous studies have focused on thermal-induced chemical degradation of PUs. Herein, the physical aging of PU coatings was investigated in the aging temperature range of 80–130 °C based on rheological and spectroscopic analyses. Dynamic mechanical analysis indicated that the thermal aging of PUs is strongly correlated with the glass transition temperature (Tg) of the hard segment (HS). For PU consisting of acrylic and ester polyols and hexamethylene diisocyanate trimer, the thermal aging at 80–130 °C increases the Tg of HS from 91 (fresh) to 118–126 °C (aged films). In particular, the aging was accelerated above the Tg. While the Tg of 130 °C-aged films exceeds 110 °C in 1 d of the aging, that of 80 °C-aged films reached over 110 °C in 7 d of the aging. This result is ascribed to the densification of hard domains: 1.7-fold higher activation energy (391 kJ mol−1) of 130 °C-aged PU than that of fresh PU (234 kJ mol−1) indicates the restricted segmental motion of HS in densified domains. In addition, constructing a two-dimensional gradient map of FTIR can be used to determine whether aging involves physical or chemical processes.