Temperature’s influence on the physical and mechanical properties of rocks is a crucial concern for the rational design of deep rock engineering structures and the assurance of their long-term stability. To systematically comprehend the impact of the evolution of mineral composition and micro characteristics on the physical and mechanical behavior of thermally damaged granite, we observed the microscopic structural defects inside the rocks with a polarizing microscope and revealed the thermal damage mechanism of granite from a microscopic perspective by combining ultrasound detection and XRD phase characteristic analysis. The results show that the physical properties of the specimens changed significantly at three characteristic temperature points: 400 °C, 800 °C, and 1000 °C. Under high temperature conditions, the diffraction intensity of all minerals in granite, except for quartz, generally decreased, and stable minerals decomposed. Albite and potash feldspar decomposed to form anorthoclase, thereby reducing the structural stability of the rock material. In addition, the peak width of various minerals decreased to varying degrees with increasing temperature. The increase in mineral volume further damaged the internal structure of the rock material while promoting the transformation from grain boundary to intergranular cracks and from intragranular cracks to transgranular cracks, ultimately forming a interconnected crack network. Thermal damage significantly reduced the longitudinal wave velocity, uniaxial compressive strength, and elastic modulus of the specimens, while the stress–strain curve relationship indicated that the specimens underwent two opposite processes of transformation from brittleness to ductility and then from ductility to brittleness. The thermal damage threshold of granite in this study was 600 °C.