Improving permeability is the crucial mechanism for further enhancing oil and gas recovery of ultra-low permeability reservoirs. To investigate the temperature sensitivity and stress sensitivity of permeability under an actual reservoir condition, heating treatment and real-time measurement of pulse-decay gas permeability (PDP) of cores collected from three different ultra-low permeability reservoirs were conducted in our self-developed high-temperature pseudo-triaxial core holder. To clarify the intrinsic relationship between the microstructure development and porosity-permeability enhancement, stereo light microscope (SLM) observation, computerized tomography (CT) scan and nuclear magnetic resonance (NMR) test were used to quantitatively characterize the dynamic variations in thermal cracks, nanopores and porosity. Based on the lithology analysis and thermal analysis, the primary mechanisms of nanopore evolution and thermal cracking for different ultra-low permeability rocks were analyzed in detail. The experimental results show that the permeability of Chang 8 ultra-low permeability sandstone and Longmaxi shale cores only increased by about one order of magnitude above the threshold temperature of 500 °C, while the permeability of Jimsar shale cores increased significantly by two orders of magnitude above the threshold temperature of 300 °C. Above their threshold temperatures, the permeability showed stronger stress sensitivity. The SLM observation and CT scan revealed that thermal cracks propagated rapidly above the threshold temperatures, which resulted in a substantial increase in temperature sensitivity and stress sensitivity of permeability. NMR test shows that the porosity of ultra-low permeability cores performed an increasing trend as the temperature rose. However, the nanopore structure in Chang 8 ultra-low permeability sandstone monotonously developed toward larger micron-sized pores, while the Longmaxi shale and Jimsar shale eventually re-evolved into smaller nanopores. We found that kerogen pyrolysis was the most critical mechanism leading to the microstructure, porosity and permeability enhancement in organic-rich shale. For organic-poor ultra-low permeability sandstone and shale, the quartz phase transition from α to β was considered as the key reaction for the thermal cracking and permeability improvement. By comparison, heating treatment has a higher thermal stimulation potential in shale cores from medium-to-low maturity shale oil reservoirs.