Effects of substrate temperature on the thermal stability of Al-doped ZnO (AZO) films have been studied. Degradation of electrical properties of AZO films by annealing under flowing N2 gas depends on their crystallinity controlled by the substrate temperature. A thin AlOx capping layer was employed to passivate the thermal degradation of the AZO layer. A strong correlation between Zn desorption and reduction in carrier concentration was observed. Thermal desorption of Zn was prevented by the AlOx layer, retaining carrier concentration. With the AlOx capping layer, the reduction in Hall mobility was prevented in samples with good c-axis orientation, while the reduction in Hall mobility was still observed in poor c-axis oriented films. However, the reduction was smaller than that in bare AZO films. The dependence of Hall mobility evolution on the substrate temperature, and therefore, on crystallinity, strongly suggests the impact of grain boundary scattering on thermal degradation. An increase in optical mobility, which was evaluated from optical spectra using the Drude model, with annealing temperatures, supports the conclusion that an increase in grain boundary scattering by annealing caused the degradation of Hall mobility. The increase in grain boundary scattering induced by Zn desorption was prevented by the capping layer, while contributions of domain alignment and other segregation of defects to the grain boundary scattering, which depend on the substrate temperature retained, leading to different evolutions of Hall mobility.