Unloading-reloading behavior is of crucial importance in multi-pass forming of metal materials, the nonlinear of which creates challenges for the analysis of springback and residual stress of the precision manufacture components. Especially for the hot bulk-formed material at elevated temperature, the thermal-mechanical coupling effect makes the unloading-reloading process more complicated and thus more difficult to model and control, which remains to be solved. In this paper, the unloading-reloading behavior dependent on strain rate and deformation temperature and its model of hot compressive deformed aluminum alloys at elevated temperature were studied from the macroscopic and microscopic perspectives. The effect of deformation temperature, strain rate and strain on the unloading-reloading behavior of hot compressive deformed aluminum alloys were revealed, and corresponding micro-mechanism was elucidated and then discussed. Macroscopically, a strain-, strain rate- and temperature-dependent nonlinear macroscopic model describing unloading–reloading process was established, which can be used to describe the unloading stress relaxation and predict springback during multi-pass hot bulk forming. Microscopically, it is suggested that the relaxations of back stress and forest dislocation hardening stress collectively affect the unloading stress relaxation, due to the decrease of mobile and immobile (forest) dislocation density in unloading, while the dislocation-related behavior before yield, i.e. the dislocations’ bowing-out under pinning, affects the reloading behavior. Based on the micro-mechanism, a physically-based micro-model for describing unloading–reloading was constructed which further is used to clarify the physical mechanism of internal stress relaxation and inelastic strain recovery under multi-pass hot compression deformation. The comparison of the calculation results of the models with those of experiments indicated that the established unloading–reloading macroscopic and microscopic models are able to predict the stress–strain response upon the unloading–reloading loop of multi-pass cyclic hot compression deformation.