Rotating backward extrusion (RBE) as a new severe plastic deformation technique is a continuous process to produce cylindrical tubes with fine grains and superior properties. In this study, the RBE process with an open punch was applied to the AZ80 alloy at 653 K, and the deformation mechanisms and the microstructure evolution were examined by the finite element method (FEM) and thermal simulation experiment. The results showed that the effective strain of the RBE process was higher than that of the conventional backward extrusion (CBE) process, and the strain increased with revolutions increasing. The strain improvement of the RBE process was related to the large cumulative plastic deformation imposed by the continuous rotation of the open punch. Furthermore, the extrusion load was reduced significantly in the RBE process comparing with the CBE process, due to the change of friction stress and stress state of the materials. And the maximum load reduction of the FEM and experiment was 47.33% and 31.6%, respectively. The average grain size of the RBE sample could be reduced by up to 90% in the region A at 30 N compared with the CBE sample. The maximum increase in microhardness of the RBE sample relative to the CBE sample was 23% at 30 N in the region A. Therefore, the grain refinement and mechanical properties of the materials can be substantially improved by the RBE process.