In this paper, a three-dimensional aerodynamics optimization system is built and applied to optimize a rotor blade to balance the conflicts between stall margin, total pressure ratio, adiabatic efficiency, and mass flow rate for the high-loading and transonic-flow fan. A novel flow diagnostic method based on vorticity dynamics theory is utilized to analyze the reasons for the improvement in aerodynamic performance in the optimized transonic fan. In the established aerodynamic optimization method, use the blade profile camber line curvature and its leading edge metal angle as the optimization variables, which are optimized by modifying the coordinates of their control points and introducing a genetic algorithm. Finally, the vorticity dynamics parameters, such as the boundary vorticity flux (BVF), azimuthal vorticity and skin-friction lines are used to diagnose the key flow features in the optimized and baseline fan passage. The results indicate that, by controlling skillfully the blade camber line curvature in the optimization improves the aerodynamic performance of the fan stage, increasing the total pressure ratio by 1.90% while increasing the mass flow rate and adiabatic efficiency by 5.82% and 4.45%, respectively. The formulas from the vorticity dynamics diagnosis method indicate a close link between the aerodynamic performance and vorticity dynamic parameters for the axial fan/compressor passage flow, and that both azimuthal vorticity and boundary vorticity flux have significant influence on fan stage performance. Moreover, the boundary layer separation flow on the rotor blade surface is accompanied by a spike of entropy and static pressure, and their derivative/gradient also suffer drastic changes under the effect of shock waves. Detailed flow information can be obtained about the on-wall with high accuracy based on the vorticity dynamics diagnosis method, which provides researchers with a novel method for the turbomachinery aerodynamic design and analysis in the aero-engine engineering development field.