Supercritical carbon dioxide (S-CO2) Brayton cycle system is a promising closed-loop energy conversion system frequently mentioned in the automotive and power generation field in recent years. To develop a suitable design methodology for S-CO2 turbines with better performance, an understanding of the vortex flow patterns and associated aerodynamic loss inside a S-CO2 turbine is essential. In this paper, a hundred-kilowatt level S-CO2 axial turbine is designed and investigated using a three-dimensional transient viscous flow simulation. The NIST Span and Wagner equation of state model that considers the real gas effects is utilized to estimate the thermodynamic properties of the supercritical fluid. The numerical methods are experimentally validated. The results indicates that the aspect ratio and tip-to-hub ratio are different in the S-CO2 turbine from that in the gas turbine, and the vortex flow patterns are influenced notably by these geometrical parameters. Both the vortex structure and moving tracks of passage vortices are changed as a result of large centrifugal force. An interaction between tip leakage vortex and hub passage vortex is observed in the impeller passage and its formation and development mechanism are revealed. To further explore the aerodynamic loss mechanism caused by vortex interaction, the energy loss in the impeller passage is analyzed with the enstrophy dissipation method, which can not only accurately calculate the energy loss but also estimate how the vortical motions occur. It is found that the enstrophy and energy loss can be effectively reduced by vortex interaction between tip leakage vortex and hub passage vortex. The results in this study would increase the knowledge of vortex flow patterns in S-CO2 turbine and the proposed enstrophy production method can be used intuitively to provide a reference for flow vortical motion study in turbines.