The mixing induced by the Rayleigh–Taylor (RT) instability occurs widely in various natural phenomena and engineering applications, such as inertial confinement fusion. The mixing transition in the RT mixing process is the key process affecting the mixing evolution. At present, research in RT mixing transition mainly involves mixing transition criteria based on global quantities, statistical analysis of mixedness parameters and kinetic energy, and so on. A few studies have paid attention to the evolution of vorticity and its intensity, enstrophy, during mixing transition process. However, previous studies have inferred that vorticity and enstrophy play important roles in mixing transition. In this paper, implicit large-eddy simulation for RT mixing is carried out to analyze the evolution of vorticity and enstrophy in mixing transition. First, the vortical motions throughout the whole mixing process are investigated by comparing the contours of mass fraction and vorticity. Then, for revealing the mechanism of vortical motions in transition stage, the vortex structures are extracted and the relationship between vortex structures and enstrophy in mixing transition is investigated. Finally, in order to quantify the vortical motions in the mixing transition, the probability density function (PDF) of enstrophy is introduced and analyzed. The main conclusions are as follows: (1) The evolution of vortical motions is closely related to the RT mixing transition process. Enstrophy can reflect the vortical motions in the mixing transition process. When the growth rate of averaged enstrophy reaches its maximum value, the transition occurs; (2) the PDFs of enstrophy can quantify the evolution of vortex structures during mixing transition and characterize the mixing transition process. The mixing transition begins when the PDF of enstrophy appears double peaks. The process of PDF right peak movement corresponds to the transition process, and the transition ends when the position of the right peak is no longer moving. Since the enstrophy studied in this paper is a local field quantity, the above results are expected to be used to construct local mixing transition criterion.