In this paper, we investigate the propagation properties of a vortex Hermite-cosh-Gaussian beam (vHChGB) passing through a fractional Fourier transform (FRFT) optical system. The analytical expression of a vHChGB in FRFT plane is derived based on the Huygens–Fresnel diffraction integral. From the obtained expression, the evolution of the intensity and phase distributions of output beam are illustrated numerically as a function of the order of the FRFT and under different beam parameters conditions. The results show that a vHChGB in FRFT plane evolves gradually and periodically versus the order of the FRFT optical system. The beam shape depends strongly on the fractional order, and it is affected by the beam order, the decentered parameter and the topological vortex charge. The results obtained may be beneficial to applications in optical trapping, micro-particle manipulation, and beam shaping.