Vortex beams provide an innovative multiplexing mechanism for the field of optical communication due to its specific orthogonality. The precise and efficient detection of orbital angular momentum (OAM) modes carried by vortex beams is an essential technology for OAM multiplexing communication systems. An OAM mode classifier based on convolutional neural network (CNN) is proposed for the mode detection of distorted high-order vortex beams (radial index p > 0). The effect of different atmospheric turbulence (AT) intensities, radial indexes, and propagation distances on the detection accuracy of the OAM modes are investigated. The ability of the classifier models for generalization after training on both specific and mixed AT dataset is analyzed. The results show that the higher-order vortex beams have a greater detection accuracy in AT environments. For the high-order vortex beam carrying OAM modes, the CNN-OAM mode classifier has an average detection accuracy of 97.3% and can detect up to 30 OAM modes under any AT intensities ranging from 1 × 10−16 m−2/3 to 1 × 10−14 m−2/3 at a propagation distance of 1000 m. The great accuracy and extensive detection range of the CNN-OAM mode classifier proposed in this paper can offer a novel approach to the recognition and detection of high-order vortex beams.