Different types of wear particles in lubricating oil are crucial information sources for describing wear properties of rotating machinery components. Solving the problem of particle classification using artificial intelligence models depends on sufficient wear samples, though the samples are difficult to obtain, especially in lubricants with real-time online monitoring. First, a high precision particle analyzer for online acquisition of multi-particle images in the circulating oil system based on the optical direct imaging method is developed. Then, a novel hybrid convolutional neural network (CNN) model based on the visual geometry group (VGG) network feature and residual block, called VRCNN, is proposed to classify nine types of metallic and non-metallic particles, including normal, sliding, fatigue, cutting, sphere, ceramic, bubble, plastic, and fiber, using their images. Finally, a dataset produced by a pin disc friction tribometer and bearing accelerated fatigue tester is used to demonstrate the approach. The proposed VRCNN model is also compared with the typical classifiers, including AlexNet, VGG16, RestNet50, SqueezeNet, MobileNetV2, and ShuffleNetV2 models in terms of classification accuracy, standard deviation of accuracy, F1-score, and detection time. The results show that the VRCNN model has a classification accuracy of more than 95%, and its response time is superior to that of the other networks. Furthermore, the operational process of the VRCNN classifier is analyzed visually using the Grad-CAM map. This work is beneficial for improving any type of rotating machinery’s wear mechanism analysis.