Steam flooding and chemical flooding have been widely proven to be the most promising methods to facilitate the development of heavy oil reservoirs. However, the remaining oil reserves after steam flooding or chemical flooding still account for the vast majority, and it is challenging and necessary to study the occurrence state and formation mechanism of the remaining oil. In this paper, an anionic Gemini surfactant and an amphoteric surfactant are synthesized and combined with a hydrophobic modified polyacrylamide-based polymer to form the surfactant-polymer (SP) flooding system. Then, the development effect of the system is investigated by the visualization experiment, and high-resolution images of the remaining oil are obtained. Finally, Mask R-CNN, an intelligent image recognition technology based on deep learning, is introduced to more accurately and quickly study the microscopic remaining oil occurrence state and macroscopic remaining oil distribution. The results show that the SP system has excellent synergistic oil displacement effect and can increase the oil recovery by 47.2% in visualization experiments. Remaining oil is classified into four categories: flake oil, columnar oil, droplet oil and membranous oil. The algorithm achieves an accuracy of 93.83% on the identification task and an intersection over union (IOU) of 91.5% for the instance segmentation. The oil remaining after steam and chemical flooding is mainly flake oil and dispersed column oil, droplet oil and film oil, respectively. This algorithm lays the foundation for the study of the microscopic occurrence state and macroscopic statistical distribution of remaining oil under different displacement methods.