Due to the large number of frames and low video resolution, tennis match videos cannot accurately identify and extract effective data, which reduces the level of fine analysis of tennis matches. In order to solve the problem of poor detection effect of small targets in tennis video, an automatic detection method of small targets in tennis video based on deep learning is proposed. Non-maximum suppression algorithm is used to determine the position of the target between different adjacent video image sequences, and SVM classifier is used to train a large number of target behaviors. According to the hierarchical structure of dataset annotation, the hierarchical structure of tennis video for deep learning is optimized. The reconstruction algorithm is used to enhance the video image in the input VOC data set and improve the fine segmentation effect of the image. The difference video image is binarized to complete the automatic detection of small targets in tennis video. The experimental results show that the proposed method has high integrity of tennis video information collection, high recognition accuracy and short detection time.