Video event detection and annotation work is an important content of video analysis and the basis of video content retrieval. Basketball is one of the most popular types of sports. Event detection and labeling of basketball videos can help viewers quickly locate events of interest and meet retrieval needs. This paper studies the application of anisotropic diffusion in video image smoothing, denoising, and enhancement. An improved form of anisotropic diffusion that can be used for video image enhancement is analyzed. This paper studies the anisotropic diffusion method for coherent speckle noise removal and proposes a video image denoising method that combines anisotropic diffusion and stationary wavelet transform. This paper proposes an anisotropic diffusion method based on visual characteristics, which adds a factor of video image detail while smoothing, and improves the visual effect of diffusion. This article discusses how to apply anisotropic diffusion methods and ideas to video image segmentation. We introduced the classic watershed segmentation algorithm and used forward-backward diffusion to process video images to reduce oversegmentation, introduced the active contour model and its improved GVF Snake, and analyzed the idea of how to use anisotropic diffusion and improve the GVF Snake model to get a new GGVF Snake model. In the study of basketball segmentation of close-up shots, we propose an improved Hough transform method based on a variable direction filter, which can effectively extract the center and radius of the basketball. The algorithm has good robustness to basketball partial occlusion and motion blur. In the basketball segmentation research of the perspective shot, the commonly used object segmentation method based on the change area detection is very sensitive to noise and requires the object not to move too fast. In order to correct the basketball segmentation deviation caused by the video noise and the fast basketball movement, we make corrections based on the peak characteristics of the edge gradient. At the same time, the internal and external energy calculation methods of the traditional active contour model are improved, and the judgment standard of the regional optimal solution and segmentation validity is further established. In the basketball tracking research, an improved block matching method is proposed. On the one hand, in order to overcome the influence of basketball’s own rotation, this article establishes a matching criterion that has nothing to do with the location of the area. On the other hand, this article improves the diamond motion search path based on the basketball’s motion correlation and center offset characteristics to reduce the number of searches and improve the tracking speed.