In order to overcome the problems of abnormal detection and low accuracy in the process of football foul feature extraction, this paper proposes a football foul feature extraction method based on deep learning algorithm to accurately identify the fouls in the process of normal competition. In this method, the background is eliminated by the difference between the input image and the background image, so as to obtain the effective detection target. According to the characteristics of football competition, the human motion tracking algorithm is proposed. Through the template representation, candidate target representation, similarity measurement calculation and search strategy, the dynamic target is tracked in real-time, and its dynamic information is obtained. Finally, the star skeleton feature is used to extract the football foul action feature, and the image feature is transformed into available data to realise the data extraction of action feature. The experimental results show that the proposed method can detect the target with low accuracy.