Combining Wearable Tracking Data and Deep Learning for Moving Camera Calibration

Abstract

AbstractThis study aimed to investigate the feasibility of using wearable tracking data to train a player detection and moving camera calibration model for Australian Football. Player tracking data were collected from multiple matches of professional Australian Football using wearable local positioning system (LPS) devices sampling at 10 Hz. Each match was also filmed from two angles with moving high-definition cameras. Image registration using the Speeded-Up Robust Features (SURF) keypoint detector and descriptor was performed to calibrate the moving camera footage and a pre-trained object detector was used to detect players in each frame. These methods required no manual annotation of the video footage and resulted in an incomplete player tracking data set. This initial data set was cross referenced against LPS player locations to identify the subset of successfully calibrated frames. This subset was used to train a deep learning based landmark detection model to track known markings on the field, and to fine tune the object detection model, by using the LPS tracking data to annotate the frames. The pre-trained object detection model had low precision and recall (<50%) and when combined with the image registration method resulted in ~ 10% of frames deemed suitable for model training. A stacked hourglass model trained on this subset of frames was able to correctly calibrate greater than 80% of all frames in most videos but under-performed in lighting conditions under-represented in the training data. Re-training the object detection model on the football player annotations from the LPS devices increased the precision and recall of player detections to > 85%. Wearable sensor-based player tracking data can be effectively used to supervise computer vision-based approaches to moving camera calibration and player detection in Australian football.KeywordsComputer visionAustralian footballDeep learning