A 500-Fps Pan-Tilt Tracking System With Deep-Learning-Based Object Detection

Abstract

In this letter, we propose a fast mirror-drive pan-tilt target tracking system that can robustly track an object whose appearance varies in a complex background at 500 fps. By assuming a small image displacement between frames, which is a property of high-frame rate vision, we develop an fast object tracking algorithm by hybridizing the convolutional-neural-network (CNN) based object detection with template-matching (TM) based tracking operating at hundreds of frames per second (fps). For object tracking with high-speed visual feedback, the proposed tracking algorithm can remarkably reduce dozens-of-milliseconds-latency in the CNN-based object detection by simultaneously executing TM-based tracking for several images at consecutive frames within a few milliseconds. In the proposed pan-tilt tracking system, when the current tracked objects are occluded or out of the camera view, it can recognize objects to be newly tracked with CNN-based object detection at the rate of 33 fps with acceleration using graphic processing units (GPUs). Controlling the pan-tilt tracking system via visual feedback at 500 Hz, fast moving objects can be robustly tracked at the center of the camera view. The effectiveness of our method was experimentally demonstrated via several results when fast-moving pre-learned objects, such as toy cars were tracked in complex backgrounds.