In the field of computer vision and robotics, scholars use object tracking technology to track objects of interest in various video streams and extend practical applications, such as unmanned vehicles, self-driving cars, robotics, drones, and security surveillance. Object tracking is a mature technology in the field of computer vision and robotics; however, there is still no one object tracking algorithm that can comprehensively and simultaneously solve the four problems encountered by tracking objects, namely deformation, illumination variation, motion blur, and occlusion. We propose an algorithm called an adaptive dynamic multi-template correlation filter (ADMTCF) which can simultaneously solve the above four difficulties encountered in tracking moving objects. The ADMTCF encodes local binary pattern (LBP) features in the HSV color space, so the encoded features can resist the pollution of the tracking image caused by illumination variation. The ADMTCF has four templates that can be adaptively and dynamically resized to maintain tracking accuracy to combat tracking problems such as deformation, motion blur, and occlusion. In this paper, we experimented with our ADMTCF algorithm and various state-of-the-art tracking algorithms in scenarios such as deformation, illumination variation, motion blur, and occlusion. Experimental results show that our proposed ADMTCF exhibits excellent performance, stability, and robustness in various scenarios.