Due to the difficulty in distinguishing the target and the noise in the visible light background and the loss of the target for a short period of time, a scheme of track association and target tracking for the small target in the sky is carried out. Multi-feature similarity is taken into account, such as the maximum intensity, the maximum brightness, the size and the moving speed. Firstly, the trajectory prediction is carried out by the least square prediction method. By updating the real-time data to deduce the trajectory parameters, the target position in the next frame image is predicted. Based on the trajectory confidence test, all current recorded trajectories are traversed. The over-confidence trajectory is detected and the trajectories of continuous lost points are deleted, as well as the trajectories below the threshold of confidence. For the trajectory of the over-confidence, the target trajectory without the significant motion feature is deleted by judging the magnitude of the moving speed, so that the effective trajectory can be displayed while the false target is eliminated. Then, the target tracking is carried out, which is based on the LOG filter and the constant false alarm rate segmentation. Considering the temporal and spatial continuity of the target trajectory and the randomness of the amplitude and position of the noise, the false alarm rate can be reduced by eliminating the high amplitude noise by using the difference between the two in terms of temporal and spatial correlation. Owing to that the setting of the division threshold is not less than 5 times the background variance of the window image, the generation of the noise by the constant false alarm rate segmentation can be avoid when there is no target. In addition, the target coordinates may be abrupt due to sudden changes in target speed and the jitter of the image acquisition equipment. Therefore, we propose a dynamic adjustment scheme for the tracking window size to enable window adaptive tracking, which can set the length and width of the rectangular window to k times of the original size according to actual needs. At the same time, the number of points of the relatively stationary target in each frame of the image can be counted in real time. Finally, the reliability of the method is verified by simulation experiments, and the running speed is relatively fast and the recognition accuracy is relatively high.