Vision-based dynamic multi-objective detection and stable tracking has been a hot topic in recent years. This paper proposed a novel multi-target detection and tracking algorithm based on false-alarms elimination with classification and detection tracking fusion. Firstly, adaptive threshold segmentation and multiple morphological processing were used to finish target detecting, and center-peripheral gray scale difference classification method was proposed to remove false alarm target; secondly, the improved spatial phase correlation algorithm was used to complete the tracking of the target one by one; and thirdly, the target detection and tracking fusion algorithm based on a minimum cost function was also proposed to reduce the false alarm rate of the target. Experiment tests show that compared with the traditional method, the target detection accuracy of the detection and tracking fusion algorithm improves by 7% and the algorithm is able to adaptively adjust the tracking frame size and recapture the target after the target is lost. It can satisfy multiple targets tracking at the same time.