In this paper, we propose a robust tracking method with a novel appearance model based on fragments-based PCA sparse representation. It samples non-overlapped local image patches within the templates in PCA subspace. Then, the candidate local image patches are sparse represented by the local template patches in PCA subspace. Finally, tracking is continued using the particle filter for propagating sample distributions over time. In addition, the templates are updated online based on incremental subspace learning .Using the fragments-based PCA templates rather than the image templates facilitates the tracker to handle significant illumination and pose change as well as occlusion. Experimental results on challenging videos show that our method can track accurately and robustly, and outperform many other state-of-the-art trackers.