Siamese-based trackers suffer from over-parameterization and online learning limitations, making it difficult to balance high performance and real-time execution when deployed on resource-constrained devices. In this paper, we propose a unified tracking framework that integrates lightweight Siamese network and template-guided learning. Specifically, we propose a two-step pruning method for compressing the Siamese network, which examines both the statistical distribution and correlation patterns of filters. By removing the filters with less importance and replaceable contribution, as well as their connected feature maps, network optimization is realized without custom architectures. Furthermore, we construct a template-guided learning model to capture target appearance information and suppress distracters. This can effectively ensure tracking performance in specific scenarios. Extensive experiments on OTB50, OTB-2013, OTB-2015, DTB70, UAV123, UAV20L, VOT2019, GOT-10K, LaSOT and TrackingNet indicate that the proposed method outperforms several state-of-the-art trackers and is faster. In particular, our lightweight Siamese network reduces the model size by 3.4× and FLOPs by 3.7× without significantly sacrificing performance while running at 116 fps.