Cathaya argyrophylla, referred as “the giant panda of plants”, is an endangered relict conifer species endemic to subtropical mountainous in China. The remanent populations of C. argyrophylla only survive in four refuges, while how the species migrated into these refuges remains unknown. To investigate the genetic structure and history of C. argyrophylla, here we genotyped four natural populations of C. argyrophylla using restriction site-related DNA sequencing (RAD-Seq). The four populations exhibited a relatively high level of nucleotide diversity (π = 0.29–0.33) and low differentiation (Fst = 0.04–0.12), with relatively high gene flow frequency (Nm = 1.86–6.00). We identified two historical migration events from high latitude populations to low latitude ones. The divergence times were corresponded to the starting time of population decline, which coincided with Naynayxungla Glaciation (∼0.8–0.5 Mya), the last glacial period (LGP, ∼115–11.7 kya) and the Holocene (∼10 kya). These results indicate that historical cooling events drove the endemic status of C. argyrphylla and its migration into low latitude refuges. Our findings provide insights into the evolutionary history and genetic diversity of this endangered species.