Castanopsis hystrix is a valuable native, broad-leaved, and fast-growing tree in South China. In this study, 15 phenotypic traits and 32 simple sequence repeat (SSR) markers were used to assess the genetic diversity and population structure of a natural population of C. hystrix and to construct a core germplasm collection by a set of 232 accessions. The results showed that the original population of C. hystrix had relatively high genetic diversity, with the number of alleles (Na), effective number of alleles (Ne), observed heterozygosity (Ho), expected heterozygosity (He), Shannon's information index (I), and polymorphism information content (PIC) averaging at 26.188, 11.565, 0.863, 0.897, 2.660, and 0.889, respectively. Three sub-populations were identified based on a STRUCTURE analysis, indicating a strong genetic structure. The results from the phylogenetic and population structures showed a high level of agreement, with 232 germplasms being classified into three main groups. The analysis of molecular variance (AMOVA) test indicated that 96% of the total variance was derived from within populations, which revealed a low differentiation among populations. A core collection composed of 157 germplasms was firstly constructed thereafter, of which the diversity parameters non-significantly differed from the original population. These results revealed the genetic diversity and population structure of C. hystrix germplasms, which have implications for germplasm management and genome-wide association studies on C. hystrix, as well as for core collection establishment applications in other wood-producing hardwood species.