Revealing the relationship between the mating system (i.e., the outcrossing/inbreeding degree) and the fitness of seeds in tree species under wild conditions is essential for understanding the ecological adaptability and evolutionary stability of the species. This study collected open-pollinated seeds from seven wild populations of Pinus yunnanensis var. tenuifolia that exhibited fragmentation in the Nanpan–Hongshui River basin, an ecologically fragile area in China. The seeds and sprouts (germinated seeds) from 20 families were genotyped (24 seeds and 24 sprouts per family) using twelve microsatellite loci to reveal the genetic diversity, mating status, and effect of inbreeding on the three seed quality indicators (thousand-seed weight, germination rate, and germination potential). The three seed quality indicators differed significantly between families (p < 0.001). Higher values of genetic diversity (except the observed heterozygosity) were observed in the sprout group than those in the seed group. Families from different populations showed a notable genetic differentiation (Φst = 0.12), and a large part of families from the common populations had a high degree of coancestry, which signified that the current habitat fragmentation is limiting gene flow between populations. High levels of outcrossing rates (tm) were observed in both the seed group (tm = 0.974) and the sprout group (tm = 0.978), indicating that a low proportion of seeds were self-fertilized. Although there was a slightly higher single outcrossing rate (ts = 0.888) and a lower proportion of biparental inbreeding (tm − ts = 0.077) in sprouts compared to the seeds (ts = 0.871, tm − ts = 0.091), indicating that a part of inbred seeds were purged during the germination stage, curve fitting between the outcrossing rate and seed quality indicators showed that a certain degree of biparental inbreeding (ts between 0.89 and 0.91 and tm − ts between 0.09 and 0.11) did have a positive effect on seed germination ability. This highlights that excessive inbreeding or outbreeding seems to be unfavorable to seed viability. The peculiar relationship between seed viability and the mating system in P. yunnanensis var. tenuifolia was likely an evolutionary consequence of a trade-off between the nature of mixed mating and its specific ecological niche.