Campomanesia xanthocarpa (Mart.) O. Berg is a South American fruit tree species with important ecological and medicinal properties, which remnants are currently found mainly in isolated forest fragments. In this study, SSR markers from three different genomic origins (gene-linked, nuclear neutral, and organellar) were used to evaluate the patterns of genetic diversity, fine-scale spatial genetic structure and historical gene flow in fragmented forest formations of C. xanthocarpa from the Atlantic Forest in southern Brazil. Our results show that the forest fragments present moderate to high levels of genetic diversity in comparison to species presenting similar life traits, although a trend opposite to expected was observed concerning gene-linked and neutral SSR markers. The fine-scale spatial genetic structure revealed different patterns in short and large distance classes, with a distinct influence of gene-linked and neutral markers in driving the genetic structure in each distance class. The presence of an isolation-by-adaptation pattern implies the need for maintenance of the current remnants to assure the conservation of the private alleles. Finally, as the genetic diversity is found predominantly within forest fragments, programs of seed collection and/or genetic rescue should prioritize a larger number of individuals within each fragment, to increase the sampled diversity.