Cacao (Theobroma cacao L.) is a tropical tree species belonging to the Malvaceae, which originated in the lowland rainforests of the Amazon. It is a major agricultural commodity, which contributes towards the Gross Domestic Product of West African countries, where it accounts for about 70% of the world’s production. Understanding the genetic diversity of genetic resources in a country, especially for an introduced crop such as cacao, is crucial to their management and effective utilization. However, very little is known about the genetic structure of the cacao germplasm from Sierra Leone and Togo based on molecular information. We assembled cacao germplasm accessions (235 from Sierra Leone and 141 from Togo) from different seed gardens and farmers’ fields across the cacao-producing states/regions of these countries for genetic diversity and population structure studies based on single nucleotide polymorphism (SNP) markers using 20 highly informative and reproducible KASP–SNPs markers. Genetic diversity among these accessions was assessed with three complementary clustering methods, including model-based population structure, discriminant analysis of principal components (DAPC), and phylogenetic trees. STRUCTURE and DAPC exhibited some consistency in the allocation of accessions into subpopulations or groups, although some discrepancies in their groupings were noted. Hierarchical clustering analysis grouped all the individuals into two major groups, as well as several sub-clusters. We also conducted a network analysis to elucidate genetic relationships among cacao accessions from Sierra Leone and Togo. Analysis of molecular variance (AMOVA) revealed high genetic diversity (86%) within accessions. A high rate of mislabeling/duplicate genotype names was revealed in both countries, which may be attributed to errors from the sources of introduction, labeling errors, and lost labels. This preliminary study demonstrates the use of KASP–SNPs for fingerprinting that can help identify duplicate/mislabeled accessions and provide strong evidence for improving accuracy and efficiency in cacao germplasm management as well as the distribution of correct materials to farmers.