Introduction: Xinjiang Brown cattle are a famous dual-purpose (dairy-beef) cultivated breed in China that occupy a pivotal position within the cattle breeding industry in Xinjiang, China. However, little information is available on the genetic background of this breed. To fill this research gap, we conducted a whole-genome screen using specific-locus amplified fragment sequencing to examine the genetic structure and diversity of 130 Xinjiang Brown cattle-grazing type (XBG, traditional type) cattle. Methods: A subsequent joint analysis incorporating two ancestral breeds, specifically 19 Brown Swiss (BS) foreign and nine Kazakh (KZ) Chinese cattle, as well as 20 Xinjiang Brown cattle-housing type (XBH) cattle, was used to explore the genetic background of the Xinjiang Brown cattle. Results: The results showed that, after nearly a century of crossbreeding, XBG cattle formed a single population with a stable genetic performance. The genetic structure, genetic diversity, and selection signature analysis of the two ancestral types showed highly different results compared to that of XBH cattle. Local ancestry inference showed that the average proportions of XGB cattle within the BS and KZ cattle lineages were 37.22% and 62.78%, respectively, whereas the average proportions of XBH cattle within the BS and KZ cattle lineages were 95.14% and 4.86%, respectively. Thus, XGB cattle are more representative of all Xinjiang Brown cattle, in line with their breeding history, which involves crossbreeding. Two complementary approaches, fixation index and mean nucleotide diversity, were used to detect selection signals in the four aforementioned cattle breeds. Finally, the analysis of 26 candidate genes in Xinjiang Brown cattle revealed significant enrichment in 19 Gene Ontology terms, and seven candidate genes were enriched in three pathways related to disease resistance (CDH4, SIRPB1, and SIRPα) and the endocrine system (ADCY5, ABCC8, KCNJ11, and KCNMA1). Finally, development of the core SNPs in XBG cattle yielded 8,379 loci. Conclusion: The results of this study detail the evolutionary process of crossbreeding in Xinjiang Brown cattle and provide guidance for selecting and breeding new strains of this species.