Cattle-yak, the hybrid offspring of yak and taurine cattle, exhibits male sterility with normal female fertility. Spermatogenesis is arrested in adult cattle-yak, and apoptosis is elevated in spermatogenic cells. Currently, the mechanisms underlying these defects remain elusive. Sertoli cells are the only somatic cells that directly interact with spermatogenic cells in the seminiferous tubules and play essential roles in spermatogenesis. The present study was designed to investigate gene expression signatures and potential roles of Sertoli cells in hybrid sterility in cattle-yak. Immunohistochemical analysis showed that the 5 mC and 5hmC signals in Sertoli cells of cattle-yaks were significantly different from those of age-matched yaks (P < 0.05). Transcriptome profiling of isolated Sertoli cells identified 402 differentially expressed genes (DEGs) between cattle-yaks and yaks. Notably, niche factor glial cell derived neurotrophic factor (GDNF) was upregulated, and genes involved in retinoic acid (RA) biogenesis were changed in Sertoli cells of cattle-yak, suggesting possible impairments of spermatogonial fate decisions. Further studies showed that the numbers of proliferative gonocytes and undifferentiated spermatogonia in cattle-yak were significantly higher than those in yak (P < 0.01). Exogenous GDNF significantly promoted the proliferation of UCHL1-positive spermatogonia in yaks. Therefore, we concluded that altered GDNF expression and RA signaling impacted the fate decisions of undifferentiated spermatogonia in cattle-yak. Together, these findings highlight the role of Sertoli cells and their derived factors in hybrid sterility.