• Determined the influence of IAA, GA, ABA, ZT on sex differentiation of Xanthoceras sorbifolium . • Identified 20 significantly differently expressed miRNAs (83 miRNA/target pairs) being related to phytohormone synthesis and signal transduction during sex differentiation. Xanthoceras sorbifolium Bunge is an important woody oilseed tree in north China. However, its yield is severely restricted by a low ratio of female to male flowers, and little is known about the physiological and molecular mechanisms underlying the sex differentiation of flowers. Therefore, carrying out the research in its sex differentiation regulation mechanism is the basis of increasing seed yield. Here, we investigated the cytology of sex differentiation, the content and their ratios of indole-3-acetic acid (IAA), gibberellins (GA 3 ), zeatin (ZT), abscisic acid (ABA), the effects of plant growth regulators on flower development, and the expression profiles of phytohormone-related miRNAs. The results showed that both functional female and male flowers were bisexual at the early stage of flower bud differentiation, while the development of the pistils of male flowers and the stamens of female flowers was arrested at later stage of sex differentiation. Furthermore, IAA, ABA and ZT were shown to be beneficial to pistil development in X. sorbifolium . GA 3 was the key phytohormone in pistil abortion. High level of ABA/ZT was beneficial for microspore mother cell meiosis. ZT/IAA was probably related to pistil abortion. High level of ZT/GA 3 contributed to ovule development. Spraying 6-BA was more effective than GA 3 in increasing female-to-male ratio regardless of CK. In addition, several miRNAs played key roles in sex differentiation, especially miR393, miR396, and miR399. They participated in the regulation of pollen maturation and multiple phytohormone pathways. The present study comprehensively identified the roles of phytohormones and their related miRNAs during sex differentiation in X. sorbifolium , which will be a valuable resource for sex regulation using genetic and physiological methods in the future.