Insomnia, a prevalent sleep disorder, detrimentally affects quality of life and is often challenging to manage with conventional treatments. This study delved into identifying and quantifying the main compounds by employing ultra-performance liquid chromatography-Q-Exactive-Orbitrap mass spectrometry, and further to evaluate the therapeutic potential of Rosa roxburghii folium (RRF) extract, with multiple pharmacological activities. Previous research had hinted at the efficacy of glycosides in influencing the γ-aminobutyric acid (GABAergic) system, which plays a pivotal role in sleep regulation. Utilizing a p-chlorophenylalanine-induced insomnia model in BALB/C mice, this investigation aimed to unravel the effects of various dosages of RRF extract on sleep quality and elucidated its mechanism of action. A total of 66 compounds in the RRF extract were analyzed. Behavioral assessments demonstrated notable enhancements in sleep duration and latency. Biochemical analyses further corroborated these findings, revealing modulation in neurotransmitter levels indicative of a potential mechanism through the GABAergic and serotoninergic pathways. Additionally, histological evaluations suggested anti-inflammatory and antioxidant effects of the RRF extract. The findings from this study underscored the therapeutic efficacy of RRF extract in combating insomnia, particularly highlighting its glycoside components' role. The extract's significant improvement in sleep duration and latency, alongside its modulation of neurotransmitter levels, showcases its potential as a natural remedy for insomnia. Through its action on the GABAergic and serotoninergic pathways, as well as its anti-inflammatory and antioxidant effects, RRF extract emerges as a promising candidate for insomnia treatment, offering a holistic approach to sleep disorder management. © 2024 Society of Chemical Industry.