To study the extraction process of agarwood active ingredients (AA) and investigate the safety and effectiveness of AA in the treatment of insomnia rats by nasal administration. A β-cyclodextrin (β-CD) inclusion compound (a-β-CD) was prepared from agarwood essential oil (AEO), and the preparation process was optimized and characterized. The safety of AA in nasal mucosa was evaluated through Bufo gargarizans maxillary mucosa and rat nasal mucosa models. Insomnia animal models were replicated by injecting p-chlorophenylalanine (PCPA), conducting behavioral tests, and detecting the expression levels of monoamine neurotransmitters (NE and 5-HT) and amino acids (GABA/Glu) in the rat hypothalamus. The optimum inclusion process conditions of β-CD were as follows: the feeding ratio was 0.35:1.40 (g:g), the inclusion temperature was 45 °C, the inclusion time was 2 h, and the ICY% and IEO% were 53.78 ± 2.33% and 62.51 ± 3.21%, respectively. The inclusion ratio, temperature, and time are the three factors that have significant effects on the ICY% and IEO% of a-β-CD. AA presented little damage to the nasal mucosa. AA increased the sleep rate, shortened the sleep latency, and prolonged the sleep time of the rats. The behavioral test results showed that AA could ameliorate depression in insomnia rats to a certain extent. The effect on the expression of monoamine neurotransmitters and amino acids in the hypothalamus of rats showed that AA could significantly reduce NE levels and increase the 5-HT level and GABA/Glu ratio in the hypothalamus of insomnia rats. The preparation of a-β-CD from AEO can reduce its irritation, improve its stability, increase its curative effect, and facilitate its storage and transport. AA have certain therapeutic effects on insomnia. The mechanism of their effect on rat sleep may involve regulating the expression levels of monoamine neurotransmitters and amino acids in the hypothalamus.