Hallucinations are a common non-motor symptom of Parkinson’s disease and various forms of dementias. Yokukansan and Yokukansankachimpihange have attracted attention due to their effectiveness in the treatment of hallucinations of dementia. To clarify which component in these formulas contribute to the effects, at first, we focused on their differences in compositions to examine the pharmacological effects on the selective 5-HT2A/2C agonist 1-(2,5-dimethoxy-4-iodophenyl)-2-aminopropane (DOI)-induced head-twitch response (HTR) in mice that has been used as animal hallucination model. Results indicated that water extract of Byaku-jutsu (Atractylodes japonica) showed a stronger inhibitory effect on DOI-induced HTR than that of So-jutsu (A. lancea) corresponding to their major constituents of atractylenolide III and β-eudesmol, and suggested that the major constituents should be active constituents contributing to the antihallucination effects of Byaku- and So-jutsu. Besides, the part B–C ring (butenolide) in atractylenolide III was found to be similar to the structure of serotonin and suggested that the B–C ring may partially play role in antagonistic activity against serotonin receptors. Thus, a novel, rational design of butenolide-related compounds may as potential lead compounds for new drug development. Analysis of the chemical components of Byaku- and So-jutsu and further study on their structure-activity relationships are currently in progress.