The long-term and large-scale mining of rare earth minerals may lead to an accumulation of rare earth elements (REEs) in the environment, posing potential health risks to residents. We collected scalp hair (n = 254) from residents of a smelting area, a mining area, and a reference area to clarify human exposure to REEs. The contents of 15 REEs investigated in human hair samples were notably higher in the mining and smelting areas than in the reference area. Significant differences between some REEs were observed between the mining and smelting areas, for instance, cerium (Ce), dysprosium, and praseodymium. In the study areas, exposure to different sources of REEs may be one of the factors that contributed to the variations of REE correlations and clusters in human hair. Furthermore, in the smelting area, Ce contents in hair decreased with increasing age of children. However, Ce contents in the hair of adults increased with age. In contrast, Ce accumulation continuously increased in the reference area residents’ hair with age. Regression results indicated that age and location were more important than sex when considering the influence on REE accumulation in residents’ hair. The results of this study may help policymakers to implement guidelines to alleviate residents’ exposure to REE in mining and smelting areas.