Rare earth elements were extracted using a sulfuric acid roasting-water leaching process. The effect of acid roasting on a new type of low-grade sedimentary rare earth ore found in Guizhou Province, China was analyzed using X-ray diffraction and scanning electron microscopy. A systematic study was conducted on process parameters such as amount of acid, roasting temperature, roasting time, water leaching temperature, and leaching time. The results reveal that the total recovery of rare earth elements reaches 81.37%, which is 3.1 times higher than that achieved through direct acid leaching, under the optimal conditions. In addition, the leaching rate of heavy rare earth elements reaches 72.53%. Rare earth elements and some other valuable metals are transformed into soluble sulfate through the local decomposition of clay minerals under the action of the sulfuric acid attack. The dissolution rates of aluminum, iron, and titanium ions are 34.94%, 17.05%, and 62.77%, respectively. The precipitation rate of Ti reaches 99%, and the loss of rare earth ions in the solution is less than 1%. Meanwhile, the results of a leaching kinetics analysis indicate that the leaching process of rare ions is controlled by diffusion. Precious metal ions such as iron and aluminum in the leaching solution can reduce the adsorption of rare earth ions by kaolinite. This study efficiently recovered rare earth ions under conditions of low calcination temperature and direct water leaching, resulting in reduced energy consumption of the extraction process and acidity of the leaching solution. These findings provide a solid foundation for the further separation and extraction of rare earth ions.