Unraveling the distributions and controls of soil carbon (C) and nitrogen (N) accumulation is conducive to providing guidance for sustainable ecosystem management. Although the factors that affect soil C and N accumulation have been widely investigated, the relative importance of these factors and whether they confer direct or indirect effects remain unclear. Herein, we examined the distributions of soil organic carbon (SOC), soil inorganic carbon (SIC), and total nitrogen (TN) stocks across the 0–200 cm soil profiles in grasslands, forestlands, and shrublands on the Loess Plateau. Moreover, we determined the relative contributions of land use, topography, and soil properties (texture, water, phosphorus, and available nitrogen) to their variations. The results showed that with increasing soil depth, SOC and TN stocks decreased, while SIC stocks fluctuated. However, large amounts of SOC, SIC, and TN were stored in the deep soil layers (100–200 cm), with values of 3.06, 26.63, and 0.50 kg/m2, respectively. In the 0–200 cm soil layers, the total SIC stock was 53.40 kg/m2 and was 6.8 times higher than the total SOC stock. Among different plant communities, grasslands had the highest SIC stock (54.83 kg/m2), while Armeniaca sibirica and Robinia pseudoacacia had the highest SOC stock (10.06 kg/m2) and TN stock (1.49 kg/m2), respectively. Furthermore, approximately 85% of the variations in SOC, SIC, and TN stocks were jointly affected by land use, topography, and soil properties, in which soil properties acted as the most important driving factors, followed by topography and land use. In particular, topography affected SOC, SIC, and TN stocks mainly through its effects on soil texture and water. Overall, vegetation restoration scenarios incorporating grasslands and Robinia pseudoacacia are highly recommended and should be tailored according to specific topographic features.