Ecological restoration has great significance on cut rock slopes, which are considered extremely degraded habitats. The development of moss–soil crusts on cut rock slopes as a critical pathway to ecological restoration in mountain areas has been poorly reported. A total of 335 quadrats were selected on cut rock slopes with formation ages between 0 and 60 years to evaluate the evolution characteristics of moss–soil crusts under various geographical conditions (e.g., aspect, lithology, and altitude) in the mountainous area of western Sichuan, Southwest China. The results suggested that moss growth decoupled from soil accumulation within the crusts and was controlled by multiple factors. Moss growth depended on lithology, altitude, and age, while soil weight was mainly influenced by slope aspect. The development of mosses on limestone was better than on sandstone. Moss biomass varied with altitude, consistent with that of rainfall with respect to moss development dependent on moisture. Furthermore, moss development under a semiarid climate was more distinctly impacted by moisture with altitude relative to a humid region, likely owing to the higher sensitivity of the mosses to moisture in the former than in the latter. Moss biomass increased with recovery time, while the rate of moss biomass development was diverse in different geographical areas. The vegetation developed rapidly in low-altitude areas (~1000 m above sea level), resulting in moss biomass increasing from 0 to 24.08 g·m−2 with formation time increasing from 0.5 to 1.5 years and subsequently being restricted by the evolution of higher plants on cut rock slopes, leading to an insignificant difference in moss biomass between 1.5 and 60 years. In high-altitude areas, when the altitude changed slightly (from 2024 to 2430 m above sea level), the moss biomass on cut rock slopes increased linearly with increasing age from 5 to 27 years. Influenced by the surrounding fertile soils and moss bioaccumulation, there were high levels of soil major nutrient content, especially the organic matter content, which reached 377.42 g·kg−1. More soils accumulated on south-facing slopes than on north-facing slopes. This study provided field data to clearly reveal the influence of geographic factors on moss–soil crust development in natural restoration processes in high-altitude mountainous areas.