With the rapid development of Unmanned Aerial Vehicle (UAV) technology, more and more UAVs have been used in forest survey. UAV (RGB) images are the most widely used UAV data source in forest resource management. However, there is some uncertainty as to the reliability of these data when monitoring height and growth changes of low-growing saplings in an afforestation plot via UAV RGB images. This study focuses on an artificial Chinese fir (Cunninghamia lancelota, named as Chinese Fir) young forest plot in Fujian, China. Divide-and-conquer (DAC) and the local maximum (LM) method for extracting seedling height are described in the paper, and the possibility of monitoring young forest growth based on low-cost UAV remote sensing images was explored. Two key algorithms were adopted and compared to extract the tree height and how it affects the young forest at single-tree level from multi-temporal UAV RGB images from 2019 to 2021. Compared to field survey data, the R2 of single saplings’ height extracted from digital orthophoto map (DOM) images of tree pits and original DSM information using a divide-and-conquer method reached 0.8577 in 2020 and 0.9968 in 2021, respectively. The RMSE reached 0.2141 in 2020 and 0.1609 in 2021. The R2 of tree height extracted from the canopy height model (CHM) via the LM method was 0.9462. The RMSE was 0.3354 in 2021. The results demonstrated that the survival rates of the young forest in the second year and the third year were 99.9% and 85.6%, respectively. This study shows that UAV RGB images can obtain the height of low sapling trees through a computer algorithm based on using 3D point cloud data derived from high-precision UAV images and can monitor the growth of individual trees combined with multi-stage UAV RGB images after afforestation. This research provides a fully automated method for evaluating the afforestation results provided by UAV RGB images. In the future, the universality of the method should be evaluated in more afforestation plots featuring different tree species and terrain.