An efficient and accurate estimation of wheat growth and yield is important for wheat assessment and field management. To improve the accuracy and stability of wheat growth and yield estimation, an estimation method based on a genetic algorithm-improved support vector regression (GA-SVR) algorithm was proposed in this study. The correlation analysis between vegetation indices calculated from spectral data and wheat growth phenotypes and yields was performed to obtain the optimal combination of vegetation indices with high correlation and good estimation performance. At the same time, the optimal model for wheat growth monitoring was screened and constructed in experiments with 12 wheat varieties and 3 gradient nitrogen fertilizer application levels. Then, the yield estimation model was established and its applicability was verified under different nitrogen fertilizer application levels. The results showed that the constructed models for the leaf area index, plant height, and yield estimation performed well, with coefficients of determination of 0.82, 0.71, and 0.70, and root mean square errors of 0.09, 2.7, and 68.5, respectively. This study provided an effective UAV remote sensing technique for monitoring wheat growth status and estimating yield. This study provides an effective unmanned aerial remote sensing technique for monitoring wheat growth and estimating yield, and provides technical support for wheat yield assessment and field management.