This study aims to investigate the impact of different environmental conditions on the stability of RGB images in ecological sites and the anti-interference properties of vegetation images on different underlying surfaces. Three vegetation types including sugarcane, forest, and karst (mainly shrub and grass) are used to segment green vegetation using machine learning, and the RGB vegetation indices are calculated using color channel data. Then, The effect of weather, season, and time period on different types of vegetation indices are studied, which provide technical references for quantitative application of RGB image data. The results indicate the following: ① For the vegetation with high canopy density, the SVM machine learning segmentation algorithm used in this study is more applicable, as the RGB image segmentation accuracy of sugarcane and forest is significantly higher than karst. For different weather conditions, the segmentation accuracy of sugarcane and forest RGB images on sunny or cloudy days is higher than that on rainy or foggy days, but the effect on sparse vegetation in karst is not obvious. Additionally, the segmentation accuracy of different vegetation types has a small increase with NLM filter processing. ② Change in weather, season, and time can affect the stability of the image index. For different weather conditions, the vegetation indices of sugarcane and forest images on sunny or cloudy days are the more stable (ExGR, outlier proportion between 3.25% and 5.63%), while on rainy and foggy days they are less stable (ExR, outlier proportion between 17.60% and 21.59%). For different seasons, the stability of the sugarcane and karst image index obtained in spring is better (ExG, outlier proportion between 3.32% and 6.88%), while the stability of the forest image index obtained in summer is better (ExR, outlier proportion = 10.32%). For different times within a day, the sugarcane image index obtained in the morning is more stable (ExGR, outlier proportion = 4.62%), while the stability of the forest image index obtained in the afternoon is better (ExGR, outlier proportion = 9.24%). ③ The stability of the sugarcane image index is more affected by weather and season. For forest, the influence of the weather is more obvious than the season. But, for karst, the effect of season on the vegetation index is greater than that of weather.