Computer vision algorithms represent an important tool in agriculture. Particularly, foliage segmentation is a fundamental step to achieve accurate estimations in more complex visual analysis such as plant health monitoring, weed detection, growth estimation, and flower and fruit classification. Unfortunately, this complexity leads to the problem of generalizing between datasets, an issue that has not yet been addressed by the state of the art. Our work aims at filling this gap by investigating the role of vegetative indexes and color spaces in different formulations of machine learning algorithms. To this end, we have considered datasets observing variations with respect to crop species, leaf color, acquisition settings and illumination. Furthermore, we performed a comparison with different state of the art approaches that include both thresholding and machine learning. From our analysis, we have proposed a new formulation that consists of combining the CIE Luv color space and support vector machines in order to benefit from contextualized information obtained through neighboring pixels. Experiments show that our approach achieves the best results.