In this paper a segmentation algorithm has been described based on Markov Random Field (MRF) processing. The images are segmented initially by growing regions of similar color values. Then a general framework for refining initial clusters in a feature space using MRF processing is presented and subsequently, the algorithm for MRF processing in the color spaces is proposed. The proposed MRF processing is shown to be working with the principles of 1-NN and K-NN classification rules among the neighbors of a pixel. This has remarkably improved the initial segmentation results. The segmentation algorithm in this work, first, uses only the chromatic information. Then experimentations using all the three color components (i.e. with the inclusion of luminance factors also) are also presented. It has been observed that, though using only chromatic information good segmentation results are obtained, the luminance information improves the quality of segmentation in some cases. Results for different color spaces such as the OHTA coordinate space (referred to as the OHTA space in the present work), YIQ, CIELAB and UVW are also presented here. On the average the performance in the OHTA space is found to be better than the others.