Color textures are among the most important visual attributes in image analysis. From the practical point view of color texture image analysis, this paper proposes an effective multi-scale color texture classification algorithm that is rotation and scale invariant using non-marginal color monogenic wavelet transform. The proposed algorithm exploits the color monogenic wavelet transform to obtain multi-scale representation of training samples for each texture class. The coefficients of color monogenic wavelet transform represent a magnitude and three phases: two phases encode local color information while the third contains geometric information of color texture image. The multi-scale feature vector is composed of mean value, standard deviation, energy and entropy at different scales of each of the directional sub-bands. The experimental results of average correct classification rates are 98.67, 99.08 and $$99.89\%$$ which are obtained from different color texture databases demonstrate its superior performance and robustness of the proposed classifier. The proposed color texture feature vector is also shown to be effective for color texture classification.