In order to solve the problem that the object template of traditional Mean Shift tracking can only be built from a single image,and difficult to update,an improved Mean Shift tracking algorithm for color image is proposed.Firstly,RGB color space is projected to HSV color space,and a unified histogram kernel model based on HSV color space is set up.Secondly,in order to achieve template online update,an online support vector machine is introduced,and a Mean Shift tracking algorithm integrated with online SVM based on HSV color space is reasoned.By above operation,object modeling is adaptive to object size,posture or illumination changes.Finally,tracking test on two groups of international general CAVIAR color image sequence is taken to verify effectiveness of the algorithm. Experiments show that the improved algorithm performs well with great changes taken in target pose, illumination or background.When the image resolution is 384pixel×288pixel(target size of about 20 pixel×80pixel),the fastest processing speed reaches 40f/s,as well as tracking precision increases by 32.1%than traditional Mean Shift.