This paper proposes a novel color medical image encryption method based on mean shift algorithm and fractional-order hyperchaotic system. Firstly, color medical images were divided into lesion area, detection area, and edge area by using mean shift clustering. Low-value pixels and zero-value pixels in the edge region are abandoned to improve the encryption efficiency. Secondly, the doctor-patient information is embedded into the pixel matrix of the lesion region by using discrete wavelet transform and singular value decomposition. Thirdly, the pixels of the lesion region and the detection region are processed by using fractional-order hyperchaotic system. Finally, the color medical image embedded with doctor-patient information can be quickly encrypted. Security analysis shows that the encryption method has not only strong robustness and security, but also good performance against various attacks. SPNA and CA results: (a) s1-DE-SPNA, (b) s2-DE-SPNA, (c) s3-DE-SPNA, (d) s4-DE-SPNA; (e) s1-DE-CA, (f) s2-DE-CA, (g) s3-DE-CA, (h) s4-DE-CA.