Recently, a global non-smooth optimization method was developed for efficiently eliminating specular highlights from single color image. Although this method was reported to be effective on both natural and medical images, its algorithm’s speed is slow due to the non-smoothness of the objective function. This paper proposes a global smooth optimization method, where two gradient regularization terms used to describe diffuse and specular component features for texture details. A new simplified optimality condition is established and thus a matrix iteration algorithm for fast specular highlight removal is proposed. By employing the gradient regularization of diffuse reflection, image texture detail can be preserved during specular highlight removal. Due to the smoothness of the objective function, the proposed matrix iteration algorithm has lower computational complexity and less storage requirement than the global non-smooth optimization algorithm. Furthermore, the proposed matrix iteration algorithm is guaranteed to converge to a global optimal solution under a fixed step length. Experimental results demonstrate that the proposed matrix iteration algorithm is much faster than the non-smooth optimization algorithm. Moreover, the proposed highlight removal algorithm is more effective on benchmark natural images and real laparoscopic images than other highlight removal algorithms.