In this paper, we comprehensively study optimality conditions for rank-constrained matrix optimization (RCMO). By calculating the Clarke tangent and normal cones to a rank-constrained set, along with the given Frechet, Mordukhovich normal cones, we investigate four kinds of stationary points of the RCMO and analyze the relations between each stationary point and local/global minimizer of the RCMO. Furthermore, the second-order optimality condition of the RCMO is achieved with the help of the Clarke tangent cone.