A systematic study of constraints in SU(2) and SU(3) Yang-Mills classical mechanics is performed. Except for the SU(2) case with vanishing spatial angular momenta they turn out to be non-holonomic. Using Dirac's constraint formalism we achieve a complete elimination of the unphysical gauge and rotational degrees of freedom. This leads to an effective unconstrained formulation both for the full SU(2) Yang-Mills classical mechanics and for the SU(3) case in the subspace of vanishing spatial angular momenta. We believe that our results are well suited for further explicit dynamical investigations.