We study ultracold color fermions with three internal states Red, Green and Blue with ${\rm SU(3)}$ symmetry in optical lattices, when color-orbit coupling and color-flip fields are present. This system corresponds to a generalization of two-internal state fermions with ${\rm SU(2)}$ symmetry in the presence of spin-orbit coupling and spin-flipping Zeeman fields. We investigate the eigenspectrum and Chern numbers to describe different topological phases that emerge in the phase diagrams of color-orbit coupled fermions in optical lattices. We obtain the phases as a function of artificial magnetic, color-orbit and color-flip fields that can be independently controlled. For fixed artificial magnetic flux ratio, we identify topological quantum phases and phase transitions in the phase diagrams of chemical potential versus color-flip fields or color-orbit coupling, where the chirality and number of midgap edge states changes. The topologically non-trivial phases are classified in three groups: the first group has total non-zero chirality and exhibit only the quantum charge Hall effect; the second group has total non-zero chirality and exhibit both quantum charge and quantum color Hall effects; and the third group has total zero chirality, but exhibit the quantum color Hall effect. These phases are generalizations of the quantum Hall and quantum spin Hall phases for charged spin-$1/2$ fermions. Lastly, we also describe the color density of states and a staircase structure in the total and color filling factors versus chemical potential for fixed color-orbit, color-flip and magnetic flux ratio. We show the existence of incompressible states at rational filling factors precisely given by a gap-labelling theorem that relates the filling factors to the magnetic flux ratio and topological quantum numbers.