A Hermitian metric on a complex manifold $(M, I)$ of complex dimension $n$ is called Calabi-Yau with torsion (CYT) or Bismut-Ricci flat, if the restricted holonomy of the associated Bismut connection is contained in ${\rm SU}(n)$ and it is called strong K\"ahler with torsion (SKT) or pluriclosed if the associated fundamental form $F$ is $\partial \overline \partial$-closed. In the paper we study the existence of left-invariant SKT and CYT metrics on compact semi-simple Lie groups endowed with a Samelson complex structure $I$. In particular, we show that if $I$ is determined by some maximal torus $T$ and $g$ is a left-invariant Hermitian metric, which is also invariant under the right action of the torus $T$, and is both CYT and SKT, then $g$ has to be Bismut flat.