In this study, the relationship between the anisotropy of toughness and microstructure in low carbon low alloy steel treated by quenching and tempering (QT) heat treatment was investigated with the aid of scanning electron microscope, electron back-scattered diffraction and transmission electron microscope combined with energy disperse spectroscopy techniques. Results show that the impact toughness of the quenched steel plate along the longitudinal and transverse directions are little different. However, after tempering, the longitudinal impact toughness of QT steel plate is improved by 88%, while the transverse impact toughness is slightly decreased, leading to a significant anisotropy of toughness. Microstructural characterizations reveal that banded microstructure with coarse grain size exists along longitudinal direction (i.e., rolling direction) of steel plate, which is attributed to co-segregation of Mn and C and the resulting uneven recrystallization during hot rolling. After tempering, fine and dispersed carbides are precipitated in matrix microstructure, but high-density coarse carbides are formed within banded microstructure. It is suggested that the coarse grains and high-density coarse carbides significantly deteriorate the resistance against crack propagation along banded microstructure, leading to the anisotropy of toughness of QT steel plate. The findings of this study will aid to design metallurgical processes including chemical composition design, hot rolling, and heat treatments to eliminate the anisotropy of toughness of low alloy steels with high strength and toughness.