Based on general symmetry considerations we investigate how the dependence of the tunneling anisotropic magnetoresistance (TAMR) on the magnetization direction is determined by the specific form of the spin-orbit coupling field. By extending a phenomenological model, previously proposed for explaining the main trends of the TAMR in (001) ferromagnet/semiconductor/normal-metal magnetic tunnel junctions (MTJs) [J. Moser et al., Phys. Rev. Lett. 99, 056601 (2007)], we provide a unified qualitative description of the TAMR in MTJs with different growth directions. In particular, we predict the forms of the angular dependence of the TAMR in (001),(110), and (111) MTJs with structure inversion asymmetry and/or bulk inversion asymmetry. The effects of in-plane uniaxial strain on the TAMR are also investigated.