We proceed further in the study of harmonicity for almost contact metric structures already initiated by Vergara-Díaz and Wood. By using the intrinsic torsion, we characterise harmonic almost contact metric structures in several equivalent ways and show conditions relating harmonicity and classes of almost contact metric structures. Additionally, we study the harmonicity of such structures as a map into the quotient bundle of the oriented orthonormal frames by the action of the structural group U(n)×1. Finally, by using a Bochner type formula proved by Bör and Hernández Lamoneda, we display some examples which give the absolute minimum for the energy.