We propose a comprehensive physical model explaining the conical character of broadband terahertz generation from femtosecond two-color laser-induced air plasma filaments. We show, that, in contrast to other models, emission is always conical, resulting from phase matching of the radiation produced inside the filament combined with a partial back reflection of the generated terahertz field from the filament itself due to the frequency-dependent critical plasma density. The obtained conical angle varies from 2${}^{\ensuremath{\circ}}$ to 10${}^{\ensuremath{\circ}}$, depending on the plasma density distribution and filament length. Unlike previously proposed models, our model shows good agreement with our experiments as well as a wide range of experimental findings from the literature.