We study lepton violating Higgs (HLFV) decays, first from the effective field theory (EFT) point of view, and then analysing the different high-energy realizations of the operators of the EFT, highlighting the most promising models. We argue why two Higgs doublet models can have a $\mathrm{BR}(h\rightarrow \tau \mu)\sim 0.01$, and why this rate is suppressed in all other realizations including vector-like leptons. We further discuss HLFV in the context of neutrino mass models: in most cases it is generated at one loop giving always $\mathrm{BR}(h\rightarrow \tau \mu) < 10^{-4}$ and typically much less, which is beyond experimental reach. However, both the Zee model and extended left-right symmetric models contain extra ${\rm SU(2)}$ doublets coupled to leptons and could in principle account for the observed excess, with interesting connections between HLFV and neutrino parameters.