Disentangling flavor violation in the top-Higgs sector at the LHC

Abstract

We study the LHC phenomenology of flavor changing Yukawa couplings between top quark, Higgs boson, and either an up or charm quark. Such $tuh$ or $tch$ couplings arise for instance in models in which Higgs sector is extended by the existence of additional Higgs bosons or by higher dimensional operators. We emphasize the importance of anomalous single top plus Higgs production in these scenarios, in addition to the more widely studied $t \to h j$ decays. By recasting existing CMS searches in multilepton and diphoton plus lepton final states, we show that bounds on $tuh$ couplings are improved by a factor of 1.5 when single top plus Higgs production is accounted for. We also recast the CMS search for vector boson plus Higgs production into new, competitive constraints on $tuh$ and $tch$ couplings, setting the limits of $BR(t\to hu) < 0.7%$ and $BR(t\to hc) <1.2%$. We then investigate the sensitivity of future searches in multilepton channel and in fully hadronic channel. In multilepton searches, studying the lepton rapidity distributions and charge assignments can be used to discriminate between $tuh$ couplings, for which anomalous single top production is relevant, and $tch$ couplings, for which it is suppressed by the parton distribution function of the charm quark. An analysis of fully hadronic $t+h$ production and $t\to h j$ decay can be competitive with the multilepton search at 100 fb$^{-1}$ of 13 TeV data if jet substructure techniques are employed to reconstruct boosted top quarks and Higgs bosons. To show this we develop a modified version of the HEPTopTagger algorithm, optimized for tagging $t \to h j$ decays. Our sensitivity estimates on $BR(t\to hu)$ ($BR(t\to hc)$) at 100 fb$^{-1}$ of 13 TeV data for multilepton searches, vector boson plus Higgs search and fully hadronic search are $0.22%$ ($0.33%$), $0.15%$ ($0.19%$) and $0.36%$ ($0.48%$), respectively.