Flavor violating top decays and flavor violating quark decays of the Higgs boson

Abstract

In the Standard Model, flavor violating decays of the top quark and of the Higgs boson are highly suppressed. Further, the flavor violating decays of the top and of the Higgs are also small in MSSM and not observable in current or in near future experiment. In this work, we show that much larger branching ratios for these decays can be achieved in an extended MSSM model with an additional vector-like quark generation. Specifically, we show that in the extended model, one can achieve branching ratios for [Formula: see text] and [Formula: see text] as large as the current experimental upper limits given by the ATLAS and the CMS Collaborations. We also analyze the flavor violating quark decay of the Higgs boson, i.e. [Formula: see text] and [Formula: see text]. Here again, one finds that the branching ratio for these decays can be as large as [Formula: see text]. The analysis is done with inclusion of the CP phases in the Higgs sector, and the effect of CP phases on the branching ratios is investigated. Specifically, the Higgs sector spectrum and mixings are computed involving quarks and mirror quarks, squarks and mirror squarks in the loops consistent with the Higgs boson mass constraint. The resulting effective Lagrangian with inclusion of the vector-like quark generation induce flavor violating decays at the tree level. In the analysis, we also include the experimental constraints from the flavor changing quark decays of the [Formula: see text] boson. The test of the branching ratios predicted could come with further data from LHC13 and such branching ratios could also be accessible at future colliders such as the Higgs factories where the Higgs couplings to fermions will be determined with greater precision.