Abstract We present a flavor model with $S_3$ modular invariance in the framework of SU(5) grand unified theory (GUT). The $S_3$ modular forms of weights $2$ and $4$ give the quark and lepton mass matrices with a common complex parameter, the modulus $\tau$. The GUT relation of down-type quarks and charged leptons is imposed by the vacuum expectation value (VEV) of the adjoint 24-dimensional Higgs multiplet in addition to the VEVs of $5$ and $\bar 5$ Higgs multiplets of SU(5). The observed Cabibbo–Kobayashi–Maskawa and Pontecorvo–Maki–Nakagawa–Sakata mixing parameters as well as the mass eigenvalues are reproduced properly. We discuss the leptonic charge–parity phase and the effective mass of the neutrinoless double beta decay with the sum of neutrino masses.