We perform a comprehensive analysis of the homogeneous finite modular group $\Gamma'_4\equiv S'_4$ which is the double covering of $S_4$ group. The weight 1 modular forms of level 4 are constructed in terms of Dedekind eta function, and they transform as a triplet $\mathbf{\hat{3}'}$ of $S'_4$. The integral weight modular forms until weight 6 are built from the tensor products of weight 1 modular forms. We perform a systematical classification of $S'_4$ modular models for lepton masses and mixing with/without generalized CP, where the left-handed leptons are assigned to triplet of $S'_4$ and right-handed charged leptons transform as singlets under $S'_4$, and we consider both scenarios where the neutrino masses arise from Weinberg operator or type I seesaw mechanism. The phenomenological implications of the minimal models for lepton masses, mixing angles, CP violation phases and neutrinoless double decay are discussed. The $S'_4$ modular symmetry is extended to quark sector, we present several predictive models which use nine or ten free parameters including real and imaginary parts of $\tau$ to describe quark masses and Cabibbo-Kobayashi-Maskawa mixing matrix. We give a quark-lepton unified model which can explain the flavor structure of quarks and leptons simultaneously for a common value of $\tau$.