We perform a comprehensive study of the homogeneous finite modular group $A'_5$ which is the double covering of $A_5$. The integral weight and level 5 modular forms have been constructed up to weight 6 and they are decomposed into the irreducible representations of $A'_5$. Then we perform a systematical analysis of the $A'_5$ modular models for lepton masses and mixing. The phenomenologically viable models with minimal number of free parameters and the results of fit are presented. We find out 15 models with 9 real free parameters which can accommodate the experimental data of lepton sector. After including generalized CP symmetry, 9 viable models with 7 free parameters are found out. We apply $A'_5$ modular symmetry to the quark sector, and a quark-lepton unification model is given. The framework of modular invariance is extended to include the rational weight modular forms of level 5. The ring of modular forms at level 5 can be generated by two algebraically independent weight $1/5$ modular forms denoted by $F_1(\tau)$ and $F_2(\tau)$. We give the expressions of the rational weight modular forms of level 5 up to weight $3$ and arrange them into the irreducible multiplets of finite metaplectic group $\widetilde{\Gamma}_5\cong A'_5\times Z_5$. A neutrino mass model with $\widetilde{\Gamma}_5$ modular symmetry is presented, and the phenomenological predictions of the model are analyzed numerically.