We perform a systematical and analytical study of lepton mixing which can be derived from the subgroups of $SU(3)$ under the assumption that neutrinos are Dirac particles. We find that type D groups can predict lepton mixing patterns compatible with the experimental data at $3\sigma$ level. The lepton mixing matrix turns out to be of the trimaximal form, and the Dirac CP violating phase is trivial. Moreover, we extend the flavor symmetry to the quark sector. The Cabibbo mixing between the first two generations of quarks can be generated by type D groups. Since all the finite subgroups of $U(3)$ which are not the subgroups of $SU(3)$ have not been classified, an exhaustive scan over all finite discrete groups up to order 2000 is performed with the help of the computer algebra system \texttt{GAP}. We find that only 90 (10) groups for Dirac (Majorana) neutrinos can generate the lepton mixing angles in the experimentally preferred ranges. The lepton mixing matrix is still the trimaximal pattern and the Dirac CP phase remains trivial. The smallest groups which lead to viable mixing angles are $[162, 10]$, $[162, 12]$ and $[162, 14]$. For quark flavor mixing, the correct order of magnitude of the CKM matrix elements can not be generated. Only the Cabibbo mixing is allowed even if we impose very loose constraints $0.1\leq|\left(V_{CKM}\right)_{12}|\leq0.3$ and $|\left(V_{CKM}\right)_{13}|\leq|\left(V_{CKM}\right)_{23}|<|\left(V_{CKM}\right)_{12}|$. The group $\Delta(6\cdot7^2)$ can predict a Cabibbo angle $\theta_q=\pi/14$ in good agreement with the best fit value. The groups which can give rise to both phenomenologically viable lepton mixing angles and acceptable Cabibbo angle are discussed, and the groups $\Delta(6\cdot9^2)$, $[648, 259]$, $[648, 260]$, $[648, 266]$ and $\Delta(6\cdot14^2)$ are especially promising.