We study matter-coupled $N=3$ gauged supergravity in four dimensions with various semisimple gauge groups. When coupled to $n$ vector multiplets, the gauged supergravity contains $3+n$ vector fields and $3n$ complex scalars parametrized by $SU(3,n)/SU(3)\times SU(n)\times U(1)$ coset manifold. Semisimple gauge groups take the form of $G_0\times H\subset SO(3,n)\subset SU(3,n)$ with $H$ being a compact subgroup of $SO(n+3-\textrm{dim}(G_0))$. The $G_0$ groups considered in this paper are of the form $SO(3)$, $SO(3,1)$, $SO(2,2)$, $SL(3,\mathbb{R})$ and $SO(2,1)\times SO(2,2)$. We find that $SO(3)\times SO(3)$, $SO(3,1)$ and $SL(3,\mathbb{R})$ gauge groups admit a maximally supersymmetric $AdS_4$ critical point. The $SO(2,1)\times SO(2,2)$ gauge group admits a supersymmetric Minkowski vacuum while the remaining gauge groups admit both half-supersymmetric domain wall vacua and $AdS_4$ vacua with completely broken supersymmetry. For the $SO(3)\times SO(3)$ gauge group, there exists another supersymmetric $N=3$ $AdS_4$ critical point with $SO(3)_{\textrm{diag}}$ symmetry. We explicitly give a detailed study of various holographic RG flows between $AdS_4$ critical points, flows to non-conformal theories and supersymmetric domain walls in each gauge group. The results provide gravity duals of $N=3$ Chern-Simons-Matter theories in three dimensions.