In this paper, we study the Cauchy problem for the critical inhomogeneous nonlinear Schrödinger (INLS) equation$ iu_{t} +\Delta u = |x|^{-b} f(u), \; u(0) = u_{0} \in H^{s} (\mathbb R^{n} ), $where $ n\ge3 $, $ 1< s<\frac{n}{2} $, $ 0<b<2 $ and $ f(u) $ is a nonlinear function that behaves like $ \lambda |u|^{\sigma } u $ with $ \lambda \in \mathbb C $ and $ \sigma = \frac{4-2b}{n-2s} $. We establish the local well-posedness as well as the small data global well-posedness and scattering in $ H^{s} (\mathbb R^{n} ) $ with $ 1<s<\frac{n}{2} $ for the critical INLS equation under some assumption on $ b $. To this end, we first establish various nonlinear estimates by using fractional Hardy inequality and then use the contraction mapping principle based on Strichartz estimates.