In this paper, a novel compact operator is derived for the approximation of the Riesz derivative with order $\alpha\in(1,2].$ The compact operator is proved with fourth-order accuracy. Combining the compact operator in space discretization, a linearized difference scheme is proposed for a two-dimensional nonlinear space fractional Schrödinger equation. It is proved that the difference scheme is uniquely solvable, stable, and convergent with order $O(\tau^2+h^4)$, where $\tau$ is the time step size, $h=\max\{h_1,h_2\}$, and $h_1,\,h_2$ are space grid sizes in the $x$ direction and the $y$ direction, respectively. Based on the linearized difference scheme, a compact alternating direction implicit scheme is presented and analyzed. Numerical results demonstrate that the compact operator does not bring in extra computational cost but improves the accuracy of the scheme greatly.