Reactive sputtering deposition has been widely applied to the preparation of oxide films, but a method is still needed to stably control the deposition process and the film growth. In this work, the plasma in the vicinity of sputtering target is studied by optical emission spectroscopy and the atomic oxygen density ([O]) is linked to the sputtering mode and the growth of NiO films. The NiO films can be deposited either in oxide mode or in the transition region, whereas the [O] value plays an important role in determining the growth behavior of NiO films and the crystallinity. In the transition region, the (111)-oriented growth of NiO films is preferential and the crytallinity is improved with the decrease in the [O] value. In the oxide mode, the NiO films are preferentially (001)-oriented and exhibit a relatively good crystallinity. By fixing the O2/Ar flow ratio at a constant, the [O] value is explored as a function of substrate temperature, showing that the oxygen desorption may have significant impacts on the [O] value as the substrate temperature is higher than 400 °C.