We have observed unusual magnetic properties of NiO (nickel oxide) nanoparticles embedded in a silica matrix. The sample was synthesized by a method based on the contribution of sol–gel and combustion processes. X-ray powder diffraction (XRPD) of the sample shows the formation of the nanocrystalline NiO phase whereas transmission electron microscope (TEM) reveals spherical-shaped nanoparticles of about 4 nm diameter. Moreover, HRTEM images show lattice fringes of the nanoparticles and defects in the crystal structure. The temperature and field dependence of the magnetization are also measured. The zero-field-cooled (ZFC) measurements show two maximums, one sharp and narrow at low temperatures ∼6.5 K and an other broad one at higher temperature ∼64 K. The FC magnetization shows a continuous increase upon lowering the temperature. The M(H) measurements reveal that NiO nanoparticles display anomalous hysteretic behaviors at low temperatures (below the low temperature maximum in the ZFC curve, 2 K and 5 K) showing that the magnetization initial curve lies below the hysteresis loop for a certain field range. Moreover, jump of the magnetization at low temperatures (2 K and 5 K) are also observed. These features represent novel magnetic properties for nanosized NiO which may be attributed to the surface spins. Moreover, these results indicate that the NiO nanoparticle consists of magnetically disorder shell and antiferromagnetically order core with an uncompensated magnetic moment.