This work is devoted to the study of the influence of controlled Si(111) surface nitridation on the epitaxial growth of AlN-on-Si nucleation layers with reduced tensile stress on ordered crystalline silicon nitride phase. The Si(111) surface nitridation process was performed at low ammonia flux and substrate temperatures in the range of 700–900 °C and was studied using RHEED and STM techniques. A universal criterion, namely the stage of the nitridation process completion is introduced, taking into account the influence of substrate temperature, ammonia flux and nitridation time. The 100 nm AlN nucleation layer on silicon substrates grown by ammonia molecular beam epitaxy is studied using AFM, XRD, HR-TEM and Raman spectroscopy techniques. The Raman data show that reducing the nitridation temperature from 900 °C to 700 °C not only deteriorates the crystalline quality of the subsequent AlN nucleation layers, but also reduces the residual tensile stress by almost 30 %. In the present contribution, micro-Raman spectroscopy is used to determine the nature of the defects formed during the high temperature growth of the AlN nucleation layers and confirms them to be inversion domains. The HR-TEM technique was used to study the AlN/Si interface in AlN-on-Si nucleation layers grown on a nitridated silicon surface at 700 °C and 900 °C at the optimum stage of the nitridation process completion. HR-TEM images of AlN nucleation layers revealed regions with different AlN/Si(111) interfaces: 1) AlN/amorph-Si3N4/Si, 2) AlN/SiN(8 × 8)/Si, and 3) AlN/Si with a sharp interface boundary. Using fast Fourier transform image analysis, it is shown that the presence of amorphous Si3N4 phase inclusions in the AlN/Si interface boundary introduces tensile stresses in the AlN nucleation layer which can be reduced by lowering the nitridation temperature. The results obtained clearly show that one of the causes of cracks in III-nitride layers grown on silicon substrates is the formation of tensile AlN layers with a high content of the amorphous Si3N4 phase at the AlN/Si interface, which is characteristic of silicon nitridation at elevated temperatures (> 700 °C).