In the current research, the silicon carbon oxynitride (SiCON) thin film was deposited on the silicon (Si) substrate by radio frequency (RF) reactive magnetron sputtering method. To comprehensively assess the impact of nitrogen flux rate on thin film characteristics, a suite of advanced analytical methods was utilized. The GIXRD analysis confirmed that the SiCON thin film is amorphous in structure. Additionally, Raman spectroscopy detected no graphite nanocrystals within the film. Ellipsometry measurements further showed that the refractive index of the thin films rises with increased nitrogen flux in the reactive gas, indicating a direct correlation between nitrogen concentration during deposition and optical properties. Based on the designs made with McLeod's software, the amount of reflection can be reduced up to 5.7 % at the wavelength of 4 μm and up to 8.5 % in the wavelength range of 3–5 μm for an optimal thickness thin film. The atomic force microscopy (AFM) examination revealed that the surface roughness of the thin films decreases as the nitrogen flux in the reactive gas mixture increases. Additionally, measurements of the water contact angle (WCA) indicated that the SiCON thin films exhibit a hydrophilic state.