Nanoindentation and photoluminescence (PL) studies were performed on hydrogenated boron carbon nitride thin films deposited using radio frequency magnetron sputtering. Dual target sputtering from B4C and BN targets was used to deposit films. The variation in the composition of films was studied using energy-dispersive X-ray spectroscopy. The influence of hydrogen gas and substrate temperature on the mechanical properties was investigated using nanoindentation measurements. Photoluminescence studies were performed on films deposited under varying hydrogen content and different deposition temperatures. The films deposited in this study exhibited hardness of 6–22 GPa and Young’s modulus of 125–140 GPa. PL spectra demonstrated two prominent emission peaks around 499 nm and 602 nm for the deposited films. Increasing the hydrogen gas ratio in the films induced PL peak shifts to longer wavelengths. Emission spectra shifted to long wavelength with increasing substrate temperature. The emission peak position shifted from 499 nm to 544 nm and from 602 nm to 655 nm as a function of substrate temperature. For the first time, BCNH based thin films PL behavior at low temperature (77 K) has been characterized in this study. The BCNH thin films show a rare phenomenon of negative thermal quenching of emission.