This study investigated the surface-enhanced Raman spectroscopy (SERS) properties and nonlinear optical (NLO) behavior of silver nanoparticles (Ag NPs) decorated hexagonal boron nitride (hBN) nanocomposites. Although Ag NPs are known for their excellent plasmonic properties, their susceptibility to oxidation in ambient conditions significantly reduces SERS activity. To overcome this, a femtosecond laser-assisted single-step novel approach is developed for synthesizing hybrid Ag-hBN nanocomposites via ablating Ag target in a solution of exfoliated hBN nanosheets, forming uniformly embedded Ag NPs on hBN nanosheets. The morphological characteristics and elemental compositions were validated by field emission scanning electron microscopy (FESEM), transmission electron microscopy (TEM), and energy dispersive X-ray spectroscopy (EDS). TEM analysis revealed that Ag NPs averaged 15 nm in size, while hBN nanosheets had lateral dimensions of approximately 250 nm. The SERS activity of the prepared nanocomposites was investigated using dye and explosive molecules, achieving detection sensitivities of 1 μM and 100 μM, respectively, with high reproducibility. These detection sensitivities are significant as they demonstrate the potential of the Ag-hBN nanocomposites for sensitive and reliable detection of trace amounts of analytes. Notably, the Ag-hBN nanocomposites showed enhanced SERS activity compared to pure Ag NPs, with a 2.5-fold improvement for Nile blue, 3.6-fold for methylene blue, and 2.4-fold for RDX. The current study also demonstrates that hBN prevents Ag NPs’ oxidation and preserves the SERS functionality even at elevated temperatures. Furthermore, the NLO properties of the nanocomposites were investigated using the standard Z-scan technique, revealing two-photon and three-photon absorption coefficients of 3×10−4 cm/GW and 6.5×10−5 cm3/GW2, respectively. The findings highlight the potential of Ag-hBN nanocomposites to enhance SERS and NLO devices, enabling future applications in sensing and photonics.