In this work, a hexagonal boron nitride (h-BN)/AuAg nanoalloy hybrids (NAHs) was synthesized to fabricate h-BN/Au1Ag3 membrane as a solid surface-enhanced Raman spectroscopy (SERS) substrate for sensitive SERS detection of melamine. The AuAg nanoalloys were in situ grown on h-BN by chemical reduction method, and the Au/Ag molar ratio was tuned to achieve optimal SERS performance. After the SERS performance of h-BN/AuAg NAHs with different Au/Ag ratios was analyzed, h-BN/Au1Ag3 NAHs were chosen for SERS analysis. The h-BN/Au1Ag3 membrane can be obtained through simple filtration of h-BN/Au1Ag3 NAHs on chromatographic paper. As expected, the solid SERS substrates of h-BN/Au1Ag3 membrane were uniform and demonstrated good selectivity, repeatability and reproducibility for SERS detection of melamine. The results demonstrate that h-BN/Au1Ag3 membrane exhibited high SERS activity for 4-mercaptobenzoic acid (4-MBA) with limit of detection (LOD) at 1.0 ng L−1, and its analytical enhancement factor (AEF) reached 3.6 × 108. The possible enhancement mechanism, including electromagnetic mechanisms (EM) and chemical mechanisms (CM) were illustrated by finite-difference time-domain (FDTD) and density functional theory (DFT) simulations in detail, respectively. The concentration of melamine in the 0.05–5.0 mg L−1 range showed good linear relationship (R2 = 0.9940) with SERS intensity with LOD of 0.01 mg L−1. Finally, the recoveries of melamine in liquid milk samples are 87.7–105.7% with relative standard deviations (RSDs) in range of 0.6–2.6%, providing precise safety evaluation of melamine in milk samples.