Surface-enhanced Raman scattering (SERS) is widely used for trace detection of substances, and the key to this technology lies in the preparation of the substrate material. In this study, a composite SERS material of Au NRs/4-MBA/PAM was constructed and characterized to better immobilize the reference molecule 4-mercaptobenzoic acid (4-MBA). Electron transmission microscopy results demonstrated that the PAM film helps Au NRs to pack closely, enhancing the stability of the material structure and reducing the interference of external environmental factors on the response of 4-MBA, thus improving the accuracy of quantitative determination. Comparative experimental results with the Au NRs/4-MBA substrate showed that the relative standard deviations (RSDs) of the detection results for MG on different batches of Au NRs/4-MBA/PAM were less than 8.0%, and the RSDs of different points on the same material were less than 10.0%, indicating that the Au NRs/4-MBA/PAM has higher uniformity, better reproducibility, and higher sensitivity in detecting malachite green (MG). Applying this material in the recovery determination of fish extract showed that the recovery rates of MG were between 75.60% and 83.24%. Therefore, the Au NRs/4-MBA/PAM substrate can accurately detect and quantify veterinary drug residue in complex matrices such as food tissue.