Clenbuterol and ractopamine can be used to improve carcass leanness in swine, but their residues in pork also pose health risks. This study used surface-enhanced Raman scatter (SERS) technology to establish MLR and PLSR prediction models for clenbuterol and ractopamine, respectively, to achieve rapid detection and identification of ractopamine (RAC) and clenbuterol hydrochloride (CL) residues. In this report, the detection of clenbuterol and ractopamine was carried out using three gold colloids with different particle sizes as enhanced substrates. It was found that the larger particle size of gold nanoparticles was, the better enhancement effect was, which indicated that gold nanoparticles with larger particle sizes were more conducive to the adsorption of RAC and CL. Then, the SERS spectra of 0.1–10 µg/mL clenbuterol and ractopamine solution were collected by using NaCl aqueous solution of 1 mol/L as aggregating compound and gold colloid A as enhanced substrate. Automatic Whittaker Filter (AWF) was used for preprocessing to reduce fluorescence background contained in the original Raman spectra collected. Finally, models were established between SERS intensity at each characteristic peak and the concentration of clenbuterol and ractopamine, respectively. The results show that the PLSR model for clenbuterol using six peaks was stable. The correlation coefficient R2 and the root mean square error of correction set were 0.99 and 0.025 µg/mL, respectively. The correlation coefficient R2 and the root mean square error of prediction set were 0.99 and 0.026 µg/mL, respectively. The MLR model for ractopamine using six peaks had a higher correlation coefficient and a lower root mean square error (RMSEC) than other models. The correlation coefficient R2 and the root mean square error of correction set were 0.99 and 0.043 µg/mL, respectively. The correlation coefficient R2 and the root mean square error of prediction set were 0.99 and 0.05 µg/mL, respectively. The detection limit (LOD) value of this method for the detection of clenbuterol and ractopamine was 18 ng/mL and 24 ng/mL, respectively. This study is helpful for further development of a simple SERS detection method of clenbuterol and ractopamine based on gold colloid.