Residual pesticides such as phosmet in fruits and vegetables are one of the major food safety concerns for consumers around the world. Surface-enhanced Raman spectroscopy (SERS) with gold-coated substrates was exploited to analyze phosmet (0.5–10 μg/g) in apples. Phosmet could be detected from its SESR spectra at as low as 1 μg/mL in standard solution and 1 μg/g in apple extracts. The calculated limit of detection for phosmet in standard solution and in apple extracts were 1.01 μg/mL and 1.44 μg/g, respectively. There were linear relationships between the concentrations of phosmet solutions and the Raman intensities of four major characteristic peaks, with the coefficient of determination (R2) ranging from 0.905 to 0.984, the root mean square error of prediction (RMSEP) ranging from 0.46 to 1.16 μg/mL, and the ratio performance deviation (RPD) from 3.22 to 8.21. Although the linear regression results for phosmet in apple extracts with Raman intensities were not as good as those for the standard solutions (R2 = 0.866–0.945, RMSEP = 0.84–1.38 μg/g, RPD = 2.65–4.33, which still showed great potential of using SERS for quantitative analysis. The presence of non-targeted compounds in apple extracts affected the sensitivity for qualitative analysis and accuracy for quantitative analysis. This study demonstrated that SERS coupled with gold nanosubstrates could be used for analysis of trace contaminants like phosmet in complex food matrices.