Glyphosate, also known as N-(phosphonomethyl)glycine, is one of the most widely used herbicides in the world. However, the controversy surrounding the toxicity of glyphosate and its main breakdown product, aminomethylphosphonic acid (AMPA), remains a serious public concern. Therefore, there is a clear need to develop a rapid, sensitive and automated alternative method for the quantification of glyphosate and AMPA. In this context, surface enhanced Raman spectroscopy (SERS) coupled with a microfluidic system for the determination of glyphosate in tap water was developed, optimized and validated. The design of the microfluidic configuration for this application was built constructed to integrate the synthesis of the SERS substrate through to the detection of the analyte. To optimize the microfluidic setup, a design of experiments approach was used to maximize the SERS signal of glyphosate. Subsequently, an approach based on the European guideline document SANTE/11312/2021 was used to validate the method in the range of 78–480 μg/L using the normalized band intensities. The limit of detection and quantification obtained for glyphosate were 40 and 78 μg/L, respectively. Recoveries were in the range 76–117%, while repeatability and intra-day reproducibility were ≤17%. Finally, the method was also tested for the determination of AMPA in tap water matrix and for the simultaneous detection of AMPA and glyphosate.