Surface enhanced Raman scattering (SERS), since its discovery in the mid-1970s, has taken on many roles in the world of analytical measurement science. From identifying known and unknown chemicals in mixtures such as pharmaceutical and environmental samples to enabling qualitative and quantitative analysis of biomolecules and biomedical disease markers (or biomarkers), furthermore expanding to tracking nanostructures in vivo for medical diagnosis and therapy. This is because SERS combines the inherent power of Raman scattering capable of molecular species identification, topped with tremendous amplification in the Raman signal intensity when the molecule of interest is positioned near plasmonic nanostructures. The higher the SERS signal amplification, the lower the limit of detection (LOD) that could be achieved for the above applications. Therefore, improving SERS sensing efficiencies is vital. The signal reproducibility and SERS enhancement factor (EF) heavily rely on plasmonic nanostructure design, which has led to tremendous work in the field. But SERS signal and EF reproducibility remain key limitations for its wider market usability. This Review will scrutinize factors, some recognized and some often overlooked, that dictate the SERS signal and are of utmost importance to enable reproducible SERS EFs. Most of the factors pertain to colloidal labeled SERS. Some critically reviewed factors include the nanostructure's surface area as a limiting factor, SERS hot-spots including optimizing the SERS EF within the hot-spot volume and positioning labels, properties of label molecules governing molecule orientation in hot-spots, and resonance effects. A better understanding of these factors will enable improved optimization and control of the experimental SERS, enabling extremely sensitive LODs without overestimating the SERS EFs. These are crucial steps toward identification and reproducible quantification in SERS sensing.