It is essential to monitor the blood level of flutamide (FLT) during prostate cancer treatment since the abnormal renal elimination due to pathophysiological factors hinders attaining steady-state drug concentration. This study reports a systematic investigation of aggregation of silver nanoparticles (AgNPs) based on Hofmeister effect and eventual change in surface-enhanced Raman spectroscopy (SERS) enhancement for the determination of FLT. Among the aggregants, MgSO4 showed the maximum enhancement. The concentration of MgSO4, pH, and duration of aggregation were optimized for FLT detection. The aggregated AgNPs (a-AgNPs) SERS substrate enhanced the Raman signal of FLT by up to five-orders of magnitude under optimized conditions. The developed technique was able to determine FLT from 0.25 × 10−3 μM to 250 μM with a limit of detection (LoD) of 1 pM. The application of the SERS substrate in practical analysis has been demonstrated using FLT-spiked human serum from 0 nM to 1000 nM with a detection limit of 0.3 nM. The reported Hofmeister series of AgNPs aggregation will assist in the development of SERS substrates for other analytes.