Hydrogen is one of the most detrimental elements in steel, therefore it must be strictly controlled, especially in the ferritic steel SA508-3 used in nuclear power plant, due to its persistent exposure to irradiated environment. However, the tests of hydrogen concentration in current industry have provided usually unstable and sparse results, which seriously perplexes practitioners. In the present study, tested hydrogen contents of a series of specimens with different machining and cleaning conditions were compared. Experimental and theoretical investigations showed that differences in surface contamination can significantly influence hydrogen analysis result, and the measured hydrogen content is the combination of the inherent hydrogen in the steel and the adsorbed hydrogen on the surface. Time-of-Flight Secondary Ion Mass Spectrometry (TOF-SIM) was then used to determine the hydrogen concentration gradient from the surface to the base of the SA508-3 steel, with the aim of confirming the existence of surface hydrogen. The result demonstrated that, if the specimen surface is in a rough and polluted state, the count intensity of hydrogen on the specimen surface, which qualitatively reflects the content of surface hydrogen can be more than 4 times that in the base of the specimen. Finally, on the basis of the analysis of tested results, a pre-treatment method combining a certain machining speed and ultrasonic cleaning is proposed in order to obtain a stable and reliable hydrogen analysis result. Furthermore, a mass of practical industry data is used to confirm the effectiveness of this pre-treatment method.