The presence of a weak remanence in Ultra-Low-Field (ULF) NMR sample containers is investigated on the basis of proton precession. The high-sensitivity magnetometer used for the NMR detection, enables simultaneously the measurement of the static field produced in the sample proximity by ferromagnetic contaminants. The presence of the latter is studied by high resolution chemical analyses of the surface, based on X-ray fluorescence spectroscopy and secondary ions mass spectroscopy. Methodologies to reduce the contamination are explored and characterized. This study is of relevance in any ULF-NMR experiment, as in the ULF regime spurious ferromagnetism becomes easily a dominant cause of artefacts.