A mirror-like reflecting surface is an important characteristic in many industrial metallic parts. Polishing is done to form a mirror surface on metals. However, the effect of the grain size of metals on surface roughness through polishing processes is not clear. Specifically, mirror surface formation of ultrafine grained materials is still unknown. Ultrafine grained steels and coarse grained steels with 0.02, 0.10, and 0.60 wt% carbon contents were prepared by warm caliber rolling and annealing. Average grain sizes were 1–2 μm and 4–40 μm. The changes in surface roughness, Sa, were measured with an atomic force microscope (AFM) via eight polishing steps, using emery papers of type #600, #1000, #1500, #2000, #2500, #4000, and free abrasive grains of 3 μm and 1 μm diamond. As the polishing process progressed, the surface unevenness was removed and the surface roughness, Sa, decreased in all steels. The differences of Sa at each polishing step were analyzed from the point of carbon content, Vickers hardness, and grain size. Carbon contents and Vickers hardness have little effect on Sa. However, grain size has a considerable effect on Sa in all steels. Ultrafine grained steels have smaller Sa in all polishing steps in all steels. This is because ultrafine grained steels have very small work hardening rate. After final polishing, Sa is 2.5–3.6 nm in coarse grained steels and 2.0–2.6 nm in ultrafine grained steels. To obtain a mirror surface with smaller Sa, grain size control is important.