Cylindrical roller bearings have been widely used in the mechanical industry due to the high radial load capacity. In some precision applications, high accuracy of cylindrical rollers, such as high profile accuracy and excellent surface quality, is strongly demanded and is even indispensable for the operational reliability of bearings. In this paper, on the basis of a lapping process, chemical mechanical polishing (CMP) technique was used to prepare cylindrical rollers with a low roundness and a nanoscale surface roughness Ra on a double-side cylindrical ultra-precision processing device, which was developed by the Ultra-precision Machining Center of Zhejiang University of Technology. Firstly, the effects of the experimental conditions, such as the rotating speeds and the down force, on the processing performance were studied. The simulation results reveal that, using the optimized rotating speeds, with the processing time going, the processing trajectory gradually distributes uniformly on the cylindrical surface of the roller along both the circular and the axial directions, and thereby, the uniform material removal can be realized; with the increase of the down force, the material removal rate (MRR) of the cylindrical roller almost linearly increases. Specially, the down force of the CMP process was set to 20 N/roller to obtain a better polishing performance. Then, on the basis of a lapping process, CMP technique was used to further improve the surface quality of the cylindrical rollers to nanoscale roughness. The results show that, after a two-step CMP process including a rough polishing step and a fine polishing step, the cylindrical surface of the rollers changes from rough and uneven state into smooth and free of micro-scratches state, and the surface roughness Ra decreases from initial 76 to 16.6 nm, and the roundness decreases from initial 0.97 to 0.40 μm. Finally, the material removal mechanism of the cylindrical rollers was investigated. During the lapping process, the material removal is predominantly driven by the mechanical abrasion from the α-alumina particles; while during the CMP process, the material removal is realized by the alternating cycles of “oxidation/complexation/passivation-abrasion-oxidation/complexation/passivation” process. In sequence, the mechanism of the roundness improvement was proposed. During the lapping and the CMP processes, the MRR at the protrudent point is larger than that at the recessed point due to the larger contact stress, and as a result, the actual surface profile of the circular section gradually reaches the average circle, and the roundness decreases.