Achieving ultra-smooth surfaces is the goal of aluminum optical manufacturing. Under certain processing conditions, improving the microstructure of aluminum and understanding its relationship with surface roughness requires systematic study. The grain structure and various types of second-phase particles are of paramount importance. This study analyzed the microstructure of 6061 alloy after undergoing severe plastic deformation under various processing conditions followed by T6 homogenization heat treatment. Utilizing a white light interferometer, a comparative analysis of the surface roughness was conducted on specimens that underwent single-point diamond turning to achieve a mirror finish. The assessment of surface roughness on machined surfaces is solely based on white light interferometry. The analysis and discussion focus on the effects of phases (causing scratches and voids), the grains and grain boundaries. Experimental findings signify: the grain size, grain boundary and residual second phase can both influence the surface quality, the increase in deformation temperature and accumulated strain both facilitate the dissolution and fragmentation of the secondary phases. However, they also contribute to some extent to grain growth, resulting in a minimum secondary phase area fraction of 0.87% and grain sizes reaching 147.8 μm. Subsequent heat treatments, while effective in reducing the negative impact of the phases, reveal noticeable step-like structures affecting the quality of surface roughness, with the lowest obtained Ra value being 0.8 nm. A proposed pretreatment method in cleaner ingot processing with lower alloy element content addresses the trade-off between reducing phases and controlling grain growth, aiming to achieve improved surface roughness, promoting the application of polycrystalline aluminum alloys in the field of optics manufacturing.