The iron and steel industry with high energy consumption and high emissions is one of the main industries of steel slag and CO2 emissions, carbonation technology is a promising way to improve the utilization rate of steel slag and reduce CO2 emission since steel slag contains a large number of substances with strong carbonation activity, such as Ca(OH)2, C3S, C2S, etc. In the study, the effects of fly ash on the pore structure improvement before and after carbonation and the formation of products (C-S-H and CaCO3) that play a major role in the strength development were analyzed in steel slag samples. The results of SEM and MIP indicated that CaCO3 produced after carbonation can fill most of the pores (>15–20 nm) in the carbonation zone, and the filling of the pores on the surface of steel slag samples hinders the carbonation of the internal non-carbonation zone. However, the addition of fly ash can prevent a part of the pores (>400–500 nm) in the steel slag samples from being blocked during carbonation, which provides a channel for the CO2 diffusion into the internal non-carbonation zone,and accelerates the carbonation depth of steel slag samples and the CO2 uptake. Moreover, XRD, FTIR, XPS analysis suggested that the addition of fly ash could significantly increase the formation of CaCO3, and promote the conversion of C-S-H gel with high Ca/Si ratio to C-S-H gel with low Ca/Si ratio in the carbonation zone.