Phase change materials (PCMs) have been proposed as a means of building energy conservation due to a large amount of heat being absorbed or released during phase-transition. Recently, the application of the encapsulated PCMs (MPCMs) in cement-based materials is drawing more and more attention for the improvement of the thermal performance of building materials. However, the thermal effect of MPCMs with inorganic silica shells on cement-based materials is rarely focused on. In this study, two types of MPCMs with silica shells respectively at nanometer and micron scale were fabricated, and the effect of MPCMs on the thermal performances of cement-based materials was systematically evaluated. Results showed that the prepared MPCMs increased the enthalpy of phase change but decreased the thermal conductivity of cementitious composites. The heat storage and release characteristics of cement pastes were noted to be strongly dependent on the volume fraction of MPCMs, while small size MPCM exhibited better thermal stability and the effect of MPCMs was strongly correlated with the temperature ramp. The prepared MPCMs significantly improved the thermal inertia of cement pastes and can be used as a thermal storage material in buildings to mitigate the indoor temperature fluctuations. To achieve better building energy conservation, MPCM with higher enthalpy is recommended.