Fast scanning calorimetry (FSC) experiments were performed to investigate physical aging in amorphous and semi-crystalline poly(l-lactic acid)s (PLLAs) that were thermally crystallized under conditions leading to the α'- or α-crystalline form, and either favouring or inhibiting the development of a rigid amorphous fraction (RAF). The enthalpy of recovery was calculated after two procedures of rescaling to the content of the whole amorphous phase and also to the only content of the mobile amorphous fraction (MAF), which helped in clarifying the contribution of the RAF. From the dependence of the structural relaxation rate on the aging temperature, two regimes were evidenced for all samples. In the aging temperature domain situated close to the glass transition, the structural relaxation occurs significantly faster in the MAF. Its rate is independent of the aging temperature and is not influenced by the microstructure. However, the distance to equilibrium is higher in samples for which the coupling is strong between crystal and amorphous, implying that the time to reach equilibrium is also higher. In contrast, at low aging temperatures, for which the whole amorphous phase can be considered as solid, MAF and RAF exhibit the same structrural relaxation rate. This convergence in the relaxation kinetics by decreasing the temperature of physical aging was interpreted as the evolution of relaxation dynamics in the MAF from segmental to local. This change is highlighted by the comparison between MAF and RAF relaxation kinetics, but it occurs similarly in a pure amorphous system.