The addition of inorganic fillers has been reported to increase the toughness of poly(l-lactide) (PLLA), but the effect of physical aging in such composites has been neglected. The present work discusses the effect of the still ongoing segmental relaxation in PLLA-based composites filled with BaSO4 inorganic particles in regard of the filler quantity. By means of differential scanning calorimetry, X-ray diffraction, and tensile testing of progressively aged PLLA filled with particles ranging from 0.5-10 wt %, we observed an increase in the mechanical energy required to activate the plastic flow of the primary structure in the PLLA matrix, which resulted in the embrittlement of the majority of composites upon enough aging. Results further clarify the role of debonding in the activation process of PLLA, and the behavior of the composite is described at the segmental level. Only an addition of 10% of particles has effectively preserved a ductile behavior of the samples beyond 150 aging days; therefore, we strongly remark the significance of studying the effect of physical aging in such composites.