The origin of “annealing peaks” in semicrystalline polymers, obtained with aging thermal protocols at temperatures between the glass transition and the melting temperature, and revealed by calorimetry as small endotherms preceding the main melting peak, has been recently debated in the literature. On the one side it is claimed that these small endotherms are related to the melting of defective/thinner crystals, on the other side it is proposed that they can be associated to the enthalpy relaxation of the rigid amorphous fraction (RAF), i.e., that part of the amorphous region directly coupled to the crystalline stems. With this work we aim to contribute to the debate, by exploiting the different thermal properties of the RAF coupled with different crystal modifications of polymorphic poly (butene-1) (PB-1). In this polymer the RAF mobilizes at largely different temperatures when it is coupled with crystals of trigonal Form I, or tetragonal Form II. By exploiting this peculiarity, we performed isochronous aging experiments with differential scanning calorimetry (DSC) on both crystalline phases, in a wide temperature range between the glass transition of the mobile (bulk) amorphous fraction and the onset of crystal melting. An endothermic peak above the aging temperature was typically observed. The trend of the enthalpy of this annealing peak with temperature can be described by a bell-shaped curve, approaching zero recovered enthalpy at temperatures of 100–110 °C, and 40–50 °C for Form I and Form II, respectively. These temperatures coincide with previous literature report about mobilization of PB-1 RAF and are thus identified as the upper limit of the RAF glass transition for the two polymorphs. Isothermal aging experiments as a function of time suggest that the enthalpy recovery of Form I RAF show the typical signature of glassy dynamics, similarly to what occurs for Form II RAF after short aging times. At longer times, the superposition of the polymorphic phase transition and RAF aging changes the kinetics of the recovered enthalpy for the latter polymorph, possibly due to the increased stress at the crystal-amorphous interface and creation of additional RAF in parallel with Form II to Form I transformation. Overall, our results demonstrate that for PB-1, at least within the investigated temperature range, the annealing peaks can be related to the RAF strive to attain thermodynamic equilibrium in its glassy state.