The correlation between the dynamic relaxation behavior and micro-inhomogeneous structure of amorphous alloy is of significance for exploring the mechanical deformation. In this work, we report the pronounced secondary (β) relaxation process in FeP(B)CCu amorphous alloys with different microstructures, which reveals the local heterogeneity at atomic scale. With the help of thermodynamic analysis, it was found that compared to the partially crystallized alloy, the more local structural heterogeneities were activated in the amorphous matrix due to the low activation energy of β relaxation and the high activation energy of crystallization. Physical aging far below the crystallization temperature demonstrated that atomic rearrangement supported the weakening of β relaxation amplitude as well as the increment of storage modulus, especially for FePBCCu amorphous alloy, which was consistent well with the increases of the reduced modulus and hardness by nanoindentation tests. Moreover, the fitted results from the Maxwell-Voigt model on creep curves revealed that the diffusion and rearrangement of atoms in loosely packed zones corresponding to the β relaxation were more active during aging. Our findings provide new insight into understanding the dynamic relaxation behavior in FePBCCu amorphous alloy and render it an ideal system to probe the correlation between relaxation and mechanical deformation.