Among the magnetocaloric materials featuring first-order phase transitions (FOPT), FeRh is considered as a reference system to study the FOPT because it is a "simple" binary system with a CsCl structure exhibiting a large adiabatic temperature change. Recently, ab initio theory predicted that changes in the Fe/Rh stoichiometry in the vicinity of equiatomic composition strongly influence the FOPT characteristics. However, this theoretical prediction was not clearly verified experimentally. Here, we investigated the composition dependence of the transitional hysteresis in FeRh. It is shown that a Fe excess of only 1 at. % induces a ferromagnetic state in the whole temperature range (from 5 K up to Tc) for a minor portion of the sample (≈10%), while 5 at. % is enough to completely eliminate the FOPT. Element-specific X-ray magnetic circular dichroism (XMCD) measurements suggest that this ferromagnetic contribution arises from residual FeRh ferromagnetic regions. We attribute the formation of such domains to Fe antisite defects, as Mössbauer spectroscopy demonstrates the presence of Fe atoms located at the 1b (Rh) sites in the CsCl-type structure. As a consequence, compared with the equiatomic composition, the slightly Fe-rich sample exhibits completely different FOPT properties, influencing the magnetocaloric performances. Thus, our study sheds light on the origin of the remarkable stoichiometric sensitivity of the FOPT behavior in FeRh. These insights have broader implications for understanding FOPT dynamics and the role of residual ferromagnetic domains.