The constitutive modelling of semi-crystalline polymers (SCP) has to consider several aspects as rate- and temperature-dependence, self-heating, and in particular, the double yield (DY) phenomenon. A full characterization of all these complex features involves prominent efforts in terms of material testing and parameter identification (PI). The contribution of the crystalline phase plays an important role in the evolution of the plastic yield in the SCPs. In this work, a constitutive model, named Unified SCP (USCP), is proposed by modifying the physically-based Boyce–Parks–Argon (BPA) glassy model. The contribution of the crystalline phase is introduced in the strain softening/hardening evolution law of the strength, providing an alternative interpretation of the underlying morphological changes caused by the crystalline phase embedded in the amorphous phase. The proposed formulation extends the BPA model with a new contribution to capture the crystalline phase. A full thermo-mechanical coupled numerical framework is developed for the USCP model validation. The DY phenomenon at different strain rates with self-heating and thermal softening effects is investigated and predicted. The proposed model extension needs three material constants with clear physical meaning. To identify them, a fast in-house optimization process based on Nelder–Mead is used, in which only a single element test is required. The model accurately predicts the experimental results for both thermosets and thermoplastics such as epoxy, nylon 101, PA6 and LDPE under monotonic loadings reported by different authors.