Continuous dynamic recrystallization (cDRX) is an effective way to refine the microstructures of materials with high stacking fault energies during their hot working. However, investigation of the influence of grain boundaries and second phases on the microstructure evolutions of these materials using conventional cDRX models is challenging. Accordingly, herein a cDRX model characterized with micron-scale second phases distributed along the grain boundaries was developed by an internal state variable plasticity-based approach. Geometrically necessary dislocation (GND) densities generated near the second phase and grain boundary were calculated by considering strain concentration in the grain, which affected the evolution of dislocation density at the subgrain boundary. The cDRX model considers an original grain as a collection of subgrains and represents the subgrains as a compound structure consisting of subgrain boundaries with relatively high dislocation densities and a subgrain interior with significantly low dislocation density. Changes in the misorientations of subgrains can be accurately determined by tracking and analyzing the evolutions of the dislocation densities of subgrain boundaries. Accuracy of the developed cDRX model was verified by comparing the hot compression data of 5052 and 2A14 Al alloys acquired using this model with the corresponding experimental results. The predicted values of variables, including flow stress, subgrain size, and the fraction of low-angle grain boundary, were in appropriate agreement with the experimental results. Effects of temperature and strain rate on the microstructure evolution of the 2A14 Al alloy were investigated. Based on the influences of strain concentration near the grain boundary and second phase on the dislocation density of the 2A14 Al alloy, the reasons for the enhancement of the occurrence of cDRX and improvement of the mechanical properties of the 2A14 Al alloy by the second phase and grain boundary are explained. The results of this study provide a direction for the subsequent improvement of the proposed cDRX model with second phases inside the grain.