Abstract To precisely predict the forming instability of a forming process of thin-walled parts is a challenging problem. In this work, a study has been conducted on modeling of the forming instability in the in-plane roll-bending process of strip (IRS) with a 3A21O aluminum alloy. During the modeling, eigenvalue buckling analysis is firstly carried out through a mechanical model with three sides simply supported and one sider free. Then, the equivalent geometric micro-defects are defined with the superposition of multiple bulking modes, which is then implanted into the FE model of the IRS process. Next, the sensitivity analysis of the equivalent geometric micro-defects is performed by considering single and the superposition of two and three bulking modes in the IRS process, consequently, the most sensitive geometrical micro-defect was determined as W = 0.3 ⊍ mode 7 + 0.3 ⊍ mode 8. Then, the 3D finite element model of the IRS process with multi-modal geometric micro-defects being considered has been established, so that the influence of multi-factors on the forming stability can be captured. With this model, four types of forming defects, such as wrinkling at outer rim, inner rim and both rims, and spatial turning as well as the stably formed ring were predicted, which are in agreement with the experimental results in terms of defects mode and size.