The in-plane bending of a strip metal workpiece under unequal compression can be developed into an advanced precision forming process with high quality, high efficiency, low consumption, and good flexibility for size changing. Moreover, the loading path and stress state of the process make it possible to develop the deformation potentiality of the workpiece to the full and to greatly improve the bending forming limit of the workpiece, which is difficult to achieve by any other conventional process. The forming limit of the workpiece depends on the modes and the conditions of instability of the process. However, up to now, literature on the instability has been scant. This paper reports research into the mechanism, the modes, and the conditions of instability of the process by a comprehensive analysis based on the minimum energy principle of plastic deformation and by an experiment with LF21M strip workpieces on apparatus made by the authors for performing in-plane bending with two conical rollers. The research results show the following: (1) Because the in-plane bending process has a specific three-dimensional compressive stress state and the resistance to instability is determined by the geometry parameters and the property values of the workpiece, there are three new modes of instability of the process, different from those of conventional bending processes. (2) For a given material the original thickness t 0, original width b 0, unequal compression width a z, and the smallest thickness of the formed workpiece t 1, there are the instability parameters of K 1, K 2, K 3, m 1, and m 2, and they satisfy the conditions of 0 < K 1 < K 2 < K 3 and 0 < m 1 < m 2. It is possible for the workpiece to produce external wrinkling when a z/ b 0 ≤ K 1 or t 1/ t 0 ≤ m 1, the turning when K 1 < a z/ b 0 ≤ K 2 and m 1 < t 1/ t 0 ≤ m 2, and the internal wrinkling when K 2 < a z/ b 0 ≤ K 3 and m 1 < t 1/ t 0 ≤ m 2. When a z/ b 0 > K 3 and m 1 < t 1/ t 0 ≤ m 2, the workpiece can perform a stable process of in-plane bending and greatly improve the bending forming limit without any mode of instability. The approximate values of K 1, K 2, K 3, m 1, and m 2 for strip workpieces LF21M alloy (sheet thickness: 1.90 and 1.50 mm) are 0.25, 0.45, 0.60, 0.30, and 0.95, respectively. (3) The greater are the elastic modulus E and the strain-hardening modulus D of the material used, the smaller are the instability values of K 1, K 2, K 3, and m 1, and the greater is the value of m 2. The achievements of this research can serve as a significant guide to the determination of the forming limit and the optimal process parameters for the practice of the relevant processes.