The paper presents a new analytical method based on the strain energy density to investigate the mechanism of wellbore instability. To establish the method, pre- and post-drilling states of strain energy densities were first formulated, and then have been applied to the wellbore wall using the strain energy balance. This innovative approach differentiates the instability prone zones into two distinct parts; the compressive, and the tensile zones that also known as borehole breakout and drilling-induced fracture, respectively. These two types of instabilities are related to the restored strain energy density (RESED) and the released strain energy density (RLSED), accordingly. Among the two types of instabilities, the geometrical characteristics of the compressive instable prone zone, that is, width, 2 θ b , and length, L b , were further formulated, examined and proposed. The results of the presently developed method have been compared with those that obtained by using the existing borehole breakout’s geometry. In each case, good agreement was found between the results of this study and the conventional methods. This method has further been compared and verified by using the experimental data. Finally, it has been demonstrated that how this method helps to determine the optimal mud pressure for a safe wellbore design.