Coal and gas outburst is a violent mechanical instability of gas-containing coal, and its mechanism is crucial for prevention of gas dynamic disasters. The macroscopic instability of gas-containing coal is the result of the expansion and connection of micro cracks driven by unloading induced tensile stress and gas pressure. This study is aimed at solving the problem of crack instability of gas-containing coal during mining unloading. To this end, first, the governing equations of unloading-induced tensile stress and gas pressure evolutions in cracks were built, and the dynamic evolutions and main controlling factors of tensile stress and gas pressure were revealed. Furthermore, the evaluation indexes of crack stability under tensile stress and shear stress were established, and the variation of crack stability during unloading was investigated. Finally, the phase diagram of crack failure mode was drawn, and the transformation of crack failure mode was analyzed. The results show that unloading-induced tensile stress is generated when θ > 54.73°, and it rises gradually with the increase of original stress, unloading rate, unloading time and crack size and dip angle. Overall, gas pressure in cracks increases first and then decreases during unloading. It increases gradually with the increase of initial gas pressure, gas desorption rate and original stress, while it decreases as the unloading rate gets higher (k from 10−4 to 10−2 s−1) and the dip angle gets larger (θ from 50 to 90°). With the increase of unloading ratio, the crack stability declines slowly first and then plunges rapidly until the crack finally becomes unstable. In addition, burial depth of coal seam, unloading rate, gas desorption rate and mechanical strength of coal have significant effects on this process. Different factors have important influences on the crack failure mode. With the increase of burial depth from 100 to 2000 m, the crack failure mode gradually transforms from tensile failure to shear failure, and the higher the unloading rate is, the more easily tensile failure occurs. With the increase of cohesion (0.1–2 MPa), the scope of the shear failure zone shrinks gradually, and the crack is less prone to shear failure. The desorption rate has a notable effect on the scope of the non-failure zone. Under a higher gas desorption rate, the scope of the non-failure zone becomes smaller, which is due to the greater gas pressure in crack for a given time. The research in this paper has certain reference value for further grasping of the instability failure of gas-containing coal and the mechanism of coal and gas outburst.