This study is aimed at solving the issue of mining under the boundary coal pillar of the close‐distance coal seam that causes roof falling. This study established a new key bearing structure model for analyzing the structural instability mechanism when mining under the coal pillar at the working face by taking Shaping Coal Mine as an example. The purpose of this study is to analyze the formation process, load transfer mechanism, and two failure types of the key bearing structure using theoretical analysis and numerical simulation. Additionally, this study discussed the timing and method of roof control for different key bearing structure failure types. Research shows that the instability of the key bearing structure composed of the coal pillar, interlayer rock, and lower coal body is the important reason behind the roof falling. The instability types of key bearing structures include the coal pillar instability type and the cantilever beam instability type. The stability width of the interlayer rock cantilever beam and the coal pillar jointly ascertain the failure type of the key bearing structure. In the 9204 working face, the key bearing structure was destroyed when the coal pillar was 14 m wide, resulting in the roof stress being as high as 31.81 MPa. The stress drop phenomenon can be used as a boundary to divide the failure process of the key bearing structure into three stages. The pressure relief of the coal pillar and interlayer rock cantilever beam is an effective way to deal with this problem, and the coal pillar instability type needs to be pressure relieved earlier than the cantilever beam instability type. The research findings offer new insights into the roof stability control of mining under the coal pillar.