Abstract Water and sand inrush pose significant threats to underground geotechnical engineering, including shallow buried resource extraction and tunnel construction. To understand the mechanisms behind these phenomena, a mud collapse accident in Chaider Coal Mine was comprehensively investigated through field exploration and laboratory-based testings. Using numerical simulation experiments, we analyzed the failure patterns and seepage characteristics of overlying strata in steeply inclined coal seam mining under various working conditions. We established a structural instability model for water and sand inrush and identified the critical conditions for sand collapse occurrence. Our research indicates that the backfill in the surface mining pit provided a substantial material source for the accident. The overall destabilization of the top coal, due to its insufficient thickness and strength, created a pathway for sand collapse. Furthermore, frequent rainfall during the flood season and the inadequate arrangement of pumping equipment acted as triggers for the sudden water collapse. Preventative measures, such as limiting the mining height, enhancing the shear strength of the top coal, and altering the working face layout, can effectively control the development height of the water-conducting fracture zone. Additionally, timely evacuation and lowering of the aquifer water level, weakening its seepage effect in the top coal area, reducing the moisture content of the bottom soil, and improving its shear strength can mitigate water and sand inrush accidents in the backfill areas of open-pit mines.