Wellbore instability is one of the significant challenges in the drilling engineering and during the development of carbonate reservoirs, especially with open-hole completion. The problems of wellbore instability such as downhole collapse and silt deposit in the fractured carbonate reservoir of Tarim Basin (Ordovician) are severe. Solid destabilization and production (SDP) was proposed to describe this engineering problem of carbonate reservoirs. To clarify the mechanism and mitigate potential borehole instability problems, we conducted particle size distribution (PSD) analysis, X-ray diffraction (XRD) analysis, triaxial compression tests, and micro-scale sand production tests based on data analysis. We found that the rock fragments and silt in the wellbore came from two sources: one from the wellbore collapse in the upper unplugged layers and the other from the production of sand particles carried by the fluid in the productive layers. Based on the experimental study, a novel method combining a geomechanical model and microscopic sand production model was proposed to predict wellbore instability and analyze its influencing factors. The critical condition and failure zone predicted by the prediction model fit well with the field observations. According to the prediction results, the management and prevention measures of wellbore instability in carbonate reservoirs were proposed. It is suggested to optimize the well track in new drilling wells while upgrading the production system in old wells. This study is of great guiding significance for the optimization of carbonate solid control and it improves the understanding of the sand production problems in carbonate reservoirs.