It is of great practical significance to accurately obtain formation collapse pressure and determine an effective three-pressure profile with a correct strength criterion in the drilling process to identify the best drilling fluid density. Taking the tight sandstone of the XuJiahe formation as an example, we conducted a series of rock mechanics tests, focusing on large-scale, high-density confining pressure triaxial experiments; determined a mathematical expression for the continuous tangent envelope of a nonlinear Mohr circle envelope based on a series of triaxial tests; and clarified the variation rules of cohesion force and internal friction angle with confining pressure. The impact of rock mechanics parameters determined by using the traditional method and the continuous tangent envelope method on wellbore stability is compared and analyzed by using the MathCAD program, and then the collapse pressure is obtained. The results show that the parabolic curve derived from the uniaxial rock mechanics test data of the XuJiahe formation is not suitable for the triaxial test results under high confining pressure. By means of the continuous tangent envelope method, the relationship between rock cohesion and internal friction angle and confining pressure is obtained; this replaces the traditional collapse pressure calculation results using geophysical logging data or uniaxial tests, and the relationship between cohesion and friction angle is more consistent with confining pressure. The MathCAD simulation analysis shows that the rock mechanics parameters determined by the continuous tangent envelope can reflect the stratum situation more truly than the linear envelope method. Compared with the linear envelope method and parabolic envelope method under the same conditions, the continuous tangent envelope method has certain advantages in determining the critical density, which provides a theoretical basis for the accuracy of sandstone formation collapse pressure calculation and can give significant guidance for the study of wellbore stability of deep sandstone formations.