Limited studies focused on the slope rock foundation, and the application results of several methods differ considerably. In this paper, the coefficients of the Bell solution are first reduced by the length ratio of the sliding failure plane and the vertical area ratio of the overloaded soil. Then, the Hoek–Brown criterion for nonlinear failure is introduced, and the tangential line tangent to the Hoek–Brown failure criterion curve is calculated to replace the original M-C failure criterion. A method for determining the instantaneous friction angle and instantaneous cohesion under normal stress with the Bell solution failure mode is established, and the relationship between the instantaneous friction angle, instantaneous cohesion, and normal stress are discussed. The instantaneous friction angle decreases with the increase in normal stress, whereas the instantaneous cohesion increases with normal stress. The influence of geological strength index (GSI) and rock empirical parameter mi on bearing capacity coefficients N σ, N q, and Nγ are also comprehensively analyzed. The coefficients increase with GSI and mi . In addition, the method is used to analyze the bearing capacity of rock foundation of a high and steep slope in an open-pit mine, and the results are compared with the slip-line method, which considers Hoek–Brown failure criterion and finite-element numerical simulation. The comparison demonstrates the acceptability of the proposed method.