Nonlinear Mohr failure criterion is re-evaluated in this paper. It has been revealed that the polynomial failure criteria constructed by the normal stress σn and the shear stress τn predict the fracture angles unreasonably, and their predictions are also inconsistent with experimental observation that the two branches of failure envelope trend to be parallel to σn axis under high hydrostatic compression. In order to overcome the problems, first, the transformation of failure functions from the principal stress space (σ1, σ3) to the stress space (σn, τn) is derived. Then the failure function is expressed in terms of the principal stresses, and it is converted from the principal stress space (σ1, σ3) to the stress space (σn, τn) based on the transformation. The present failure criterion predicts more reasonable failure angles, and its envelope trends to be parallel to σn axis under high hydrostatic compression. In particular, the paper reveals that for brittle materials the fracture angle under the uniaxial tension is not 0° although the fracture surface is nearly perpendicular to the direction of the loading; under the pure shear stress the angle between the fracture surface and the direction of the loading is not 45° although it is close to 45°. The present criterion predicts this physical phenomenon reasonably.