The generalized Hoek–Brown criterion (GHB) is recognized as one of the standard failure criteria in rock engineering and its validity extends to a wide range of rock mass quality. A drawback of this criterion is the difficulty of transforming it into an explicit form defining the Mohr failure envelope when its strength parameter a is not equal to 0.5. The information on the functional form of the Mohr envelope for the full range of rock mass conditions enables the implementation of classical engineering approaches, such as the limit equilibrium method and limit analysis, in the framework of the GHB criterion. Knowing that for a≠0.5 the exact closed-form representation of the Mohr envelope is not feasible, an alternative is to express it in an approximate analytical form. The main purpose of this study is to propose a new improved method to define an approximate Mohr envelope of the GHB criterion that is much more accurate compared with the recently published approximations. The idea behind the formulation is to expand the Balmer’s equation, which defines the relationship between the normal stress and minor principal stress at failure, by invoking the finite Taylor series centered at the known solution for a=0.5. The formulation is then completed by substituting this solution into another Balmer’s equation, defining the relationship between the shear strength and the minor principal stress. The Taylor polynomial approximations of up to third degree are considered in the formulation. The accuracy of the shear strength prediction is shown to be much better than that of the approximate formula of Lee and Pietruszczak proposed in 2021. An illustrative example of limit equilibrium analysis of rock slope stability, incorporating the new approximate expression for the Mohr envelope, is provided. The analysis incorporates a modified version of the Bishop approach, which is simpler and more rigorous than the original nonlinear expression. The study confirms that the new approximate representation of the Mohr failure envelope can facilitate the application of the GHB criterion to a range of practical rock engineering calculations.