Abstract Vibrations such as bit-bouncing, stick–slip, and whirl motion can significantly reduce downhole drilling's performance and efficiency. These phenomena are likely to arise once the dynamical states of drilling fall into undesired operating regimes, as verified by both theoretical analyses and experimental studies. To effectively avoid such undesired operating conditions of a downhole drilling process, this paper introduces an advanced nonlinear state-constrained control method to formulate a setpoint tracking problem of high-order directional drilling dynamics under state constraints. These constraints are carefully defined using the field test results, and their shapes are in complex state-space regions, causing additional complexity to the control design. Moreover, unlike vertical drilling, the directional drilling system requires modeling of coupled axial and torsional dynamics with a higher degree-of-freedom (DOF). In this study, the proposed method will tackle these challenges by converting the original high-order constrained control problem into a standard nonlinear control problem. Leveraging on the linear parameter varying (LPV) method that is applied to the converted system, we can actively prevent drilling from falling into the undesired regimes, to achieve the constrained control objective.