Abstract This paper examines a nonlinear dynamics model to explore the significant factors influencing the coupled torsional-axial vibrations of a drill string. The primary focus lies in effectively modeling, analyzing, and controlling both torsional and axial vibrations occurring in a rotary oil well drilling system. The model employed incorporates a wave equation with a damping term (PDE) connected to an ordinary differential equation (ODE) through a nonlinear function representing the interaction between the drill bit and the rock. We proceed to establish the well-posedness of the coupled PDE-ODE in an appropriate functional space. As a PDE-ODE coupled system, we specifically investigate the stability problem concerning the velocity at the bottom extremity, considering both the presence and absence of the damping term in the wave PDE. To address this, we propose a systematic method based on a Lyapunov approach for designing feedback controllers. These controllers ensure system stability and effectively suppress harmful vibrations.