Hydrocarbon exploration and production activities are guaranteed through various operations including the drilling process, which is realized by using rotary drilling systems. The process involves crushing the rock by rotating the drill bit along a drill string to create a borehole. However, during this operation, violent vibrations can occur at the level of the drill string due to its random interaction with the rocks. According to their axes of occurrence, there are three types of vibrations: axial, lateral, and torsional, where the relentless status of the torsional vibrations is terminologically known as the stick-slip phenomenon. Such a phenomenon can lead to increased fatigue of the drill string and cause its abortive fracture, in addition to reducing the efficiency of the drilling process and consequently making the exploration and production operations relatively expensive. Thus, the main objective of this paper is to eliminate the severe stick-slip vibrations that appear along the drill string of the rotary drilling system according to the LQG observer-based controller approach. The rock–bit interaction term is highly nonlinear, and the bit rotational velocity is unmeasurable; an observer was first designed to estimate the unknown inputs of the model, and then the controller was implemented in the drill string model with 10 degrees of freedom. The estimation process was essentially based on surface measurements, namely, the current and rotational velocity of the top drive. Thereafter, the performance of the proposed observer-based LQG controller was tested for different simulation scenarios in a SimScape/Matlab environment, for which the controller demonstrated good robustness in suppressing the severe stick-slip vibrations. Furthermore, the simulation and experimental results were compared to other controllers designed for the same model; the proposed observer-based LQG controller showed better performance, and it was less sensitive to structured disturbances than H∞. Thence, it is highly recommended to use the proposed approach in smart rotary drilling systems.