The significance of improving the drilling productivity and reducing the cost and non-productive time of drilling process, substantially relies on the efficiency of drilling performance. This paper provides a comprehensive understanding of drilling process, aiming to predict drilling performance and investigate drilling parameters using a validated finite element (FE) model. Experimental validation of the FE model was achieved through testing on a laboratory drilling rig, ensuring the accuracy and reliability of the numerical simulations. To accurately capture the nonlinear characteristics of bit-rock interaction, the Riedel–Hiermaier–Thoma model was adopted as a material model, and its parameters were identified through a series of carefully conducted experimental tests. The numerical results obtained from the FE rock failure model during the compressive and tensile tests demonstrated a robust correlation with the experimental data. The verified material model was then employed into another FE drilling model to simulate rock breaking in an actual drilling scenario. This analysis sheds light on the impact of drill-bit interaction with the rock formation, providing valuable insights into its behaviour during drilling operations. The FE drilling model was further utilised in a parametric study to predict the effects of critical drilling parameters, like loading rate and rotary speed, on the weight on the bit, torque on the bit, and rate of penetration. Both the FE drilling and experimental results provided a significant consistency when the drilling parameters were compared, and nonlinear dynamic phenomena, such as stick–slip and bit-bouncing, were observed. By investigating these effects, this study contributes to optimising drilling operations, enabling better control of premature vibrations and enhancing drilling efficiency.