Short glass fiber composites, particularly sheet molding compound (SMC) materials, are becoming increasingly important alternative in various contemporary aerospace, automotive, and electronic applications. For these manufacturing industries, the quality of the machined SMC composite is still a challenging target. The article proposes a new tool design with an offset between teeth to minimize friction, limit damage and promote chip removal when drilling composite materials. The effects of the tool’s geometric parameters, especially the rake, the inclination and the complementary side cutting edge angles on the material removal process, as well as the cutting and thrust forces, are investigated. A 3D finite element model of a representative multi-tooth tool is developed using the ABAQUS\\Explicit code. The results show that fine-tuning the geometric parameters of the tool reduces the induced machining damage and enhances the chip removal and the flow evolution. The rake angle has a significant influence on the cutting and thrust forces. However, both forces are insensitive to the inclination angle. The complementary side cutting edge angle influences only the thrust force. The presented outcomes not only give insights into the cutting process, but also improve the SMC machinability.