The toolholders featuring large length-diameter ratio and variable cross-sections are growingly required to conduct certain machining tasks in face milling operation. However, the weak dynamic characteristics of this kind of toolholder can easily cause chatter, and its special structure creates further obstacles to chatter suppression. This paper develops a novel toolholder with high dynamic stiffness to improve the chatter stability. Taking the effect of multiple order vibration modes into consideration, a dynamic model for face milling operation is established to investigate the relationship between the stability performance and modal parameters of the machining system. The theoretical analysis shows that the increasing dynamic stiffness can enhance the limit axial depth of cut. Based on this finding, the novel toolholder is designed with the assistance of the embedded stair-step strips and damping core, and its dynamic characteristics are qualitatively analysed by the extended strain energy method and cantilever beam theory. Then, the optimal design of geometrical dimensions and related materials of the novel toolholder is performed by the finite element method (FEM). Finally, the novel toolholder is manufactured, and then verified by tool point dynamics tests and milling experiments. Compared with the conventional toolholder, the measured results show that the dynamic stiffness and material removing rate (MRR) of the novel toolholder are increased by 3.75 and 2.81 times, respectively.