Cutting depth and spacing are two important parameters for the efficiency of a roadheader in mining operation. In this study, the case of a roadheader excavating a coal gallery is taken into account. Based on explicit finite element analysis (FEA), the minimum cutting specific energy (SE) was obtained by a series of numerical simulations. According to the actual cutting process, a three-dimensional (3D) double-pick cutting model was established. The validation for the cutting model showed that it was not only reliable to predict the value of cutting SE, but also capable of accurately simulating the cutting morphology. The variation of cutting moments, stress distributions, and character of coal fragment formation were investigated. The reasons for the different SEs are explained and an optimum design for cutting depth and spacing is given. The results show that SE shrinks by 22.82% using the optimum design compared to the original parameters. Overall, it is believed that the novel 3D double-pick cutting model and the optimization method used in this study are highly appropriate to better understand coal fragmentation and improved mining efficiency.