Aiming at the problems of low threshing rate and high loss rate in threshing and separating device of millet combine harvester, an axial-flow threshing and separating device with front-end rasp bar and rear-end nail tooth structure was designed. The difference of threshing performance and separating performance between rasp bar and the nail tooth was obtained, based on the mathematical model of threshing and separating performance. The threshing and separating performance of the rotary roller of full stripe bar, full nail tooth, and combined-stripe bar and nail tooth were compared by model analysis. The earhead model was established by using Hertz-Mindlin with bonding model. The Discrete Element Method (DEM) simulation test was carried out, and its results combined with that of model analysis to optimize the rotary roller structure parameters. The simulation single factor tests were carried out to obtain the influence rules of rotary roller speed, threshing clearance and concave grid spacing on the operation performance. Finally, a three-factor five-level quadratic orthogonal rotation combination test was carried out. Using the method of variance analysis and multi-objective optimization, it was calculated the best operating parameters and optimal performance indicators of the device. The verification test of the optimization results shows that they are basically consistent with the optimization results, which shows that the rasp bar and nail tooth rotary roller can meet the requirements of millet threshing and separation operations. Mathematical model analysis and DEM can provide reference for the optimization design of millet threshing and separation device.