This study investigated the effect of temperature and atmosphere on the grindability of the semicoke prepared via a mid–low temperature pyrolysis of low-rank Shenmu coal, analyzed the variation in its grinding characteristics with the increase of ball grinding duration, and explored the influence mechanism of the pyrolysis conditions on the grindability of semicoke based on the pyrolysis process and semicoke structure evolution. The results show that as the pyrolysis temperature increases (400–700 °C), grindability and grinding mechanism of the semicoke change significantly; the grindability first increases and then decreases with the increase of the pyrolysis temperature. As the grinding duration increased, high-Hardgrove grindability index (HGI) semicoke underwent intensive bulk grinding at a specific grinding time. Additionally, the varying contents of the proximate analysis during devolatilization are not the main cause of the varying HGI during pyrolysis. Using thermogravimetric and thermomechanical analyses, the optimal grindability of the semicoke was found to occur at the transition stage of the pyrolysis to pyrocondensation reactions (approximately 600 °C). During the transition, the intense pyrolysis reaction led the semicoke to have abundant pores and cracks, a highly disordered chemical structure, and low matrix hardness. The porous semicoke which has high grindability first underwent surface grinding; subsequently, the loose cores were more prone to bulk grinding, whereby the optimal grindability was obtained. HGIs of the semicoke prepared in the reductive atmospheres such as H2, CO and CH4 are lower than that of the semicoke prepared in N2 atmosphere at the same pyrolysis temperature. Moreover, the reductive atmospheres inhibited pore development in the semicoke and made its carbon structure more ordered, which densified the structure and decreased the grindability of the semicoke.