Polyetheretherketone (PEEK) is a promising thermo-plastic polymer material due to its excellent mechanical properties. To further improve the mechanical properties of PEEK, different kinds of short fibers are added into the PEEK matrix. The grinding machinability of short-fiber-reinforced PEEK varies with the effect of fiber type and content. Therefore, it is crucial to investigate the surface quality and removal mechanism of fiber-reinforced PEEK in ultra-precision grinding. In this paper, different fiber types and mass fractions of short-fiber-reinforced PEEK, including carbon-fiber-reinforced PEEK (CF/PEEK) and glass-fiber-reinforced PEEK (GF/PEEK), are employed. The grinding machinability of short-fiber-reinforced PEEK was investigated using grinding experiments with grinding wheels of different grit sizes. The effects of the fiber type and mass fraction on the surface quality and removal mechanism during grinding were discussed. The results showed that the brittle–ductile transition depth of carbon fiber was much larger than that of glass fiber, so it was easier to achieve ductile removal in grinding with the carbon fiber. Therefore, the ground surface roughness of CF/PEEK was smaller than that of GF/PEEK under the same grinding conditions. With the increase in carbon fiber mass fraction, the ground surface roughness of CF/PEEK decreased due to the higher hardness. The brittle–ductile transition depth of glass fiber was small, and it was easy to achieve brittle removal when grinding. When the glass fiber removal mode was brittle removal, the GF/PEEK surface roughness increased with the increase in glass fiber content.