Ultrafine-grained (UFG) refractory metals are promising materials for applications in aerospace, microelectronics, nuclear energy, and many others under extreme environments. Powder metallurgy (PM) allows to produce such materials with well-controlled chemistry and microstructure at multiple length scales and near-net shape manufacturing. However, sintering refractory metals to full density while maintaining a fine microstructure is still challenging due to the high sintering temperature and the difficulty to separate the kinetics of densification versus grain growth. Here an overview of the sintering issues, microstructural design rules, and PM practices towards UFG and nanocrystalline refractory metals are sought to be provided. The previous efforts shall be reviewed to address the processing challenges, including the use of fine/nanopowders, second-phase grain growth inhibitors, and field-assisted sintering techniques. Recently, pressureless two-step sintering has been successfully demonstrated in producing dense UFG refractory metals down to ≈300nm average grain size with a uniform microstructure and this technological breakthrough shall be reviewed. PM progresses in specific materials systems shall be next reviewed, including elementary metals (W and Mo), refractory alloys (W-Re), refractory high-entropy alloys, and their composites. Last, future developments and the endeavor towards UFG and nanocrystalline refractory metals with exceptionally uniform microstructure and improved properties are outlined.