Even though China has increased grain output in the past decade and greatly contributed to reducing global hunger, the country is still confronted with intense food security pressure due to its huge population base and severe loss of farmland. It has become increasingly important to fully exploit yield gaps (the difference between potential and actual yields) of all staple grains and to identify priority areas in order to achieve the UN sustainable development goal of zero hunger by 2030. The objective of this study was to calculate the production and yield gaps for rice (Oryza sativa L.), wheat (Triticum aestivum L.), and corn (Zea mays L.) across 31 provinces of mainland China, and then to propose sustainable ways to exploit the production and yield gaps in relation to the availability of natural resources and the ecological conditions of the various regions. The yield potential was estimated by assuming that certificated cultivars were adopted in each agricultural ecological zone (AEZ) and that state-of-the-art technologies and best management practices were used. The results suggested that the gross potential productivity of the three staple grains of mainland China in 2016 was 8.86 × 108 tons, with a production gap of 2.99 × 108 tons. Corn exhibited the greatest yield gap. The highest potential productivity was observed in the Northeast Plain and the Huang-Huai-Hai/North China Plain, accounting for 26.35% and 35.91% of the country’s total potential productivity, respectively. The greatest yield gaps were also found in these two plains. To narrow yield gaps, farmland infrastructure, especially irrigation facilities, should be improved, and field management should be strengthened for the provinces with the greatest gaps. However, in view of natural resource constraints (e.g., water shortages in Northern China), water-saving measures and techniques have been encouraged throughout mainland China and the planting of high water-consuming crops such as rice has been discouraged and should be reduced and replaced by other crops in the North China Plain and Northwest China. Using presently available cultivars and field management technologies, China still has great potential to increase grain production and improve food security by closing exploitable yield gaps through the use of suitable production methods based on existing natural resource capacity and ecological status.