In Ethiopia, smallholder farmers are responsible for most food production. Though yield levels in grain crops have improved greatly over the years, they are still much lower than their potential. The source of yield improvements and the causes of those yield gaps are not well understood. To explain the drivers of yield gaps and current sources of yield improvements in four major cereals (teff, maize, wheat, and sorghum) and three grain legumes (faba bean, common bean, and soybean), we accessed the databases of the Global Yield Gap Atlas, the Food and Agriculture Organization of the United Nations, and the Central Statistical Agency of Ethiopia. Refereed journal articles and grey literature were sought in online databases using keywords. The results showed large increases in production of grain crops with little or no increase in areas of production. The yield increases were primarily attributed to genetic gain rather than agronomic improvements. Farmers’ yields remain far lower than those from on-farm trials and on-station trials and the calculated water-limited yield potential. Currently, yields of wheat, maize, sorghum, and common bean in Ethiopia are about 26.8, 19.7, 29.3, and 35.5% of their water-limited yield potentials. Significant portions of the yield gaps stem from low adoption and use of improved varieties, low application of inputs, continual usage of un-optimized crop management practices, and uncontrolled biotic and abiotic stresses. Proper application of fertilizers and use of improved varieties increase yield by 2 to 3 fold and 24–160%, respectively. Cereal-legume intercropping and crop rotation practices increase yield while reducing severity of pests and the need for application of synthetic fertilizers. In contrast, abiotic stresses cause yield reductions of 20–100%. Hence, dissection of the water-limited yield gap in terms of technology, resource, and efficiency yield gaps will allow the prioritization of the most effective intervention areas.