Yield gaps are pervasive in African smallholder agriculture, and are large for almost all crops in all regions. There is consensus that poor soil fertility and nutrient availability are the major biophysical limitations to agricultural production in the continent. We identify two major yield gaps: (1) the gap between actual yields (YA) and the water-limited yield potential (Yw), which is the maximum yield achievable under rainfed conditions without irrigation if soil water capture and storage is optimal and nutrient constraints are released, and (2) The gap between YA, and a locally attainable yield (YL) which corresponds to the water and nutrient-limited yields that can be measured in the most productive fields of resource endowed farmers in a community. Estimates of these two yield gaps are given for major crops, together with a framework for how yield gaps can be estimated in a pragmatic way for different farming systems. The paradigm of ecological intensification which focuses on yield potential, soil quality and precision agriculture is explored for the African context. Our analysis suggests that smallholder farmers are unable to benefit from the current yield gains offered by plant genetic improvement. In particular, continued cropping without sufficient inputs of nutrients and organic matter leads to localised but extensive soil degradation and renders many soils in a non-responsive state. The lack of immediate response to increased inputs of fertiliser and labour in such soils constitutes a chronic poverty trap for many smallholder farmers in Africa. This necessitates a rethink for development policy aimed to improve productivity and address problems of food insecurity.