• Short term fallows enhance productivity, improve soil fertility and serve dual purpose in crop-livestock systems. • Green manuring increased wheat grain yield by 18–26 % over mineral fertilizers. • Root biomass of green manures alone increased yield of subsequent crop by 25 %. • Green manuring improved soil water, soil pH, organic carbon and nutrients. • Residual effect of green manuring compensates 33 % of N demands of wheat. Soil fertility depletion is a major constraint of smallholder farming in sub-Saharan Africa. We tested the aftereffects of green manures, namely vetch (Vicia sativa L.), lupin (Lupinus polyphyllus L.), and lablab ( Lablab purpureus L.) incorporated into the soil compared to three fertilizer levels (0/0, 23/0, and 78/20 kg N/P ha −1 ) and evaluated their effect on soil fertility status and wheat yield in acidic Nitisols of southern Ethiopia, for three consecutive years (2017–2019). The treatments were laid out in a randomized complete block design with three replications. In 2017 and 2018, green manures were sown in March and April, respectively, using short rains and incorporated into the soil during late flowering stage, either (i) the whole shoot and root biomass were plowed under, (ii) shoot biomass was transferred to non-treated plots or (iii) only the below ground root biomass was used. Wheat was sown during long rains during the same growing season a month after the incorporation of green manures. In 2019, wheat was grown on the residual plots with the application of an additional 64/20 kg N/P ha −1 . Results revealed that in 2017 and 2018 the application of vetch and lupin green manure resulted in grain yield advantages of 49 and 32 % and 34 and 19 %, respectively, over 0/0 and 23/0 kg N/P ha −1 , though it produced less grain yield compared to higher fertilizer rates. In 2019, the addition of vetch and lupin whole biomass treatments gave significantly higher wheat yield over fertilizer treatments, with yield advantages of 18–26 %. Similarly, root biomass only also produced a significantly higher yield than fertilized plots. The yield benefits from green manures were due to improved soil water content, improved P-availability, significantly increased exchangeable K, Ca, and Mg, and increased pH by about 0.5 units. The residual effect of green manures could compensate for up to 33 % of the recommended rate of 78 N kg ha −1 . We conclude that niche-based integration of green manures could improve yield, enhance soil carbon sequestration and sustain crop-livestock systems, whereby the above ground biomass could serve as quality livestock feed without compromising soil fertility benefits.