The introduction of conservation agriculture (CA) for smallholders increased the competition for crop residues between crop and livestock enterprises of the mixed smallholder farming system. Smallholders practicing CA have resorted to using grass and leaf litter in addition to available crop residues. The effect of these different mulching materials on maize (Zea mays L.) growth and yield is not well documented in smallholder CA systems of southern Africa. A two-year experiment was run in 2012/13 and 2013/14 seasons to evaluate the effect of maize residues, grass (Hyparrhenia filipendula (L.) Stapf.) and leaf litter that farmers are currently using and residues from leguminous species, sunhemp (Crotolaria juncea L.) and Tephrosia (Tephrosia vogelii ((Hook) f.)) on maize nitrogen (N) uptake, growth and yield. Significant differences in soil water content across treatments were only observed during March in 2012/13 season. Maize residues retained more soil water and Tephrosia had the lowest soil water content when seasonal rainfall pattern was erratic. Grass and Tephrosia treatments had the lowest chlorophyll content. Conventional ploughing, maize residues and leaf litter had similar chlorophyll content which was significantly higher than grass and Tephrosia treatments. At a site with higher initial soil fertility conventional ploughing treatment out yielded the other treatments by 727–1265 kg ha−1. With more degraded sandy soil conventional practice had 119–430 kg ha−1 more maize grain than the CA treatments. With adequate fertilization, the mulching materials have a similar effect on maize growth in basins and direct seeding. Further studies on different application rates of mulching materials and mineral N fertilizer, and nutrient release patterns of these residues are critical in order to better understand soil fertility management under smallholder CA systems.