Background Cassava is an outcrossing, highly heterozygous plant that is reported to suffer from inbreeding depression. However, unraveling recessive traits and exploring additive genes would require a limited level of inbreeding for the genetic improvement of cassava. Method The impact of inbreeding depression (ID) on agronomic and biotic tolerance of cassava was evaluated using the S1 progenies of five African cassava varieties (TMS 30572, TME 419, TMS 98/0505, TMS 01/1371, and TMS 98/0002) at the seedling and clonal evaluation stages. Results At both trial stages, the effects of ID were severe on average performance of fresh root yield and fresh foliage yield; moderate on harvest index, dry matter content, vigor, and tolerance to cassava mosaic disease, cassava bacterial blight, and cassava anthracnose diseases; and less severe on plant height. Further examination of S1 families using molecular markers showed varying levels of heterozygosity at several loci across the genome. In addition, the degree of heterozygosity and ID varied by the S1 family. The TMS 01/1371 family showed the lowest degree of heterozygosity and ID on the average performance of different agronomic attributes, indicating that inbreeding may be strategically explored in this family to increase genetic gain and identify useful recessive traits. Lastly, the observed depression from inbreeding was higher in seedling evaluation than in clonal evaluation trials for fresh root yield, fresh foliage yield, harvest index, and dry matter content. Conclusion S1 individuals showing relatively low heterozygosity based on SNP data were selected as parents to advance genetic gain in cassava. Generally, reduced heterozygosity was prominent in traits with severe ID impacts on average phenotypic performance. Our results highlight the relative importance of exploring non-additive genetic effects and transgressive segregations for favorable allele combinations in cassava improvement.