A diversified diet in early childhood that comprises plant and animal source foods contributes to better health outcomes throughout the life course. However, inadequate infant and young child-feeding practices are serious public health problems in Africa. Evidence on the magnitude and determinants of inadequate animal source food consumption (ASF) in sub-Saharan Africa (SSA) is not yet well established. Hence, this study aimed to estimate the pooled magnitude and determinants of ASF consumption in SSA. A total weighted sample of 92,309 children aged 6–23 months from the Demographic and Health Survey dataset of 35 SSA countries was used for this study. A multilevel mixed-effect model was fitted. The intra-class correlation coefficient (ICC) and likelihood ratio (LR) tests were used to assess the presence of the clustering effect. The model comparison was made using deviance information criteria (DIC). A p-value of p < 0.05 was used to declare statistical significance. The pooled magnitude of ASF consumption among children aged 6–23 months in SSA was 23.7% (95% CI: 23.4, 23.9). Across the SSA regions, the Southern Africa region had the highest consumption of ASF (51.35%), and the lowest was from the Eastern Africa region (21.41%). Younger children aged 6–9 months (AOR = 3.39; 95% CI: 3.14, 3.65), mothers who had no formal education (AOR = 1.46; 95% CI: 1.37, 1.56), children from mothers with no media exposure (AOR = 1.17; 95% CI: 1.11, 1.24), children from the poorest households (AOR = 2.59; 95% CI: 2.36, 2.84), rural residence (AOR = 1.20; 95% CI: 1.12, 1.28), living in low-income countries (AOR = 2.43; 95% CI: 2.03, 2.90), and children from the East Africa region (AOR = 1.24; 95% CI: 1.14, 1.35) were among the significant predictors for not consuming ASF. In this study, ASF consumption was markedly low. Almost three-fourths of children aged 6–23 months in SSA did not consume ASF in the previous day. Age of the child, ANC visits, maternal educational status, maternal marital status, media exposure, and household wealth index were significant individual-level predictors of ASF consumption, whereas the place of residence, country’s income level, and regions of SSA were found to be significant community-level predictors of ASF consumption. Thus, the study highlighted that policymakers, governments, and their partners should demonstrate rigorous efforts to improve dietary diversification with due consideration to the identified predictors of ASF.