The literature argues that increasing the total factor productivity in agriculture is prior to off-farm growth and poverty alleviation. The debate persists on a robust estimate of agricultural productivity growth that includes multifactor. So, this study applies the stochastic frontier analysis with heteroscedasticity correction, to examine the total factor productivity growth in a sample of 23 countries in Sub-Saharan Africa over the 1991-2015 period. The results show that the estimated elasticity of production with respect to land, machinery capital and fertilizer are positive and statistically significant. Whereas, the estimated elasticity of production to labour is positive but not significant. This result calls into question the productivity of labour in the African agricultural system. After the 2000s, the average annual growth rate of agricultural productivity is estimated at 3.13% for the sample countries. A net increase in total factor productivity is accompanied by a significant increase in technological change. Scale efficiency change is the major barrier to TFP growth in African agriculture. So, it is recommended that public or private decision-makers must stress more on: (i) human capital investment along with more efficient management of cultivated land and, (ii) the acceleration of agricultural technology adoption.