Dealing with small sets of laboratory test replicates for Improved Cooking Stoves (ICSs): Insights for a robust statistical analysis of results

Abstract

Improved Cooking Stoves (ICSs) represent the most commonly promoted solution to alleviate the burden associated with the use of traditional biomass in a short-term perspective. However, criticism is raising about the methodologies used for assessing their performance, with a particular focus on laboratory-based testing protocols. One of the key weaknesses of current protocols consists in the inaccurate and biased approach adopted for reporting and statistically analysing test results, which can lead to misleading conclusions about the actual improvements ensured by ICSs. This study proposes a robust procedure to deal with the statistical analysis of small sample sizes, and subsequently verify it through its application to an experimental comparison – based on the Water Boiling Test – between three models of stove. The results show that the current practice based on 3 or 5 replicates often produces biases in the analyses, as at least 13 replicates might be needed to achieve reliable results. Moreover, the study shows how the t-test is in most cases improperly applied, while the proposed procedure allows to deal both with normally and of non-normally distributed data sets in a robust way. In one case, the apparent improvement of an ICS model as compared to the three-stone fire, is refuted by the application of our procedure.