Accounting for uncertainty in multi-criteria sustainability assessments at the farm level: Improving the robustness of the SMART-Farm Tool

Abstract

Many farm sustainability assessments use multi-criteria methods for aggregating indicators based on performance scores and importance weights. One of these is the SMART-Farm Tool, which measures the degree of goal achievement of farms across 327 indicators nested within 4 sustainability dimensions, 21 themes and 58 sub-themes of the FAO-SAFA Guidelines (Sustainability Assessment of Food and Agriculture). This study aims to improve the empirical foundation of the SMART-Farm Tool by (i) evaluating the uncertainty of indicator weights obtained via expert opinion and (ii) integrating this uncertainty into SMART assessment results. An adapted Delphi process was implemented, involving a group of 67 experts from 21 countries divided into thematic and regional sub-groups. Experts within each sub-group rated the importance of indicators of all 58 SAFA sub-themes, and self-assessed their own competence. Contrary to expectations, the uncertainty in expert opinions was relatively high for environmental indicator weights (although lowest for animal welfare), being comparable to social and governance indicators. Considerable uncertainty remained in indicator weights, even after two rounds of discussion and exchange of views. This is attributed to regional variation and inherent system complexity (i.e. experts having legitimate but diverging viewpoints based on contradictory evidence) rather than scientific ignorance (i.e. a lack of research evidence). Nevertheless, it is expected that the levels of uncertainty could be reduced by limiting the number of indicators to be evaluated and thus by allowing for more in-depth discussions among the experts. Monte Carlo Simulations were used to translate residual indicator weight uncertainty into SMART assessment results at the farm level for four example farms from both developed and developing countries. The resulting comparisons revealed several cases where substantial apparent differences between farms in sustainability scores for a specific sub-theme (up to 23%) were not statistically significant, while in other cases differences of 5% were significant. This emphasizes the general importance of considering uncertainty in multi-criteria assessment tools, with clear implications real-world applications, such as product certification, labelling and marketing. Finally, this study provides important methodological suggestions for implementing expert-based assessments in multi-criteria assessments efficiently.