Multiple method modelling reveals lack of robustness in natural resource management research

Abstract

Research on natural resource management like fisheries, irrigation systems or forestry traditionally uses case studies providing us with a rich, in-depth perspective on many single systems. This comes with a disadvantage - lacking comparability as differences between studies exist in variables examined, their operationalization or methods used. Thus, studies often disagree on important drivers for ecological success. However, due to design differences the reasons behind different results often remain unknown. One reason might be the impact of method choice. Hence, this article tests the influence of methods on model results. We use a high-quality data set, the Nepal Irrigation Institutions and Systems database (NIIS), developed at the Ostrom Workshop. It contains 263 cases, each record having information on around 600 variables. Multiple machine learning methods - random forests (RF), gradient boosting (GBM), shallow neural networks (SNN) and deep neural networks (DNN) - are compared with a standard statistical approach (multivariate linear regressions (MLR)). We try to answer the question whether these methods differ in estimating the relevance for success of such well-known concepts like participation of users, resource size, relations with other groups, and social capital among others. The results indicate that both agreements and substantial differences exist across methods which casts doubt on the robustness of previous results. Hence, we advise more caution in interpreting existing results. We see this research as a step towards increasing the robustness of results and improving both generalisability and reproducibility of natural resource management research.