Technical risk model of machine learning based software project development - A multinational empirical study using modified Delphi-AHP method

Abstract

ContextThe development of machine learning (ML) based software projects has increased significantly over the past decade, introducing new technical risks that rarely or never appear in traditional software development projects. ObjectiveThis research aims to identify and prioritize the technical risk factors that may lead to the failure of ML-based software development projects. MethodFirst, a literature review was conducted to compile a preliminary list of technical risk factors for ML-based software project development. Then, two rounds of the modified Delphi process were conducted with 17 ML experts to review and verify the completeness and appropriateness of the preliminary technical risk factors. A hierarchy of five technical risk categories with 22 technical risk factors was concluded for the analytic hierarchy process (AHP). Then, three rounds of online AHP questionnaires were administered. The consistency ratio (CR) was used to check the respondents’ answers, and the quartile deviation (QD) was applied to assess the consensus on all 96 questions. Finally, we prioritized the technical risk categories and associated technical risk factors. ResultsWe found that "data availability and quality" ranked as the top technical risk category in terms of severity, probability, and impact rankings of the five technical risk categories. Furthermore, all four technical risk factors within this category also occupied the top four positions of impact ranking. ConclusionThe research results highlight the crucial role of the four data availability and quality risk factors for the failure of ML-based software project development. The proposed technical risk model of ML-based software project development with the identified severity and probability priorities may provide practitioners and research community with a clear overview, highlighting areas demanding priority attention to effectively mitigate project failure risks. These findings have broader implications for improving the success rates of ML-based software projects across various domains.