Predicting outcomes for big data projects: Big Data Project Dynamics (BDPD): Research in progress

Abstract

The number and importance of Big Data projects is increasing, but unfortunately, a large proportion of Big Data projects are failing. The ability of organizations to manage these projects has so far not kept pace — they need better ways to analyze the behavior of their Big Data projects and positively affect outcomes. The objective of this paper is to identify the important dynamic characteristics of Big Data projects, and explore how a modeling and simulation technique called system dynamics (SD) can be applied these characteristics. The approach draws from applicable concepts in the domains of traditional project management, Agile software development and Lean product development, and proposes to develop a model called Big Data Project Dynamics (BDPD) incorporating these insights. The BDPD model is organized into sectors: Core Rework Cycle, Iterative & Incremental, Exploration & Learning, Economic Value, and Policy Actions & Consequences. Given ADPD, practitioners can simulate Big Data project behavior from initial conditions, and probabilistically predict project outcomes.