Data journeys: Identifying social and technical barriers to data movement in large, complex organisations

Abstract

Managers in complex organisations often have to make decisions on whether new software developments are worth undertaking or not. Such decisions are hard to make, especially at an enterprise level. Both costs and risks are regularly underestimated, despite the existence of a plethora of software and systems engineering methodologies aimed at predicting and controlling them. Our objective is to help managers and stakeholders of large, complex organisations (like the National Health Service in the UK) make better informed decisions on the costs and risks of planned new software systems that will reuse or extend their existing information infrastructure.We analysed case studies describing new software developments undertaken by providers of health care services in the UK, looking for common points of risk and high cost. The results highlighted the movement of data within and between organisations as a key factor. Data movement can be hindered by numerous technical barriers, but also by other challenges arising from social aspects of the organisation. These latter aspects are often harder to predict, and are ignored by many of the more common software engineering methodologies.In this paper, we propose data journey modelling, a new method aiming to predict places of high cost and risk when existing data needs to move to a new development. The method is lightweight and combines technical and social aspects, but relies only on information that is likely to be already known to key stakeholders, or will be cheap to acquire.To assess the effectiveness of our method, we conducted a retrospective evaluation in an NHS Foundation Trust hospital. Using the method, we were able to predict most of the points of high cost/risk that the hospital staff had identified, along with several other possible directions that the staff did not identify for themselves, but agreed could be promising.