Improving validity and reliability in longitudinal case study timelines

Abstract

Management Information Systems researchers rely on longitudinal case studies to investigate a variety of phenomena such as systems development, system implementation, and information systems-related organizational change. However, insufficient attention has been spent on understanding the unique validity and reliability issues related to the timeline that is either explicitly or implicitly required in a longitudinal case study. In this paper, we address three forms of longitudinal timeline validity: time unit validity (which deals with the question of how to segment the timeline – weeks, months, years, etc.), time boundaries validity (which deals with the question of how long the timeline should be), and time period validity (which deals with the issue of which periods should be in the timeline). We also examine timeline reliability, which deals with the question of whether another judge would have assigned the same events to the same sequence, categories, and periods. Techniques to address these forms of longitudinal timeline validity include: matching the unit of time to the pace of change to address time unit validity, use of member checks and formal case study protocol to address time boundaries validity, analysis of archival data to address both time unit and time boundary validity, and the use of triangulation to address timeline reliability. The techniques should be used to design, conduct, and report longitudinal case studies that contain valid and reliable conclusions.