Too complex to test? Using exploratory trials to identify relevant contexts and mechanisms prior to larger scale evaluations

Abstract

Introduction : Designing appropriate studies for evaluating complex interventions, like eHealth solutions to support integrated care, remains a methodological challenge. Although randomized control trials RCTs persist as the gold standard, they may not always be appropriate. Identification of contexts and mechanisms that can influence outcomes of an intervention typical in realist evaluations can help to identify appropriate methods for subsequent evaluation studies. Exploratory trials can be used to uncover these determinants, occurring at macro policy/system, meso organizational/team and micro person/provider levels in complex interventions prior to designing a full evaluation plan. Methods : An exploratory trial of the electronic Patient Reported Outcome ePRO tool was conducted to uncover contexts, mechanisms and relevant outcomes prior to full-scale evaluation. ePRO is a mobile application and portal designed to support goal-oriented care in inter-disciplinary primary health care practices. The four month trial was conducted in two practices in Toronto, Canada. Patients were randomized into control and intervention groups and compared pre and post on outcome measures quality of life and activation. Semi-structured interviews were conducted with providers and patients in the intervention group to capture relevant contexts, mechanisms and outcomes. Results : Eight providers and 16 patients 7-control, 9-intervention participated in the study. Outcomes were captured using the AQoL-4D and PAM surveys; analyzed using descriptive statistics. Interview transcripts were analyzed using interpretive descriptive methods. There was little change in outcome measures, with a high baseline, suggesting patients were higher-functioning than anticipated. Qualitative data highlighted factors for study consideration. Micro level factors included: participant characteristics, comfort with technology, training, and patient-provider relationship. Meso level factors included: provider workflows and norms, and organizational resources and culture. Macro level factors included: the technology policy environment, and system structure. Notably the research context and processes themselves were also influential. Enabling mechanisms occurred mainly at the micro level around motivation and readiness to set goals, and feelings regarding technology. Discussion : Multiple contextual variables and mechanisms were found to be potentially influential on study outcomes. Identifying relevant theories has been suggested as a way to address context and mechanisms in large scale control studies. In the ePRO example, we see the potential for dozens of potentially relevant theories ranging from individual level behaviour change theories, to organizational level innovation adoption theories. Conclusion : The number of complex, overlapping influencing variables suggests that complex interventions like ePRO may not be well suited to controlled evaluation designs. Realist, mixed-methods, and case studies which adopt ethnographic and narratives approaches may be more appropriate. Lessons Learned : Context variables and mechanisms can occur at multiple levels and may change over time of the intervention. Limitations : Although our analysis methods were informed by realist approaches, the study was not set up as a realist evaluation. Opportunities for observation and follow-up that would have allowed for more in-depth exploration may have been missed. Suggestions for future research : Studies of complex interventions, liked eHealth enabled models of integrated care, should consider alternatives to RCT approaches when “controlling” all relevant variables is unlikely. A case-study ethnographic approach will be taken as part of the next phase of ePRO evaluation.