Measurement Feedback Systems Research Articles

Leveraging Health Information Technology to Meet The Joint Commission's Standard for Measurement-Based Care: A Case Study

Three decades of research have shown that routinely collecting patient-reported outcomes throughout treatment to inform clinical decision making or measurement-based care (MBC) can improve clinical outcomes, yet widespread adoption continues to be elusive. This article describes how a community behavioral health center addressed Element of Performance (EP) 1 of The Joint Commission's revised MBC standard using health information technology (HIT)-facilitated MBC and a comprehensive implementation plan grounded in the Consolidated Framework for Implementation Research. Across the initial 15-month implementation period, 96.8% of patients who had an intake evaluation also completed baseline measurements via an HIT known as a measurement feedback system (MFS), and 91.5% (78.6%-100%) completed at least one repeated measure. MFS reduces many of the logistical barriers of MBC, but implementation of MFS-facilitated MBC requires a comprehensive implementation plan that includes strategies to address barriers across all relevant domains for successful uptake.

Middle and High School Student Perspectives on Digitally-Delivered Mental Health Assessments and Measurement Feedback Systems.

Implementation of measurement-based care (MBC) by child-serving community mental health providers, particularly school-based providers, is low. To inform user-centered design of measurement feedback systems (MFSs) and MBC implementation more broadly, semi-structured interviews were conducted with 61 middle and high school students. Interviews explored student preferences for and perceived helpfulness of different assessment methods and use of MFS in counseling. Results indicate that student preference for digitally-administered assessment is equivocal, with preferences being influenced by student perceptions of the ease of use, impersonalization, and confidentiality. Students with exposure to the MFS found it helpful when used by their provider to share assessment feedback.

Measurement training and feedback system for implementation of evidence-based treatment for adolescent externalizing problems: protocol for a randomized trial of pragmatic clinician training

BackgroundInnovations in clinical training and support that enhance fidelity to evidence-based treatment (EBT) for adolescent behavior problems are sorely needed. This study will develop an online training system to address this gap: Measurement Training and Feedback System for Implementation (MTFS-I). Using procedures intended to be practical and sustainable, MTFS-I is designed to increase two aspects of therapist behavior that are fundamental to boosting EBT fidelity: therapist self-monitoring of EBT delivery, and therapist utilization of core techniques of EBTs in treatment sessions. This version of MTFS-I focuses on two empirically supported treatment approaches for adolescent conduct and substance use problems: family therapy and cognitive behavioral therapy (CBT).Methods/designMTFS-I expands on conventional measurement feedback systems for client outcomes by adding training in observational coding to promote EBT self-monitoring and focusing on implementation of EBT treatment techniques. It has two primary components. (1) The training component, delivered weekly in two connected parts, involves self-monitored learning modules containing brief clinical descriptions of core EBT techniques and mock session coding exercises based on 5–8 min video segments that illustrate delivery of core techniques. (2) The feedback component summarizes aggregated therapist-reported data on EBT techniques used with their active caseloads. MTFS-I is hosted online and requires approximately 20 min per week to complete for each treatment approach. This randomized trial will first collect data on existing delivery of family therapy and CBT techniques for youth in outpatient behavioral health sites (Baseline phase). It will then randomize site clinicians to two study conditions (Implementation phase): Training Only versus Training + Feedback + Consultation. Therapists will choose whether to train in family therapy, CBT, or both. Study aims will compare clinician performance across study phase and between study conditions on MTFS-I uptake, reliability and accuracy in EBT self-monitoring, and utilization of EBT techniques in treatment sessions (based on observer coding of audiotapes).DiscussionStudy contributions to implementation science and considerations of MTFS-I sustainability are discussed.Trial registrationClinicalTrials.gov, NCT03722654. Registered on 29 October 2018.

Multilevel Predictors of Case Managers' Assessment Administration Behavior in a Precursor to a Measurement Feedback System.

This study examines factors associated with administration of a treatment monitoring assessment measure (the Ohio Scales) in the context of a precursor to a measurement feedback system in a youth public mental health setting. 82% of all state case managers (N = 46) completed interviews and administered at least one Ohio Scale over a 12-month period. A multi-level model accounting for variance between both case managers and their administrative offices indicated that case manager characteristics (lower self-reported burnout, more experience) and monthly caseload characteristics (fewer active cases, younger average age of youth, and increased time since initial administration) predicted increased monthly administration proportions.

Measurement Training and Feedback System for Implementation of family-based services for adolescent substance use: protocol for a cluster randomized trial of two implementation strategies

BackgroundThis article describes a study protocol for testing the Measurement Training and Feedback System for Implementation (MTFS-I) and comparing two implementation strategies for MTFS-I delivery. MTFS-I is a web-based treatment quality improvement system designed to increase the delivery of evidence-based interventions for behavioral health problems in routine care settings. This version of MTFS-I focuses on family-based services (FBS) for adolescent substance use. FBS, comprising both family participation in treatment and family therapy technique use, have achieved the strongest evidence base for adolescent substance use and are a prime candidate for upgrading treatment quality in outpatient care. For FBS to fulfill their potential for widespread dissemination, FBS implementation must be bolstered by effective quality procedures that support sustainable delivery in usual care.Methods/designAdapted from measurement feedback systems for client outcomes, MTFS-I contains three synergistic components: (a) weekly reporter training modules to instruct therapists in reliable post-session self-reporting on FBS utilization; (b) weekly mock session videos of FBS interventions (5–8 min) for supportive training in, and practice coding of, high-quality FBS; and (c) monthly feedback reports to therapists and supervisors displaying aggregated data on therapist-reported FBS use. MTFS-I is hosted online and requires approximately 20 min per week to complete. The study will experimentally compare two well-established implementation strategies designed to foster ongoing MTFS-I usage: Core Training, consisting of two 3-h training sessions focused on FBS site mapping, selecting FBS improvement goals, and sustaining MTFS-I, followed by routine remote technical assistance; and Core + Facilitation, which boosts Core Training sessions with collaborative phone-based clinical consultation and on-site facilitation meetings for 1 year to promote FBS goal achievement. The study design is a cluster randomized trial testing Core Training versus Core + Facilitation in ten substance use treatment clinics. Study aims will compare conditions on MTFS-I uptake, FBS delivery (based on therapist-report and observational data), and 1-year client outcomes.DiscussionStudy contributions to implementation science and considerations of MTFS-I sustainability are discussed.Trial registrationClinicalTrials.govNCT03342872. Registered 10 November 2017

Community Study of Outcome Monitoring for Emotional Disorders in Teens (COMET): A comparative effectiveness trial of a transdiagnostic treatment and a measurement feedback system

Emotional disorders, encompassing a range of anxiety and depressive disorders, are the most prevalent and comorbid psychiatric disorders in adolescence. Unfortunately, evidence-based psychosocial therapies typically focus on single disorders, are rarely adopted by community mental health center clinicians, and effect sizes are modest. This article describes the protocol for a comparative effectiveness study of two novel interventions designed to address these challenges. The first intervention is a transdiagnostic treatment (the Unified Protocol for Transdiagnostic Treatment of Emotional Disorders in Adolescents, UP-A), a promising new approach that uses a small number of common strategies to treat a broad range of emotional disorders, and their underlying shared emotional vulnerabilities. The second intervention is a standardized measurement feedback system, the Youth Outcomes Questionnaire (YOQ), designed to improve clinical decision making using weekly symptom and relational data. The three study arms are treatment as usual (TAU), TAU plus the YOQ (TAU+), and UP-A (used in combination with the YOQ). The primary aims of the study are to [1] compare the effects of the UP-A and TAU+ to TAU in community mental health clinics, [2] to isolate the effects of measurement and feedback by comparing the UP-A and TAU+ condition, and [3] to examine the mechanisms of action of both interventions. Design considerations and study methods are provided to inform future effectiveness research.

A Digital Feedback System to Support Implementation of Measurement-Based Care by School-Based Mental Health Clinicians

Evidence supports the utility of measurement-based care (MBC) to improve youth mental health outcomes, but clinicians rarely engage in MBC practices. Digital measurement feedback systems (MFS) may reflect a feasible strategy to support MBC adoption and sustainment. This pilot study was initiated to evaluate the impact of a MFS and brief consultation supports to facilitate MBC uptake and sustainment among mental health clinicians in the education sector, the most common mental health service delivery setting for youth. Following an initial training in MBC, 14 clinicians were randomized to either a digital MFS and brief consultation supports or control. Baseline ratings of MBC attitudes, skill, and use were collected. In addition, daily assessment ratings tracked 2 core MBC practices (i.e., assessment tool administration, provision of feedback) over a 6-month follow-up period. Clinicians in the MFS condition demonstrated rapid increases in both MBC practices, whereas the control group did not significantly change. For clinicians in the MFS group, consultation effects were significant for feedback and approached significance for administration. Over the follow-up period, average decreases in the current study were moderate with only 1 of the 2 outcome variables (administration) decreasing significantly. Inspection of individual clinician trajectories revealed substantial within-group trend variation. MFS may represent an effective MBC implementation strategy beyond initial training, although individual clinician response is variable. Identifying feasible and impactful implementation strategies is critical given the ability of MBC to support precision health care.

14.3 Leveraging Technology to Support Training, Treatment Adherence, and Outcomes

Use of fidelity and outcome monitoring in the dissemination of evidence-based psychosocial treatments (EBTs) in community settings has become gold standard in taking EBTs to scale. Sustainability of EBT posttraining is high stakes to agencies and providers. Over time, adherence to EBTs tends to drift, thereby diluting program effectiveness. To support clinicians in implementing EBTs with fidelity, effective measurement methods that are feasible within service settings, as well as a feedback system coupled with supportive quality improvement, are needed. Strong interest lies in using technology to support real-time fidelity assessment and the collection of client-level data to document service dose and content. Technology-based outcome monitoring can enhance the integration of outcome measurement into care, as it enables rapid use of outcome reports, decision support tools, and measurement feedback systems. Mandates in the Affordable Care Act will require practicing clinicians to consistently use EBTs and demonstrate quality of care through systematic assessment of client outcomes. NIMH small business innovation research pilot data will be presented in an online implementation support system (EBT Performance Outcome Platform, EBT-POP, 2.0) for trauma-focused (TF)-CBT that supports high-fidelity clinical service delivery and ongoing monitoring of performance and outcomes at the case and aggregate level.

System-Wide Implementation of Routine Outcome Monitoring and Measurement Feedback System in a National Network of Operational Stress Injury Clinics.

This manuscript reviews the development and implementation process of the Client Reported Outcome Monitoring Information System in the VA Canada Operational Stress Injury National Network, and reports on outcomes of an evaluation to assess usage, barriers, and facilitators to implementation. The majority of clinicians reported regular use of routine outcomes monitoring, although objective data suggested somewhat lower actual use. In general, clinicians endorsed all barriers and most facilitators as influencing their use of routine outcomes monitoring in a minor way. However, users and non-users differed in their endorsement of facilitators and barriers. Implications for research and implementation efforts are discussed.

Facilitators and Barriers of Implementing a Measurement Feedback System in Public Youth Mental Health.

This study examines implementation facilitators and barriers of a statewide roll-out of a measurement feedback system (MFS) in a youth public mental health system. 76% of all state care coordinators (N=47) completed interviews, which were coded via content analysis until saturation. Facilitators (e.g., recognition of the MFS's clinical utility) and barriers (e.g., MFS's reliability and validity) emerged paralleling the Exploration, Adoption/Preparation, Implementation, and Sustainment framework outlined by Aarons et al. (Adm Policy Mental Health Mental Health Serv Res, 38:4-23, 2011). Sustainment efforts may leverage innovation fit, individual adopter, and system related facilitators.

Capabilities and Characteristics of Digital Measurement Feedback Systems: Results from a Comprehensive Review.

Measurement feedback systems (MFS) are a class of health information technology (HIT) that function as an implementation support strategy for integrating measurement based care or routine outcome monitoring into clinical practice. Although many MFS have been developed, little is known about their functions. This paper reports findings from an application of health information technology-academic and commercial evaluation (HIT-ACE), a systematic and consolidated evaluation method, to MFS designed for use in behavioral healthcare settings. Forty-nine MFS were identified and subjected to systematic characteristic and capability coding. Results are presented with respect to the representation of characteristics and capabilities across MFS.

Evolution of Decision Making Processes in Command during Irregular Warfare Operations

Stability operations and irregular war are increasingly complex and continually changing constructs which are no longer served by the traditional processes of the western military forces. This essay examines the significant work that has been carried out in recent years in an effort to develop new problem framing constructs that hope to offer the commander processes for utilizing and manipulating complex adaptive systems. What is becoming apparent is that the process of developing frameworks will not be of value to the military commander unless these frameworks can be embedded in the decision making process at an instinctive and experiential level. For the past 5 years The technical cooperation programme (TTCP) has been operating with the objective of harnessing complexity within the defense arena. The introduction of unknown risks associated with the incomplete and shifting information, and continuous readjustments due to the multi cell structure of irregular warfare would create significant base level noise in an adaptive system framework. This low noise introduced into the operational art of command would have the potential to damage the measurement schemas and feedback systems of the immediate adaptive framework and alter the responsiveness and sustainability of advancements based on these decisions. At present, significant work has been done to translate the science of complex adaptive systems into tools the commander can incorporate into his decision making process in high pressure scenarios. This has not yet been successful.

Health Information Technologies-Academic and Commercial Evaluation (HIT-ACE) methodology: description and application to clinical feedback systems.

BackgroundHealth information technologies (HIT) have become nearly ubiquitous in the contemporary healthcare landscape, but information about HIT development, functionality, and implementation readiness is frequently siloed. Theory-driven methods of compiling, evaluating, and integrating information from the academic and commercial sectors are necessary to guide stakeholder decision-making surrounding HIT adoption and to develop pragmatic HIT research agendas. This article presents the Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE) methodology, a structured, theory-driven method for compiling and evaluating information from multiple sectors. As an example demonstration of the methodology, we apply HIT-ACE to mental and behavioral health measurement feedback systems (MFS). MFS are a specific class of HIT that support the implementation of routine outcome monitoring, an evidence-based practice.ResultsHIT-ACE is guided by theories and frameworks related to user-centered design and implementation science. The methodology involves four phases: (1) coding academic and commercial materials, (2) developer/purveyor interviews, (3) linking putative implementation mechanisms to hit capabilities, and (4) experimental testing of capabilities and mechanisms. In the current demonstration, phase 1 included a systematic process to identify MFS in mental and behavioral health using academic literature and commercial websites. Using user-centered design, implementation science, and feedback frameworks, the HIT-ACE coding system was developed, piloted, and used to review each identified system for the presence of 38 capabilities and 18 additional characteristics via a consensus coding process. Bibliometic data were also collected to examine the representation of the systems in the scientific literature. As an example, results are presented for the application of HIT-ACE phase 1 to MFS wherein 49 separate MFS were identified, reflecting a diverse array of characteristics and capabilities.ConclusionsPreliminary findings demonstrate the utility of HIT-ACE to represent the scope and diversity of a given class of HIT beyond what can be identified in the academic literature. Phase 2 data collection is expected to confirm and expand the information presented and phases 3 and 4 will provide more nuanced information about the impact of specific HIT capabilities. In all, HIT-ACE is expected to support adoption decisions and additional HIT development and implementation research.

At the Intersection of Health Information Technology and Decision Support: Measurement Feedback Systems…and Beyond

We select and comment on concepts and examples from the target articles in this special issue on measurement feedback systems, placing them in the context of some of our own insights and ideas about measurement feedback systems, and where those systems lie at the intersection of technology and decision making. We contend that, connected to the many implementation challenges relevant to many new technologies, there are fundamental design challenges that await a more elaborate specification of the clinical information and decision models that underlie these systems. Candidate features of such models are discussed, which include referencing multiple evidence bases, facilitating observed and expected value comparisons, fostering collaboration, and allowing translation across multiple ontological systems. We call for a new metaphor for these technologies that goes beyond measurement feedback and encourages a deeper consideration of the increasingly complex clinical decision models needed to manage the uncertainty of delivering clinical care.

Challenges in the Implementation of Measurement Feedback Systems.

This commentary on the articles published in the special section on the development and implementation of measurement feedback systems (MFSs) discusses three challenging themes in the process of MFS implementation: design and planning, organizational context, and sustainability and unintended consequences. It is argued that the implementation of MFSs is complex, but is an important step in improving outcomes in routine care for children and young persons.

Annual Research Review: Building a science of personalized intervention for youth mental health.

Within the past decade, health care service and research priorities have shifted from evidence-based medicine to personalized medicine. In mental health care, a similar shift to personalized intervention may boost the effectiveness and clinical utility of empirically supported therapies (ESTs). The emerging science of personalized intervention will need to encompass evidence-based methods for determining which problems to target and in which order, selecting treatments and deciding whether and how to combine them, and informing ongoing clinical decision-making through monitoring of treatment response throughout episodes of care. We review efforts to develop these methods, drawing primarily from psychotherapy research with youths. Then we propose strategies for building a science of personalized intervention in youth mental health. The growing evidence base for personalizing interventions includes research on therapies adapted for specific subgroups; treatments targeting youths' environments; modular therapies; sequential, multiple assignment, randomized trials; measurement feedback systems; meta-analyses comparing treatments for specific patient characteristics; data-mining decision trees; and individualized metrics. The science of personalized intervention presents questions that can be addressed in several ways. First, to evaluate and organize personalized interventions, we propose modifying the system used to evaluate and organize ESTs. Second, to help personalizing research keep pace with practice needs, we propose exploiting existing randomized trial data to inform personalizing approaches, prioritizing the personalizing approaches likely to have the greatest impact, conducting more idiographic research, and studying tailoring strategies in usual care. Third, to encourage clinicians' use of personalized intervention research to inform their practice, we propose expanding outlets for research summaries and case studies, developing heuristic frameworks that incorporate personalizing approaches into practice, and integrating personalizing approaches into service delivery systems. Finally, to build a richer understanding of how and why treatments work for particular individuals, we propose accelerating research to identify mediators within and across RCTs, to isolate mechanisms of change, and to inform the shift from diagnoses to psychopathological processes. This ambitious agenda for personalized intervention science, although challenging, could markedly alter the nature of mental health care and the benefit provided to youths and families.

Development and Piloting of a Classroom-Focused Measurement Feedback System.

The present study used a community partnered research method to develop and pilot a classroom-focused measurement feedback system (MFS) for school mental health providers to support teachers' use of effective universal and target classroom practices related to student emotional and behavioral issues. School personnel from seven urban elementary and middle school classrooms participated. Phase I involved development and refinement of the system through a baseline needs assessment and rapid-cycle feedback. Phase II involved detailed case study analysis of pre-to-post quantitative and implementation process data. Results suggest that teachers who used the dashboard along with consultation showed improvement in observed classroom organization and emotional support. Results also suggest that MFS use was tied closely to consultation dose, and that broader support at the school level was critical. Classroom-focused MFSs are a promising tool to support classroom improvement, and warrant future research focused on their effectiveness and broad applicability.

Implementing a Measurement Feedback System: A Tale of Two Sites.

A randomized experiment was conducted in two outpatient clinics evaluating a measurement feedback system called contextualized feedback systems. The clinicians of 257 Youth 11-18 received feedback on progress in mental health symptoms and functioning either every 6months or as soon as the youth's, clinician's or caregiver's data were entered into the system. The ITT analysis showed that only one of the two participating clinics (Clinic R) had an enhanced outcome because of feedback, and only for the clinicians' ratings of youth symptom severity on the SFSS. A dose-response effect was found only for Clinic R for both the client and clinician ratings. Implementation analyses showed that Clinic R had better implementation of the feedback intervention. Clinicians' questionnaire completion rate and feedback viewing at Clinic R were 50% higher than clinicians at Clinic U. The discussion focused on the differences in implementation at each site and how these differences may have contributed to the different outcomes of the experiment.

Implementing a Measurement Feedback System in Community Mental Health Clinics: A Case Study of Multilevel Barriers and Facilitators.

Measurement feedback systems (MFSs) have been proposed as a means of improving practice. The present study examined the implementation of a MFS, the Contextualized Feedback System (CFS), in two community-based clinic sites. Significant implementation differences across sites provided a basis for examining factors that influenced clinician uptake of CFS. Following the theoretical implementation framework of Aarons et al. (Adm Policy Mental Health Mental Health Serv Res 38(1):4-23, 2011), we coded qualitative data collected from eighteen clinicians (13 from Clinic U and 5 from Clinic R) who participated in semi-structured interviews about their experience with CFS implementation. Results suggest that clinicians at both clinics perceived more barriers than facilitators to CFS implementation. Interestingly, clinicians at the higher implementing clinic reported a higher proportion of barriers to facilitators (3:1 vs. 2:1); however, these clinicians also reported a significantly higher level of organizational and leadership supports for CFS implementation. Implications of these findings are discussed.

The Contextualized Technology Adaptation Process (CTAP): Optimizing Health Information Technology to Improve Mental Health Systems.

Health information technologies have become a central fixture in the mental healthcare landscape, but few frameworks exist to guide their adaptation to novel settings. This paper introduces the contextualized technology adaptation process (CTAP) and presents data collected during Phase 1 of its application to measurement feedback system development in school mental health. The CTAP is built on models of human-centered design and implementation science and incorporates repeated mixed methods assessments to guide the design of technologies to ensure high compatibility with a destination setting. CTAP phases include: (1) Contextual evaluation, (2) Evaluation of the unadapted technology, (3) Trialing and evaluation of the adapted technology, (4) Refinement and larger-scale implementation, and (5) Sustainment through ongoing evaluation and system revision. Qualitative findings from school-based practitioner focus groups are presented, which provided information for CTAP Phase 1, contextual evaluation, surrounding education sector clinicians' workflows, types of technologies currently available, and influences on technology use. Discussion focuses on how findings will inform subsequent CTAP phases, as well as their implications for future technology adaptation across content domains and service sectors.