Health Information Technology Evaluation Research Articles

The Telemedicine Experience in Primary Care Practices in the United States: Insights From Practice Leaders.

The need to rapidly implement telemedicine in primary care during the coronavirus disease 2019 (COVID-19) pandemic was addressed differently by various practices. Using qualitative data from semistructured interviews with primary care practice leaders, we aimed to report commonly shared experiences and unique perspectives regarding telemedicine implementation and evolution/maturation since March 2020. We administered a semistructured, 25-minute, virtual interview with 25 primary care practice leaders from 2 health systems in 2 states (New York and Florida) included in PCORnet, the Patient-Centered Outcomes Research Institute clinical research network. Questions were guided by 3 frameworks (health information technology evaluation, access to care, and health information technology life cycle) and involved practice leaders' perspectives on the process of telemedicine implementation in their practice, with a specific focus on the process of maturation and facilitators/barriers. Two researchers conducted inductive coding of qualitative data open-ended questions to identify common themes. Transcripts were electronically generated by virtual platform software. Twenty-five interviews were administered for practice leaders representing 87 primary care practices in 2 states. We identified the following 4 major themes: (1) the ease of telemedicine adoption depended on both patients' and clinicians' prior experience using virtual health platforms, (2) regulation of telemedicine varied across states and differentially affected the rollout processes, (3) visit triage rules were unclear, and (4) there were positive and negative effects of telemedicine on clinicians and patients. Practice leaders identified several challenges to telemedicine implementation and highlighted 2 areas, including telemedicine visit triage guidelines and telemedicine-specific staffing and scheduling protocols, for improvement.

Preparing for the Improving Medicare Post-Acute Care Transformation Act: Incorporating A Sociotechnical Perspective to Enhance a Health Information Technology Evaluation Framework Applied to Post-Acute Care.

The Improving Medicare Post-Acute Care Transformation Act, which mandates electronic sharing of standardized patient data by post-acute care clinical settings, will likely spur further health information technology adoption and evaluation. To support evaluation, the study objective was to clarify components of an evidence-based health information technology evaluation framework, Health Information Technology Reference-based Evaluation Framework, by using the framework in home healthcare and incorporating a sociotechnical perspective in the health information technology evaluation. With 36 observations among three diverse home healthcare agencies, researchers conducted a recorded think-aloud process as nurses documented the home healthcare admission in the EHR. Thematic analysis revealed 15 themes that provided clarification for almost one-third of Health Information Technology Reference-based Evaluation Framework components and added a new concept. All themes reflected a sociotechnical perspective. The new theme added to the Health Information Technology Reference-based Evaluation Framework reflected the sociotechnical perspective: routine use. We anticipate the enhanced Health Information Technology Reference-based Evaluation Framework to be used by evaluators from diverse disciplines, to further facilitate context-dependent health information technology adoption in post-acute care settings.

Staff perspectives on the usability of electronic patient records for planning and delivering dementia care in nursing homes: a multiple case study

BackgroundThe electronic patient record (EPR) has been introduced into nursing homes in order to facilitate documentation practices such as assessment and care planning, which play an integral role in the provision of dementia care. However, little is known about how the EPR facilitates or hinders these practices from the end-user’s perspective. Therefore, the objective of this qualitative study was to explore the usability issues associated with the EPR for assessment and care planning for people with dementia in nursing homes from a staff perspective.MethodsAn exploratory, qualitative research design with a multiple case study approach was used. Contextual Inquiry was carried out with a variety of staff members (n = 21) who used the EPR in three nursing homes situated in Belgium, Czech Republic and Spain. Thematic analysis was used to code interview data, with codes then sorted into a priori components of the Health Information Technology Evaluation Framework: device, software functionality, organisational support. Two additional themes, structure and content, were also added.ResultsStaff provided numerous examples of the ways in which EPR systems are facilitating and hindering assessment and care planning under each component, particularly for people with dementia, who may have more complex needs in comparison to other residents. The way in which EPR systems were not customisable was a common theme across all three homes. A comparison of organisational policies and practices revealed the importance of training, system support, and access, which may be linked with the successful adoption of the EPR system in nursing homes.ConclusionsEPR systems introduced into the nursing home environment should be customisable and reflect best practice guidelines for dementia care, which may lead to improved outcomes and quality of life for people with dementia living in nursing homes. All levels of nursing home staff should be consulted during the development, implementation and evaluation of EPR systems as part of an iterative, user-centred design process.

A Sociotechnical Framework for Safety-Related Electronic Health Record Research Reporting: The SAFER Reporting Framework.

Electronic health record (EHR)-based interventions to improve patient safety are complex and sensitive to who, what, where, why, when, and how they are delivered. Success or failure depends not only on the characteristics and behaviors of individuals who are targeted by an intervention, but also on the technical characteristics of the intervention and the culture and environment of the health system that implements it. Current reporting guidelines do not capture the complexity of sociotechnical factors (technical and nontechnical factors, such as workflow and organizational issues) that confound or influence these interventions. This article proposes a methodological reporting framework for EHR interventions targeting patient safety and builds on an 8-dimension sociotechnical model previously developed by the authors for design, development, implementation, use, and evaluation of health information technology. The Safety-related EHR Research (SAFER) Reporting Framework enables reporting of patient safety-focused EHR-based interventions while accounting for the multifaceted, dynamic sociotechnical context affecting intervention implementation, effectiveness, and generalizability. As an example, an EHR-based intervention to improve communication and timely follow-up of subcritical abnormal test results to operationalize the framework is presented. For each dimension, reporting should include what sociotechnical changes were made to implement an EHR-related intervention to improve patient safety, why the intervention did or did not lead to safety improvements, and how this intervention can be applied or exported to other health care organizations. A foundational list of research and reporting recommendations to address implementation, effectiveness, and generalizability of EHR-based interventions needed to effectively reduce preventable patient harm is provided. The SAFER Reporting Framework is not meant to replace previous research reporting guidelines, but rather provides a sociotechnical adjunct that complements their use.

Creating synthetic patient data to support the design and evaluation of novel health information technology.

To ensure that new health information technology supports its intended users, researchers and developers need to follow human-centered methods during all stages of the software development lifecycle, including early stage evaluations. These evaluations need to include realistic testing scenarios to ensure that they provide valuable and accurate feedback to system developers. However, obtaining realistic patient data to support these evaluations has many challenges, including the risk of re-identifying anonymized patients as well as the costs associated with connecting test systems with production ready clinical databases. Here we present a novel five-step process to create highly structured and realistic synthetic patient data to support the evaluation and comparison of early to middle stage health information technology prototypes. We applied this method to evaluate and compare three novel health information technology prototypes designed to support clinicians during the identification of high-priority patients when answering the question: "What patient should I see first?" Our novel approach fills an important gap in the evaluation of health information technology and assists designers in creating high-quality software that best supports its end users.

Evaluation of Health Information Technology - Key Elements in the Framework.

Evaluation of Health Information Technology - Key Elements in the Framework.

Using Mixed Methods in Health Information Technology Evaluation.

With the increasing adoption of interactive systems in healthcare, there is a need to ensure that the benefits of such systems are formally evaluated. Traditionally quantitative research approaches have been used to gather evidence on measurable outcomes of health technology. Qualitative approaches have also been used to analyze how or why particular interventions did or did not work in specific healthcare contexts. Mixed methods research provides a framework for carrying out both quantitative and qualitative approaches within a single research study. In this paper an international group of four informatics scholars illustrate some of the benefits and challenges of using mixed methods in evaluation. The diversity of the research experience provides a broad overview of approaches in combining robust analysis of outcome data with qualitative methods that provide an understanding of the processes through which, and the contexts in which, those outcomes are achieved. This paper discussed the benefits that mixed methods brought to each study.

Evidence-based Health Informatics Frameworks for Applied Use.

Health Informatics frameworks have been created surrounding the implementation, optimization, adoption, use and evaluation of health information technology including electronic health record systems and medical devices. In this contribution, established health informatics frameworks are presented. Important considerations for each framework are its purpose, component parts, rigor of development, the level of testing and validation its undergone, and its limitations. In order to understand how to use a framework effectively, it's often necessary to seek additional explanation via literature, documentation, and discussions with the developers.

A framework for contexual design and evaluation of health information technology.

Poor contextual fit is a significant cause of health information technology (HIT) implementation issues. While the need for better fit of HIT and context has been well described there is a shortcoming of approaches for how to do it. While the diversity of the contexts where HIT is used prevents us from designing HIT to fit all contexts, if we had better ways of understanding and modelling contexts we could design and evaluate HIT to better fit contexts of use. This paper addresses the above need by developing a framework consisting of a set of terminology and concepts for modelling contextual structures and behaviours to support HIT design. The framework provides a way of binding contextual considerations to allow us to better model contexts as part of HIT design and evaluation.

Big data and smart health strategies: findings from the health information systems perspective.

To summarize excellent current research in the field of Health Information Systems. Creation of a synopsis of the articles selected for the 2014 edition of the IMIA Yearbook. Four papers from international peer reviewed journals were selected and are summarized. Selected articles illustrate current research regarding the impact and the evaluation of health information technology and the latest developments in health information exchange.

Practice-centred evaluation and the privileging of care in health information technology evaluation.

BackgroundElectronic Patient Records (EPRs) and telemedicine are positioned by policymakers as health information technologies that are integral to achieving improved clinical outcomes and efficiency savings. However, evaluating the extent to which these aims are met poses distinct evaluation challenges, particularly where clinical and cost outcomes form the sole focus of evaluation design. We propose that a practice-centred approach to evaluation - in which those whose day-to-day care practice is altered (or not) by the introduction of new technologies are placed at the centre of evaluation efforts – can complement and in some instances offer advantages over, outcome-centric evaluation models.MethodsWe carried out a regional programme of innovation in renal services where a participative approach was taken to the introduction of new technologies, including: a regional EPR system and a system to support video clinics. An ‘action learning’ approach was taken to procurement, pre-implementation planning, implementation, ongoing development and evaluation. Participants included clinicians, technology specialists, patients and external academic researchers. Whilst undergoing these activities we asked: how can a practice-centred approach be embedded into evaluation of health information technologies?DiscussionOrganising EPR and telemedicine evaluation around predetermined outcome measures alone can be impractical given the complex and contingent nature of such projects. It also limits the extent to which unforeseen outcomes and new capabilities are recognised. Such evaluations often fail to improve understanding of ‘when’ and ‘under what conditions’ technology-enabled service improvements are realised, and crucially, how such innovation improves care.SummaryOur contribution, drawn from our experience of the case study provided, is a protocol for practice-centred, participative evaluation of technology in the clinical setting that privileges care. In this context ‘practice-centred’ evaluation acts as a scalable, coordinating framework for evaluation that recognises health information technology supported care as an achievement that is contingent and ongoing. We argue that if complex programmes of technology-enabled service innovation are understood in terms of their contribution to patient care and supported by participative, capability-building evaluation methodologies, conditions are created for practitioners and patients to realise the potential of technologies and make substantive contributions to the evidence base underpinning health innovation programmes.

Tree testing of hierarchical menu structures for health applications

To address the need for greater evidence-based evaluation of Health Information Technology (HIT) systems we introduce a method of usability testing termed tree testing. In a tree test, participants are presented with an abstract hierarchical tree of the system taxonomy and asked to navigate through the tree in completing representative tasks. We apply tree testing to a commercially available health application, demonstrating a use case and providing a comparison with more traditional in-person usability testing methods. Online tree tests (N=54) and in-person usability tests (N=15) were conducted from August to September 2013. Tree testing provided a method to quantitatively evaluate the information structure of a system using various navigational metrics including completion time, task accuracy, and path length. The results of the analyses compared favorably to the results seen from the traditional usability test. Tree testing provides a flexible, evidence-based approach for researchers to evaluate the information structure of HITs. In addition, remote tree testing provides a quick, flexible, and high volume method of acquiring feedback in a structured format that allows for quantitative comparisons. With the diverse nature and often large quantities of health information available, addressing issues of terminology and concept classifications during the early development process of a health information system will improve navigation through the system and save future resources. Tree testing is a usability method that can be used to quickly and easily assess information hierarchy of health information systems.

A task-fit measure of health information technology use

Rationale, aims and objectives: Introducing profession concerns into the evaluation of health information technology (HIT) use is an important and developing practice. A comprehensive evaluation should include the intellect elements of HIT use. This paper proposes a task-fit measure of HIT that integrates an information/knowledge quality scale into a validated judicious HIT use measure. It also presents some statistics that have implications for policy-making and curriculum development.Methods: Statistical analyses were performed on a subset of survey data. A structural equation modelling technique was applied to examine the associations among intent to use HIT, professional concerns and information/knowledge quality.Results: The statistical results show that altruism, autonomy, physician-patient relationship, (subconscious) autonomy, digestible information and medical history associate with each other to different extents. Only altruism and medical history show to be significant determinants of intent to use at P<0.001 and P<0.05 respectively. The scaled χ2 difference test shows that this model is not significantly different from the judicious HIT use model.Conclusion: The statistical results suggest that professional concerns, digestible information and person-related information are HIT use decision factors. Perhaps physicians may prefer HITs considered to be compatible with practising the science, humanism and ethics of medicine simultaneously. This research direction will potentially contribute to identifying the task-fit HITs and the corresponding policies for re-orientating medicine to be a science-using and compassionate practice in this eHealth era, thereby promoting the development of person-centered healthcare.

Clinical trials of health information technology interventions intended for patient use: Unique issues and considerations

Despite the proliferation of health information technology (IT) interventions, descriptions of the unique considerations for conducting randomized trials of health IT interventions intended for patient use are lacking. Our purpose is to describe the protocol to evaluate Pocket PATH (Personal Assistant for Tracking Health), a novel health IT intervention, as an exemplar of how to address issues that may be unique to a randomized controlled trial (RCT) to evaluate health IT intended for patient use. An overview of the study protocol is presented. Unique considerations for health IT intervention trials and strategies are described to maintain equipoise, to monitor data safety and intervention fidelity, and to keep pace with changing technology during such trials. The sovereignty granted to technology, the rapid pace of changes in technology, ubiquitous use in health care, and obligation to maintain the safety of research participants challenge researchers to address these issues in ways that maintain the integrity of intervention trials designed to evaluate the impact of health IT interventions intended for patient use. Our experience evaluating the efficacy of Pocket PATH may provide practical guidance to investigators about how to comply with established procedures for conducting RCTs and include strategies to address the unique issues associated with the evaluation of health IT for patient use.

Change in Adoption of Electronic Health Records by US Children’s Hospitals

OBJECTIVES To assess electronic health record (EHR) adoption and meaningful use among US children's hospitals through 2011 and compare these outcomes with adult hospitals and among subgroups of children's hospitals. We hypothesized that children's hospitals would show progress since our initial evaluation of health information technology (HIT) implementation in 2008. We identified children's hospitals using the membership directory of the Children's Hospital Association and analyzed their responses from 2008 to 2011 to the American Hospital Association's annual HIT survey. EHR adoption rates were determined by using previously specified definitions of the essential functionalities comprising an EHR. Achievement of meaningful use was evaluated based on hospitals' ability to fulfill 12 core meaningful use criteria. We compared these outcomes in 2011 between children's and adult hospitals and among subgroups of children's hospitals. Of 162 children's hospitals, 126 (78%) responded to the survey in 2011. The proportion of children's hospitals with an EHR increased from 21% (in 2008) to 59% (in 2011). In 2011, 29% of children's hospitals met the 12 core criteria in our meaningful use proxy measure. EHR adoption rates and meaningful use were significantly higher for children's hospitals than for adult hospitals as a whole but similar for children's and adult major teaching hospitals. Among children's hospitals, major teaching hospitals were significantly more likely to have an EHR. Children's hospitals have achieved substantial gains in HIT implementation although minor teaching and nonteaching institutions are not keeping pace.

Health information technology and health information exchange in New York State: New initiatives in implementation and evaluation

More research is needed to understand the effects of health information technology (HIT) and health information exchange (HIE) on quality, safety, efficiency, finances, consumers and providers in community-based settings. New York State is investing heavily in HIT and HIE adoption through the HEAL NY program. It has already provided $53 million in seed money and requires that grantee organizations match the funds. HITEC (The Health Information Technology Evaluation Collaborative) was established to measure systematically the effects of HIT and HIE on consumers, providers, health care quality, patient safety, public health, and financial return on investment in New York State, as no individual grantee is able to conduct cross-cutting evaluations. The results of these evaluations should inform decisions made by leaders in HIT and HIE in New York State and across the nation.