Evaluation Of Health Information Systems Research Articles

A qualitative analysis of the key contextual factors for the scaling-up of the ADLIFE toolbox in the Basque Health Service

Introduction: The translation of new digital solutions in different settings requires a preliminary analysis of contextual factors. The results might contribute to the development of strategies addressing barriers, for a successful implementation.
 ADLIFE is an EU funded project to improve the quality of life of older people with advanced chronic diseases by providing integrated personalised care via a toolbox including innovative digital solutions. The contextual factors of the participating organizations were evaluated prior to implementing this toolbox to identify the most relevant ones for the translation of this innovation action into routine practice.
 To structure this evaluation process, a bespoke framework for implementation assessment was developed, based on the Health Information System evaluation framework HOT-fit and the Consolidated Framework for Implementation Research (CFIR). These frameworks provide a structured and systematic way to identify constructs influencing the implementation of ADLIFE.
 This work presents the analysis of the contextual factors that would influence the uptake and effectiveness of ADLIFE tools in the Basque Health Service.
 Methods: Semi-structured interviews were conducted with different stakeholders before the implementation of the ADLIFE toolbox. A standardized set of questions was utilized, based on the main dimensions of the developed framework (human, organization, technology).
 These interviews were conducted online. They were recorded, transcribed, pseudonymized and subsequently analyzed using qualitative content analysis. These analyses were conducted using the software MAXQDA.
 Results: A total of 9 interviews were conducted: 2 primary care physicians, 2 specialized care physicians, 2 nurses, 2 IT staff and a manager.
 Through the implementation of the ADLIFE toolbox, interviewees expect improvements in communication between professionals and with patients, as well as facilitation of patient empowerment.
 Training about the new systems and time to get used to them were considered essential, which would help to overcome the digital gap that currently influences the lack of adherence to this type of intervention.
 Finally, direct linkage between the existing and the new systems were considered key to motivate professionals to participate and to facilitate the integration of patient care.
 Discussion and conclusions: The scaling up of the ADLIFE toolbox in the Basque Health Service requires it to be integrated into the digital systems currently used for the management of the Electronic Health Record. ADLIFE can serve to promote an improvement, not only in quality of life, but also in the quality of care, provided that professionals who care for these patients are involved.
 This knowledge will help to establish the most effective strategies to facilitate the intervention in the Basque Health Service.
 Acknowledgements
 We are grateful for the participation of the professionals who were interviewed.
 ADLIFE (GA 875209) received funding from the EU Horizon 2020. The authors thank all partners within ADLIFE.

Identification and classification of indicators for evaluating health information systems

BackgroundSeveral evaluation methods are used to determine the advantages and disadvantages of healthcare information systems and their contribution to attaining organizational goals. Despite the existence of many evaluation frameworks, there is no comprehensive set of indicators that evaluate different dimensions of information systems. This study aimed to develop a set of indicators for evaluating health information systems. MethodsThis research was conducted in three phases. First, based on a literature review of PubMed, Web of Science, Scopus, and Embase databases, studies using the health information system evaluation methods were extracted. Second, consecutive focus group meetings were held with scientific and executive experts to discuss the list of evaluation indicators extracted from the studies. In these meetings, the experts agreed on including, removing, adding, combining, and grouping the indicators. Third, the indicators were weighted using the Analytical Network Process (ANP) method, and the set of evaluation indicators was finalized. ResultsThe review of 177 relevant articles resulted in the extraction of 360 indicators. During the focus group meetings, 174 overlapping and duplicate indicators were eliminated and 61 indicators were added to the model based on experts’ suggestions. The remaining 247 indicators were classified into a four-level hierarchy. The final set consisted of 4 dimensions, 16 criteria, 47 markers, and 180 indicators. ConclusionWe developed a comprehensive general set of indicators that helps researchers, designers, and developers of health information systems to evaluate different dimensions of these systems. This set can also be used to improve the design of relevant systems.

Usability evaluation of emergency information systems in educational hospitals in Kerman, Iran

BackgroundSmart and practical health information systems and applications with fewer errors are crucial for healthcare facilities. One method that ensures the proper design of health information systems (HIS) and applications is usability evaluation.ObjectiveThis study aimed to evaluate the usability of the emergency information systems used at the emergency departments of four educational hospitals in Kerman, Iran.MethodThis study was conducted in two phases. In the first phase, the information systems' errors and shortages were identified using a semi-structured questionnaire by users (nurses and the IT staff). In the second phase, based on the results of the first phase, two questionnaires were designed for each group of users to their opinions about the usability of the emergency information systems.ResultsThe average score of “reducing and facilitating user’s daily activities” was significantly different among hospitals (p = 0.03). Shahid Beheshti Hospital obtained the lowest usability score (17.5), and Afzalipour Hospital received the highest usability score (21.75). Moreover, the average score in “use of the HIS” for nurses and IT staff was 2.93 and 3.54 on a scale of 5, respectively.ConclusionUsability evaluation of health information systems is essential to ensure that these systems provide sufficient and accurate information and requirements for users and health care providers. Also, modifying health information systems based on the user views and expectations improves the quality of the system and user-system Interaction.

Health information system evaluation tools: a scoping review and tool development

Health information system evaluation tools: a scoping review and tool development

Health Information Systems Evaluation Criteria: Overview of Systematic Reviews

Introduction: Health information systems play an important role in improving the quality of patient care and patient safety. to ensure their effectiveness and efficiency, they need to be evaluated. Although HIS evaluation has been investigated in many studies, there is no consensus on which aspects of HIS to evaluate. The aim of this study is to identify the indicators for the evaluation of health information systems and to provide an overview of the criteria devised and studies conducted.Methods: An umbrella review was performed exploring databases PubMed, Science direct, Web of Science, Science, and IEEE while following the PRISMA protocol. Articles were reviewed by two authors independently using the covidence tool to check the inclusion criteria and to extract the data items. Risk of bias was assessed using ROBIS and AMSTAR.Result: All included studies showed a high risk of bias according to ROBIS criteria. The extracted evaluation criteria were classified into 13 categories. Most of the studies believe that a more reliable and standardized tool is needed for the evaluation of health information systems. Two studies mentioned that surveys and questionnaires were the most commonly used method for evaluation of the systems. Summative evaluation was the most used method in two studies and the least used method in another one study.Conclusion: All the included studies had high risk of bias. Accordingly, further research and evidence is needed in this field. Most of the studies highlighted the need for more reliable and standardized tools for evaluation of health information systems.

Evaluasi Sistem Informasi Kesehatan dengan Pendekatan Health Metrics Network di Dinas Kesehatan Kota Pagar Alam

Data and information are strategic resources in health development and management. A health information system is considered effective if it can give information support in a decision making process through all levels. Health information system evaluation aims to ensure the efficiency of a health information system and its ability to provide quality information. The study aimed to determine description of health information system evaluation results in Pagar Alam Public Health Office. The study was carried on by using Health Metric Network (HMN) as tools. The study is a descriptive evaluative design with mixed method approach. The study subjects were 9 Health Information System workers in Pagar Alam Public Health Office. Study data was collected by doing interview and observation. Data analysis was done by using Health Metric Network (HMN) version 4.00. The result of health information system evaluation using six components within HMN instrument was within the range of 41-60% which means that the system exists but inadequate. The conclusion of this study is that health information system in Pagar Alam Public Health Office has been carried on well enough, but requires improvements in its data management

Evaluation of health information systems with ISO 9241-10 standard: A systematic review and meta-analysis

This study aimed to summarize available data about health information system evaluation using the ISO 9241–110 (previously 9241–10) standard. A systematic review of literature from January 2000 to November 2020 was conducted in PubMed, Scopus, ScienceDirect, Web of Science, and Google Scholar as well as two Iranian national databases, Scientific Information Database and Magiran. Pooled estimates were calculated using random-effects meta-analysis. Data analyses were conducted using Stata version 14. From total of 179 studies subjected to initial screening, fifteen studies were eligible to enter the meta-analysis. The pooled mean estimates of the health information systems' overall scores (3.11; 95% confidence interval [CI], 2.82–3.41), suitability for individualization (3.82; 95% CI, 2.59–3.04), and controllability (3.82; 95% CI, 2.91–3.47) commonly had better performance compared with other dimensions. High heterogeneity was seen among the seven dimensions ( I 2 ≥ 98.6%, p < .001). According to the viewpoints of HIS users, suitability for individualization (3.82; 95% CI, 2.59–3.04) and controllability (3.82; 95% CI, 2.91–3.47) had better performance compared with other dimensions, and the results of the subgroup meta-analysis confirmed this difference ( z = 4.75, p < .001). For each dimension, Egger's and Begg's tests showed no publication bias ( p > .1). Most of the studies used the ISO 9241-110 standard as a summative assessment for evaluation of the usability of the HIS and results indicated the relative success of the evaluated systems based on this standard. • ISO 9241 part 10 is a valid standard for formative and summative evaluation of health information systems. • Most of the studies that have used the ISO9241 part 10 have focused on hospital information systems. • Few evaluations have been conducted using the formative version of ISO 9241 part 10. • Most of the evaluation studies have used the summative approach of the ISO 9241 part 10. • Because of the importance, it is suggested that researchers consider formative approaches.

EVALUATION OF HEALTH INFORMATION SYSTEMS IN THE HEALTH MEDICAL RECORD BIROBULI PALU

ABSTRACT At this time, information technology is one of technology that is growing rapidly, with the advancement of information technology, data or information that provided can be fast to access, efficient and accurate. For example the results of advances in information technology is the development of the Internet network that allows all mankind worldwide using data provided or connected in the network together, meanwhile for Community Health Centers of Birobuli, data entry system still done manually, only BPJS already using a computerized system, so this study aims to determine the evaluation of health information systems at Community Health Centers of Birobuli Palu in medical record room.The method that used is qualitative, qualitative sense is depth research on employee community health center about training, processing SIK, activities program and benefits as well as developments SIK in community health centers of Birobuli Palu. The number of informants that used was 7 people of one key informant, three informant regular and three additional informants. The analysis of data using analytical approach (analysis content) with the matrix technique where the information was obtained on though in the table. It refers that input (for an educational background in the medical record average DIII of Public Health and for training of SIK at the community health center have never done the training because which provide training was from the Department of Health. Process (in data processing is done in two ways: manually and computerized, with a network system used is a LAN network, and for the delivery of report conducted by the online system and manual system). Output (many benefits of SIK for community health centers, so that the expected for community health center can immediately apply the SIMPUS program for SIK and provide training to SIK employees).The conclusion of this study is the application of SIMPUS program hasn’t been implemented at the community health center of Birobuli Palu because the community health center employee have never been training that given by the Department of Health. Expected to governments and health authorities can implement and optimize the use of SIK adequately in order to achieve maximum health services Keywords: Input, Process, Output, Health information systems

Performance and cost evaluation of health information systems using micro-costing and discrete-event simulation.

Innovation and health-care funding reforms have contributed to the deployment of Information and Communication Technology (ICT) to improve patient care. Many health-care organizations considered the application of ICT as a crucial key to enhance health-care management. The purpose of this paper is to provide a methodology to assess the organizational impact of high-level Health Information System (HIS) on patient pathway. We propose an integrated performance evaluation of HIS approach through the combination of formal modeling using the Architecture of Integrated Information Systems (ARIS) models, a micro-costing approach for cost evaluation, and a Discrete-Event Simulation (DES) approach. The methodology is applied to the consultation for cancer treatment process. Simulation scenarios are established to conclude about the impact of HIS on patient pathway. We demonstrated that although high level HIS lengthen the consultation, occupation rate of oncologists are lower and quality of service is higher (through the number of available information accessed during the consultation to formulate the diagnostic). The provided method allows also to determine the most cost-effective ICT elements to improve the care process quality while minimizing costs. The methodology is flexible enough to be applied to other health-care systems.

Integrated AHP-TOPSIS Method for the Assessment of Health Management Information Systems Efficiency

Abstract The aim of this paper is to analyse and estimate the efficiency of health information systems in the provision of health care services. The evaluation of health information systems is conducted in the case of three most widely used softwares in the electronic health care. This evaluation is based on multi-criteria analysis of the health information systems efficiency using the AHP-TOPSIS method. This method, based on common attributes and their respective values for all three software solutions, individually determines the best rated software solution. Top rated software solution of electronic health care is not necessarily the best for the implementation and development, given the fact that each health care organisation has its own characteristics. Functional and evolutionary-minded hardware and software applicative infrastructure contributes to the consistency of electronic health concept that all system users provides a comfortable software solutions, which ultimately leads to the timely and quick medical services in real time.

Evaluating Health Information Systems Using Ontologies.

BackgroundThere are several frameworks that attempt to address the challenges of evaluation of health information systems by offering models, methods, and guidelines about what to evaluate, how to evaluate, and how to report the evaluation results. Model-based evaluation frameworks usually suggest universally applicable evaluation aspects but do not consider case-specific aspects. On the other hand, evaluation frameworks that are case specific, by eliciting user requirements, limit their output to the evaluation aspects suggested by the users in the early phases of system development. In addition, these case-specific approaches extract different sets of evaluation aspects from each case, making it challenging to collectively compare, unify, or aggregate the evaluation of a set of heterogeneous health information systems.ObjectivesThe aim of this paper is to find a method capable of suggesting evaluation aspects for a set of one or more health information systems—whether similar or heterogeneous—by organizing, unifying, and aggregating the quality attributes extracted from those systems and from an external evaluation framework.MethodsOn the basis of the available literature in semantic networks and ontologies, a method (called Unified eValuation using Ontology; UVON) was developed that can organize, unify, and aggregate the quality attributes of several health information systems into a tree-style ontology structure. The method was extended to integrate its generated ontology with the evaluation aspects suggested by model-based evaluation frameworks. An approach was developed to extract evaluation aspects from the ontology that also considers evaluation case practicalities such as the maximum number of evaluation aspects to be measured or their required degree of specificity. The method was applied and tested in Future Internet Social and Technological Alignment Research (FI-STAR), a project of 7 cloud-based eHealth applications that were developed and deployed across European Union countries.ResultsThe relevance of the evaluation aspects created by the UVON method for the FI-STAR project was validated by the corresponding stakeholders of each case. These evaluation aspects were extracted from a UVON-generated ontology structure that reflects both the internally declared required quality attributes in the 7 eHealth applications of the FI-STAR project and the evaluation aspects recommended by the Model for ASsessment of Telemedicine applications (MAST) evaluation framework. The extracted evaluation aspects were used to create questionnaires (for the corresponding patients and health professionals) to evaluate each individual case and the whole of the FI-STAR project.ConclusionsThe UVON method can provide a relevant set of evaluation aspects for a heterogeneous set of health information systems by organizing, unifying, and aggregating the quality attributes through ontological structures. Those quality attributes can be either suggested by evaluation models or elicited from the stakeholders of those systems in the form of system requirements. The method continues to be systematic, context sensitive, and relevant across a heterogeneous set of health information systems.

Quality of health information for consumers on the web: A systematic review of indicators, criteria, tools, and evaluation results

The quality of online health information for consumers has been a critical issue that concerns all stakeholders in healthcare. To gain an understanding of how quality is evaluated, this systematic review examined 165 articles in which researchers evaluated the quality of consumer‐oriented health information on the web against predefined criteria. It was found that studies typically evaluated quality in relation to the substance and formality of content, as well as to the design of technological platforms. Attention to design, particularly interactivity, privacy, and social and cultural appropriateness is on the rise, which suggests the permeation of a user‐centered perspective into the evaluation of health information systems, and a growing recognition of the need to study these systems from a social‐technical perspective. Researchers used many preexisting instruments to facilitate evaluation of the formality of content; however, only a few were used in multiple studies, and their validity was questioned. The quality of content (i.e., accuracy and completeness) was always evaluated using proprietary instruments constructed based on medical guidelines or textbooks. The evaluation results revealed that the quality of health information varied across medical domains and across websites, and that the overall quality remained problematic. Future research is needed to examine the quality of user‐generated content and to explore opportunities offered by emerging new media that can facilitate the consumer evaluation of health information.

Does e-pain plan improve management of sickle cell disease associated vaso-occlusive pain crisis? A mixed methods evaluation

PurposeThere is limited application and evaluation of health information systems in the management of vaso-occlusive pain crises in sickle cell disease (SCD) patients. This study evaluates the impact of digitization of paper-based individualized pain plans on process efficiency and care quality by examining both objective patient data and subjective clinician insights. MethodsRetrospective, before and after, mixed methods evaluation of digitization of paper documents in Children's Hospital of Pittsburgh of UPMC. Subjective perceptions are analyzed using surveys completed by 115 clinicians in emergency department (ED) and inpatient units (IP). Objective effects are evaluated using mixed models with data on 1089 ED visits collected via electronic chart review 28 months before and 22 months after the digitization. ResultsSurveys indicate that all clinicians perceived the digitization to improve the efficiency and quality of pain management. Physicians overwhelmingly preferred using the digitized plans, but only 44% of the nurses had the same response. Analysis of patient records indicates that adjusted time from analgesic order to administration was significantly reduced from 35.50 to 26.77min (p<.05). However, time to first dose and some of the objective quality measures (time from administration to relief, relief rate, admission rate, and ED re-visit rate) were not significantly affected. DiscussionThe relatively simple intervention, high baseline performance, and limited accommodation of nurses’ perspectives may account for the marginal improvements in process efficiency and quality outcomes. Additional efforts, particularly improved communication between physicians and nurses, are needed to further enhance quality of pain management. ConclusionThis study highlights the important role of health information technology (HIT) on vaso-occlusive pain management for pediatric patients with sickle cell disease and the critical challenges in accommodating human factor considerations in implementing and evaluating HIT effects.

Call for discussion about the framework for categorizing economic evaluations of health information systems and assessing their quality

In the recent JAMIA article ‘Measuring value for money: a scoping review on economic evaluation of health information systems (HIS),’ Bassi and Lau1 identified 42 studies conducted from different perspectives within public, private and mixed healthcare systems and assessed 33 of them as good quality. The importance of establishing value for money of large HIS investments is indisputable. Numerous assessments of economic aspects of different HIS implementations have been conducted; however, there is a paucity of reviews classifying and summarizing their results. Bassi and Lau's review makes a useful contribution by introducing a new methodological framework, which classifies HIS studies with an ‘economic component’ and assesses their quality. In our view, the framework has some limitations that may confuse readers and result in misguided conclusions. Here we briefly discuss these limitations and we hope this will stimulate further discussion. One of the central challenges is that many HIS studies, including those identified by Bassi and Lau, contain some information on benefits and/or costs, but such studies were often not designed to conduct an economic evaluation or assess economic efficiency as defined by the economics discipline. The stated objectives and the correspondent methods (eg, econometric modelling) may be sufficient to quantify ‘cost savings' associated with investing in HIS, but cannot show …

Measuring value for money: a scoping review on economic evaluation of health information systems

ObjectiveTo explore how key components of economic evaluations have been included in evaluations of health information systems (HIS), to determine the state of knowledge on value for money for HIS, and provide guidance for future evaluations.Materials and methodsWe searched databases, previously collected papers, and references for relevant papers published from January 2000 to June 2012. For selection, papers had to: be a primary study; involve a computerized system for health information processing, decision support, or management reporting; and include an economic evaluation. Data on study design and economic evaluation methods were extracted and analyzed.ResultsForty-two papers were selected and 33 were deemed high quality (scores ≥8/10) for further analysis. These included 12 economic analyses, five input cost analyses, and 16 cost-related outcome analyses. For HIS types, there were seven primary care electronic medical records, six computerized provider order entry systems, five medication management systems, five immunization information systems, four institutional information systems, three disease management systems, two clinical documentation systems, and one health information exchange network. In terms of value for money, 23 papers reported positive findings, eight were inconclusive, and two were negative.ConclusionsWe found a wide range of economic evaluation papers that were based on different assumptions, methods, and metrics. There is some evidence of value for money in selected healthcare organizations and HIS types. However, caution is needed when generalizing these findings. Better reporting of economic evaluation studies is needed to compare findings and build on the existing evidence base we identified.

Context sensitive health informatics: concepts, methods and tools.

Context is a key consideration when designing and evaluating health information technology (HIT) and cannot be overstated. Unintended consequences are common post HIT implementation and even well designed technology may not achieve desired outcomes because of contextual issues. While context should be considered in the design and evaluation of health information systems (HISs) there is a shortcoming of empirical research on contextual aspects of HIT. This conference integrates the sociotechnical and Human-Centered-Design (HCD) approaches and showcases current research on context sensitive health informatics. The papers and presentations outlines theories and models for studying contextual issues and insights on how we can better design HIT to accommodate different healthcare contexts.

Applying GEP-HI for the planning of an evaluation study: a case study walkthrough (workshop).

A Good evaluation practice in Health Informatics (GEP-HI) Evaluation Practice guideline has been developed through a consensus making process. The guideline lists a set of 60 issues that are relevant for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover the phases of an evaluation study: Study exploration, first study design, operationalization of methods, detailed study design, execution and finalization of an evaluation study. In this seminar we walk through a case study to present how to plan a health information system evaluation study applying the good evaluation practice guideline.

Medical informatics: Past, present, future

Objective To reflect about medical informatics as a discipline. To suggest significant future research directions with the purpose of stimulating further discussion. Methods Exploring and discussing important developments in medical informatics from the past and in the present by way of examples. Reflecting on the role of IMIA, the International Medical Informatics Association, in influencing the discipline. Results Medical informatics as a discipline is still young. Today, as a cross-sectional discipline, it forms one of the bases for medicine and health care. As a consequence considerable responsibility rests on medical informatics for improving the health of people, through its contributions to high-quality, efficient health care and to innovative research in biomedicine and related health and computer sciences. Current major research fields can be grouped according to the organization, application, and evaluation of health information systems, to medical knowledge representation, and to the underlying signal and data analyses and interpretations. Yet, given the fluid nature of many of the driving forces behind progress in information processing methods and their technologies, progress in medicine and health care, and the rapidly changing needs, requirements and expectations of human societies, we can expect many changes in future medical informatics research. Future research fields might range from seamless interactivity with automated data capture and storage, via informatics diagnostics and therapeutics, to living labs with data analysis methodology, involving sensor-enhanced ambient environments. The role of IMIA, the International Medical Informatics Association, for building a cooperative, strongly connected, and research-driven medical informatics community worldwide can hardly be underestimated. Conclusions Health care continuously changes as the underlying science and practice of health are in continuous transformation. Medical informatics as a discipline is strongly affected by these changes and is in a position to be a key, active contributor in these changes.

Mapping stakeholders for system evaluation - the case of the Electronic Prescription Service in England.

This paper discusses the process of identifying stakeholders for the evaluation of health information systems through a map. Defining the multiplicity of stakeholders associated with a new system as well as the nature of their relationships is an important aspect of evaluating any intervention. We report a study of the Electronic Prescription Service (EPS) in primary care in England. We describe the complexity associated with the process of identifying stakeholders and illustrating their dynamic relationships. Reflecting upon our experience of map-making and map-using, we discuss the role of a stakeholder map to generate and communicate knowledge. The EPS stakeholder map - in its variety of possible alternative representations - reveals the complexity of the electronic prescribing scenario and the challenge of its evaluation. Recognising the drawbacks of a static two dimensional representation, we argue that a dynamic use of a stakeholder map and a reflective map-making practice is useful and important for the evaluation of IT programmes in healthcare.

Towards a National Health Information System evaluation.

Most EU Member States have a documented policy on eHealth. Documented follow-up and evaluation strategies for assessing whether national level systems have reached their set aims and outcomes are, however, rare. Methodologies for large scale information system assessment and evaluation are poorly established. This article describes the approach used to generate the Finnish National Health Information System (NHIS) evaluation plan. The core elements of the plan are illustrated, discussing also challenges and solutions in implementation. The article is based on NHIS evaluation planning project [15] and its presentation in the MIE workshop in Sarajevo in 2009 [16], where core issues and challenges of large-scale evaluations were discussed using the Finnish NHIS evaluation plan as a frame of reference.The Finnish plan offers other countries tools with which to assess their own plans and generate national methodologies for NHIS evaluation.