Evaluation of health information systems with ISO 9241-10 standard: A systematic review and meta-analysis

Abstract

This study aimed to summarize available data about health information system evaluation using the ISO 9241–110 (previously 9241–10) standard. A systematic review of literature from January 2000 to November 2020 was conducted in PubMed, Scopus, ScienceDirect, Web of Science, and Google Scholar as well as two Iranian national databases, Scientific Information Database and Magiran. Pooled estimates were calculated using random-effects meta-analysis. Data analyses were conducted using Stata version 14. From total of 179 studies subjected to initial screening, fifteen studies were eligible to enter the meta-analysis. The pooled mean estimates of the health information systems' overall scores (3.11; 95% confidence interval [CI], 2.82–3.41), suitability for individualization (3.82; 95% CI, 2.59–3.04), and controllability (3.82; 95% CI, 2.91–3.47) commonly had better performance compared with other dimensions. High heterogeneity was seen among the seven dimensions ( I 2 ≥ 98.6%, p < .001). According to the viewpoints of HIS users, suitability for individualization (3.82; 95% CI, 2.59–3.04) and controllability (3.82; 95% CI, 2.91–3.47) had better performance compared with other dimensions, and the results of the subgroup meta-analysis confirmed this difference ( z = 4.75, p < .001). For each dimension, Egger's and Begg's tests showed no publication bias ( p > .1). Most of the studies used the ISO 9241-110 standard as a summative assessment for evaluation of the usability of the HIS and results indicated the relative success of the evaluated systems based on this standard. • ISO 9241 part 10 is a valid standard for formative and summative evaluation of health information systems. • Most of the studies that have used the ISO9241 part 10 have focused on hospital information systems. • Few evaluations have been conducted using the formative version of ISO 9241 part 10. • Most of the evaluation studies have used the summative approach of the ISO 9241 part 10. • Because of the importance, it is suggested that researchers consider formative approaches.