A Scale to Assess the Methodological Quality of Studies Assessing Usability of Electronic Health Products and Services: Delphi Study Followed by Validity and Reliability Testing.

Anabela G Silva,Alexandra Queirós,Patrícia Simões,Rita Santos,Mário Rodrigues,Nelson P Rocha

doi:10.2196/14829

Abstract

BackgroundThe usability of electronic health (eHealth) and mobile health apps is of paramount importance as it impacts the quality of care. Methodological quality assessment is a common practice in the field of health for different designs and types of studies. However, we were unable to find a scale to assess the methodological quality of studies on the usability of eHealth products or services.ObjectiveThis study aimed to develop a scale to assess the methodological quality of studies assessing usability of mobile apps and to perform a preliminary analysis of of the scale’s feasibility, reliability, and construct validity on studies assessing usability of mobile apps, measuring aspects of physical activity.MethodsA 3-round Delphi panel was used to generate a pool of items considered important when assessing the quality of studies on the usability of mobile apps. These items were used to write the scale and the guide to assist its use. The scale was then used to assess the quality of studies on usability of mobile apps for physical activity, and it assessed in terms of feasibility, interrater reliability, and construct validity.ResultsA total of 25 experts participated in the Delphi panel, and a 15-item scale was developed. This scale was shown to be feasible (time of application mean 13.10 [SD 2.59] min), reliable (intraclass correlation coefficient=0.81; 95% CI 0.55-0.93), and able to discriminate between low- and high-quality studies (high quality: mean 9.22 [SD 0.36]; low quality: mean 6.86 [SD 0.80]; P=.01).ConclusionsThe scale that was developed can be used both to assess the methodological quality of usability studies and to inform its planning.

Highlights

BackgroundMethodological quality can be defined as “the extent to which study authors conducted their research to the highest possible standards” [1]
The critical assessment of the quality of studies is a complex process that must consider several different aspects of the study, which may vary depending on the type of study and on the subject of research [1,3,4]. This process is usually performed with the aid of critical appraisal tools previously developed for that specific purpose. This is common practice in the field of health, where a number of critical appraisal tools exist to assist the assessment of the methodological quality of studies [1,3,4,5]
We were unable to find any critical tool to guide the assessment of methodological quality of usability studies, neither for electronic health applications nor for general applications

Summary

Introduction

BackgroundMethodological quality can be defined as “the extent to which study authors conducted their research to the highest possible standards” [1]. The critical assessment of the quality of studies is a complex process that must consider several different aspects of the study, which may vary depending on the type of study and on the subject of research [1,3,4] This process is usually performed with the aid of critical appraisal tools previously developed for that specific purpose. Methods: A 3-round Delphi panel was used to generate a pool of items considered important when assessing the quality of studies on the usability of mobile apps. These items were used to write the scale and the guide to assist its use. Conclusions: The scale that was developed can be used both to assess the methodological quality of usability studies and to inform its planning

Objectives

Methods

Results

Conclusion