Usability Metrics Research Articles

Family health history collected by virtual conversational agents: An empirical study to investigate the efficacy of this approach.

Family health history (FHx) is one of the simplest and most cost-effective and efficient ways to collect health information that could help diagnose and treat genetic diseases at an early stage. This study evaluated the efficacy of collecting such family health histories through a virtual conversational agent (VCA) interface, a new method for collecting this information. Standard and VCA interfaces for FHx collection were investigated with 50 participants, recruited via email and word of mouth, using a within-subject experimental design with the order of the interfaces randomized and counterbalanced. Interface workload, usability, preference, and satisfaction were assessed using the NASA Task Load Index workload instrument, the IBM Computer System Usability Questionnaire, and a brief questionnaire derived from the Technology Acceptance Model. The researchers also recorded the number of errors and the total task completion time. It was found that the completion times for 2 of the 5 tasks were shorter for the VCA interface than for the standard one, but the overall completion time was longer (17min 44s vs. 16min 51s, p=.019). We also found the overall workload to be significantly lower (34.32 vs. 42.64, p=.003) for the VCA interface, and usability metrics including overall satisfaction (5.62 vs. 4.72, p<.001), system usefulness (5.76 vs. 4.84, p=.001), information quality (5.43 vs. 4.62, p<.001), and interface quality (5.66 vs. 4.64, p<.001) to be significantly higher for this interface as well. Approximately 3 out of 4 participants preferred the VCA interface to the standard one. Although the overall time taken was slightly longer than with standard interface, the VCA interface was rated significantly better across all other measures and was preferred by the participants. These findings demonstrate the advantages of an innovative VCA interface for collecting FHx, validating the efficacy of using VCAs to collect complex patient-specific data in health care.

Comparative Usability Analysis of Selected Data Entry Methods for Web Systems

This paper addresses usability problems of data entry techniques and patterns in web systems and web sites. In order to carry out a comparative analysis of the solutions a dedicated web application was developed as an experimental tool for performing usability testing. The application was designed in the form of a test divided into scenarios and exercises on filling up selected models. Furthermore, the application recorded usability metrics, questionnaires, open and closed questions, as well as users’ personal data. 40 participants, who took part remotely in the experiment, were split into two distinct groups and completed two separate tests that differed in input data and tested solutions. Some measures referring to the effectiveness, efficiency, and user satisfaction were collected. The results were thoroughly analyzed and the recommendations for the usage of particular models were formulated.

An Arabic Adaptation of the Usability Metric for User Experience (UMUX)

ABSTRACT Improving user experience can be achieved by developing accurate tools for tracking and measuring usability. The accuracy of these measurements will be higher when usability practitioners use tools developed in their native language. The Usability Metric for User Experience (UMUX) is a standard usability questionnaire that is developed in English. UMUX measures perceived usability and conforms to the ISO definition of usability. This work aims to translate UMUX into the Arabic language, which is the main speaking language in 22 Arabic countries with a population of more than 450 million people. This translation was based on the cross-cultural adaptation method. Forward and backward translation was performed by Arabic native speakers who are either specialized in the English language or use English as a second language. The Arabic-UMUX (A-UMUX) underwent a psychometric evaluation; the Cronbach’s alpha coefficient value was 0.879, the content scale validity index (S-CVI) value was 0.925, and the scale face value index (FVI) value was 0.95. These results conclude that A-UMUX may be deemed an adequate valid and reliable tool to be used by Arabic usability practitioners.

The Effects of UTAUT and Usability Qualities on Students’ Use of Learning Management Systems in Saudi Tertiary Education

Aim/Purpose: This study proposes a theoretical framework that amalgamates Unified Theory of Acceptance and Use of Technology (UTAUT) variables with usability metrics to investigate the impact on students’ intention and use of the Learning Management System (LMS) in Saudi higher education. Background: There is a dearth of academic research on Saudi higher education to examine the effects of usability factors on students use of LMSs, so significant issues have not yet been examined. Methodology: Based on survey data from 605 respondents, the Partial Least Squares Structural Equation Modelling (PLS-SEM) technique was employed to assess the model. Contribution: The findings of the study may help colleges and universities to gain insights into the best way to promote e-learning system perceived usefulness and acceptance among students. Findings: The results confirmed that the UTAUT parameters are valid and robust in the context of LMS in Saudi Arabia. The dimension of social influence emerged to significantly influence the students’ intention and usage behaviour. The performance expectancy was affected by information quality and the system interactivity whereas the effort expectancy was influenced by system navigation, system learnability, and instructional assessment. The usability feature of interactivity was also demonstrated to influence students’ willingness to use the system. Recommendations for Practitioners: University policymakers are expected to benefit from this research for e-learning system acceptance in an academic setting and eliminate any impediments to its implementation. University students will be able to identify the factors and motivations driving their adoption of the system. In particular, usability, social, and organisational factors that affect their use of an e-learning system would be better understood. Recommendation for Researchers: The study should aid the research community in technology acceptance and usability studies to determine the students’ perceptions and experiences towards e-learning usability, social, and organisational factors that influence their acceptance, specifically in a Saudi context where students have unique psychological and social characteristics. Administrators and designers could also better understand areas of improvement for usability issues and develop design solutions based on the findings of this study. Impact on Society: The suggestions have been offered in order to accelerate and increase the use of e-learning services in Saudi higher education. System designers and administrators should have a better insight into the user interface design, considering system-independent metrics that could enhance user acceptance of e-learning systems. Future Research: The study focused on the students’ perspective, a natural progression of this work is to involve other e-learning stakeholders (teachers and administrators). This could enrich the research by providing a better understanding of undisclosed issues, offering different views about the implementation and use of an e-learning system in Saudi Arabia.

Measuring User Experience With 3, 5, 7, or 11 Points : Does It Matter?

To assess versions of the shorter form variant of Usability Metric for User Experience (UMUX-LITE) questionnaire differing in the number of response options for the items (3, 5, 7, and 11). The UMUX-LITE is an efficient (two-item) standardized questionnaire that measures perceived usability. A growing body of evidence shows it closely corresponds to one of the most widely used standardized usability questionnaires, the System Usability Scale (SUS), with regard to both correlation and magnitude of concurrently collected means. Although the "standard" version of the UMUX-LITE uses items with seven response options, there is some variance in practice. Members of a corporate user experience panel (n = 242) completed surveys rating a recent Web site experience with the SUS and UMUX-LITE, also providing ratings of overall experience and likelihood-to-recommend. Scale reliabilities were acceptable (coefficient α >.70) with the exception of UMUX-LITE with three response options. All UMUX-LITE correlations with SUS, overall experience, and likelihood-to-recommend were highly significant. For likelihood-to-recommend, there was a significant difference in the magnitude of correlations, with 11 response options higher than three. Although some statistically significant differences were observed in correspondence between SUS and UMUX-LITE scores, these did not seem to translate to practically significant differences. The number of UMUX-LITE response options does not matter much, especially in practice. Because the version with three response options showed some weakness with regard to reliability and correlation with likelihood-to-recommend, practitioners should avoid it. Unless there is a strong reason to do otherwise, use the "standard" version with seven response options.

Usability Assessment of the Missouri Cancer Registry's Published Interactive Mapping Reports: Round Two.

Health-related data's users have trouble understanding and interpreting combined statistical and mapping information. This is the second round of a usability study conducted after we modified and simplified our tested maps based on the first round's results. To explore if the tested maps' usability improved by modifying the maps according to the first round's results. We recruited 13 cancer professionals from National American Central Cancer registries (NACCR) 2016 conference. The study involved three phases per participant: A pretest questionnaire, the multi-task usability test, and the System Usability Scale (SUS). Software was used to record the computer screen during the trial and the users' spoken comments. We measured several qualitative and quantitative usability metrics. The study's data was analyzed using spreadsheet software. In the current study, unlike the previous round, there was no significant statistical relationship between the subjects' performance on the study test and the experience in GIS tools (P = .17 previously was .03). Three out of the four (75%) of our subjects with a bachelor's degree or less accomplished the given tasks effectively and efficiently. This study developed a comparable satisfaction results to the first round study, despite that the previous round's participants were highly educated and more experienced with GIS. By considering the round one's results and by updating our maps, we made the tested maps simpler to be used by subjects who have little experience in using GIS technology, and have little spatial and statistical knowledge.

Dry EEG in Sports Sciences: A Fast and Reliable Tool to Assess Individual Alpha Peak Frequency Changes Induced by Physical Effort.

Novel state-of-the-art amplifier and cap systems enable Electroencephalography (EEG) recording outside of stationary lab systems during physical exercise and body motion. However, extensive preparation time, cleaning, and limited long-term stability of conventional gel-based electrode systems pose significant limitations in out-of-the-lab conditions. Dry electrode systems may contribute to rapid and repetitive mobile EEG acquisition with significantly reduced preparation time, reduced cleaning requirements, and possible self-application by the volunteer but are known for higher channel failure probability and increased sensitivity to movement artifacts. We performed a counterbalanced repeated measure endurance cycling study to objectively validate the performance and applicability of a novel commercially available 64-channel dry electrode cap for sport science. A total of 17 healthy volunteers participated in the study, performing an endurance cycling paradigm comprising five phases: (I) baseline EEG, (II) pre-cycling EEG, (III) endurance cycling, (IV) active recovery, and (V) passive recovery. We compared the performance of the 64-channel dry electrode cap with a commercial gel-based cap system in terms of usability metrics, reliability, and signal characteristics. Furthermore, we validated the performance of the dry cap during a realistic sport science investigation, verifying the hypothesis of a systematic, reproducible shift of the individual alpha peak frequency (iAPF) induced by physical effort. The average preparation time of the dry cap was one-third of the gel-based electrode caps. The average channel reliability of the dry cap varied between 80 ± 15% (Phase I), 66 ± 19% (Phase III), and 91 ± 10% (Phase V). In comparison, the channel reliability of the gel-based cap varied between 95 ± 3, 85 ± 9, and 82 ± 9%, respectively. No considerable differences were evident for the comfort evaluations nor the signal characteristics of both caps. A within-volunteers repeated measure analysis of variance (RM-ANOVA) did not show significant effects of the electrode type on the iAPF [F(1,12) = 1.670, p = 0.221, = 0.122, Power = 0.222]. However, a significant increase of the iAPF exists from Phase II to Phases IV and V due to exhaustive physical task. In conclusion, we demonstrated that dry electrode cap is equivalent to the gel-based electrode cap based on signal characteristics, comfort, and signal information content, thereby confirming the usefulness of dry electrodes in sports science and other mobile applications involving ample movement.

Usability Measures in Mobile-Based Augmented Reality Learning Applications: A Systematic Review

The implementation of usability in mobile augmented reality (MAR) learning applications has been utilized in a myriad of standards, methodologies, and techniques. The usage and combination of techniques within research approaches are important in determining the quality of usability data collection. The purpose of this study is to identify, study, and analyze existing usability metrics, methods, techniques, and areas in MAR learning. This study adapts systematic literature review techniques by utilizing research questions and Boolean search strings to identify prospective studies from six established databases that are related to the research context area. Seventy-two articles, consisting of 45 journals, 25 conference proceedings, and two book chapters, were selected through a systematic process. All articles underwent a rigorous selection protocol to ensure content quality according to formulated research questions. Post-synthesis and analysis, the output of this article discusses significant factors in usability-based MAR learning applications. This paper presents five identified gaps in the domain of study, modes of contributions, issues within usability metrics, technique approaches, and hybrid technique combinations. This paper concludes five recommendations based on identified gaps concealing potential of usability-based MAR learning research domains, varieties of unexplored research types, validation of emerging usability metrics, potential of performance metrics, and untapped correlational areas to be discovered.

Analysis and measurement of internal usability metrics through code annotations

Nowadays, usability can be meant as an important quality characteristic to be considered throughout the software development process. A great variety of usability techniques have been proposed so far, mostly intended to be applied during analysis, design, and final testing phases in software projects. However, little or no attention has been paid to the analysis and measurement of usability in the implementation phase. Most of the time, usability testing is traditionally executed in advanced stages. However, the detection of usability flaws during the implementation is of utmost importance to foresee and prevent problems in the utilization of the software and avoid significant cost increases. In this paper, we propose a feasible solution to analyze and measure usability metrics during the implementation phase. Specifically, we have developed a framework featuring code annotations that provides a systematic evaluation of the usability throughout the source code. These annotations are interpreted by an annotation processor to obtain valuable information and automatically calculate usability metrics at compile time. In addition, an evaluation with 32 participants has been carried out to demonstrate the effectiveness and efficiency of our approach in comparison with the manual process of analyzing and measuring internal usability metrics. Perceived satisfaction was also evaluated, demonstrating that our approach can be considered as a valuable tool for dealing with usability metrics during the implementation phase.

Redesigning the user interface of a healthcare management system for the elderly with a systematic usability testing method

ABSTRACTMyriad medical advances and lengthening human longevity increasingly draw our focus toward ways to stay healthy. Healthcare management systems, both videos and apps, prosper now in the marketplace under that focus. Usability and interfaces have been identified as two critical components of these systems, yet to date, few studies have focused on improving these components for older users, who comprise a unique, expanding ethnic group. This study aimed to integrate both objective and subjective perspectives to analyze a health management system’s usability and its interface for senior. We developed an integrative usability metrics set collating both psychological and physiological data. An empirical study verified a resulting improvement of the system and its interface. Results showed more consistent data when the system user’s cognition was positive with respect to usability. Suggestions for enhancing the system’s usability among the older persons are given.

An Evaluation Model for Measuring the Usability of Mobile Office Applications through User Interface Design Metrics

Usage of mobile devices and particularly smart phones has seen an enormous hike due to the advancement of mobile phone technology in recent times. People of different age groups, one way or the other, are now connected to different mobile phone applications, such as, Social Networking, Chatting, VoIP (Voice Over Internet Protocol) applications, Gaming etc. This rapid advancement has made it necessary for user interface designers of mobile applications to design user friendly interfaces for their applications, so that users can interact and use those applications with ease irrespective of their location. Usability plays a vital role for measuring the usefulness of such applications. After examining different experimental studies on usability assessment techniques imparted by various research workers, it has been determined that there is still a great deal of requirement where application designers have to guarantee more adept and improved usability of offline mobile applications, such as, Conversion Apps, offline encyclopedia, Translation apps, business use applications (office applications) etc. The primary objective of this study is to render a model for usability metrics of measuring the usability of office applications for smart phones. The effectiveness, usefulness and reliability of the proposed model is measured through two office applications named Office Suite Pro7 for Android and Office 365 for Windows 8 touch screen Smartphone.The results of usability testing and t-test show the significance of the proposed approach. The model in this study will enable the application designers to guarantee a more adept and enhanced usability of office applications for smart phones during the designing stage.

Scavenge: an intelligent multi-agent based voice-enabled virtual assistant for LMS

ABSTRACT The existing Learning Management Systems (LMSs) are profoundly effective in empowering the organization of e-learning, however, lacking in usability and learnability. The complex navigation and an immature search system are catalysing the issues that needs vigorous improvement. This paper aims to enhance the usability of LMSs by introducing an intelligent multi-agent based voice-enabled virtual assistant, named Scavenge. The Scavenge can work with LMS to speed-up user tasks through natural voice-command (or text). It can perform many internal and external actions without a mouse or keyboard clicks, and capable to provide the results of user search queries. A prototype system was developed with Moodle LMS and conducted a usability test based on ISO/IEC 9126-4 usability metrics that are effectiveness, efficiency, and satisfaction. A group of University students were selected, who were involved in the usability testing of the proposed system. The results have shown that the proposed system has a positive impact on students’ performance and can enhance the usability of LMS. The approach and techniques used in Scavenge are novel and innovative that will be a valuable contribution to the overall field of knowledge.

Product Quality Monitoring Software Based on Usability for Manufacturing Enterprise

This paper has discussed the software usability of product quality monitoring software. In the past, software usability is not considered seriously only to take care of implementation. Otherwise, in this paper, the metrics of software usability is put forward briefly, and the usability of product quality monitoring software is investigated deeply, including: correctness usability, UI usability, IO usability, and operation usability. At the same time, how to achieve these goals in product quality monitoring software is discussed for programming with visual c. Then, this paper has also introduced the general development of product quality monitoring software based on software usability for popular architecture of manufacturing enterprise.

Feasibility study of an EHR-integrated mobile shared decision making application

IntroductionIntegrating mobile applications (apps) into users’ standard electronic health record (EHR) workflows may be valuable, especially for apps that both read and write data. This report details the lessons learned during the integration of a patient decision aid - prostate specific antigen (PSA) testing for prostate cancer screening - into our users' standard EHR workflow for a small usability assessment. Materials and MethodsThis feasibility study included two steps. First we enabled realtime, secure bidirectional data exchange between the mobile app and EHR for 14 data elements, and second we pilot tested the production environment app with 9 primary care patients aged 60–65 years. Our primary usability metric was a net promoter score (NPS), based on users’ recommendation of the app to a friend or family member; we also assessed the proportion of users who 1) updated their prostate cancer risk factor information present in the EHR and 2) submitted more than one unique response regarding their preference to have PSA testing. ResultsThe seven web services necessary to read and write data required considerable configuration, but successfully delivered risk factor-specific educational content and recorded patients’ values and decision preference directly within the EHR. Seven of the 9 patients (78%) would recommend this app to a friend/family member (NPS = 55.6%), one patient used the app to update risk factor information, and 4/9 (44%) changed their decision preference while using the app. ConclusionsIt is feasible to implement a decision aid directly into users’ standard EHR workflow for limited usability testing. Broad scale implementation may have a positive effect on patient engagement and improve shared decision making, but several challenges exist with proprietary EHR vendor application programming interfaces (API)s.

Usability Evaluation Metrics of Tourism Mobile Applications

The use of mobile phone applications for our touristic activity is very common nowadays with the simplification of smartphones. The tourism mobile applications currently can be argued to be one of the most useful applications that can facilitate the movement of travelers. However, existing usability evaluation metrics are too general to be applied to a more specific application, such as mobile tourism application. Thus, the objective of the study is to propose usability evaluation metrics for tourism mobile applications. The study employs four phases: identifying problem and the objective, encompassing the techniques of developing usability measurement of metrics, selecting the usability metrics of tourism mobile application and conducting expert review and verification. The verification phase was conducted using expert review approach to measure the proposed metrics in terms of its consistency ease of use, understandable, verifiable and overall impression. The finding revealed that the proposed metrics have been well received by the experts in terms of consistency, ease to use, understandable, verifiability and overall impression. Finally, this study presented usability metrics for the tourism mobile applications that can be used by designers or usability practitioners in creating a useable mobile application for the tourists.

The Effectiveness of a Customized Online Collaboration Tool for Teaching and Learning

Aim/Purpose: There is a huge array of educational technology tools that are now in use today. These tools have changed the way teachers teach and the way students learn. Among the many educational technology tools that are gaining popularity are the online collaboration tools. Online collaboration tools are web-based tools that allow individuals to do things together online like messaging, file sharing, and assessment. However, when new educational technology is integrated in the classroom, its effects must be determined as this is an essential component for evaluation. Having the tool is not enough, there has to be an evaluation of its quality to make it more effective. It is on this premise that the customized online collaboration tool of one university in the Philippines was assessed. The study specifically aimed to identify the perceived effectiveness of the customized online collaboration tool; and identify the effectiveness of using Usability Metrics for Effectiveness. Background: Most studies provide evaluation of newly developed software using a set of quality standards such as functionality and usability. However, there is limited evidence where online collaboration tools are evaluated on their effectiveness using user’s perception as well as assessing their usability using a set of effectiveness metrics. Methodology: The study employed questionnaire-based (n=127) and metrics-based (n=81) approaches. Data gathered were analyzed using simple and relative frequencies, weighted mean, ANOVA, and Tukey HSD (Honestly Significant Difference). Contribution: This study supplements the literature regarding the effectiveness of online collaboration tools that are used for teaching and learning particularly those that are custom-made for an institution. It provides additional information on other ways to evaluate the effectiveness of customized online collaboration tools. It likewise provides information on the difference between what the users report about the tool’s effectiveness and what the metrics show. Findings: The customized online collaboration tool was perceived to be very effective for collaboration, teaching and learning. But there is a significant difference on the students’ perception based from their courses and year levels. The customized online collaboration tool’s effectiveness as regards its application performance is poor because of design interface faults. However, the customized online collaboration tool has good completion rate. Recommendations for Practitioners: Findings suggest that the customized online collaboration tool can be an alternative tool to open-source learning platforms which faculty members may use for teaching and learning as students find it very effective not only for collaboration but for teaching and learning as well. However, errors caused by interface design problems must be addressed to yield a higher application performance rating and higher completion rate. Recommendation for Researchers: Further assessment on the tool’s effectiveness using an empirical study is recommended to provide additional definitive evidence which may support the benefits of using online collaboration tools in the classroom.

Set of Usability Heuristics for Quality Assessment of Mobile Applications on Smartphones

The innovations proposed by the cell phone market have grown steadily in recent years, along with the increasing complexity of the hardware, operating systems, and applications available in this market. These changes bring new challenges related to usability that need to be considered during the development process of these applications since the new forms of user-application interactions increasingly require adapting the behavior of smartphone users. In this situation, usability is an important issue that depends on factors such as the Users, their characteristics and abilities, the Task which the users intend to achieve and also the application usage Context. This work presents a systematic literature review with the objective of identifying the heuristics and usability metrics used in the literature and/or industry. Based on the review results, this work presents another contribution with a proposal of a set of usability heuristics focused in mobile applications on smartphones, considering the User, Task and Context as usability factors and Cognitive Load as an important attribute of usability. The components of this set are detailed in a model intended to be used in empirical validations allowing to dynamically incorporate improvements to the proposal.

Usability of “Traysi”: A Web Application for Tricycle Commuters

This study measured the usability of a web application for tricycle commuters that was developed using Hypertext Markup Language (HTML), Cascading Style Sheet (CSS) and Javascript (JS) with the aid of Google Artificial Programming Interfaces (APIs). Toward this goal, the effectiveness, efficiency and user satisfaction were measured using common usability metrics. The effectiveness was measured in terms of task completion rate and user errors while efficiency was measured in terms of time on task. For user satisfaction, the post-task questionnaire, Single Ease Question (SEQ), was used. In order to check whether the web application will be usable even for first time users, the usability test was conducted three times. The result revealed that the usability of the web application is acceptable on the first trial. However, the usability improved on the next used which is evident on the third trial that yielded a 93.33% task completion rate with only one user error. The average time on task on every trial was lower than the maximum acceptable task time and the user satisfaction was high (x = 6.00). Thus, the web application was highly usable in doing its intended purpose especially if it is repeatedly used.

SOCA-DSEM: A well-structured SOCA development systems engineering methodology

Service-oriented Software Engineering (SOSE) is a software engineering paradigm focused on Service-oriented Computing Applications (SOCAs), for what SOCA development methodologies are required. Recent studies on SOCA development methodologies revealed theoretical and practical deficiencies. Thus, academicians and practitioners must adapt development methodologies from other paradigms or use the available partial SOCA development methodologies. Also, since the high acceptance of agile approaches, we claim new well-structured and balanced agility-rigor methodologies are required. Then, this paper proposes a new SOCA Development Systems Engineering Methodology, including its description, the explanation of its theoretical foundations and the illustration of its use with a prototype of a running example. Two pilot empirical evaluations on usability metrics are also reported. Findings support both theoretical adequacy and positive perceptions from the evaluators. While further empirical tests are required for gaining more conclusive evidences our preliminary results are encouraging.

A challenge-response mechanism for securing online social networks against social bots

Social bots is fast becoming a serious security threat to online social networks (OSNs). Social bots are automated software tools able to execute malicious activities in OSNs systems in an automated fashion. It can perform auto-sharing and posting, sending fake friend requests, harvesting private information, etc. There is evidence that social bots play a crucial role in penetrating privacy and security of social networks. Hence, these malicious software tools represent a big security challenge against social network service provider (SNSP). In this paper, we introduce a novel antibot mechanism called Necklace CAPTCHA for securing OSNs platforms against the automated behaviours of social bots. Necklace CAPTCHA is an Image-based CAPTCHA, which utilises the functionality of Necklace Graph approach to generate its challenge-response tests in a novel fashion. The results demonstrated that Necklace CAPTCHA is an effective and secure anti-bot mechanism compared with other CAPTCHAs in the literature with respect to the usability and security metrics.