Usability Evaluation Methods Research Articles

Innovation Through Universal Design in Agile UX Software Development Teams. A Collaborative Case Study of an Under Graduate AR Tourist Guide Project.

The study has the objective of designing AR tourist guide mobile app within an academic teaching framework facilitating collaborative (e.g. external commercial partners), cooperative (i.e. external academic experts) and user-centred design (UCD). [1]The tourist guide app, VisitAR, is a digitized tour application that portrays information in the form of landmarks and information windows. VisitAR provides a seamless walking experience in real-time by using your location, and triggering pop up information windows while you walk at Carlingford Ireland. The application testing was completed by using several usability evaluation methods i.e. technical field testing, living lab testing including speaking thoughts out loud, usability focus group testing and usability analysis As a result, by teaching UD within an experiential, living lab, a more realistic design context is provided, addressing realistic UX and SD, allowing deployment of potentially commercially viable solutions, which address the needs of a more diverse range of end users. As part of this case study, both qualitative and quantitative data related to UX, usability and SD from each stage of development was evaluated.

Systematic review of applied usability metrics within usability evaluation methods for hospital electronic healthcare record systems: Metrics and Evaluation Methods for eHealth Systems.

Background and objectivesElectronic healthcare records have become central to patient care. Evaluation of new systems include a variety of usability evaluation methods or usability metrics (often referred to interchangeably as usability components or usability attributes). This study reviews the breadth of usability evaluation methods, metrics, and associated measurement techniques that have been reported to assess systems designed for hospital staff to assess inpatient clinical condition.MethodsFollowing Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) methodology, we searched Medline, EMBASE, CINAHL, Cochrane Database of Systematic Reviews, and Open Grey from 1986 to 2019. For included studies, we recorded usability evaluation methods or usability metrics as appropriate, and any measurement techniques applied to illustrate these. We classified and described all usability evaluation methods, usability metrics, and measurement techniques. Study quality was evaluated using a modified Downs and Black checklist.ResultsThe search identified 1336 studies. After abstract screening, 130 full texts were reviewed. In the 51 included studies 11 distinct usability evaluation methods were identified. Within these usability evaluation methods, seven usability metrics were reported. The most common metrics were ISO9241‐11 and Nielsen's components. An additional “usefulness” metric was reported in almost 40% of included studies. We identified 70 measurement techniques used to evaluate systems. Overall study quality was reflected in a mean modified Downs and Black checklist score of 6.8/10 (range 1–9) 33% studies classified as “high‐quality” (scoring eight or higher), 51% studies “moderate‐quality” (scoring 6–7), and the remaining 16% (scoring below five) were “low‐quality.”ConclusionThere is little consistency within the field of electronic health record systems evaluation. This review highlights the variability within usability methods, metrics, and reporting. Standardized processes may improve evaluation and comparison electronic health record systems and improve their development and implementation.

A Review of Website Measurement for Website Usability Evaluation

There are many website usability measurement. Therefore, researcher that focus on usability measurement need to select the best method for usability evaluation to get the best result. This paper review website measurement that has classify to usability inspection, usability inquiry and usability testing. Some of usability measurement focus on experts only to measure the evaluation and some of measurement focus on real users. Each of website measurement have advantage and disadvantage. Researcher need to determine the scope, context and the reason when evaluate usability for website. This paper found that survey is common method that used by researcher because it save cost and can gather a population easily.

Research on User Experience Based on Competition Websites

The design of the website interface has an impact on the user experience in all aspects. A set of relatively perfect interactive interface is often in line with the needs, cognition and behavior of the target user to form a good experience. In order for users to have a good experience in their use, subjective evaluation and usability evaluation methods are usually used to test and tune after the interface is designed. This analysis method has its limitations. For example, it cannot reveal the thinking process and psychological activities of the user and the interface in the interaction process for the designer, and provide the tester with the time and space information of the user’s search and processing process. This paper adopts more cutting-edge research methods. First, we analyze the usability of three websites of the same type and different structure, such as evaluating the effectiveness by task completion rate and error rate, and evaluating efficiency by operating time and operating path. Secondly, conduct eye movement experiments on users, further assist and re-verify the usability evaluation method, and analyze its dynamic interaction characteristics, and conduct a comprehensive user experience research on competition websites from qualitative and quantitative perspectives.

A methodology for domain and culture-oriented heuristics creation and validation

ABSTRACT Culture fit is an essential consideration while designing websites in this technologically evolving era, yet it is too often ignored in current usability evaluation methods. The heuristic evaluation is one such method, in which, generally, experts use a set of heuristics as a guide. These heuristics are well designed to pinpoint the general usability design faults; however, they limit identifying potential usability problems for specific domains and cultures. This article presents a formal methodology with a sequence of steps of how domain and culture-specific heuristics can be designed. This methodology is validated and applied in the context of websites for universities in Pakistan.

Evaluation of Tourism Information Accuracy Assessment Framework

The objective of this paper to evaluate the framework’s suitability, efficiency, satisfaction, adaptability and safety for tourism information. The usability evaluation method includes the usability attributes of framework in tourism information phenomena’s such as information source, information development, information gathering, information collection and information assessment. The usability evaluation has been conducted using the expert review methodology to gather for recommendations and improvements of the developed framework. The framework expert review comprises of five evaluation components which are the expert selection criteria, expert selection, issue familiarization of experts, collection of responses and result reporting. Furthermore, we also analysis the usability of the framework based on the experts score for each area for the assessments. We have used the polar chart to show the overall usability measurement of the framework and the results has been discussed. The identified usability issues been addressed accordingly to ensure the developed framework is useful for tourism information environments in producing accurate tourism information.

Develop and evaluate an augmented reality posture training tool to promote work safety

Conventional work posture training tools included pamphlets, one-time training orientation, and/or videos. These tools did not always yield satisfactory training outcomes, and the incident rate of work-related musculoskeletal disorders did not substantially lower. In this research, modern augmented reality (AR) technology was leveraged to deliver interactive, holistic, whole-body visual information to convey safe work postures. The developmental procedure followed DMAIC by first defining specifications of training content, which led to the development of the training tool, including 3D reconstruction of a virtual instructor and building of user interface based on user-centered framework. This AR training tool was measured and analyzed through usability evaluation, and quantitative and qualitative data were obtained for cross-validation and usability issue source identification. Findings revealed the utility of 3D reconstruction of a virtual instructor and practicality of adopting conventional usability evaluation method for AR user interface usability evaluation. Feedback from the usability evaluation via questionnaire, think aloud, and post-task open-ended responses are employed to iteratively design the next version of the AR posture training tool.

Integrating heuristics and think-aloud approach to evaluate the usability of game-based learning material

The purpose of this study was to evaluate the usability of a game-based learning material. The heuristics and think-aloud approach were employed to determine the usability of the learning material. Four experts used the learning material independently and completed a heuristic evaluation questionnaire based on a design and use perspective. Forty-five university students used the learning material independently and completed a heuristic evaluation questionnaire based on their experiences. The heuristic evaluation questionnaire was comprised of six constructs with a total of twelve heuristic evaluation criteria. The experts and students provided data about usability problems while using the learning material. The usability problems that were found included: (a) inconsistent interface styles on different learning units (i.e., game scenes) that affected the learnability, memorability and operation of interfaces; (b) inadequate operational efficiency and guidance could be improved; (c) small games and instructional videos could probably increase learning times and loads; (d) different guidance and supports for errors and inconsistent display styles of error messages for each learning units; and (e) inappropriate feedback on errors and inadequate interaction on instructional videos. The usability problems were identified and improved with the usability evaluation methods. Educational implication and research recommendation are discussed.

VGI VISUALISATION TO SUPPORT PARTICIPATORY LAKE MONITORING: THE CASE STUDY OF SIMILE PROJECT

Abstract. This study focuses on the role of VGI visualization in encouraging participation to environmental Citizen Science, taking as a case study the mobile application developed within SIMILE (Informative System for the Integrated Monitoring of Insubric Lakes and their Ecosystems) project activities. The main project goal is the protection of water quality for Lugano, Maggiore and Como lakes through a geoinformatics coordination of existing monitoring systems with new data collection methods (satellite images, high frequency sensors and citizen science). This tool enables the producers to perform a community-based monitoring side by side with local authorities and research bodies improving the efficiency of the current protocols.Referring to ontologies of geospatial data, the VGI samples are analysed under different aspects as their quality, quantity, variety, granularity, spatial and temporal distribution, highlighting their temporary, moving and changing nature. This analysis shows the presence of potential contradictions existing between the application use and the perceived lake water quality, to be mitigated mainly working on the user interface of the map view. So, a study of existing conventions, especially in terms of colours, is compared with the results coming from the application of consolidated survey methodologies to this specific context. Reflections on markers, symbols and attributes of features and maps are presented introducing geovisual analytical solutions for dynamic phenomena. The design solutions adopted are finally presented and validated with a combination of different usability evaluation methods like heuristics and field tests.

A Review of Usability Evaluation Methods and Their Use for Testing eHealth HIV Interventions.

To provide a comprehensive review of usability testing of eHealth interventions for HIV. We identified 28 articles that assessed the usability of eHealth interventions for HIV, most of which were published within the past 3years. The majority of the eHealth interventions for HIV was developed on a mobile platform and focused on HIV prevention as the intended health outcome. Usability evaluation methods included eye-tracking, questionnaires, semi-structured interviews, contextual interviews, think-aloud protocols, cognitive walkthroughs, heuristic evaluations and expert reviews, focus groups, and scenarios. A wide variety of methods is available to evaluate the usability of eHealth interventions. Employing multiple methods may provide a more comprehensive assessment of the usability of eHealth interventions as compared with inclusion of only a single evaluation method.

Usability Evaluation Model for Mobile Visually Impaired Applications

The usability of the mobile applications is the most important factor in developing, so the key to develop successful mobile applications is usability, especially for users have specific needs such as visually impaired. However, developers do not focus on visually impaired users. Moreover, there are limited studies and usability evaluation models for mobile applications for visually impaired so developers use just a modified usability evaluation methods which are not enough and useful to evaluate mobile applications for visually impaired, or they use general usability evaluation models. Therefore, using these methods or models is difficult for evaluator and not useful for visually impaired users. This study conducts Systematic Literature Review (SLR) to identify usability dimensions that help mobile applications developers and evaluators to evaluate mobile application for users which have moderate and severe visual impairment. The result shows that, six dimensions that have a significant impact on moderate and severe visually impaired users' satisfaction, who use mobile applications. These dimensions namely efficiency, effectiveness, satisfaction, errors, accessibility and understandability.

효율적인 스마트러닝 앱 사용성 평가 방법 연구

효율적인 스마트러닝 앱 사용성 평가 방법 연구

Understanding usability evaluation setup for VR products in industry

VR development practices have a diverse set of practices compared to traditional software development. Tasks like scene design, acoustic design, vergence manipulation, image depth, etc. are specific to VR apps and hence require evaluation processes that may be different from the traditional means. Usability Evaluation is one such process which is being executed in an unconventional way by Industrial Practitioners today. In this paper, the researchers detail a Systematic Literature Review of the Usability Evaluation Methods practised by Industrial researchers while building VR Products. The researchers found that VR Product teams follow unique methods to improve usability in their products. Further, the researchers consolidate these methods and provide insights into choosing the best to build a real-world VR Product based on the defined product constraints

ANALISIS USER INTERFACE TERHADAP WEBSITE AKTA ONLINE BANYUWANGI MENGGUNAKAN METODE HEURISTIC EVALUATION

Banyuwangi Regency has a new innovation called " Lahir Procot Pulang Bawa Akta " which in this innovation is realized in the form of an Online Deed Website. Every information technology has an interface that can be a link between the user and the technology itself. Interface formation is influenced by needs, and information technology has different interface designs according to the needs of its users. The User Interface has the aim of making it easier for users to operate information technology that can make users feel comfortable using the application or technology. Heuristic Evaluation is one of the Usability evaluation methods that can be used to determine the extent to which a system is used by users to achieve certain goals with effectiveness, efficiency and satisfaction. This research is a research that is focused on the use of Heuristic Evaluation based on user interface design aspects of application usability through observation, interviews and questionnaires to users.

An Automated Usability Evaluation Method Based on Structural Compatibility of Information Resource Distribution and Screen Transition Design with User Tasks

An Automated Usability Evaluation Method Based on Structural Compatibility of Information Resource Distribution and Screen Transition Design with User Tasks

Software developers need help too! Developing a methodology to analyse cognitive dimension-based feedback on usability

ABSTRACT Software developers use various methods to evaluate usability and identify usability issues that exist in systems they develop. Cognitive dimensions framework (CDF) based usability evaluation is one of the popular usability evaluation methods. It uses an open-ended questionnaire to collect qualitative feedback from users after using a system. To identify usability issues, evaluators should analyse this qualitative feedback. However, the approach to follow when performing this analysis is not explored in detail. We conducted a systematic literature review and reviewed 70 studies that used various CDF questionnaires for usability evaluations and investigated how those studies have analysed CDF questionnaire responses to identify usability issues. This revealed five methods that previous research has used for data analysis and four methods for identifying usability issues from CDF questionnaire responses. We applied the results of the literature review to develop a methodology and a set of guidelines to analyse qualitative feedback collected via a CDF questionnaire that targets evaluating security application programming interfaces. We tested the developed guidelines by conducting an empirical investigation. The results of the experiment revealed that using the proposed guidelines helps to identify significantly more usability issues with a higher validity.

Cognitive Usability Evaluation of Electronic Health Record Systems (CUE-E)

This project’s purpose was to develop cognitively-focused usability evaluation methods for electronic health records (EHRs). This research involved developing a conceptual framework for evaluating EHR usability in terms of clinical cognition and operationalizing the framework in the form of two novel EHR usability evaluation methods. The two evaluation methods - one observational and one lab-based - are described in a suite of protocol materials and recommendations for EHR evaluation and design. This resulting body of work is referred to as CUE-E: Cognitive Usability Evaluation - EHR. This paper describes the process behind the development of the CUE-E evaluation methods, summarizes the use of both, and discusses directions for future work. The two CUE-E evaluation methods are currently ready for pilot applications to assess their reliability and validity and identify opportunities for further improvement.

Applying Semantic Network Analysis to Develop User Experience Assessment Model for Smart TV

The user experience (UX) concept has widely been adopted in the field of development and evaluation of products and services. One reason behind this emerging trend is the limitation in traditional usability evaluation methods (UEMs), which emphasizes performance measures such as task completion time and error rate. To overcome the limitation, users’ perceived value must be set to reflect users’ various experiences. In order to address this issue, this study proposes a network analysis-based method to investigate a value structure to assess the level of user experience in using a product. Gesture interaction on a smart TV is used as a case study to demonstrate the process of obtaining a quantified value structure from the transcriptions of users’ in-depth interview. A major contribution of this study lies in the transformation from qualitative data to a quantified value structure. Product or service designers, in the industrial field, can develop value structure based on network analysis when evaluating their design alternatives within a properly fitted evaluation framework.

Before Teaching Cybersecurity in the University: Usability Analysis of the Cloud Platform with Learner’s Perspective

The aim of this research is to apply usability evaluation of the Cloud Service Providers (CSP) established for various development purposes, CSP also are used in teaching cybersecurity courses for undergraduate program. In order to apply usability evaluation, two CSP products are used, namely Google Cloud and Microsoft Azure. We have applied usability evaluation method to measure its appropriateness with respect to learning and the use of a product for teaching. We have categorized students in two groups, namely expert and novice. Our evaluation suggests that before using graphical user interfaces of CSP, one should have knowledge and skills in implementing Cybersecurity project.

Usability Evaluation of Software Tools for Engineering Design

AbstractMuch of the work in design research focusses on the development of methods and tools to support engineering designers. Many of these tools are nowadays implemented in software. Due to the strongly growing use of computers and smart devices in the last two decades, the expectations of users increased dramatically. In particular users expect good usability, for example little effort for learning to apply the software. Therefore, the usability evaluation of design software tools is crucial. A software tool with bad usability will not be used in industrial practice. Recommendations for usability evaluation of software often stem from the field of Human Computer Interaction. The aim of this paper is to tailor these general approaches to the specific needs of engineering design. In addition, we propose a method to analyse the results of the evaluation and to derive suggestions for improving the design software tool. We apply the usability evaluation method on a use case - the KoMBi software tool for bio-inspired design. The case study provides additional insights with regards to problem, causes and improvement categories.