Usability Evaluation Techniques Research Articles

Usability of Standards for Scaffolding in a Health Sciences Programme: A feasibility Study

BackgroundStandards contribute to comprehensive and programmatic implementation of educational strategies, such as scaffolding. Although the development of educational standards follows a rigorous consensus approach, they are socially constructed and could result in varied interpretations by users. Reports of varied implementation of standards in health professions education underscore the need to test the developed standards for scaffolding in health sciences programmes. Usability entails determining whether a product like standards works as intended under the expected conditions and contexts. This study aimed to describe the usability of standards for scaffolding in a health sciences programme through a pilot study.MethodsA multi-method design employing user and expert-based usability evaluation techniques sought to describe the usability of the standards for scaffolding in a three-year pre-registration nursing programme. The user sample of nurse educators drawn from the programme, conducted a self-assessment on scaffolding practices in the programme using a developed standards checklist. For the expert sample, three-panel members with an understanding of the discipline and programme context were purposively sampled. These panelists studied the users’ self-assessment reports before completing an author-generated heuristics checklist to support or refute any of the standards. Descriptive statistics, comparative and content analysis were applied to analyse data from users’ interviews and expert’s completed heuristics checklist, determining the standards’ usability, and identifying the usability flaws or strengths.ResultsThe users had three or more years of teaching experience in the competency-based curriculum for nursing. The experts shared an average of 16 years of experience in teaching in higher education, and seven years of experience in quality assurance and programme accreditation. The four standards had a usability score of above average (68%). Seven usability strengths and four usability flaws were identified. Usability flaws related to misinterpretation of some criteria statements and terminologies, multiple meanings, and users’ challenges in generating evidence for some criteria.ConclusionsThe pilot study revealed the context-based ‘truth’ regarding the fidelity of a health sciences programme evaluation on scaffolding, as well as identifying the ideal contextual conditions in which the standards for scaffolding health sciences programmes would work best. The identified usability flaws highlighted the need for further revisions of the standards. Future research on the feasibility of the standards in other health sciences programmes and contexts is recommended.

Consensus on the Terms and Procedures for Planning and Reporting a Usability Evaluation of Health-Related Digital Solutions: Delphi Study and a Resulting Checklist.

Usability evaluation both by experts and target users is an integral part of the process of developing and assessing digital solutions. Usability evaluation improves the probability of having digital solutions that are easier, safer, more efficient, and more pleasant to use. However, despite the widespread recognition of the importance of usability evaluation, there is a lack of research and consensus on related concepts and reporting standards. The aim of the study is to generate consensus on terms and procedures that should be considered when planning and reporting a study on a usability evaluation of health-related digital solutions both by users and experts and provide a checklist that can easily be used by researchers when conducting their usability studies. A Delphi study with 2 rounds was conducted with a panel of international participants experienced in usability evaluation. In the first round, they were asked to comment on definitions, rate the importance of preidentified methodological procedures using a 9-item Likert scale, and suggest additional procedures. In the second round, experienced participants were asked to reappraise the relevance of each procedure informed by round 1 results. Consensus on the relevance of each item was defined a priori when at least 70% or more experienced participants scored an item 7 to 9 and less than 15% of participants scored the same item 1 to 3. A total of 30 participants (n=20 females) from 11 different countries entered the Delphi study with a mean age of 37.2 (SD 7.7) years. Agreement was achieved on the definitions for all usability evaluation-related terms proposed (usability assessment moderator, participant, usability evaluation method, usability evaluation technique, tasks, usability evaluation environment, usability evaluator, and domain evaluator). A total of 38 procedures related to usability evaluation planning and reporting were identified across rounds (28 were related to usability evaluation involving users and 10 related to usability evaluation involving experts). Consensus on the relevance was achieved for 23 (82%) of the procedures related to usability evaluation involving users and for 7 (70%) of the usability evaluation procedures involving experts. A checklist was proposed that can guide authors when designing and reporting usability studies. This study proposes a set of terms and respective definitions as well as a checklist to guide the planning and reporting of usability evaluation studies, constituting an important step toward a more standardized approach in the field of usability evaluation that may contribute to enhancing the quality of planning and reporting usability studies. Future studies can contribute to further validating this study work by refining the definitions, assessing the practical applicability of the checklist, or assessing whether using this checklist results in higher-quality digital solutions.

Domain Usability Evaluation

Contemporary software systems focus on usability and accessibility from the point of view of effectiveness and ergonomics. However, the correct usage of the domain dictionary and the description of domain relations and properties via their user interfaces are often neglected. We use the term domain usability (DU) to describe the aspects of the user interface related to the terminology and domain. Our experience showed that poor domain usability reduces the memorability and effectiveness of user interfaces. To address this problem, we describe a method called ADUE (Automatic Domain Usability Evaluation) for the automated evaluation of selected DU properties on existing user interfaces. As a prerequisite to the method, metrics for formal evaluation of domain usability, a form stereotype recognition algorithm, and general application terms filtering algorithm have been proposed. We executed ADUE on several real-world Java applications and report our findings. We also provide proposals to modify existing manual usability evaluation techniques for the purpose of domain usability evaluation.

Improving the Usability of Organizational Data Systems

For research data repositories, web interfaces are usually the primary, if not the only, method that data users have to interact with repository systems. Data users often search, discover, understand, access, and sometimes use data directly through repository web interfaces. Given that sub-par user interfaces can reduce the ability of users to locate, obtain, and use data, it is important to consider how repositories’ web interfaces can be evaluated and improved in order to ensure useful and successful user interactions. This paper discusses how usability assessment techniques are being applied to improve the functioning of data repository interfaces at the National Center for Atmospheric Research (NCAR). At NCAR, a new suite of data system tools is being developed and collectively called the NCAR Digital Asset Services Hub (DASH). Usability evaluation techniques have been used throughout the NCAR DASH design and implementation cycles in order to ensure that the systems work well together for the intended user base. By applying user study, paper prototype, competitive analysis, journey mapping, and heuristic evaluation, the NCAR DASH Search and Repository experiences provide examples for how data systems can benefit from usability principles and techniques. Integrating usability principles and techniques into repository system design and implementation workflows helps to optimize the systems’ overall user experience.

Usability Evaluation of Mobile Tracking Applications: A Systematic Review

In the recent years, mobile is found to be useful since they allow the user to perform more tasks in a mobile context. However, this usefulness come at the expense of the usability of these devices in some contexts. Thus, usability becomes an important factor in the adoption of mobile applications (such as mobile tracking applications, mobile health and mobile learning), which are often used by people who have problems when using mobile devices and who have a limited experience of technology. It’s critical in the Mobile domain to ensure the success of mobile applications. Despite some studies review usability evaluation, regrettably there are no systematic literature reviews focused on usability evaluation of mobile tracking applications. Consequently, in this study, the researchers strive to present the results of a systematic literature review (SLR) involving the use of usability evaluation techniques in Mobile tracking applications. Based on three research questions, the results indicated that, there are three kinds of the Mobile tracking applications namely: information tracking, location tracking and information and location tracking. In addition, and regard to the techniques used to evaluate the usability, the experimental study was the regularly used in the selected studies. Finally, the results shown that, for characteristics were frequently exploited to measure the usability of mobile tracking application are: Effectiveness, Efficiency, Usefulness and Accurate. This study is valuable for the researchers who need to evaluate the usability for any new mobile application through explored the appropriate methods used to evaluation the usability and indicators required for this purpose.

A new optimization model for design of traditional cockpit interfaces

PurposeReal flight is cognitively demanding; accordingly, both indicators and display panel layout should be user-friendly to improve pilot-aircraft interaction. Poor pilot-interface interactions in aircrafts could result in accidents. Although a general reason of accidents is improper displays, relatively few studies were conducted on interfaces. This study aims to present an optimization model to create intuitively integrated user-friendly cockpit interfaces.Design/methodology/approachSubjectivity within most usability evaluation techniques could bring about interface design problems. A priori information about indicator’s possible locations may be available or unavailable. Thus different analytical approaches must be applied for modifications and new interface designs. Relative layout design (RLD) model was developed and used in new interface designs to optimize locations of indicators. This model was based on layout optimization and constructed in accordance with design requirements, ergonomic considerations with the pilot preferences. RLD model optimizes interface design by deploying indicators to the best locations to improve usability of display panel, pilot-aircraft interaction and flight safety.FindingsOptimum interfaces for two problem instances were gathered by RLD model in 15.77 CPU(s) with 10 indicators and 542.51 CPU(s) with 19 indicators. A comparison between relative and existing cockpit interfaces reveals that locations of six navigation and four mechanical system indicators are different. The differences may stem from pilots’ preferences and relativity constraints. Both interfaces are more similar for the central part of the display panel. The objective function value of relative interface design (Opt: 527938) is far better than existing interface (737100). The RLD model improved usability of existing interface (28.61 per cent considering decrease in the objective function values from 737100 to 527938.Practical implicationsFuture cockpit and new helicopter interface designs may involve RLD model as an alternative interface design tool. Furthermore, other layout optimization problems, e.g. circuit boards, microchips and engines, etc. could be handled in a more realistic manner by RLD model.Originality/valueOriginality and impact of this study related to development and employment of a new optimization model (RLD) on cockpit interface design for the first time. Engineering requirements, human factors, ergonomics and pilots’ preferences are simultaneously considered in the RLD model. The subjectivity within usability evaluation techniques could be diminished in this way. The contributions of RLD model to classical facility layout models are relativity constraints with the physical constrictions and ergonomic objective function weights. Novelty of this paper is the development and employment of a new optimization model (RLD) to locate indicators.

Employing a user-centered cognitive walkthrough to evaluate a mHealth diabetes self-management application: A case study and beginning method validation.

Self-management of chronic diseases using mobile health (mHealth) systems and applications is becoming common. Current evaluation methods such as formal usability testing can be very costly and time-consuming; others may be more efficient but lack a user focus. We propose an enhanced cognitive walkthrough (CW) method, the user-centered CW (UC-CW), to address identified deficiencies in the original technique and perform a beginning validation with think aloud protocol (TA) to assess its effectiveness, efficiency and user acceptance in a case study with diabetes patient users on a mHealth self-management application. A total of 12 diabetes patients at University of Utah Health, USA, were divided into UC-CW and think aloud (TA) groups. The UC-CW method included: making the user the main evaluator for detecting usability problems, having a dual domain facilitator, and using three other improved processes: validated task development, higher level tasks and a streamlined evaluation process. Users interacted with the same mHealth application for both methods. Post-evaluation assessments included the NASA RTLX instrument and a set of brief interview questions. Participants had similar demographic characteristics. A total of 26 usability problems were identified with the UC-CW and 20 with TA. Both methods produced similar ratings: severity across all views (UC-CW = 2.7 and TA = 2.6), numbers of problems in the same views (Main View [UC-CW = 11, TA = 10], Carbohydrate Entry View [UC-CW = 4, TA = 3] and List View [UC-CW = 3, TA = 3]) with similar heuristic violations (Match Between the System and Real World [UC-CW = 19, TA = 16], Consistency and Standards [UC-CW = 17, TA = 15], and Recognition Rather than Recall [UC-CW = 13, TA = 10]). Both methods converged on eight usability problems, but the UC-CW group detected five critical issues while the TA group identified two. The UC-CW group identified needed personalized features for patients' disease needs not identified with TA. UC-CW was more efficient on average time per identified usability problem and on the total evaluation process with patients. NASA RTLX scores indicated that participants experienced the UC-CW half as cognitively demanding. Common themes from interviews indicated the UC-CW as enjoyable and easy to perform while TA was considered somewhat awkward and more cognitively challenging. UC-CW was effective for finding severe, recurring usability problems and it highlighted the need for personalized user features. The method was also efficient and had high user acceptance. These results indicate UC-CW's utility and user acceptance in evaluating a mHealth self-management application. It provides an additional usability evaluation technique for researchers.

Evaluation of Usability Dimensions of Smartphone Applications

This study analyses different techniques used for evaluation of various usability dimensions of software applications (apps) being used on the smartphones. The scope of this study is to evaluate various aspects of the usability techniques employed in the domain of smartphone apps. Usability assessment methodologies are evaluated for different types of applications running on different operating systems like Android, Blackberry and iOS, etc. Usability evaluation techniques and methodologies with respect to usability heuristics like field experiments, laboratory experiments models and usability standards are discussed in detail. The issues for evaluation of usability of smartphone apps are identified by considering limitations and areas of improvement outlined in the contemporary literature. A conceptual framework for usability evaluation of smartphone apps is also designed which would be validated through experimentation in the thesis work. This study is particularly useful to comprehend usability issues and their likely remedies to produce high quality smartphone apps.

Heuristic Evaluation of Serious Game Application for Slow-reading Students

The findings of preliminary studies found that conventional approaches were still relevant but students showed weak and moderate interest and quickly lost focus rather than technology approaches such as serious games were used especially for slow reading students (SRS). Most teachers use interventions that are not specifically designed to help SRS. They usually use teaching aids below the literacy level of the SRS. Therefore, an easy and user-friendly games application called “Mari Membaca” or M2M was developed. The objective is to make sure the application is free from design and interface problems by demonstrating the application of expert-based usability evaluation techniques such as Heuristic evaluation. This paper reports the experimental heuristic evaluation of M2M for SRS among expert evaluators includes remedial teachers and game developers. This study adopted ten Usability Heuristics and seven brain-compatible instructional phases of brain-based learning to be included in the questionnaire. The overall result derived from the evaluation is 14 out of 17 (3.41-5.00) above average mean score, which are neutral (2.61-3.40) in one domain. Several comments and feedback from the experts were essentials for further improvement of the game application to ensure meets the user requirement and expectation.

EVALUATING CONSTRUCTION DEFECT MOBILE APPLICATION USING THINK ALOUD

Construction Defect mobile application (app) is used by Ministry of Defense's Construction Defect Inspection Team to help the team in recording construction defect. However, the mobile app has never been evaluated formally to identify any usability problems. The objectives of this study are to identify usability problems of Construction Defect mobile app using think aloud and to recommend design improvement of Construction Defect mobile app based on the identified usability problems. The think aloud study involved 15 participants. During the evaluation, every participant carried out the given tasks and gave his impressions as he went along the tasks. Three usability problems were identified. Some recommendations have been proposed to improve the design of Construction Defect mobile app. As for the future work, the study may be conducted using different usability evaluation technique.

The design of manual domain usability evaluation techniques

AbstractIn this paper we design novel techniques for manual domain usability evaluation. Domain usability is the aspect of a particular user interface that relates to its terminology, hierarchy of terms, feature descriptions and icons, used language and consistency. However, domain usability is often neglected not only by software developers, but also by many researchers. The design of formal means to evaluate existing user interfaces would aid the development of better, domain-usable user interfaces. We designed six qualitative evaluation techniques and one formal evaluation technique based on System Usability Scale (SUS). To show the viability of the designed techniques, we demonstrate each of them in the domain of gospel music. Two of the techniques were used to experimentally verify the impact of specific domain usability aspects on usability and user experience. In this paper we also focus on the equality of the domain usability aspects with the goal of designing domain usability metrics. The preliminary design of the metrics is also presented in this paper and areas of future research are suggested.

Combining usability evaluations to highlight the chain that leads from usability flaws to usage problems and then negative outcomes

Poor usability of health technology is thought to diminish work system performance, increase error rates and, potentially, harm patients. The present study (i) used a combination of usability evaluation methods to highlight the chain that leads from usability flaws to usage problems experienced by users and, ultimately, to negative patient outcomes, and (ii) validated this approach by studying two different discharge summary production systems. To comply with quality guidelines, the process of drafting and sending discharge summaries is increasingly being automated. However, the usability of these systems may modify their impact (or the absence thereof) in terms of production times and quality, and must therefore be evaluated. Here, we applied three successive techniques for usability evaluation (heuristic evaluation, user testing and field observation) to two discharge summary production systems (underpinned by different technologies). The systems’ main usability flaws led respectively to an increase in the time need to produce a discharge summary and the risk of patient misidentification. Our results are discussed with regard to the possibility of linking the usability flaws, usage problems and the negative outcomes by successively applying three methods for evaluating usability (heuristic evaluation, user testing and in situ observations) throughout the system development life cycle.

Heuristic Evaluation of an Institutional E-learning System: A Nigerian Case

Many African academic institutions have adopted the use of e-learning systems, since they enable students to learn at their own pace, time, and without restriction to the classroom. However, evidence of usability evaluation of e-learning systems in Africa is mostly lacking in the literature. This paper reports the experimental heuristic evaluation of the e-learning system of a Nigerian University. The objective is to demonstrate the application of expert-based usability evaluation techniques such as Heuristic evaluation for assessing the attributes of existing e-learning systems. The study revealed that while the e-learning systems has strong credentials in terms of support for Web 2.0 activities, good learning content and boasts of useful e-learning features, improvements are necessary in other areas such as interactive learning, assessment and feedback, and quality of learning content. The study adds to the body of extant knowledge in the area of usability evaluation of e-learning systems in African institutions.

Quality assurance of academic websites using usability testing: an experimental study with AHP

Considering the competitive business scenario of the IT world, websites are the key aspect for satisfying the user’s demands. Thus, usability evaluation is the hidden measure to succeed in this competitive environment. Measuring user satisfaction is a hindering job because of lack of appropriate methodologies. This paper addresses these issues by considering AHP based usability evaluation technique to measure the usability score of a website. The proposed methodology evaluates user satisfaction for three different websites based on feedback mechanism for different usability attributes. Feedback is collected using Questionnaire methodology. AHP is a multi-criteria decision making algorithm that proves to be the reliable way for the user to make a decision for choosing the best website that fulfills user satisfaction.

A Systematic Mapping Review of Usability Evaluation Methods for Software Development Process

Given that usability is one of the most important aspects of software quality, several methods have been developed in order to establish techniques capable of evaluating this attribute from early phases of the software development process. However, the choice of the most appropriate method for a particular scenario is still a difficult decision, due to the existence of a vast number of approaches that are described in the literature for this purpose. Therefore, a systematic mapping review was conducted in order to identify the most commonly used usability evaluation techniques in software developments. A total of 1169 studies were identified, of which only 215 studies were selected for this review. According to the analysis, most cases studies establish the use of usability questionnaires as assessment tool. In addition, health informatics and Web applications are the software domain and type of application that are frequently reported in these evaluations. This work has allowed to reach promising results in this area. It is intended to be a guide for specialists to support the choice of the most suitable method for a particular scenario.

Feasibility and usability of a wearable orthotic for stroke survivors with hand impairment

Purpose: The concept of a vibrating wristband, to improve dextrous hand function of stroke survivors, was recently proposed with clinical results and is referred to as ‘TheraBracelet’ in this paper. The purpose of this study was to demonstrate feasibility of a portable, wearable TheraBracelet, and to apply usability evaluation techniques to assess potential demands of TheraBracelet and to identify critical improvement needs of the prototype. Method: A prototype was developed with a vibrating element housed in an elastic wristband and connected to a wearable electronics box via a cable. Expectation for TheraBracelet and evaluation of the prototype were obtained from 10 chronic stroke survivors using surveys before and after using the prototype and House of Quality analysis. Results: The survey for expectation showed stroke survivors’ willingness to try out TheraBracelet at a low cost. The survey evaluating the prototype showed that the current prototype was overall satisfactory with a mean rating of 3.7 out of 5. The House of Quality analysis revealed that the priority improvement needs for the prototype are to improve clinical knowledge on long-term effectiveness, reduce cost, ease donning/doffing and waterproof.Conclusions: This study presents a potential for a low-cost wearable hand orthotic likable by stroke survivors.Implications for RehabilitationFeasibility for a portable wearable wristband-type hand orthotic was demonstrated.The survey showed stroke survivors are willing to try such an orthotic at low cost.The current prototype was rated overall satisfactory by stroke survivors.This study provides a potential for a low-cost wearable hand orthotic likable by stroke survivors.

Usability Evaluation Technique of HMI Based on Bidirectional Primary-Secondary Task Method Using Driver’s Decision Vagueness Index in Vehicle Following

Usability Evaluation Technique of HMI Based on Bidirectional Primary-Secondary Task Method Using Driver’s Decision Vagueness Index in Vehicle Following

A quantitative approach to evaluate usability of academic websites based on human perception

In this competitive world, websites are considered to be a key aspect of any organization’s competitiveness. In addition to visual esthetics, usability of a website is a strong determinant for user’s satisfaction and pleasure. However, lack of appropriate techniques and attributes for measuring usability may constrain the usefulness of a website. To address this issue, we conduct a statistical study to evaluate the usability and accessibility levels of three popular academic websites based on human (user) perception. Two types of usability evaluation techniques were employed in this study. First one is Questionnaire-based evaluation and second one is performance-based evaluation. Usability assessment was performed by analyzing the results from the observed task success rates, task completion times, post-task satisfaction ratings and feedback. We also investigate the possibility of there being any impact of task completion times on participant’s satisfaction levels. The results of the questionnaire based evaluation were observed to be consistent with the results of performance-based evaluation. Accessibility evaluation was carried out by testing the degree of compliance of the web pages as per WCAG 2.0 guidelines.

Usability Testing of an E- Learning System: A Comparative study of two Evaluation Techniques

E- Learning is the most feasible way to hold distance learning by providing learning materials over the internet; thus trend is becoming popular in developed and developing countries. This learning style becomes difficult to use by learners, usually by vague interface design. Due to which learner/user becomes reluctant towards E- Learning system. Usability testing of E- Learning systems targets its interface design, usability and human computer interaction. This study is an attempt to uncover the problems that occur in the use of such systems at user end. A survey from learners of an E- Learning system is carried out and result are compared by another usability evaluation technique named Heuristic Evaluation commonly employed by experts. Both end users and usability expert participated in this research by which two different evaluation methods are used to evaluate the usability of Virtual University Learning Management System. At the end some suggestions are given in order to improve the usability of target system after validating the results.

Using an empirical study to evaluate the feasibility of a new usability inspection technique for paper based prototypes of web applications

Usability is one of the most important factors that determine the quality of Web applications, which can be verified performing usability inspection. This paper presents the Web Design Usability Evaluation (Web DUE) technique, which allows the identification of usability problems in low-fidelity prototypes (or mockups) of Web applications during the design phases of the development. We have also proposed the Mockup Design Usability Evaluation (Mockup DUE) tool which is able to assist inspectors using the Web DUE technique. In order to verify the feasibility of these technologies, we have performed two empirical studies. During the first study, we compared the effectiveness and efficiency indicators of the Web DUE technique with the ones of its predecessor, the Web Design Perspective (WDP) based usability inspection technique. Also, during the second study, experienced inspectors used the Mockup DUE tool and answered a questionnaire aiming at identifying improvement opportunities in its design. The analysis of the quantitative data showed that the Web DUE technique allowed the identification of more usability problems in lesser time when compared to the WDP technique. Moreover, the qualitative data from the second empirical study provided information on the tool’s perceived ease of use, indicating that inspectors were satisfied and that they would use it to perform a usability inspection with the Web DUE technique. These results showed that the DUE technologies could be applied in the identification of usability problems early in the design of Web applications. Thus, their use could enable the correction of such problems before the source code of the application is written.