Usability Evaluation Methodology Research Articles

Using Eye Tracking to Measure Overall Usability of Online Grocery Shopping Websites

This article examines usability evaluation methodologies, then presents a non-conscious behavioral indicator based on user eye movements and pupil dilation. The authors test how gender and online buying history affect the behavioral index's usability scores. This study uses three Iranian online food retailers. Thirty participants were asked to add things from predetermined grocery stores to virtual shopping carts before the experiment took them to the other two websites in a random order to collect eye movement data. Each group's presentation order was randomized. The number of fixations, number of saccades, total duration of fixations, scan-path length, pupil size, and task time were inversely linked with self-report usability measures. This research evaluates groups with different levels of online shopping expertise and gender based on experienced usability. Differences between groups suggest that user demographics affect usability.

Remote Usability Testing to Facilitate the Continuation of Research.

Usability testing has historically been an in-person activity where test participants and evaluation researchers are co-located. Recruiting participants into usability studies can be a challenging endeavor especially when potential participants are concerned about time commitments and social distancing. The global COVID-19 pandemic has driven the development of remote usability testing methods. In this paper, we describe remote usability testing as it evolved during a pre-pandemic research study. We adapted our in-person usability evaluation methodology for a commercially available mHealth app to a remote usability testing methodology to accommodate potential participants during a more convenient participant-identified time. In doing so we met the needs, preferences, and availability of our participants and maintained research progress. Adapting to patient-centered needs through remote usability testing has the potential to facilitate continued research and engage potential participants due to its convenience, flexibility, and decrease constraints presented by geographic limits.

Heuristic Evaluation of Peruvian Government Web Portals, used within the State of Emergency

The development of web platforms is very abundant at present, this event has developed exponentially due to the state of emergency. Therefore, there is a need to evaluate the quality of these platforms to ensure a good user experience, especially if these platforms are governmental. Currently, there are several platforms that have been developed by governments of different nations, which are related to the theme of covid-19; which should be evaluated from a heuristic perspective to detect usability problems that can occur when users interact with a product and identify ways to solve them. This article presents a heuristic evaluation of Peruvian government web portals, used within the state of emergency; for this purpose, a heuristic evaluation is carried out using a list of 15 heuristic principles, proposed by Toni Granollers, from two government platforms: Aprendo en casa and Covid19 Minsa. In this way, it was identified that the Peruvian platform Aprendo en casa has fewer usability problems compared to the Covid19 Minsa platform. Therefore, there is a need to renew or update the Covid19 Minsa platform using the results of the heuristic evaluation performed; on the other hand, although a heuristic evaluation of these Peruvian government platforms is being carried out, it is recommended to continue with a research that uses other usability evaluation methodologies for other platforms of daily use, such as the S/ 380 bonus platform, Bonus for independent or AFP withdrawal.

Using cognitive dimensions to evaluate the usability of security APIs: An empirical investigation

Abstract Context Usability issues of security Application Programming Interfaces (APIs) are a main factor for mistakes programmers make that could result in introducing security vulnerabilities into applications they develop. This has become a common problem as there is no methodology to evaluate the usability of security APIs. A usability evaluation methodology for security APIs would allow API developers to identify usability issues of security APIs and fix them. A Cognitive Dimensions Framework (CDF) based usability evaluation methodology has been proposed in previous research to empirically evaluate the usability of security APIs. Objective In this research, we evaluated the proposed CDF based methodology through four security APIs (Google Authentication API, Bouncy Castle light weight Crypto API, Java Secure Socket Extension API, OWASP Enterprise Security API). Method We conducted four experiments where in each experiment we recruited programmers and they completed a programming task using one of the four security APIs. Participants’ feedback on each cognitive dimension of the particular API was collected using the cognitive dimensions questionnaire. Usability issues of each API was identified based on this feedback. Results Results of the four experiments revealed that over 83% of the usability issues in a security API could be identified by this methodology with a considerably good validity and reliability. Conclusion The proposed CDF based usability evaluation methodology provides a good platform to conduct usability evaluation for security APIs.

Evaluation of Usability Dimensions of Smartphone Applications

This study analyses different techniques used for evaluation of various usability dimensions of software applications (apps) being used on the smartphones. The scope of this study is to evaluate various aspects of the usability techniques employed in the domain of smartphone apps. Usability assessment methodologies are evaluated for different types of applications running on different operating systems like Android, Blackberry and iOS, etc. Usability evaluation techniques and methodologies with respect to usability heuristics like field experiments, laboratory experiments models and usability standards are discussed in detail. The issues for evaluation of usability of smartphone apps are identified by considering limitations and areas of improvement outlined in the contemporary literature. A conceptual framework for usability evaluation of smartphone apps is also designed which would be validated through experimentation in the thesis work. This study is particularly useful to comprehend usability issues and their likely remedies to produce high quality smartphone apps.

Metodología de evaluación de usabilidad de interfaces humano-máquina

Este artículo presenta la aplicación y validación de una metodología para la evaluación de la usabilidad de interfaces humano-máquina (IHM) en sistemas de supervisión industrial. Inicialmente, se presentan los aspectos fundamentales de evaluación de usabilidad de sistemas basados en software. Luego, se describe la metodología, que utiliza un sistema de adquisición de datos para supervisión y control de procesos (SCADA), un sistema de registro de eventos (EVENT LOGGER), un programa de clasificación de eventos en criterios de usabilidad, y un sistema adaptativo de inferencia neuro-difusa (ANFIS). La metodología fue aplicada a varias interfaces de operador para la monitorización de una planta. Finalmente, para comprobar la validez de la metodología se realizó una prueba de hipótesis frente a otras dos metodologías tradicionales: inspección por el experto e indagación a los usuarios. Se determinó que la metodología propuesta tiene equivalencia con las otras dos, pero tiene la ventaja de ser completamente automatizada. Los resultados permitirán ser retomados para su aplicación generalizada en el diseño de interfaces humano-máquina en cualquier sistema tecnológico para mejorar las prestaciones de la evaluación.

Merits of usability testing for PACS selection

ObjectivesTo compare the usability of different Picture Archiving and Communication System (PACS) workstations, determine whether a usability test has added value with respect to the traditional way of comparing PACSs based on functional requirements, and to evaluate the appropriateness of a task-based methodology for a PACS usability test. MethodsA task-based usability test of four PACS workstations was performed. Radiologists’ subjective responses to the PACSs and their performance on the tasks were measured. To mimic the traditional PACS selection process, two functional requirements were defined which the PACSs met in varying degrees. The focus of the usability test was on the aspects of the PACS related to these requirements. The usability results were compared to the PACSs’ ability to meet the functional requirements. ResultsOne PACS outperformed the other PACSs both in terms of subjective preference and task performance, indicating its superior usability. There were differences in usability between PACSs with identical functionality. Also, a PACS with theoretically advantageous functionality for a given task did not necessarily have better usability for this task than a PACS without this functionality. There was a discrepancy between participants’ subjective preferences and their task performance, which indicates that it is vital to include performance measures in the usability assessment so that it accurately reflects the efficiency of interaction. ConclusionsThe differences in usability between PACSs with identical functionality indicate that functional requirements alone are insufficient to determine a PACS's overall quality. A usability test should therefore be used in addition to a functional requirement list in a PACS selection process to ensure that a hospital buys the PACS with the highest quality. A task-based usability evaluation methodology, which yields both subjective preference data and objective performance data of radiologists interacting with the PACS, is very suitable for such a usability test.

Una Metodología Automatizada para la Evaluación de Usabilidad de Interfaces de Supervisión Industrial

The development, application and validation of a methodology for usability evaluation of interfaces of industrial supervision systems are presented. The fundamental aspects of usability evaluation software based systems are described and the methodology is presented. The methodology was applied in a laboratory to monitor various interfaces of a liquid dosing process. The system is used to determine the value of the supervision interface for the interaction between the operator and the process. The ratings were useful for classifying interface usability. To check the validity of the methodology, a hypothesis test was done and results compared to two traditional methods: expert inspection and inquiry to users. It was determined that the proposed methodology is equivalent to the other two methods, but it has the advantage of being completely automated.

Avaliação heurística de usabilidade em jornais online: estudo de caso em dois sites

O trabalho foca a usabilidade em sites de jornais eletrônicos. São descritas as principais metodologias de avaliação de usabilidade propostas na literatura, e adota-se, dentre elas, a avaliação heurística de Nielsen, para analisar a usabilidade dos sites dos jornais Estado de São Paulo e Folha de São Paulo. Uma vez detalhados os resultados das avaliações, apresentam-se as considerações finais, nas quais destacam-se a concentração de violações nas heurísticas sobre visibilidade do estado do sistema; a consistência e padrões; e a prevenção de erros.

An open source usability maturity model (OS-UMM)

User satisfaction has always been a major factor in the success of software, regardless of whether it is closed proprietary or open source software (OSS). In open source projects, usability aspects cannot be improved unless there are ways to test and measure them. Hence, the increasing popularity of open source projects among novice and non-technical users necessitates a usability evaluation methodology. Consequently, this paper presents a usability maturity model specifically aimed at usability-related issues for open source projects. In particular, the model examines the degree of coordination between open source projects and their usability aspects. The measuring instrument of the model contains factors that have been selected from four of our empirical studies, which examine the perspectives of OSS users, developers, contributors and the industry. In addition to presenting the usability maturity model, this paper discusses assessment questionnaires, a rating methodology and two case studies.

Ergonomics and Fashion: the OIKOS methodology for usability and comfort evaluation in clothing and fashion

The purpose of this article is to present a methodological proposition, named OIKOS, to evaluate comfort and us- ability in clothing and fashion products. The development of the methodology was based on the need to underlay the project of clothing and fashion products with ergonomic and usability principles in all the stages of the product development, since its conception. The theoretical foundation lays on the introduction of ergonomics and usability as fundamental variables while developing clothing and fashion product projects, to obtain comfort. The OIKOS Methodology proposition incorporates ergo- nomic features, usability principles and comfort criteria for clothing and fashion products. These will be guidelines to fulfill the user different requirements relating to the clothing and fashion products adjustment and quality. Besides reaching the in- tended comfort, this methodological proposition contributes to extend the product's life cycle, reducing the environment im- pact caused by its early disposal, and at the same time establishes an emotional connection between the user and the product.

Combining experimental and inquiry methods in software usability evaluation

PurposeThe purpose of this paper is to present a methodology combining experimental and inquiry methods used for software usability evaluation. The software product of LeViS project funded by the European Commission (Socrates/Lingua II) is used as an evaluation paradigm. The aim of the paper is twofold: to present the results of the usability evaluation using this software as an example and to suggest a number of improvements for the next version of the software tool; and to portray the advantages of combining methods from different evaluation approaches and the experiences from their application.Design/methodology/approachThe evaluation for this experiment combined different usability methods, both experimental and inquiry ones. More specifically, the methods employed were the Thinking Aloud Protocol and the User Logging, which were performed in a usability evaluation laboratory, as well as the inquiry methods of Interview and Focus Group.FindingsIn this study, usability problems regarding the Learning via Subtitling (LvS) educational software were revealed as well as issues regarding the use of Thinking Aloud Protocol method and involving users with a specific profile. The research findings presented in this paper constitute an innovative and effective methodology for software usability evaluation and are useful for laboratories aiming to conduct similar evaluations.Research limitations/implicationsAlthough this methodology has been successfully applied for over 20 software products, due to practical purposes related to this paper's extent, only one software is used as an example.Originality/valueThrough the evaluation process, apart from discovering certain usability problems related to the software, there are a number of important conclusions drawn, regarding the methods used and the methodology followed in software usability evaluation.

A complex physiology-based empirical usability evaluation method in practice

This paper outlines the INTERFACE usability evaluation methodology developed by researchers of our department. It is based on the simultaneous assessment of Heart Period Variability (HPV), Skin Conductance (SC), and other data. One of the highlights of this methodology is its capability to identify quality attributes of software elements with a time-resolution of only a few seconds: in particular cases it can assess 2- or 3-second events. The Department of Ergonomics and Psychology at the Budapest University of Technology and Economics carried out applied research projects assessing very various software. After these, we can show different types of typical software problems identified by our method. The method of analysis allows us not only to decide what types of problems are significant to the users; however, on the other hand, the method allows us to decide, to what extent the found problems and their assessed severity concern all the users in general, or how these things depend on the type and characteris tics of the users.

Field-Based Usability Evaluation Methodology for Mobile Geo-Applications

An important aspect of a Dutch research project into usable (and well scaled) mobile maps for consumers is presented: the development of an appropriate field-based usability evaluation methodology for the prototype of a geo-mobile application that will be the result of a user centred design approach. Automatic generalisation, required for the user's orientation in space, but also for progressive data transfer, will be an important aspect of the prototype. What is reported here is an effective and technically unique user research methodology, based on a combination of video observation, thinking aloud and semi-structured interviewing. The experiments that have led to this outcome revealed some interesting usability issues that deserve further investigation.

Embedding accessibility and usability: considerations for e-learning research and development projects

This paper makes the case that if e-learning research and development projects are to be successfully adopted in real-world teaching and learning contexts, then they must effectively address accessibility and usability issues; and that these need to be integrated throughout the project. As such, accessibility and usability issues need to be made explicit in project documentation, along with allocation of appropriate resources and time. We argue that accessibility and usability are intrinsically inter-linked. An integrated accessibility and usability evaluation methodology that we have developed is presented and discussed. The paper draws on a series of mini-case studies from e-learning projects undertaken over the past 10 years at the Open University.DOI: 10.1080/09687760701673659

Embedding accessibility and usability: considerations for e-learning research and development projects

This paper makes the case that if e-learning research and development projects are to be successfully adopted in real-world teaching and learning contexts, then they must effectively address accessibility and usability issues; and that these need to be integrated throughout the project. As such, accessibility and usability issues need to be made explicit in project documentation, along with allocation of appropriate resources and time. We argue that accessibility and usability are intrinsically inter-linked. An integrated accessibility and usability evaluation methodology that we have developed is presented and discussed. The paper draws on a series of mini-case studies from e-learning projects undertaken over the past 10 years at the Open University.DOI: 10.1080/09687760701673659

An approach to usability evaluation of e-learning applications

Despite recent advances of electronic technologies in e-learning, a consolidated evaluation methodology for e-learning applications is not available. The evaluation of educational software must consider its usability and more in general its accessibility, as well as its didactic effectiveness. This work is a first step towards the definition of a methodology for evaluating e-learning applications. Specific usability attributes capturing the peculiar features of these applications are identified. A preliminary user study involving a group of e-students, observed during their interaction with an e-learning application in a real situation, is reported. Then, the proposal is put forward to adapt to the e-learning domain a methodology for systematic usability evaluation, called SUE. Specifically, evaluation patterns are proposed that are able to drive the evaluators in the analysis of an e-learning application.

Portal corporativo: conceitos e características

Conceitos a respeito de uma nova tecnologia conhecida como portal corporativo, cuja proposta é facilitar o acesso às informações digitais no contexto organizacional. Este artigo apresenta definições, características básicas, arquitetura e tipos de portais corporativos identificados na literatura especializada, constituindo-se em parte da revisão de literatura de projeto de pesquisa em andamento, cujo objetivo geral é analisar métodos para avaliação de usabilidade de portais corporativos

A broadband multimedia collaborative system for advanced teleradiology and medical imaging diagnosis

This paper presents a new telemedicine system currently in routine clinical usage, developed within the European Union (EU) ACTS BONAPARTE project (1). The telemedicine system is developed on an asynchronous transfer mode (ATM) multimedia hardware/software platform comprising the following set of telemedicine services: synchronous cooperative work, high-quality video conference, multimedia mail, medical image digitizing, processing, storing and printing, and local and remote transparent database access. The medical information handled by the platform conforms to the Digital Imaging and Communications in Medicine (DICOM) 3.0 medical imaging standard. The telemedicine system has been installed for clinical routines in three Spanish hospitals since November 1997 and has been used in an average of one/two clinical sessions per week. At each clinical session, a usability and clinical evaluation of the system was carried out. Evaluation is carried out through direct observation of interactions and questionnaire-based subjective data. The usability evaluation methodology and the results of the system usability study are also presented in this article. The experience gained from the design, development, and evaluation of the telemedicine system is providing an indepth knowledge of the benefits and difficulties involved in the installation and clinical usage of this type of high-usability and advanced multimedia telemedicine system in the field of teleradiology and collaborative medical imaging diagnosis.

A systematic method for hypermedia usability inspection

This paper will discuss a usability inspection method for hypermedia applications, that explicitly takes into account the particular features of this class of systems. The proposed technique is a specialisation for hypermedia of a general methodology for usability evaluation of interactive systems, named SUE -Systematic Usability Evaluation. Our approach aims to support a systematic evaluation process, making it well organised, fast and cheap, and is based on the use of a hypermedia model, a set of hypermedia-specific usability attributes, and a set of patterns of inspection activities, called abstract tasks. The model identifies the ‘dimensions’ along which hypermedia usability can be analysed, defines the application constituents that must be inspected, and allows a precise formulation of usability attributes and abstract tasks. tasks provide operational guidelines for systematically checking usability attributes throughout the application. To exemplify our approach, we will report some inspection results concerning nine commercially available hypermedia CD-ROMs