Usability Metrics Research Articles

Sirius: A heuristic-based framework for measuring web usability adapted to the type of website

The unquestionable relevance of the web in our society has led to an enormous growth of websites offering all kinds of services to users. In this context, while usability is crucial in the development of successful websites, many barely consider the recommendations of experts in order to build usable designs. Including the measurement of usability as part of the development process stands out among these recommendations. One of the most accepted methods for usability evaluation by experts is heuristic evaluation. There is abundant literature on this method. However, there is a lack of clear and specific guidelines to be used in the development and evaluation process. This is probably an important factor contributing to the aforementioned generalized deficiency in web usability.We miss an evaluation method based on heuristics whose measure is adapted to the type of evaluated website. In this paper we define Sirius, an evaluation framework based on heuristics to perform expert evaluations that takes into account different types of websites. We also provide a specific set of evaluation criteria, and a usability metric that quantifies the usability level achieved by a website depending on its type.

Comparing two anesthesia information management system user interfaces: a usability evaluation

Anesthesia information management systems (AIMS) have been developed by multiple vendors and are deployed in thousands of operating rooms around the world, yet not much is known about measuring and improving AIMS usability. We developed a methodology for evaluating AIMS usability in a low-fidelity simulated clinical environment and used it to compare an existing user interface with a revised version. We hypothesized that the revised user interface would be more useable. In a low-fidelity simulated clinical environment, twenty anesthesia providers documented essential anesthetic information for the start of the case using both an existing and a revised user interface. Participants had not used the revised user interface previously and completed a brief training exercise prior to the study task. All participants completed a workload assessment and a satisfaction survey. All sessions were recorded. Multiple usability metrics were measured. The primary outcome was documentation accuracy. Secondary outcomes were perceived workload, number of documentation steps, number of user interactions, and documentation time. The interfaces were compared and design problems were identified by analyzing recorded sessions and survey results. Use of the revised user interface was shown to improve documentation accuracy from 85.1% to 92.4%, a difference of 7.3% (95% confidence interval [CI] for the difference 1.8 to 12.7). The revised user interface decreased the number of user interactions by 6.5 for intravenous documentation (95% CI 2.9 to 10.1) and by 16.1 for airway documentation (95% CI 11.1 to 21.1). The revised user interface required 3.8 fewer documentation steps (95% CI 2.3 to 5.4). Airway documentation time was reduced by 30.5 seconds with the revised workflow (95% CI 8.5 to 52.4). There were no significant time differences noted in intravenous documentation or in total task time. No difference in perceived workload was found between the user interfaces. Two user interface design problems were identified in the revised user interface. The usability of anesthesia information management systems can be evaluated using a low-fidelity simulated clinical environment. User testing of the revised user interface showed improvement in some usability metrics and highlighted areas for further revision. Vendors of AIMS and those who use them should consider adopting methods to evaluate and improve AIMS usability.

Analysis of Accessibility Initiatives Applied to the Web

The objective of this study is to investigate the current accessibility initiatives applied to the Web and identify its main characteristics and trends. The method used was a literature review focused on work in the last twenty years in the United States, United Kingdom, Australia, Canada and Brazil, ranked according to their approaches, in six lines of study. As a result, points out the lack of studies that allow effective analysis of the accessibility of websites, and methods for continuous improvement of technical standards and recommendations. It follows, showing the application of usability metrics by Nielsen and ISO-9241 and the incorporation of research models of information behavior seeking for assessment of gaps in accessibility.

He Says, She Says: Does Voice Affect Usability?

Interactive voice response systems (IVRs) are ubiquitous user interfaces that allow customers to gather information and execute transactions. Because the entire user interaction is auditory-based, the voice used in the IVR is of high importance. Entities that field IVRs go to great lengths to select appropriate voices, usually on the basis of aesthetics of the voice, or on branding considerations. While much has been written on the usability of these IVR systems from the perspective of navigation, the impact of the voice on the usability of IVRs has not been well explored. Towards this end, we investigated the impact of voice personality and speaker gender on the usability of an interactive voice response system. Results suggest that voice characteristics can have an impact on the perceived usability of the system, with male voices resulting in higher usability metrics than female voices.

A direct touch table-top display as a multi-user information kiosk: Comparing the usability of a single display groupware either by a single user or people cooperating as a group

This study aimed to investigate the usability of a direct touch table-top display (D3TD) application that was designed to be used as a multi-user information kiosk (MUIK). We examined its use by either a single person or multiple user groups to determine the effect of people cooperating as a group on its usability. We compared both groups’ user performance and usability survey results. A one-way MANOVA of overall usability metrics emphasised a significant difference between the groups. Given the significance of the overall test, examining the univariate main effects revealed that the difference between the groups arose in the earlier system use stages due to differences in the success rate. There was no difference in the task completion time between the groups. Participants in the multiple user group achieved a higher task completion rate in the first phase and lower error rates in the following steps. However, the usability survey results did not reveal a significant difference between the groups. We observed that users working on the same individual task tended to help each other when working in a multi-user environment. These mutual aids occurred mostly in the earlier stages, influencing the users’ performance. The results show that a D3TD has a potential to enhance usability in terms of effectiveness when deployed as a MUIK.

Enterprise Transformation to Enable University–Industry Collaboration: A Case Study in Complexity and Usability

Enterprises need to make continuous fundamental transformations—such as improving current business processes, performing entirely different tasks, and conducting automated business processes—to maintain or gain competitive advantage. These transformations may increase value or decrease time, costs, and uncertainties. However, it is difficult to choose transformations that deserve major investment without assessing the relative value of alternative transformations. Analyzing and redesigning business processes to ensure consistency with business requirements and information technology (IT) specifications is a critical factor for successful enterprise transformation. This paper provides an evolution methodology based on process complexity to implement effective and efficient best practices for enterprise transformation. This paper uses a process complexity and usability metrics, combining software science and cognitive science, to evaluate the cognitive loading of the business processes. Furthermore, to illustrate the metric, this paper describes an IT-driven enterprise transformation to enable university–industry collaboration. The purpose of this study is to evaluate the need for conducting operations with and without the use of information technology. The complexity model shows a more than 60% decrease in the complexity, suggesting that the IT-integrated process is less complex than earlier processes.

Use of In Vivo Confocal Microscopy to Assess Allergic Inflammation following Conjunctival Allergen Challenge

In this study, we used confocal microscopy following conjunctival allergen challenge (CAC) to track the progress of the acute allergic response and the subsequent ocular late phase inflammation. Imaging data provided the basis for development of a reproducible scale for assessment of late phase allergy. Patients (10) with history of allergic conjunctivitis were enrolled in this pilot study. The ocular surface of subjects was imaged before (as a baseline) and after a single CAC, conducted according to established protocols. Confocal images were captured at regular intervals from 3 minutes to 180 minutes post-challenge using a Nidek Confoscan scanning confocal microscope. Image analysis was used to develop a scale for extent of inflammatory cell infiltration. Clinical symptoms were scored in parallel to imaging. This scale was then used to score individual subjects (in a blinded fashion) to provide numerical comparators to baseline measures. Conjunctival Allergen Challenge (CAC) caused a significant increase in cellular infiltration compared to baseline. While the sample size was small, comparison of all measured parameters suggests that the late-phase scales accurately reflect the subject symptomology. The use of in vivo confocal microscopy has great potential to aid clinical diagnosis of anterior segment diseases. This study represents an important step in development of usable metrics to assess the time course and extent of allergic ocular inflammation. Scales developed in this study should be useful in diagnosing presence of allergic disease and in assessing disease severity as well as therapeutic efficacies of potential treatments.

Analysis of Accessibility Initiatives Applied to the Web

The objective of this study is to investigate the current accessibility initiatives applied to the Web and identify its main characteristics and trends. The method used was a literature review focused on work in the last twenty years in the United States, United Kingdom, France, Canada and Brazil, ranked according to their approaches, in six lines of study. As a result, points out the lack of studies that allow effective analysis of the accessibility of websites, and methods for continuous improvement of technical standards and recommendations. It follows, showing the application of usability metrics by Nielsen and ISO-9241 and the incorporation of research models of information behavior seeking for assessment of gaps in accessibility.

A Usability Comparison of SMS and IVR as Digital Banking Channels

In this paper, the authors compare the usability of SMS mobile banking and automated IVR telephone banking. Participants (N = 116) used SMS banking and IVR banking to find their account balance in a repeated-measures experiment. IVR banking scored higher for usability metrics: effectiveness, attitude, and quality. There was no clear difference in rank order of preference between the two channels. Participants gave positive comments regarding speed and efficiency with SMS banking, but had serious doubts over the security of the SMS channel, impacting consumer trust in SMS banking. The authors argue that usability problems and security concerns are a major factor in the low adoption of SMS mobile banking. Older users were less positive in general to SMS banking compared with the more established IVR banking. Older users had lower first time completion rates for SMS banking and gave IVR banking higher attitude and quality scores.

On the usability of grid middleware and security mechanisms

Usability is an all too often neglected aspect of grid computing, although it is one of the principal factors militating against the widespread uptake of distributed computing. Many resource providers on a grid infrastructure deploy a standard middleware stack and expect users to invoke the default client tools for that middleware stack to access their resources. Unfortunately, many of these middleware client tools have been developed as an afterthought, and are widely considered difficult to use. Such tools typically require a user to interact with a machine, to stage data and launch jobs, and to use digital certificates. Our experience of working with grids over many years has led us to propose a new model of grid interaction, which we call the user-application interaction model. Similar considerations have also led us to develop environments that remove digital certificates from the user's experience, replacing them with familiar username and password authentication credentials. In this paper, we investigate the usability of this interaction model and its security system through a series of tests, which compare the usability of our systems with commonly deployed middleware tools using five usability metrics. Our middleware and security solutions are judged to be more usable than the systems in use by most of today's computationalgrids.

Benchmarking of virtual reality performance in mechanics education

The paper explores the potentialities of virtual reality (VR) to improve the learning process of mechanical product design. It is focused on the definition of a proper experimental VR-based set-up whose performance matches mechanical design learning purposes, such as assemblability and tolerances prescription. The method consists of two main activities: VR technologies benchmarking based on sensory feedback and evaluation of how VR tools impact on learning curves. In order to quantify the performance of the technology, an experimental protocol is defined and an testing plan is set. Evaluation parameters are divided into performance and usability metrics to distinguish between the cognitive and technical aspects of the learning process. The experimental VR-based set up is tested on students in mechanical engineering through the application of the protocol.

A usabilidade de tarefas típicas de seleção do destino em sistemas de navegação GPS automotivos

O sistema de navegação GPS automotivo é um sistema veicular que auxilia o motorista, através de mapas e indicadores de manobra, o caminho que ele deve percorrer até o seu destino. Este artigo apresenta um estudo sobre a usabilidade de tarefas típicas de seleção do destino em três sistemas vendidos no Brasil, com métodos de entrada de dados e interfaces distintos. A pesquisa teve como objetivo avaliar se problemas de usabilidade também ocorriam em navegadores brasileiros. Para tanto, foram aplicados testes de usabilidade com usuários experientes e não experientes, com o intuito de medir a eficácia, a eficiência e a satisfação de uso (métricas de usabilidade) das três interfaces. Com os resultados dos testes, pôde-se observar diversos problemas de usabilidade no projeto das interfaces avaliadas. Também, com base neles, foi possível sugerir soluções adequadas para a organização e navegação de menus e comandos.

Developing usable web interfaces with the aid of automatic verification of their formal specification

The development of interactive visual applications is a complex work, usually performed with the help of advanced visual programming environments. Although a number of tools are available to support designers and developers in the specification of a GUI's layout and behavior, and in the generation of the corresponding code that implements the interface, theoretical guidelines and/or semi-automatic mechanisms rely upon the knowledge of the designer to manage usability and accessibility issues. Indeed, the evaluation of the visual environments is traditionally performed by means of expert-based evaluations or by testing with end users. In this work we describe a methodology to specify and evaluate interactive visual environments, in particular web interfaces, based on the SR-Action Grammars formalism and we present a bottom – up approach to aid the designer to develop graphical applications that automatically respect a significant number of usability rules before the software is released and tested by standard methods. We show how it is possible to assess the usability metrics of consistency, completeness and user control by means of checks performed at a high level of abstraction. VALUTA (Automatic Tool for the Usability Verification at Abstract Level ) is the implemented tool that allows developers to generate the formal specification of an interactive visual application in automatic manner, so to perform the related usability controls at a very early stage. Thanks to usability controls automatically performed at formal level, the designer can use the evaluation results to perform feedback analysis of the visual environment. The tool is applicable to an already existing interface, allowing the designer to evaluate its usability in the development of a more usable version. We have analyzed the home page of three web sites ( www.lycos.it, www.google.it, http://it.mail.yahoo.com/) as a case study and we show the related evaluation report generated by the described approach.

Development and testing of new upper-limb prosthetic devices: Research designs for usability testing

The purposes of this article are to describe usability testing and introduce designs and methods of usability testing research as it relates to upper-limb prosthetics. This article defines usability, describes usability research, discusses research approaches to and designs for usability testing, and highlights a variety of methodological considerations, including sampling, sample size requirements, and usability metrics. Usability testing is compared with other types of study designs used in prosthetic research.

Assessment of usability metrics for object-oriented software system

The demand for efficient software is increasing day by day. For this reason software developers need appropriate metrics for the development of software applications. Usability is one of the most important fields in software engineering and a highly focused quality factor. It is a key factor in the development of successful software applications. Object-oriented design techniques have become one of the most powerful mechanisms to develop efficient software system. Object-oriented software can play important role in usability for software applications. It can not only help in reducing the cost but also in developing highly usable software systems. This paper focuses some important issues and analyzes the relationship between usability and object-oriented metrics

The effect of previous exposure to technology on acceptance and its importance in usability and accessibility engineering

In Usability and Accessibility Engineering, metric standards are vital. However, the development of a set of reciprocal metrics—which can serve as an extension of, and supplement to, current standards—becomes indispensable when the specific needs of end-user groups, such as the elderly and people with disabilities, are concerned. While ISO 9126 remains critical to the usability of a product, the needs of the elderly population are forcing the integration of other factors. Familiarity and recognisability are not relevant to someone with no experience and therefore no referent; however, acceptance becomes a major factor in their willingness to learn something new and this acceptance requires trust based on association. Readability and legibility are of less relevance to a blind person than to someone with failing eyesight. This paper describes some usability metrics ascertained on the basis of experiments made with applications for elderly people throughout the summer term of 2007. The factors that influence the older users’ acceptance of software, including the extent of their previous exposure to technology, are evaluated in order to provide short guidelines for software developers on how to design and develop software for the elderly. The evaluation of the expectations, behavior, abilities, and limitations of prospective end-users is considered of primary importance for the development of technology. A total of N = 31 participants (22 women/9 men) took part in various tests. The participants’ ages ranged from 49 to 96 years with an average age of 79. Five of the tests were designed for a PDA or cellular phone, one test was designed for a laptop PC. Of the total of 55 tests, 52 tests provided sufficient data to evaluate the results. In 23 of the tests, all tasks were completed. As a main outcome, it can be experimentally proved that the acceptance is related to a factor, which is this paper is called PET (Previous Exposure to Technology). This is discussed in light of the aforementioned metrics.

Scaling Usability in Terms of Requirements: A Method for Evaluating User Interfaces

Software usability should be scaled in terms of the probability that a user interface will meet established usability requirements. This paper describes a scaling procedure that first estimates the distribution of common usability metrics and then calculates the probability that a user interface will meet a specific requirement. A series of Monte Carlo simulations showed that even with a small sample of usability test participants, the scaling procedure remained unbiased and could accurately differentiate usable from unusable user interfaces.

EVALUATION OF THE GLOBO.COM PORTAL EFFICIENCY: A CASE STUDY IN LIGHT OF THE FUZZY SET THEORY

The emergence of Internet has revolutionized the way people communicate. However, due to the large number of websites available, web users usually feel lost and very often frustrated for not achieving their goals. This paper aims to develop a model which is capable of evaluating the level of flexibility and efficiency of the Globo.com portal, as one of the usability metrics considered by Nielsen. Forty one attributes were assessed based on the opinions of 10 developers and 8 users of the Globo.com portal. The fuzzy set theory was the tool used to address uncertainties. As a result, a Flexibility and Efficiency Index of the Portal was obtained and some changes are recommended to improve its use.

Innovations in user sensitive design, research and development

Computing is making an increasingly significant impact on modern society, underpinned by the tremendous expansion of the capacity of information and communication systems (ICT) as predicted by Moore’s Law. This has led to a profusion of functions, new tasks and new and multiple ways to perform tasks. However, this has not always resulted in improvements in the user experience. There are currently a large number of ICT systems which should not, but do, require a great deal of technological skill, knowledge and high levels of sensory, motor and cognitive abilities. They have become more and more exclusive. Too often function has won at the expense of design. An understandable enthusiasm for what software is capable of, plus current hardware innovations, has blinded designers to crucial user interface issues. Although mainstream players are paying increasing attention to the relevance and acceptability of their offerings to their intended users, many have a very narrow description of their potential user base, ignoring the characteristics, needs and wants of a significant proportion of the population—that is those people who either because of age, disability, social, educational and/or geographical considerations find the use of ICT difficult or have poor computer literacy. It is not always fully realized that these groups offer a growing, but not yet saturated, market for ICT systems. Modern developers now meet the triple challenges of (a) the burgeoning capacity of IST, (b) increasingly demanding expectations of purchasers and users and (c) the requirements for an increasingly inclusive Information Society. Responding to these challenges requires new methods for design, research and development. Although user-centred design is becoming more accepted as best practice, it often concentrates on the typical user, overlooking the atypical user referred to above. Taking into account a larger user base may be significant in market share and even mandatory under emerging legislation. If society is to benefit fully from technological advances, it is important that a greater weight is placed on atypical users. This special issue suggests a number of ways towards addressing this issue. The keystone paper by Newell, Gregor, Morgan, Pullin and Macaulay argues for greater empathy with users by developers—an empathy that should be reflected in current methods for design, research and development. They demonstrate how user-centred design can be developed into user sensitive inclusive design to include a much wider range of users. Older adults are one such ‘‘atypical’’ group of users who are becoming both more prevalent and significant. Holzinger, Searle and Wernbacher describe some new usability metrics developed on the basis of experiments made on applications for elderly people. They evaluate factors that influence the older adults’ acceptance of software products, considering the extent of their previous exposure to technology and providing short guidelines for software developers for the design and development of software for elderly people. They conclude that acceptance is related to ‘‘Previous Exposure to Technology’’ (PET). The paper provided by Dickinson, Eisma and Gregor also explores this user group, showing how non-experimental methods can contribute valuable research findings. They R. Adams (&) CIRCUA, Middlesex University, School of Engineering and Information Sciences, The Burroughs, Hendon, London NW4 4BT, UK e-mail: adams@churchillians.net

The Usability Metric for User Experience

The Usability Metric for User Experience (UMUX) is a four-item Likert scale used for the subjective assessment of an application’s perceived usability. It is designed to provide results similar to those obtained with the 10-item System Usability Scale, and is organized around the ISO 9241-11 definition of usability. A pilot version was assembled from candidate items, which was then tested alongside the System Usability Scale during usability testing. It was shown that the two scales correlate well, are reliable, and both align on one underlying usability factor. In addition, the Usability Metric for User Experience is compact enough to serve as a usability module in a broader user experience metric.