Human Machine Interface Evaluation Research Articles

A Framework for the Evaluation of Human Machine Interfaces of Robot-Assisted Colonoscopy.

This article presents a framework to compare and evaluate different HMIs for robotic colonoscopy, with the objective of identifying the optimal HMI that minimises the clinician's effort and maximises the clinical outcomes. The framework comprises a 1) a virtual simulator (clinically validated), 2) wearable sensors measuring the cognitive load, 3) a data collection unit of metrics correlated to the clinical performance, and 4) questionnaires exploring the users' impressions and perceived stress. The framework was tested with 42 clinicians investigating the optimal device for tele-operated control of robotic colonoscopes. Two control devices were selected and compared: a haptic serial-kinematic device and a standard videogame joypad. The haptic device was preferred by the endoscopists, but the joypad enabled better clinical performance and reduced cognitive and physical load. The framework can be used to evaluate different aspects of a HMI, both hardware and software, and determine the optimal HMI that can reduce the burden on clinicians while improving the clinical outcome. The findings of this study, and of future studies performed with this framework, can inform the design and development of HMIs for intraluminal robots, leading to improved clinical performance, reduced physical and mental stress for clinicians, and ultimately better patient outcomes.

An evaluation method for HMI of deep-sea manned submersible based on human reliability

Improving the human reliability of the human–machine interface (HMI) of deep-sea manned submersible is of great importance for the development of the deep-sea field. Based on the SHEL (Software S, Hardware H, Environment E, Liveware L) model, this study classifies the performance shaping factors (PSF) that affect the human reliability of submersible HMIs and builds a PSF system. The interpretative structural model (ISM) is used to matrix the interactions between the elements that make up the system of PSF. A multi-level recursive structure is obtained by building the corresponding adjacency matrix. The Noisy-OR model is introduced to construct a Bayesian network in order to build a new HMI evaluation method. A real case of Bayesian network causal inference verifies the validity of the built method. This study proposes a set of HMI human reliability evaluation methods applicable to deep-sea manned submersible, which provides a new idea for human reliability assessment.

Performance Index Based on Predicted Auditory Reaction Time Analysis for the Evaluation of Human-Machine Interface in Flight Control.

With the rapid development of complex equipment, such as airplanes, the appropriate design of the human-machine interface is often upgraded, thus emerged many methods to evaluate whether such an upgrade is effective. Most researches focus on the time accumulation effect of the human state during the interaction to evaluate the interface. However, in the aviation application, the performance of the pilot's instantaneous reactions also reveals the design efficiency of the interface, since the difficulty level of obtaining the useful information would severely influence the reaction time in some voice command tasks or emergency situations. Besides, there are so many flight scenarios that are impossible to be simulated in experiments or in a laboratory environment. Also, voice commands are too numerous to be traversed simulated. This paper introduced predicted auditory reaction time as an index to evaluate human-machine interface design. The proposed method has two advantages. On the one hand, it effectively measures the pilot's auditory reaction time based on the eye movement tracking; thus, the data can be taken in flight task scenarios, and the experiment would not cause interference to the subjects. On the other hand, a prediction model is proposed, in which the pilot's reaction time under more generalized voice command can be estimated based on a small-size sample set.

Human-Centered Design for an In-Vehicle Truck Driver Fatigue and Distraction Warning System

Driver fatigue and distraction are major road safety issues globally; developing driver state detection and warning technology to help reduce impairment-related incidents is a promising approach. The aim of this case study was to design an effective Human Machine Interface (HMI) for a near-market driver warning system primarily aimed at commercial truck driving. A human-centered design (HCD) process was employed for the development and evaluation. Application of HCD here was a multi-stage iterative process: a comprehensive literature review; developing a context of use description; undertaking truck driver interviews; identifying user needs and associated design requirements; conducting two design workshops; operationalizing the design; running HMI evaluation studies, and finalizing the HMI concepts. As a result of the iterative HCD process, the HMI comprises a multi-modal warning system (visual, auditory and tactile) with two levels for driver fatigue and an escalating system for driver distraction. Following this successful HCD process, further on-road evaluation work is proposed before the final version of the HMI is manufactured.

Study on Ergonomic Digital Evaluation System for the Naval Shipborne Command Cabin Based on Extended JACK

The naval ship command cabin(NSCC) is an important part and the nerve centre of the whole ship. In order to achieve better evaluation and design improvements of the human-machine interface(HMI) of the naval shipborne command cabin in the design phase, we preliminarily expounded the ergonomic digital evaluation system of the NSCC based on extended JACK. The NSCC human-machine interface evaluation index system was constructed by using the improved Delphi method, and the quantitative and qualitative evaluation method was studied. The ergonomic digital evaluation system for the naval shipborne command cabin was developed by extending the JACK software, in which the mannequin module, human-machine interface design module, evaluation module and design standards module were integrated. Finally, the feasibility and applicability of the proposed ergonomic digital evaluation system were verified by an illustrated example.

Checklist for Expert Evaluation of HMIs of Automated Vehicles—Discussions on Its Value and Adaptions of the Method within an Expert Workshop

Within a workshop on evaluation methods for automated vehicles (AVs) at the Driving Assessment 2019 symposium in Santa Fe; New Mexico, a heuristic evaluation methodology that aims at supporting the development of human–machine interfaces (HMIs) for AVs was presented. The goal of the workshop was to bring together members of the human factors community to discuss the method and to further promote the development of HMI guidelines and assessment methods for the design of HMIs of automated driving systems (ADSs). The workshop included hands-on experience of rented series production partially automated vehicles, the application of the heuristic assessment method using a checklist, and intensive discussions about possible revisions of the checklist and the method itself. The aim of the paper is to summarize the results of the workshop, which will be used to further improve the checklist method and make the process available to the scientific community. The participants all had previous experience in HMI design of driver assistance systems, as well as development and evaluation methods. They brought valuable ideas into the discussion with regard to the overall value of the tool against the background of the intended application, concrete improvements of the checklist (e.g., categorization of items; checklist items that are currently perceived as missing or redundant in the checklist), when in the design process the tool should be applied, and improvements for the usability of the checklist.

Behavioural validity of driving simulators for prototype HMI evaluation

In-vehicle interfaces are now part of the vast majority of production vehicles. Such interfaces need to be thoroughly evaluated to ensure they do not pose any risks to the drivers using them. Driving simulators have extensively been used in such a context, yet their reliability in terms of how realistic a driving behaviour they elicit is still in question. An investigation on driving simulator behavioural validity in the context of prototype human-machine interface evaluation is presented in this study. Using data collected in a dual setting driving study (driving simulator and real world), as well as results from existing related literature, a comparison between driving behaviour in different types of driving simulators and in reality was carried out, for a variety of behavioural metrics. The results are presented in the form of a `validity matrix' that aggregates the level of behavioural validity different simulator settings can achieve for different behavioural metrics.

Takeover Requests in Highly Automated Truck Driving: How Do the Amount and Type of Additional Information Influence the Driver–Automation Interaction?

Vehicle automation is linked to various benefits, such as increase in fuel and transport efficiency as well as increase in driving comfort. However, automation also comes with a variety of possible downsides, e.g., loss of situational awareness, loss of skills, and inappropriate trust levels regarding system functionality. Drawbacks differ at different automation levels. As highly automated driving (HAD, level 3) requires the driver to take over the driving task in critical situations within a limited period of time, the need for an appropriate human–machine interface (HMI) arises. To foster adequate and efficient human–machine interaction, this contribution presents a user-centered, iterative approach for HMI evaluation of highly automated truck driving. For HMI evaluation, a driving simulator study [n = 32] using a dynamic truck driving simulator was conducted to let users experience the HMI in a semi-real driving context. Participants rated three HMI concepts, differing in their informational content for HAD regarding acceptance, workload, user experience, and controllability. Results showed that all three HMI concepts achieved good to very good results in these measures. Overall, HMI concepts offering more information to the driver about the HAD system showed significantly higher ratings, depicting the positive effect of additional information on the driver–automation interaction.

Potential Use of HMI Evaluation Methods in HRA

A human reliability analysis (HRA) is defined as “any method by which human reliability is estimated” [1][1,p. 301] and generally consist of three parts: 1) identifying possible human errors and contributors, 2) model human error, and 3) quantify human error probabilities. Many HRA methods quantify human error probabilities through the use of performance shaping factors (PSFs) that increase or decrease these probabilities. One of the factors that are often evaluated is the quality of the human machine interface (HMI) and how it affects performance. However, as evaluating HMI can be a complicated task the descriptions found in HRA methods are often not sufficient to perform the evaluation. This problem has increased lately as most current HRA methods are based on classical non-computerized control rooms creating a mismatch between the descriptions and the real world [2]. In this paper different HMI evaluation methods are discussed in terms of their usability in situations where the descriptions provided in HRA methods are not sufficient.

Analysis of Usability of ECDIS Human-Machine Interface

Usability is a key indicator in the evaluation of Human-Machine Interface (HMI). ECDIS has a typical HMI for e-Navigation system. In order to improve the usability of ECDIS HMI, it is necessary to carry out usability evaluation. This paper investigated the time spent in executing typical navigation operations using two common designs, which are based on single document and dialog box. The accuracy of the theoretical prediction was verified against actual test results, and the efficiency of the two design schemes was compared. The research results can provide scientific guidance to improve the usability of ECDIS and design of HMI.

Analysis of Usability of ECDIS Human-Machine Interface

Usability is a key indicator in the evaluation of Human-Machine Interface (HMI). ECDIS has a typical HMI for e-Navigation system. In order to improve the usability of ECDIS HMI, it is necessary to carry out usability evaluation. This paper investigated the time spent in executing typical navigation operations using two common designs, which are based on single document and dialog box. The accuracy of the theoretical prediction was verified against actual test results, and the efficiency of the two design schemes was compared. The research results can provide scientific guidance to improve the usability of ECDIS and design of HMI.

Virtual Comfort Evaluation as a New Paradigm of Health and Safety in Industrial Engineering

New VP (Virtual Prototyping) techniques and their development and diffusion in industrial environments make possible, since 1990, designing and redesigning of work-place in virtual environment; DHM (Digital Human Modelling) technique allow to virtually evaluate the interaction between humans and the main commands/hand tools of machine/workplace. Those devices, together with ergonomics and comfort evaluation models, may allow the preventive evaluation of Human Machine Interfaces (HMI) both in terms of comfort/ ergonomics while processing a product than in terms of comfort/ ergonomics while using the product.

A human–machine interface evaluation method: A difficulty evaluation method in information searching (DEMIS)

A human–machine interface (HMI) evaluation method, which is named “difficulty evaluation method in information searching (DEMIS)”, is proposed and demonstrated with an experimental study. The DEMIS is based on a human performance model and two measures of attentional-resource effectiveness in monitoring and detection tasks in nuclear power plants (NPPs). Operator competence and HMI design are modeled to be most significant factors to human performance. One of the two effectiveness measures is fixation-to-importance ratio (FIR) which represents attentional resource (eye fixations) spent on an information source compared to importance of the information source. The other measure is selective attention effectiveness (SAE) which incorporates FIRs for all information sources. The underlying principle of the measures is that the information source should be selectively attended to according to its informational importance. In this study, poor performance in information searching tasks is modeled to be coupled with difficulties caused by poor mental models of operators or/and poor HMI design. Human performance in information searching tasks is evaluated by analyzing the FIR and the SAE. Operator mental models are evaluated by a questionnaire-based method. Then difficulties caused by a poor HMI design are evaluated by a focused interview based on the FIR evaluation and then root causes leading to poor performance are identified in a systematic way.

Analytical Model Based Evaluation of Human Machine Interfaces Using Cognitive Modeling

Cognitive models allow predicting some aspects of util- ity and usability of human machine interfaces (HMI), and simulating the interaction with these interfaces. The action of predicting is based on a task analysis, which investigates what a user is required to do in terms of actions and cognitive processes to achieve a task. Task analysis facilitates the understanding of the system's functionalities. Cognitive models are part of the analytical approaches, that do not associate the users during the development process of the interface. This article presents a study about the evaluation of a human machine interaction with a contextual assistant's interface using ACT- R and GOMS cognitive models. The present work shows how these techniques may be applied in the evaluation of HMI, design and research by emphasizing firstly the task analysis and secondly the time execution of the task. In order to validate and support our results, an experimental study of user performance is conducted at the DOMUS laboratory, during the interaction with the contextual assistant's interface. The results of our models show that the GOMS and ACT-R models give good and excellent predictions respectively of users performance at the task level, as well as the object level. Therefore, the simulated results are very close to the results obtained in the experimental study.