Adaptive Human Machine Interface Research Articles

Virtual/augmented reality-based human–machine interface and interaction modes in airport control towers

The concept of an innovative human–machine interface and interaction modes based on virtual and augmented reality technologies for airport control towers has been developed with the aim of increasing the human performances and situational awareness of air traffic control operators. By presenting digital information through see-through head-mounted displays superimposed over the out-of-the-tower view, the proposed interface should stimulate controllers to operate in a head-up position and, therefore, reduce the number of switches between a head-up and a head-down position even in low visibility conditions. This paper introduces the developed interface and describes the exercises conducted to validate the technical solutions developed, focusing on the simulation platform and exploited technologies, to demonstrate how virtual and augmented reality, along with additional features such as adaptive human–machine interface, multimodal interaction and attention guidance, enable a more natural and effective interaction in the control tower. The results of the human-in-the-loop real-time validation exercises show that the prototype concept is feasible from both an operational and technical perspective, the solution proves to support the air traffic controllers in working in a head-up position more than head-down even with low-visibility operational scenarios, and to lower the time to react in critical or alerting situations with a positive impact on the human performances of the user. While showcasing promising results, this study also identifies certain limitations and opportunities for refinement, aimed at further optimising the efficacy and usability of the proposed interface.

Using Eye-Tracking for Adaptive Human-Machine Interfaces for Pilots: A Literature Review and Sample Cases

This paper explores the potential of eye-tracking technology in adaptive human-machine interfaces for pilots in aviation. We argue that an interface able to adjust its layout and elements based on pilots’ real-time eye-tracking data can prevent errors and enhance their performance. The study presents a literature review on the use of eye-tracking for various pilot cases, including flight simulator games, drone pilots, and cockpit pilots. Results in most cases showed that eye-tracking has been employed to improve interactions, enhance spatial awareness, guide pilots’ gaze to relevant areas, and provide insights into pilots’ information processing and task load. The paper discusses two sample cases demonstrating the potential of eye-tracking in adaptive human-machine interfaces. In the first case, during challenging drone simulations, eye-tracking identified areas where an adaptive human-machine interface could aid navigation and reduce cognitive load. In the second one, based on real drone flights, when signal loss incidents occurred, eye-tracking data showed that the interface should adapt to pilots’ needs by providing critical information to help them to improve situational awareness. The paper concludes that eye-tracking technology has significant potential in adaptive human-machine interfaces for aviation, emphasising the importance of refining these technologies to meet pilots’ specific needs and enhance flight safety.

Structured dataset of human-machine interactions enabling adaptive user interfaces

This article introduces a dataset of human-machine interactions collected in a controlled and structured manner. The aim of this dataset is to provide insights into user behavior and support the development of adaptive Human-Machine Interfaces (HMIs). The dataset was generated using a custom-built application that leverages formally defined User Interfaces (UIs). The resulting interactions underwent processing and analysis to create a suitable dataset for professionals and data analysts interested in user interface adaptations. The data processing stage involved cleaning the data, ensuring its consistency and completeness. A data profiling analysis was conducted for checking the consistency of elements in the interaction sequences. Furthermore, for the benefit of researchers, the code used for data collection, data profiling, and usage notes on creating adaptive user interfaces are made available. These resources offer valuable support to those interested in exploring and utilizing the dataset for their research and development efforts in the field of human-machine interfaces.

Nudges-Based Design Method for Adaptive HMI to Improve Driving Safety

This study introduces a new operational tool based on the AEIOU observational framework to support the design of adaptive human machine interfaces (HMIs) that aim to modify people’s behavior and support people’s choices, to improve safety using emotional regulation techniques, through the management of environmental characteristics (e.g., temperature and illumination), according to an approach based on the nudging concept within a design thinking process. The proposed approach focuses on research in the field of behavioral psychology that has studied the correlations between human emotions and driving behavior, pushing towards the elicitation of those emotions judged to be most suitable for safe driving. The main objective is to support the ideation of scenarios and/or design features for adaptive HMIs to implement a nudging strategy to increase driving safety. At the end, the results from a collaborative workshop, organized as a case study to collect concept ideas in the context of sports cars, will be shown and evaluated to highlight the validity of the proposed methodology, but also the limitations due to the requirement of prototypes to evaluate the actual effectiveness of the presented nudging strategies.

Using Glance Behaviour to Inform the Design of Adaptive HMI for Partially Automated Vehicles

Partially automated vehicles present a large range of information to the driver in order to keep them in-the-loop and engaged with monitoring the vehicle’s actions. However, existing research shows that this causes cognitive overload and disengagement from the monitoring task. Adaptive Human Machine Interfaces (HMIs) are an emerging technology that might address this problem, by prioritising the information presented. To date, research aiming to define the driver’s glance fixation behaviour in a partially automated vehicle to contribute towards an adaptive interface is scarce. This study used a unique three-day longitudinal driving simulator study design to explore which information drivers in a partially automated vehicle require. Twenty-seven participants experienced nine partially automated driving simulations over three consecutive days. Nine information types, developed from standards, previous studies and industry collaboration, were displayed as discrete icons and presented on a surrogate in-vehicle display. Unique to the literature, this study showed that the recorded eye-tracking data demonstrated that usage of the information types changed with longitudinal driving simulator use. This study provides three key contributions: first, the longitudinal study design suggest that single exposure HMI evaluations may be limited in their assessment. Secondly, this study has methodologically shortlisted a list of nine information types that can be used in future studies to represent future partially automated vehicle interfaces. Finally, this is one of the first studies to characterise glance behaviour for partially automated vehicles. With this knowledge, this study contributes important design recommendations for the development of adaptive interfaces.

Adaptive attention-based human machine interface system for teleoperation of industrial vehicle

This study proposes a Human Machine Interface (HMI) system with adaptive visual stimuli to facilitate teleoperation of industrial vehicles such as forklifts. The proposed system estimates the context/work state during teleoperation and presents the optimal visual stimuli on the display of HMI. Such adaptability is supported by behavioral models which are developed from behavioral data of conventional/manned forklift operation. The proposed system consists of two models, i.e., gaze attention and work state transition models which are defined by gaze fixations and operation pattern of operators, respectively. In short, the proposed system estimates and shows the optimal visual stimuli on the display of HMI based on temporal operation pattern. The usability of teleoperation system is evaluated by comparing the perceived workload elicited by different types of HMI. The results suggest the adaptive attention-based HMI system outperforms the non-adaptive HMI, where the perceived workload is consistently lower as responded by different categories of forklift operators.

Improving maintenance efficiency and safety through a human-centric approach

This paper presents an adaptive human-machine interface (HMI) that can provide appropriate sets of digital maintenance information and guidance to an operator during maintenance. It takes into consideration the expertise level of the operator and the maintenance context and progress. The proposed human-centric methodology considers the heart rate, intention, and expertise level of the operator, which can be captured using sensors during maintenance. A set of rules is formulated based on the sensor data to infer the state of the operator during a maintenance task. Based on the operator state, the adaptive HMI can augment the operator’s senses using a scheme that combines visual, audio, and haptic guidance cues during maintenance to enhance the operator’s ability to perceive information and perform maintenance tasks. Various schemes of visual, audio, and haptic cues are developed based on a comparison of the best practices obtained from experienced operators.

Influence of Adaptive Human–Machine Interface on Electric-Vehicle Range-Anxiety Mitigation

The electrification of vehicles is without a doubt one of the milestones of today’s automotive technology. Even though industry actors perceive it as a future standard, acceptance, and adoption of this kind of vehicles by the end user remain a huge challenge. One of the main issues is the range anxiety related to the electric vehicle’s remaining battery level. In the scope of the H2020 ADAS&ME project, we designed and developed an intelligent Human Machine Interface (HMI) to ease acceptance of Electric Vehicle (EV) technology. This HMI is mounted on a fake autonomous vehicle piloted by a hidden joystick (called Wizard of Oz (WoZ) driving). We examined 22 inexperienced EV drivers during a one-hour driving task tailored to generate range anxiety. According to our protocol, once the remaining battery level started to become critical after manual driving, the HMI proposed accurate coping techniques to inform the drivers how to reduce the power consumption of the vehicle. In the following steps of the protocol, the vehicle was totally out of battery, and the drivers had to experience an emergency stop. The first result of this paper was that an intelligent HMI could reduce the range anxiety of the driver by proposing adapted coping strategies (i.e., transmitting how to save energy when the vehicle approaches a traffic light). The second result was that such an HMI and automated driving to a safe spot could reduce the stress of the driver when an emergency stop is necessary.

Designing an Adaptive Interface: Using Eye Tracking to Classify How Information Usage Changes Over Time in Partially Automated Vehicles

While partially automated vehicles can provide a range of benefits, they also bring about new Human Machine Interface (HMI) challenges around ensuring the driver remains alert and is able to take control of the vehicle when required. While humans are poor monitors of automated processes, specifically during `steady state' operation, presenting the appropriate information to the driver can help. But to date, interfaces of partially automated vehicles have shown evidence of causing cognitive overload. Adaptive HMIs that automatically change the information presented (for example, based on workload, time or physiologically), have been previously proposed as a solution, but little is known about how information should adapt during steady-state driving. This study aimed to classify information usage based on driver experience to inform the design of a future adaptive HMI in partially automated vehicles. The unique feature of this study over existing literature is that each participant attended for five consecutive days; enabling a first look at how information usage changes with increasing familiarity and providing a methodological contribution to future HMI user trial study design. Seventeen participants experienced a steady-state automated driving simulation for twenty-six minutes per day in a driving simulator, replicating a regularly driven route, such as a work commute. Nine information icons, representative of future partially automated vehicle HMIs, were displayed on a tablet and eye tracking was used to record the information that the participants fixated on. The results found that information usage did change with increased exposure, with significant differences in what information participants looked at between the first and last trial days. With increasing experience, participants tended to view information as confirming technical competence rather than the future state of the vehicle. On this basis, interface design recommendations are made, particularly around the design of adaptive interfaces for future partially automated vehicles.

A practical augmented reality-assisted maintenance system framework for adaptive user support

Abstract This paper describes an Augmented Reality Maintenance System (ARMS) that can provide adaptive support and feedback to a user during maintenance. An innovative wrist-based haptic tracking methodology is implemented to provide maintenance step awareness to the maintenance operator when he/she is working within the maintenance workspace. This is integrated with an image-based step detection module to facilitate the determination on whether a step has been conducted correctly. To further adapt to the user's visualisation requirements during maintenance, a novel adaptive human-machine interface module has been incorporated. The result is a seamless AR-assisted maintenance platform that can better cater to the needs of the operators during maintenance by providing useful feedback to the operators. The system has been demonstrated through a case study to highlight the viability of the methodology in actual industrial scenarios.

Experimental characterisation of eye-tracking sensors for adaptive human-machine systems

Adaptive Human-Machine Interfaces and Interactions (HMI2) are closed-loop cyber-physical systems comprising a network of sensors measuring human, environmental and mission parameters, in conjunction with suitable software for adapting the HMI2 (command, control and display functions) in response to these real-time measurements. Cognitive HMI2 are a particular subclass of these systems, which support dynamic HMI2 adaptations based on the user’s cognitive state. These states are estimated in real-time using various neuro-physiological parameters from gaze, cardiorespiratory and brain signals, which are processed by an Adaptive Neuro-Fuzzy Inference System (ANFIS). However, the accuracy and precision of neuro-physiological measurements are affected by a variety of environmental factors and therefore need to be accurately characterised prior to operational use. This paper describes the characterisation activities performed on two types of eye tracking devices used in the Aerospace Intelligent and Autonomous Systems (AIAS) laboratory of RMIT University to support the development of cognitive human-machine systems. The uncertainty associated with the ANFIS outputs is quantified by propagating the uncertainties in the input data (determined experimentally) through the inference engine. This process is of growing relevance because similar machine learning techniques are now being developed for an increasing number of applications including aerospace, transport, biomedical and defence cyber-physical systems.

Integration of an Adaptive Infotainment System in a Vehicle and Validation in Real Driving Scenarios

More services, functionalities, and interfaces are increasingly being incorporated into current vehicles and may overload the driver capacity to perform primary driving tasks adequately. For this reason, a strategy for easing driver interaction with the infotainment system must be defined, and a good balance between road safety and driver experience must also be achieved. An adaptive Human Machine Interface (HMI) that manages the presentation of information and restricts drivers’ interaction in accordance with the driving complexity was designed and evaluated. For this purpose, the driving complexity value employed as a reference was computed by a predictive model, and the adaptive interface was designed following a set of proposed HMI principles. The system was validated performing acceptance and usability tests in real driving scenarios. Results showed the system performs well in real driving scenarios. Also, positive feedbacks were received from participants endorsing the benefits of integrating this kind of system as regards driving experience and road safety.

Adaptive human machine interface based on the detection of driver's cognitive state using machine learning approach

Cognitive distraction during the driving task might cause impairment of detection performance and of the recognition and/or response selection, increasing the risk of road crashes. In order to avoid or mitigate the negative effects related to cognitive distraction, this paper describes the development and testing of a Cooperative Lane Change Assistant (C-LCA) system: it takes into account the real-time driver's cognitive state by means of a cognitive distraction classifier expressly designed and it implements road cooperation between the vehicles thanks to a cooperative driver model. Three different test sessions were conducted on a static driving simulator and, in each test session, the participants carried out several analogous runs of a reference protocol test, derived from the Lane Change Task. Using the data collected during the first test session, the cognitive distraction classifier was developed using Machine Learning techniques. In the remaining two sessions, a specific C-LCA HMI prototype with visual and acoustic interfaces has been evaluated. The results show that the C-LCA reduced the workload during the lane change manoeuvres compared both with the baseline and with the assistance of a non-cooperative warning system. As well, the users expressed satisfaction about the Visual Interface and Acoustic Interface designed for the C-LCA.

Remote Control Station Design and Testing for Tele-Operated Space-Missions

Space environ ment is inherently hostile and dangerous for astronauts. Since extra -vehicu lar activity (EVA ) should be limited or assisted as much as possible, the significant role of robot ic systems is of notable relevance. Tele-navigation of ground rovers in hostile and unknown environments is one of the applicable p roblems for modern planetary exp loration missions. Pilots involved in these missions should be supported with intelligent and adaptive Hu man-Machine Interfaces (HM Is) able to guarantee a high level of situation awareness and an affordable mental workload. This paper aims to illustrate the approach used to design, test and assess a Remote Control Station (RCS) fo r ground missions . The RCS is used to remotely operate a 4-wheel drive scaled vehicle, representing a min i-rover for space exp loration. The complete system includes the control station and the vehicle, all based on off-the-shelf components. The rover is used as a tool to assess operators situation awareness provided through the Graphic User Interface (GUI). This research activity is part of the project STEPS - a research project co-financed by Pied mont Region.

Co-Adaptive and Affective Human-Machine Interface for Improving Training Performances of Virtual Myoelectric Forearm Prosthesis

The real-time adaptation between human and assistive devices can improve the quality of life for amputees, which, however, may be difficult to achieve since physical and mental states vary over time. This paper presents a co-adaptive human-machine interface (HMI) that is developed to control virtual forearm prosthesis over a long period of operation. Direct physical performance measures for the requested tasks are calculated. Bioelectric signals are recorded using one pair of electrodes placed on the frontal face region of a user to extract the mental (affective) measures (the entropy of the alpha band of the forehead electroencephalography signals) while performing the tasks. By developing an effective algorithm, the proposed HMI can adapt itself to the mental states of a user, thus improving its usability. The quantitative results from 16 users (including an amputee) show that the proposed HMI achieved better physical performance measures in comparison with the traditional (nonadaptive) interface ({\rm p\hbox{-}value}<0.001). Furthermore, there is a high correlation (correlation coefficient < 0.9, {\rm p\hbox{-}value} < .01) between the physical performance measures and self-report feedbacks based on the NASA TLX questionnaire. As a result, the proposed adaptive HMI outperformed a traditional HMI.

満足感に適応するヒューマン・マシンインタフェースの開発 作業方式の違いによる満足感の比較

This paper proposes an adaptive human-machine interface to match human satisfaction which is evaluated by the satisfaction measurement system with neural network based on measuring electroencephalogram (EEG). This adaptive human-machine interface can adapt to individual operators and varying context based on human satisfaction. A prototype system was developed using a pacing mental task to test the proposed human-machine interface. The result of experiments showed that the adaptive human-machine interface which understands individual human satisfaction in real time and takes adaptive action based on operator's satisfaction change has the possibility in heightening or maintaining human satisfaction rather than other work styles such as machine paced mode and self-paced mode.

Development of Adaptive Human-Machine Interface to Match Human Satisfaction

The 21st century is likely to see many robots and computer systems coming into the sphere of daily life. Such robots and systems must possess the ability to cooperate and coexist along with humans. Adaptive human-machine interfaces that can adapt to the current operator and the current context are requested. In this paper an adaptive human-machine interface based on satisfaction measures estimated by electroencephalogram (EEG) measurement has been proposed. The results of experiments showed the possibility of the adaptive human-machine interface based on satisfaction measures.

An Attempt to Measure Human Satisfaction by EEG Measurement.

We are part of sociotechnical systems consisting of individual persons, organizations and technology. Therefore a methodology of synthesis based on balancing technological, organizational and human factors should be constructed as a discipline. Hence we have proposed a design method that takes human satisfaction into consideration. This paper describes the development of a satisfaction measurement system using a neural network which estimates human satisfaction based on electroencephalogram (EEG) measurement. The results of measurements demonstrated that the system was successfully evaluate satisfaction level. It is expected that this system can be applied in many fields such as analysis, design, and evaluation of products or systems, task and job design, human-oriented bio-feedback and adaptive human machine interface.

Design method considering human satisfaction: Development of adaptive human-machine interface based on satisfaction measures

We are part of a sociotechnical system consisting of individual persons, organizations, and technology. Therefore, a methodology of synthesizing and balancing technological, organizational, and human factors becomes necessary as a discipline. The authors have proposed a design method that takes human satisfaction into consideration, and to this end, have developed a satisfaction measurement system using a neural network that estimates human satisfaction from electroencephalogram measurements. This article discusses the design method and the satisfaction measurement system, and describes the development of a real-time adaptive human–machine interface based on satisfaction measures as an example of the design method proposed. © 1999 John Wiley & Sons, Inc.