Human-machine Interaction Design Research Articles

Exploring the Human-Centric Interaction Paradigm: Augmented Reality-Assisted Head-Up Display Design for Collaborative Human-Machine Interface in Cockpit

Human-cybernetic interfaces (HCI) focuses on the integration and synergy of human-centric design, human factors (HFs) and economics, and human–machine interaction (HMI). The scrutiny of single pilot operations (SPO) approval from civil aviation authorities (CAAs) is subject to the aviation safety and pilot capability during exceptional handling and flight emergency. Airliners advocate the possibility from dual pilot operations (DPO) to SPO because of the financial considerations and captain shortage in the post-pandemic era. One could expect that SPO can only realise when reduced crew operations initiative is proven to be safe and airworthy. Public concerns pilot mental workload and potentially low situational awareness (SA) when handling overwhelmed multi-source information from the HMI in a cockpit, leading to degraded skills and flying performance. Advanced head up display (HUD) design, an integrated transparent display, can assist pilots in maintaining their optimal performance when SPO exercised. In this study, we are interested to design a SPO-favoured cockpit design with HUD and investigate the potential of integrating augmented reality (AR) and HUD in SPO in flight operations. An experiment involving 21 pilots was conducted to assess the effectiveness of AR-HUD on SA and workload in DPO, SPO, and SPO with AR-HUD scenarios. The Situation Awareness Present Assessment Method (SPAM) and a self-rated workload scale were used to evaluate SA and workload levels. Eye-tracking data was also collected to analyse behavioural performance. Results indicated that the AR-HUD scenario achieved optimal workload and SA levels and demonstrated superior performance in handling emergencies. Eye-tracking data revealed that AR-HUD facilitated a more equitable distribution of gaze between instruments and external view. In conclusion, the integration of AR in cockpit design introduces a human-centric interaction paradigm, contributing to aviation safety, SPO implementation and adaptive HMI design.

Supporting cognitive intelligence for smart manufacturing systems using HMI design: challenges and fundamental issues

As the rate of digitalisation grows in the manufacturing sector, there is a need to address the manner in which design for the human experience is conceptualised. In industrial systems, user experience/user interface (UX/UI) as a paradigm does not fit in squarely, as industrial technologies require a blend of humans and technology for productivity. Thus, the aim of this article is to highlight the challenges and fundamental issues related to human-machine interaction (HMI) design in smart manufacturing from the disciplines of Human Factors and Interaction Design in terms of major themes: (1) designing for operator experience, (2) meaning-based analytics and design for complexity, and (3) human-centered design process for enabling reliability, maintainability, availability, and safety. These various challenges are depicted using a case study of HMI concept design for monitoring an additive manufacturing setup. The article concludes by discussing future steps for addressing HMI design in the manufacturing sector.

Design of parallel coil arrays with identifiable eigenfrequency elements for wearable human-machine interactions

Wearable electronic devices are considered as a promising platform for efficient human-machine interactions (HMI) in the advancing era of the Internet of Things (IoT). However, the limited capacity often necessitates the arraying of individual devices, which inevitably increase the number of connecting wires, communication channels, and physical chaos. Herein, we propose a high-capacity HMI interface which is composed of parallel-arranged flexible coils, a magnetized composite and a pair of electrodes for electrical connection. When the coils were deformed, the inherent oscillation occurs spontaneously once the mechanical constraint was released. Upon this, the magnetic flux variation caused by the oscillation results in induced current within the coil as defined by the law of electromagnetic induction. Assisted by theoretical model of cantilever beams, eigenfrequency could be effectively regulated and the integration of distinct coils permits a single device to transmit multiple commands and information. Taking advantage of the design, spatial position sensing, virtual piano playing and robotic control have been demonstrated as practical applications, which are mainly based on one communication channel and a pair of electrodes. We expect that the eigenfrequency-dependent mechanism can provide an inspiring solution for the design of high-capacity, effective, and comfortable HMI interfaces in the future.

Influence of automation level of human-machine system on operators’ mental load

The appropriate automation in armored vehicles is vital for the operational efficiency and personnel security of operators. In this study, fifty subjects conducted over-the-horizon strike and N-back tests at different automation levels based on a virtual simulation system for armored vehicles. Physiological signals and subjective assessments were recorded. The mental load and task performance of operators were related to different automation levels. Results suggested that the mental load decreased with the increase of automation levels. Apart from object destruction time, heart rate and standard deviation of NN intervals (SDNN), other indexes were all significantly affected by the automation level of subtasks (p ​< ​0.01). The NASA-TLX scores, object destruction time, response time of abnormal states, and reaction time in N-back tests decreased by at least 2.9 ​%, 8.2 ​%, 11.2 ​% and 1.3 ​% respectively, while the mean accuracy in N-back tests increased by 0.1 ​%. Furthermore, there existed several automation levels of tasks where the task performance remained almost unchanged under normal operation. The function of task automation on decreasing mental load reduced in the following order: A3-B3-C2-D2-E2, A2-B2-C2-D2-E2, and A3-B3-C1-D1-E1. The main contribution of this research was to provide a qualitative method and framework for the evaluation of influences of automation level on operators’ mental load, and the design of human-machine interaction and adaptive automation in automated systems.

Recent Advancements of Embedded System in HMI

Human-Machine Interaction (HMI) refers to the interface and interaction between humans and machines, often in the context of technology-driven systems. In the realm of vehicular systems, HMI plays a crucial role in facilitating communication between drivers, passengers, and the vehicle itself. The driving experience is completely transformed by the quick developments in embedded systems, with the Human-Machine Interface (HMI) having a major influence on how drivers and cars interact. This study examines the most current developments in HMI design and execution, emphasizing how they have improved the usability, safety, and intuitiveness of vehicle operation interfaces.

Developing a Multimodal HMI Design Framework for Automotive Wellness in Autonomous Vehicles

With the development of autonomous technology, the research into multimodal human-machine interaction (HMI) for autonomous vehicles (AVs) has attracted extensive attention, especially in automotive wellness. To support the design of HMIs for automotive wellness in AVs, this paper proposes a multimodal design framework. First, three elements of the framework were envisioned based on the typical composition of an interactive system. Second, a five-step process for utilizing the proposed framework was suggested. Third, the framework was applied in a design education course for exemplification. Finally, the AttrakDiff questionnaire was used to evaluate these interactive prototypes with 20 participants who had an affinity for HMI design. The questionnaire responses showed that the overall impression was positive and this framework can help design students to effectively identify research gaps and expand design concepts in a systematic way. The proposed framework offers a design approach for the development of multimodal HMIs for autonomous wellness in AVs.

Sweeping-responsive interface using the intrinsic polarity of magnetized micropillars for self-powered and high-capacity human-machine interaction

The rapid development of wearable tactile sensors in human-machine interaction (HMI) is revolutionizing the way that people communicate with intelligent terminals. Compared with “tapping”, another daily operation, “directional sweeping”, is rarely explored in tactile sensors for HMI due to the limits from device configuration. Herein, we designed and optimized the in-plane magnetized flexible micropillars as the self-powered interface that can perceive the sweeping with distinguishable signals according to the operation directions. Based on Faraday’s law of induction, the intrinsic magnetic polarity was applied as an avenue to rapidly produce diverse electromotive forces of “+ /-” and “-/+ ” through the bi-directional micropillar deformation. With the self-powered interface, several interesting HMI applications, e.g., smartphone page flip, Morse code communication, and game playing, were demonstrated via habitual finger sweeping with high efficiency, accuracy, intuitive experience, and control diversity. Furthermore, the unique behavior of “+ /-” and “-/+ ” enables the build-up of ternary system with broader command capacity of 3 n if n devices were integrated in parallel. Along with the merits such as high sensitivity, applicable diversity (humid condition), mechanical robustness, and coding accuracy, we believe that the study would bring inspiration of future HMI design for a more fascinating, effective and intelligent living.

A user study of directional tactile and auditory user interfaces for take-over requests in conditionally automated vehicles

General introduction of unconditionally and conditionally automated vehicles is expected to have a highly positive impact on the society, from increased accessibility to mobility and road traffic safety, to decreased environmental and economic negative impacts. However, there are several obstacles and risks slowing down the adoption of this technology, which are primarily related to the human–machine interaction (HMI) and exchange of control between the vehicle and the human driver. In this article, we present key takeaways for HMI design of take-over requests (TOR) that the vehicle issues to inform the driver to take over control of the vehicle. The key takeaways were developed based on the results of a user study, where directional tactile-ambient (visual) and auditory-ambient (visual) TOR user interfaces (UI) were evaluated with regards to commonly used take-over quality aspects (attention redirection, take-over time, correct interpretation of stimuli, off-road drive, brake application, lateral acceleration, minimal time-to-collision and occurrence of collision). 36 participants took part in the mixed design study, which was conducted in a driving simulator. The results showed that drivers’ attention was statistically significantly faster redirected with the auditory-ambient UI, however using the tactile-ambient UI resulted in less off-road driving and slightly less collisions. The results also revealed that drivers correctly interpreted the directional TOR stimuli more often than the non-directional one. Based on the study results, a list of key takeaways was developed and is presented in the conclusion of the paper. The results from this study are especially relevant to the TOR UI designers and the automotive industry, which tend to provide the most usable UI for ensuring safer end efficient human-vehicle interaction during the TOR task.

Recognition of visual-related non-driving activities using a dual-camera monitoring system

For a Level 3 automated vehicle, according to the SAE International Automation Levels definition (J3016), the identification of non-driving activities (NDAs) that the driver is engaging with is of great importance in the design of an intelligent take-over interface. Much of the existing literature focuses on the driver take-over strategy with associated Human-Machine Interaction design. This paper proposes a dual-camera based framework to identify and track NDAs that require visual attention. This is achieved by mapping the driver's gaze using a nonlinear system identification approach, on the object scene, recognised by a deep learning algorithm. A novel gaze-based region of interest (ROI) selection module is introduced and contributes about a 30% improvement in average success rate and about a 60% reduction in average processing time compared to the results without this module. This framework has been successfully demonstrated to identify five types of NDA required visual attention with an average success rate of 86.18%. The outcome of this research could be applicable to the identification of other NDAs and the tracking of NDAs within a certain time window could potentially be used to evaluate the driver's attention level for both automated and human-driving vehicles.

Sleep inertia in automated driving: Post-sleep take-over and driving performance

Sleep is emerging as a new driver state in automated driving. Post-sleep performance impairments due to sleep inertia, the transitional phase from sleep to wakefulness that can take up to 30 min, are a potential safety issue. Take-over performance immediately after sleep is impaired and drivers perceive the take-over as critical. The aim of the presented study was to assess take-over behavior immediately after sleep and driving behavior during the 10 min after sleep. A study with N = 31 drivers was conducted in a high-fidelity driving simulator. Take-over performance and driving performance were assessed a) under alert baseline conditions and b) after awakening from electroencephalography-confirmed stable sleep. Take-over performance 15 s after awakening was impaired resulting in more driving errors compared to the alert baseline. Lane keeping was dramatically impaired in the first 3 min after sleep and recovered rapidly. Drivers drove slower after sleep and speed keeping was less stable for at least 10 min. The results suggest that human-machine interaction design should account for the drivers’ impaired post-sleep driving performance.

Einsatz digitaler Assistenzsysteme in der Produktion

One major challenge of digitalisation is the human-centred design of human-machine interactions. As an example the introduction of digital assistance systems requires the design of changed processes, tasks and qualification requirements. In this paper, classification schemes, requirements and success criteria with regard to the introduction of assistance systems will be pointed out on the basis of a case study from the automotive sector.

Building Trust in Autonomous Vehicles: Role of Virtual Reality Driving Simulators in HMI Design

The investigation of factors contributing at making humans trust Autonomous Vehicles (AVs) will play a fundamental role in the adoption of such technology. The user's ability to form a mental model of the AV, which is crucial to establish trust, depends on effective user-vehicle communication; thus, the importance of Human-Machine Interaction (HMI) is poised to increase. In this work, we propose a methodology to validate the user experience in AVs based on continuous, objective information gathered from physiological signals, while the user is immersed in a Virtual Reality-based driving simulation. We applied this methodology to the design of a head-up display interface delivering visual cues about the vehicle' sensory and planning systems. Through this approach, we obtained qualitative and quantitative evidence that a complete picture of the vehicle's surrounding, despite the higher cognitive load, is conducive to a less stressful experience. Moreover, after having been exposed to a more informative interface, users involved in the study were also more willing to test a real AV. The proposed methodology could be extended by adjusting the simulation environment, the HMI and/or the vehicle's Artificial Intelligence modules to dig into other aspects of the user experience.

Investigating the effects of age and disengagement in driving on driver’s takeover control performance in highly automated vehicles

ABSTRACTDriving is closely linked to older people’s mobility and independence. However, age-related functional decline reduces their safe driving abilities and thereby their wellbeing may decline. The rapid development of vehicle automation has the potential to enhance the mobility of older drivers by enabling them to continue driving safer for longer. So far only limited work has been carried out to study older drivers’ interaction with highly automated vehicles (HAV). This study investigates the effect of age and level of driving disengagement on the takeover control performance in HAV. A driving simulation study with 76 drivers has been conducted. Results showed that 20 s was sufficient for drivers to take over control from HAV. Older drivers take longer to respond and make decisions than younger drivers. The age effect on some aspects of takeover quality, in terms of operating steering wheel and pedals, is still pronounced. In addition, complete disengagement in driving in HAV leads to a longer takeover time and worse takeover quality, and it affects older drivers more seriously than younger drivers. The results highlight that an age-friendly design of human-machine interaction is important for enhancing the safety and comfort of older drivers when interacting with HAVs.

New trends in the design of human-machine interaction for CNC machines

The development of numerical control technology, the increasing efficiency of production processes and the high automation level have allowed a great improvement of Computer Numerical Control (CNC) machines. Although CNC machines allow short working times and the precise repeatability of the processing, they suffer from some major disadvantages: they need to be controlled and managed by an expert operator who has to interact with the user interface to complete her/his tasks and set-up the machine. Unfortunately, to the best of our knowledge, there are no thorough studies on how human operators manage and perceive the interaction with these machines. As a result, the interfaces of most current CNC machines present several limitations. Considering the advantages in adding ergonomics and usability analysis to increase quality, productivity and market competitiveness, the industrial world is moving to develop new interfaces for CNC machines, to improve users’ experience. Starting from our experience with woodworking CNC machines, this paper reports, first, an analysis of the CNC process from the point of view of human-machine interaction; specifically, main issues and criticalities are outlined. Then, major new trends in modern interfaces for CNC machines are presented, highlighting the tendency to integrate user experience in the design of interfaces.

Investigation of older driver's takeover performance in highly automated vehicles in adverse weather conditions

Driving is important for older people to maintain mobility. To reduce age-related functional decline, older drivers may adjust their driving by avoiding difficult situations. One of these situations is driving in adverse weather conditions such as in the rain, snow and fog which reduce the visual clarity of the road ahead. The upcoming highly automated vehicle (HAV) has the potential of supporting older people. However, only limited work has been done to study older drivers' interaction with HAV, especially in adverse weather conditions. This study investigates the effect of age and weather on takeover control performance among drivers from HAV. A driving simulation study with 76 drivers has been implemented. The participants took over the vehicle control from HAV under four weather conditions clear weather, rain, snow and fog, where the time and quality of the takeover control are quantified and measured. Results show age did affect the takeover time (TOT) and quality. Moreover, adverse weather conditions, especially snow and fog, lead to a longer TOT and worst takeover quality. The results highlighted that a user-centred design of human-machine interaction would have the potential to facilitate a safe interaction with HAV under the adverse weather for older drivers.

Modular Simulation Design of Human-machine Interaction for Civil Aircraft Cockpit

A design method for human-machine interaction (HMI) simulation of the civil aircraft cockpit has been introduced, including model-based design of user interface, logic dependency of control panel and display, and warning conditions under various scenarios. The civil aircraft cockpit HMI simulation can be used for cockpit layout, static and dynamic evaluation of flight tasks, and dynamic assessment platform. Additionally, the modular design has the advantages of flexible modification, cost saving, and quick access to the user’s requirement. Finally, a prototype of high lift system (HLS) of the civil aircraft has been used to illustrate the cockpit HMI simulation with the slat-flap retraction and extraction process.

Modular Simulation Design of Human-machine Interaction for Civil Aircraft Cockpit

Modular Simulation Design of Human-machine Interaction for Civil Aircraft Cockpit

Creating Appropriate Trust in Automated Vehicle Systems: A Framework for HMI Design

While automated vehicle technology progresses, potentially leading to a safer and more efficient traffic environment, many challenges remain within the area of human factors, such as user trust for automated driving (AD) vehicle systems. The aim of this paper is to investigate how an appropriate level of user trust for AD vehicle systems can be created via human-machine interaction (HMI). A guiding framework for implementing trust-related factors into the HMI interface is presented. This trust-based framework incorporates usage phases, AD events, trust-affecting factors, and levels explaining each event from a trust perspective. Based on the research findings, the authors recommend that HMI designers and automated vehicle manufacturers take a more holistic perspective on trust rather than focusing on single, “isolated” events, for example understanding that trust formation is a dynamic process that starts long before a user's first contact with the system, and continues long thereafter. Furthermore, factors-affecting trust change, both during user interactions with the system and over time; thus, HMI concepts need to be able to adapt. Future work should be dedicated to understanding how trust-related factors interact, as well as validating and testing the trust-based framework.

A human–machine interaction design and evaluation method by combination of scenario simulation and knowledge base

ABSTRACT A method of designing and evaluating HMI (human–machine interaction) is proposed for the design in supervisory control of fully digitalized I&C (instrumentation and control) and digitalized human–machine interface system, which is a large-scale complex system in the NPPs (nuclear power plants). The proposed method consists of plant accident scenario simulation, knowledge base establishment, and interaction simulation. The plant accident scenario simulation is to analyze the plant behavior and system sequences under the predefined conditions; the knowledge base is modeled based on the simulation results as human and machine roles; and the interaction simulation is to simulate the interactions such as between operator and plant, operator and technical advisor. The proposed method utilizes the object-oriented software named plant DiD (defense-in-depth) risk monitor with the combination of accident simulation by an advanced nuclear safety analysis code such as RELAP5/MOD4. The practical developments for the details are demonstrated using an example practice for the SBLOCA (small break loss of coolant accident) case of passive safety PWR (pressurized water reactor) AP1000.

A formal framework of human–machine interaction in proactive maintenance – MANTIS experience

ABSTRACTThe general concept of MANTIS project is to provide a proactive maintenance service platform architecture that allows to monitor essential system parameters and schedule maintenance in order to predict and prevent imminent failures. Human–machine interaction (HMI) is an important integral part of the platform by providing the right information in the right modality to the users when needed. As MANTIS comprises 11 distinct industrial use cases, the design of such HMI presents a great challenge. The framework presented in this paper originates from the scenario-based design and can be treated as a part of the overall scenario-based usability engineering approach. The framework has been conceived from an extensive list of HMI features extracted from the descriptions of use-case scenarios provided by each industrial partner. Due to the broad range of representative industry environments including production asset maintenance, vehicle maintenance, energy production management and health equipment maintenance we believe that the resulting HMI framework can be applied in different cases in practice and the paper would also be of general interest to the readers.