Human-machine System Performance Research Articles

Editorial: Interdisciplinary approaches to the structure and performance of interdependent autonomous human machine teams and systems

Editorial: Interdisciplinary approaches to the structure and performance of interdependent autonomous human machine teams and systems

Human Systems Integration: A Review of Concepts, Applications, Challenges, and Benefits

This paper provides a review of the concepts, applications, and challenges of Human Systems Integration (HSI). HSI is a multidisciplinary field that aims to optimize the performance of human-machine systems by considering the characteristics and capabilities of both human operators and technological systems. The review begins with a discussion of the fundamental concepts of HSI, including human factors engineering, human-computer interaction, and cognitive engineering. It then examines the application of HSI in specific domains such as aviation, transportation, and medicine. The paper concludes by discussing the challenges and future directions of HSI research. The paper highlights the importance of considering the cognitive and affective aspects of human operators in the design of systems and the challenges related to the integration of multiple systems and the consideration of social and organizational factors. Keywords: Human Systems Integration, Healthcare, Manufacturing, Transportation, Benefits, Challenges. DOI: 10.7176/JESD/14-4-04 Publication date: February 28 th 2023

METHOD OF GEOMETRIC INTERPRETATION OF FACIAL EXPRESSIONS FOR EMOTIONS RECOGNITION

Emotional expressions serve a crucial role in interpersonal communication between people while improving social life. In particular, information security systems for visual surveillance that aim to recognize human emotional facial states are highly relevant today. Facial expressions are among the most effective and straightforward means of nonverbal interaction in systems with a human-machine interface. Despite significant scientific and engineering advances in emotion recognition, there are still several challenges in improving the performance of real-time human-machine systems that might work more effectively. In this work, a method of facial geometric feature representation is proposed to improve the operation of security systems. The method is designed to automatically reflect the facial expressions of human emotions in the form of quantitative characteristics of geometric shapes. It uses software-generated landmarks for constructing specific geometric characteristics of the face, which serve as input for the method. Our method consists in forming seven geometric shapes based on predefined landmarks, with the subsequent quantitative expression of these shapes. It was established within the method that the movement of the landmarks when changing facial expressions directly changed the value of each geometric shape. The method outputs the quantitative features of seven shapes, later used to classify emotional facial states. Finally, our method was validated using hyperplane classification. The results of computational experiments confirmed the effectiveness of the proposed method for identifying changes in a person’s emotional state by facial expressions. In addition, the use of simple mathematical calculations in our method has significantly reduced the computational complexity against analogs.

Adaptive automation and its health and safety challenges

Abstract Introduction: This article is a short communique on the subject of Adaptive Automation. Autonomous systems function reliably and effectively, but are workers involved? …better protected? A discussion about when and how workers can, will and/or should integrate with adaptive automated and autonomous systems. Method: A partial literature review was conducted and it was coupled with the author’s thoughts and opinions on the concept and application of adaptive automation. The author’s research of the changing roles of humans in the control systems has been incorporated. Results and conclusions: Human error has driven us to explore automation. The expectation has been that with the human out of the loop, risk of error and injury will decline, however, it is not yet well understood what that decline in risk will look like or what the impact on worker health and safety will be, especially in individual cases. It is generally accepted that human involvement will become more critical, however, concern still exists around skill-decay type incidents when the worker is removed from the control loop and then has to rapidly rejoin the loop if and when something goes wrong. Practical Application: Adaptive automation responds to the engagement question essential to human–machine system performance and therefore, an understanding of the concept and application can be used to inform design decisions about function allocation as control systems are being designed and built.

Automation reliability, human–machine system performance, and operator compliance: A study with airport security screeners supported by automated explosives detection systems for cabin baggage screening

Using a simulated X-ray screening task, we tested 122 airport security screeners working with the support of explosives detection systems for cabin baggage screening (EDSCB) as low-level automation. EDSCB varied systematically on three automation reliability measures: accuracy, d', and positive predictive value (PPV). Results showed that when unaided performance was high, operator confidence was high, and automation provided only small benefits. When unaided performance was lower, operator confidence was lower, and automation with higher d' provided large benefits. Operator compliance depended on the PPV of automation: We found lower compliance for lower PPV. Automation with a high false alarm rate of 20% and a low PPV of .3 resulted in operators ignoring about one-half of the true automation alarms on difficult targets—a strong cry-wolf effect. Our results suggest that automation reliability described by d' and PPV is more valid than using accuracy alone. When the PPV is below .5, operators should receive clear instructions on how to respond to automation alarms.

Sensor Networks for Aerospace Human-Machine Systems.

Intelligent automation and trusted autonomy are being introduced in aerospace cyber-physical systems to support diverse tasks including data processing, decision-making, information sharing and mission execution. Due to the increasing level of integration/collaboration between humans and automation in these tasks, the operational performance of closed-loop human-machine systems can be enhanced when the machine monitors the operator’s cognitive states and adapts to them in order to maximise the effectiveness of the Human-Machine Interfaces and Interactions (HMI2). Technological developments have led to neurophysiological observations becoming a reliable methodology to evaluate the human operator’s states using a variety of wearable and remote sensors. The adoption of sensor networks can be seen as an evolution of this approach, as there are notable advantages if these sensors collect and exchange data in real-time, while their operation is controlled remotely and synchronised. This paper discusses recent advances in sensor networks for aerospace cyber-physical systems, focusing on Cognitive HMI2 (CHMI2) implementations. The key neurophysiological measurements used in this context and their relationship with the operator’s cognitive states are discussed. Suitable data analysis techniques based on machine learning and statistical inference are also presented, as these techniques allow processing both neurophysiological and operational data to obtain accurate cognitive state estimations. Lastly, to support the development of sensor networks for CHMI2 applications, the paper addresses the performance characterisation of various state-of-the-art sensors and the propagation of measurement uncertainties through a machine learning-based inference engine. Results show that a proper sensor selection and integration can support the implementation of effective human-machine systems for various challenging aerospace applications, including Air Traffic Management (ATM), commercial airliner Single-Pilot Operations (SIPO), one-to-many Unmanned Aircraft Systems (UAS), and space operations management.

Human Work and Status Evaluation Based on Wearable Sensors in Human Factors and Ergonomics: A Review

Humans are an important component of human-machine systems. A better understanding of the role and the status of humans can facilitate and improve the overall human-machine system performance, as well as ensure the well-being of humans. In human factors and ergonomics (HF/E), conventional methods of human performance evaluation usually require the efforts of trained personnel for data collection, data analysis, and the explanation of results. There is an emerging need for a novel and cost-efficient method of assessing human work and status in various systems. The development of wearable technologies has improved the potential for developing a smarter and automatic solution to performing relevant evaluations. In this paper, the authors conduct a scoping review of the studies published from 2001 to 2017 that focused on wearable sensor technology and propose a framework to summarize the research topics. The main steps in the framework include data collection, data processing, and system feedback. Specifically, considering the gaps between current HF/E studies of wearables and the available resources, the authors conducted a detailed review of the application of wearables in HF/E human work evaluation. The opportunities and challenges of introducing wearable sensors into HF/E evaluation are discussed.

The Tides of the Zodiac MK VI HD: Comparing the Usability of Inflatable Boats for Seaborne Operations

OCCUPATIONAL APPLICATIONSThis user-centered study compared the usability of old and new models of an inflatable boat during a multi-day high-pressure military exercise. Participants represented the entire population of Special Operations soldiers who were experienced with both models, providing assessments on the effectiveness, efficiency, and user satisfaction of the boats and their central console, one of the main components that was updated on the new model. The new model was rated as significantly more usable in terms of effectiveness, efficiency, and user satisfaction. Although the response to the new model was overwhelmingly positive, follow-up interviews identified specific problems associated with changes introduced with the new model. These included: (a) visibility problems with the windshield, caused by the accumulation of spray and salt; (b) increased difficulty of moving/installing the central console, due to the doubling of its weight; and (c) fuel uptake issues, resulting from the new flexible fuel bladder.TECHNICAL ABSTRACTBackground: The military has been used as a test bed for human-system performance for decades. The present study used the upgrade of the Zodiac MK VI Heavy-duty (Zodiac MK VI HD) inflatable boat, issued to a platoon of a Special Operation Engineers, as a unique opportunity to compare the old and new model’s usability during multi-day seaborne exercises. Purpose: To compare the old and new Zodiac MK VI HD (with its upgraded central console) in terms of effectiveness, efficiency, and user satisfaction, by involving expert users in their most extreme context of use. The assessment implied both a formal empirical measurement that enabled a data-driven comparison between both boat types, and an interview-based problem discovery phase to identify future design improvements. Method: Using a customized questionnaire based on the ISO 9241 standard, a repeated-measures field study was completed using all Special Operation Engineers (N = 17), using both Zodiac MK VI HD types interchangeably, in their most extreme environment of use. Remaining design flaws were explored using open-ended survey questions and interviews immediately following usage. Results: The new Zodiac MK VI HD, including all upgrades (e.g., console, digital controls, LED-based displays) was rated significantly higher in terms of effectiveness, efficiency, and user satisfaction. Additionally, problem discovery efforts revealed that design issues remained with the opaque windshield during seaborne operations, weight of the console, flexible fuel bladder, and power loss that affected the console’s digital speed and position display. Conclusions: These results highlight the importance of comparative usability research and problem discovery, directly involving end-users in high-pressure operational circumstances. Broadening its inferences to many other high-reliability occupations (e.g., law enforcement, fire fighters, and medical first responders), effectiveness, efficiency, satisfaction, ease of maintenance, and feelings of increased safety are important considerations, because they affect the performance of human-machine systems.

A hybrid reinforced learning system to estimate resilience indicators

This paper describes a learning system based on resilience indicators. It proposes a hybrid learning system to estimate Human–Machine System performance when facing unprecedented situations. Collected data from various criteria are compared with data estimated using the local and the global resilience indicators, to give both instantaneous and over-time Human–Machine System states. The learning system can be composed of two different, separate reinforcement functions; the first allowing reinforcement of its own system knowledge and the second allowing reinforcement of its estimation function. When used together in a hybrid approach, the resilience indicator estimation should be improved. The learning system is then applied in a simulated air transport context and the impact of each reinforcement function on resilience indicator estimation is assessed. The hypothesis on performance of hybrid reinforcement learning is confirmed and it provides better results than those obtained by the knowledge based reinforcement or the estimation based reinforcement alone.

Information Model and Architecture Specification for Context Awareness Interaction Decision Support in Cyber-Physical Human–Machine Systems

This paper aims to contribute to situation, activity, and goal awareness in cyber-physical human–machine systems (HMS) by presenting a new information model and specifications for a decision-making component that can be integrated in current system architectures. The objective of this work is to improve the efficacy, acceptance, adaptability, and overall performance of HMS and human–system interaction (HSI) applications using a context-based approach. Our hypothesis is that we can enhance current interaction functionalities by integrating context and interaction information models into a decision-making component that behaves as a supervision process for controlling interaction. In HSI, we aim to define a general human model that may lead to principles and algorithms, allowing more natural and effective interaction between humans and artificial agents. The approach was implemented and tested targeting application in the domain of active and assisted living. The challenge of user acceptance is of vital importance for future solutions and is still one of the major reasons for reluctance to adopt cyber-physical systems in this domain.

Designing intelligent manufacturing systems through Human-Machine Cooperation principles: A human-centered approach

Since the start of industrialization, machine capabilities have increased in such a way that human control of processes has evolved from simple (with mechanization) to cognitive (with computerization), and even emotional (with semi/full automation). The processes have also evolved from simple to complicated, and now complex systems, in the emerging context of Industry 4.0. This is notably the case with Intelligent Manufacturing Systems in which processes have become so autonomous that humans are unaware of the processes running, while they may need to intervene to update the manufacturing plan or modify the process configuration if a machine breaks down, or to assist process-intelligent entities when they find themselves in a deadlock. This paper highlights the lack of attention paid to the correct integration of humans in Intelligent Manufacturing Systems and provides solutions based on Human-Machine Cooperation principles to retain humans in the process control loop with different levels of involvement identified by the levels of automation. The aim of these principles is to propose a human-centered approach to design and evaluate systems, processes, and their interactions with humans. Herein, these principles are detailed and applied to Intelligent Manufacturing Systems using Artificial Self-Organizing systems (ASO) as an example. An assistance system was designed to support cooperation between ASO and human operators. Experiments were conducted to evaluate the system and its utility in improving the performance of Human-Machine Systems, as well as its acceptability with regard to human factors. The results presented highlight the advantages of the approach within the context of Industry 4.0.

Modeling and control of operator functional state in a unified framework of fuzzy inference petri nets

Background and objectiveIn human-machine (HM) hybrid control systems, human operator and machine cooperate to achieve the control objectives. To enhance the overall HM system performance, the discrete manual control task-load by the operator must be dynamically allocated in accordance with continuous-time fluctuation of psychophysiological functional status of the operator, so-called operator functional state (OFS). The behavior of the HM system is hybrid in nature due to the co-existence of discrete task-load (control) variable and continuous operator performance (system output) variable. MethodsPetri net is an effective tool for modeling discrete event systems, but for hybrid system involving discrete dynamics, generally Petri net model has to be extended. Instead of using different tools to represent continuous and discrete components of a hybrid system, this paper proposed a method of fuzzy inference Petri nets (FIPN) to represent the HM hybrid system comprising a Mamdani-type fuzzy model of OFS and a logical switching controller in a unified framework, in which the task-load level is dynamically reallocated between the operator and machine based on the model-predicted OFS. Furthermore, this paper used a multi-model approach to predict the operator performance based on three electroencephalographic (EEG) input variables (features) via the Wang-Mendel (WM) fuzzy modeling method. The membership function parameters of fuzzy OFS model for each experimental participant were optimized using artificial bee colony (ABC) evolutionary algorithm. Three performance indices, RMSE, MRE, and EPR, were computed to evaluate the overall modeling accuracy. ResultsExperiment data from six participants are analyzed. The results show that the proposed method (FIPN with adaptive task allocation) yields lower breakdown rate (from 14.8% to 3.27%) and higher human performance (from 90.30% to 91.99%). ConclusionThe simulation results of the FIPN-based adaptive HM (AHM) system on six experimental participants demonstrate that the FIPN framework provides an effective way to model and regulate/optimize the OFS in HM hybrid systems composed of continuous-time OFS model and discrete-event switching controller.

Augmented-Reality-Based Indoor Navigation: A Comparative Analysis of Handheld Devices Versus Google Glass

Navigation systems have been widely used in outdoor environments, but indoor navigation systems are still in early development stages. In this paper, we introduced an augmented-reality-based indoor navigation application to assist people navigate in indoor environments. The application can be implemented on electronic devices such as a smartphone or a head-mounted device. In particular, we examined Google Glass as a wearable head-mounted device in comparison with handheld navigation aids including a smartphone and a paper map. We conducted both a technical assessment study and a human factors study. The technical assessment established the feasibility and reliability of the system. The human factors study evaluated human–machine system performance measures including perceived accuracy, navigation time, subjective comfort, subjective workload, and route memory retention. The results showed that the wearable device was perceived to be more accurate, but other performance and workload results indicated that the wearable device was not significantly different from the handheld smartphone. We also found that both digital navigation aids were better than the paper map in terms of shorter navigation time and lower workload, but digital navigation aids resulted in worse route retention. These results could provide empirical evidence supporting future designs of indoor navigation systems. Implications and future research were also discussed.

A Robust Operator Functional State Fuzzy Modeling Approach Based on EEG Data

Operator functional state (OFS) is defined as the time-variable ability that an operator completes his/her assigned tasks. To evaluate the OFS in safety-critical human-machine systems, it is modeled by using the Wang-Mendel-based fuzzy system paradigm in this paper. The fuzzy model is constructed to correlate three EEG features (as model inputs) to the human-machine system performance (as model output). To derive a fuzzy model for real-time OFS assessment, the Gaussian membership function membership crossover point membership gradeδis found to be an essential parameter that controls the robustness of data-driven fuzzy models. The fuzzy models with differentδare applied to the OFS fuzzy modeling. The results have demonstrated that an appropriate value ofδcan be selected to derive robust fuzzy models. Compare with the results obtained by fuzzy models based on symmetric Gaussian membership functions, the new approach based on asymmetric Gaussian membership function leads to considerably improved robustness performance.

Task partitioning effects in semi-automated human–machine system performance

Twelve professional pilots performed a flight simulation consisting of three component sub-tasks: (i) tracking, (ii) monitoring and (iii) targeting, respectively. The targeting sub-task required (i) target identification, (ii) weapon selection and then (iii) weapon release. Pilots performed in a fully manual condition, a partial automation condition or a fully automated condition. Automated assistance was provided for the targeting sub-task only, while tracking and monitoring sub-tasks were always performed manually. During full automation, the computer located the target, identified it and released the appropriate weapon without any pilot input. During partial automation, the computer located and identified the target while the pilot retained final control over weapon release. Significantly higher levels of tracking error distinguished manual from both automated conditions and also between the two levels of automation. Monitoring response times were also sensitive to the degree of automation engaged, with the partial-automation condition exhibiting faster responses than full automation. Findings support a design principle in which pilots retain control over final weapons release directly on the basis of objective performance outcome. These collective results support the contention that effective and principled task-partitioning should represent a central strategy for the evolution of complex human–machine systems. Practitioner Summary: Advantages of partitioning tasks between human and automated control are contingent upon the overall context of performance and the actual way the partitioning is accomplished. Simple algorithms, for example, automate on every feasible occasion, are poor design heuristics and may even prove actively harmful to overall response capacity. Transitioning humans from active controllers to passive monitors can be a problematic design choice, especially when that individual is socially deemed to retain overall responsibility for ultimate system effects in the real world.

Three-dimensional spatial stimulus–response (S–R) compatibility for visual signals with hand and foot controls

This study investigated the performance of 24 participants in eight different three-dimensional spatial stimulus–response (S–R) mapping conditions. The stimulus visual signals were positioned at the corners of an imaginary cube, and responses were made by hands and feet. There was a significant interaction of visual signal position and response device position which demonstrated the existence of a strong spatial S–R compatibility effect in the task. Relatively faster reaction times and lower error percentages were obtained for the compatible S–R conditions. The spatial compatibility effect in the up–down dimension was the strongest, and the effect in the left–right dimension was stronger than the effect in the front–rear dimension. The findings of this study could be translated into practical and useful ergonomics recommendations for designing visual displays and control devices in three-dimensional space for improving overall human–machine system performance.

Simulator-based human-machine interaction design

Human-in-the-loop simulators have been used for some decades in the automotive area. Historically, the most prevalent motive for this, beside training purposes, has been to study human performance in various situations. We suggest another focus, to evaluate human-machine system performance for various car system solutions as a fundamental part of simulator-based design (SBD). Our technical background is human-machine interaction (HMI), which is only one area where SBD could be deployed. However, in combination with human-in-the-loop simulation, HMI should be the most relevant perspective. In order to facilitate this SBD approach, it is necessary to have simulators (software and hardware) that support this way to implement new car system solutions to work seamlessly together with the entire vehicle system. Unfortunately, there are few such simulators available on the market today. We will expand this theme in the article and also address important questions concerning fidelity and validity.

Spatial stimulus–response (S-R) compatibility for foot controls with visual displays

This experiment examined the performance of 32 Chinese right-handed/right-footed participants for four spatial stimulus–response (S-R) mapping conditions with visual signals and foot controls in transverse and longitudinal orientations. The results showed that reaction times for the four S-R mappings were significantly different from each other. There was a significant interaction of visual signal position and response pedal key position which demonstrated the existence of a spatial S-R compatibility effect for this task. Compared to other non-correspondent S-R mappings, an improvement of 135 ms in reaction time was obtained with the correspondence of the stimulus and response key in both the transverse and longitudinal orientations. The findings suggested that the relative positions of visual signals and foot pedal response keys should be spatially compatible for the best human machine system performance. There was also evidence that spatial compatibility for the right–left dimension was stronger than for the front/rear dimension, and that responses with the right foot were faster than with left foot for the right-footed subjects used here. Relevance to industry Most of the previous studies on spatial S-R compatibility have been limited to the use of hand controls. The results of this study provide useful ergonomics guidelines for designing foot controls and visual displays, and this will be valuable for improving efficiency and performance in human–machine systems.

Intelligent Adaptive Interfaces for the Control of Multiple UAVs

A lack of guidance for designing complex, dynamic networked systems presents challenges to the development of such systems to maximize overall human-machine system performance. An intelligent adaptive interface (IAI) concept and associated technologies have been developed to address this problem. In order to support effective decision making, a typical IAI is driven by software agents that can change the display and/or control characteristics to react to the changes of mission and operator states in real time. This work investigated the efficacy of IAIs in a multi-uninhabited aerial vehicle (UAV) scenario. The IAI was modeled as part of the UAV tactical workstations found in a maritime patrol aircraft. A performance model was developed to compare the difference in mission activities with and without IAI agents. A prototype IAI experimental environment was implemented for a human-in-the-loop empirical investigation. Both simulation and experiment results revealed that the control of multiple UAVs is a cognitively complex task with high workload. IAIs facilitated a significant reduction in workload and an improvement in situation awareness, thus allowing operators to continue working under high time pressure. This research revealed IAI triggering conditions under different cognitive workload situations.

An exploratory study of chaos in human-Machine system dynamics

The human-machine system behavior and performance are dynamic, nonlinear, and possibly chaotic. Various techniques have been used to describe such dynamic and nonlinear system characteristics. However, these techniques have rarely been able to accommodate the chaotic behavior of such a nonlinear system. Therefore, this study proposes the use of nonlinear dynamic system theory as one possible technique to account for the dynamic, nonlinear, and possibly chaotic human-machine system characteristics. It briefly describes some of the available nonlinear dynamic system techniques and illustrates how their application can explain various properties of the human-machine system. A pilot's heart interbeat interval (IBI) and altitude tracking error time series data are used in the illustration. Further, the possible applications of the theory in various domains of human factors for on-line assessment, short-term prediction, and control of human-machine system behavior and performance are discussed.