Human-machine Cooperative System Research Articles

Human–Machine Cooperative Driving and Real-Time Algorithm Implementation Based on Double-Loop Control System

The double loop structure has been implemented to verify the feasibility of human-machine cooperative control in real environment in this paper, which can fully reflect the performance of the controller and realization of the sharing steering control between human driver and automatic control system. The outer loop adopts fuzzy proportional-integral-derivative(PID) controller to deal with different working conditions, on which the complexity of the algorithm is reduced and the real-time performance of the controller is improved. The inner loop adopts the active disturbance rejection controller (ADRC), which can effectively suppress the interference such as drivers’ faulty operations etc. and improve the tracking accuracy and system performance. The stability for extended state observe(ESO) and ADRC is described in theory to ensure the feasibility of the controller in the real world environment. The experiment results show that the human-machine cooperative control system based on the double-loop controller can effectively reduce the lateral error of the vehicle and the steering torque of the driver in the process. And the control performance of the vehicle is improved.

Special Issue on Human Digital Twin Technology

With the advancement of information technologies such as the Internet of Things and artificial intelligence, cyber-physical systems are being introduced into society. At the core of these systems is the digital twin, a computer model (twin) of a physical entity built in cyberspace for simulation-based prediction. Currently, the digital twin mainly targets artificial objects, such as aircraft engines and factories. However, if it can be extended to humans, it could lead to the realization of human-machine cooperative systems and health promotion services, thereby solving social issues stemming from the aging of society. However, humans are the weakest link in the system, and many technical problems remain to be solved, such as realizing the measurement, modeling, and prediction of human behavior, if the human digital twin is to become reality. This special issue contains 9 papers on developing essential technologies for the human digital twin and constructing human-machine systems for specific applications. The topics covered include learning algorithms, motion measurement and analysis techniques, human perception, system development, and platform software for system development. These clearly show that cross-disciplinary efforts are essential to the realization of the human digital twin. We thank the authors of the papers submitted for this special issue. We are confident that the information provided by the authors is suggestive and informative for both specialists and non-specialists alike. We also sincerely appreciate the efforts of the reviewers. Their contributions helped to make this special issue possible. We hope that this special issue will catalyze sharing across the boundaries of research fields along the path to realizing a human digital twin.

Robust and Realtime Large Deformation Ultrasound Registration Using End-to-End Differentiable Displacement Optimisation

Image registration for temporal ultrasound sequences can be very beneficial for image-guided diagnostics and interventions. Cooperative human-machine systems that enable seamless assistance for both inexperienced and expert users during ultrasound examinations rely on robust, realtime motion estimation. Yet rapid and irregular motion patterns, varying image contrast and domain shifts in imaging devices pose a severe challenge to conventional realtime registration approaches. While learning-based registration networks have the promise of abstracting relevant features and delivering very fast inference times, they come at the potential risk of limited generalisation and robustness for unseen data; in particular, when trained with limited supervision. In this work, we demonstrate that these issues can be overcome by using end-to-end differentiable displacement optimisation. Our method involves a trainable feature backbone, a correlation layer that evaluates a large range of displacement options simultaneously and a differentiable regularisation module that ensures smooth and plausible deformation. In extensive experiments on public and private ultrasound datasets with very sparse ground truth annotation the method showed better generalisation abilities and overall accuracy than a VoxelMorph network with the same feature backbone, while being two times faster at inference.

A Supervised Machine Learning Approach to Operator Intent Recognition for Teleoperated Mobile Robot Navigation

In applications that involve human-robot interaction (HRI), human-robot teaming (HRT), and cooperative human-machine systems, the inference of the human partner's intent is of critical importance. This paper presents a method for the inference of the human operator's navigational intent, in the context of mobile robots that provide full or partial (e.g., shared control) teleoperation. We propose the Machine Learning Operator Intent Inference (MLOII) method, which a) processes spatial data collected by the robot's sensors; b) utilizes a supervised machine learning algorithm to estimate the operator's most probable navigational goal online. The proposed method's ability to reliably and efficiently infer the intent of the human operator is experimentally evaluated in realistically simulated exploration and remote inspection scenarios. The results in terms of accuracy and uncertainty indicate that the proposed method is comparable to another state-of-the-art method found in the literature.

Coordination Control Strategy for Human-Machine Cooperative Steering of Intelligent Vehicles: A Reinforcement Learning Approach

This paper proposes a novel coordination control strategy through reinforcement learning approach for human-machine cooperative steering of intelligent vehicles, so as to realize a more flexible and efficient way for the human driver and the automated driving system to jointly complete the path-following. Firstly, the human-machine cooperative steering is modeled by the optimal preview driver model with adaptive preview time and the MPC lateral tracking model with adaptive prediction horizon and step, as well as the steering system with a realistic dual-motor steering-by-wire structure is also built into the human-machine cooperative steering system. Then, the human-machine coordination control strategy is designed and trained based on two reinforcement learning agents to further optimize the allocation of cooperative steering weights. Finally, the strategy is fully verified in lane departure simulation scenarios and also in real-world Steering-in-the-Loop experiments with real-time controller, which shows that both agents can effectively keep the vehicle with the small lateral and yaw errors in the simulation scenarios, and also can be successfully applied in the realistic human-machine cooperative system to maintain the lateral and yaw errors within a safe range during path-following.

Traded and combined cooperative control of a smart wheelchair

AbstractThis article deals with a human–machine cooperative system for the control of a smart wheelchair for people with motor disabilities. The choice of a traded control mode is first argued. The paper then pursues two objectives. The first is to describe the design of the cooperative system by focusing on the dialogue and the interaction between the pilot and the robot. The second objective is to introduce a new cooperative mode. In this one, three features are proposed: two semi-autonomous features, a wall following and a doorway crossing, during which the user can intervene punctually to rectify a trajectory or a path, and an assisted mode where, conversely, the machine intervenes in a manual control to avoid obstacles. This mode of intervention of an entity, human or machine, supervising a movement controlled by the other is referred as “combined control.” Examples of scenarios exploiting the cooperative capabilities of the system are presented and discussed.

Erato: Cooperative Data Story Editing via Fact Interpolation.

As an effective form of narrative visualization, visual data stories are widely used in data-driven storytelling to communicate complex insights and support data understanding. Although important, they are difficult to create, as a variety of interdisciplinary skills, such as data analysis and design, are required. In this work, we introduce Erato, a human-machine cooperative data story editing system, which allows users to generate insightful and fluent data stories together with the computer. Specifically, Erato only requires a number of keyframes provided by the user to briefly describe the topic and structure of a data story. Meanwhile, our system leverages a novel interpolation algorithm to help users insert intermediate frames between the keyframes to smooth the transition. We evaluated the effectiveness and usefulness of the Erato system via a series of evaluations including a Turing test, a controlled user study, a performance validation, and interviews with three expert users. The evaluation results showed that the proposed interpolation technique was able to generate coherent story content and help users create data stories more efficiently.

Utility assessment in automated driving for cooperative human–machine systems

Currently, car manufacturers, suppliers, and IT companies are surpassing each other with ambitious plans regarding their driving automation technology. However, even the most optimistic announcements grant that, for a certain time, a human driver cannot be replaced in all driving situations. Hence, human drivers will still be a part of future traffic by working together with automation systems. Analyzing the joint decision-making process of such a human–machine system in automated driving provides a better understanding of the resulting traffic system. In this paper, a driving simulator study with 33 participants focusing on the utility of cooperative driver–vehicle systems with the use case of highway driving is presented. Based on the study’s results, a model that explains the linkage between subjective measures such as the perceived utility and objective driving data is derived. Moreover, on an individual level, models are parameterized by using driving states as predictors and the individual utility perceived in a driving situation as response. This individual utility can be used for predicting driving actions such as the initiation of overtaking maneuvers.

Real-time detection of driver distraction: random projections for pseudo-inversion-based neural training

There is an accumulating evidence that distracted driving is a leading cause of vehicle crashes and accidents. In order to support safe driving, numerous methods of detecting distraction have been proposed, which are empirically focused on certain driving contexts and gaze behaviour. This paper aims at illustrating a method for the non-intrusive and real-time detection of visual distraction based on vehicle dynamics data and environmental data, without using eye-tracker information. Experiments are carried out in the context of the automotive domain of the European project Holides, which addresses development and qualification of adaptive cooperative human–machine systems, and is co-funded by ARTEMIS Joint Undertaking and Italian University, Educational and Research Department. The collected data are analysed by a single-layer feedforward neural network trained through pseudo-inversion methods, characterized by direct determination of output weights given randomly set input weights and biases. One main feature of our work is the convenient setting of input weights by the so-called sparse random projections: the presence of a great number of null elements in the involved matrices makes especially parsimonious the use at run time of the trained network. Moreover, we use a genetic approach to better explore the input weights network space. The obtained results show better performance with respect to classical pseudo-inversion methods and effective and parsimonious use of memory resources.

Analysis of Upper-Extremity Motion and Muscle and Brain Activation During Machine Operation in Consideration of Mass and Friction

There have been numerous studies on human–machine cooperative systems. In such studies, it is generally preferable to consider human maneuverability and motion during machine operation as well as considering mechanical parameters. However, many of the current studies only evaluate the mechanical parameters. Although some studies have evaluated human parameters, these remain subjective. Therefore, in this study, upper extremity movement and muscle and brain activation during machine operation with inertia and friction were measured quantitatively. The experimental conditions were divided into nine classes. These were then combined into the following frictional and mass conditions: the friction during machine operation was ${\text{3.3}}\pm {\text{1.2}},{\text{8.6}}\pm {\text{2.7}}$ , or $\text{11.5}\pm \text{3.8}$ N; the mass of the machine was $\text{2.2}\pm \text{0.1},\text{3.2}\pm \text{0.2}$ , or $\text{4.4}\pm \text{0.2}$ kg. The experimental results revealed significant differences in upper extremity muscle activation with both friction and mass $(p . Although the brain analysis results showed no significant difference with friction, significant differences were found with mass $(p . The experimental results showed that the trajectory of the hand position, upper extremity muscle activation, and brain activation tended to be affected by mass.

Changes in brain activation associated with backlash magnitude: a step toward quantitative evaluation of maneuverability

In recent years, there have been many studies on human–machine cooperative systems. Studies evaluating human maneuverability by questionnaire and operating performance have been reported; however, quantitative analyses of human perception when operating a machine have not been conducted. The present study focuses on mechanical backlash to evaluate maneuverability quantitatively. The purpose of this study is to propose a method to quantitatively evaluate the perception of mechanical backlash. To evaluate the degree of backlash perception, biological information of subjects was measured while they used the fingertips of their right hand to rotate a mechanism under different magnitudes of backlash: 1.3, 1.0, 0.7, and 0.4 mm. Biological information including higher brain functions and surface potential signals were measured using near-infrared spectroscopy and surface electromyography, respectively. On a questionnaire after each trial, the subjects reported if they could perceive the mechanical backlash using a four-point scale (1: hardly; 2: a little; 3: somewhat; 4: clearly). The results of the questionnaire on the magnitude of the backlash indicated that subjects were able to perceive all of the backlash conditions at nearly the same level. We confirmed that changing the magnitude of the backlash changed the brain activation state.

FML-based Dynamic Assessment Agent for Human-Machine Cooperative System on Game of Go

In this paper, we demonstrate the application of Fuzzy Markup Language (FML) to construct an FML-based Dynamic Assessment Agent (FDAA), and we present an FML-based Human–Machine Cooperative System (FHMCS) for the game of Go. The proposed FDAA comprises an intelligent decision-making and learning mechanism, an intelligent game bot, a proximal development agent, and an intelligent agent. The intelligent game bot is based on the open-source code of Facebook’s Darkforest, and it features a representational state transfer application programming interface mechanism. The proximal development agent contains a dynamic assessment mechanism, a GoSocket mechanism, and an FML engine with a fuzzy knowledge base and rule base. The intelligent agent contains a GoSocket engine and a summarization agent that is based on the estimated win rate, real-time simulation number, and matching degree of predicted moves. Additionally, the FML for player performance evaluation and linguistic descriptions for game results commentary are presented. We experimentally verify and validate the performance of the FDAA and variants of the FHMCS by testing five games in 2016 and 60 games of Google’s Master Go, a new version of the AlphaGo program, in January 2017. The experimental results demonstrate that the proposed FDAA can work effectively for Go applications.

A hierarchical framework to analyze shared control conflicts between human and machine

Abstract: Shared Control, where the machine and the human share tasks and control the situation together, and its extension cooperative automation are promising approaches to overcome automation-induced problems, such as lack of situation awareness and degradation of skill. However, the design of Shared Controllers and/or cooperative human-machine systems should be done in a very careful manner. One of the major issues is conflicts between the human and the machine: how to detect these conflicts, and how to resolve them, if necessary? A complicating factor is that when the human is right, conflicts are undesirable (resulting in nuisance, degraded performance, etc), but when the machine is right, conflicts are desirable (warning the operator, or proper assistance or overruling). Research has pointed out several types and causes of conflicts, but offers no coherent framework for design and evaluation guidelines. In this paper, we propose such a theoretical framework in order to structure and relate different types of conflicts. The framework is inspired by a hierarchical task analysis, and identifies five possible sources of conflicts: intent, information gathering, information processing, decision-making and action implementation. Examples of conflicts in several application domains such as automobile, telerobotics, and surgery are discussed to illustrate the applicability of this framework.

Implementation of Human System Integration (HSI) and ‘Non-functional Characteristics’ into the Systems Engineering Lifecycle - A Practical Approach at Airbus Defence and Space

This paper provides an overview of the process, practices, developments and synergies that Airbus Defence and Space are implementing Human System Integration (HSI) into the Continuous Engineering /Model Based System Engineering (MBSE) throughout the company. The paper highlights these developments, which include internal research programmes, European Union funded research projects, Airbus internal Synergy project. This research includes an EU funded 3 year project HoliDes - Holistic Human Factors and System Design of Adaptive Cooperative Human Machine Systems. Continuous engineering within Airbus has included various corporate and divisional Synergy programs and lean and agile initiatives. As our business is defined by Multiple Portfolio products, supplied from multi countries, the MBSE process had to speed up delivery and reduce overall cost. The Airbus process models needed significant amendment due to products supplied via multiple business lines throughout the company. This problem is exacerbated by varying requirements of four national defence requirements, who impose restrictions on the use of Architectural Frameworks (NAF/MODAF/DODAF) and supporting tools. Our Human System Integration (HSI) is continuing from the initial Human View Architectural Framework research to include the nine HSI domains. These nine domains overlap with our current Human factors research and will now allow us to pull in our Non-functional work into our Systems Engineering lifecycle. This paper highlights the research projects Airbus Space and Defence are incorporating in to our overall strategy, these include internal R&D funding and European Union funded research programs. These research projects have highlighted several areas of interest these include, stochastic modelling with respect to Human System Integration and Human views, verification and analysis of emergent behaviour and non-functional properties, which conforms with our own work on non-functional requirements and finally the use of System of System Formal Methods for parts of the lifecycle.

Pattern Classification of Human Cognitive State in Man-Machine Systems Using Multiple Sources of Pyschophysiological Data and Fuzzy Clustering Approach

The primary goal of this work is to classify Human Cognitive State (HCS) in human-machine cooperative control systems. A total of 10 young and fit volunteers were used as the experimental subjects. A set of 9 process control task conditions were programmed on an automation-enhanced Cabin Air Management System (aCAMS) originally developed to simulate with high-fidelity the life support system in aerospace applications. The psychophysiological and performance data of the subjects were recorded while they performed process control operations in collaboration with computer-based automatic control systems. The fuzzy C-Means (FCM) algorithm was used to classify the momentary HCSs into three categories: “Good”, “Average” and “Risky” with certain degree of membership. The classification results indicated that the FCM-based classifier can achieve accurate HCS classification if the influential features are properly selected. The method proposed has potential to be applied to design adaptive task (or functional) allocation strategy in adaptive/intelligent human-machine control systems.

Classifying human operator functional state based on electrophysiological and performance measures and fuzzy clustering method.

The human operator's ability to perform their tasks can fluctuate over time. Because the cognitive demands of the task can also vary it is possible that the capabilities of the operator are not sufficient to satisfy the job demands. This can lead to serious errors when the operator is overwhelmed by the task demands. Psychophysiological measures, such as heart rate and brain activity, can be used to monitor operator cognitive workload. In this paper, the most influential psychophysiological measures are extracted to characterize Operator Functional State (OFS) in automated tasks under a complex form of human-automation interaction. The fuzzy c-mean (FCM) algorithm is used and tested for its OFS classification performance. The results obtained have shown the feasibility and effectiveness of the FCM algorithm as well as the utility of the selected input features for OFS classification. Besides being able to cope with nonlinearity and fuzzy uncertainty in the psychophysiological data it can provide information about the relative importance of the input features as well as the confidence estimate of the classification results. The OFS pattern classification method developed can be incorporated into an adaptive aiding system in order to enhance the overall performance of a large class of safety-critical human-machine cooperative systems.

A human-machine cooperative system for generating sign language animation using thermal image processing, a fuzzy algorithm, and simulated annealing

We propose a method for a sign language animation by skin region detection applied to an infrared thermal image. In a system incorporating the proposed method, a 3D CG model corresponding to a person's characteristic posture while using sign language is generated automatically by pattern recognition of the thermal image, and then a person's hand in the CG model is set. The hand part is made manually beforehand. If necessary, the model can be replaced manually by a more appropriate model corresponding to training key frames, and/or the same generated model can be refined manually. In our experiments, three hearing-impaired people, who were experienced in using sign language, recognized the Japanese sign language gestures of 70 words expressed as animations with 94.3% accuracy. We further improved the system by correcting the position and direction of the hand of the automatically generated model through the use of a fuzzy algorithm and simulated annealing.

Designing dynamic distributed cooperative Human-Machine Systems

The paper presents a new approach to the development of cooperative human-machine systems in the Transportation domain which is currently researched in the European project D3CoS. A necessary precondition for the acceptance of cooperative human-machine systems with shared control is the confidence and trust of the user into the system. D3CoS tackles this important issue by addressing the cooperative system as the object and the target of the system development process. This new perspective, along with corresponding innovative methods, techniques and tools, shall allow the identification of optimal task and authority sharing approaches supported by intuitive human-machine interaction and user interfaces at an early stage of system development. This will support powerful teamwork between humans and machines or between machines and machines that is transparent, intuitive and easy to understand. The paper describes the research dimensions for the development of the methods, techniques and tools as well as first results.

Towards a dynamic balance between humans and automation: authority, ability, responsibility and control in shared and cooperative control situations

Progress enables the creation of more automated and intelligent machines with increasing abilities that open up new roles between humans and machines. Only with a proper design for the resulting cooperative human–machine systems, these advances will make our lives easier, safer and enjoyable rather than harder and miserable. Starting from examples of natural cooperative systems, the paper investigates four cornerstone concepts for the design of such systems: ability, authority, control and responsibility, as well as their relationship to each other and to concepts like levels of automation and autonomy. Consistency in the relations between these concepts is identified as an important quality for the system design. A simple graphical tool is introduced that can help to visualize the cornerstone concepts and their relations in a single diagram. Examples from the automotive domain, where a cooperative guidance and control of highly automated vehicles is under investigation, demonstrate the application of the concepts and the tool. Transitions in authority and control, e.g. initiated by changes in the ability of human or machine, are identified as key challenges. A sufficient consistency of the mental models of human and machines, not only in the system use but also in the design and evaluation, can be a key enabler for a successful dynamic balance between humans and machines.

A modelling formalism for human-machine cooperative systems

This paper presents a collection of models of humans involved in complex systems with a focus on control of the system while allowing human participation in decisions for system operation. The existence of a formal model that captures human behaviour in a complex system allows for the efficient development of modelling and control software of man-machine systems. This paper provides a foundation for modelling and software development for complex systems that includes human activities. A modelling formalism, based on the automata theory for human-machine cooperative systems is demonstrated in this paper in consideration with the ecological concept of affordance.