Human-machine Collaborative Systems Research Articles

Towards Implementation of Emotional Intelligence in Human–Machine Collaborative Systems

Social awareness and relationship management components can be seen as a form of emotional intelligence. In the present work, we propose task-related adaptation on the machine side that accounts for a person’s momentous cognitive and emotional state. We validate the practical significance of the proposed approach in person-specific and person-independent setups. The analysis of results in the person-specific setup shows that the individual optimal performance curves for that person, according to the Yerkes–Dodson law, are displaced. Awareness of these curves allows for automated recognition of specific user profiles, real-time monitoring of the momentous condition, and activating a particular relationship management strategy. This is especially important when a deviation is detected caused by a change in the person’s state of mind under the influence of known or unknown factors.

A digital twin-driven flexible scheduling method in a human-machine collaborative workshop based on hierarchical reinforcement learning.

Under the influence of the global COVID-19 pandemic, the demand for medical equipment and epidemic prevention materials has increased significantly, but the existing production lines are not flexible and efficient enough to dynamically adapt to market demand. The human-machine collaboration system combines the advantages of humans and machines, and provides feasibility for implementing different manufacturing tasks. With dynamic adjustment of robots and operators in the production line, the flexibility of the human-machine collaborative production line can be further improved. Therefore, a parallel production line is set up as a parallel community, and the digital twin community model of the intelligent workshop is constructed. The fusion and interaction between the production communities enhance the production flexibility of the manufacturing shop. Aiming at the overall production efficiency and load balancing state, a digital twin-driven intra-community process optimization algorithm based on hierarchical reinforcement learning is proposed, and as a key framework to improve the production performance of production communities, which is used to optimize the proportion of human and machine involvement in work. Finally, taking the assembly process of ventilators as an example, it is proved that the intelligent scheduling strategy proposed in this paper shows stronger adjustment ability in response to dynamic demand as well as production line changes.

Automatic Assessments of Parkinsonian Gait with Wearable Sensors for Human Assistive Systems.

The rehabilitation evaluation of Parkinson's disease has always been the research focus of human assistive systems. It is a research hotspot to objectively and accurately evaluate the gait condition of Parkinson's disease patients, thereby adjusting the actuators of the human-machine system and making rehabilitation robots better adapt to the recovery process of patients. The rehabilitation evaluation of Parkinson's disease has always been the research focus of rehabilitation robots. It is a research hotspot to be able to objectively and accurately evaluate the recovery of Parkinson's disease patients, thereby adjusting the driving module of the human-machine collaboration system in real time, so that rehabilitation robots can better adapt to the recovery process of Parkinson's disease. The gait task in the Unified Parkinson's Disease Rating Scale (UPDRS) is a widely accepted standard for assessing the gait impairments of patients with Parkinson's disease (PD). However, the assessments conducted by neurologists are always subjective and inaccurate, and the results are determined by the neurologists' observation and clinical experience. Thus, in this study, we proposed a novel machine learning-based method of automatically assessing the gait task in UPDRS with wearable sensors as a more convenient and objective alternative means for PD gait assessment. In the design, twelve gait features, including three spatial-temporal features and nine kinematic features, were extracted and calculated from two shank-mounted IMUs. A novel nonlinear model is developed for calculating the score of gait task from the gait features. Twenty-five PD patients and twenty-eight healthy subjects were recruited for validating the proposed method. For comparison purpose, three traditional models, which have been used in previous studies, were also tested by the same dataset. In terms of percentages of participants, 84.9%, 73.6%, 73.6%, and 66.0% of the participants were accurately assigned into the true level with the proposed nonlinear model, the support vector machine model, the naive Bayes model, and the linear regression model, respectively, which indicates that the proposed method has a good performance on calculating the score of the UPDRS gait task and conformance with the rating done by neurologists.

Enabling human–machine collaboration in infrastructure inspections through mixed reality

In this study, we propose a novel end-to-end system called Human–Machine Collaborative Inspection (HMCI) to enable collaboration between inspectors with Mixed Reality (MR) headsets and a robotic data collection platform (robot) for structural inspections. We utilize the MR headset’s holographic display and precise head tracking to allow inspectors to visualize and localize information (e.g., structural defect) on the real scenes, which are gathered by the robot and processed by an offsite computational server. The primary use case of HMCI is to enable the inspector to visualize, supervise, and improve results produced by automated defect detection algorithms in near real-time. The workflow in HMCI starts with collecting images and depth data to generate 3D maps of the site from the robot. A technique called single-shot localization is developed to create visual anchors for real-time spatial alignment between the robot and the MR headset. The 3D map and images are then sent to the computational server for analysis to detect defects and their locations. Then, the information is received by the MR headset and overlaid on the actual scenes to visualize it with spatial context. An experimental study is conducted in a lab environment to demonstrate HMCI using Microsoft HoloLens 2 (HL2) as the MR headset and Turtlebot2 as the robot. We start with the reconstruction of a 3D environment using a 3D depth sensor (Azure Kinect) on Turtlebot2 and visually detect fiducial markers as regions-of-interest (replicating structural damage) along a predefined inspection path. Then, regions-of-interest are successfully anchored to the real scene and visualized through HL2. To our knowledge, HMCI is one of the first human–machine collaborative systems that can integrate robots and inspectors with the MR headset, which has been developed, tested, and presented for structural inspection.

Review on Haptic Assistive Driving Systems Based on Drivers’ Steering-Wheel Operating Behaviour

With the availability of modern assistive techniques, ambient intelligence, and the Internet of Things (IoT), various innovative assistive environments have been developed, such as driving assistance systems (DAS), where the human driver can be provided with physical and emotional assistance. In this human–machine collaboration system, haptic interaction interfaces are commonly employed because they provide drivers with a more manageable way to interact with other components. From the view of control system theory, this is a typical closed-loop feedback control system with a human in the loop. To make such a system work effectively, both the driving behaviour factors, and the electrical–mechanical components should be considered. However, the main challenge is how to deal with the high degree of uncertainties in human behaviour. This paper aims to provide an insightful overview of the relevant work. The impact of various types of haptic assistive driving systems (haptic guidance and warning systems) on driving behaviour performance is discussed and evaluated. In addition, various driving behaviour modelling systems are extensively investigated. Furthermore, the state-of-the-art driving behaviour controllers are analysed and discussed in driver–vehicle–road systems, with potential improvements and drawbacks addressed. Finally, a prospective approach is recommended to design a robust model-free controller that accounts for uncertainties and individual differences in driving styles in a haptic assistive driving system. The outcome indicated that the haptic feedback system applied to the drivers enhanced their driving performance, lowered their response time, and reduced their mental workload compared to a system with auditory or visual signals or without any haptic system, despite some annoyances and system conflicts. The driving behaviour modelling techniques and the driving behaviour control with a haptic feedback system have shown good matching and improved the steering wheel’s base operation performance. However, mathematical principles, a statistical approach, and the lack of consideration of the individual differences in the driver–vehicle–road system make the modelling and the controller less robust and inefficient for different driving styles.

Human–Machine Collaboration for Automated Driving Using an Intelligent Two‐Phase Haptic Interface

Automated Driving In article number 2000229, Chen Lv and co-workers propose a novel human-machine collaboration system based on an intelligent haptic interface for addressing the takeover control problem for automated vehicles. The intelligent haptic torque is applied to the steering wheel and switches its functionality between predictive guidance and haptic assistance according to the varying state and control ability of human drivers. This helps drivers gradually resume manual control during takeover.

Decoding Single-Hand and Both-Hand Movement Directions From Noninvasive Neural Signals.

Decoding human movement parameters from electroencephalograms (EEG) signals is of great value for human-machine collaboration. However, existing studies on hand movement direction decoding concentrate on the decoding of a single-hand movement direction from EEG signals given the opposite hand is maintained still. In practice, the cooperative movement of both hands is common. In this paper, we investigated the neural signatures and decoding of single-hand and both-hand movement directions from EEG signals. The potentials of EEG signals and power sums in the low frequency band of EEG signals from 24 channels were used as decoding features. The linear discriminant analysis (LDA) and support vector machine (SVM) classifiers were used for decoding. Experimental results showed a significant difference in the negative offset maximums of movement-related cortical potentials (MRCPs) at electrode Cz between single-hand and both-hand movements. The recognition accuracies for six-class classification, including two single-hand and four both-hand movement directions, reached 70.29%± 10.85% by using EEG potentials as features with the SVM classifier. These findings showed the feasibility of decoding single-hand and both-hand movement directions. This work can lay a foundation for the future development of an active human-machine collaboration system based on EEG signals and open a new research direction in the field of decoding hand movement parameters from EEG signals.

Digital twin improved via visual question answering for vision-language interactive mode in human–machine collaboration

The human–machine collaboration system is a key means of manufacturing. Its surveillance, prognostic, and health management are related to safety and manufacturing persistence. This paper begins with the mission requirements of intelligent manufacturing. The study is based on the visual question answering (VQA) technology with a digital twin to increase efficiency. The research contents are as follows: (1) A method of modeling human–machine collaboration based on digital twins is proposed. (2) A VQA is adopted in the digital twin. The video and neural language are considered. (3) VQA technology is introduced into the modeling of the human–machine collaboration system for consistent integration. With VQA technology, humans and machines can collaborate. Human–machine interaction and product counting are implemented in a case study to provide a comprehensive perception.

Bio-signal based elbow angle and torque simultaneous prediction during isokinetic contraction

It is of great importance to decode motion dynamics of the human limbs such as the joint angle and torque in order to improve the functionality and provide more intuitive control in human-machine collaborative systems. In order to achieve feasible prediction, both the surface electromyography (sEMG) and A-mode ultrasound were applied to detect muscle deformation and motor intent. Six abled subjects were recruited to perform five trails elbow isokinetic flexion and extension, and each trail contained five repetitions, with muscle deformation and sEMG signals recorded simultaneously. The experimental datasets were categorized as: the ultrasound-EMG combined datasets, ultrasound-only datasets and EMG-only datasets. The support vector machine (SVM) regression model was developed for both elbow joint angle and torque prediction, based on the above three kinds of datasets. The root-mean-square error (RMSE) and the correlation coefficients (R) were applied to evaluate the prediction accuracy. The results across all the subjects for different datasets indicated that the combined datasets and the ultrasound datasets were superior to the sEMG datasets both on elbow joint angle and torque prediction, and there were no significant differences between the combined datasets and the ultrasound datasets. It turns out that elbow angle and torque can be reconstructed by A-mode ultrasound, and the significant findings pave the way towards the application of musculature-driven human-machine collaborative systems.

Haptic-Based Manipulation Scheme of Magnetic Nanoparticles in a Multi-Branch Blood Vessel for Targeted Drug Delivery.

Magnetic drug targeting is a promising technique that can deliver drugs to the diseased region, while keeping the drug away from healthy parts of body. Introducing a human in the control loop of a targeted drug delivery system and using inherent bilateralism of a haptic device at the same time can considerably improve the performance of targeted drug delivery systems. In this paper, we suggest a novel intelligent haptic guidance scheme for steering a number of magnetic nanoparticles (MNPs) using forbidden region virtual fixtures and a haptic rendering scheme with multi particles. Forbidden region virtual fixtures are a general class of guidance modes implemented in software, which help a human-machine collaborative system accomplish a specific task by constraining a movement into limited regions. To examine the effectiveness of our proposed scheme, we implemented a magnetic guided drug delivery system in a virtual environment using a physics-based model of targeted drug delivery including a multi-branch blood vessel and realistic blood dynamics. We performed user studies with different guidance modes: unguided, semi virtual fixture and full virtual fixture modes. We found out that the efficiency of targeting was significantly improved using the forbidden region virtual fixture and the proposed haptic rendering of MNPs. We can expect that using intelligent haptic feedback in real targeted drug delivery systems can improve the targeting efficiency of MNPs in multi-branch vessels.

Intelligent Assembly Technology of Railway Vehicle Bogies

Due to the complexity and highly demand in quality of the bogie assembly of railway vehicles, CRRC Nanjing Puzhen Co., Ltd has developed an intelligent assembly system for railway vehicle bogies. The system can accomplish the intelligent transportation of the bogie parts by the robot on the shelf and the software system developed by ourselves. At the same time, it can also carry out wireless information interaction with the control center and generate process files automatically according to process tasks transmitted by the control center. Besides, a series of key technologies including 3D process documents intelligent push technology, intelligent material box online detection technology, human-machine collaboration technology and system integration control technology have been conquered, which realized the high efficient production and reduced costs of railway vehicle bogie assembly.

Human machine collaborative support scheduling system of intelligence information from multiple unmanned aerial vehicles based on eye tracker

Many human-machine collaborative support scheduling systems are used to aid human decision making by providing several optimal scheduling algorithms that do not take operator’s attention into consideration. However, the current systems should take advantage of the operator’s attention to obtain the optimal solution. In this paper, we innovatively propose a human-machine collaborative support scheduling system of intelligence information from multi-UAVs based on eye-tracker. Firstly, the target recognition algorithm is applied to the images from the multiple unmanned aerial vehicles (multi-UAVs) to recognize the targets in the images. Then, the support system utilizes the eye tracker to gain the eye-gaze points which are intended to obtain the focused targets in the images. Finally, the heuristic scheduling algorithms take both the attributes of targets and the operator’s attention into consideration to obtain the sequence of the images. As the processing time of the images collected by the multi-UAVs is uncertain, however the upper bounds and lower bounds of the processing time are known before. So the processing time of the images is modeled by the interval processing time. The objective of the scheduling problem is to minimize mean weighted completion time. This paper proposes some new polynomial time heuristic scheduling algorithms which firstly schedule the images including the focused targets. We conduct the scheduling experiments under six different distributions. The results indicate that the proposed algorithm is not sensitive to the different distributions of the processing time and has a negligible computational time. The absolute error of the best performing heuristic solution is only about 1%. Then, we incorporate the best performing heuristic algorithm into the human-machine collaborative support systems to verify the performance of the system.

Pattern Classification of Instantaneous Cognitive Task-load Through GMM Clustering, Laplacian Eigenmap, and Ensemble SVMs.

The identification of the temporal variations in human operator cognitive task-load (CTL) is crucial for preventing possible accidents in human-machine collaborative systems. Recent literature has shown that the change of discrete CTL level during human-machine system operations can be objectively recognized using neurophysiological data and supervised learning technique. The objective of this work is to design subject-specific multi-class CTL classifier to reveal the complex unknown relationship between the operator's task performance and neurophysiological features by combining target class labeling, physiological feature reduction and selection, and ensemble classification techniques. The psychophysiological data acquisition experiments were performed under multiple human-machine process control tasks. Four or five target classes of CTL were determined by using a Gaussian mixture model and three human performance variables. By using Laplacian eigenmap, a few salient EEG features were extracted, and heart rates were used as the input features of the CTL classifier. Then, multiple support vector machines were aggregated via majority voting to create an ensemble classifier for recognizing the CTL classes. Finally, the obtained CTL classification results were compared with those of several existing methods. The results showed that the proposed methods are capable of deriving a reasonable number of target classes and low-dimensional optimal EEG features for individual human operator subjects.

어포던스 기반의 인간-기계 협업 모델을 이용한 제조 시스템 구현 연구

Yeong Gwang Oh․Ikchan Ju․Wooyeol Lee․Namhun KimThe Department of Human and Systems Engineering, UNIST(Ulsan National Institute of Science and Technology) Modeling and control of human-involved manufacturing systems poses a huge challenge on how to model all possible interactions among system components within the time and space dimensions. As the manufacturing environment are getting complicated, the importance of human in the manufacturing system is getting more and more spotlighted to incorporate the manufacturing flexibility. This paper presents a formal modeling methodo-logy of affordance-based MPSG (Message-based Part State Graph) for a human-machine collaboration system incorporating supervisory control scheme for flexible manufacturing systems in automotive industry. Basically, we intend to extend the existing model of affordance-based MPSG to the real industrial application of human- machine cooperative environments. The suggested extension with the real industrial example is illustrated in three steps; first, the manufacturing process and relevant data are analyzed in perspectives of MABA-MABA and the supervisory control; second, the manufacturing processes and task allocation between human and machine are mapped onto the concept of MABA-MABA; and the last, the affordance-based MPSG of human- machine collaboration for the manufacturing process is presented with UMLs for verification.

Negotiating Agents

Negotiation is a complex emotional decision-making process aiming to reach an agreement to exchange goods or services. From an agent technological perspective creating negotiating agents that can support humans with their negotiations is an interesting challenge. Already more than a decade, negotiating agents can outperform human beings (in terms of deal optimality) if the negotiation space is well-understood. However, the inherent semantic problem and the emotional issues involved make that negotiation cannot be handled by artificial intelligence alone, and a human-machine collaborative system is required. This article presents research goals, challenges, and an approach to create the next generation of negotiation support agents.

Tuning Trust Using Cognitive Cues for Better Human-Machine Collaboration

In human-machine collaborative systems, human operators usually disuse, misuse, or abuse the assistance from machines. They distrust the systems with poor reliability and may be over-dependent on the highly reliable systems. Human's trust of the assistance of machines currently seems trapped in a dilemma. It prevents achieving harmonious human-machine collaboration. Considering the characteristics of human cognitive process, this paper suggests providing extra cognitive cues to help operators to tune their trust and decision making. The cues are expressed through variable multi-modal human-machine interfaces and serve as intelligent machines' human-like self-confidence, or confidence, in short. To test this approach, a car-following driving experiment was conducted to investigate drivers' perception of unreliable visual/auditory assistance. Experimental results showed that communication of confidence significantly ( α = 0.05) affected human drivers' cognitive processing, especially for decision-making—oriented auditory assistance. Human drivers felt the machines were more trustworthy and useful when they showed variable self-confidence. Meanwhile, a rise in trust did not significantly induce a fall in self-reliance and over-dependence on the machines. Variable self-confidence continuously reminded human drivers that the assistance was not always reliable and they should maintain sufficient awareness of their situation. Using these extra cognitive cues, human operators are more comfortable to tune their trust in real-time situations and to build better human-machine collaboration relationships.

Spatial Motion Constraints Using Virtual Fixtures Generated by Anatomy

This paper describes a spatial-motion-constraints-generation approach for a human-machine collaborative surgical-assistant system from registered computer tomography models. We extend constrained optimization formulation incorporating task goals, anatomy-based constraints, "no fly zones," etc. We use a fast potential-collision-constraint-detection method based on a 3-D surface model and covariance tree data structure. These boundary constraints, along with task behaviors and joint limits, serve as constraint conditions for constrained robot control. We are able to follow a complex path inside a human skull, phantom represented by a surface model composed of 99 000 vertices and 182 000 triangles in real time. Our approach enables real-time task-based control of a surgical robot in a precise interactive minimally invasive surgery task. We illustrate our approach based on two example tasks which are analogous to the procedures in endoscopic sinus surgery, and analyze the user's performance on both teleoperation and cooperative control for one of the example tasks. The experimental results show that a robotic assistant employing our approach on spatial motion constraints can assist the user in skilled manipulation tasks, while maintaining desired properties. Our approach is equally applicable to teleoperative and cooperative controlled robots

MIND MODEL AND SOCIAL INTELLIGENCE IN HUMAN-MACHINE COLLABORATIVE SYSTEMS

Even though social intelligence has not been clearly defined yet, consideration of this new type of intelligence should be important for realizing a new generation of human-machine collaborative systems based on human-centered system design policy. In this article, social intelligence and mind model for implementing socially intelligent agents are studied. At first, emotional and affective aspect of social intelligence is mainly discussed because the aspect is considered as more important in contrast with goal-oriented intelligence. Then, implementation of the mind model and its application as an experiment system is discussed with considering the experiment results.

Special Issue on Intelligent Control in Coming New Generation

In the early 1970s, a concept of intelligent control was proposed by Fu, and since then the advancement of control technologies as a migrate of control theory, artificial intelligence and operations research has been actively attempted. The breakthrough of this concept was to integrate a human judgment and a concept of value as well as management theory into conventional control theoretic approaches, and synthesize these as artificial intelligence. A number of unconventional control techniques have evolved, offering solutions to many difficult control problems in industry and manufacturing. Saridis proposed a general architecture for intelligent control and proposed a design principle of such a hierarchical system as the principle of Increasing Precision with Decreasing Intelligence. During the first generation of intelligent control, a number of intelligent methodologies besides the purely symbolic and logical processing of human knowledge were introduced. They are broadly called soft computing techniques that include artificial neural networks, fuzzy logic, genetic algorithm, and chaos theory. These techniques have contributed much to the advancement of intelligent control from the viewpoint of its ""intelligence"" part, but no solutions are provided from a control theoretic viewpoint, and the definition of intelligence in terms of control theory is still left questionable. To discuss this issue, we initiated a specialist's meeting on survey of intelligent control in 1997 organized under the Institute of Electrical Engineers of Japan, and discussed the current status as well as future perspectives of intelligent control. Some of the papers contributed to this special issue are results obtained in this series of meetings. During that time, the framework of intelligent control has entered the second generation. In the first stage, this framework was discussed in terms of utilized methodologies such as control theory, artificial intelligence, and operations research seeking optimal combinations of these methodologies wherein a distinction is made between the controller, the plant, and the external environment and representations as well as state concepts utilized were a priorily determined and fixed without flexibility. In contrast, the second generation intelligent control system must emphasize a biologically inspired architecture that can accommodate the flexible and dynamic capabilities of living systems including human beings. That is, it must be able to grow and develop increasing capabilities of self-control, self-awareness of representation and reasoning about self and of constructing a coherent whole out of different representations. Actually, a new branch of research on artificial life and system theory of function emergence has shifted the perspectives of intelligence from conventional reductionism to a new design principle based on the concept of ""emergence"". Thus, their approach is quite new in that they attempt to build models that bring together self-organizing mechanisms with evolutionary computation. Such a trend has forced us to reconsider the biological system and/or natural intelligence. In this special issue, we focus on the aspects of semiosis within a multigranular architecture and of emergent properties and techniques for human-machine and/or multiagent collaborative control systems in the coming new generation. These topics are mutually interrelated; the role of multivariable and multiresolutional quantization and clustering for designing intelligent controllers is essential for realizing the abilities to learn unknown multidimensional functions and/or for letting a joint system, which consists of an external environment, a human, and a machine, self-organize distinctive roles in a bottom-up and emerging fashion. This special issue includes papers on proposals of conceptual architecture, methodologies and reports from practical field studies on the hierarchical architecture of machines for realizing hierarchical collaboration and coordination among machine and human autonomies. We believe that these papers will lead to answers to the above questions. We sincerely thank the contributors and reviewers who made this special issue possible. Thanks also go to the editor-in-chief of the Journal of Robotics and Mechatronics, Prof. Makoto Kaneko (Hiroshima University), who provided the opportunity for editing this special issue.