Human-robot Collaboration Framework Research Articles

A Comprehensive Framework for Human-Robot Collaboration in Industrial Environments

A Comprehensive Framework for Human-Robot Collaboration in Industrial Environments

An ethical framework for human-robot collaboration for the future people-centric manufacturing: A collaborative endeavour with European subject-matter experts in ethics

Envisioning humans and (smart) robots collaboratively working on the manufacturing shop floor, sharing spaces, tasks and objectives, reflects the ambitious goal that the ideal factory of the future aspires to attain. However, ensuring the effective implementation of this novel form of labour organisation remains an ongoing area of research. Key aspects such as the future role of workers, potential psychological risks, and the overall ethical considerations of human-robot (H-R) collaboration warrant further investigation until the underpinning safety challenges have been addressed. This study presents a novel ethical framework for H-R collaboration in manufacturing, which involved 30 subject-matter experts in ethics within the European context in a collaborative design process conducted through a year-long three-round data collection qualitative Delphi study. The ethical framework adopts a human-centric approach, recognising the influences that expand beyond the specific context of H-R dynamics on the shop floor, towards organisational and societal governance for a more responsible integration of (smart) robotics into the professional settings. Ethics, in this regard, aims to foster ethical awareness and accountability in the processes and practices of design and innovation, involving all stakeholders who play a role in shaping the future of Industry 5.0.

Multi-scale control and action recognition based human-robot collaboration framework facing new generation intelligent manufacturing

Facing the new generation intelligent manufacturing, traditional manufacturing models are transitioning towards large-scale customized productions, improving the efficiency and flexibility of complex manufacturing processes. This is crucial for enhancing the stability and core competitiveness of the manufacturing industry, and human-robot collaboration systems are an important means to achieve this goal. At present, mainstream manufacturing human-robot collaboration systems are modeled for specific scenarios and actions, with poor scalability and flexibility, making it difficult to flexibly handle actions beyond the set. Therefore, this article proposes a new human-robot collaboration framework based on action recognition and multi-scale control, designs 27 basic gesture actions for motion control, and constructs a robot control instruction library containing 70 different semantics based on these actions. By integrating static gesture recognition, dynamic action process recognition, and You-Only-Look-Once V5 object recognition and positioning technology, accurate recognition of various control actions has been achieved. The recognition accuracy of 27 types of static control actions has reached 100%, and the dynamic action recognition accuracy of the gearbox assembly process based on lightweight MF-AE-NNOBJ has reached 90%. This provides new ideas for simplifying the complexity of human-robot collaboration problems, improving system accuracy, efficiency, and stability.

Augmented Reality and Human–Robot Collaboration Framework for Percutaneous Nephrolithotomy: System Design, Implementation, and Performance Metrics

Augmented Reality and Human–Robot Collaboration Framework for Percutaneous Nephrolithotomy: System Design, Implementation, and Performance Metrics

Contamination Detection Using a Deep Convolutional Neural Network with Safe Machine—Environment Interaction

In the food and medical packaging industries, clean packaging is crucial to both customer satisfaction and hygiene. An operational Quality Assurance Department (QAD) is necessary for detecting contaminated packages. Manual examination becomes tedious and may lead to instances of contamination being missed along the production line. To address this issue, a system for contamination detection is proposed using an enhanced deep convolutional neural network (CNN) in a human–robot collaboration framework. The proposed system utilizes a CNN to identify and classify the presence of contaminants on product surfaces. A dataset is generated, and augmentation methods are applied to the dataset for nine classes such as coffee, spot, chocolate, tomato paste, jam, cream, conditioner, shaving cream, and toothpaste contaminants. The experiment was conducted using a mechatronic platform with a camera for contamination detection and a time-of-flight sensor for safe machine–environment interaction. The results of the experiment indicate that the reported system can accurately identify contamination with 99.74% mean average precision (mAP).

Human–Robot Collaboration Framework Based on Impedance Control in Robotic Assembly

Human–robot (HR) collaboration (HRC) is an emerging research field because of the complementary advantages of humans and robots. An HRC framework for robotic assembly based on impedance control is proposed in this paper. In the HRC framework, the human is the decision maker, the robot acts as the executor, while the assembly environment provides constraints. The robot is the main executor to perform the assembly action, which has the position control, drag and drop, positive impedance control, and negative impedance control modes. To reveal the characteristics of the HRC framework, the switch condition map of different control modes and the stability analysis of the HR coupled system are discussed. In the end, HRC assembly experiments are conducted, where the HRC assembly task can be accomplished when the assembling tolerance is 0.08 mm or with the interference fit. Experiments show that the HRC assembly has the complementary advantages of humans and robots and is efficient in finishing complex assembly tasks.

Deep Learning Framework for Controlling Work Sequence in Collaborative Human–Robot Assembly Processes

The human-robot collaboration (HRC) solutions presented so far have the disadvantage that the interaction between humans and robots is based on the human's state or on specific gestures purposely performed by the human, thus increasing the time required to perform a task and slowing down the pace of human labor, making such solutions uninteresting. In this study, a different concept of the HRC system is introduced, consisting of an HRC framework for managing assembly processes that are executed simultaneously or individually by humans and robots. This HRC framework based on deep learning models uses only one type of data, RGB camera data, to make predictions about the collaborative workspace and human action, and consequently manage the assembly process. To validate the HRC framework, an industrial HRC demonstrator was built to assemble a mechanical component. Four different HRC frameworks were created based on the convolutional neural network (CNN) model structures: Faster R-CNN ResNet-50 and ResNet-101, YOLOv2 and YOLOv3. The HRC framework with YOLOv3 structure showed the best performance, showing a mean average performance of 72.26% and allowed the HRC industrial demonstrator to successfully complete all assembly tasks within a desired time window. The HRC framework has proven effective for industrial assembly applications.

Uncertainty-propagated Cartesian coordinated human–robot collaboration on Riemannian manifold with hidden state-space model

This paper proposes a novel Cartesian coordinated human–robot collaboration framework derived from the hidden state-space models, which are established on the behaviour cloning of both the human and robot in a nonparametric form by using dual quaternion on Riemannian manifold. Based on the Taylor linearisation, this framework could provide an analytical approximation of the posterior distribution and hence infer six DoF Cartesian hidden state variables of the collaborative manipulator given human observation and its uncertainties. As the full Cartesian pose (special Euclidean group, SE(3)) includes translation and rotation (special orthogonal group, SO(3)), which is not Euclidean, directly encoding the Cartesian motions with nonparametric regression in Euclidean space could cause significant errors. This paper addresses this issue by defining both human and robot behaviours using modified quaternion and cloning them in the tangent space of the Riemannian manifold. Not akin to other coordinated human–robot collaboration methods, the collaboration framework not only preserves the adaptation functionalities but also propagates full Cartesian state variables and their uncertainties during real-time coordinated collaboration implementation. Leveraging on the Gaussian mixture model on the Riemannian manifold, the multiple task recognition is further addressed to enhance the generalisation capability of the framework presented. The application feasibility is demonstrated in both theoretical comparison simulation and experiments.

A Framework and Algorithm for Human-Robot Collaboration Based on Multimodal Reinforcement Learning.

Despite the emergence of various human-robot collaboration frameworks, most are not sufficiently flexible to adapt to users with different habits. In this article, a Multimodal Reinforcement Learning Human-Robot Collaboration (MRLC) framework is proposed. It integrates reinforcement learning into human-robot collaboration and continuously adapts to the user's habits in the process of collaboration with the user to achieve the effect of human-robot cointegration. With the user's multimodal features as states, the MRLC framework collects the user's speech through natural language processing and employs it to determine the reward of the actions made by the robot. Our experiments demonstrate that the MRLC framework can adapt to the user's habits after repeated learning and better understand the user's intention compared to traditional solutions.

Imitation learning for coordinated human–robot collaboration based on hidden state-space models

This paper proposes a novel coordinated human–robot collaboration framework based on the hidden state-space model, which probabilistically clones the human behaviour and presents dynamic features in a nonparametric form. Derived from the filter prediction techniques and the theory of exact moment matching, this framework could provide an analytical approximation of the posterior distribution, and hence infer the hidden state variables of the collaborative robot given the external observation and its uncertainties. Not akin to the other cutting-edge movement-primitive based algorithms or coordinated human–robot collaboration methods, our collaboration framework not only preserves the adaptation functionalities of imitation learning but also propagates state variables and their uncertainties during real-time coordinated implementation. By leveraging on the binary Gaussian process classification, additional functionality, such as multiple task recognition is proposed to enhance the generalisation capability of our framework. The application feasibility is verified from both theoretical comparison simulation and real-world experiments.

A Human-Robot Collaboration Framework for Improving Ergonomics During Dexterous Operation of Power Tools

A Human-Robot Collaboration Framework for Improving Ergonomics During Dexterous Operation of Power Tools

Semantic knowledge based reasoning framework for human robot collaboration

Human robot collaborative assemblies, where humans and robots work together, are becoming the new frontier in industrial robotics. However, the aspect of humans to easily and intuitively interact/collaborate with a robot, especially for teaching new tasks, is still open. In this work, we present a semantic knowledge-based-reasoning framework that first learns to recognize human activities using perception data (action recognition and object tracking) and semantically links them to an assembly process. Thus deriving the intention of the human interaction in the teaching process. The framework then extracts relevant parameters for the robot action execution using an interactive skill-based programming approach. To resolve ambiguities during the teaching process, the reasoning framework initiates an interaction at an (semantically) abstract level with the user by exploiting previous knowledge and the current environmental setup. The reasoning framework is demonstrated in two different application scenarios involving human-robot collaborative teaching and a user guidance system respectively.

A Human-Robot Collaboration Framework for Improving Ergonomics During Dexterous Operation of Power Tools

• A human-robot collaboration takes into account ergonomic aspects of the human co-worker during power tool operations. • The human overloading joint torques are estimated and monitored online using a whole-body dynamic state model. • The collaborative robot motion that brings the human into the suitable ergonomic working configuration is obtained by an optimisation method to minimise the overloading joint torques. • The human arm muscular manipulability and safety of the collaborative task are used in the optimisation process as the constraints to achieve a task-relevant optimised configuration. • An ergonomic human-robot collaboration framework enable to provide not only the co-worker’s lower joints overloading effect but also high manipulability capacity. In this work, we present a novel control approach to human-robot collaboration that takes into account ergonomic aspects of the human co-worker during power tool operations. The method is primarily based on estimating and reducing the overloading torques in the human joints that are induced by the manipulated external load. The human overloading joint torques are estimated and monitored using a whole-body dynamic state model. The appropriate robot motion that brings the human into the suitable ergonomic working configuration is obtained by an optimisation method that minimises the overloading joint torques. The proposed optimisation process includes several constraints, such as the human arm muscular manipulability and safety of the collaborative task, to achieve a task-relevant optimised configuration. We validated the proposed method by a user study that involved a human-robot collaboration task, where the subjects operated a polishing machine on a part that was brought to them by the collaborative robot. A statistical analysis of ten subjects as an experimental evaluation of the proposed control framework is provided to demonstrate the potential of the proposed control framework in enabling ergonomic and task-optimised human-robot collaboration.

Towards Efficient Human-Robot Collaboration With Robust Plan Recognition and Trajectory Prediction

Human-robot collaboration (HRC) is becoming increasingly important as the paradigm of manufacturing is shifting from mass production to mass customization. The introduction of HRC can significantly improve the flexibility and intelligence of automation. To efficiently finish tasks in HRC systems, the robots need to not only predict the future movements of human, but also more high-level plans, i.e., the sequence of actions to finish the tasks. However, due to the stochastic and time-varying nature of human collaborators, it is quite challenging for the robot to efficiently and accurately identify such task plans and respond in a safe manner. To address this challenge, we propose an integrated human-robot collaboration framework. Both plan recognition and trajectory prediction modules are included for the generation of safe and efficient robotic motions. Such a framework enables the robots to perceive, predict and adapt their actions to the human's work plan and intelligently avoid collisions with the human. Moreover, by explicitly leveraging the hierarchical relationship between plans and trajectories, more robust plan recognition performance can be achieved. Physical experiments were conducted on an industrial robot to verify the proposed framework. The results show that the proposed framework could accurately recognize the human workers' plans and thus significantly improve the time efficiency of the HRC team even in the presence of motion classification noises.

Integral Sliding-Mode Observer-Based Disturbance Estimation for Euler–Lagrangian Systems

In this article, a novel integral sliding-mode observer is proposed to estimate the external disturbance and velocity of Euler-Lagrangian systems. This method provides high bandwidth and precise estimation with only commanded input and position measurement. A system velocity measurement is not required to construct the sliding-mode manifold. The convergence of the estimation error to zero is theoretically in finite time, which is proven by a direct Lyapunov method utilizing the passivity property of Euler-Lagrangian systems. An integral sliding manifold is designed to reduce the reaching phase, such that the robustness of the estimation is enhanced. The method has been applied to a robot manipulator to estimate the joint velocity and external contact forces in a physical human-robot task. Simulations and experiments reveal that this novel method provides fast, precise, and robust estimation results and can be used to replace the measurement of an external force sensor. The successful application of this observer to a force-sensor-less admittance controller for a manipulator contributes to the implementation of a sensor-free safety framework for human-robot collaboration (HRC).

A Deadband-Based Method for User Effort Reduction in Human-Robot Shared Control

In a human-robot collaboration framework, the authority over the execution of tasks can be shared between the human user and the automatic controller. In case of disagreement, the human user may exert a significant amount of effort opposing the robotic counterpart. Therefore, a prompt assessment of user intent is of the utmost importance in shared systems. In this paper a novel strategy, inspired by anti-windup techniques, is proposed to adjust the component of automatic control according to the sensed human effort. A haptic human-robot interface is developed and the method is validated in a shared control application in which both the human user and the automatic controller can steer a simulated ground vehicle. Evidence is shown that the proposed method can reduce the required human effort, with performance comparable to the state-of-the-art.

Performance guaranteed human-robot collaboration with POMDP supervisory control

Abstract Human-Robot Collaboration (HRC) studies how to achieve effective collaborations between human and robots to take advantage of the flexibility from human and the autonomy from robots. Many applications involving HRC, such as joint assembly manufacturing systems and advanced driver assistance systems, need to achieve high-level tasks in a provably correct manner. These applications motivate the requirements of HRC to have the performance guarantee to assure the task completion and safety of both human and robots. In this paper, a correct-by-design HRC framework is built to enable a performance guaranteed HRC. To model the uncertainties from human, robots and the environment, partially observable Markov decision process (POMDP) is used as the model. Based on the POMDP modeling, a supervisory control framework is applied and designed to be adaptive to modeling uncertainties. To reduce the model checking complexity involved in the supervisor synthesis process, an abstraction method for POMDP is integrated to find a quotient system with a smaller size of state space. Based on the abstraction method, a verification adaptation technique is developed with simulation relation checking algorithms to deal with possible online model changing. If the verification adaptation indicates the necessity to update the supervisor, supervisor adjustment methods are given. Altogether, it leads to a semi-online adaptation approach for system model changing. Examples are given to illustrate this framework.

ACT-R-typed human–robot collaboration mechanism for elderly and disabled assistance

SUMMARYThis work aims to propose an innovative mechanism of human–robot collaboration (HRC) for mobile service robots in the application of elderly and disabled assistance. Previous studies on HRC mechanism usually focused on integrating decision-making intelligence of human beings by qualitative judgment and reasoning intelligence of robots by quantitative calculation. Instead, novelties of the proposed methodology include (1) constructing an HRC framework by taking reference from the Adaptive Control of Thought – Rational (ACT-R) human cognitive architecture; (2) establishing semantic webs of cognitive reasoning through human–robot interaction (HRI) and HRC to plan and implement complex tasks; and (3) realizing human–robot intelligence fusion by mutual encouragement, connect, and integration of modules of human, robot, perception, HRI, and HRC in the ACT-R architecture. Its technical feasibility is validated by some selected experiments within a “pouring” scenario. Further, although this study is oriented to mobile service robots, the modularized design of hardware and software makes its extensive use feasible in other types of service robots like smart rehabilitation beds, wheelchairs, and cleaning equipments.