Human-robot Collaboration Research Articles

Efficient human activity recognition: A deep convolutional transformer-based contrastive self-supervised approach using wearable sensors

Artificial intelligence has advanced the applications of sensor-based human motion capture and recognition technology in various engineering fields, such as human–robot collaboration and health monitoring. Deep learning methods can achieve satisfactory recognition results when provided with sufficient labeled data. However, labeling a large dataset is expensive and time-consuming in practical applications. To address this issue, this paper proposes a deep convolutional transformer-based contrastive self-supervised (DCTCSS) model under the bootstrap your own latent (BYOL) framework. The DCTCSS model aims to achieve reliable activity recognition using only a small amount of labeled data. Firstly, a deep convolutional transformer (DCT) model is proposed as the backbone of DCTCSS model, to learn high-level feature representations from unlabeled data in pre-training period. Subsequently, a simple linear classifier is trained with supervised fine-tuning using a limited amount of labeled data to recognize activities. In addition, this paper experimentally formulates a random data augmentation strategy to increase the diversity of input data. The performance of the DCTCSS model is evaluated and compared with several state-of-the-art algorithms on three datasets widely used in daily life, medical monitoring, and intelligent manufacturing. Experimental results show that the DCTCSS model achieves mean F1 scores of 95.64%, 88.39%, and 98.40% on the UCI-HAR, Skoda, and Mhealth datasets, respectively, using only 10% of the labeled data. These results outperform both supervised and unsupervised baseline models. Consequently, the DCTCSS model demonstrates its effectiveness in reducing the dependence on large amounts of labeled data while still achieving competitive activity recognition performance.

Regressionsbasierte Ermittlung von Ausführungszeiten in Mensch-Roboter-Kollaboration

Abstract The intensifying competitive pressures in manufacturing companies, combined with the expanding array of product variants, are propelling the integration of human-robot collaboration in operational frameworks. Effectively planning assembly processes demands particularly streamlined methods for assessing execution times, especially in the initial planning phases. While predetermined motion time systems are designed for human tasks, this contribution puts forth an approach for the time-economic analysis of robot movements without resorting to simulation.

Long horizon episodic decision making for cognitively inspired robots

The Human decision-making process works by recollecting past sequences of observations and using them to decide the best possible action in the present. These past sequences of observations are stored in a derived form which only includes important information the brain thinks might be useful in the future, while forgetting the rest. we propose an architecture that tries to mimic the human brain and improve the memory efficiency of transformers by using a modified TransformerXL architecture which uses Automatic Chunking which only attends to the relevant chunks in the transformer block. On top of this, we use ForgetSpan which is technique to remove memories that do not contribute to learning. We also theorize the technique of Similarity based forgetting to remove repetitive memories. We test our model in various tasks that test the abilities required to perform well in a human–robot collaboration scenario.

Learning from demonstration for 7-DOF anthropomorphic manipulators without offset via analytical inverse kinematics

Learning from demonstration (LfD) has been widely studied as a convenient method for robot learning. In the LfD paradigm for redundant manipulators, the reproduced trajectories ought to be similar to human demonstrations in both task and joint space. Despite many advancements achieved in the context of learning in task space, the solutions for generating similar reproductions in joint space are still largely open. In this paper, a novel non-parametric LfD framework for 7-DOF anthropomorphic manipulators with high computational efficiency is proposed. The proposed method leverages redundancy resolution and kernel-based approaches to formulate an efficient model characterized by a limited set of open parameters. Experiments were conducted to evaluate the performance of the proposed method and compare it with the commonly used ‘LfD+IK’ solution. The results indicated that the proposed method behaves much better in terms of the similarity between the demonstration and reproduction with high computing efficiency. As the proposed method can learn from human demonstrations effectively in both task and joint space, it has the potential to significantly enhance human–robot collaboration, streamline assembly line processes, or improving robot learning. An important future challenge will be extending the proposed method for general-purpose redundant manipulators and considering task constraints to perform complex tasks.

Socially adaptive cognitive architecture for human-robot collaboration in industrial settings.

This paper introduces DAC-HRC, a novel cognitive architecture designed to optimize human-robot collaboration (HRC) in industrial settings, particularly within the context of Industry 4.0. The architecture is grounded in the Distributed Adaptive Control theory and the principles of joint intentionality and interdependence, which are key to effective HRC. Joint intentionality refers to the shared goals and mutual understanding between a human and a robot, while interdependence emphasizes the reliance on each other's capabilities to complete tasks. DAC-HRC is applied to a hybrid recycling plant for the disassembly and recycling of Waste Electrical and Electronic Equipment (WEEE) devices. The architecture incorporates several cognitive modules operating at different timescales and abstraction levels, fostering adaptive collaboration that is personalized to each human user. The effectiveness of DAC-HRC is demonstrated through several pilot studies, showcasing functionalities such as turn-taking interaction, personalized error-handling mechanisms, adaptive safety measures, and gesture-based communication. These features enhance human-robot collaboration in the recycling plant by promoting real-time robot adaptation to human needs and preferences. The DAC-HRC architecture aims to contribute to the development of a new HRC paradigm by paving the way for more seamless and efficient collaboration in Industry 4.0 by relying on socially adept cognitive architectures.

A framework for neurosymbolic robot action planning using large language models.

Symbolic task planning is a widely used approach to enforce robot autonomy due to its ease of understanding and deployment in engineered robot architectures. However, techniques for symbolic task planning are difficult to scale in real-world, highly dynamic, human-robot collaboration scenarios because of the poor performance in planning domains where action effects may not be immediate, or when frequent re-planning is needed due to changed circumstances in the robot workspace. The validity of plans in the long term, plan length, and planning time could hinder the robot's efficiency and negatively affect the overall human-robot interaction's fluency. We present a framework, which we refer to as Teriyaki, specifically aimed at bridging the gap between symbolic task planning and machine learning approaches. The rationale is training Large Language Models (LLMs), namely GPT-3, into a neurosymbolic task planner compatible with the Planning Domain Definition Language (PDDL), and then leveraging its generative capabilities to overcome a number of limitations inherent to symbolic task planners. Potential benefits include (i) a better scalability in so far as the planning domain complexity increases, since LLMs' response time linearly scales with the combined length of the input and the output, instead of super-linearly as in the case of symbolic task planners, and (ii) the ability to synthesize a plan action-by-action instead of end-to-end, and to make each action available for execution as soon as it is generated instead of waiting for the whole plan to be available, which in turn enables concurrent planning and execution. In the past year, significant efforts have been devoted by the research community to evaluate the overall cognitive capabilities of LLMs, with alternate successes. Instead, with Teriyaki we aim to providing an overall planning performance comparable to traditional planners in specific planning domains, while leveraging LLMs capabilities in other metrics, specifically those related to their short- and mid-term generative capabilities, which are used to build a look-ahead predictive planning model. Preliminary results in selected domains show that our method can: (i) solve 95.5% of problems in a test data set of 1,000 samples; (ii) produce plans up to 13.5% shorter than a traditional symbolic planner; (iii) reduce average overall waiting times for a plan availability by up to 61.4%.

Research on the Human–Robot Collaborative Disassembly Line Balancing of Spent Lithium Batteries with a Human Factor Load

The disassembly of spent lithium batteries is a prerequisite for efficient product recycling, the first link in remanufacturing, and its operational form has gradually changed from traditional manual disassembly to robot-assisted human–robot cooperative disassembly. Robots exhibit robust load-bearing capacity and perform stable repetitive tasks, while humans possess subjective experiences and tacit knowledge. It makes the disassembly activity more adaptable and ergonomic. However, existing human–robot collaborative disassembly studies have neglected to account for time-varying human conditions, such as safety, cognitive behavior, workload, and human pose shifts. Firstly, in order to overcome the limitations of existing research, we propose a model for balancing human–robot collaborative disassembly lines that take into consideration the load factor related to human involvement. This entails the development of a multi-objective mathematical model aimed at minimizing both the cycle time of the disassembly line and its associated costs while also aiming to reduce the integrated smoothing exponent. Secondly, we propose a modified multi-objective fruit fly optimization algorithm. The proposed algorithm combines chaos theory and the global cooperation mechanism to improve the performance of the algorithm. We add Gaussian mutation and crowding distance to efficiently solve the discrete optimization problem. Finally, we demonstrate the effectiveness and sensitivity of the improved multi-objective fruit fly optimization algorithm by solving and analyzing an example of Mercedes battery pack disassembly.

A Survey of Wearable Lower Extremity Neurorehabilitation Exoskeleton: Sensing, Gait Dynamics, and Human–Robot Collaboration

A Survey of Wearable Lower Extremity Neurorehabilitation Exoskeleton: Sensing, Gait Dynamics, and Human–Robot Collaboration

Placement Optimization of Flexible Proximity Sensors for Human-Robot Collaboration

Placement Optimization of Flexible Proximity Sensors for Human-Robot Collaboration

Learning to Communicate Functional States With Nonverbal Expressions for Improved Human-Robot Collaboration

Learning to Communicate Functional States With Nonverbal Expressions for Improved Human-Robot Collaboration

Impact of AI-Driven Digital Twins in Industry 4.0: an Exploratory Analysis

Human society is witnessing a revolutionary growth of digital twin (DT) and artificial intelligence (AI) technologies, which has greater impact on Industry 4.0 revolution specially in academia and industry. DT is a digital representation of a physical entity, with data and infrastructure serving as its foundation, algorithms, and models as its core, and software and services as its application. The methodical and thorough integration of domain-specific expertise is even more essential to the foundations of DT and AI in industrial sectors. This paper provides a thorough analysis of more than 30 articles on AI-driven DT technologies employed in Industry 4.0 over the previous five years. It also describes the general advances of these technologies and the current status of AI integration in the domains of advanced robotics and smart manufacturing which are affecting human society. These include established methods like industrial automation as well as complex mechanism like 3D printing and human-robot collaboration. Additionally, the benefits of AI-powered DTs are explained in relation to sustainable development. The development potential and practical difficulties of AI-driven DTs are examined, with varying emphasis on various levels.

Evaluation of Biomechanical and Mental Workload During Human-Robot Collaborative Pollination Task.

The purpose of this study is to identify the potential biomechanical and cognitive workload effects induced by human robot collaborative pollination task, how additional cues and reliability of the robot influence these effects and whether interacting with the robot influences the participant's anxiety and attitude towards robots. Human-Robot Collaboration (HRC) could be used to alleviate pollinator shortages and robot performance issues. However, the effects of HRC for this setting have not been investigated. Sixteen participants were recruited. Four HRC modes, no cue, with cue, unreliable, and manual control were included. Three categories of dependent variables were measured: (1) spine kinematics (L5/S1, L1/T12, and T1/C7), (2) pupillary activation data, and (3) subjective measures such as perceived workload, robot-related anxiety, and negative attitudes towards robotics. HRC reduced anxiety towards the cobot, decreased joint angles and angular velocity for the L5/S1 and L1/T12 joints, and reduced pupil dilation, with the "with cue" mode producing the lowest values. However, unreliability was detrimental to these gains. In addition, HRC resulted in a higher flexion angle for the neck (i.e., T1/C7). HRC reduced the physical and mental workload during the simulated pollination task. Benefits of the additional cue were minimal compared to no cues. The increased joint angle in the neck and unreliability affecting lower and mid back joint angles and workload requires further investigation. These findings could be used to inform design decisions for HRC frameworks for agricultural applications that are cognizant of the different effects induced by HRC.

A vision-language-guided robotic action planning approach for ambiguity mitigation in human–robot collaborative manufacturing

Human–robot collaboration (HRC) has been recognized as a potent pathway towards mass personalization in the manufacturing sector, by leveraging the synergy of human creativity and robotic precision. Previous approaches rely heavily on visual perception to autonomously comprehend the HRC environment. However, the inherent ambiguity in human–robot communication cannot be consistently neutralized by relying solely on visual cues. With the recently soaring popularity of large language models (LLMs), the consideration of language data as a complementary information source has increasingly drawn research attention, while the application of such large models, particularly within the context of HRC in manufacturing, remains largely under-explored. In response to this gap, a vision-language reasoning approach is proposed to mitigate the communication ambiguity prevalent in human–robot collaborative manufacturing scenarios. A referred object retrieval model is first designed to alleviate the object–reference ambiguity in the human language command. This model is then seamlessly integrated into an LLM-based robotic action planner to achieve an improved HRC performance. The effectiveness of the proposed approach is demonstrated empirically through a series of experiments conducted on the object retrieval model and its application in a human–robot collaborative assembly case.

Assessing the Suitability of Automation Using the Methods–Time–Measurement Basic System

Due to its high complexity and the varied assembly processes, hybrid assembly systems characterized by human–robot collaboration (HRC) are meaningful. Suitable use cases must be identified efficiently to ensure cost-effectiveness and successful deployment in the respective assembly systems. This paper presents a method for evaluating the potential of HRC to derive automation suitability based on existing or to-be-collected time data. This should enable a quick and favorable statement to be made about processes, for efficient application in potential analyses. The method is based on the Methods–Time–Measurement Basic System (MTM-1) procedure, widely used in the industry. This ensures good adaptability in an industrial context. It extends existing models and examines how much assembly activities and processes can be optimized by efficiently allocating between humans and robots. In the process model, the assembly processes are subdivided and analyzed with the help of the specified MTM motion time system. The suitability of the individual activities and sub-processes for automation are evaluated based on criteria derived from existing methods. Two four-field matrices were used to interpret and classify the analysis results. The process is assessed using an example product from electrolyzer production, which is currently mainly assembled by hand. To achieve high statement reliability, further work is required to classify the results comprehensively.

Human-robot collaboration assembly personalised safety detection method based on digital twin

Human-robot collaboration (HRC) in assembly is becoming increasingly popular in the field of intelligent manufacturing. However, safety concerns remain the most critical issue to be addressed. The complex assembly environment and the individual differences of workers make it challenging to ensure safety effectively. Furthermore, the concept of human-centric manufacturing also places higher requirements on safety and efficiency of HRC assembly. The application of digital twin (DT) technology has bolstered the cognitive and decision-making capabilities in the context of HRC assembly. This technological advancement provides essential support for addressing safety concerns in HRC assembly processes guided by a human-centric approach. A personalised safety detection method for HRC assembly based on DT technology is proposed. This approach involves personalised modelling and simulation of real objects within the HRC process, establishing a collision detection model based on skeleton points. Subsequently, human motion prediction methods drive safety monitoring between humans and robots. Ultimately, personalised safety warning strategies are applied to ensure the safety of the assembly process.

Advancing human-robot collaboration: Predicting operator trajectories through AI and infrared imaging

In industrial applications demanding close interaction between humans and robots, accurate perception of the robot's surroundings is vital to adjust its behavior appropriately. This need becomes especially pertinent when manipulators operate autonomously and execute both separate and common tasks with human operators. To ensure operator safety in such Human-Robot Collaboration (HRC) environments, the present paper proposes an innovative approach that leverages advanced collaborative robotic systems capable of sensing the operator's position to effectively prevent or mitigate potential contact. Our solution combines infrared-thermal imaging technology with a workspace monitoring system, utilizing thermal and depth cameras, to track the operator's current location and predict their future trajectory using a multi-layer Long Short-Term Memory (LSTM) neural network. We also introduce a technique for simulating both human and robot movements to generate a synthetic dataset. This data is shared with the motion planning system to generate trajectories that do not intersect with the operator's future paths, minimizing downtime due to safety stoppages. The solution is integrated and evaluated within an industrial high payload collaborative robot context, and the results have shown improvements in cycle time efficiency and operator approval. This approach offers a promising direction for enhancing safety and productivity in HRC workspaces.

How does the status of errant robot affect our desire for contact? – The moderating effect of team interdependence

Technological breakthroughs such as artificial intelligence and sensors make human-robot collaboration a reality. Robots with highly reliable, specialised skills gain informal status in collaborative teams, but factors such as unstructured work environments and task requirements make robot error inevitable. So how do status differences of errant robots affect the desire for contact, and do team characteristics also have an impact? This paper describes an intergroup experiment using the Experimental Vignette Method (EVM), based on the Expectation Violation Theory, 214 subjects were invited to test the following hypotheses: (1) Errant robot status has an influence on employees' desire for contact and support for robotics research through negative emotions; (2) Team interdependence is a boundary condition for the effect of errant robot status on negative emotions. This paper contributes to the literature on employee reactions to robot errors in human-robot collaboration and provides suggestions for robot status design.

Balancing parallel assembly lines with human-robot collaboration: problem definition, mathematical model and tabu search approach

Human-robot collaboration (HRC), as an emerging production mode, has garnered significant attention in the context of the growing smart manufacturing. This study aims to investigate the application of human-robot collaboration in parallel assembly lines and explore its potential for improving productivity, resource utilisation and flexibility. To the best of our knowledge, this is one of the first attempts to consider the combination of parallel assembly line balancing problem (PALBP) and human-robot collaboration. In this study, the parallel assembly line balancing problem with human-robot collaboration (PALBP-HRC) is introduced and characterised. The collaboration mode allows human workers and robots to execute tasks both in parallel and in collaboration. A mixed-integer programming model (MIP) that minimises the cycle time is formulated and a lower bound is proposed. Due to the NP-hardness, we develop an adapted tabu search (TS) algorithm specifically tailored for problem characterisation. Experimental results indicate that the proposed TS algorithm achieves competing performance compared to the MIP and other algorithms, including genetic algorithm (GA) and simulated annealing (SA) algorithm. The comparison between PALBP and PALBP-HRC demonstrates that the integration of human-robot collaboration to parallel assembly line can achieve significant improvements in efficiency and flexibility.

A new XR-based human‐robot collaboration assembly system based on industrial metaverse

The industrial metaverse is an emerging paradigm of integrated information technologies. A human-robot collaboration assembly system is developed using extended reality (XR) and blockchain technologies. This system includes the physical, virtual reality (VR), augmented reality (AR) supporting, and blockchain systems. Virtual avatars represent physical operators and mobile robots in the XR environment. The VR system emulates real-time physical operations, facilitating pre-planning, online task optimization, and converting strategic decisions into executable instructions for physical robots. The AR system enhances the cognitive experience of intelligent operators, contributing to decision-making and VR system control through multi-modal interactions. The integration of the VR and AR systems in perception-decision-control enables real-time process optimization and the reverse control of physical operations. The system is implemented using a decentralized blockchain and end-edge-cloud collaborative network architecture to regulate interaction permissions for each scene characters, ensuring systematic functionality, robust communication, and comprehensive data management. A case study of a gearbox assembly is used to demonstrate the system’s development and efficacy. The results indicate the VR system's comprehensive perception and the AR system's detailed insights, yielding reliable information on global and local levels. The integration of artificial intelligence (AI) algorithms and spatiotemporal visualization during VR decision-making provides computational advantages. Incorporating human input into the AR system enables human-in-the-loop operational decision-making, providing optimal instructions to improve human-robot collaboration performance. Notably, the fusion of blockchain and end-edge-cloud architectures augments inter-system data exchange by efficiently managing and processing human-robot interactions and their diverse avatars. This system offers an innovative approach to human-robot collaboration assembly in manufacturing, with implications for analogous contexts.

Conformal, stretchable, breathable, wireless epidermal surface electromyography sensor system for hand gesture recognition and rehabilitation of stroke hand function

Surface electromyography (sEMG) plays a significant role in the everyday practice of clinic hand function rehabilitation. The materials and design of current typical clinic sEMG electrodes are rigid Ag/AgCl or flexible polyimide (PI) film, which cannot provide a stable interface between electrodes and skin for adequate long-term high-quality data. Thus, conformal, soft, breathable, wireless epidermal sEMG sensor systems have broad potential relevance to clinic rehabilitation settings. Herein, we demonstrate a stretchable epidermal sEMG sensor array system with optimized materials and structure strategies for hand gesture recognition and hand function rehabilitation. The optimized serpentine structures with marvelous stretchability and improved fill ratio, provide lower impedance and high-quality sEMG signals. Moreover, the easy-to-use airhole method further ensures stable and long-term contact with the skin for recording. In addition, integrated with a customized flexible wireless data acquisition system, the capability for real-time 8-channel sEMG monitoring is developed, and taking together with the CNN-based algorithm, the system can automatically and reliably realize the 7 kinds of hand gestures with an accuracy of 81.02%. Moreover, the low-cost yet high-performance epidermal sEMG sensor system demonstrated its conceptual feasibility in quantitatively evaluation of stroke patient’s hand and facilitating human-robot collaboration in hand rehabilitation by proof-of-the-concept clinical testing.