Human-robot Environment Research Articles

Implementations of Digital Transformation and Digital Twins: Exploring the Factory of the Future

In the era of rapid technological advancement and evolving industrial landscapes, embracing the concept of the factory of the future (FoF) is crucial for companies seeking to optimize efficiency, enhance productivity, and stay sustainable. This case study explores the concept of the FoF and its role in driving the energy transition and digital transformation within the automotive sector. By embracing advancements in technology and innovation, these factories aim to establish a smart, sustainable, inclusive, and resilient growth framework. The shift towards hybrid and electric vehicles necessitates significant adjustments in vehicle components and production processes. To achieve this, the adoption of lighter materials becomes imperative, and new technologies such as additive manufacturing (AM) and artificial intelligence (AI) are being adopted, facilitating enhanced efficiency and innovation within the factory environment. An important aspect of this paradigm involves the development and utilization of a modular, affordable, safe human–robot interaction and highly performant intelligent robot. The introduction of this intelligent robot aims to foster a higher degree of automation and efficiency through collaborative human–robot environments on the factory floor and production lines, specifically tailored to the automotive sector. By combining the strengths of human and robotic capabilities, the future factory aims to revolutionize manufacturing processes, ultimately driving the automotive industry towards a more sustainable and technologically advanced future. This study explores the implementation of automation and the initial strides toward transitioning from Industry 4.0 to 5.0, focusing on three recognized, large, and automotive companies operating in the north of Portugal.

Graph-based meta-learning for context-aware sensor management in nonlinear safety-critical environments

This study introduces a novel framework for optimizing energy efficiency and computational load in safety-critical robotic systems operating in nonlinear domains. Leveraging Graph Attention Networks for state awareness and decision-making, the framework employs adaptive sensor and filter toggling strategies to dynamically manage system resources through real-time inferential processes. Our framework maintains continuous robot operation in the presence of sensor noise and environmental disturbances by activating additional sensors, thus preventing system shutdowns or stalls. Few-shot meta-learning techniques further augment the model's adaptability, allowing it to generalize and make real-time decisions across varying operational conditions. An extensive evaluation reveals a reduced average energy consumption, compared to ‘always-on’ configurations, by 13.71% and CPU utilization by 29.07%, without compromising system performance and safety. We also introduce Matching Networks and Siamese Networks with different loss functions to assess the system's capability to adapt to different levels of criticality. Our experiments demonstrate that the system prioritizes performance and safety in high-critical scenarios while maximizing energy efficiency in less critical situations. The framework's real-time decision-making capability is particularly crucial in human–robot environments and holds significant implications for future applications in nonlinear control systems and resilient robotic systems.

Conflict Avoidance in Social Navigation—a Survey

A major goal in robotics is to enable intelligent mobile robots to operate smoothly in shared human-robot environments. One of the most fundamental capabilities in service of this goal is competent navigation in this “social” context. As a result, there has been a recent surge of research on social navigation; and especially as it relates to the handling of conflicts between agents during social navigation. These developments introduce a variety of models and algorithms, however as this research area is inherently interdisciplinary, many of the relevant papers are not comparable and there is no shared standard vocabulary. This survey aims at bridging this gap by introducing such a common language, using it to survey existing work, and highlighting open problems. It starts by defining the boundaries of this survey to a limited, yet highly common type of social navigation—conflict avoidance. Within this proposed scope, this survey introduces a detailed taxonomy of the conflict avoidance components. This survey then maps existing work into this taxonomy, while discussing papers using its framing. Finally, this article proposes some future research directions and open problems that are currently on the frontier of social navigation to aid ongoing and future research.

Unified Human–Robot–Environment Interaction Control in Contact-Rich Collaborative Manipulation Tasks via Model-Based Reinforcement Learning

Many safety-critical or performance-demanding systems are human-in-the-loop, i.e., the robot interacts with human being and environment, in which human-in-the-loop control becomes a key research topic. In this paper, a unified optimal interaction control in joint space is presented for multi-point human-robot-environment interaction (HREI) problems, which are very common in human-robot collaborative manipulation tasks. Specifically, model-based reinforcement learning method is leveraged for obtaining optimal interaction control. For multi-point human-robot-environment interaction, the interaction forces exerted on each link are isolated and estimated via the backward generalized momentum observer method. In human-robot-environment interaction problems, the environmental as well as the human arm's dynamics parameters are usually unknown, stochastic, and time-varying. To obviate the dependence on these parameters, the GMM/GMR method is employed to learn the unknown external dynamics. It is noteworthy the interaction forces are considered as system states, the time derivatives of which are computed based on the GMM/GMR learning results via chain-rule. Then the iterative linear quadratic Gaussian with learned external dynamics (ILQG-LED) method is utilized to realize optimal multi-point human-robot-environment interaction control. The validity of the proposed method is verified through experimental studies.

Probabilistic Dual-Space Fusion for Real-Time Human-Robot Interaction.

For robots in human environments, learning complex and demanding interaction skills from humans and responding quickly to human motions are highly desirable. A common challenge for interaction tasks is that the robot has to satisfy both the task space and the joint space constraints on its motion trajectories in real time. Few studies have addressed the issue of hyperspace constraints in human-robot interaction, whereas researchers have investigated it in robot imitation learning. In this work, we propose a method of dual-space feature fusion to enhance the accuracy of the inferred trajectories in both task space and joint space; then, we introduce a linear mapping operator (LMO) to map the inferred task space trajectory to a joint space trajectory. Finally, we combine the dual-space fusion, LMO, and phase estimation into a unified probabilistic framework. We evaluate our dual-space feature fusion capability and real-time performance in the task of a robot following a human-handheld object and a ball-hitting experiment. Our inference accuracy in both task space and joint space is superior to standard Interaction Primitives (IP) which only use single-space inference (by more than 33%); the inference accuracy of the second order LMO is comparable to the kinematic-based mapping method, and the computation time of our unified inference framework is reduced by 54.87% relative to the comparison method.

A systematic automated grasping approach for automatic manipulation of fabric with soft robot grippers

PurposeThe garment industry will be one of the major beneficiaries of advances in smart manufacturing, as it is highly labor-intensive and heavily depends on labor force. Manipulating robots in human environments has made great strides in recent years. However, the main research has focused on rigid, solid objects and core capabilities such as grasping, placing remain a challenging problem when dealing with soft textiles. The experimental results indicate that adopting the proposed bionic soft finger will provide garment manufacturers with smart manufacturing capabilities. Then, the purpose of this paper is to utilize the flexibility of the soft finger to transfer fabric layer by layer without damage in garment automation.Design/methodology/approachIn this paper, a new way to separate layer by layer pieces of fabric has been inspired by the rise of soft robotics and their applications in automation. Fabric gripping is accomplished by wiping deformation and pinching the fabric. A single fabric piece is separated from cutting pile by the soft finger in four steps: making an arch by pressing, wiping deformation, grasping and separating, and placing.FindingsThe case study demonstrated that the soft finger arrangement for automated grasping of fabric pieces of a garment can be successfully applied to delicate fabric. A combination of cloth shape and weight determines the number of soft fingers. In addition, the soft finger was tested on different types of fabrics to determine its performance and application capabilities. The technology may be used to produce clothing intelligently in the future, such as intelligent stacking, intelligent transportation and intelligent packaging, to increase clothing industry productivity.Originality/valueAn industrial bionic soft finger gripping system is proposed in this paper for application in the field of fabric automatic manipulation. A piece of fabric could be picked up and released layer by layer from a stack by the proposed gripper without creating any damage to it. Soft grippers have the right proportion of softness and rigidity like a human being. A soft finger has a potential affinity for soft materials such as fabrics without damaging either their surface or their properties.

Audio–visual language instruction understanding for robotic sorting

For robot in human environment, it has always been expected that the robot can execute specified tasks following language instructions. Most current methods only rely on visual perception to understand the language instruction, while it may be not sufficient to fully interpret some language instructions when visually identical objects exist. In this paper, we propose a task of audio–visual language instruction understanding for robotic sorting, in which the robot is able to use both the visual and audio information to fully understand and execute the given instruction. To solve the proposed task, an audio–visual fusion framework is developed, which combines the visual localization and audio recognition models together for the robotic sorting task following language instruction. We have also collected a multimodal dataset for evaluation, and extensive experiments are conducted within the dataset and generalized to new scenarios in physical world demonstrating the effectiveness of the proposed framework.

Collaboration effectiveness-based complex operations allocation strategy towards to human–robot interaction

Under the background of the fourth industrial revolution driven by the new generation information technology and artificial intelligence, human–robot collaboration has become an important part of smart manufacturing. The new “human–robot–environment” relationship conducts industrial robots to collaborate with workers to adapt to environmental changes harmoniously. How to determine a reasonable human–robot interaction operations allocation strategy is the primary problem, by comprehensively considering the workers’ flexibility and industrial robots’ automation. In this paper, a human–robot collaborative operation framework based on CNC (Computer Number Control) machine tool was proposed, which divided into three stages: pre-machining, machining and post-machining. Then, an action-based granularity decomposition method was used to construct the human–robot interaction hierarchical model. Further, a collaboration effectiveness-based operations allocation function was established through normalizing the time, cost, efficiency, accuracy and complexity of human–robot interaction. Finally, a simulated annealing algorithm was adopted to solve preferable collaboration scheme; a case was used to verify the feasibility and effectiveness of the proposed method. It is expected that this study can provide useful guidance for human–robot interaction operations allocation on CNC machine tools.

Energy-aware redundant actuation for safe spring-assisted modular and reconfigurable robot

AbstractA spring-assisted modular and reconfigurable robot (SA-MRR) has been recently developed at our laboratory to reinforce its performance and enable safe and dexterous operation in human–robot environments. Multiple working mode (MWM) control enables each SA-MRR joint module to switch independently between working in a primary actuation mode and a secondary, spring-assisted mode that may improve task-specific energy performance measures and safety in a variety of manipulation tasks. The spring-assisted mode is characterized by synergy of spring and motor energy and may be summoned to offset motor energy demands or to safeguard a reconfigurable set of secondary joint limits. In this research work, two spring-assisted working mode strategies are proposed, and their characteristics have been investigated for SA-MRR actuation energy advantages while safe robot segregation in collaboration tasks is maintained. One MWM strategy has been designed to safeguard task-specific joint limits and is able to decrease motor energy consumption in some tasks. Another MWM strategy has been designed for energy efficiency and was able to reduce motor energy per cycle by $ \text{72}$ % in a simulated manipulation task while maintaining spatial safety constraints. Numerical simulations have demonstrated the effectiveness of the proposed spring-assisted working mode strategies for energy-aware safe manipulation applications.

Generating predicate suggestions based on the space of plans: an example of planning with preferences

Task planning in human–robot environments tends to be particularly complex as it involves additional uncertainty introduced by the human user. Several plans, entailing few or various differences, can be obtained to solve the same given task. To choose among them, the usual least-cost plan criteria is not necessarily the best option, because here, human constraints and preferences come into play. Knowing these user preferences is very valuable to select an appropriate plan, but the preference values are usually hard to obtain. In this context, we propose the Space-of-Plans-based Suggestions (SoPS) algorithms that can provide suggestions for some planning predicates, which are used to define the state of the environment in a task planning problem where actions modify the predicates. We denote these predicates as suggestible predicates, of which user preferences are a particular case. The first algorithm is able to analyze the potential effect of the unknown predicates and provide suggestions to values for these unknown predicates that may produce better plans. The second algorithm is able to suggest changes to already known values that potentially improve the obtained reward. The proposed approach utilizes a Space of Plans Tree structure to represent a subset of the space of plans. The tree is traversed to find the predicates and the values that would most increase the reward, and output them as a suggestion to the user. Our evaluation in three preference-based assistive robotics domains shows how the proposed algorithms can improve task performance by suggesting the most effective predicate values first.

Explainable robotic systems: understanding goal-driven actions in a reinforcement learning scenario

Robotic systems are more present in our society everyday. In human–robot environments, it is crucial that end-users may correctly understand their robotic team-partners, in order to collaboratively complete a task. To increase action understanding, users demand more explainability about the decisions by the robot in particular situations. Recently, explainable robotic systems have emerged as an alternative focused not only on completing a task satisfactorily, but also on justifying, in a human-like manner, the reasons that lead to making a decision. In reinforcement learning scenarios, a great effort has been focused on providing explanations using data-driven approaches, particularly from the visual input modality in deep learning-based systems. In this work, we focus rather on the decision-making process of reinforcement learning agents performing a task in a robotic scenario. Experimental results are obtained using 3 different set-ups, namely, a deterministic navigation task, a stochastic navigation task, and a continuous visual-based sorting object task. As a way to explain the goal-driven robot’s actions, we use the probability of success computed by three different proposed approaches: memory-based, learning-based, and introspection-based. The difference between these approaches is the amount of memory required to compute or estimate the probability of success as well as the kind of reinforcement learning representation where they could be used. In this regard, we use the memory-based approach as a baseline since it is obtained directly from the agent’s observations. When comparing the learning-based and the introspection-based approaches to this baseline, both are found to be suitable alternatives to compute the probability of success, obtaining high levels of similarity when compared using both the Pearson’s correlation and the mean squared error.

Disentangled Multi-Relational Graph Convolutional Network for Pedestrian Trajectory Prediction

Pedestrian trajectory prediction is one of the important tasks required for autonomous navigation and social robots in human environments. Previous studies focused on estimating social forces among individual pedestrians. However, they did not consider the social forces of groups on pedestrians, which results in over-collision avoidance problems. To address this problem, we present a Disentangled Multi-Relational Graph Convolutional Network (DMRGCN) for socially entangled pedestrian trajectory prediction. We first introduce a novel disentangled multi-scale aggregation to better represent social interactions, among pedestrians on a weighted graph. For the aggregation, we construct the multi-relational weighted graphs based on distances and relative displacements among pedestrians. In the prediction step, we propose a global temporal aggregation to alleviate accumulated errors for pedestrians changing their directions. Finally, we apply DropEdge into our DMRGCN to avoid the over-fitting issue on relatively small pedestrian trajectory datasets. Through the effective incorporation of the three parts within an end-to-end framework, DMRGCN achieves state-of-the-art performances on a variety of challenging trajectory prediction benchmarks.

Research on Real-Time Joint Stiffness Configuration of a Series Parallel Hybrid 7-DOF Humanoid Manipulator in Continuous Motion

In this paper, the real-time joint stiffness configuration strategy of a series parallel hybrid 7-DOF (degree of freedom) humanoid manipulator with flexible joints in continuous motion is studied. Firstly, considering the potential human robot accidental collision, combined with the manipulator safety index (MSI) and human body injury thresholds, the motion speed and joint stiffness of the robot are optimized in advance. Secondly, using hyperbolic tangent function for reference, the relationship between joint torques and passive joint deflection angles of the robot is given, which is beneficial for the real-time calculation of joint stiffness and obtain reasonable joint stiffness. Then, the structural model of the selected humanoid manipulator is described, on this basis, the relationship between the joint space stiffness and the Cartesian space stiffness of the humanoid manipulator is analyzed through Jacobian matrix, and the results show that the posture and joint space stiffness of the humanoid manipulator directly affect the Cartesian space stiffness of the humanoid manipulator. Finally, according to whether the humanoid manipulator works in the human-robot interaction environment, the real-time joint stiffness configuration of the humanoid manipulator in continuous motion is simulated and analyzed. The research shows that the humanoid manipulator with flexible joints can adjust the joint stiffness in real-time during continuous motion, and the joint stiffness configuration strategy can effectively improve the safety of human body in human-robot collision. In addition, in application, when the joint space stiffness of the robot is lower, the position accuracy can be improved by trajectory compensation.

An improved gesture recognition algorithm for human-robot collaboration

In this paper, an improved gesture recognition algorithm framework based on the improved SSD algorithm and OpenPose model for solving the robot’s mobile grasping problem in human-robot environment is proposed. Improved SSD algorithm by optimizing the front network of SSD algorithm and adjusting the prediction box is designed to identify the position of the hand rapidly, and to reduce the time for the training model. Testing experiments are perfomed for recognizing the hand’s key points information and eight designed gestures are compared with the traditional algorithm. The results show that the method can greatly improve the recognition distance of the original model under the dense environment, and also enhance the robustness and accuracy of the model, and have improved the ability of long-distance human-robot collaboration to a certain extent.

An agile multi-body additively manufactured soft actuator for soft manipulators


 
 
 With the introduction of collaborative robots in production environments, the harm to workers by using traditional robots with rigid links is inherent. A new generation of robots made from flexible soft materials that decreases collision danger by self-deforming actions has been proposed as a promising solution for the human-robot collaboration environments. Recently, by the development of additive manufacture of elastic soft materials, new design opportunities arise for these so-called soft robots. However, robustness that is required for production environments is still not achieved. This paper presents a design approach of a fully additively manufactured three-axis soft pneumatic actuator. For its use in flexible soft robotic manipulator systems, design guidelines, a direct 3D printing process with elastic materials and a low-level PLC semi-automated pressure regulation control system are presented. To validate the proposed design, the actuator is manufactured and tested for maximum contact force, bending motion reaction and its signal response.
 
 

Deep Reinforcement Learning with Interactive Feedback in a Human–Robot Environment

Robots are extending their presence in domestic environments every day, it being more common to see them carrying out tasks in home scenarios. In the future, robots are expected to increasingly perform more complex tasks and, therefore, be able to acquire experience from different sources as quickly as possible. A plausible approach to address this issue is interactive feedback, where a trainer advises a learner on which actions should be taken from specific states to speed up the learning process. Moreover, deep reinforcement learning has been recently widely used in robotics to learn the environment and acquire new skills autonomously. However, an open issue when using deep reinforcement learning is the excessive time needed to learn a task from raw input images. In this work, we propose a deep reinforcement learning approach with interactive feedback to learn a domestic task in a Human–Robot scenario. We compare three different learning methods using a simulated robotic arm for the task of organizing different objects; the proposed methods are (i) deep reinforcement learning (DeepRL); (ii) interactive deep reinforcement learning using a previously trained artificial agent as an advisor (agent–IDeepRL); and (iii) interactive deep reinforcement learning using a human advisor (human–IDeepRL). We demonstrate that interactive approaches provide advantages for the learning process. The obtained results show that a learner agent, using either agent–IDeepRL or human–IDeepRL, completes the given task earlier and has fewer mistakes compared to the autonomous DeepRL approach.

Collision-free path planning based on new navigation function for an industrial robotic manipulator in human-robot coexistence environments

ABSTRACT This paper presents a method for real-time collision-free path planning that uses a new navigation function for environments in which humans and robots cooperate or coexist, which guarantees the safety of a human operator who works with a collaborative robot. This method follows 10 skeletal joints in a human operator, which are detected using an RGB-D sensor, in order to detect humans. A new navigation function ensures collision-free planning for a robotic manipulator that avoids local minima and generates a shorter path. Simulation results show the effectiveness of the proposed method in comparison to two existing methods. The applicability and practicability of the proposed method are demonstrated by experiments using a 6-DoF industrial manipulator HIWIN RA605 with an RGB-D vision sensor (Kinect V1), a real-time operation system extension (RTX64) and an EtherCAT network protocol.

Learning Densities in Feature Space for Reliable Segmentation of Indoor Scenes

Deep learning has enabled remarkable advances in scene understanding, particularly in semantic segmentation tasks. Yet, current state of the art approaches are limited to a closed set of classes, and fail when facing novel elements, also known as out of distribution (OoD) data. This is a problem as autonomous agents will inevitably come across a wide range of objects, all of which cannot be included during training. We propose a novel method to distinguish any object (foreground) from empty building structure (background) in indoor environments. We use normalizing flow to estimate the probability distribution of high-dimensional background descriptors. Foreground objects are therefore detected as areas in an image for which the descriptors are unlikely given the background distribution. As our method does not explicitly learn the representation of individual objects, its performance generalizes well outside of the training examples. Our model results in an innovative solution to reliably segment foreground from background in indoor scenes, which opens the way to a safer deployment of robots in human environments.

Unified Neural Adaptive Control for Multiple Human–Robot–Environment Interactions

To go from human demonstration to robot independent operation, there are generally three phases of interaction to undergo, including human-robot interaction (HRI), human-robot-environment interaction (HREI), and robot-environment interaction (REI). Most existing methods address problems of a single stage. In this article, a unified neural adaptive control method that organically fuses multiple interactions is proposed. HRI, REI, and their coupling effects in HREI are comprehensively considered. First, the iterative least squares method is used for robot dynamics identification based on the linearized momentum observer. The accuracy of external force observation is improved to deal with dynamic uncertainties. The human force and the environmental force are achieved and decoupled by using only a force sensor, a momentum observer, and a selection matrix S. Next, the neural adaptive control method compensating position errors caused by the model uncertainty is addressed. The control system is proved to be stable based on the Lyapunov theorem. The trajectory tracking error under the model uncertainties is reduced. Then, the adaptive admittance control method is introduced. The interaction force of HRI is minimized and the interaction force control of REI is realized. Finally, the proposed method is verified by simulations and experiments.

Capture Point-Based Controller Using Real-Time Zero Moment Point Manipulation for Stable Bipedal Walking in Human Environment.

For collaboration of humans and bipedal robots in human environments, this paper proposes a stability control method for dynamically modifiable bipedal walking using a capture point (CP) tracking controller. A reasonable reference CP trajectory for the CP tracking control is generated using the real-time zero moment point (ZMP) manipulation without information on future footstep commands. This trajectory can be modified at any time during the single support phase according to a given footstep command. Accordingly, this makes it possible for the robot to walk stably with dynamically modifiable walking patterns, including sudden changes in navigational commands during the single support phase. A reference CP trajectory during the double support phase is also generated for continuity. The CP of the robot is controlled to track the reference trajectory using a ZMP-based CP tracking controller. The ZMP while walking is measured by the force-sensing resistor sensors mounted on the sole of each foot. A handling method for infeasible footstep commands is utilized so that the manipulated ZMP satisfies the allowable ZMP region for stability. The validity of the proposed method is verified through simulations and experiments.