Complex Assembly Tasks Research Articles

The Impact of Fidget Spinners on Fine Motor Skills in Individuals with and without ADHD: An Exploratory Analysis

Fidget spinners have been marketed as repetitive motion devices that improve attention and motor performance, and as such, they have become quite appealing to individuals with Attention Deficit Hyperactive Disorder (ADHD). To date, no studies have explored changes in brain activity that may occur due to fidgeting in ADHD. Our aim was to use functional Near-Infrared Spectroscopy (fNIRS) to examine the prefrontal cortex (PFC) during the performance of a standardized fine motor skills test after using a fidget spinner. Eight right-handed adults with ADHD and eight age and gender matched adults without ADHD (4F/4M, 4 control/4 fidget) performed the Purdue Pegboard Test (PPT) while their brain oxygenation was monitored using fNIRS. Relative neural efficiency (RNE) and involvement (RNI) were calculated and analyzed for all subtasks of PPT including the less cognitively demanding fine motor subtasks and more complex assembly tasks. The fidget spinner improved both task performance and RNE in the ADHD group but not the non-ADHD group for the less cognitively demanding subtasks. Our results indicate Fidget spinners may improve both relative neural efficiency and fine motor performance in adults with ADHD for less cognitively demanding tasks.

A Shared Control Method for Collaborative Human-Robot Plug Task

Humans are superior to robots in performing complex assembly tasks primarily due to their dexterity and sensorimotor abilities. When humans and robots collaborate to complete an assembly task, it is critical to provide the robot with sufficiently precise manipulation capabilities and formulate a shared control method that coordinates human and robot actions. In this study, we are investigating human-robot collaboration in completing an assembly task, namely the plug task. The plug task, motivated by the 2015 DARPA Robotics Challenge Finals, involves inserting a plug connected with a cable into a paired socket. The human holds the socket while the robot is grasping the cable and manages to insert the plug. We estimate an initial socket pose based on human subjects' data. With the statistically calculated initial guess, the robot could act instantly from the beginning to move the plug towards an initial target. The actual socket pose is updated using the Kalman filter in real-time. Besides, the system continuously updates the cable model and tracks the plug to enable visual servoing to complete the plug task. The result of an extensive experimental study demonstrates that the proposed approach demonstrates fast and accurate performance as well as flexibility when compared to a state-of-the-art method based on interactive probabilistic movement primitives.

InsertionNet - A Scalable Solution for Insertion

Complicated assembly processes can be described as a sequence of two main activities: grasping and insertion. While general grasping solutions are common in industry, insertion is still only applicable to small subsets of problems, mainly ones involving simple shapes in fixed locations and in which the variations are not taken into consideration. Recently, RL approaches with prior knowledge (e.g., LfD or residual policy) have been adopted. However, these approaches might be problematic in contact-rich tasks since interaction might endanger the robot and its equipment. In this letter, we tackled this challenge by formulating the problem as a regression problem. By combining visual and force inputs, we demonstrate that our method can scale to 16 different insertion tasks in less than 10 minutes. The resulting policies are robust to changes in the socket position, orientation or peg color, as well as to small differences in peg shape. Finally, we demonstrate an end-to-end solution for 2 complex assembly tasks with multi-insertion objectives when the assembly board is randomly placed on a table.

Magnetic Actuators: Reconfigurable Magnetic Origami Actuators with On‐Board Sensing for Guided Assembly (Adv. Mater. 25/2021)

In article number 2008751, Joseph B. Tracy, Denys Makarov, and co-workers fabricate ultrathin and reconfigurable magnetic origami actuators equipped with highly compliant magnetic field sensors allowing detection of their magnetization state and control over complex assembly tasks. This functionality is of major importance to assure a proper fault-free operation of origami-based reconfigurable smart actuators, which are in the heart of strain-engineered 3D architectures and soft robotics.

Dynamic Task Allocation based on Individual Abilities - Experiences from Developing and Operating an Inclusive Assembly Line for Workers With and Without Disabilities

Digital assistive systems, enable workers with disabilities to perform complex industrial work. However, the previously presented systems considered only a single workplace and a single user. This paper presents an assembly line that enables a joint processing of complex tasks by multiple workers with and without disabilities. The aim was to investigate the use of interaction technologies such as in-situ projections and hand-tracking to enable the processing of complex assembly tasks by work teams with highly heterogeneous abilities. The developed assembly line assists users and coordinates the joint work by distributing single assembly steps to workers based on the individual workers' abilities. Besides presenting the concept and implementation of the assembly line, we report our findings after six months of operation. Our results indicate that using the assistive assembly line has positive impacts, such as increased satisfaction and independence of the workers combined with a higher productivity.

Evaluation of an Augmented Reality Instruction for a Complex Assembly Task

Abstract This study compares the use of a marker-based AR instruction with a paper instruction commonly used in manual assembly. Hypotheses were tested as to whether the instruction type affects assembly time, number of errors, usability, and employee strain. Instead of student participants and artificial assembly tasks (e. g. Lego assemblies), the study was conducted with 16 trainees in a real workplace for the assembly of emergency door release handles in rail vehicles. Five assembly runs were performed. Assembly times and assembly errors were determined from recorded videos. Usability (SUS) and strain (NASA-TLX) were recorded with questionnaires. After a slower assembly at the beginning, the AR group assembled significantly faster in the fifth run. The comparable number of errors, usability and strain make marker-based AR applications interesting for knowledge transfer in manual assembly, especially due to the easy entrance and low costs.

Function Block-Based Multimodal Control for Symbiotic Human–Robot Collaborative Assembly

AbstractIn human–robot collaborative assembly, robots are often required to dynamically change their preplanned tasks to collaborate with human operators in close proximity. One essential requirement of such an environment is enhanced flexibility and adaptability, as well as reduced effort on the conventional (re)programming of robots, especially for complex assembly tasks. However, the robots used today are controlled by rigid native codes that cannot support efficient human–robot collaboration. To solve such challenges, this article presents a novel function block-enabled multimodal control approach for symbiotic human–robot collaborative assembly. Within the context, event-driven function blocks as reusable functional modules embedded with smart algorithms are used for the encapsulation of assembly feature-based tasks/processes and control commands that are transferred to the controller of robots for execution. Then, multimodal control commands in the form of sensorless haptics, gestures, and voices serve as the inputs of the function blocks to trigger task execution and human-centered robot control within a safe human–robot collaborative environment. Finally, the performed processes of the method are experimentally validated by a case study in an assembly work cell on assisting the operator during the collaborative assembly. This unique combination facilitates programming-free robot control and the implementation of the multimodal symbiotic human–robot collaborative assembly with the enhanced adaptability and flexibility.

Evaluation of Fatigue and Workload among Workers Conducting Complex Manual Assembly in Manufacturing

OCCUPATIONAL APPLICATIONS We conducted a study to evaluate fatigue and workload among workers performing complex assembly tasks. We investigate several predictors of fatigue, including subjective workload estimates, sleep duration, the shift being worked, and production levels. High levels of fatigue were reported in one-third of the shifts evaluated. The main predictors of high fatigue were workload estimates, working evening shifts, and baseline fatigue. Among the six dimensions of workload, only mental demand and frustration were predictors of high fatigue. Mental demand was also rated highest. Participants reported less than seven hours of sleep in 60% of the nights evaluated. These results suggest that managers and supervisors should consider cognitive workload as a key contributing factor to fatigue in complex manual assembly. Similarly, work schedule planning should consider shift duration, start times, and end times, because of the negative influence on fatigue and the potential disruptions on sleep among workers.

Hybrid Trajectory and Force Learning of Complex Assembly Tasks: A Combined Learning Framework

Complex assembly tasks involve nonlinear and low-clearance insertion trajectories with varying contact forces at different stages. For a robot to solve these tasks, it requires a precise and adaptive controller which conventional force control methods cannot provide. Imitation learning is a promising method for learning controllers that can solve the nonlinear trajectories from human demonstrations without needing to explicitly program them into the robot. However, the force profiles obtain from human demonstration via tele-operation tend to be sub-optimal for complex assembly tasks, thus it is undesirable to imitate such force profiles. Reinforcement learning learns adaptive control policies through interactions with the environment but struggles with low sample efficiency and equipment tear and wear in the physical world. To address these problems, we present a combined learning-based framework to solve complex robotic assembly tasks from human demonstrations via hybrid trajectory learning and force learning. The main contribution of this work is the development of a framework that combines imitation learning, to learn the nominal motion trajectory, with a reinforcement learning-based force control scheme to learn an optimal force control policy, which can satisfy the nominal trajectory while adapting to the force requirement of the assembly task. To further improve the imitation learning part, we develop a hierarchical architecture, following the idea of goal-conditioned imitation learning, to generate the trajectory learning policy on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">skill</i> level offline. Through experimental validations, we corroborate that the proposed learning-based framework can generate high-quality trajectories and find suitable force control policies which adapt to the tasks’ force requirements more efficiently.

A digital twin-based frame work for task planning and robot programming in HRC

This paper presents a digital twin-based framework for complex assembly tasks in human-robot collaborative assembly system. In this research, the system is developed to support the planning, decision, and implementation of human-robot collaboration. Virtual simulation is applied from task planning to robot control, including assembly task allocation and scheduling, robot trajectory planning and programming. Online optimization is conducted for real-time control based the status of operators and robots from the real production system. An industrial case is presented to validate the performance of the proposed digital twin framework and collaborative assembly system.

Leveraging multimodal data for intuitive robot control towards human-robot collaborative assembly

In human-robot collaborative assembly, robots help human operators complete complex assembly tasks. However, controlling these robots is not intuitive given the constraints posed by the close proximity of the operator. In response to this need, a novel approach using multimodal data is developed for human-centred robot control in human-robot collaborative assembly. An interface design is developed to fuse multimodal communication channels for robust and adaptive robot control and then multimodal data are defined as control input for assembly task execution. This control scheme offers human operators symbiotic multimodal tools for proactive HRC with enhanced flexibility and adaptability.

Analytical Joining Models for Learning Contact-Rich Cabinet Assembly Tasks from Simulation

Recent advances in Machine Learning introduce promising solutions for the so-called intelligent automation. Learning robot control offline in a physics simulation environment presents one suitable approach, even for complex assembly tasks with high variance in the products to be manufactured. Beside the continuous development and improvement of efficient learning algorithms, the synthetically generation of feasible training data is of crucial importance. Therefore, this paper focuses on defining and evaluating analytical process models for the assembly of electrical cabinet terminals. Using the geometric and material product data, the predicted assembly forces are described mathematically as a function of the joining path. The presented work covers the development, evaluation and validation of accurate joining models for snap hook-based assembly which in a subsequent step can be implemented in any suitable state of the art physics simulation.

Measuring the Performance Impact of Using the Microsoft HoloLens 1 to Provide Guided Assembly Work Instructions

Abstract The human performance benefits of four types of guided assembly instructions, including the Microsoft HoloLens 1, were analyzed in the context of a realistic assembly task. Several studies have confirmed the benefits of using augmented reality (AR) work instructions over traditional digital or paper instructions, but few have compared the effects of different AR hardware, including head-mounted displays, for complex assembly tasks. Participants completed a mock wing assembly task using the Microsoft HoloLens 1, and completion time, error count, and net promoter score (NPS) were recorded. These data were compared to data from previous research which employed desktop model-based instructions (MBIs), tablet MBI, and tablet AR instructions for the same task. The use of HoloLens AR instructions led to time saving of 16% over the tablet AR instructions. HoloLens users also had lower error rates than non-AR users. Despite the performance benefits of the HoloLens AR instructions, this condition had a lower NPS than the tablet AR group. The qualitative data showed that some users thought the HoloLens device was uncomfortable and that the tracking was not always exact. Although the users favored the tablet AR condition, the HoloLens condition had significantly faster assembly times. The authors recommend using the HoloLens for complex guided assembly instructions with minor changes, such as allowing the user to toggle the AR instructions on and off at will. The results of this paper can help manufacturing stakeholders understand the benefits of different AR technologies for manual assembly tasks.

Joining Force of Human Muscular Task Planning With Robot Robust and Delicate Manipulation for Programming by Demonstration

Recently, programing by demonstration (PbD) received much attention for its capacity of fast programming with increasing demands in the robot manipulation area, especially in industrial applications. However, one of the biggest challenges of PbD is the recognition of demonstrator's finger high-fidelity motions especially in the environments with uncertainties, which limits the efficiency and accuracy of PbD. In this article, inspired by human dexterity, a novel PbD approach using the implicit muscular task planning strategy is presented to extract features from the arms’ giant movement and the hands’ fine motions during the demonstrator's operation. Furthermore, we integrate a deep reinforcement learning control method that further improves the manipulations’ adaptive ability in the unknown or dynamic environments. The experimental results show that our proposed approach can deal with relative complex assembly tasks with a success rate of more than 67% within a fit tolerance of 4.2 mm by one-shot demonstration.

Composing Diverse Policies for Temporally Extended Tasks

Robot control policies for temporally extended and sequenced tasks are often characterized by discontinuous switches between different local dynamics. These change-points are often exploited in hierarchical motion planning to build approximate models and to facilitate the design of local, region-specific controllers. However, it becomes combinatorially challenging to implement such a pipeline for complex temporally extended tasks, especially when the sub-controllers work on different information streams, time scales and action spaces. In this letter, we introduce a method that can automatically compose diverse policies comprising motion planning trajectories, dynamic motion primitives and neural network controllers. We introduce a global goal scoring estimator that uses local, per-motion primitive dynamics models and corresponding activation state-space sets to sequence diverse policies in a locally optimal fashion. We use expert demonstrations to convert what is typically viewed as a gradient-based learning process into a planning process without explicitly specifying preand post-conditions. We first illustrate the proposed framework using an MDP benchmark to showcase robustness to action and model dynamics mismatch, and then with a particularly complex physical gear assembly task, solved on a PR2 robot. We show that the proposed approach successfully discovers the optimal sequence of controllers and solves both tasks efficiently.

Smart augmented reality instructional system for mechanical assembly towards worker-centered intelligent manufacturing

Quality and efficiency are crucial indicators of any manufacturing company. Many companies are suffering from a shortage of experienced workers across the production line to perform complex assembly tasks. To reduce time and error in an assembly task, a worker-centered system consisting of multi-modal Augmented Reality (AR) instructions with the support of a deep learning network for tool detection is introduced. The integrated AR is designed to provide on-site instructions including various visual renderings with a fine-tuned Region-based Convolutional Neural Network, which is trained on a synthetic tool dataset. The dataset is generated using CAD models of tools and displayed onto a 2D scene without using real tool images. By experimenting the system to a mechanical assembly of a CNC carving machine, the result of a designed experiment shows that the system helps reduce the time and errors of the given assembly tasks by 33.2 % and 32.4 %, respectively. With the integrated system, an efficient, customizable smart AR instruction system capable of sensing, characterizing requirements, and enhancing worker’s performance has been built and demonstrated.

Learning to Assemble: Estimating 6D Poses for Robotic Object-Object Manipulation

In this letter we propose a robotic vision task with the goal of enabling robots to execute complex assembly tasks in unstructured environments using a camera as the primary sensing device. We formulate the task as an instance of 6D pose estimation of template geometries, to which manipulation objects should be connected. In contrast to the standard 6D pose estimation task, this requires reasoning about local geometry that is surrounded by arbitrary context, such as a power outlet embedded into a wall. We propose a deep learning based approach to solve this task alongside a novel dataset that will enable future work in this direction and can serve as a benchmark. We experimentally show that state-of-the-art 6D pose estimation methods alone are not sufficient to solve the task but that our training procedure significantly improves the performance of deep learning techniques in this context.

Skill-based Programming of Force-controlled Assembly Tasks using Deep Reinforcement Learning

The selection of parameters for force-controlled assembly tasks remains a time consuming process. Skill-based approaches offer a guidance in parameter space, but still require expert knowledge to tune the parameters accordingly. Recently, Deep Reinforcement Learning algorithms have been used more frequently to learn the parameters of complex assembly tasks. For the industrial application of Reinforcement Learning, it is crucial to increase the training efficiency. Most approaches are based on engineered force-controllers or contact-models and focus on one specific robot. In this paper, we propose a framework, using state-of-the-art model-free algorithms and manipulation skills to learn force and position parameters in a simulation environment.

Symbiotic human-robot collaboration: multimodal control using function blocks

Complex assembly tasks require increased flexibility and adaptability, as well as higher effort on the conventional (re)programing of robots. To solve such challenges, this paper presents a function block-enabled multimodal control scheme for symbiotic human-robot collaborative assembly. Data/event-driven function blocks with smart decision algorithms are used for human-centred robot control with multimodal fusion. Then, multimodal control commands in the form of haptics, gesture and voice are defined as the inputs of the function blocks to trigger task execution. This novel scheme facilitates the implementation of the multimodal symbiotic human-robot collaborative assembly with enhanced stability and flexibility.

Symbolic-Based Recognition of Contact States for Learning Assembly Skills.

Imitation learning is gaining more attention because it enables robots to learn skills from human demonstrations. One of the major industrial activities that can benefit from imitation learning is the learning of new assembly processes. An essential characteristic of an assembly skill is its different contact states (CS). They determine how to adjust movements in order to perform the assembly task successfully. Humans can recognize CSs through haptic feedback. They execute complex assembly tasks accordingly. Hence, CSs are generally recognized using force and torque information. This process is not straightforward due to the variations in assembly tasks, signal noise and ambiguity in interpreting force/torque (F/T) information. In this research, an investigation has been conducted to recognize the CSs during an assembly process with a geometrical variation on the mating parts. The F/T data collected from several human trials were pre-processed, segmented and represented as symbols. Those symbols were used to train a probabilistic model. Then, the trained model was validated using unseen datasets. The primary goal of the proposed approach aims to improve recognition accuracy and reduce the computational effort by employing symbolic and probabilistic approaches. The model successfully recognized CS based only on force information. This shows that such models can assist in imitation learning.