Robotic Tasks Research Articles

Microstimulation-based path tracking control of pigeon robots through parameter adaptive strategy

Research on animal robots utilizing neural electrical stimulation is a significant focus within the field of neuro-control, though precise behavior control remains challenging. This study proposes a parameter-adaptive strategy to achieve accurate path tracking. First, the mapping relationship between neural electrical stimulation parameters and corresponding behavioral responses is comprehensively quantified. Next, adjustment rules related to the parameter-adaptive control strategy are established to dynamically generate different stimulation patterns. A parameter-adaptive path tracking control strategy (PAPTCS), based on fuzzy control principles, is designed for the precise path tracking tasks of pigeon robots in open environments. The results indicate that altering stimulation parameter levels significantly affects turning angles, with higher UPN and PTN inducing changes in the pigeons' motion state. In experimental scenarios, the average control efficiency of this system was 82.165%. This study provides a reference method for the precise control of pigeon robot behavior, contributing to research on accurate target path tracking.

Optimal robot task scheduling in cluttered environments considering mechanical advantage

Abstract In various industrial robotic applications, the effective traversal of a manipulator amidst obstacles and its ability to reach specific task-points are imperative for the execution of predefined tasks. In certain scenarios, the sequence in which the manipulator reaches these task-points significantly impacts the overall cycle time required for task completion. Moreover, some tasks necessitate significant force exertion at the end-effector. Therefore, establishing an optimal sequence for the task-points reached by the end-effector’s tip is crucial for enhancing robot performance, ensuring collision-free motion and maintaining high-force application at the end-effector’s tip. To maximize the manipulator’s manipulability, which serves as a performance index for assessing its force capability, we aim to establish an optimal collision-free task sequence considering higher mechanical advantage. Three optimization criteria are considered: the cycle time, collision avoidance and the manipulability index. Optimization is accomplished using a genetic algorithm coupled with the Bump-Surface concept for collision avoidance. The effectiveness of this approach is confirmed through simulation experiments conducted in 2D and 3D environments with obstacles employing both redundant and non-redundant robots.

Real-world cooking robot system from recipes based on food state recognition using foundation models and PDDL

Although there is a growing demand for cooking behaviors as one of the expected tasks for robots, a series of cooking behaviors based on new recipe descriptions by robots in the real world has not yet been realized. In this study, we propose a robot system that integrates real-world executable robot cooking behavior planning using the Large Language Model (LLM) and classical planning of PDDL descriptions, and food ingredient state recognition learning from a small number of data using the Vision Language model (VLM). We succeeded in experiments in which PR2, a dual-armed wheeled robot, performed cooking from arranged new recipes in a real-world environment, and confirmed the effectiveness of the proposed system.

ORPP—An Ontology for Skill-Based Robotic Process Planning in Agile Manufacturing

Ontology plays a significant role in AI (Artificial Intelligence) and robotics by providing structured data, reasoning, action understanding, context awareness, knowledge transfer, and semantic learning. The structured framework created by the ontology for knowledge representation is crucial for enabling intelligent behavior in robots. This paper provides a state-of-the-art analysis on the existing ontology approaches and at the same time consolidates the terms in the robotic task planning domain. The major gap identified in the literature is the need to bridge higher-level robotic process management and lower-level robotic control. This gap makes it difficult for operators/non-robotic experts to integrate robots into their production processes as well as evaluate key performance indicators (KPI) of the processes. To fill the gap, the authors propose an ontology for skill-based robotics process planning (ORPP). ORPP not only provides a standardization in the robotic process planning in the agile manufacturing domain but also enables non-robotic experts to design and plan their production processes using an intuitive Process-Task-Skill-Primitive structure to control low-level robotic actions. On the performance level, this structure provides traceability of the KPIs down to the robot control level.

A developmental model of audio-visual attention (MAVA) for bimodal language learning in infants and robots

A social individual needs to effectively manage the amount of complex information in his or her environment relative to his or her own purpose to obtain relevant information. This paper presents a neural architecture aiming to reproduce attention mechanisms (alerting/orienting/selecting) that are efficient in humans during audiovisual tasks in robots. We evaluated the system based on its ability to identify relevant sources of information on faces of subjects emitting vowels. We propose a developmental model of audio-visual attention (MAVA) combining Hebbian learning and a competition between saliency maps based on visual movement and audio energy. MAVA effectively combines bottom-up and top-down information to orient the system toward pertinent areas. The system has several advantages, including online and autonomous learning abilities, low computation time and robustness to environmental noise. MAVA outperforms other artificial models for detecting speech sources under various noise conditions.

Sparse Convolution-Based 6D Pose Estimation for Robotic Bin-Picking With Point Clouds

Abstract Estimating the orientation and position of objects is a crucial step in robotic bin-picking tasks. The challenge lies in the fact that, in real-world scenarios, a diverse array of objects is often randomly stacked, resulting in significant occlusion. This study introduces an innovative approach aimed at predicting 6D poses by processing point clouds through a two-stage neural network. In the initial stage, a network for scenes with low-textured environments is designed. Its purpose is to perform instance segmentation and provide an initial pose estimation. Entering the second stage, a pose refinement network is suggested. This network is intended to enhance the precision of pose prediction, building upon the output from the first stage. To tackle the challenge of resource-intensive annotation, a simulation technique is employed to generate a synthetic dataset. Additionally, a dedicated software tool has been developed to annotate real point cloud datasets. In practical experiments, our method demonstrated superior performance compared to baseline methods such as PointGroup and Iterative Closest Point. This superiority is evident in both segmentation accuracy and pose refinement. Moreover, practical grasping experiments have underscored the method's efficacy in real-world industrial robot bin-picking applications. The results affirm its capability to successfully address the challenges produced by occluded and randomly stacked objects.

Trajectory tracking control based on deep reinforcement learning and ensemble random network distillation for robotic manipulator

Abstract In general, the trajectory tracking of robotic manipulator is exceptionally challenging due to the complex and strongly coupled mechanical architecture. In this paper, precise track control of the robotic manipulator is formulated as a dense reward problem for reinforcement learning(RL). A deep RL(DRL) approach combining the soft actor-critic (SAC) algorithm and ensemble random network distillation (ERND) is proposed to address the tracking control problem for robotic manipulator. Firstly, an ERND model is designed, consisting of a module list of multiple RND models. Each RND model obtains the error by learning the target features and the predicted features of the environment. The resulting error serves as internal rewards that drive the robotic agent to explore unknown and unpredictable environmental states. The ensemble model obtains the total internal reward by summing the internal rewards of each RND model, thereby obtaining more accurately reflecting the characteristics of the manipulator in tracking control tasks and improving control performance. Secondly, combining the SAC algorithm with ERND facilitates more robust exploration capabilities in environments with input saturation and joint angle constraints, thereby enabling faster learning of effective policies and enhancing the performance and efficiency of robotic manipulator tracking control tasks. Finally, the simulation results demonstrate that the robotic manipulator tracking control task is effectively completed in dense reward problems through the combination of the SAC algorithm and ERND.

Multiobjective optimization of collaborative robotic task sequence assignment problems under collision-free constraints

This paper proposes a multiobjective optimization approach to address the challenge of collaborative manufacturing with multiple robot arms. Given the necessity for multiple robot arms to work together, the potential for collisions between robotic motions is a significant concern, and the automated task sequence assignment for robots becomes increasingly complex. Previous research has either simplified the collision-free conditions in a limited working area, or employed a master-slave approach to obtain only a local solution. Consequently, we propose a unified global optimization approach for simultaneously addressing various collaborative manufacturing issues, including robotic task sequence assignment (RTSA), multiple inverse kinematics (IK) selection, joint-space collision-free operations and multiple manufacturing objectives. As the optimal collaborative RTSA problem is a combinatorial optimization problem with non-deterministic polynomial-time hard (NP-hard) complexity, this paper presents a hybrid nondominated sorting genetic algorithm III (NSGA-III) method that integrates a Hamming-distance-based method and a greedy strategy within NSGA-III to improve population diversity and solution quality. To validate the efficacy of the proposed approach, simulation experiments were conducted on cooperative manufacturing scenarios, with two objectives: task completion time and task load balancing. The experimental results demonstrate that the proposed approach is effective in obtaining collision-free Pareto solutions. Furthermore, the proposed hybrid NSGA-III method obtains superior solutions compared to the original method in the studied problem, as measured by two performance indices.

An Implementation of Communication, Computing and Control Tasks for Neuromorphic Robotics on Conventional Low-Power CPU Hardware

An Implementation of Communication, Computing and Control Tasks for Neuromorphic Robotics on Conventional Low-Power CPU Hardware

Adaptive spatiotemporal neural networks through complementary hybridization

Processing spatiotemporal data sources with both high spatial dimension and rich temporal information is a ubiquitous need in machine intelligence. Recurrent neural networks in the machine learning domain and bio-inspired spiking neural networks in the neuromorphic computing domain are two promising candidate models for dealing with spatiotemporal data via extrinsic dynamics and intrinsic dynamics, respectively. Nevertheless, these networks have disparate modeling paradigms, which leads to different performance results, making it hard for them to cover diverse data sources and performance requirements in practice. Constructing a unified modeling framework that can effectively and adaptively process variable spatiotemporal data in different situations remains quite challenging. In this work, we propose hybrid spatiotemporal neural networks created by combining the recurrent neural networks and spiking neural networks under a unified surrogate gradient learning framework and a Hessian-aware neuron selection method. By flexibly tuning the ratio between two types of neurons, the hybrid model demonstrates better adaptive ability in balancing different performance metrics, including accuracy, robustness, and efficiency on several typical benchmarks, and generally outperforms conventional single-paradigm recurrent neural networks and spiking neural networks. Furthermore, we evidence the great potential of the proposed network with a robotic task in varying environments. With our proof of concept, the proposed hybrid model provides a generic modeling route to process spatiotemporal data sources in the open world.

Robotic task specific training for upper limb neurorehabilitation: a mixed methods feasibility trial reporting achievable dose.

Robotic devices for upper-limb neurorehabilitation allow an increase in intensity of practice, often relying on video game-based training strategies with limited capacity to individualise training and integrate functional training. This study shows the development of a robotic Task Specific Training (TST) protocol and evaluate the achieved dose. Mixed-method study. A 3D robotic device for the upper limb, was made available to therapists for use during neurorehabilitation sessions. A first phase allowed clinicians to define a dedicated session protocol for TST. In a second phase the protocol was applied and the achieved dose was measured. First phase (N = 5): a specific protocol, using deweighting for assessment, followed by customised passive movements and then active movement practice was developed. Second phase: the protocol was successfully applied with all participants (N = 10). Intervention duration: 4.5 ± 0.8 weeks, session frequency: 1.4 ± 0.2sessions/week, session length: 42 ± 9mins, session density: 39 ± 13%, intensity: 214 ± 84 movements/session, difficulty: dn = 0.77 ± 0.1 (normalised reaching distance) and Ɵ = 6.3 ± 23° (transverse reaching angle). Sessions' density and intensity were consistent across participants but clear differences of difficulty were observed. No changes in metrics were observed over the intervention. Robotic systems can support TST with high therapy intensity by modulating the practice difficulty to participants' needs and capabilities.

Experimental Comparison of Two 6D Pose Estimation Algorithms in Robotic Fruit-Picking Tasks

This paper presents an experimental comparison between two existing methods representative of two categories of 6D pose estimation algorithms nowadays commonly used in the robotics community. The first category includes purely deep learning methods, while the second one includes hybrid approaches combining learning pipelines and geometric reasoning. The hybrid method considered in this paper is a pipeline of an instance-level deep neural network based on RGB data only and a geometric pose refinement algorithm based on the availability of the depth map and the CAD model of the target object. Such a method can handle objects whose dimensions differ from those of the CAD. The pure learning method considered in this comparison is DenseFusion, a consolidated state-of-the-art pose estimation algorithm selected because it uses the same input data, namely, RGB image and depth map. The comparison is carried out by testing the success rate of fresh food pick-and-place operations. The fruit-picking scenario has been selected for the comparison because it is challenging due to the high variability of object instances in appearance and dimensions. The experiments carried out with apples and limes show that the hybrid method outperforms the pure learning one in terms of accuracy, thus allowing the pick-and-place operation of fruits with a higher success rate. An extensive discussion is also presented to help the robotics community select the category of 6D pose estimation algorithms most suitable to the specific application.

Multimodal tactile sensing fused with vision for dexterous robotic housekeeping

As robots are increasingly participating in our daily lives, the quests to mimic human abilities have driven the advancements of robotic multimodal senses. However, current perceptual technologies still unsatisfied robotic needs for home tasks/environments, particularly facing great challenges in multisensory integration and fusion, rapid response capability, and highly sensitive perception. Here, we report a flexible tactile sensor utilizing thin-film thermistors to implement multimodal perceptions of pressure, temperature, matter thermal property, texture, and slippage. Notably, the tactile sensor is endowed with an ultrasensitive (0.05 mm/s) and ultrafast (4 ms) slip sensing that is indispensable for dexterous and reliable grasping control to avoid crushing fragile objects or dropping slippery objects. We further propose and develop a robotic tactile-visual fusion architecture that seamlessly encompasses multimodal sensations from the bottom level to robotic decision-making at the top level. A series of intelligent grasping strategies with rapid slip feedback control and a tactile-visual fusion recognition strategy ensure dexterous robotic grasping and accurate recognition of daily objects, handling various challenging tasks, for instance grabbing a paper cup containing liquid. Furthermore, we showcase a robotic desktop-cleaning task, the robot autonomously accomplishes multi-item sorting and cleaning desktop, demonstrating its promising potential for smart housekeeping.

The Impacts of Gradually Dismantled Scaffolding on Learning Performance, Learning Engagement, and Self-Regulated Learning in Elementary School mBot Courses

In Taiwan, there is an increasing prevalence of robotics and programming integration in education, propelled by the national curriculum’s emphasis on competency development. This study focuses on the utilization of mBot robots in elementary school education to investigate the efficacy of incorporating self-study record sheets with gradually dismantled scaffolding for technology-assisted autonomous learning. The study explores the impact on learning performance, engagement, and self-regulated learning in mBot courses. The study employed a quasi-experimental pretest-posttest design with nonequivalent control group methodology. The experimental group utilized gradually dismantled scaffolding worksheets during technology-assisted autonomous learning, while the control group did not. Data analysis demonstrated that using scaffolding significantly improved learning performance and engagement. However, no significant impact on self-regulated learning was observed. Future research can explore the interaction between instructional strategies and student characteristics and investigate the effects of different types of scaffolding on learning outcomes in robotics tasks. Additionally, examining transfer effects and the impact of increased practice examples can further enhance students’ autonomous learning in robotics education.

PRF: A Program Reuse Framework for Automated Programming by Learning from Existing Robot Programs

This paper explores the problem of automated robot program generation from limited historical data when neither accurate geometric environmental models nor online vision feedback are available. The Program Reuse Framework (PRF) is developed, which uses expert-defined motion classes, a novel data structure introduced in this work, to learn affordances, workspaces, and skills from historical data. Historical data comprise raw robot joint trajectories and descriptions of the robot task being completed. Given new tasks, motion classes are then used again to formulate an optimization problem capable of generating new open-loop, skill-based programs to complete the tasks. To cope with a lack of geometric models, a technique to learn safe workspaces from demonstrations is developed, allowing the risk of new programs to be estimated before execution. A new learnable motion primitive for redundant manipulators is introduced, called a redundancy dynamical movement primitive, which enables new end-effector goals to be reached while mimicking the whole-arm behavior of a demonstration. A mobile manipulator part transportation task is used throughout to illustrate each step of the framework.

MosViT: towards vision transformers for moving object segmentation based on Lidar point cloud

Abstract Moving object segmentation is fundamental for various downstream tasks in robotics and autonomous driving, providing crucial information for them. Effectively extracting spatial-temporal information from consecutive frames and addressing the scarcity of dataset is paramount for learning-based 3D LiDAR Moving Object Segmentation (LIDAR-MOS). In this work, we propose a novel deep neural network based on Vision Transformers (ViTs) to tackle this problem. We first validate the feasibility of Transformer networks for this task, offering an alternative to CNNs. Specifically, we utilize a dual-branch structure based on range-image data to extract spatial-temporal information from consecutive frames and fuse it using a motion-guided attention mechanism. Furthermore, we employ the ViT as the backbone, keeping its architecture unchanged from what is used for RGB images. This enables us to leverage pre-trained models from RGB images to improve results, addressing the issue of limited LIDAR point cloud data, which is cheaper compared to acquiring and annotating point cloud data. We validate the effectiveness of our approach on the LIDAR-MOS benchmark of SemanticKitti and achieve comparable results to methods that use CNNs on range image data. The source code and trained models are available at https://github.com/mafangniu/MOSViT.git.

A Petri nets-based modeling method for multi robot path planning

The construction of lunar bases is one of the core enabling technologies in current lunar exploration and development plans of various countries. However, to eliminate the constraints of high transportation costs and limited manned space technology, a new research plan is to employ multi robot teams to build lunar bases. The key to this solution is how to achieve path planning for complex tasks for multiple robots. Therefore, this article takes the exploration and collection area, lunar soil collection, and lunar soil transportation in the lunar base construction scene as complex task inputs, and studies a multi robot path planning modeling method based on Petri net model. Firstly, a Petri net model for multi robot motion was constructed. Meanwhile, linear temporal logic(LTL) language was used to describe the related tasks of lunar base construction. Finally, simulation validation was conducted in Matlab software and compared with the modeling method using switching systems. The results show that the total modeling time required for using the Petri net model is reduced by two orders of magnitude compared to the modeling time for a single task switching system model, indicating that the established Petri net multi robot model has advantages such as avoiding dimension explosion and computational efficiency.

End-to-end deep reinforcement learning and control with multimodal perception for planetary robotic dual peg-in-hole assembly

The planetary construction is necessary for long-term scientific deep space exploration and resource utilization in the future. The planetary robotic assembly control is a key technology that must be broken through in future planetary surface construction. The paper focuses on the most representative dual peg-in–hole assembly, which has sufficiently complex contact interaction, wide range of applications and good method portability. To address the challenges brought by the unstructured planetary environment and the features of the construction tasks, the paper proposes an end-to-end deep reinforcement learning and control method with multimodal perception for planetary robotic assembly tasks. A staged reward function based on the visual virtual target point for policy learning is designed. The effectiveness and feasibility of the proposed control method have been verified through simulation experiments and ground real robot experiments. It provides a feasible control method of robotic operations for future planetary surface construction.

Integrating Speech and Gesture for Generating Reliable Robotic Task Configuration

Integrating Speech and Gesture for Generating Reliable Robotic Task Configuration

A lightweight and accurate recognition algorithm of pointer meter based on improved Deeplabv3+ for inspection robots

Pointer meter is widely utilized in the fields of modern industry. Nowadays, intelligent inspection robots are gradually employed in place of labors for inspection task. Pointer meter recognition is one of the most important tasks of inspection robots. This article presents a lightweight pointer meter recognition algorithm, which is suitable for deployment on inspection robots. First, pointer meter is identified by pruned YOLOv5. Afterwards, the dial and the pointer are segmented through improved Deeplabv3+ in which the JPU and depthwise separable convolutions are utilized in lieu of dilated convolutions. After perspective transform and central-line extraction of pointer, the reading of the pointer meter can be determined using the angle method. Experimental results verify the FPS of pruned YOLOv5 improves 13.18% compared to original YOLOv5 and the FPS of improved Deeplabv3+ improves 45.74% compared to original Deeplabv3+. Additionally, the reading accuracy of the algorithm is 98.40% and average fiducial error is 0.32%, which indicate good accuracy. The relative standard deviation of reading is less than 1.4%, which indicates good stability of proposed algorithm. This study proposes a lightweight and accurate pointer meter recognition algorithm based on improved Deeplabv3+, the algorithm is ported to NVIDIA Jetson TX2 NX to verify its stability and accuracy.