Robot Operating Research Articles

Multistage Synchronous Telescopic Manipulator With End‐Effector–Biased Rotating‐Pulling Mode for Damage‐Free Robotic Picking

ABSTRACTFruit picking is one of the most time‐consuming and labor‐intensive stages of fruit production, characterized by high labor demands and significant labor costs. Traditional fruit‐picking robotic manipulators typically adopt configurations similar to general‐purpose industrial robots, following a predefined path and employing a direct‐pulling mode to detach the fruit. However, due to the constraints of the orchard environment and the varying conditions of the fruit, manipulators should be designed to accommodate the specific horticultural characteristics of the trees to improve picking efficiency. Additionally, the picking process should be optimized based on the biological characteristics of the fruit to ensure quality. In this study, a five‐degree‐of‐freedom manipulator based on a multistage synchronous telescopic mechanism is proposed for fruit picking. Workspace analysis indicates that the manipulator can cover more than 80% of the fruit distribution on the trees. To ensure motion accuracy, a FreeRTOS‐based motion control system is developed for the manipulator. To evaluate picking efficiency and quality, fruit‐picking experiments are conducted in an apple orchard. A rope‐driven, three‐finger end‐effector is mounted in a biased position at the end of the manipulator, complemented by an RGB‐D camera for fruit detection and a ROS‐based control system for robotic operation. The performance of two picking modes (direct‐pulling and biased rotating‐pulling) are compared in these experiments. The results demonstrate that the biased rotating‐pulling mode yields a higher picking success rate and a lower stem damage rate compared with the direct‐pulling mode. Specifically, the damage‐free success rate for the biased rotating‐pulling mode is 80%, with a 9.18% reduction in stem damage compared with the direct‐pulling mode. Furthermore, the average picking cycle time is approximately 14.5 s. In conclusion, the manipulator and its motion control system successfully achieve efficient, nondestructive fruit picking with a high success rate, offering valuable insights for the development of fully automated fruit‐picking robots in the future.

Integration of Deep Learning Vision Systems in Collaborative Robotics for Real-Time Applications

Collaborative robotics and computer vision systems are increasingly important in automating complex industrial tasks with greater safety and productivity. This work presents an integrated vision system powered by a trained neural network and coupled with a collaborative robot for real-time sorting and quality inspection in a food product conveyor process. Multiple object detection models were trained on custom datasets using advanced augmentation techniques to optimize performance. The proposed system achieved a detection and classification accuracy of 98%, successfully processing more than 600 items with high efficiency and low computational cost. Unlike conventional solutions that rely on ROS (Robot Operating System), this implementation used a Windows-based Python framework for greater accessibility and industrial compatibility. The results demonstrated the reliability and industrial applicability of the solution, offering a scalable and accurate methodology that can be adapted to various industrial applications.

Tube-MPC based Trajectory Tracking Control for Substation Inspection Robot

Abstract A Tube-MPC based trajectory tracking control method was developed to enhance the capabilities of substation inspection robots. First, the kinematic models of the inspection robots were established, and the general form of the optimization control problem for Tube-MPC was introduced. The nominal control law for the nominal system was derived using a linearized nominal model, which was employed to solve the Tube-MPC cost function, ensuring accurate tracking of the desired trajectory. The algorithm was then simulated and tested. Simulation results demonstrate that the Tube-MPC approach significantly outperforms conventional MPC algorithms in terms of trajectory tracking performance and robustness for substation inspection robots. Experimental results further confirm that the Tube-MPC based method effectively enhances both robustness and tracking accuracy during inspection robot operations.

A Lightweight Detection Model Without Convolutions for Complex Stacked Grasping Tasks

In complex environments with multi-object stacking, the spatial relationships between objects necessitate a sequential grasping strategy to ensure both the safety of the target objects and the efficiency of robotic arm operations. To address this challenge, this study introduces a Visual Manipulation Relationship Network (VMRN) to determine the optimal grasping sequence. Traditional VMRN frameworks typically rely on convolutional neural networks (CNNs) for feature extraction, which often struggle with high-frequency feature extraction, long-tail data distributions, and real-time computational demands in multi-object stacking scenarios. To overcome these limitations, we propose a lightweight, convolution-free Transformer-based feature extraction network integrated into the visual detection model. This model is specifically designed for visual reasoning, with a focus on lightweight optimization to enhance the extraction of features for stacked objects. The proposed network incorporates local window attention, global information aggregation and broadcasting, and a dual-dimensional attention-based feedforward network to improve feature representation. Additionally, a novel loss function is designed to address the performance degradation in detecting long-tail categories, effectively mitigating the over-suppression of rare objects in imbalanced datasets. Experimental results demonstrate that the proposed model significantly improves both detection accuracy and computational efficiency, making it particularly suitable for real-time robotic grasping tasks in complex environments due to its lightweight design.

IC-GLI: a real-time, INS-centric GNSS-LiDAR-IMU localization system for intelligent vehicles

Abstract The integration of multiple sensors plays a vital role in achieving accurate vehicle positioning. However, light detection and ranging (LIDAR) and global navigation satellite system (GNSS) are susceptible to environmental effects, unlike inertial navigation system (INS), which remains unaffected by external factors. To enhance the precise positioning of intelligent vehicles in diverse environments and maximize the use of inertial measurement unit (IMU) data, this study presents a LIDAR, IMU, and GNSS-based positioning method centered around INS. The proposed method employs an IMU pre-integration model that includes the Earth’s rotation and utilizes a factor graph optimization framework to fuse the measurement data from LIDAR, IMU, and GNSS. By assigning appropriate weights to the residuals and dynamically adjusting the contributions of each sensor based on the prevailing environmental conditions, the optimal localization results are obtained through the solution of the maximum a posteriori estimate and the update of the INS state. Finally, the proposed system’s localization performance is validated using intelligent vehicles in the Robot Operating System framework. The results demonstrate that the system achieves high accuracy and robustness across various environments.

A Multi-Drone System Proof of Concept for Forestry Applications

This study presents a multi-drone proof of concept for efficient forest mapping and autonomous operation, framed within the context of the OPENSWARM EU Project. The approach leverages state-of-the-art open-source simultaneous localisation and mapping (SLAM) frameworks, like LiDAR (Light Detection And Ranging) Inertial Odometry via Smoothing and Mapping (LIO-SAM), and Distributed Collaborative LiDAR SLAM Framework for a Robotic Swarm (DCL-SLAM), seamlessly integrated within the MRS UAV System and Swarm Formation packages. This integration is achieved through a series of procedures compliant with Robot Operating System middleware (ROS), including an auto-tuning particle swarm optimisation method for enhanced flight control and stabilisation, which is crucial for autonomous operation in challenging environments. Field experiments conducted in a forest with multiple drones demonstrate the system’s ability to navigate complex terrains as a coordinated swarm, accurately and collaboratively mapping forest areas. Results highlight the potential of this proof of concept, contributing to the development of scalable autonomous solutions for forestry management. The findings emphasise the significance of integrating multiple open-source technologies to advance sustainable forestry practices using swarms of drones.

Development of an Autonomous Driving Path-Generation Algorithm for a Crawler-Type Ridge-Forming Robot

The agricultural sector is currently facing problems including a decline in the agricultural population, labor shortages, and an aging population. To solve these problems and increase agricultural productivity, the development and distribution of autonomous agricultural machinery is necessary. Since autonomous agricultural machinery is operated along a pre-defined path, it is essential to generate an autonomous driving path that takes into account the driving and working methods of the agricultural machinery. In this study, an autonomous driving path-generation algorithm for the autonomous operation of a crawler-type ridge-forming robot is proposed. The proposed algorithm defines the field boundary using the geodetic coordinates of the field boundary points and the size of the robot, generates working line segments within the field boundary, and generates three types of waypoints, which constitute an autonomous driving path based on the autonomous driving operating scenario. To verify the proposed algorithm, tests were conducted using four types of field boundary points with different shapes, and the results are presented. As a result of the simulation test, when a ridge was created using the generated autonomous driving path, the area occupied by the ridge in the total field area according to the field types of a rectangle, trapezoid, pentagon, and hexagon was indicated to be 80, 77, 85, and 77%, respectively.

Autonomous Teleoperated Robotic Arm Based on Imitation Learning Using Instance Segmentation and Haptics Information

Teleoperated robots are attracting attention as a solution to the pressing labor shortage. To reduce the burden on the operators of teleoperated robots and improve manpower efficiency, research is underway to make these robots more autonomous. However, end-to-end imitation learning models that directly map camera images to actions are vulnerable to changes in image background and lighting conditions. To improve robustness against these changes, we modified the learning model to handle segmented images where only the arm and the object are preserved. The task success rate for the demonstration data and the environment with different backgrounds was 0.0% for the model with the raw image input and 66.0% for the proposed model with segmented image input, with the latter having achieved a significant improvement. However, the grasping force of this model was stronger than that during the demonstration. Accordingly, we added haptics information to the observation input of the model. Experimental results show that this can reduce the grasping force.

Performance Comparison of ROS2 Middlewares for Multi-robot Mesh Networks in Planetary Exploration

Recent advancements in Multi-Robot Systems (MRS) and mesh network technologies pave the way for innovative approaches to explore extreme environments. The Artemis Accords, a series of international agreements, have further catalyzed this progress by fostering cooperation in space exploration, emphasizing the use of cutting-edge technologies. In parallel, companies across various sectors’ widespread adoption of the Robot Operating System 2 (ROS 2) underscores its robustness and versatility. This paper evaluates the performances of available ROS 2 MiddleWare (RMW), such as FastRTPS, CycloneDDS and Zenoh, over a mesh network with a dynamic topology. The final choice of RMW is determined by the one that would most fit the scenario: an exploration of the extreme extra-terrestrial environment using a Multi-Robot Systems (MRS). The conducted study in a real environment highlights Zenoh as a potential solution for future applications, showing a reduced delay, reachability, data overhead and CPU usage while being competitive on the RAM usage over a dynamic mesh topology.

Analysis and Simulation of Polishing Robot Operation Trajectory Planning

Analysis and Simulation of Polishing Robot Operation Trajectory Planning

A portable EEG signal acquisition system and a limited-electrode channel classification network for SSVEP

Brain-computer interfaces (BCIs) have garnered significant research attention, yet their complexity has hindered widespread adoption in daily life. Most current electroencephalography (EEG) systems rely on wet electrodes and numerous electrodes to enhance signal quality, making them impractical for everyday use. Portable and wearable devices offer a promising solution, but the limited number of electrodes in specific regions can lead to missing channels and reduced BCI performance. To overcome these challenges and enable better integration of BCI systems with external devices, this study developed an EEG signal acquisition platform (Gaitech BCI) based on the Robot Operating System (ROS) using a 10-channel dry electrode EEG device. Additionally, a multi-scale channel attention selection network based on the Squeeze-and-Excitation (SE) module (SEMSCS) is proposed to improve the classification performance of portable BCI devices with limited channels. Steady-state visual evoked potential (SSVEP) data were collected using the developed BCI system to evaluate both the system and network performance. Offline data from ten subjects were analyzed using within-subject and cross-subject experiments, along with ablation studies. The results demonstrated that the SEMSCS model achieved better classification performance than the comparative reference model, even with a limited number of channels. Additionally, the implementation of online experiments offers a rational solution for controlling external devices via BCI.

Terrain Traversability via Sensed Data for Robots Operating Inside Heterogeneous, Highly Unstructured Spaces.

This paper presents a comprehensive approach to evaluating the ability of multi-legged robots to traverse confined and geometrically complex unstructured environments. The proposed approach utilizes advanced point cloud processing techniques integrating voxel-filtered cloud, boundary and mesh generation, and dynamic traversability analysis to enhance the robot's terrain perception and navigation. The proposed framework was validated through rigorous simulation and experimental testing with humanoid robots, showcasing the potential of the proposed approach for use in applications/environments characterized by complex environmental features (navigation inside collapsed buildings). The results demonstrate that the proposed framework provides the robot with an enhanced capability to perceive and interpret its environment and adapt to dynamic environment changes. This paper contributes to the advancement of robotic navigation and path-planning systems by providing a scalable and efficient framework for environment analysis. The integration of various point cloud processing techniques into a single architecture not only improves computational efficiency but also enhances the robot's interaction with its environment, making it more capable of operating in complex, hazardous, unstructured settings.

Comparative Study of Digital Twin Developed in Unity and Gazebo

Digital twin (DT) technology has become a cornerstone in the simulation and analysis of real-world systems, offering unparalleled insights into the lifecycle management of physical assets. By providing a real-time synchronized replica of the physical entity, DTs enable predictive maintenance, performance optimization, and lifecycle extension, which are pivotal for industries aiming for digital transformation. This paper presents a comprehensive comparative study of DT development of a robotic arm using two prominent simulation platforms: Unity and Gazebo. Unity, with its roots in the gaming industry, offers robust real-time rendering and a user-friendly interface, making it a versatile choice for various industries. Gazebo, traditionally used in robotics, provides detailed physics simulations and sensor data emulation, which is ideal for precise engineering applications. We explored the performance of both platforms in creating accurate and dynamic digital replicas. Through qualitative and quantitative analyses, this study evaluates each platform’s strengths and limitations. The study assesses these platforms across key performance metrics such as accuracy, latency, graphic quality, and integration with the Robot Operating System (ROS). The DTs were developed using a consistent physical setup and communication layer to ensure fair comparisons. The results indicate that Unity performed better in terms of accurately mimicking the robotic arm with lower latency, making it ideal for applications requiring high-fidelity visualizations and real-time responsiveness. However, Gazebo excels in its ease of ROS integration and cost-effectiveness, making it a suitable choice for smaller robotics and automation projects. This study conducts an empirical comparison of these platforms in terms of their performance in creating DTs of robotic arms which is not readily available. This paper aims to guide developers and organizations in selecting the appropriate platform for their DT initiatives, ensuring efficient resource utilization and optimal outcomes.

Are YouTube Videos Useful in Robot-Assisted Segmentectomy Education?

Segmentectomy operation became a preferable operation for small lesions due to the importance of saving lung parenchyma. Using robotic technology has too many advantages for segmentectomy operations. Websites such as YouTube have become educational tools for surgical trainees. The aim of our study is to analyze YouTube videos for accurate and up-to-date information about robotic segmentectomy operations. The videos on www.youtube.com, which were reached on July 11, 2024, by using the keywords 'Robot segmentectomy' and 'Robotic segmentectomy lung', were evaluated in this research. The videos were evaluated by using the Journal of the American Medical Association (JAMA) scoring system, critical view of safety (CVS), and LAParoscopic surgery Video Educational GuidelineS (LAP-VEGaS). Eighty-one videos were included. Almost half of the videos (n = 42) were affiliated with university hospitals. Preoperative imaging was seen in 49% of all videos; however, the rates were 32% and 20.9% for patients' demographics and preoperative assessment information, respectively. Only 29.6% of the videos presented the placement of trocars during the presentation. It has become possible to record high-quality videos easily with developing technology. However, our results showed that many of the videos don't include the parameters, especially related to education. Therefore, we believe that those videos are inadequate for trainees.

Adaptive Robot Motion Planning for Smart Manufacturing Based on Digital Twin and Bayesian Optimization-Enhanced Reinforcement Learning

Abstract Advanced motion planning is crucial for safe and efficient robotic operations in various scenarios of smart manufacturing, such as assembling, packaging and palletizing. Compared to traditional motion planning methods, Reinforcement Learning (RL) shows better adaptability to complex and dynamic working environments. However, the training of RL models is often time-consuming and the determination of well-behaved reward function parameters is challenging. To tackle these issues, we propose an adaptive robot motion planning approach based on digital twin and reinforcement learning. The core idea is to adaptively select geometry-based or RL-based method for robot motion planning through a real-time distance detection mechanism, which can reduce the complexity of RL model training and accelerate the training process. In addition, we integrate Bayesian Optimization within RL training to refine the reward function parameters. We validate the approach with a Digital Twin enabled robot system through five kinds of tasks (Pick and Place, Drawer Open, Light Switch, Button Press, Cube Push) in dynamic environments. Experiment results show that our approach outperforms traditional RL-based method with improved training speed and guaranteed task performance. This work contributes to the practical deployment of adaptive robot motion planning in smart manufacturing.

Simulation of Autonomous Tractor using ROS

Farmers encounter a multitude of obstacles in agriculture, such as operating in challenging weather conditions and unpredictable surroundings, which can have a significant impact on sustainability and productivity. To tackle these hurdles, autonomous agricultural vehicles are indispensable. This paper introduces the digital modelling and testing of a self-driving tractor system tailored for agricultural tasks. By utilizing the Robot Operating System (ROS) and Gazebo simulator, the study delves into the incorporation of cutting-edge navigation algorithms, including the GMapping algorithm for Simultaneous Localization and Mapping (SLAM), Dijkstra algorithm, and the Dynamic Window Approach (DWA) for efficient path planning. The tractor prototype, developed using Fusion 360, undergoes thorough assessments to evaluate its mapping accuracy, localization precision, path planning effectiveness, and obstacle avoidance capabilities. The outcomes showcase successful mapping and localization, precise trajectory planning, adept obstacle avoidance, and visualization through RViz and Gazebo co-simulation. The study aims to propel the realm of autonomous agriculture by offering insights into the design, enhancement, and virtual implementation of autonomous tractor systems, ultimately enhancing the efficiency and sustainability of agricultural practices. Keywords: Autonomous Vehicles, Autonomous Tractors, Path Planning, GMapping, AMCL, ROS, Gazebo, RViz, Dijkstra’s Algorithm, DWA Path Planning, PID Controller, Simulation

Integrating humanoid robots with human musicians for synchronized musical performances

Entertainment robotics has garnered significant attention in recent years, with researchers focusing on developing robots capable of performing a variety of tasks, including magic, drawing, dancing, and music. This article presents our research on forming a musical band that includes both humanoid robots and human musicians, with the goal of achieving natural synchronization and collaboration during musical performances. We utilized two of our humanoid robots for this project: Polaris, a mid-sized humanoid robot, as the drummer, and Oscar, a Robotis-OP3 humanoid robot, as the keyboardist. The technical implementation incorporated essential components such as visual servoing, human-robot interaction, and Robot Operating System (ROS), enabling seamless communication and coordination between the humanoid robots and the human musicians. The success of this collaborative effort can be both seen and heard through the following YouTube link: https://youtu.be/pFOyt1KKCfY?feature=shared.

Designing an Autonomous Robot for Monitoring Open-Mouth Behavior of Chickens in Commercial Chicken Farms

HighlightsA mobile robot was designed to autonomously cruise in typical commercial chicken farms in Taiwan.The localization of the robot reached a mean absolute error of 0.17 m.The mobile robot automatically monitored the open-mouth behavior of chickens.A trained YOLO v7 model achieved an AP of 86.1% in the detection of chickens with open-mouth behavior.The analysis indicates that the open-mouth behavior ratio was correlated to ambient temperature.Abstract. In the conventional management of commercial floor-rearing chicken farms, farmers regularly patrol to observe chicken status and to check the environment. For countries with a subtropical climate (e.g., Taiwan), heat stress (HS) is a common environmental stressor for chickens due to high temperature and humidity. Open-mouth behavior (OMB) of chickens is an indicator of HS and is closely monitored in patrols. Manual patrols are laborious and time-consuming. Moreover, frequent patrols may increase the risk of pathogen spread while entering and exiting chicken farms. Therefore, this study developed a mobile robot to autonomously patrol in chicken farms and to automatically monitor OMB of chickens. Considering the commercial chicken farms in Taiwan usually have narrow aisles between feed buckets, a compact robot was designed. An ultra-wideband (UWB) module was utilized to localize the mobile robot. The mobile robot performed navigation and obstacle avoidance using a depth camera and the navigation stack of the Robot Operating System (ROS). For monitoring OMB of chickens, an image acquisition component was developed to acquire chicken images. A YOLO v7 model was trained to detect the chickens with OMB in the collected images. Subsequently, OMB ratios, defined as the ratios of chickens with OMB, were calculated to quantify this behavior. Experimental results showed that the mean absolute error of the UWB was 0.17 m in robot localization, complying with the requirements for navigating the robot in the narrow aisles of chicken farms. The trained YOLO v7 model achieved an average precision of 86.1%. Experimental results indicated that the mean OMB ratios between summer and winter were significantly different. In addition, the OMB ratio and temperature in summer were found to be moderately and positively correlated, indicating that OMB ratio was an effective indicator to represent chicken behaviors under different temperatures. The developed mobile robot provides a solution to automate the regular patrols performed by chicken farmers. Keywords: Deep learning, Depth image, Heat stress, Indoor positioning, Ultra-wideband (UWB), Unmanned vehicle.

Efficacy of Minimally Invasive Techniques versus Open Surgery for Recurrent Inguinal Hernias: A Meta-Analysis

Background: Recurrent inguinal hernias present difficult tasks in surgical approaches to address the issue and enhance patient prognosis. Laparoscopic and other forms of obsolete, less invasive surgery are increasingly being preferred to traditional open surgery. Objective: This meta-analysis aims to synthesize the effectiveness of minimal approach strategies compared with the open approach for recurrent inguinal hernia repair. Methods: The present study adhered to the PRISMA flowchart and databases search was conducted in PubMed, Cochrane, Embase, and Scopus. A total of 29 articles were used from the literature published between 2000 and 2024, which compared minimally invasive techniques and open surgery. Meaningfully, assessment parameters drawn from the study were recurrence rates, post-surgery pain, lengths of hospital stay, complications, and, satisfaction. Structural analysis was performed and combined ordinal data were analyzed for comparison of relative effectiveness. Results: It can be seen that, over the different parameters, minimally invasive approaches provided better results. Repeat rate was smaller (6.06% versus 11.14%), early postoperative pain (3.1 versus 5.4 days); shorter hospital stay (2.5 versus 4.8 days). In the following analysis we observed that the complication rate was significantly less in the minimally invasive group (4.2% vs 8.9%) and patient satisfaction index was slightly higher (89.4 vs 76.3). Further, cross-sectional analysis demonstrated cost effectiveness of laparoscopic techniques and precision in complicated cases offered by robotic operations. Conclusion: The results of the present study showed that minimally invasive approaches are superior to open surgery in treating recurrent inguinal hernias in terms of postoperative recurrence, recovery, and patient satisfaction. Patient and surgery based tailored treatment planning should be vital for promoting best treatment results.

基于改进RRT算法的葡萄采摘机械臂路径规划研究

The robot's operation in a grape orchard environment is often disrupted by obstacles such as vines and leaves, resulting in low fruit picking efficiency. To achieve stable obstacle avoidance, an improved RRT algorithm based on global adaptive step size and target-biased sampling was developed. First, the kinematic equations of the grape-picking robotic arm were established using the PoE method, and both forward and inverse kinematics calculations were performed to determine the robot's workspace. Then, to address the issues of lack of target orientation and other shortcomings in the traditional RRT algorithm when planning collision-free paths, dynamic updating and global adaptive step size strategies were proposed. Simulation experiments conducted using MATLAB software demonstrated that our improved RRT algorithm, compared to the RRT, RRT_informed, and RRT_star algorithms, offered advantages in terms of lower planning time, fewer sampling points, and shorter path lengths in both 2D and 3D map scenarios. Finally, grape-picking experiments were conducted in both a laboratory setting and a real orchard. The results demonstrated that the average path planning time using the proposed algorithm was shorter compared to baseline algorithms, effectively validating the efficiency and practicality of the algorithm.