Robot Operating Research Articles

Deep diffusion learning of mutual-reflective structured light patterns for multi-body three-dimensional imaging

Simultaneous structured light imaging of multiple objects has become more demanding and widely in many scenarios involving robot operations in intelligent manufacturing. However, it is challenged by pattern aliasing caused by mutual reflection between high-reflective objects. To this end, we propose to learn clear fringe patterns from aliased mutual-reflective observations by diffusion models for achieving high-fidelity multi-body reconstruction in line with typical phase-shift algorithms. Regarding mutual reflection imaging as a formation of adding significant noise, we build a supervised generative learning framework based on diffusion models and then train a self-attention-based deep network with a U-Net-like skip-connected encoder-decoder architecture. We demonstrate the generalization capability of the trained model in fringe pattern recovery and its performance in phase and three-dimensional (3D) shape reconstruction. Both experimental results show that the proposed method has the expected feasibility and accuracy, heralding a promising solution for addressing the current challenge in various multi-body mutual-reflective 3D reconstruction tasks.

Endoscopic Robotic Mitral Operating Room as a Microsystem for Safety and Sustainability.

Safety and sustainability are key elements of a robotic mitral valve (MV) program at any stage of development. Challenges include the positioning of the surgeon at the robotic console, increasing patient complexity, and upstream administrative staffing difficulties. We instituted a systems approach to maximize patient safety and maintain robotic service viability. A single dedicated robotic operating room (OR) was equipped as a microsystem with team training in the operative steps, ergonomics, digital tools, and an explicit culture of safety. Outcomes of all robotic mitral procedures including concomitant procedures in the microsystem OR by a single surgeon were retrospectively reviewed. From January 2014 through December 2023, 1,529 consecutive MV patients were operated with an endoscopic robotic approach. Ten patients (0.65%) were converted to conventional approaches. Overall, 1,300 MV repairs (85%) were performed with residual MV regurgitation of none to trace in 1,205 patients (92.7%), mild in 92 patients (7.1%), and moderate in 3 patients (0.23%). MV replacements were performed in 229 patients (15%) with no paravalvular leaks. Mortality was 0.08% in the repair group and 0.87% in the replacement group. No deaths have occurred in the last 38 months. Stroke occurred in 0.31% of repair patients and 1.3% of replacement patients. One patient developed transient renal failure. Organization of the robotic OR as a microsystem is associated with surgical efficacy and very low morbidity and mortality. A comparable microsystem approach using all or select components may promote safety and sustainability for robotic MV programs at all levels.

Deep Learning-Driven Robot Arm Control Fusing Convolutional Visual Perception and Predictive Modeling for Motion Planning

The wide application of robotic technology in various industries from industrial automation to medical assistance is gradually changing our production and lifestyle, and has attracted widespread attention in many fields. However, existing robotic control systems often grapple with limited flexibility and poor adaptability to complex environments, particularly in highly dynamic operational contexts. Against this backdrop, the integration of deep learning technologies offers new possibilities in enhancing robotic perception and decision-making, especially in visual perception and motion planning. To address these challenges, we have introduced a novel robotic arm control network model, MPC-WGAN-Faster R-CNN, which combines Model Predictive Control concepts with Wasserstein Generative Adversarial Networks and Faster R-CNN visual recognition technology. This integration aims to improve the precision and adaptability of robotic arm operations in complex environments.

A Study of Library Mobile Robot Book Classification and Transportation by Integrating DA and RMM

As the complexity of modern library management tasks increases, it is difficult for traditional mobile robots to meet the task of moving and classifying books. In order to design a mobile robot that can autonomously classify and transport books, the study realizes the tasks of book classification and transportation in libraries by fusing the differential speed algorithm and the robot motion model algorithm. First, the robot operating system is utilized to scan the books, classify the books, and obtain the category information of the books. Then, the differential speed algorithm is used to control the motion of the robot to ensure that the robot can accurately transport the books to the designated location. At the same time, combined with the robot motion model algorithm, the motion trajectory of the robot is planned to ensure that the robot can avoid obstacles and stably complete the book transportation task. Finally, the deep reinforcement learning algorithm is used to train the decision-making model of the robot to improve the intelligence level of the robot. The results of simulation experiments show that the research method has the highest accuracy, with an average accuracy of 99.98%, and the robot is able to accurately categorize the books and quickly avoid obstacles with strong stability. The results of the application experiments show that the research method has the shortest moving distance, with an average moving distance of 132 m and an average completion time of 34 seconds, which are lower than the remaining three types of robots. The research robot showed high accuracy in the task of returning books in four time periods within 10 days in the library, with an average accuracy of 99.58%. The experimental results validate the superiority of the research methodology and show that the robots are capable of accurately recognizing and classifying books and can autonomously perform transportation tasks in libraries. The research results help to improve the automation level and management efficiency of libraries and have important application value.

Review and analysis of software simulators for robotic information systems

The rapid development of educational technologies requires the active involvement of new means and effective teaching methods. One of the innovative tools of learning in the field of robotics is the use of robot simulators in the educational process, which open up many opportunities for better assimilation of knowledge and development of practical skills among students. Robot simulators are software complexes that allow you to visualize and simulate the behavior of real robots. With these tools, students can: program robots in a virtual environment, test code and debug without the risk of damaging real equipment; to study robotics and the principles of robot operation, exploring different types of sensors, actuators and algorithms of robot behavior; to solve problems in various fields, such as production automation, logistics, construction, etc.; practices your skills in a safe and controlled environment. Among the benefits of using robotic simulators are next: availability for a large number of students, as simulators are much cheaper than real jobs; safety, as the virtual environment eliminates the risks associated with working with real robots; flexibility that allows students to work at their own pace, explore different scenarios and experiment with different parameters; interactivity with a realistic 3D-environment and visualization. This work presents an overview and analysis of software that allows modeling robot control systems and simulating their operation for various algorithms. A review and analysis of the proposed software tools for simulating robotic systems, which are used to teach bachelor's and master's degree students in the specialties of "Computer Science", "Information Systems", "Industrial Mechanical Engineering" and "Applied Mechanics" and have proven their effectiveness and effectiveness in distance learning at the Kyiv National University of Construction and Architecture.

Compact convolutional transformers- generative adversarial network for compound fault diagnosis of industrial robot

The safe operation of Industrial robots is a major concern in intelligent manufacturing. Accurate compound fault diagnosis is essential to the safe operation of industrial robots, while it is challenging to achieve since the compound fault samples are hard to be collected. Generative adversarial network (GAN) is a useful tool for addressing the data imbalance issue. However, the computation efficiency of GAN in addressing the data imbalance issue has not been investigated. Hence, this study proposes a lightweight GAN named compact convolutional Transformers-GAN (CCT-GAN) to alleviate the data imbalance issue in compound fault diagnosis modelling. Firstly, the feedback current signals collected from the industrial robot are transformed into time-frequency images via continuous wavelet transformation (CWT). Secondly, CCT-GAN is designed to achieve high-quality fake data generation and compound fault diagnosis modelling without large computational costs. Thirdly, the relation between a single fault and the compound fault is considered in the compound fault diagnosis modelling via multi-hot representation to alleviate the data imbalance issue. An experimental study based on the real-world compound fault dataset of industrial robots reveals that the proposed CCT-GAN shows merits in compound fault diagnosis modelling in comparison with the prevailing algorithms. The results indicate that CCT-GAN can performance of compound fault diagnosis when only 100 data samples from each compound fault category are available.

Classification of infection grade for anthracnose in mango leaves under complex background based on CBAM-DBIRNet

Anthracnose is one of the most harmful fungal diseases in mango orchards. Accurate and rapid detection of anthracnose infection grade for mango leaves in complex natural environments is a prerequisite of precise variable pesticide spraying by robots in intelligent orchards. In this paper, A lightweight double branch inverted residual attention network (CBAM-DBIRNet) model was proposed by combining the inverted residual network and the attention mechanism, which was used to detect the infection grade of anthracnose for mango leaves in natural environment. The model designed two parallel residual branch structures to extract multi-scale features of mango leaves with different infection grades through different dilation coefficients. The inverted residual structure and depthwise separable convolution were used to reduce the complexity of the model. The CBAM attention module was integrated into the model to suppress the interference of complex background information. The mango leaves collected in the natural orchard were used for experiments. The results showed that the accuracy, recall and F1 of the CBAM-DBIRNet model for the classification of anthracnose infection grade were 98.42 %, 97.80 % and 97.89 %, respectively. The model size and parameter were 0.64 MB and 0.15 M, respectively. This study provides technical support for smart orchard management and robotic precision variable application operations.

Design and Validation of an Ambulatory User Support Gait Rehabilitation Robot: NIMBLE

Relearning to walk requires progressive training in real scenarios—overground—along with assistance in basic tasks, such as balancing. In addition, user ability must be maximized through compliant robotic assistance as needed. Despite decades of research, gait rehabilitation robotic devices yield controversial results. This article presents the conceptual design of a novel walking assistance and rehabilitation robot, the NIMBLE robot, aimed at providing ambulatory, bodyweight-supported gait training, assisting the user’s center of mass trajectory to aid weight transfer and dynamic balance during walking. NIMBLE consists of a robotic mobile frame, a partial bodyweight support (PBWS) system, an ambulatory lower-limb exoskeleton (Exo-H3) and a cable-driven pelvis-assisting robot. Designed as a modular structure, it differentiates hierarchical communication levels through a Robot Operating System (ROS) 2 network. We present the mechatronic design and experimental results assessing the impact of the mechatronic coupling between the robotic modules on the walking kinematics and the frame movement control performance. The robotic frame hardly affects the walking kinematics up to 2 degrees in both the sagittal and frontal planes, making it feasible for lateral balance and weight translation training. Moreover, it successfully tracks and follows user trajectories. The NIMBLE robotic frame assessment shows promising results for ambulatory gait rehabilitation.

Investigating Human-Robot Interactions with a Quadruped Robot

Robots are becoming more prevalent in our daily lives. As robots become more common in society, we must understand how people react and interact with them. This study leverages Boston Dynamics' Spot robot to examine human responses to quadruped robots. Previous studies had Spot walking alongside participants at fixed speeds, independent of their gait. Improving upon this method, I developed a program that enables Spot to detect and follow fiducial markers (AprilTags). Using the Robot Operating System (ROS), the depth and position of the tag within the camera’s frame were extracted and translated into motion commands for the robot. The tag is attached to an apparatus on the hip that extends the tag near the subject’s knee, allowing Spot to track the participant’s trunk speed and maintain a consistent distance, offering a more adaptive walking experience. The program allows for variations in the acceleration, height, and camera selection (front/back, left/right) through a custom Graphical User Interface (GUI), allowing for diverse trial conditions. Extensive testing identified considerations to optimize the walking experience. Factors included gradual acceleration to allow Spot to react and maintain detections, and the set requirement to run only a single camera at a time for optimized processing. Additionally, keeping the tag unobstructed and square to the camera is critical to avoid failures in detection. Overall, Spot's new capabilities will assist in further studies of human and robot interactions.

Using Mixed Reality for Control and Monitoring of Robot Model Based on Robot Operating System 2

This article presents the design and implementation of an innovative human–machine interface (HMI) in mixed reality for a robot model operating within Robot Operating System 2 (ROS 2). The interface is specifically developed for compatibility with Microsoft HoloLens 2 hardware and leverages the Unity game engine alongside the Mixed Reality Toolkit (MRTK) to create an immersive mixed reality application. The project uses the Turtlebot 3 Burger model robot, simulated within the Gazebo virtual environment, as a representative mechatronic system for demonstration purposes. Communication between the mixed reality application and ROS 2 is facilitated through a publish–subscribe mechanism, utilizing ROS TCP Connector for message serialization between nodes. This interface not only enhances the user experience by allowing for the real-time monitoring and control of the robotic system but also aligns with the principles of Industry 5.0, emphasizing human-centric and inclusive technological advancements. The practical outcomes of this research include a fully functional mixed reality application that integrates seamlessly with ROS 2, showcasing the potential of mixed reality technologies in advancing the field of industrial automation and human–machine interaction.

DewROS2: A platform for informed Dew Robotics in ROS

With the shift from Cloud to Fog and Dew Robotics a lot of emphasis of the research community has been devoted to task offloading. Effective and efficient resource monitoring is however necessary for such offloading and it is also fundamental for other important safety and security tasks. Despite this, robot monitoring has received little attention in general and also for Robot Operating System (ROS) the most employed framework in robotics. In this paper DewROS2 is presented, a platform for Dew Robotics that comprises entities to monitor the system status and to share it with interested applications. The design and implementation of the platform is presented together with the monitoring entities created. DewROS2 has been deployed on different real devices, including an unmanned aerial vehicle and an industrial router, to move from theory to practice and to analyze the impact of monitoring on robot resources. DewROS2 has also been tested in a search and rescue use case where robots are used to collect and transmit videos to spot signs of humans in trouble. Results in controlled and uncontrolled conditions show that the monitoring nodes do not have a significant impact on the performance while providing important and measurable benefits to the applications. Accurately monitoring of robot resources, for example, allows the search and rescue application to almost double the utilization of the network, therefore collecting video at a much higher resolution.

Measuring unit for synchronously collecting air dose rate and measurement position

This paper describes a measuring unit for synchronously collecting the air dose rate and measurement position for efficient dosimetry surveying and data logging. The developed prototype comprises a three-dimensional light detection and ranging-based mapping part and dosimetry part, which are integrated into a single measuring unit through an embedded computer that installs a ROS (robot operating system) framework. The unit can function as a standalone system with embedded batteries. Since it is portable, on-line data gathering in the workspace can be realized, thereby maintaining consistency between the air dose rate and measurement position. In this paper, we describe the functional requirements for the measuring unit, the prototype system configuration, and the experimental results obtained in the mockup environment and nuclear facility to discuss its performance.

Consensus guidelines on the bedside assistant skills required in robotic surgery

BackgroundWhile bedside assistants play a critical role in many robotic operations, substantial heterogeneity remains in bedside assistant training pathways. As such, this study aimed to develop consensus guidelines for bedside assistant skills required for team members in robotic operations.MethodsWe designed a study using the Delphi process to develop consensus guidelines around bedside assistant skills. We generated an initial list of bedside assistant skills from the literature, training materials, and expert input. We selected experts for the Delphi process based on prior scholarship in the area of robotic bedside assistant education and experience facilitating robotic bedside assistant training. For each item, respondents specified which robotic team members should have the skill from a list of “basic” bedside assistants, “advanced” bedside assistants, surgeons, surgical technologists, and circulating nurses. We conducted two rounds of the Delphi process and defined 80% agreement as sufficient for consensus.ResultsFourteen experts participated in two rounds of the Delphi process. By the end of the second round, the group had reached consensus on 253 of 305 items (83%). The group determined that “basic” bedside assistants should have 52 skills and that “advanced” bedside assistants should have 60 skills. The group also determined that surgeons should have 54 skills, surgical technologists should have 25 skills, and circulating nurses should have 17 skills. Experts agreed that all participants should have certain communication skills and basic knowledge of aspects of the robotic system.ConclusionsWe developed consensus guidelines on the skills required during robotic surgery by bedside assistants and other team members using the Delphi process. These findings can be used to design training around bedside assistant skills and assess team members to ensure that each team member has the appropriate skills. Hospitals can also use these guidelines to standardize expectations for robotic team members.

Comparative study of adaptive trajectory tracking controller for four-wheel mobile robot with prescribed-prediction performance

The nonlinear external disturbances and unmodeled dynamics characteristics have crucial impacts on trajectory tracking control accuracy of a four-wheel mobile robot (FWMR) under complicated working conditions. In this work, an adaptive trajectory tracking controller is designed for the FWMR to achieve the prescribed-prediction performance. On the basis of establishing the FWMR’s dynamics equations, an enhanced prescribed performance function (EPPF) is constructed to restrain the tracking errors of the FWMR within a certain range without requiring the exact initial conditions, thus guaranteeing the transient performance of the control system. Then, an optimal-predictive control (OPC) approach is presented to fulfill the asymptotic stability of the tracking errors of the FWMR. Specifically, the radial basis function neural network (RBFNN) incorporating a minimum parameter learning approach that are implanted into the expected controller is designed to attenuate the nonlinear external disturbances and the unmodeled dynamics of the FWMR. Lastly, comparative simulation investigations are carried out to illustrate the superiority of the proposed EPPF-OPC controller, and moreover, the comparative experiments are further performed to validate the practical effectiveness of the EPPF-OPC controller based on a self-established robot operating system (ROS) test platform of the FWMR.

Transforming Agriculture with Advanced Robotic Decision Systems via Deep Recurrent Learning

Global farming methods have improved owing to supply demand, human resource shortages, and seasonal fluctuations. Integrating automated mechanics, smart controllers, and artificial intelligence has become imperative to tackle these problems. Through self-learning and wise decision-making, this integration speeds up processing and encourages sustainability. To help modern farming techniques become more efficient and environmentally friendly, this paper presents a Pliant Decision System (PDS) specifically made for Farming Robots (FRs). The FRs are designed to be time-synchronized and multi-operational, which allows for guided agricultural practices that provide sustainable results. The robotic operations are planned to finish the agricultural processes with individual sustainability verification once they are first classified according to traditional techniques. Two layers of deep recurrent learning are used to evaluate sustainability. Verifying individual completion and sustainability via timely execution and cumulative effect is the primary emphasis of the first layer. In the second layer, the seasonal consistency of production improvements is compared using dual analyses. By training the learning model to synchronize and complete robotic activities promptly, people can execute judgments and synchronize successive agricultural operations throughout the year. The proposed method outwits the existing methods by leveraging sustainable factors by 8.24%, completion by 11.1%, promptness by 7.79%, synchronization by 6.75%, and reducing decision time by 12.07% for the varying process times.

VT-VT: a slip detection model for transformer-based visual-tactile fusion

Slip detection plays a crucial role in robotic operations and has received increasing attention in the field of robotics. We introduced a slip detection model called VT-VT, based on the fusion of vision and touch using a transformer architecture. The model leverages the advantages of a 'divided spatial-temporal attention' mechanism, capturing global contextual information more effectively and exhibiting high sensitivity to potential temporal features during the slipping process. To validate the effectiveness of VT-VT, we conducted experiments on a public slip detection dataset, achieving a test accuracy of up to 90.52%, significantly outperforming a convolutional neural network–long short-term memory model combined with different feature extraction networks. Furthermore, this study revealed the impact of different patch size values, different modal perception modes, and varying lighting conditions on the performance of the VT-VT model. Moreover, we analyzed the real-time capabilities of the VT-VT model.

Centralized multi-robot logistic system: An approach using the island model genetic algorithm as task scheduler

The practicality allied with technological and logistical advances led customers increasingly to join e-commerce. This phenomenon was even more expanded during the COVID-19 pandemic. Because of the growing demand for logistic services, warehouses must adapt and seek faster and more efficient processes. A means to achieve that is to use mobile robots in transportation-related tasks. However, their effectiveness is only achieved if the system has an efficient work distributor. This research developed a centralized coordination architecture using the island model genetic algorithm for multi-robot logistic task allocation. The coordination receives, allocates, and monitors tasks performed by robots with different payload capacities and average speeds. The architecture was developed upon the robotic operating system. Thus, any robot with a robotic operating system-based internal system can work under the coordination of this architecture. On the results, the task scheduler, developed in previous work, had the evaluation complemented. The scheduler based on the island model genetic algorithm allocates more tasks than the standard genetic algorithm when using the same heuristic. Regarding coordination architecture, the system successfully managed a group of robots in a simulated environment. It also detected and notified failures during task execution. Therefore, this system provides a complete task scheduling, allocation, and monitoring solution.

532. OPERATIVE TRENDS AND CLINICAL OUTCOMES OF OPEN, LAPAROSCOPIC AND ROBOTIC APPROACHES TO HIATAL AND PARAOESOPHAGEAL HERNIAS-A STUDY OF 1834 PATIENTS

Abstract Background Indications for repair of hiatal and paraesophageal hernia (HH/PEH) repair evolved significantly over the past two decades, impacted by the increasing application of minimally invasive surgical (MIS) approaches. This transition underscores the importance of evaluating the operative trends and clinical outcomes associated with open (open), laparoscopic (lap), and robotic surgeries especially in cases of giant hernias (over 50% intrathoracic stomach). Understanding these dynamics is crucial, especially as MIS becomes increasingly prevalent in managing complex cases, including giant hernias. The aim of this study is to assess critical outcomes achieved during the evolution from open to lap to robotic HH/PEH repair in a high volume center. Methods This is a retrospective review of 1834 patients undergoing open (958), lap (390), and robot (486), consecutive HH/PEH repairs prospectively entered into an IRB approved database from 2000 to 2023 at a high-volume center. Patient demographics, operative techniques, and top three most common presenting symptoms were studied overtime. Impact of operative technique, hernia type and size, operative outcomes including length of stay (LOS), radiographic recurrence (esophagram at 3 and 12 months), revisional surgery, on post-operative symptom resolution was evaluated. Results Annual volume increased overtime, with 418 repairs (78% robotic) in the last two years. Average hernia size decreased over the study period. The majority of patients (99.1%) presented with symptoms (Fig 1) and robotic repair had significantly better symptomatic resolution than open surgery (p<0.05). This advantage persists with symptoms in giant hernias (p<.001) LOS decreased over the study period (Fig 1). Overall radiographic recurrence (8.6%) and reoperative rate (2.4%) did not differ among operative techniques, however robotic repair had higher radiographic recurrence rate in case of giant hernia (18%, p<.001). Conclusion The operative volume of HH/PEH repair has increased overtime associated with the increased application along with advancements in MIS and robotic operative techniques. Surgical interventions markedly ameliorate HH/PEH symptoms, with greater improvement observed in large hernia cases. Significant improvement in LOS and at least equivalent clinical results favors MIS, particularly robotic surgery. Although the incidence of anatomic recurrent hernia and reoperation is relatively low continued assessment of especially robotic operations to decrease this incidence further is warranted.

Clinical practice guideline for robotic surgical system-assisted thyroid and parathyroid surgery (2024 edition)

The use of robotic operating systems is an advancement of intelligent precision, minimally invasive surgery. It has been used in the field of thyroid surgery with satisfactory results. Robotic surgery system assisted thyroid and parathyroid surgical expert consensus(2016) as played an important role in the standardization and clinical popularization of robotic surgical system-assisted thyroid and parathyroid surgery. With the deepening of clinical practice, updates in minimally invasive concept, the replacement of robotic platforms and the continuous improvement of technology, robotic thyroid and parathyroid surgery has been further developed. Notably, it has made substantial progress in expanding indications and the training of robotic surgeons and teams. Based on the 2016 Chinese expert consensus, combined with recent related articles and clinical studies, the Clinical Practice Guideline for Robotic Surgical System-Assisted Thyroid and Parathyroid Surgery (2024 edition) was formed. The surgical team training, indications, preoperative evaluation, patient position and space establishment, thyroidectomy procedures, neck lymph node dissection skills were summarized and recommended. Furthermore, reasonable suggestions on reoperation, parathyroid surgery and management of postoperative complications were also put forward, aiming to better guide clinical practice.

Multipoint Variable parameter compliant control of redundant manipulator based on the equivalent twin model of flexible contact dynamics

To elucidate the dynamic coupling mechanism involved in multipoint/arbitrary point contact during human–robot interactions with redundant manipulators, we introduce a compliant control strategy that adapts to variable parameters. This strategy is based on the flexible contact dynamic equivalent twin system for human–robot interactions. Using the Virtual Elastic Element Representation model for arbitrary point contact dynamics and leveraging the skeleton principle for contact force conversion, we construct a flexible contact dynamic equivalent twin system tailored for redundant manipulators with multipoint/arbitrary point contact capabilities. Within this system, we implement real-time adjustments to the impedance stiffness coefficient in conjunction with the virtual end velocity of the manipulator. We optimize the joint angular velocity of the manipulator while considering the motion allocation of redundant degrees of freedom. This optimization enables multipoint contact variable parameter compliance control for safe human–robot interactions. To validate our approach, we analyze the multipoint contact dynamic model of the redundant manipulator's interaction with humans and assess the feasibility of the variable parameter compliance control method using robot operating system simulations. Ultimately, we verify the effectiveness and practicality of our proposed compliance control method through simulation and experimental results.