Robot Middleware Research Articles

Inquiring the robot operating system community on the state of adoption of the ROS 2 robotics middleware

AbstractThe robot operating system (ROS) is a well-established and popular open-source robotics middleware used for rapid prototyping of robotic applications. However, ROS has several known weaknesses, such as lack of support for real-time systems and limitations in fully distributed deployments with multiple robots. To address these weaknesses, ROS underwent a major update and the first version of ROS 2 was released back in 2015, being lingeringly adopted by the community in recent years. Yet, long-term support for ROS 1 will come to an end on May 2025. As such, it is more important than ever to analyze and explore the features of ROS 2, and understand its technological readiness. This work focuses on the state of adoption of ROS 2. Specifically, the article presents a user experience questionnaire targeting the ROS community to understand the more eminent needs with respect to ROS 2, determine the current levels of adoption, and identify what is holding the community back from migrating their ROS 1 applications to ROS 2. Results with more than 100 participants from around the world show that the community is aware of ROS 2 and the middleware is addressing most user expectations. We unveil the main reasons given for not migrating to ROS 2, further understand the community’s views on ROS 2, and identify what is missing to anticipate ROS 1 users migrating to ROS 2 and make ROS 2 widely adopted. Moreover, we gather important impressions on the appropriateness of ROS 2 for multi-robot system use cases.

Distributed Petri nets ROS

AbstractVerifying industrial robotic systems is a complex task because those systems are distributed and solely defined by their implementation instead of models of the system to be verified. Some technologies mitigate parts of this problem, e.g., robotic middleware such as the Robotic Operating System (ROS) or concrete solutions such as automata-based specification of robot behavior. However, they all lack the required modeling depth to describe the structure, behavior, and communication of the system. We introduce an improved version of our previous model-driven approach based on Petri nets, integrating these three aspects of ROS-based systems. Using a formal modeling language enables verification of the described system and the generation of complete system parts in the form of ROS nodes. This reduces testing effort because the specification of component workflows and interfaces remains formally proven, while only changed implementations have to be revalidated. We extended our previous approach with novel model transformations, which considerably improved our approach’s performance and memory requirements. We evaluate our approach in a case study involving multiple industrial robotic arms and show that the structure of and communication between ROS nodes can be described and verified.

ManiWare: An Easy-to-Use Middleware for Cooperative Manipulator Teams

Manipulator teams are frequently employed in various industrial applications to handle challenging cooperative tasks. The complicated interaction between manipulators makes it difficult to design applications from scratch. Although robotics middleware has emerged as the key to lowering the development complexity of manipulator applications, existing works still have limitations in controlling multiple manipulators to carry out tasks cooperatively. To overcome the limitations, middleware should provide programming abstraction support, coordination mechanism, and dynamic reconfiguration of motion controllers so that a team of manipulators can work together efficiently. This work proposes ManiWare, an easy-to-use middleware that provides the team-level programming abstraction and the manipulator-level plugin mechanism for programming and configuring manipulator applications. The team-level programming abstraction can facilitate the development process by invoking the functions from the fundamental cooperation components, which drives the developers to focus on designing application logic. Besides, the plugin mechanism dynamically configures and manages the motion controller of different parts of manipulators, making the reconfiguration feasible. This work implements ManiWare and evaluates the task execution performance with three case studies in the high-fidelity simulation platform. The experimental results demonstrate that ManiWare facilitates cooperative tasks with a high success rate, efficient completion time, and marginal runtime overhead. The source code is at.

How can social robot use cases in healthcare be pushed - with an interoperable programming interface

IntroductionResearch into current robot middleware has revealed that most of them are either too complicated or outdated. These facts have motivated the development of a new middleware to meet the requirements of usability by non-experts. The proposed middleware is based on Android and is intended to be placed over existing robot SDKs and middleware. It runs on the android tablet of the Cruzr robot. Various toolings have been developed, such as a web component to control the robot via a webinterface, which facilitates its use.MethodsThe middleware was developed using Android Java and runs on the Cruzr tablet as an app. It features a WebSocket server that interfaces with the robot and allows control via Python or other WebSocket-compatible languages. The speech interface utilizes Google Cloud Voice text-to-speech and speech-to-text services. The interface was implemented in Python, allowing for easy integration with existing robotics development workflows, and a web interface was developed for direct control of the robot via the web.ResultsThe new robot middleware was created and deployed on a Cruzr robot, relying on the WebSocket API and featuring a Python implementation. It supports various robot functions, such as text-to-speech, speech-to-text, navigation, displaying content and scanning bar codes. The system’s architecture allows for porting the interface to other robots and platforms, showcasing its adaptability. It has been demonstrated that the middleware can be run on a Pepper robot, although not all functions have been implemented yet. The middleware was utilized to implement healthcare use cases and received good feedback.ConclusionCloud and local speech services were discussed in regard to the middleware’s needs, to run without having to change any code on other robots. An outlook on how the programming interface can further be simplified by using natural text to code generators has been/is given. For other researchers using the aforementioned platforms (Cruzr, Pepper), the new middleware can be utilized for testing human-robot interaction. It can be used in a teaching setting, as well as be adapted to other robots using the same interface and philosophy regarding simple methods.

A BPMN-driven framework for Multi-Robot System development

Programming robotic systems is often a challenging task requiring advanced skills, especially when the goal is to ensure loosely-coupled coordination in heterogeneous Multi-Robot Systems (MRSs). Model-driven approaches for robotic system engineering have shown their benefits in facilitating the development of robots’ behavior, controllers, and system components. However, the state of the art still lacks contributions addressing crucial aspects of the model-driven approach applied to MRSs, such as developing robots’ distributed cooperation through models supporting the communication among robots.In this paper, we present a novel framework for modeling, configuring and enacting the cooperative behaviors of MRSs through collaboration diagrams as provided by the BPMN 2.0 standard. The advantages of our solution lie, indeed, in the use of BPMN, which provides easily understandable and highly expressive diagrams for representing the cooperation among distributed robots, and benefits from a wide list of supporting tools. Starting from the selection of BPMNelements, we define a set of guidelines for driving the developer in modeling an MRS mission using BPMN. The developer configures the resulting collaboration diagram to link elements in the model to the robotic middleware, ROS2 in the toolchain we implemented. Finally, the configured model is enacted by BPMN engines integrated into the ROS2 middleware run by each robot involved in the MRS, thus obtaining a fully distributed cooperation. We assess our framework’s effectiveness through experiments in simulated and real environments.

SonifyIt: Towards Transformative Sound for All Robots

Transformative robot sound yields perceptual, functional, and social benefits in human-robot interactions, but broader research and implementation related to this topic is impeded by the lack of a common sound generation system for robots. Such a system could enable a wide array of situated robot sound studies, smoother collaborations with sound designers than current state of the art methods, and broader adoption of transformative robot sound. Based on other successful open-source projects in the robotics community, we integrated Robot Operating System, a popular robotics middleware, and Pure Data, a visual programming language for multimedia, to enable live sound synthesis and sample playback for robots. This sound generation system synthesized sound in an in-the-wild pilot study with positive qualitative results. Furthermore, an online within-subjects survey study with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N=96$</tex-math></inline-formula> showed that the proposed sound system made the robot seem warmer, happier, and more energetic. This work benefits robotics researchers by providing the current sound system as a validated artifact and demonstrating its potential impact on broader robotics applications. We plan to develop this software into an open-source package: SonifyIt.

Scalable and heterogenous mobile robot fleet-based task automation in crowded hospital environments-a field test.

In hospitals, trained medical staff are often, in addition to performing complex procedures, spending valuable time on secondary tasks such as transporting samples and medical equipment; or even guiding patients and visitors around the premises. If these non-medical tasks were automated by deploying mobile service robots, more time can be focused on treating patients or allowing well-deserved rest for the potentially overworked healthcare professionals. Automating such tasks requires a human-aware robotic mobility system that can among other things navigate the hallways of the hospital; predictively avoid collisions with humans and other dynamic obstacles; coordinate task distribution and area coverage within a fleet of robots and other IoT devices; and interact with the staff, patients and visitors in an intuitive way. This work presents the results, lessons-learned and the source code of deploying a heterogeneous mobile robot fleet at the Tartu University Hospital, performing object transportation tasks in areas of intense crowd movement and narrow hallways. The primary use-case is defined as transporting time-critical samples from an intensive care unit to the hospital lab. Our work builds upon Robotics Middleware Framework (RMF), an open source, actively growing and highly capable fleet management platform which is yet to reach full maturity. Thus this paper demonstrates and validates the real-world deployment of RMF in an hospital setting and describes the integration efforts.

Developing Intelligent Robots that Grasp Affordance.

Humans and robots operating in unstructured environments both need to classify objects through haptic exploration and use them in various tasks, but currently they differ greatly in their strategies for acquiring such capabilities. This review explores nascent technologies that promise more convergence. A novel form of artificial intelligence classifies objects according to sensory percepts during active exploration and decides on efficient sequences of exploratory actions to identify objects. Representing objects according to the collective experience of manipulating them provides a substrate for discovering causality and affordances. Such concepts that generalize beyond explicit training experiences are an important aspect of human intelligence that has eluded robots. For robots to acquire such knowledge, they will need an extended period of active exploration and manipulation similar to that employed by infants. The efficacy, efficiency and safety of such behaviors depends on achieving smooth transitions between movements that change quickly from exploratory to executive to reflexive. Animals achieve such smoothness by using a hierarchical control scheme that is fundamentally different from those of conventional robotics. The lowest level of that hierarchy, the spinal cord, starts to self-organize during spontaneous movements in the fetus. This allows its connectivity to reflect the mechanics of the musculoskeletal plant, a bio-inspired process that could be used to adapt spinal-like middleware for robots. Implementation of these extended and essential stages of fetal and infant development is impractical, however, for mechatronic hardware that does not heal and replace itself like biological tissues. Instead such development can now be accomplished in silico and then cloned into physical robots, a strategy that could transcend human performance.

Zoro: A robotic middleware combining high performance and high reliability

With the significant advances of AI technology, robotic systems have achieved remarkable development and profound effects. The robotic middleware plays a critical role in robotic systems to provide message definition, data transmission, and service discovery functions. As the modern robotic systems are usually operated in safety critical environments, such as various autonomous driving scenarios, it requires the design of robotic middleware to combine both high performance and high reliability in order to achieve system reliability and product safety. However, conventional robotic middleware used in the majority of robotic systems is based on an inefficient socket communication mechanism and relies on a hasty service discovery design, which leads to system instability and high resource usage. In this work, we propose a sophisticated robotic middleware, Zoro, to fulfill both high performance and high reliability. For communication, we employ shared memory for performance improvement and propose a socket-based communication control algorithm to improve reliability during data transmission. Also, a hierarchical memory protection mechanism is proposed to address safety problems caused by shared memory. Furthermore, we design a light-weight service discovery, to achieve high performance and a weak centralized mechanism for high reliability. Experiments show the communication latency of Zoro significantly outperforms state-of-the-art robotic middleware such as ROS2 and CyberRT by up to 41%. In terms of service discovery, Zoro reduces CPU usage by up to 44% compared to ROS. Zoro achieves reliability with respect to communication and service discovery.

GraphWare: A graph‐based middleware enabling multi‐robot cooperation

SummaryMulti‐robot systems are widely used to handle complex and cooperative missions in various industrial applications. Although robotic middleware has become the key to reducing the complexity of multi‐robot application development, existing works still have limitations in controlling multiple robots to perform missions cooperatively. To enable multi‐robot cooperation, middleware should provide high‐level abstraction support, dynamic configuration, communication, and synchronization. In this article, we propose GraphWare, a novel middleware that provides a graph‐based programming abstraction and its underlying runtime kernel for programming and building multi‐robot cooperation applications. The graph‐based programming abstraction can express cooperative missions without exposing the complexity of managing multiple robots. The runtime kernel configures and manages multiple heterogeneous robots to intelligently perform cooperative missions. We implement GraphWare and evaluate its performance with ball collection missions which are cooperatively accomplished by a group of mobile robots, and study the fault‐tolerance, flexibility, and scalability of the middleware in the realistic simulation. The experimental results demonstrate that GraphWare facilitates the multi‐robot cooperative mission with efficient mission completion time, high success rate, and marginal runtime overhead.

Research on AI (Artificial Intelligence) with the Robotic Arm

Robot operating system (ROS) is robot middleware. Although ROS is not an operating system, it provides the necessary services designed for a heterogeneous cluster of servers, controlling low-end devices, deploying functions, transmitting messages, and managing low-end devices. A set of ROS-based processes represented as architectural graphs that can receive and concatenate sensor data, state control, scheduling, actuators, and other messages. The application of ROS to the product is in order to test and simulate the operation of the robot arm, thereby supporting researchers, improving, developing and deploying applications on robots smoothly. Stemming from the above analysis, the topic completely meets the urgency posed by modern society.

Robotics Middleware Extensions for Support Various Communication Protocols

Robotics Middleware Extensions for Support Various Communication Protocols

Survey on Cloud Robotics and Robot Operating System

The paper proposes cloud robotics which intends five parts which are for software architecture, trends and challenges, design based on a web technology, cloud computing platforms and cloud robotics in practical smart city applications. Software architecture involves lots of different cloud technologies like cloud storage and shared services for robotics for Robot Operating System. There are lots of trends in cloud robotics as it uses cloud technologies for robotics. The use of the cloud for automation brings some largely ameliorating the performance of robotics. Besides, the trends, there are some challenges in cloud robotics. Robot Operating System has been a standard tool in the design for cloud robotics. It always runs at the on-board computer but some resources of CPU do not allow the ability of robotics. In this problem, a good solution is to transfer the task to the cloud. Hence, the design of a cloud robotics middleware is based on web technology. Robotics with cloud computing gives lots of research scopes and research fields. In Smart City Applications, Real-Time Cloud Robotics is present. Real Time Cloud Robotics has emerged as an efficient computing means in data processing applications. By this we introduce CR20 cloud robotics platform that uses real time stream technology virtually to connect to different devices.

CHORD: Distributed Data-Sharing via Hybrid ROS 1 and 2 for Multi-Robot Exploration of Large-Scale Complex Environments

A well-structured and reliable communication system is key to the successful operations of multi-robot systems. In this letter, we present our design and implementation of a multi-robot communication architecture CHORD (Collaborative High-bandwidth Operations with Radio Droppables) based on two popular robotics middleware, ROS 1 and ROS 2. We discuss the benefit and best practices of combining two different frameworks that share the same spirit and show its performance from large-scale real-world experiments. The proposed system is developed as part of Team CoSTAR's effort for the DARPA Subterranean (SubT) Challenge. The system has been field-proved and demonstrated in the Urban Circuit event, where team CoSTAR won first place. To our knowledge, this work is the first real-world demonstration of a ROS 2-based multi-robot system in such large-scale extreme environments. From the significant improvement of the communication performance and the ease of transition from existing ROS 1 systems, this work encourages wider adoption of ROS 2 in field robotics applications.

Development of a Basic Educational Kit for Robotic System with Deep Neural Networks.

In many robotics studies, deep neural networks (DNNs) are being actively studied due to their good performance. However, existing robotic techniques and DNNs have not been systematically integrated, and packages for beginners are yet to be developed. In this study, we proposed a basic educational kit for robotic system development with DNNs. Our goal was to educate beginners in both robotics and machine learning, especially the use of DNNs. Initially, we required the kit to (1) be easy to understand, (2) employ experience-based learning, and (3) be applicable in many areas. To clarify the learning objectives and important parts of the basic educational kit, we analyzed the research and development (R&D) of DNNs and divided the process into three steps of data collection (DC), machine learning (ML), and task execution (TE). These steps were configured under a hierarchical system flow with the ability to be executed individually at the development stage. To evaluate the practicality of the proposed system flow, we implemented it for a physical robotic grasping system using robotics middleware. We also demonstrated that the proposed system can be effectively applied to other hardware, sensor inputs, and robot tasks.

Craft to site

The paper presents a craft-oriented middleware for assisting in situ robotic fabrication. This middleware addresses the need for sensor-based feedback in robotic fabrication involving uncertain materials, non-structured environments, or post-production surfaces. The presented middleware addresses these fabrication challenges by allowing the robot to react to its environment without the need for a predetermined program. The middleware and its components are presented, followed by experiments demonstrating in situ capacities. The first experiment demonstrates autonomous carving—performing an iterative, non-prescribed robotic fabrication process. The second experiment presents integrated fabrication—a sequence of subtractive and additive techniques performed on a non-even substrate. Together, these experiments contribute to the implementation of sensible robotic middleware in architecture and assist in restoring the lost link between workmanship and construction sites.

RESTCub: A Language Independent Middleware for Cognitive Robot

Since the last few decades, research in the area of robotics technology has been emphasizing in the modeling and development of cognitive machines. A cognitive machine can have multiple cognitive capabilities to be programmed to make it artificially intelligent. Numerous cognitive modules interact to mimic human behavior in machines and result in such a heavily coupled system that a minor change in logic or hardware may affect a large number of its modules. To address such a problem, several middlewares exist to ease the development of cognitive machines. Although these layers decouple the process of logic building and communication infrastructure of modules, they are language-dependent and have their limitations. A cognitive module developed for one research work cannot be a part of another research work resulting in the re-invention of the wheel. This paper proposes a RESTful technology-based framework that provides language-independent access to low-level control of the iCub’s sensory-motor system. Moreover, the model is flexible enough to provide hybrid communications between cognitive modules running on different platforms and operating systems. Furthermore, a cognitive client is developed to test the proposed model. The experimental analysis performed by creating different scenarios shows the effectiveness of the proposed framework.

Interoperability Among Unmanned Maritime Vehicles: Review and First In-field Experimentation.

Complex maritime missions, both above and below the surface, have traditionally been carried out by manned surface ships and submarines equipped with advanced sensor systems. Unmanned Maritime Vehicles (UMVs) are increasingly demonstrating their potential for improving existing naval capabilities due to their rapid deployability, easy scalability, and high reconfigurability, offering a reduction in both operational time and cost. In addition, they mitigate the risk to personnel by leaving the man far-from-the-risk but in-the-loop of decision making. In the long-term, a clear interoperability framework between unmanned systems, human operators, and legacy platforms will be crucial for effective joint operations planning and execution. However, the present multi-vendor multi-protocol solutions in multi-domain UMVs activities are hard to interoperate without common mission control interfaces and communication protocol schemes. Furthermore, the underwater domain presents significant challenges that cannot be satisfied with the solutions developed for terrestrial networks. In this paper, the interoperability topic is discussed blending a review of the technological growth from 2000 onwards with recent authors' in-field experience; finally, important research directions for the future are given. Within the broad framework of interoperability in general, the paper focuses on the aspect of interoperability among UMVs not neglecting the role of the human operator in the loop. The picture emerging from the review demonstrates that interoperability is currently receiving a high level of attention with a great and diverse deal of effort. Besides, the manuscript describes the experience from a sea trial exercise, where interoperability has been demonstrated by integrating heterogeneous autonomous UMVs into the NATO Centre for Maritime Research and Experimentation (CMRE) network, using different robotic middlewares and acoustic modem technologies to implement a multistatic active sonar system. A perspective for the interoperability in marine robotics missions emerges in the paper, through a discussion of current capabilities, in-field experience and future advanced technologies unique to UMVs. Nonetheless, their application spread is slowed down by the lack of human confidence. In fact, an interoperable system-of-systems of autonomous UMVs will require operators involved only at a supervisory level. As trust develops, endorsed by stable and mature interoperability, human monitoring will be diminished to exploit the tremendous potential of fully autonomous UMVs.

A hardware intelligent processing accelerator for domestic service robots

ABSTRACT We present a method for implementing hardware intelligent processing accelerator on domestic service robots. These domestic service robots support human life; therefore, they are required to recognize environments using intelligent processing. Moreover, the intelligent processing requires large computational resources. Therefore, standard personal computers (PCs) with robot middleware on the robots do not have enough resources for this intelligent processing. We propose a ‘connective object for middleware to an accelerator (COMTA),’ which is a system that integrates hardware intelligent processing accelerators and robot middleware. Herein, by constructing dedicated architecture digital circuits, field-programmable gate arrays (FPGAs) accelerate intelligent processing. In addition, the system can configure and access applications on hardware accelerators via a robot middleware space; consequently, robotic engineers do not require the knowledge of FPGAs. We conducted an experiment on the proposed system by utilizing a human-following application with image processing, which is commonly applied in the robots. Experimental results demonstrated that the proposed system can be automatically constructed from a single-configuration file on the robot middleware and can execute the application 5.2 times more efficiently than an ordinary PC.

BOLeRo: Behavior optimization and learning for robots

Reinforcement learning and behavior optimization are becoming more and more popular in the field of robotics because algorithms are mature enough to tackle real problems in this domain. Robust implementations of state-of-the-art algorithms are often not publicly available though, and experiments are hardly reproducible because open-source implementations are often not available or are still in a stage of research code. Consequently, often it is infeasible to deploy these algorithms on robotic systems. BOLeRo closes this gap for policy search and evolutionary algorithms by delivering open-source implementations of behavior learning algorithms for robots. It is easy to integrate in robotic middlewares and it can be used to compare methods and develop prototypes in simulation.