Collaborative Inspection Research Articles

Application of intelligent self-organizing algorithms in UAV cooperative inspection of power distribution networks

In the rapidly evolving technological landscape, the advent of collaborative Unmanned Aerial Vehicle (UAV) inspections represents a revolutionary leap forward in the monitoring and maintenance of power distribution networks. This innovative approach harnesses the synergy of UAVs working together, marking a significant milestone in enhancing the reliability and efficiency of infrastructure management. Despite its promise, current research in this domain frequently grapples with challenges related to efficient coordination, data processing, and adaptive decision-making under complex and dynamic conditions. Intelligent self-organizing algorithms emerge as pivotal in addressing these gaps, offering sophisticated methods to enhance the autonomy, efficiency, and reliability of UAV collaborative inspections. In response to these challenges, we propose the MARL-SOM-GNNs network model, an innovative integration of Multi-Agent Reinforcement Learning, Self-Organizing Maps, and Graph Neural Networks, designed to optimize UAV cooperative behavior, data interpretation, and network analysis. Experimental results demonstrate that our model significantly outperforms existing approaches in terms of inspection accuracy, operational efficiency, and adaptability to environmental changes. The significance of our research lies in its potential to revolutionize the way power distribution networks are inspected and maintained, paving the way for more resilient and intelligent infrastructure systems. By leveraging the capabilities of MARL for dynamic decision-making, SOM for efficient data clustering, and GNNs for intricate network topology understanding, our model not only addresses current shortcomings in UAV collaborative inspection strategies but also sets a new benchmark for future developments in autonomous infrastructure monitoring, highlighting the crucial role of intelligent algorithms in advancing UAV technologies.

Research on collaborative inspection key technologies for multi-UAVs and multi-nests

The single individual capability and insufficient endurance of unmanned aerial vehicles (UAVs) have always been problems in power inspection operations. This article is based on the new development needs of UAV inspection and proposes a new model of multi-UAVs collaborative inspection. On this basis, this article studies the decoupling and intelligent matching technology between airports and UAVs, task merging and splitting technology in complex multi-task situations, and proposes a safety inspection strategy for multi-UAVs collaborative operations to ensure the safety of operations in the multi-UAVs multi-nests collaborative inspection mode. Finally, a complete multi-UAV, multi-nests collaborative inspection strategy is constructed.

A Voice-Enabled ROS2 Framework for Human–Robot Collaborative Inspection

Quality inspection plays a vital role in current manufacturing practice since the need for reliable and customized products is high on the agenda of most industries. Under this scope, solutions enhancing human–robot collaboration such as voice-based interaction are at the forefront of efforts by modern industries towards embracing the latest digitalization trends. Current inspection activities are often based on the manual expertise of operators, which has been proven to be time-consuming. This paper presents a voice-enabled ROS2 framework towards enhancing the collaboration of robots and operators under quality inspection activities. A robust ROS2-based architecture is adopted towards supporting the orchestration of the process execution flow. Furthermore, a speech recognition application and a quality inspection solution are deployed and integrated to the overall system, showcasing its effectiveness under a case study deriving from the automotive industry. The benefits of this voice-enabled ROS2 framework are discussed and proposed as an alternative way of inspecting parts under human–robot collaborative environments. To measure the added value of the framework, a multi-round testing process took place with different parameters for the framework’s modules, showcasing reduced cycle time for quality inspection processes, robust HRI using voice-based techniques and accurate inspection.

A New Generation of Collaborative Immersive Analytics on the Web: Open-Source Services to Capture, Process and Inspect Users’ Sessions in 3D Environments

Recording large amounts of users’ sessions performed through 3D applications may provide crucial insights into interaction patterns. Such data can be captured from interactive experiences in public exhibits, remote motion tracking equipment, immersive XR devices, lab installations or online web applications. Immersive analytics (IA) deals with the benefits and challenges of using immersive environments for data analysis and related design solutions to improve the quality and efficiency of the analysis process. Today, web technologies allow us to craft complex applications accessible through common browsers, and APIs like WebXR allow us to interact with and explore virtual 3D environments using immersive devices. These technologies can be used to access rich, immersive spaces but present new challenges related to performance, network bottlenecks and interface design. WebXR IA tools are still quite new in the literature: they present several challenges and leave quite unexplored the possibility of synchronous collaborative inspection. The opportunity to share the virtual space with remote analysts in fact improves sense-making tasks and offers new ways to discuss interaction patterns together, while inspecting captured records or data aggregates. Furthermore, with proper collaborative approaches, analysts are able to share machine learning (ML) pipelines and constructively discuss the outcomes and insights through tailored data visualization, directly inside immersive 3D spaces, using a web browser. Under the H2IOSC project, we present the first results of an open-source pipeline involving tools and services aimed at capturing, processing and inspecting interactive sessions collaboratively in WebXR with other analysts. The modular pipeline can be easily deployed in research infrastructures (RIs), remote dedicated hubs or local scenarios. The developed WebXR immersive analytics tool specifically offers advanced features for volumetric data inspection, query, annotation and discovery, alongside spatial interfaces. We assess the pipeline through users’ sessions captured during two remote public exhibits, by a WebXR application presenting generative AI content to visitors. We deployed the pipeline to assess the different services and to better understand how people interact with generative AI environments. The obtained results can be easily adopted for a multitude of case studies, interactive applications, remote equipment or online applications, to support or accelerate the detection of interaction patterns among remote analysts collaborating in the same 3D space.

Enabling human–machine collaboration in infrastructure inspections through mixed reality

In this study, we propose a novel end-to-end system called Human–Machine Collaborative Inspection (HMCI) to enable collaboration between inspectors with Mixed Reality (MR) headsets and a robotic data collection platform (robot) for structural inspections. We utilize the MR headset’s holographic display and precise head tracking to allow inspectors to visualize and localize information (e.g., structural defect) on the real scenes, which are gathered by the robot and processed by an offsite computational server. The primary use case of HMCI is to enable the inspector to visualize, supervise, and improve results produced by automated defect detection algorithms in near real-time. The workflow in HMCI starts with collecting images and depth data to generate 3D maps of the site from the robot. A technique called single-shot localization is developed to create visual anchors for real-time spatial alignment between the robot and the MR headset. The 3D map and images are then sent to the computational server for analysis to detect defects and their locations. Then, the information is received by the MR headset and overlaid on the actual scenes to visualize it with spatial context. An experimental study is conducted in a lab environment to demonstrate HMCI using Microsoft HoloLens 2 (HL2) as the MR headset and Turtlebot2 as the robot. We start with the reconstruction of a 3D environment using a 3D depth sensor (Azure Kinect) on Turtlebot2 and visually detect fiducial markers as regions-of-interest (replicating structural damage) along a predefined inspection path. Then, regions-of-interest are successfully anchored to the real scene and visualized through HL2. To our knowledge, HMCI is one of the first human–machine collaborative systems that can integrate robots and inspectors with the MR headset, which has been developed, tested, and presented for structural inspection.

Designs of human–robot interaction using depth sensor-based hand gesture communication for smart material-handling robot operations

With rapid developments in biometric recognition, a great deal of attention is being paid to robots which interact smartly with humans and communicate certain types of biometrical information. Such human–machine interaction (HMI), also well-known as human–robot interaction (HRI), will, in the future, prove an important development when it comes to automotive manufacturing applications. Currently, hand gesture recognition-based HRI designs are being practically used in various areas of automotive manufacturing, assembly lines, supply chains, and collaborative inspection. However, very few studies are focused on material-handling robot interactions combined with hand gesture communication of the operator. The current work develops a depth sensor-based dynamic hand gesture recognition scheme for continuous-time operations with material-handling robots. The proposed approach properly employs the Kinect depth sensor to extract features of Hu moment invariants from depth data, through which feature-based template match hand gesture recognition is developed. In order to construct continuous-time robot operations using dynamic hand gestures with concatenations of a series of hand gesture actions, the wake-up reminder scheme using fingertip detection calculations is established to accurately denote the starting, ending, and switching timestamps of a series of gesture actions. To be able to perform typical template match on continuous-time dynamic hand gesture recognition with the ability of real-time recognition, representative frame estimates using centroid, middle, and middle-region voting approaches are also presented and combined with template match computations. Experimental results show that, in certain continuous-time periods, the proposed complete hand gesture recognition framework can provide a smooth operation for the material-handling robot when compared with robots controlled using only extractions of full frames; presented representative frames estimated by middle-region voting will maintain fast computations and still reach the competitive recognition accuracy of 90.8%. The method proposed in this study can facilitate the smart assembly line and human–robot collaborations in automotive manufacturing.

Combined inspection strategy of bionic substation inspection robot based on improved Biological Inspired Neural Network

Aiming at the problem of complex and low efficiency of intelligent patrol paths in 220/500 kV bionic substations, this paper proposes a method for collaborative inspection of multiple patrol robots. Firstly, the deficiencies of the Biological Inspired Neural Network algorithm(BINN) in complex paths and large corner scenarios are analyzed, and an improved method for solving the neuron activity value near the boundary and an improved method for the neuron activity value near the obstacle are proposed, in order to solve the phenomenon that the decrease of neurons that can transmit activity values in the neighborhood leads to generally low neuron activity values, which leads to unreasonable path planning; Secondly, establish a substation area segmentation model based on the variable tangent method, divide the substation area into several sub-intervals without obstacles, and use the priority heuristic algorithm to select the robot travel path, so as to realize the multi-robot joint inspection task ; Finally, the method proposed in this paper reduces the path length, number of turns and rotation angle by 11.38%, 38,89% and 20.51, respectively, compared with the BINN algorithm, and by 5.75%, 18.18%, and 16.68 compared with the A* algorithm, which verifies the feasibility and effectiveness of the proposed method.

Joint Inspection of HD Video and Robot in Substation Based on OCR Technology

The research on the joint inspection technology of HD video and robot in substation is of great significance in the operation and maintenance management of substation. The related research on multidimensional collaborative inspection technology of substation has attracted great attention. The substation monitoring system is designed to realize multidimensional collaborative inspection of HD video and inspection robot. OCR technology is used to improve the abnormal recognition system of inspection robot equipment. According to the collection of video information, DSP comprehensive information processing is carried out, and the detection information is analyzed by frequency domain filtering method through human-computer interaction interface. Meanwhile, the main component features of substation detection components are extracted by pattern recognition method, and the resolution model of similar features of video monitoring images is constructed. It plays an important role in the daily operation and maintenance of substation to construct the reliability visual tracking and identification model and optimize the multidimensional cooperative inspection system model.

Robust Trajectory Tracking Control for Underactuated Autonomous Underwater Vehicles in Uncertain Environments

This article addresses the tracking control problem of 3-D trajectories for underactuated underwater robotic vehicles operating in a constrained workspace including obstacles. More specifically, a robust nonlinear model predictive control (NMPC) scheme is presented for the case of underactuated autonomous underwater vehicles (AUVs) (i.e., unicycle-like vehicles actuated only in the surge, heave, and yaw). The purpose of the controller is to steer the unicycle-like AUV to the desired trajectory with guaranteed input and state constraints (e.g., obstacles, predefined vehicle velocity bounds, and thruster saturations) inside a partially known and dynamic environment where the knowledge of the operating workspace is constantly updated via the vehicle’s onboard sensors. In particular, considering the sensing range of the vehicle, obstacle avoidance with any of the detected obstacles is guaranteed by the online generation of a collision-free trajectory tracking path, despite the model dynamic uncertainties and the presence of external disturbances representing ocean currents and waves. Finally, realistic simulation studies verify the performance and efficiency of the proposed framework. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article was motivated by the problem of robust trajectory tracking for an autonomous underwater vehicle (AUV) operating in an uncertain environment where the knowledge of the operating workspace (e.g., obstacle positions) is constantly updated online via the vehicle’s onboard sensors (e.g., multibeam imaging sonars and laser-based vision systems). In addition, there may be other system limitations (e.g., thruster saturation limits) and other operational constraints, induced by the need of various common underwater tasks (e.g., a predefined vehicle speed limit for inspecting the seabed, and mosaicking), where it should also be considered into the control strategy. However, based on the existing trajectory tracking control approaches for underwater robotics, there is a lack of an autonomous control scheme that provides a complete and credible control strategy that takes the aforementioned issues into consideration. Based on this, we present a reliable control strategy that takes into account the aforementioned issues, along with dynamic uncertainties of the model and the presence of ocean currents. In future research, we will extend the proposed methodology for multiple AUV performing collaborative inspection tasks in an uncertain environment.

A collaborative augmented reality annotation tool for the inspection of prefabricated buildings

The inspection of prefabricated buildings involves different stages and tasks such as the collection of measurements, the visual inspection of components and the written annotation of defects. Traditionally, inspectors have documented the process, the kind of defects and the proposed correction measures in paper format, hindering the collaboration with other experts (either simultaneously or asynchronously) and the collection of other types of annotations (e.g. images, 3D elements). In this paper, we present an AR tool designed to aid inspectors during this process. The tool has many benefits, as it allows simultaneously performing a collaborative inspection, taking multitype and geolocated annotations, their monitoring and edition, and performing in situ augmented visualizations. The quantitative and qualitative user evaluation carried out with our tool in a real environment (including usability and satisfaction evaluations) shows the relevance that such a technology might bring to the field and prove that our tool is usable and fulfils most of the inspectors’ expectations.

A Method of Collaborative Inspection Planning by Integrating a Production Planning System

Quality improvement activities have been promoted not only in the inspection sector but also in other sectors such as the production sector with the aim of zero defects under the concept of TQC/TQM. Besides improving quality, quality improvement activities contribute to the improvement of the capabilities of manufacturers to deliver at a lower cost. However, because of the increasing severity of global competition, manufacturers face increased pressure to reduce costs, shorten due dates, and increase quality requirements. Thus, these days, the production sector tends to design the production plan with less slack time to reduce production cost and in-process times. As a result, the flexibility of making the inspection plan by the inspection sector is reduced, which may increase the chance for internal and external failures. In this paper, we propose a method of collaborative inspection planning that integrates the inspection plan with the production plan in the production sector. In the method, slack times are intentionally inserted in the production plan, and inspection levels are selected for the inspection plan by the collaborative efforts in consideration of production cost, due dates, and internal and external failure costs. The method generates the strategy map that shows the expected profit based on the possible combinations of the slack times and the inspection levels. The combinations that maximize the expected profit are selected by using the strategy map, and the inspection plan is made based on the inspection levels. The effectiveness of the proposed method is demonstrated through numerical experiments using a simple manufacturing model.

Multirobot inspection of industrial machinery

Inspection of aircraft and power generation machinery using a swarm of miniature robots is a promising application both from an intellectual and a commercial perspective. Our research is motivated by a case study concerned with the inspection of a jet turbine engine by a swarm of miniature robots. This article summarizes our efforts that include multirobot path planning, modeling of self-organized robotic systems, and the implementation of proof-of-concept experiments with real miniature robots. Although other research tackles challenges that arise from moving within three-dimensional (3-D) structured environments at the level of the individual robotic node, the emphasis of our work is on explicitly incorporating the potential limitations of the individual robotic platform in terms of sensor and actuator noise into the modeling and design process of collaborative inspection systems. We highlight difficulties and further challenges on the (lengthy) path toward truly autonomous parallel robotic inspection of complex engineered structures.

A COLLABORATIVE SOFTWARE CODE INSPECTION: THE DESIGN AND EVALUATION OF A REPEATABLE COLLABORATION PROCESS IN THE FIELD

The use of software products in today's world has increased dramatically making quality an important aspect of software development. There is a continuous need to develop processes to control and increase software quality. Software code inspection is one way to pursue this goal. This paper presents a collaborative code inspection process that was designed during an action research study using Collaboration Engineering principles and techniques. Our inspection process was implemented as a sequence of thinkLets, chunks of facilitation skill, that were subsequently field tested in a traditional paper-based and Group Support System (GSS)-based environment. Four inspections were performed on four different pieces of software code in two different organizations. Results show that regardless of the implementation, the process was found to be successful in uncovering many major, minor, and false-positive defects in inspected pieces of code. Overall observations and feedback suggest that the collaborative inspection process was considered to be productive and satisfactory. GSS inspections were more effective, especially in terms of major defects. GSS inspections were also found to be more efficient. Finally, the GSS inspections outperformed the paper inspections from a practical perspective: logging and managing defects in a GSS was far superior.

Enhancing collaboration in virtual reality applications

We derive a complete component framework for transforming standalone virtual reality (VR) applications into full-fledged multithreaded collaborative virtual reality environments (CVREs), after characterizing existing implementations into a feature-rich superset. Our main contribution is placing over the existing VR tool a very concise and extensible class framework as an add-on component that provides emerging collaboration features. The enhancements include: a scalable arbitrated peer-to-peer topology for scene sharing; multi-threaded components for graphics rendering, user interaction and network communications; a streaming message protocol for client communications; a collaborative user interface model for session handling; and interchangeable user roles with multicamera perspectives, avatar awareness and shared 3D annotations. We validate the framework by converting the existing ALICE VR Navigator into complete CVRE, with experimental results showing good performance in the collaborative inspection and manipulation of complex models.