Next Best View Research Articles

FF3D: A Rapid and Accurate 3D Fruit Detector for Robotic Harvesting.

This study presents the Fast Fruit 3D Detector (FF3D), a novel framework that contains a 3D neural network for fruit detection and an anisotropic Gaussian-based next-best view estimator. The proposed one-stage 3D detector, which utilizes an end-to-end 3D detection network, shows superior accuracy and robustness compared to traditional 2D methods. The core of the FF3D is a 3D object detection network based on a 3D convolutional neural network (3D CNN) followed by an anisotropic Gaussian-based next-best view estimation module. The innovative architecture combines point cloud feature extraction and object detection tasks, achieving accurate real-time fruit localization. The model is trained on a large-scale 3D fruit dataset and contains data collected from an apple orchard. Additionally, the proposed next-best view estimator improves accuracy and lowers the collision risk for grasping. Thorough assessments on the test set and in a simulated environment validate the efficacy of our FF3D. The experimental results show an AP of 76.3%, an AR of 92.3%, and an average Euclidean distance error of less than 6.2 mm, highlighting the framework's potential to overcome challenges in orchard environments.

RL-NBV: A deep reinforcement learning based next-best-view method for unknown object reconstruction

The Next-Best-View (NBV) algorithm is a key component in autonomous unknown object reconstruction. It iteratively determines the optimal sensor pose to capture the maximum information about the object under reconstruction. However, prevailing deep reinforcement learning (DRL) based NBV algorithms tend to transform point cloud, the raw sensor data, into different representations, thereby obscuring natural invariances of the data. In this work, we propose an innovative DRL-based method, denoted as RL-NBV, to learn NBV policy directly from the raw point cloud data. Specifically, we interpret the observation space as the current state of the reconstructed object represented by point clouds and current view selection states. Experimental results indicate that our method outperforms existing methods in terms of reconstruction performance. Moreover, our method significantly improves efficiency over ray-casting-based algorithms as time-consuming ray casting and data transformation are unnecessary.

Active robotic search for victims using ensemble deep learning techniques

In recent years, legged quadruped robots have proved to be a valuable support to humans in dealing with search and rescue operations. These robots can move with great ability in complex terrains, unstructured environments or regions with many obstacles. This work employs the quadruped robot A1 Rescue Tasks UPM Robot (ARTU-R) by Unitree, equipped with an RGB-D camera and a lidar, to perform victim searches in post-disaster scenarios. Exploration is done not by following a pre-planned path (as common methods) but by prioritising the areas most likely to harbour victims. To accomplish that task, both indirect search and next best view techniques have been used. When ARTU-R gets inside an unstructured and unknown environment, it selects the next exploration point from a series of candidates. This operation is performed by comparing, for each candidate, the distance to reach it, the unexplored space around it and the probability of a victim being in its vicinity. This probability value is obtained using a Random Forest, which processes the information provided by a convolutional neural network. Unlike other AI techniques, random forests are not black box models; humans can understand their decision-making processes. The system, once integrated, achieves speeds comparable to other state-of-the-art algorithms in terms of exploration, but concerning victim detection, the tests show that the resulting smart exploration generates logical paths—from a human point of view—and that ARTU-R tends to move first to the regions where victims are present.

The surface edge explorer (SEE): A measurement-direct approach to next best view planning

High-quality observations of the real world are crucial for a variety of applications, including producing 3D printed replicas of small-scale scenes and conducting inspections of large-scale infrastructure. These 3D observations are commonly obtained by combining multiple sensor measurements from different views. Guiding the selection of suitable views is known as the Next Best View (NBV) planning problem. Most NBV approaches reason about measurements using rigid data structures (e.g., surface meshes or voxel grids). This simplifies next best view selection but can be computationally expensive, reduces real-world fidelity and couples the selection of a next best view with the final data processing. This paper presents the Surface Edge Explorer (SEE), a NBV approach that selects new observations directly from previous sensor measurements without requiring rigid data structures. SEE uses measurement density to propose next best views that increase coverage of insufficiently observed surfaces while avoiding potential occlusions. Statistical results from simulated experiments show that SEE can attain similar or better surface coverage with less observation time and travel distance than evaluated volumetric approaches on both small- and large-scale scenes. Real-world experiments demonstrate SEE autonomously observing a deer statue using a 3D sensor affixed to a robotic arm.

BIMBot for Autonomous Laser Scanning in Built Environments

Accurate and complete 3D point clouds are essential in creating as-built building information modeling (BIM) models, although there are challenges in automating the process for 3D point cloud creation in complex environments. In this paper, an autonomous scanning system named BIMBot is introduced, which integrates advanced light detection and ranging (LiDAR) technology with robotics to create 3D point clouds. Using our specially developed algorithmic pipeline for point cloud processing, iterative registration refinement, and next best view (NBV) calculation, this system facilitates an efficient, accurate, and fully autonomous scanning process. The BIMBot’s performance was validated using a case study in a campus laboratory, featuring complex structural and mechanical, electrical, and plumbing (MEP) elements. The experimental results showed that the autonomous scanning system produced 3D point cloud mappings in fewer scans than the manual method while maintaining comparable detail and accuracy, demonstrating its potential for wider application in complex built environments.

An autonomous Unmanned Aerial Vehicle exploration platform with a hierarchical control method for post‐disaster infrastructures

AbstractCatastrophic natural disasters like earthquakes can cause infrastructure damage. Emergency response agencies need to assess damage precisely while repeating this process for infrastructures with different shapes and types. The authors aim for an autonomous Unmanned Aerial Vehicle (UAV) platform equipped with a 3D LiDAR sensor to comprehensively and accurately scan the infrastructure and map it with a predefined resolution r. During the inspection, the UAV needs to decide on the Next Best View (NBV) position to maximize the gathered information while avoiding collision at high speed. The authors propose solving this problem by implementing a hierarchical closed‐loop control system consisting of a global planner and a local planner. The global NBV planner decides the general UAV direction based on a history of measurements from the LiDAR sensor, and the local planner considers the UAV dynamics and enables the UAV to fly at high speed with the latest LiDAR measurements. The proposed system is validated through the Regional Scale Autonomous Swarm Damage Assessment simulator, which is built by the authors. Through extensive testing in three unique and highly constrained infrastructure environments, the autonomous UAV inspection system successfully explored and mapped the infrastructures, demonstrating its versatility and applicability across various shapes of infrastructure.

Online Scene CAD Recomposition via Autonomous Scanning

Autonomous surface reconstruction of 3D scenes has been intensely studied in recent years, however, it is still difficult to accurately reconstruct all the surface details of complex scenes with complicated object relations and severe occlusions, which makes the reconstruction results not suitable for direct use in applications such as gaming and virtual reality. Therefore, instead of reconstructing the detailed surfaces, we aim to recompose the scene with CAD models retrieved from a given dataset to faithfully reflect the object geometry and arrangement in the given scene. Moreover, unlike most of the previous works on scene CAD recomposition requiring an offline reconstructed scene or captured video as input, which leads to significant data redundancy, we propose a novel online scene CAD recomposition method with autonomous scanning, which efficiently recomposes the scene with the guidance of automatically optimized Next-Best-View (NBV) in a single online scanning pass. Based on the key observation that spatial relation in the scene can not only constrain the object pose and layout optimization but also guide the NBV generation, our system consists of two key modules: relation-guided CAD recomposition module that uses relation-constrained global optimization to get accurate object pose and layout estimation, and relation-aware NBV generation module that makes the exploration during the autonomous scanning tailored for our composition task. Extensive experiments have been conducted to show the superiority of our method over previous methods in scanning efficiency and retrieval accuracy as well as the importance of each key component of our method.

Decentralized Multi-UAV Cooperative Exploration Using Dynamic Centroid-Based Area Partition

Efficient exploration is a critical issue in swarm UAVs with substantial research interest due to its applications in search and rescue missions. In this study, we propose a cooperative exploration approach that uses multiple unmanned aerial vehicles (UAVs). Our approach allows UAVs to explore separate areas dynamically, resulting in increased efficiency and decreased redundancy. We use a novel dynamic centroid-based method to partition the 3D working area for each UAV, with each UAV generating new targets in its partitioned area only using the onboard computational resource. To ensure the cooperation and exploration of the unknown, we use a next-best-view (NBV) method based on rapidly-exploring random tree (RRT), which generates a tree in the partitioned area until a threshold is reached. We compare this approach with three classical methods using Gazebo simulation, including a Voronoi-based area partition method, a coordination method for reducing scanning repetition between UAVs, and a greedy method that works according to its exploration planner without any interaction. We also conduct practical experiments to verify the effectiveness of our proposed method.

Integrated design-sense-plan architecture for autonomous geometric-semantic mapping with UAVs.

This article presents a complete solution for autonomous mapping and inspection tasks, namely a lightweight multi-camera drone design coupled with computationally efficient planning algorithms and environment representations for enhanced autonomous navigation in exploration and mapping tasks. The proposed system utilizes state-of-the-art Next-Best-View (NBV) planning techniques, with geometric and semantic segmentation information computed with Deep Convolutional Neural Networks (DCNNs) to improve the environment map representation. The main contributions of this article are the following. First, we propose a novel efficient sensor observation model and a utility function that encodes the expected information gains from observations taken from specific viewpoints. Second, we propose a reward function that incorporates both geometric and semantic probabilistic information provided by a DCNN for semantic segmentation that operates in close to real-time. The incorporation of semantics in the environment representation enables biasing exploration towards specific object categories while disregarding task-irrelevant ones during path planning. Experiments in both a virtual and a real scenario demonstrate the benefits on reconstruction accuracy of using semantics for biasing exploration towards task-relevant objects, when compared with purely geometric state-of-the-art methods. Finally, we present a unified approach for the selection of the number of cameras on a UAV, to optimize the balance between power consumption, flight-time duration, and exploration and mapping performance trade-offs. Unlike previous design optimization approaches, our method is couples with the sense and plan algorithms. The proposed system and general formulations can be be applied in the mapping, exploration, and inspection of any type of environment, as long as environment dependent semantic training data are available, with demonstrated successful applicability in the inspection of dry dock shipyard environments.

A Shadowcasting-Based Next-Best-View Planner for Autonomous 3D Exploration

In this letter, we address the problem of autonomous exploration of unknown environments with an aerial robot equipped with a sensory set that produces large point clouds, such as LiDARs. The main goal is to gradually explore an area while planning paths and calculating information gain in short computation time, suitable for implementation on an on-board computer. To this end, we present a planner that randomly samples viewpoints in the environment map. It relies on a novel and efficient gain calculation based on the Recursive Shadowcasting algorithm. To determine the Next-Best-View (NBV), our planner uses a cuboid-based evaluation method that results in an enviably short computation time. To reduce the overall exploration time, we also use a dead end resolving strategy that allows us to quickly recover from dead ends in a challenging environment. Comparative experiments in simulation have shown that our approach outperforms the current state-of-the-art in terms of computational efficiency and total exploration time. The video of our approach can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://www.youtube.com/playlist?list =PLC0C6uwoEQ8ZDhny1VdmFXLeTQOSBibQl</uri> .

A Global Max-Flow-Based Multi-Resolution Next-Best-View Method for Reconstruction of 3D Unknown Objects

Many robot tasks, such as grasping and inspection, may require complete 3D models with enough surface details. Fully autonomous environment exploration and 3D reconstruction of unknown objects are challenging for a robot when little or no knowledge about an object is known as a priori. Previous work updated sensor measurements from a greedy ordered set of views into a map with an unchangeable voxel size, leading to the lack of details on the object surface, sampling problems, and no adaption to small objects. We propose a global max-flow-based multi-resolution next-best-view (NBV) method to improve performance on these problems. In particular, it utilizes a max-flow-based global view quality function to obtain optimal NBVs, and a multi-resolution strategy to optimize the reconstruction quality and efficiency. Results of comparative simulation experiments with state-of-the-art (SOTA) methods show that our method achieves a higher voxel coverage under the same resolution. Parametric and ablation studies confirm that the global view function and multi-resolution strategy are effective. Results of real-world experiments show that our method achieves complete reconstruction of small objects with only 6–8 views.

SCVP: Learning One-Shot View Planning via Set Covering for Unknown Object Reconstruction

The view planning (VP) problem in robotic active vision enables a robot system to automatically perform object reconstruction tasks. Lacking prior knowledge, next-best-view (NBV) methods are typically used to plan a view sequence, with the goal of covering as many object surface areas as possible in an unknown environment. However, such methods have two problems: (1) they are unable to perform global path planning; and (2) the reconstruction process is inefficient because of time-consuming ray casting and high movement cost. We propose a neural network, SCVP, to pre-learn prior knowledge via set covering (SC) based training so as to achieve one-shot view planning. The SCVP network takes the volumetric occupancy grid as input and directly predicts a small (ideally minimum) number of views that cover all surface areas. Given object 3D models as a priori geometric knowledge, the training dataset is automatically labeled by the set covering optimization method. We propose a global path planning method to reconstruct objects without redundant movement. Comparative experiments on multiple datasets of 3D models show that, under the condition of similar or better surface coverage, the proposed method can outperform state-of-the-art NBV methods in terms of movement cost and inference time. Real-world experiments confirm that the proposed method can achieve faster object reconstruction than other methods.

Multi-Robot Hybrid Coverage Path Planning for 3D Reconstruction of Large Structures

Coverage Path Planning (CPP) is an essential capability for autonomous robots operating in various critical applications such as fire fighting, and inspection. Performing autonomous coverage using a single robot system consumes time and energy. In particular, 3D large structures might contain some complex and occluded areas that shall be scanned rapidly in certain application domains. In this paper, a new Hybrid Coverage Path Planning (HCPP) approach is proposed to explore and cover unknown 3D large structures using a decentralized multi-robot system. The HCPP approach combines a guided Next Best View (NBV) approach with a developed Long Short Term Memory (LSTM) waypoint prediction approach to decrease the CPP exploration time at each iteration and simultaneously achieve high coverage. The hybrid approach is the new ML paradigm which fosters intelligence by balancing between data efficiency and generality allowing the exchange of some CPP parts with a learned model. The HCPP uses a stateful LSTM network architecture which is trained based on collected paths that cover different 3D structures to predict the next viewpoint. This architecture captures the dynamic dependencies of adjacent viewpoints in the long term sequences like the coverage paths. The HCPP switches between these methods triggered by either the number of iterations or an entropy threshold. In the decentralized multi-robot system, the proposed HCPP is embedded in each robot where each one of them shares its global 3D map ensuring robustness. The results performed in a realistic Gazebo robotic simulator confirmed the advantage of the proposed HCPP approach by achieving high coverage on different 3D unknown structures in a shorter time compared to conventional NBV.

Online coverage and inspection planning for 3D modeling

In this study, we address an exploration problem when constructing complete 3D models in an unknown environment using a Micro-Aerial Vehicle. Most previous exploration methods were based on the Next-Best-View (NBV) approaches, which iteratively determine the most informative view, that exposes the greatest unknown area from the current partial model. However, these approaches sometimes miss minor unreconstructed regions like holes or sparse surfaces (while these can be important features). Furthermore, because the NBV methods iterate the next-best path from a current partial view, they sometimes produce unnecessarily long trajectories by revisiting known regions. To address these problems, we propose a novel exploration algorithm that integrates coverage and inspection strategies. The suggested algorithm first computes a global plan to cover unexplored regions to complete the target model sequentially. It then plans local inspection paths that comprehensively scans local frontiers. This approach reduces the total exploration time and improves the completeness of the reconstructed models. We evaluate the proposed algorithm in comparison with other state-of-the-art approaches through simulated and real-world experiments. The results show that our algorithm outperforms the other approaches and in particular improves the completeness of surface coverage.

Autonomous Scene Exploration for Robotics: A Conditional Random View-Sampling and Evaluation Using a Voxel-Sorting Mechanism for Efficient Ray Casting.

Carrying out the task of the exploration of a scene by an autonomous robot entails a set of complex skills, such as the ability to create and update a representation of the scene, the knowledge of the regions of the scene which are yet unexplored, the ability to estimate the most efficient point of view from the perspective of an explorer agent and, finally, the ability to physically move the system to the selected Next Best View (NBV). This paper proposes an autonomous exploration system that makes use of a dual OcTree representation to encode the regions in the scene which are occupied, free, and unknown. The NBV is estimated through a discrete approach that samples and evaluates a set of view hypotheses that are created by a conditioned random process which ensures that the views have some chance of adding novel information to the scene. The algorithm uses ray-casting defined according to the characteristics of the RGB-D sensor, and a mechanism that sorts the voxels to be tested in a way that considerably speeds up the assessment. The sampled view that is estimated to provide the largest amount of novel information is selected, and the system moves to that location, where a new exploration step begins. The exploration session is terminated when there are no more unknown regions in the scene or when those that exist cannot be observed by the system. The experimental setup consisted of a robotic manipulator with an RGB-D sensor assembled on its end-effector, all managed by a Robot Operating System (ROS) based architecture. The manipulator provides movement, while the sensor collects information about the scene. Experimental results span over three test scenarios designed to evaluate the performance of the proposed system. In particular, the exploration performance of the proposed system is compared against that of human subjects. Results show that the proposed approach is able to carry out the exploration of a scene, even when it starts from scratch, building up knowledge as the exploration progresses. Furthermore, in these experiments, the system was able to complete the exploration of the scene in less time when compared to human subjects.

Automated 3D Reconstruction Using Optimized View-Planning Algorithms for Iterative Development of Structure-from-Motion Models

Unsupervised machine learning algorithms (clustering, genetic, and principal component analysis) automate Unmanned Aerial Vehicle (UAV) missions as well as the creation and refinement of iterative 3D photogrammetric models with a next best view (NBV) approach. The novel approach uses Structure-from-Motion (SfM) to achieve convergence to a specified orthomosaic resolution by identifying edges in the point cloud and planning cameras that “view” the holes identified by edges without requiring an initial model. This iterative UAV photogrammetric method successfully runs in various Microsoft AirSim environments. Simulated ground sampling distance (GSD) of models reaches as low as 3.4 cm per pixel, and generally, successive iterations improve resolution. Besides analogous application in simulated environments, a field study of a retired municipal water tank illustrates the practical application and advantages of automated UAV iterative inspection of infrastructure using 63 % fewer photographs than a comparable manual flight with analogous density point clouds obtaining a GSD of less than 3 cm per pixel. Each iteration qualitatively increases resolution according to a logarithmic regression, reduces holes in models, and adds details to model edges.

Efficient Autonomous Exploration With Incrementally Built Topological Map in 3-D Environments

Autonomous 3-D exploration with unmanned aerial vehicles (UAVs) is increasingly prevalent for environment monitoring without human intervention. In this article, we present a systematic solution toward efficient UAV exploration in 3-D environments. Innovatively, a road map is incrementally built and maintained along with the exploration process, which explicitly exhibits the topological structure of the 3-D environment. By simplifying the environment, the road map can efficiently provide the information gain and the cost-to-go for a candidate region to be explored, which are two quantities for next-best-view (NBV) evaluation, thus prompting the efficiency for NBV determination. In addition, with reference to the global plan queried on the road map, we propose a local planner based on the potential field method that drives the robot to the information-rich area during the navigation process, which further improves the exploration efficiency. The proposed framework and its composed modules are verified in various 3-D environments, which exhibit their distinctive features in NBV selection and better performance in improving the exploration efficiency than other methods.

Victim Localization in USAR Scenario Exploiting Multi-Layer Mapping Structure

Urban search and rescue missions require rapid intervention to locate victims and survivors in the affected environments. To facilitate this activity, Unmanned Aerial Vehicles (UAVs) have been recently used to explore the environment and locate possible victims. In this paper, a UAV equipped with multiple complementary sensors is used to detect the presence of a human in an unknown environment. A novel human localization approach in unknown environments is proposed that merges information gathered from deep-learning-based human detection, wireless signal mapping, and thermal signature mapping to build an accurate global human location map. A next-best-view (NBV) approach with a proposed multi-objective utility function is used to iteratively evaluate the map to locate the presence of humans rapidly. Results demonstrate that the proposed strategy outperforms other methods in several performance measures such as the number of iterations, entropy reduction, and traveled distance.

Guided Next Best View for 3D Reconstruction of Large Complex Structures

In this paper, a Next Best View (NBV) approach with a profiling stage and a novel utility function for 3D reconstruction using an Unmanned Aerial Vehicle (UAV) is proposed. The proposed approach performs an initial scan in order to build a rough model of the structure that is later used to improve coverage completeness and reduce flight time. Then, a more thorough NBV process is initiated, utilizing the rough model in order to create a dense 3D reconstruction of the structure of interest. The proposed approach exploits the reflectional symmetry feature if it exists in the initial scan of the structure. The proposed NBV approach is implemented with a novel utility function, which consists of four main components: information theory, model density, traveled distance, and predictive measures based on symmetries in the structure. This system outperforms classic information gain approaches with a higher density, entropy reduction and coverage completeness. Simulated and real experiments were conducted and the results show the effectiveness and applicability of the proposed approach.

Plant Phenotyping by Deep-Learning-Based Planner for Multi-Robots

Manual plant phenotyping is slow, error prone, and labor intensive. In this letter, we present an automated robotic system for fast, precise, and noninvasive measurements using a new deep-learning-based next-best view planning pipeline. Specifically, we first use a deep neural network to estimate a set of candidate voxels for the next scanning. Next, we cast rays from these voxels to determine the optimal viewpoints. We empirically evaluate our method in simulations and real-world robotic experiments with up to three robotic arms to demonstrate its efficiency and effectiveness. One advantage of our new pipeline is that it can be easily extended to a multi-robot system where multiple robots move simultaneously according to the planned motions. Our system significantly outperforms the single robot in flexibility and planning time. High-throughput phenotyping can be made practically.