Viewpoint Planning Research Articles

Viewpoint Planning of Robotic Measurement System for Free-Form Surfaces Based on Visibility Cone Space Explorer

Viewpoint Planning of Robotic Measurement System for Free-Form Surfaces Based on Visibility Cone Space Explorer

Research on Path Planning Technology of a Line Scanning Measurement Robot Based on the CAD Model

With the development of robotics and vision measurement technology, the use of robots with line laser scanners for 3D scanning and measurement of parts has become a mainstream trend in the field of industrial inspection. Traditional scanning and measuring robots mainly use the teach-in scanning method, which has unstable scanning quality and low scanning efficiency. In this paper, the adaptive sampling method for a free-form surface, which can realize the adaptive distribution of surface measurement points according to the curvature features of free-form surfaces, is proposed first. Then, integrated with the proposed adaptive sampling method, the automatic path planning method is proposed. This method consists of adaptive sampling, scanning attitude calculation based on a quaternion, scanning viewpoint planning based on viewable cones, and scan path generation based on bi-directional scanning. Based on the proposed automatic path planning method, the scanning and measuring robot can obtain complete 3D information of the surface to be measured with high measurement accuracy and efficiency. The performance index of the laser scanner can be fully reached.

Assessing and Improving Automated Viewpoint Planning for Static Laser Scanning Using Optimization Methods

Abstract. The preparation of laser scanning missions is important for efficiency and data quality. Furthermore, it is a prerequisite for automated data acquisition, which has numerous applications in the built environment, including autonomous inspections and monitoring of construction progress and quality criteria. The scene and potential scanning locations can be discretized to facilitate the analysis of visibility and quality aspects. The remaining mathematical problem to generate an economic scan strategy is the Viewpoint Planning Problem (VPP), which asks for a minimum number of scanning locations within the given scene to cover the scene under pre-defined requirements. Solutions for this problem are most commonly found using heuristics. While these efficient methods scale well, they cannot generally return globally optimal solutions. This paper investigates the VPP based on a problem description that considers quality-constrained visibility in 3D scenes and suitable overlaps between individual viewpoints for targetless registration of acquired point clouds. The methodology includes the introduction of a preprocessing method designed to simplify the input data without losing information about the problem. The paper details various solution methods for the VPP, encompassing conventional heuristics and a mixed-integer linear programming formulation, which is solved using Benders decomposition. Experiments are carried out on two case study datasets, varying in specifications and sizes, to evaluate these methods. The results show the actual quality of the obtained solutions and their deviation from optimality (in terms of the estimated optimality gap) for instances where exact solutions can not be achieved.

Fast Multi-View 3D reconstruction of seedlings based on automatic viewpoint planning

Three-dimensional models of plants provide valuable phenotypic information. Phenotypic measurements of plant structures are essential for monitoring plant growth and understanding plant responses to environmental changes. Existing 3D reconstruction techniques have achieved accurate reconstruction models of some plants such as corn and soybeans. However, several drawbacks exist in current plant 3D reconstruction systems, including high cost, fixed capture perspectives and complex operation. To address these issues and considering the influence of camera capture angles on model reconstruction accuracy, we investigated a viewpoint planning reconstruction method. We designed a plant seedling reconstruction system that utilizes a consumer-grade L515 LiDAR sensor and a precision turntable. We establish a viewpoint rule based on the camera imaging model of the three-dimensional spatial structure of leaves, constructing minimum constraints on the leaf normal vector and the camera optical axis. This enables to determine the optimal capture viewpoint for each leaf to obtain the maximum leaf information. The angle β between the leaf normal vector and the camera optical axis serves as the basis for the turntable rotation, systematically planning the rotation of the turntable to enable the camera to capture the maximum useful leaf information. Unlike static camera-based turntable reconstruction systems that capture images at fixed angle intervals, we plan the rotation of the turntable based on the acquired information. This allows us to obtain more effective point cloud information of plant seedlings from the same or even fewer images, reducing the collection of redundant images and reconstructing more accurate plant seedling 3D models. Additionally, we measured commonly used leaf traits in plant phenotypic studies, such as leaf length, leaf width, and leaf area. The leaf area measured based on the reconstructed seedling model exhibited high accuracy (R2 > 0.99). The results of this study demonstrate that the consumer-grade L515 LiDAR sensor combined with the proposed viewpoint planning method can effectively reconstruct accurate seedling 3D models and measure phenotypic information. Due to the absence of error accumulation caused by adjacent view registration, this approach offers advantages such as shorter reconstruction time, higher efficiency and increased accuracy compared to some multi-view point cloud registration and reconstruction methods. Therefore, this system can be applied to three-dimensional reconstruction and phenotype analysis of plant seedlings due to its high efficiency and accuracy.

Object positional uncertainty modelling in 2D viewpoint planning

Viewpoint planning of terrestrial laser scanning surveys is often guided by the level of accuracy, which is a measure of radiated point precision that does not account for all factors affecting positional uncertainty. We propose the use of modelled object precision for viewpoint planning in a methodology that rigorously combines the three sources of positional uncertainty: the observations; registration; and the observation distribution. Our method is demonstrated on three 2D datasets by analyzing object precision in comparison to United States General Services Administration accuracy criterion. The results show the method's ability to 1) represent the precision for all objects in the environment; 2) identify problem areas where the level of accuracy specification is not met; and 3) inform densification measures needed to improve the design. Detailed explanations of the causes of design weaknesses accompany the results to illustrate the challenges of scanning complex built environments and to highlight solutions.

Viewpoint Planning for Range Sensors Using Feature Cluster Constrained Spaces for Robot Vision Systems.

The efficient computation of viewpoints for solving vision tasks comprising multi-features (regions of interest) represents a common challenge that any robot vision system (RVS) using range sensors faces. The characterization of valid and robust viewpoints is even more complex within real applications that require the consideration of various system constraints and model uncertainties. Hence, to address some of the challenges, our previous work outlined the computation of valid viewpoints as a geometrical problem and proposed feature-based constrained spaces (C-spaces) to tackle this problem efficiently for acquiring one feature. The present paper extends the concept of C-spaces to consider multi-feature problems using feature cluster constrained spaces (GC-spaces). A GC-space represents a closed-form, geometrical solution that provides an infinite set of valid viewpoints for acquiring a cluster of features satisfying diverse viewpoint constraints. Furthermore, the current study outlines a generic viewpoint planning strategy based on GC-spaces for solving vision tasks comprising multi-feature scenarios effectively and efficiently. The applicability of the proposed framework is validated on two different industrial vision systems used for dimensional metrology tasks.

Research on Viewpoint Planning Method for Multi-view Image 3D Reconstruction

Research on Viewpoint Planning Method for Multi-view Image 3D Reconstruction

Viewpoint planning and 3D image stitching algorithms for inspection of panels

Non-Destructive Testing using optical methods such as Thermography and Laser Shearography are gaining importance in the inspection of aerospace composite structures due to their non-contact, area based and non-hazardous nature. Here we address the problem of fully covering a curved panel using a robotic system which holds the NDT system up against the panel, and stitching the acquired images together so that the images with defects can be visualized in a single model. A formal way of defining view-points in 3D based on the structure’s 3D model and stitching the gathered image data back onto the 3D model has been lacking in the literature. We present algorithms for defining a set of viewpoints for robotic systems using Area based NDT and for doing full 3D image stitching on the pointcloud generated from the 3D model of the structure. A virtual model or digital twin of the inspection process taking into account the details of the structural panel, robot and camera model is developed and used for getting inputs from the user for the viewpoint planning process as well as displaying the output after image stitching. We show excellent results for synthetic image capturing process as well as good results while using a prototype 5-axis cartesian robot with an IP camera to represent the NDT system.

Viewpoint planning with transition management for active object recognition.

Active object recognition (AOR) provides a paradigm where an agent can capture additional evidence by purposefully changing its viewpoint to improve the quality of recognition. One of the most concerned problems in AOR is viewpoint planning (VP) which refers to developing a policy to determine the next viewpoints of the agent. A research trend is to solve the VP problem with reinforcement learning, namely to use the viewpoint transitions explored by the agent to train the VP policy. However, most research discards the trained transitions, which may lead to an inefficient use of the explored transitions. To solve this challenge, we present a novel VP method with transition management based on reinforcement learning, which can reuse the explored viewpoint transitions. To be specific, a learning framework of the VP policy is first established via the deterministic policy gradient theory, which provides an opportunity to reuse the explored transitions. Then, we design a scheme of viewpoint transition management that can store the explored transitions and decide which transitions are used for the policy learning. Finally, within the framework, we develop an algorithm based on twin delayed deep deterministic policy gradient and the designed scheme to train the VP policy. Experiments on the public and challenging dataset GERMS show the effectiveness of our method in comparison with several competing approaches.

A Brief Study on Lane Detection using Lane Boundary Marker Network

Both advanced driver assistance systems and self-driving automobiles rely heavily on lane detection. While current techniques utilize a scope of highlights from low-level to profound highlights separated from convolutional brain networks, they all experience the ill effects of the issue of impediment and battle to distinguish paths with low or no proof on the street. Key points along the lane boundaries are identified using a lane boundary marker network in this paper. An backwards viewpoint planning is assessed utilizing street geometry which is then applied to the recognized markers and lines/bends are fitted together on the amended focuses. Finally, missing path limits are anticipated utilizing path geometry limitations i.e., equidistant and parallelism. Reciprocal weighted averaging guarantees path limits with solid proof rule their anticipated other options. This demonstrates our algorithm's resistance to occluded and missing lanes. We also demonstrate that our algorithm can be used in conjunction with other lane detectors to enhance their lane retrieval capabilities.

A Digital Twin-Based Intelligent Robotic Measurement System for Freeform Surface Parts

This paper proposed a novel framework of a digital twin-based robotic measurement system for freeform surface parts, which can improve the efficiency, autonomy, and intelligence of the robotic measurement system. This framework provides core services for robotic measurement tasks including visibility computation, collision detection, sensor simulation, path planning, and 3D inspection. These services improve the adaptability and the simulation capability of digital twin-based robot measurement system for freeform surface parts. Based on the proposed framework, an effective viewpoint planning method combining viewpoint sampling strategy and nest-best-view planner (VS-NBV) for freeform surface parts is proposed and verified. And case studies demonstrated the advantages of the digital twin-based robotic measurement system in terms of viewpoint planning, measurement path planning, object reconstruction, and 3D inspection.

Viewpoint planning of surface structured light scanning for complex surface parts

Viewpoint planning of surface structured light scanning for complex surface parts

Optimization Scheme for Online Viewpoint Planning of Active Optical Navigation System in Orthopedic Surgeries

Optical tracking system (OTS) is widely used in robot orthopedic surgeries. During the surgery, the locomotion of the surgical instrument along different osteotomy planes and the adjustment of the patient’s body will lead to obvious variations on the spatial relationship among the positioning tools. Usually, the OTS is manually placed at a fixed viewpoint. As the size of the measurement volume (MV) of the OTS is limited, some positioning tools might move towards the boundaries of the MV, increasing the risk of navigation failure. This paper proposes a two-phase optimization scheme for the active online adjusting of the OTS viewpoint to guarantee the consistency of navigation. A novel optimization structure is proposed with the consideration of continuous state inequality constraints. The dynamic accuracy of the OTS during the active adjusting is also demonstrated to be reliable. Ultimately, simulations and experiments are performed on an active navigation system to verify the effectiveness of the proposed two-phase optimization scheme. Experimental results show that the OTS can be moved to the optimal viewpoint smoothly when confronted with significant variations on the spatial relationship among the positioning tools, guaranteeing the consistency of navigation. The navigation robot successfully keeps the positioning tools around the central region of the MV at different surgical phases.

A Practical Algorithm for the Viewpoint Planning of Terrestrial Laser Scanners

Applications using terrestrial laser scanners (TLS) have been skyrocketing in the past two decades. In a scanning project, the configuration of scans is a critical issue as it has significant effects on the project cost and the quality of the product. In this paper, a practical strategy is proposed to resolve the problem of the optimal placement of the terrestrial laser scanner. The method attempts to reduce the number of viewpoints under the premise that the scenes are fully covered. In addition, the approach is designed in a way that the solutions can be efficiently explored. The method has been tested on 540 polygons simulated with different sizes and complexities. The results have also been compared with a benchmark strategy in terms of the optimality of the solutions and runtime. It is concluded that our proposed algorithm ties or reduces the number of viewpoints in the benchmark paper in 85.6% of the 540 tests. For complex environments, the method can potentially reduce the project cost by 10%. Although with relatively lower efficiency, our method can still reach the solution within a few minutes for a polygon with up to 500 vertices.

Viewpoint Planning for Object Identification Using Visual Experience According to Long-Term Activity

In this paper, we propose a viewpoint planning method for object identification. We introduce the policy of maximizing the posterior probability of the orientation of an object observed after a robot moves its viewpoint and show a novel formulation of viewpoint planning. In addition, we propose criteria for viewpoint selection based on past sensing experience. Finally, we confirm the effectiveness of the proposed method via simulations using a mobile manipulator.

Viewpoint Planning Based on Long-Term Experience for Identification of Object by Mobile Manipulator

Viewpoint Planning Based on Long-Term Experience for Identification of Object by Mobile Manipulator

Continuous Viewpoint Planning in Conjunction with Dynamic Exploration for Active Object Recognition.

Active object recognition (AOR) aims at collecting additional information to improve recognition performance by purposefully adjusting the viewpoint of an agent. How to determine the next best viewpoint of the agent, i.e., viewpoint planning (VP), is a research focus. Most existing VP methods perform viewpoint exploration in the discrete viewpoint space, which have to sample viewpoint space and may bring in significant quantization error. To address this challenge, a continuous VP approach for AOR based on reinforcement learning is proposed. Specifically, we use two separate neural networks to model the VP policy as a parameterized Gaussian distribution and resort the proximal policy optimization framework to learn the policy. Furthermore, an adaptive entropy regularization based dynamic exploration scheme is presented to automatically adjust the viewpoint exploration ability in the learning process. To the end, experimental results on the public dataset GERMS well demonstrate the superiority of our proposed VP method.

Viewpoints Planning for Active 3-D Reconstruction of Profiled Blades Using Estimated Occupancy Probabilities (EOP)

Reconstructing 3-D models of blades plays an increasingly important role in manufacturing. However, the scanning equipment cannot acquire a complete model of a blade with one scanning due to its discontinuous and complex spatial surface. How to plan the viewpoints in presence of the thin walled and free-form properties of blades is a challenging task. In this article, we propose a novel viewpoint planning algorithm using estimated occupancy probabilities. Existing viewpoint planning methods do not consider the enough overlaps, which are important for data registration. We predict surface overlaps using an estimated OctoMap. OctoMap is an efficient probabilistic framework for 3-D occupancy grid mapping. However, the occupancy probabilities of occluded voxels in the original OctoMap remain unknown, which makes it impossible to calculate the overlap of candidate viewpoints. Hence, a recursive Bayesian filter is designed to estimate occupancy probabilities of occluded voxels. With more specific occupancy probabilities, the overlap of a viewpoint is predicted by the ray tracing technology. Experiments on four different profiled blades synthetic datasets show that our algorithm outperforms existing methods in terms of controlling overlap rate. The efficiency of our method is confirmed by real world experiments with low registration failure frequency in reconstruction.

Research on Aerial View Point Planning of Drone based on Multi-View

For multi-view viewpoint planning research, Firstly, using the UAV aerial multi-view image, the viewpoint view is first to establish a candidate view area according to the shooting distance. Secondly, the sub-viewpoint set is filtered and the view points are complemented by the region where the candidate view cannot be photographed, so that the three-dimensional reconstruction data has better integrity. Thirdly, the influence of camera incident angle on image pixel error is analyzed, and the corresponding 3D model is established. Finally, the stereoscopic overlap is used to perform viewpoint screening under the premise of ensuring image quality and shooting coverage. The experimental results show that the proposed viewpoint planning method has high coverage and the reconstruction accuracy is below 0.1% of the target size.

Viewpoint planning for object search on the desk using a mobile robot

Viewpoint planning for object search on the desk using a mobile robot