Omnidirectional Vision Sensor Research Articles

Design of hexapod robot equipped with omnidirectional vision sensor for defect inspection of pipeline’s inner surface

Abstract Defect detection of inner surface of precision pipes is a crucial aspect of ensuring production safety. Currently, pipeline defect detection primarily relies on recording video for manual recognition, with urgent need to improve automation, quantification and accuracy. This paper presents a hexapod in-pipe robot with carrying capacity designed to transport the omnidirectional vision sensor to specified location within unreachable pipelines. The feasibility of the robot’s mechanical design and sensor load-carrying module is analyzed using theory calculations, motion simulations and finite element method. To address the challenges of small pixel ratio and weak background changes in panoramic images, a tiny defect segmentor (TDS) based on ResNet is proposed for detecting tiny defects on the inner surface of pipelines. The hardware and software systems are implemented, and the motion performance of the pipeline robot is validated through experiments. The results demonstrate that the robot achieves stable movement at a speed of over 0.1m/s and can adapt to pipe diameter ranging from of 110 to 130 mm. The novelty of the robot lies in providing stable control of the loaded vision sensor, with control precision of the rotation angle and the displacement recorded at 1.84% and 0.87%, respectively. Furthermore, the proposed method achieves a detection accuracy of 95.67% for tiny defects with a diameter less than 3 mm and provides defect location information. This pipeline robot serves as an essential reference for development of in-pipe 3D vision inspection system.

Spacecraft relative navigation with an omnidirectional vision sensor

With the onset of autonomous spacecraft formation flying missions, the ability of satellites to autonomously navigate relatively to other space objects has become essential. To implement spacecraft relative navigation, relative measurements should be taken, and processed using relative state estimation. An efficient way to generate such information is by using vision-based measurements. Cameras are passive, low-energy, and information-rich sensors that do not actively interact with other space objects. However, pointing cameras with a conventional field-of-view to other space objects requires much a-priori initialization data; in particular, dedicated attitude maneuvers are needed, which may interfere with the satellite’s main mission. One way to overcome these difficulties is to use an omnidirectional vision sensor, which has a 360-degree horizontal field of view. In this work, we present the development of an omnidirectional vision sensor for satellites, which can be used for spacecraft relative navigation, formation flying, and space situational awareness. The study includes the development of the measurement equations, dynamical models, and state estimation algorithms, as well as a numerical study, an experimental investigation, and a space scalability analysis.

REAL TIME SEARCH OF AGRICULTURAL MACHINERY BASED ON MATRIX SEQUENCE SENSOR

Omni-directional vision sensor can provide information within the sensor range, and the directional angle of an object can be accurately obtained through omni-directional images. Based on this characteristic, an automatic navigation and positioning system for agricultural machinery is developed, and a three-dimensional positioning algorithm for agricultural wireless sensor networks based on cross particle swarm optimization is proposed. The method mainly includes three stages: convergence node selection, measurement distance correction and node location. Using the idea of crossover operation of genetic algorithm for reference, the diversity of particles is increased, and the influence of ranging error and the number of anchor nodes on positioning results is effectively improved. The location algorithm has the ability of global search. On the positioning node, the symmetric bidirectional ranging algorithm based on LFM (Linear frequency modulation) spread spectrum technology is used to calculate the distance between the positioning node and each beacon node, and the trilateral centroid positioning algorithm is used to calculate the coordinate position information of unknown nodes. Finally, the Kalman filter algorithm is used to superimpose the observed values of the target state to solve the influence of measurement noise on the positioning accuracy.

Image Sensory Experience of Artistic Design Based on the Role of Omnidirectional Vision Sensors

As a sensor with a wide field of view, the panoramic vision sensor is efficient and convenient in perceiving the characteristic information of the surrounding environment and plays an important role in the experience of artistic design of images. The transformation of visual and other sensory experiences in art design is to integrate sound, image, texture, taste, and smell with each other through reasonable rules, to create more excellent crossborder art design works. To improve the sensory experience that art design works bring to the audience, the combination of vision and other sensory experiences can maximize the advantages of multiple information dissemination methods and combine the omnidirectional visual sensor with the sensory experience of art design images. In the method part, this article introduces the omnidirectional vision sensor, art design image, and sensory experience modes and content and introduces the hyperbolic concave mirror theory and the Micusik perspective projection imaging model. In the experimental part, the experimental environment, experimental objects, and experimental procedures of this article are introduced. In the analysis part, this article analyzes the six aspects of image database dependency test, performance, comparison of different distortion types, false detection rate and missing detection rate, algorithm time‐consuming comparison, sensory experience analysis, and feature point screening. Among the feelings of the art design image, for the first image, 87.21% of the audience’s feelings are happy, indicating that the main idea of this image can bring joy to people. In the second image, the audience’s feelings are mostly sad. For the third image, more than half of the audience’s feelings are melancholy. For the fourth image, 69.34% of the audience’s inner feelings are calm. It explains that the difference in the content of art design images can bring different sensory experiences to people.

Creating Incremental Models of Indoor Environments through Omnidirectional Imaging

In this work, an incremental clustering approach to obtain compact hierarchical models of an environment is developed and evaluated. This process is performed using an omnidirectional vision sensor as the only source of information. The method is structured in two loop closure levels. First, the Node Level Loop Closure process selects the candidate nodes with which the new image can close the loop. Second, the Image Level Loop Closure process detects the most similar image and the node with which the current image closed the loop. The algorithm is based on an incremental clustering framework and leads to a topological model where the images of each zone tend to be clustered in different nodes. In addition, the method evaluates when two nodes are similar and they can be merged in a unique node or when a group of connected images are different enough to the others and they should constitute a new node. To perform the process, omnidirectional images are described with global appearance techniques in order to obtain robust descriptors. The use of such technique in mapping and localization algorithms is less extended than local features description, so this work also evaluates the efficiency in clustering and mapping techniques. The proposed framework is tested with three different public datasets, captured by an omnidirectional vision system mounted on a robot while it traversed three different buildings. This framework is able to build the model incrementally, while the robot explores an unknown environment. Some relevant parameters of the algorithm adapt their value as the robot captures new visual information to fully exploit the features’ space, and the model is updated and/or modified as a consequence. The experimental section shows the robustness and efficiency of the method, comparing it with a batch spectral clustering algorithm.

Relative Altitude Estimation Using Omnidirectional Imaging and Holistic Descriptors

Currently, many tasks can be carried out using mobile robots. These robots must be able to estimate their position in the environment to plan their actions correctly. Omnidirectional vision sensors constitute a robust choice to solve this problem, since they provide the robot with complete information from the environment where it moves. The use of global appearance or holistic methods along with omnidirectional images constitutes a robust approach to estimate the robot position when its movement is restricted to the ground plane. However, in some applications, the robot changes its altitude with respect to this plane, and this altitude must be estimated. This work focuses on this problem. A method based on the use of holistic descriptors is proposed to estimate the relative altitude of the robot when it moves upwards or downwards. This descriptor is constructed from the Radon transform of omnidirectional images captured by a catadioptric vision system. To estimate the altitude, the descriptor of the image captured from the current position is compared with the descriptor of the reference image, previously built. The framework is based on the use of phase correlation to calculate relative orientation and a method based on the compression-expansion of the columns of the holistic descriptor to estimate relative height. Only an omnidirectional vision sensor and image processing techniques are used to solve these problems. This approach has been tested using different sets of images captured both indoors and outdoors under realistic working conditions. The experimental results prove the validity of the method even in the presence of noise or occlusions.

Omnidirectional-Vision-Based Distributed Optimal Tracking Control for Mobile Multirobot Systems With Kinematic and Dynamic Disturbance Rejection

Although various methods of controlling mobile robots have been studied, the distributed tracking control problem for uncertain nonholonomic mobile multirobot (NMMR) systems in an optimal manner with disturbance rejection for both kinematics and dynamics has not been thoroughly solved. This paper, therefore, devotes a novel method to solve the problem with application to real NMMR systems equipped with omnidirectional vision sensors, of which parameters are unknown or uncalibrated. First, the distributed optimal tracking control problem of a separate system in the presence of kinematic and dynamic disturbances is transformed into the equivalent optimal regulation with disturbance rejection of an integrated system. Then, the theory of differential games is utilized to formulate the integrated system into coupled Hamilton–Jacobi–Isaac equations, of which the solutions are approximated in real time by designed algorithms. By the Lyapunov theory, it is proven that the algorithms converge, and the closed-loop systems are stable. Finally, compared simulations and experiments for a group of three robots are provided to show the effectiveness of the proposed algorithms.

Detection of morphology defects in pipeline based on 3D active stereo omnidirectional vision sensor

There are many kinds of defects in pipes, which are difficult to detect with a low degree of automation. In this work, a novel omnidirectional vision inspection system for detection of the morphology defects is presented. An active stereo omnidirectional vision sensor is designed to obtain the texture and depth information of the inner wall of the pipeline in real time. The camera motion is estimated and the space location information of the laser points are calculated accordingly. Then, the faster region proposal convolutional neural network (Faster R-CNN) is applied to train a detection network on their image database of pipe defects. Experimental results demonstrate that system can measure and reconstruct the 3D space of pipe with high quality and the retrained Faster R-CNN achieves fine detection results in terms of both speed and accuracy.

Modeling Environments Hierarchically with Omnidirectional Imaging and Global-Appearance Descriptors

In this work, a framework is proposed to build topological models in mobile robotics, using an omnidirectional vision sensor as the only source of information. The model is structured hierarchically into three layers, from one high-level layer which permits a coarse estimation of the robot position to one low-level layer to refine this estimation efficiently. The algorithm is based on the use of clustering approaches to obtain compact topological models in the high-level layers, combined with global appearance techniques to represent robustly the omnidirectional scenes. Compared to the classical approaches based on the extraction and description of local features, global-appearance descriptors lead to models that can be interpreted and handled more intuitively. However, while local-feature techniques have been extensively studied in the literature, global-appearance ones require to be evaluated in detail to test their efficacy in map-building tasks. The proposed algorithms are tested with a set of publicly available panoramic images captured in realistic environments. The results show that global-appearance descriptors along with some specific clustering algorithms constitute a robust alternative to create a hierarchical representation of the environment.

An Omnidirectional Vision Sensor Based on a Spherical Mirror Catadioptric System.

The combination of mirrors and lenses, which defines a catadioptric sensor, is widely used in the computer vision field. The definition of a catadioptric sensors is based on three main features: hardware setup, projection modelling and calibration process. In this paper, a complete description of these aspects is given for an omnidirectional sensor based on a spherical mirror. The projection model of a catadioptric system can be described by the forward projection task (FP, from 3D scene point to 2D pixel coordinates) and backward projection task (BP, from 2D coordinates to 3D direction of the incident light). The forward projection of non-central catadioptric vision systems, typically obtained by using curved mirrors, is usually modelled by using a central approximation and/or by adopting iterative approaches. In this paper, an analytical closed-form solution to compute both forward and backward projection for a non-central catadioptric system with a spherical mirror is presented. In particular, the forward projection is reduced to a 4th order polynomial by determining the reflection point on the mirror surface through the intersection between a sphere and an ellipse. A matrix format of the implemented models, suitable for fast point clouds handling, is also described. A robust calibration procedure is also proposed and applied to calibrate a catadioptric sensor by determining the mirror radius and center with respect to the camera.

Design of an omnidirectional single-point photodetector for large-scale spatial coordinate measurement

In high precision and large-scale coordinate measurement, one commonly used approach to determine the coordinate of a target point is utilizing the spatial trigonometric relationships between multiple laser transmitter stations and the target point. A light receiving device at the target point is the key element in large-scale coordinate measurement systems. To ensure high-resolution and highly sensitive spatial coordinate measurement, a high-performance and miniaturized omnidirectional single-point photodetector (OSPD) is greatly desired. We report one design of OSPD using an aspheric lens, which achieves an enhanced reception angle of −5 deg to 45 deg in vertical and 360 deg in horizontal. As the heart of our OSPD, the aspheric lens is designed in a geometric model and optimized by LightTools Software, which enables the reflection of a wide-angle incident light beam into the single-point photodiode. The performance of home-made OSPD is characterized with working distances from 1 to 13 m and further analyzed utilizing developed a geometric model. The experimental and analytic results verify that our device is highly suitable for large-scale coordinate metrology. The developed device also holds great potential in various applications such as omnidirectional vision sensor, indoor global positioning system, and optical wireless communication systems.

Development of Height Indicators using Omnidirectional Images and Global Appearance Descriptors

Nowadays, mobile robots have become a useful tool that permits solving a wide range of applications. Their importance lies in their ability to move autonomously through unknown environments and to adapt to changing conditions. To this end, the robot must be able to build a model of the environment and to estimate its position using the information captured by the different sensors it may be equipped with. Omnidirectional vision sensors have become a robust option thanks to the richness of the data they capture. These data must be analysed to extract relevant information that permits estimating the position of the robot taking into account the number of degrees of freedom it has. In this work, several methods to estimate the relative height of a mobile robot are proposed and evaluated. The framework we present is based on the global appearance of the scenes, which has emerged as an efficient and robust alternative comparing to methods based on local features. All the algorithms have been tested with some sets of images captured under real working conditions in several indoor and outdoor spaces. The results prove that global appearance descriptors provide a feasible alternative to estimate topologically the relative altitude of the robot.

Development of a modeling method for monitoring tunnel deformation based on active panoramic vision technique

Aiming at the characteristics of long-distance tunnel, long maintenance intervals, limited repairing time and small changes in the deformation data, this paper presents a detection method of tunnel deformation based on active omni-directional vision tunnel section deformation. Firstly, the active panoramic vision sensor (ASODVS) is installed at the tunnel and detecting device scans the cross section of tunnel to acquire tunnel section panoramic images. Secondly, the sub-pixel center of the panoramic image is extracted by the improved Gauss curve fitting method, and the smooth processing is performed by using the Bezier curves. Thirdly, the system analyzes the geometry information of the tunnel cross-section of the inner wall through calibration results of omnidirectional vision sensor. The tunnel cross section 3D point cloud data are used for 3D reconstruction. Finally, the precision of the tunnel reconstruction model is analyzed. The experimental results show the method has the advantages of high speed acquisition, real-time, comprehensive data and good visualization. It can meet the needs of the rapid qualitative and quantitative analyses.

A State-of-the-Art Review on Mapping and Localization of Mobile Robots Using Omnidirectional Vision Sensors

Nowadays, the field of mobile robotics is experiencing a quick evolution, and a variety of autonomous vehicles is available to solve different tasks. The advances in computer vision have led to a substantial increase in the use of cameras as the main sensors in mobile robots. They can be used as the only source of information or in combination with other sensors such as odometry or laser. Among vision systems, omnidirectional sensors stand out due to the richness of the information they provide the robot with, and an increasing number of works about them have been published over the last few years, leading to a wide variety of frameworks. In this review, some of the most important works are analysed. One of the key problems the scientific community is addressing currently is the improvement of the autonomy of mobile robots. To this end, building robust models of the environment and solving the localization and navigation problems are three important abilities that any mobile robot must have. Taking it into account, the review concentrates on these problems; how researchers have addressed them by means of omnidirectional vision; the main frameworks they have proposed; and how they have evolved in recent years.

Method for SLAM Based on Omnidirectional Vision: A Delayed-EKF Approach

This work presents a method for implementing a visual-based simultaneous localization and mapping (SLAM) system using omnidirectional vision data, with application to autonomous mobile robots. In SLAM, a mobile robot operates in an unknown environment using only on-board sensors to simultaneously build a map of its surroundings, which it uses to track its position. The SLAM is perhaps one of the most fundamental problems to solve in robotics to build mobile robots truly autonomous. The visual sensor used in this work is an omnidirectional vision sensor; this sensor provides a wide field of view which is advantageous in a mobile robot in an autonomous navigation task. Since the visual sensor used in this work is monocular, a method to recover the depth of the features is required. To estimate the unknown depth we propose a novel stochastic triangulation technique. The system proposed in this work can be applied to indoor or cluttered environments for performing visual-based navigation when GPS signal is not available. Experiments with synthetic and real data are presented in order to validate the proposal.

An omnidirectional 3D sensor with line laser scanning

An active omnidirectional vision owns the advantages of the wide field of view (FOV) imaging, resulting in an entire 3D environment scene, which is promising in the field of robot navigation. However, the existing omnidirectional vision sensors based on line laser can measure points only located on the optical plane of the line laser beam, resulting in the low-resolution reconstruction. Whereas, to improve resolution, some other omnidirectional vision sensors with the capability of projecting 2D encode pattern from projector and curved mirror. However, the astigmatism property of curve mirror causes the low-accuracy reconstruction. To solve the above problems, a rotating polygon scanning mirror is used to scan the object in the vertical direction so that an entire profile of the observed scene can be obtained at high accuracy, without of astigmatism phenomenon. Then, the proposed method is calibrated by a conventional 2D checkerboard plate. The experimental results show that the measurement error of the 3D omnidirectional sensor is approximately 1mm. Moreover, the reconstruction of objects with different shapes based on the developed sensor is also verified.

基于主动式全景视觉的管道形貌缺陷检测系统

基于主动式全景视觉的管道形貌缺陷检测系统

Using Omnidirectional Vision to Create a Model of the Environment: A Comparative Evaluation of Global-Appearance Descriptors

Nowadays, the design of fully autonomous mobile robots is a key discipline. Building a robust model of the unknown environment is an important ability the robot must develop. Using this model, this robot must be able to estimate its current position and to navigate to the target points. The use of omnidirectional vision sensors is usual to solve these tasks. When using this source of information, the robot must extract relevant information from the scenes both to build the model and to estimate its position. The possible frameworks include the classical approach of extracting and describing local features or working with the global appearance of the scenes, which has emerged as a conceptually simple and robust solution. While feature-based techniques have been extensively studied in the literature, appearance-based ones require a full comparative evaluation to reveal the performance of the existing methods and to tune correctly their parameters. This work carries out a comparative evaluation of four global-appearance techniques in map building tasks, using omnidirectional visual information as the only source of data from the environment.

基于主动式全景视觉的管道形貌缺陷检测系统

基于主动式全景视觉的管道形貌缺陷检测系统

3D panorama stereo visual perception centering on the observers

For existing three-dimensional (3D) laser scanners, acquiring geometry and color information of the objects simultaneously is difficult. Moreover, the current techniques cannot store, modify, and model the point clouds efficiently. In this work, we have developed a novel sensor system, which is called active stereo omni-directional vision sensor (ASODVS), to address those problems. ASODVS is an integrated system composed of a single-view omni-directional vision sensor and a mobile planar green laser generator platform. Driven by a stepper motor, the laser platform can move vertically along the axis of the ASODVS. During the scanning of the laser generators, the panoramic images of the environment are captured and the characteristics and space location information of the laser points are calculated accordingly. Based on the image information of the laser points, the 3D space can be reconstructed. Experimental results demonstrate that the proposed ASODVS system can measure and reconstruct the 3D space in real-time and with high quality.