Omnidirectional Vision System Research Articles

Client-Oriented Blind Quality Metric for High Dynamic Range Stereoscopic Omnidirectional Vision Systems

A high dynamic range (HDR) stereoscopic omnidirectional vision system can provide users with more realistic binocular and immersive perception, where the HDR stereoscopic omnidirectional image (HSOI) suffers distortions during its encoding and visualization, making its quality evaluation more challenging. To solve the problem, this paper proposes a client-oriented blind HSOI quality metric based on visual perception. The proposed metric mainly consists of a monocular perception module (MPM) and binocular perception module (BPM), which combine monocular/binocular, omnidirectional and HDR/tone-mapping perception. The MPM extracts features from three aspects: global color distortion, symmetric/asymmetric distortion and scene distortion. In the BPM, the binocular fusion map and binocular difference map are generated by joint image filtering. Then, brightness segmentation is performed on the binocular fusion image, and distinctive features are extracted on the segmented high/low/middle brightness regions. For the binocular difference map, natural scene statistical features are extracted by multi-coefficient derivative maps. Finally, feature screening is used to remove the redundancy between the extracted features. Experimental results on the HSOID database show that the proposed metric is generally better than the representative quality metric, and is more consistent with the subjective perception.

A Hybrid Calibration Method for the Binocular Omnidirectional Vision System

Omnidirectional three-dimensional (3D) measurement based on the binocular omnidirectional vision system attracts increasing attention in computer vision. A 360° horizontal field of view (FOV) and 180° vertical FOV are obtained by the binocular omnidirectional vision system with only a pair of images. However, a nonlinear relationship exists between points on image planes and in 3D space for the binocular omnidirectional vision system, and traditional calibration methods are no longer applicable. This study proposes a hybrid calibration approach, which effectively fuses the unified camera model and the backpropagation neural network with a virtual 3D target, in which the neural network improved with genetic algorithm compensates all kinds of distortions and errors. First, the unified camera model is used to calibrate the intrinsic parameters, extrinsic parameters, and a transformation matrix. Second, the pixel coordinates on image planes are converted to 3D coordinates by 3D reconstruction through binocular vision. The errors between actual and calculated 3D coordinates are obtained. In addition, the neural network improved with genetic algorithm is established and trained with pixel coordinates on image planes and errors. Experimental results indicate that the average measurement errors in the X and Y directions of the central area are respectively reduced to 0.1057 and 0.1548 mm, and those of the edge area are reduced to 0.1992 and 0.2748 mm. Compared with mathematical and neural network methods, the hybrid calibration method provides more stable, robust, and accurate results. It is proven suitable for complex calibration scenes with large FOV, high distortion, and strong nonlinearity.

Calibration and three-dimensional reconstruction with a photorealistic simulator based on the omnidirectional vision system

Recent advancements in deep learning require a large amount of the annotated training data containing various terms and conditions of the environment. Thus, developing and testing algorithms for the navigation of mobile robots can be expensive and time-consuming. Motivated by the aforementioned problems, this article presents a photorealistic simulator for the computer vision community working with omnidirectional vision systems. Built using unity, the simulator integrates sensors, mobile robots, and elements of the indoor environment and allows one to generate synthetic photorealistic data sets with automatic ground truth annotations. With the aid of the proposed simulator, two practical applications are studied, namely extrinsic calibration of the vision system and three-dimensional reconstruction of the indoor environment. For the proposed calibration and reconstruction techniques, the processes themselves are simple, robust, and accurate. Proposed methods are evaluated experimentally with data generated by the simulator. The proposed simulator and supporting materials are available online: http://www.ilabit.org .

Edge detection algorithm for omnidirectional images, based on superposition laws on Blach’s sphere and quantum entropy

This paper presents an edge detection algorithm for omnidirectional images based on superposition law on Bloch’s sphere and quantum local entropy. Omnidirectional vision system has become an essential tool in computer vision, duo to its large field of view. However, classical image processing algorithms are not suitable to be applied directly in this type of images without taking into account the spatial information around each pixel. To show the performance of the proposed method, a set of experimentation was done on synthetic and real images devoted to agriculture applications. Later, Fram & Deutsh criterion has been adopted to evaluate its performance against three algorithms proposed on the literature and developed for omnidirectional images. The results show a good performance of the proposed method in term of edge quality, edge community and sensibility to noise.

The Role of Global Appearance of Omnidirectional Images in Relative Distance and Orientation Retrieval.

Over the last few years, mobile robotics has experienced a great development thanks to the wide variety of problems that can be solved with this technology. An autonomous mobile robot must be able to operate in a priori unknown environments, planning its trajectory and navigating to the required target points. With this aim, it is crucial solving the mapping and localization problems with accuracy and acceptable computational cost. The use of omnidirectional vision systems has emerged as a robust choice thanks to the big quantity of information they can extract from the environment. The images must be processed to obtain relevant information that permits solving robustly the mapping and localization problems. The classical frameworks to address this problem are based on the extraction, description and tracking of local features or landmarks. However, more recently, a new family of methods has emerged as a robust alternative in mobile robotics. It consists of describing each image as a whole, what leads to conceptually simpler algorithms. While methods based on local features have been extensively studied and compared in the literature, those based on global appearance still merit a deep study to uncover their performance. In this work, a comparative evaluation of six global-appearance description techniques in localization tasks is carried out, both in terms of accuracy and computational cost. Some sets of images captured in a real environment are used with this aim, including some typical phenomena such as changes in lighting conditions, visual aliasing, partial occlusions and noise.

Mobile Robot Self-Localization Using Omnidirectional Vision with Feature Matching from Real and Virtual Spaces

This paper presents a novel self-localization technique for mobile robots based on image feature matching from omnidirectional vision. The proposed method first constructs a virtual space with synthetic omnidirectional imaging to simulate a mobile robot equipped with an omnidirectional vision system in the real world. In the virtual space, a number of vertical and horizontal lines are generated according to the structure of the environment. They are imaged by the virtual omnidirectional camera using the catadioptric projection model. The omnidirectional images derived from the virtual and real environments are then used to match the synthetic lines and real scene edges. Finally, the pose and trajectory of the mobile robot in the real world are estimated by the efficient perspective-n-point (EPnP) algorithm based on the line feature matching. In our experiments, the effectiveness of the proposed self-localization technique was validated by the navigation of a mobile robot in a real world environment.

Soccer‐Assisted Training Robot Based on Image Recognition Omnidirectional Movement

With the continuous emergence and innovation of computer technology, mobile robots are a relatively hot topic in the field of artificial intelligence. It is an important research area of more and more scholars. The core of mobile robots is to be able to realize real‐time perception of the surrounding environment and self‐positioning and to conduct self‐navigation through this information. It is the key to the robot’s autonomous movement and has strategic research significance. Among them, the goal recognition ability of the soccer robot vision system is the basis of robot path planning, motion control, and collaborative task completion. The main recognition task in the vision system is the omnidirectional vision system. Therefore, how to improve the accuracy of target recognition and the light adaptive ability of the robot omnidirectional vision system is the key issue of this paper. Completed the system construction and program debugging of the omnidirectional mobile robot platform, and tested its omnidirectional mobile function, positioning and map construction capabilities in the corridor and indoor environment, global navigation function in the indoor environment, and local obstacle avoidance function. How to use the local visual information of the robot more perfectly to obtain more available information, so that the “eyes” of the robot can be greatly improved by relying on image recognition technology, so that the robot can obtain more accurate environmental information by itself has always been domestic and foreign one of the goals of the joint efforts of scholars. Research shows that the standard error of the experimental group’s shooting and dribbling test scores before and the experimental group’s shooting and dribbling test results after the standard error level is 0.004, which is less than 0.05, which proves the use of soccer‐assisted robot‐assisted training. On the one hand, we tested the positioning and navigation functions of the omnidirectional mobile robot, and on the other hand, we verified the feasibility of positioning and navigation algorithms and multisensor fusion algorithms.

Indoor Key Point Reconstruction Based on Laser Illumination and Omnidirectional Vision

Efficient and precise three-dimensional (3D) measurement is an important issue in the field of machine vision. In this paper, a measurement method for indoor key points is proposed with structured lights and omnidirectional vision system and the system can achieve the wide field of view and accurate results. In this paper, the process of obtaining indoor key points is as follows: Firstly, through the analysis of the system imaging model, an omnidirectional vision system based on structured light is constructed. Secondly, the full convolution neural network is used to estimate the scene for the dataset. Then, according to the geometric relationship between the scenery point and its reference point in structured light, for obtaining the 3D coordinates of the unstructured light point is presented. Finally, combining the full convolution network model and the structured light 3D vision model, the 3D mathematical representation of the key points of the indoor scene frame is completed. The experimental results proved that the proposed method can accurately reconstruct indoor scenes, and the measurement error is about 2%.

Gas Pipeline Inspection Using Autonomous Robots With Omni-Directional Cameras

Maintenance of pipelines is a major concern for oil and gas companies and millions of dollars are spent on preventive inspections every year. The type of equipment used in carrying out inspections is expensive and very sophisticated requiring very specialized manpower. Here, a novel approach to pipeline inspection is presented using a small mobile robot and a catadioptric omni-directional vision system. The system captures panoramic videos using conical and/or hyperbolic mirrors, which are transmitted to a monitoring station to examine the inner surface of gas pipelines. The videos are prepped, unwrapped, and un-warped to obtain a realistic flat image of the whole inner surface of the pipeline. The Simulated Annealing optimization method is implemented to optimize the system parameters using different mirror shapes and unwrapping and un-warping techniques. Simulations and experimental studies using a mobile robot equipped with a CCD camera and a conical mirror are presented here to demonstrate the viability of this inspection technique. The developed robotic system can be used to visually pre-scan pipelines before using much more expensive tools such as magnetic flux leakage (MFL) or ultrasound (UT) based inspection systems. These latter tools cost tens of thousands of dollars per hour and will be used only after suspicious areas are visually detected. This will reduce the inspection time and cost significantly.

Creating Incremental Models of Indoor Environments through Omnidirectional Imaging

In this work, an incremental clustering approach to obtain compact hierarchical models of an environment is developed and evaluated. This process is performed using an omnidirectional vision sensor as the only source of information. The method is structured in two loop closure levels. First, the Node Level Loop Closure process selects the candidate nodes with which the new image can close the loop. Second, the Image Level Loop Closure process detects the most similar image and the node with which the current image closed the loop. The algorithm is based on an incremental clustering framework and leads to a topological model where the images of each zone tend to be clustered in different nodes. In addition, the method evaluates when two nodes are similar and they can be merged in a unique node or when a group of connected images are different enough to the others and they should constitute a new node. To perform the process, omnidirectional images are described with global appearance techniques in order to obtain robust descriptors. The use of such technique in mapping and localization algorithms is less extended than local features description, so this work also evaluates the efficiency in clustering and mapping techniques. The proposed framework is tested with three different public datasets, captured by an omnidirectional vision system mounted on a robot while it traversed three different buildings. This framework is able to build the model incrementally, while the robot explores an unknown environment. Some relevant parameters of the algorithm adapt their value as the robot captures new visual information to fully exploit the features’ space, and the model is updated and/or modified as a consequence. The experimental section shows the robustness and efficiency of the method, comparing it with a batch spectral clustering algorithm.

Omnidirectional Vision System With Laser Illumination in a Flexible Configuration and Its Calibration by One Single Snapshot

An omnidirectional vision system consisting of a fisheye camera and structured light projector has its own distinctive advantages toward other vision systems. A vision system provides a wide range of information by one single capture. Moreover, the laser feature is easy to detect and extract from the given image for further analysis. In order to obtain the reliable results, the vision system must be calibrated. There have been improvements toward calibration techniques for vision systems with laser illumination. However, these techniques are not suitable for an omnidirectional vision system in a particular configuration. Motivated by the abovementioned problem, this article presents a novel omnidirectional vision system with laser illumination in a flexible configuration and a method for its extrinsic calibration. Flexible configuration means that both camera and laser plane are no longer parallel to the floor. The calibrated vision system is evaluated experimentally by using simulated and real data to demonstrate that the proposed calibration method is more robust and provides a higher measurement accuracy.

Comparison of hand segmentation approaches in grasp images obtained with an omnidirectional vision system

Comparison of hand segmentation approaches in grasp images obtained with an omnidirectional vision system

Optical Design of a Reflecting Omnidirectional Vision System for Long-wavelength Infrared Light

Optical Design of a Reflecting Omnidirectional Vision System for Long-wavelength Infrared Light

A discriminant method of single-optical-axis omnidirectional vision system

Generally, due to the instrumental error of omnidirectional camera, it is very difficult to satisfy the standard single-optical-axis request. Thus, it is necessary to evaluate whether a single-optical-axis camera lens is aligned or has tangent distortion. In this paper, we propose a discriminant function of single-optical-axis omnidirectional vision system based on checkerboard, which is only related to the image points and does not involve any camera parameters. Firstly, under single-optical-axis omnidirectional camera, the geometric invariance among image points of collinear and equidistant space points is derived. Next, based on the derived geometric invariance in a single image, we construct the object function to discriminate the standard single-optical-axis omnidirectional camera. Finally, a discriminant method of single-optical-axis omnidirectional vision system is proposed based on the checkerboard. Experimental results on both simulated and real data have demonstrated the usefulness and effectiveness of our method.

Sea-skyline-based image stabilization of a buoy-mounted catadioptric omnidirectional vision system

Marine monitoring systems have the requirements of a large field of view, low power consumption, real-time viewing, and economical and automatic functionality. This paper establishes an omnidirectional vision system used in marine buoys that meets these requirements. We present a framework for image stabilization, which is achieved by omnidirectional sea-skyline detection in a marine environment. We propose an optimal edge estimation method to calculate the sea-skyline ellipsis according to the sea-skyline characteristics in panoramic images. We construct a compact panoramic image stabilization model based on the sea-skyline and propose a reconstruction method for the invalid regions using the key frame. The experimental results and analysis show that the proposed approach is capable of acquiring stable video in real-time marine monitoring tasks and that the target detection is sufficiently effective, efficient, and accurate for a real-time ship target detection application.

A target tracking and location robot system based on omnistereo vision

PurposeBecause of their large field of view, omnistereo vision systems have been widely used as primary vision sensors in autonomous mobile robot tasks. The purpose of this article is to achieve real-time and accurate tracking by the omnidirectional vision robot system.Design/methodology/approachThe authors provide in this study the key techniques required to obtain an accurate omnistereo target tracking and location robot system, including stereo rectification and target tracking in complex environment. A simple rectification model is proposed, and a local image processing method is used to reduce the computation time in the localization process. A target tracking method is improved to make it suitable for omnidirectional vision system. Using the proposed methods and some existing methods, an omnistereo target tracking and location system is established.FindingsThe experiments are conducted with all the necessary stages involved in obtaining a high-performance omnistereo vision system. The proposed correction algorithm can process the image in real time. The experimental results of the improved tracking algorithm are better than the original algorithm. The statistical analysis of the experimental results demonstrates the effectiveness of the system.Originality/valueA simple rectification model is proposed, and a local image processing method is used to reduce the computation time in the localization process. A target tracking method is improved to make it suitable for omnidirectional vision system. Using the proposed methods and some existing methods, an omnistereo target tracking and location system is established.

Helicopter blades running elevation measurement using omnidirectional vision

Omnidirectional dynamic space parameters of high-speed rotating helicopter blades are precise 3D vector description of the blades. In particular, the elevation difference is directly related to the aerodynamic performance and maneuverability of the helicopter. The state of the art detection techniques based on optics and common vision have several drawbacks, such as high demands on devices but poor extensibility, limited measurement range and fixed measurement position. In this paper, a novel approach of helicopter blades running elevation measurement is proposed based on omnidirectional vision. With the advantages of panoramic visual imaging integration, 360° field of view and rotation in-variance, high-resolution images of all rotating blades positions are obtained at one time. By studying the non-linear calibration and calculation model of omnidirectional vision system, aiming at solving the problem of inaccurate visual space mapping model, the omnidirectional and full-scale measurement of the elevation difference are finalized. Experiments are carried out on our multifunctional simulation blades test system and the practical blades test tower, respectively. The experimental results demonstrate the effectiveness of the proposed method and show that the proposed method can considerably reduce the complexity of measurement.

Target-tracking algorithm for omnidirectional vision

Omnidirectional vision with the advantage of a large field-of-view overcomes the problem that a target is easily lost due to the narrow sight of perspective vision. We improve a target-tracking algorithm based on discriminative tracking features in several aspects and propose a target-tracking algorithm for an omnidirectional vision system. (1) An elliptical target window expression model is presented to represent the target’s outline, which can adapt to the deformation of an object and reduce background interference. (2) The background-weighted linear RGB histogram target feature is introduced, which decreases the weight of the background feature. (3) The Bhattacharyya coefficients-based feature identification method is employed, which reduces the computation time of the tracking algorithm. (4) An adaptive target scale and orientation measurement method is applied to adapt to severe deformations of the target’s outline. (5) A model update strategy is put forward, which is based on similarity measurements to achieve an effective and accurate model update. The experimental results show the proposed algorithm can achieve better performance than the state-of-the-art algorithms when using omnidirectional vision to perform long-term target-tracking tasks.

Calibration of a paracatadioptric camera by projection imaging of a single sphere.

This paper introduces an omnidirectional vision system calibration method that uses the central catadioptric projection properties of a single sphere. A central catadioptric camera with a single effective viewpoint images a single sphere as a closed conic. The intrinsic parameters of the camera can be obtained based on the imaged circular points and orthogonal vanishing points. The feasibility and validity of three algorithms were demonstrated through simulations and experiments with real data.

Design and implementation of an omnidirectional vision system for robot perception

To meet the demand of surrounding detection of a humanoid robot, we developed an omnidirectional vision system for robot perception (OVROP) with 5 Degrees of Freedom (DOFs). OVROP has a modular design and mainly consists of three parts: hardware, control architecture and visual processing part (omnidirectional vision and stereovision). As OVROP is equipped with universal hardware and software interfaces it can be applied to various types of robots. Our performance evaluation proves that OVROP can accurately detect and track an object with 360° field of view (FOV). Besides, undistorted omnidirectional perception of surroundings can be achieved through calibrations of both monocular and stereo cameras. Furthermore, our preliminary experimental results show that OVROP can perceive a desired object within 160 ms in most cases. As a result, OVROP can provide detailed information on surrounding environment for full-scope and real-time robot perception.