Monocular Vision Sensor Research Articles

Research on application method of intelligent driving technology based on monocular vision sensor

With the development of driverless cars, intelligent driving technology is increasingly used in the automotive industry, monocular vision sensor plays an indispensable role in intelligent driving technology because of its simple structure, low cost and abundant information. This paper discusses and optimizes the application of the monocular vision sensor in intelligent driving. The basic principle and key technologies of the monocular vision sensor are described in detail. In the specific application of the monocular vision sensor, this paper focuses on the monocular vision sensor's depth learning network, multi-information fusion technology, improved target detection and tracking algorithm. Through in-depth research and analysis, a series of optimization strategies based on the monocular vision sensor, such as the FAST Region Convolutional Neural Network (FASTRCNN) vehicle target detection method and improved Scale-Invariant Feature Transform (SIFT) algorithm, are proposed. Finally, this paper summarizes the intelligent driving technology based on the monocular vision sensor and suggests that the monocular vision sensor will play a more important role in intelligent driving technology. Future research shall focus on improving the accuracy of the algorithm, such as the development of end-to-end convolutional neural network fusion methods, the proposed depth multi-modal sensor fusion network, and so on.

Monocular Absolute Depth Estimation from Motion for Small Unmanned Aerial Vehicles by Geometry-Based Scale Recovery.

In recent years, there has been extensive research and application of unsupervised monocular depth estimation methods for intelligent vehicles. However, a major limitation of most existing approaches is their inability to predict absolute depth values in physical units, as they generally suffer from the scale problem. Furthermore, most research efforts have focused on ground vehicles, neglecting the potential application of these methods to unmanned aerial vehicles (UAVs). To address these gaps, this paper proposes a novel absolute depth estimation method specifically designed for flight scenes using a monocular vision sensor, in which a geometry-based scale recovery algorithm serves as a post-processing stage of relative depth estimation results with scale consistency. By exploiting the feature correspondence between successive images and using the pose data provided by equipped navigation sensors, the scale factor between relative and absolute scales is calculated according to a multi-view geometry model, and then absolute depth maps are generated by pixel-wise multiplication of relative depth maps with the scale factor. As a result, the unsupervised monocular depth estimation technology is extended from relative depth estimation in semi-structured scenes to absolute depth estimation in unstructured scenes. Experiments on the publicly available Mid-Air dataset and customized data demonstrate the effectiveness of our method in different cases and settings, as well as its robustness to navigation sensor noise. The proposed method only requires UAVs to be equipped with monocular camera and common navigation sensors, and the obtained absolute depth information can be directly used for downstream tasks, which is significant for this kind of vehicle that has rarely been explored in previous depth estimation studies.

AUTONOMOUS VEHICLES LOCALISATION BASED ON SEMANTIC MAP MATCHING METHOD

Abstract. In autonomous driving systems, a positioning method that can be used in scenarios with no satellite signals or long signal interruptions is a must. In this paper, we address the problems in map construction methods and map matching methods in scenes without satellite signals or long signal interruptions, and construct a semantic map matching-based localization method to meet the localization requirements by means of monocular vision sensors on the basis of weighing the accuracy and cost of map construction and localization. In this paper, firstly, the method of map construction is studied and a static semantic map construction method based on monocular camera is constructed. Then the map matching localization method is studied, and a semantic map matching based localization method is constructed to align the local map built during localization with the pre-built global semantic map to obtain the current location information. Finally, this paper constructs a method to fuse the visual odometry and map matching localization results, so as to obtain more accurate localization results.

GNSS Reconstrainted Visual–Inertial Odometry System Using Factor Graphs

Monocular vision sensors are often affected by the rapid direction change in load platform and violent illumination change when the mobile device moves autonomously with high maneuverability. The images collected by the visual sensor will also have a lot of dynamic blur, which together with the weak texture environment reduces the continuity and accuracy of the visual autonomous navigation system. To enhance the stability of the system, in the letter, we propose a vision-led multisource data fusion navigation algorithm. The system combines the visual information for trajectory estimation, adds the inertial measurement unit (IMU) measurement information to the sliding window for optimization, and finally uses the global navigation satellite system (GNSS) data as a reconstraint condition through factor graph optimization to further optimize the trajectory accuracy. Experiments on the public datasets containing a variety of different scene categories show that the trajectory tracking results generated by our algorithm are more complete and stable and can better meet the system’s autonomous navigation requirements.

Monitoring multiple geometrical dimensions in WAAM based on a multi-channel monocular visual sensor

Monitoring multiple geometrical dimensions is a crucial premise to realize precise control of layer size in wire and arc additive manufacturing (WAAM). Different from conventional single-target detection using a single sensor or multiple-target detection using multiple sensors, this study designs a novel multi-channel monocular vision sensor to monitor different geometrical sizes, i.e., current layer to nozzle distance, previous layer to nozzle distance, and layer width. A Marr-Hildreth algorithm is employed to extract boundaries of the molten pool and the previous layer. A homography matrix is solved for height calibration, and detected heights are introduced to assist layer width calibration. The sensor’s accuracy is verified via variable-size templates, and the maximum errors of detected height and width are 3.39% and 3.98%, respectively. Finally, the sensor is applied to monitor the layer size of a 5th-layered wall. This study lays a solid foundation for high-precision control of multiple geometrical sizes in WAAM.

Extension of the Observability Rank Condition to Time-Varying Nonlinear Systems

This article provides an extension of the observability rank condition to time-varying nonlinear systems. Previous conditions to check the state observability only hold for nonlinear systems that do not explicitly depend on time, or, for time-varying systems, they only account for the linear case. In this article, a general analytic condition is provided. The new condition reduces to the well-known observability rank condition in the two simpler cases of time-varying linear systems and time-invariant nonlinear systems. The proposed condition is applied to aerospace robotics to study the observability of a lunar-module-type system equipped with monocular vision and inertial sensors.

An Attitude Estimation Method Based on Monocular Vision and Inertial Sensor Fusion for Indoor Navigation

The attitude of moving objects has a predictive function for navigation and positioning in a dim indoor environment. Currently, the accuracy of vision and inertial sensors’ attitude estimation is low, as it mainly affected by factors such as motion image blur, attitude angle processing algorithms, and data synchronization. Firstly, in this paper we propose a novel continuous multi-frame evaluation and registration algorithm to obtain high signal-to-noise ratio (SNR) images in regular indoor environments without global positioning system (GPS). The coordinates of vanishing points and plumb lines were extracted in the strong texture image. Then, a visual attitude model was constructed based on the characteristic points and lines. A pre-integrated gyro sensor was used to establish the inertial attitude model. The visual and gyro attitude models were used to obtain high-precision attitude information using multi-rate filtering. Finally, we build a hardware attachment composed of a consumer camera, an inexpensive inertial sensor and hardware circuit based on digital signal processing, which effectively demonstrates that the experiment result compares favourably with the more traditional methods.

SFA-MDEN: Semantic-Feature-Aided Monocular Depth Estimation Network Using Dual Branches.

Monocular depth estimation based on unsupervised learning has attracted great attention due to the rising demand for lightweight monocular vision sensors. Inspired by multi-task learning, semantic information has been used to improve the monocular depth estimation models. However, multi-task learning is still limited by multi-type annotations. As far as we know, there are scarcely any large public datasets that provide all the necessary information. Therefore, we propose a novel network architecture Semantic-Feature-Aided Monocular Depth Estimation Network (SFA-MDEN) to extract multi-resolution depth features and semantic features, which are merged and fed into the decoder, with the goal of predicting depth with the support of semantics. Instead of using loss functions to relate the semantics and depth, the fusion of feature maps for semantics and depth is employed to predict the monocular depth. Therefore, two accessible datasets with similar topics for depth estimation and semantic segmentation can meet the requirements of SFA-MDEN for training sets. We explored the performance of the proposed SFA-MDEN with experiments on different datasets, including KITTI, Make3D, and our own dataset BHDE-v1. The experimental results demonstrate that SFA-MDEN achieves competitive accuracy and generalization capacity compared to state-of-the-art methods.

Fast Reconstruction of 3D Point Cloud Model Using Visual SLAM on Embedded UAV Development Platform

In recent years, the rapid development of unmanned aerial vehicle (UAV) technologies has made data acquisition increasingly convenient, and three-dimensional (3D) reconstruction has emerged as a popular subject of research in this context. These 3D models have many advantages, such as the ability to represent realistic scenes and a large amount of information. However, traditional 3D reconstruction methods are expensive, and require long and complex processing. As a result, they cannot rapidly respond when used in time-sensitive applications, e.g., those for such natural disasters as earthquakes, debris flow, etc. Computer vision-based simultaneous localization and mapping (SLAM) along with hardware development based on embedded systems, can provide a solution to this problem. Based on an analysis of the principle and implementation of the visual SLAM algorithm, this study proposes a fast method to quickly reconstruct a dense 3D point cloud model on a UAV platform combined with an embedded graphics processing unit (GPU). The main contributions are as follows: (1) to resolve the contradiction between the resource limitations and the computational complexity of visual SLAM on UAV platforms, the technologies needed to compute resource allocation, communication between nodes, and data transmission and visualization in an embedded environment were investigated to achieve real-time data acquisition and processing. Visual monitoring to this end is also designed and implemented. (2) To solve the problem of time-consuming algorithmic processing, a corresponding parallel algorithm was designed and implemented based on the parallel programming framework of the compute unified device architecture (CUDA). (3) The visual odometer and methods of 3D “map” reconstruction were designed using under a monocular vision sensor to implement the prototype of the fast 3D reconstruction system. Based on preliminary results of the 3D modeling, the following was noted: (1) the proposed method was feasible. By combining UAV, SLAM, and parallel computing, a simple and efficient 3D reconstruction model of an unknown area was obtained for specific applications. (2) The parallel SLAM algorithm used in this method improved the efficiency of the SLAM algorithm. On the one hand, the SLAM algorithm required 1/6 of the time taken by the structure-from-motion algorithm. On the other hand, the speedup obtained using the parallel SLAM algorithm based on the embedded GPU on our test platform was 7.55 × that of the serial algorithm. (3) The depth map results show that the effective pixel with an error less than 15cm is close to 60%.

Dynamic pose estimation of uncooperative space targets based on monocular vision.

A novel algorithm of dynamic pose estimation for monocular visual sensor is proposed in this paper. The sensor is principally composed of two 1D turntables, one collimated laser, and one industrial camera. In particular, the proposed algorithm is suitable for the cases of uncooperative targets. By analyzing the motion of a laser beam based on quaternion, the functional detection algorithm is derived from the position information of multiple scanning points. Furthermore, the depth recovery based on a nonparametric model is a key step in the pose calculation, which is unnecessary to make use of the calibration parameters of an industrial camera. It is, however, effective to avoid the influence of camera distortion and calibration error. After establishing a test platform, simulation and experiments for pose estimation are carried out. The experimental results show that the maximum error is 0.98° at a range of 500 mm, which proves that the proposed algorithm is accurate and effective.

A flexible 3D point reconstruction with homologous laser point array and monocular vision

A 3D recovery approach is achieved by the monocular vision sensor and the homologous laser point array with the arbitrary relative pose to the 2D reference. In the modeling process, firstly, the 3D texture board is moved to different positions in the view field of the vision sensor. Therefore, the laser rays intersect with the cubic board and generate laser points on it. Then, the Plücker matrices of the laser rays are determined by the laser points. The algebraic solution of the joint point of laser rays, the optic center of the projector, is contributed by the singular value decomposition. Thirdly, the laser rays are refined by the algebraic solution of the joint point and represented in the 2D-reference coordinate frame. Finally, the reconstruction is achieved by the calibrated laser rays of the projector. The reconstruction method is experimentally conducted to verify the performance and precision in various measurement conditions. The reconstruction error averages are 0.942 mm, 1.055 mm, 1.374 mm, 2.063 mm, under the 700 mm–1000 mm measuring distances between the vision sensor and the optic center. As there is no strict requirement for the relative installation position and orientation between the laser-ray projector and the 2D reference, the approach prompts the flexibility for the on-site active-vision measurement.

Convolutional Neural Network for Monocular Vision-based Multi-target Tracking

This paper addresses multi-target tracking using a monocular vision sensor. To overcome the fundamental observability issue of the monocular vision, a convolutional neural network (CNN)-based method is proposed. The method combines a CNN-based multi-target detection into a model-based multi-target tracking framework. While previous CNN applications to image-based object recognition and tracking focused on prediction of region of interest (RoI), the proposed method allows for prediction of the three-dimensional position information of the moving objects of interest. This is achieved by appropriately construct a network tailored to the moving object tracking problems with potentially occluded objects. In addition, the cubature Kalman filter integrated with a data association scheme is adopted for effective tracking of nonlinear motion of the objects with the measurements information from the learned network. A virtual simulator that generates the trajectories of the target motions and a sequence of images of the scene has been developed and used to test and verify the proposed CNN scheme. Simulation case studies demonstrate that the proposed CNN improves the position accuracy in the depth direction substantially.

Obstacle Detection Based on Heterogeneous Data Fusion

Obstacle detection is very important for the autonomous movement of robots. Variety of sensors are used in obstacle detection to obtain information about obstacles. Different types of data obtained by sensors cannot be directly analyzed. This paper proposed an obstacle detection method based on heterogeneous data fusion. An image processing method is adopted to process the image of obstacle obtained by monocular vision sensor, and the size and position of the obstacle can be obtained. Then using infrared distance sensors are used to obtain distance of obstacles. Finally the location and distance information of obstacles can be obtained by a specific algorithm. Two experiments are performed for verifing the effect of proposed method. Experiments shown that the proposed method can effectively obtain the position and distance of obstacles.

Target Recognition and Localization of Mobile Robot with Monocular PTZ Camera

The target recognition and location based on the vision sensor is a kind of more intuitive and effective method. The paper designs a human mobile robot and the monocular PTZ camera fixed on the differential driver of the mobile robot for the target recognition and localization. The camera parameters are calibrated using the Zhang Zhengyou calibration method; target recognition is based on the combination of the color and the edge detection. First, the objects with the same color are extracted by the color recognition method (by setting an appropriate threshold value for the HSV color space of the acquired image). Then the target is further extracted by the edge detection (by Hough circle transformation); target location is based on the Similar Triangle Principle because the PTZ camera has pitch, tilt, and zoom characteristics and the monocular vision sensor of the camera with different pitching angles is used to measure the distance between the robot and the target. According to the characteristics of the monocular PTZ camera the mobile robot realizes the target localization and tracking. The simulation and experimental results demonstrate that the mobile robot shows better target recognition and localization in mobile tracking target process and proves the effectiveness of the method.

A Monocular Vision Sensor-Based Efficient SLAM Method for Indoor Service Robots

This paper presents a new implementation method for efficient simultaneous localization and mapping using a forward-viewing monocular vision sensor. The method is developed to be applicable in real time on a low-cost embedded system for indoor service robots. In this paper, the orientation of a robot is directly estimated using the direction of the vanishing point. Then, the estimation models for the robot position and the line landmark are derived as simple linear equations. Using these models, the camera poses and landmark positions are efficiently corrected by a local map correction method. The performance of the proposed method is demonstrated under various challenging environments using dataset-based experiments using a desktop computer and real-time experiments using a low-cost embedded system. The experimental environments include a real home-like setting. These conditions contain low-textured areas, moving people, or changing environments. The proposed method is also tested using the robotics advancement through web publishing of sensorial and elaborated extensive datasets benchmark dataset.

A novel multifunctional visual sensor based on combined laser structured lights and its anti-jamming detection algorithms

Based on the combined laser structured lights, a novel multifunctional monocular visual sensor was proposed. Firstly, a mathematical model of the proposed visual sensor was established based on the optical triangulation measurement principle. The imaging characteristics of the visual sensor and the influence of structure parameters of the designed visual sensor on the detection errors were studied. Then, the new algorithms were derived for the accurate detection of the welding groove cross-sectional dimensions and the welding torch height even under the circumstances with existing of the installation and robot motion errors, which use only one photograph in monocular vision. Finally, the experiments designed for the detection were carried out. The experimental results show that the proposed visual sensor has high feasibility and anti-jamming capability in the detection, and the errors can also be detected by the sensor simultaneously. Therefore, this novel visual sensor could be applied for the improvement of precision and intelligence in the robotic welding manufacturing.

Principle of an innovative visual sensor based on combined laser structured lights and its experimental verification

To promote development of sensing technology applied in the welding process and improvement of production automation in the welding manufacturing industry, an innovative multi-functional monocular visual sensor based on combined laser structured lights (single-line & cross-lines) is proposed and devised. It can realize the detection of cross-sectional size parameters of welding groove, space position and posture of welding torch relative to the weldment or welding groove by using monocular vision. Its calibration makes full use of the intrinsic parameters, such as its internal structure parameters and the focal length of industrial CCD camera. Therefore, it is unnecessary to re-calibrate when the position and posture of welding torch have some variations during the welding process or the visual sensor reassembles with the welding torch, which availably improves the adaptability of this sensor in engineering application, so that it can apply to the welding process with complex space welding trajectory. Furthermore, with the use of combined laser structured lights, it effectively eliminates the problem of information loss in depth commonly existing in usual monocular visual sensor. Meanwhile, all the detection functions of this sensor are derived from the processing results of a single image captured by industrial CCD camera, which avoids the problems of time-consuming and large amount of calculation resulting from multi-images processing (multi-vision method or multi-angle time-sharing imaging in monocular vision), and thus significantly improves its real-time performance in engineering application. Its functional detection algorithms for the cross-sectional size parameters of welding groove and the height of welding torch derived from the principle of optical triangulation are given. After establishing a test platform, we carried out a series of detecting experiments for different types or sizes of V-type butt groove. By image processing, recognition and extraction of the feature points, the cross-sectional size parameters of welding groove and the height of welding torch calculate out through the derived detection algorithms. The acquired detection results have enough accuracy and their maximum relative errors are not exceeding 3.9%, which demonstrate the rationality of the physical design for the proposed visual sensor and the validity of the derived detection algorithms.

Robotic assembly of smartphone back shells with eye-in-hand visual servoing

Abstract Research concerning smartphone assembly has attracted increasing attention from the manufacturing industry. This paper presents an automatic back shell assembly system using real-time visual servoing in an eye-in-hand configuration. Look-then-move and look-and-move control approaches are sequentially proposed to handle distal and proximal assembly tasks, respectively. During distal assembly, binocular vision is employed to assist a robotic manipulator to locate and reconstruct the positions and orientations of a cellular phone and its back shell in the work space. After lifting the back shell up and then reaching the top of the cellular phone, there is no binocular field of view. A seemingly novel way of 3-D reconstruction by two uniocular fields of view of the two cameras is proposed. With the pair of monocular vision sensors, two monocular visual servo control laws are employed sequentially to perform the proximal assembly task autonomously. Based on the experimental results, the robotic manipulator can be driven to hold the back shell by a vacuum absorption device and position it precisely onto the smartphone to accomplish the automatic assembly task even if the vision system is only roughly calibrated. The proposed control approach for distal and proximal assembly appears to be efficient, flexible, and reliable for potential applications in industrial manufacturing.

Combined measurement system for double shield tunnel boring machine guidance based on optical and visual methods

In order to detect the position of the cutting shield at the head of a double shield tunnel boring machine (TBM) during the excavation, this paper develops a combined measurement system which is mainly composed of several optical feature points, a monocular vision sensor, a laser target sensor, and a total station. The different elements of the combined system are mounted on the TBM in suitable sequence, and the position of the cutting shield in the reference total station frame is determined by coordinate transformations. Subsequently, the structure of the feature points and matching technique for them are expounded, the position measurement method based on monocular vision is presented, and the calibration methods for the unknown relationships among different parts of the system are proposed. Finally, a set of experimental platforms to simulate the double shield TBM is established, and accuracy verification experiments are conducted. Experimental results show that the mean deviation of the system is 6.8mm, which satisfies the requirements of double shield TBM guidance.

Monocular Vision-Based Underwater Object Detection.

In this paper, we propose an underwater object detection method using monocular vision sensors. In addition to commonly used visual features such as color and intensity, we investigate the potential of underwater object detection using light transmission information. The global contrast of various features is used to initially identify the region of interest (ROI), which is then filtered by the image segmentation method, producing the final underwater object detection results. We test the performance of our method with diverse underwater datasets. Samples of the datasets are acquired by a monocular camera with different qualities (such as resolution and focal length) and setups (viewing distance, viewing angle, and optical environment). It is demonstrated that our ROI detection method is necessary and can largely remove the background noise and significantly increase the accuracy of our underwater object detection method.