Stereo Camera Sensor Research Articles

Robust Real-Time AUV Self-Localization Based on Stereo Vision-Inertia

Autonomous underwater vehicles (AUVs) play an important role in deep-sea exploration, in which AUV self-localization is a key component. However, due to poor visibility caused by challenging marine environments, AUVs are often equipped with high-cost and heavy-weight acoustic sensors to accomplish localization tasks. We propose a robust real-time AUV self-localization method based on stereo camera and inertial sensor, which merges point and diagonal features, as well as inertial measurements to overcome the challenges of poor visibility. Our method also includes an underwater loop detection algorithm based on the combination of points and diagonal segments, which can extract effective binary descriptors in low-textured underwater scenarios. Furthermore, we develop an AUV self-localization system based on a real-time, portable, low-cost, and small volume sensor suite. Finally, we test the proposed method in a real underwater environment using our sensor suite, and the experimental results demonstrate the effectiveness of the proposed method under dramatically changing underwater scenarios.

Development and testing of a navigation solution for Autonomous Underwater Vehicles based on stereo vision

Autonomous Underwater Vehicles (AUVs) employed for inspection and monitoring applications require a reliable estimate of their navigation status to successfully accomplish the scheduled mission. Considering that they are typically endowed with vision systems to collect images of the surveyed environment, approaches based on Visual Odometry (VO) allow to utilise the optical payload also to perform robot navigation. Despite a large body of prior research, the topic of navigation based on stereo vision is still an open problem in the marine domain. Within this context, this paper proposes an underwater navigation framework utilising a stereo camera as a linear velocity sensor. The purpose is to assess the potentiality of a stereo vision system to indirectly measure the linear velocity of an AUV, and provide this information to the navigation framework of the vehicle. The proposed strategy consists in a visual inertial odometry solution fusing linear velocity estimates, derived from optical information, with attitude and depth measurements to retrieve the robot navigation status. Such linear velocity estimates are computed exploiting a stereo VO algorithm, relying on a 3D-to-2D approach, whose source code is made available online. The developed strategy is evaluated on real underwater datasets acquired during monitoring surveys performed by an AUV equipped with Doppler Velocity Log (DVL), bottom-looking stereo camera, inertial unit, and depth sensor. The results, obtained through a comparison with the DVL taken as benchmark, confirm the approach as feasible and exploitable in underwater inspection and monitoring contexts.

Agricultural Robot-Centered Recognition of Early-Developmental Pest Stage Based on Deep Learning: A Case Study on Fall Armyworm (Spodoptera frugiperda)

Accurately detecting early developmental stages of insect pests (larvae) from off-the-shelf stereo camera sensor data using deep learning holds several benefits for farmers, from simple robot configuration to early neutralization of this less agile but more disastrous stage. Machine vision technology has advanced from bulk spraying to precise dosage to directly rubbing on the infected crops. However, these solutions primarily focus on adult pests and post-infestation stages. This study suggested using a front-pointing red-green-blue (RGB) stereo camera mounted on a robot to identify pest larvae using deep learning. The camera feeds data into our deep-learning algorithms experimented on eight ImageNet pre-trained models. The combination of the insect classifier and the detector replicates the peripheral and foveal line-of-sight vision on our custom pest larvae dataset, respectively. This enables a trade-off between the robot's smooth operation and localization precision in the pest captured, as it first appeared in the farsighted section. Consequently, the nearsighted part utilizes our faster region-based convolutional neural network-based pest detector to localize precisely. Simulating the employed robot dynamics using CoppeliaSim and MATLAB/SIMULINK with the deep-learning toolbox demonstrated the excellent feasibility of the proposed system. Our deep-learning classifier and detector exhibited 99% and 0.84 accuracy and a mean average precision, respectively.

Detecting Darting Out Pedestrians With Occlusion Aware Sensor Fusion of Radar and Stereo Camera

Early and accurate detection of crossing pedestrians is crucial in automated driving in order to perform timely emergency manoeuvres. However, this is a difficult task in urban scenarios where pedestrians are often occluded (not visible) behind objects, e.g., other parked vehicles. We propose an occlusion aware fusion of stereo camera and radar sensors to address scenarios with crossing pedestrians behind such parked vehicles. Our proposed method adapts both the expected rate and properties of detections in different areas according to the visibility of the sensors. In our experiments on a real-world dataset, we show that the proposed occlusion aware fusion of radar and stereo camera detects the crossing pedestrians on average 0.26 seconds earlier than using the camera alone, and 0.15 seconds earlier than fusing the sensors without occlusion information. Our dataset containing 501 relevant recordings of pedestrians behind vehicles will be publicly available on our website for non-commercial, scientific use.

3D Pose Recognition System of Dump Truck for Autonomous Excavator

The purpose of an excavator is to dig up materials and load them onto heavy-duty dump trucks. Typically, an excavator is positioned at the rear of the dump truck when loading. In order to automate this process, this paper proposes a system that employs a combined stereo camera and two LiDAR sensors to determine the three-dimensional (3D) position of the truck’s cargo box and to analyze its loading space. Sparse depth information acquired from the two LiDAR sensors is used to detect points on the door of the cargo box and establish the plane on its rear side. Dense depth information of the cargo box acquired from the stereo camera sensor is projected onto the plane of the box’s rear to estimate its initial 3D position. In the next step, the point cloud sampled along the long shaft of the edge of the cargo box is used as the input of the Iterative Closest Point algorithm to calculate a more accurate cargo box position. The data collected from the stereo camera are then used to determine the 3D position of the cargo box and provide an estimate of the volume of the load along with the 3D position of the loading space to the excavator. In order to demonstrate the efficiency of the proposed method, a mock-up of a heavy-duty truck cargo box was created, and the volume of the load in the cargo box was analyzed.

Redundant Multi-Object Detection for Autonomous Vehicles in Structured Environments

This paper presents a redundant multi-object detection method for autonomous driving, exploiting a combination of Light Detection and Ranging (LiDAR) and stereocamera sensors to detect different obstacles. These sensors are used for distinct perception pipelines considering a custom hardware/software architecture deployed on a self-driving electric racing vehicle. Consequently, the creation of a local map with respect to the vehicle position enables development of further local trajectory planning algorithms. The LiDAR-based algorithm exploits segmentation of point clouds for the ground filtering and obstacle detection. The stereocamerabased perception pipeline is based on a Single Shot Detector using a deep learning neural network. The presented algorithm is experimentally validated on the instrumented vehicle during different driving maneuvers.

An application to simulate and control industrial robot in virtual reality environment integrated with IR stereo camera sensor

The main goal of this research is to test for potential ways to control KUKA KR10 industrial arm manipulator using Virtual Reality technology and check for the advantages of applying this control methods. The final version of this application aims to achieve this goal by establishing an interaction between the user and the manipulator inside a virtual environment developed using the game engine Unity3D and the HTC VIVE Pro headset for the virtual visualization. By applying this control method, the user does not have to operate on site and instead he can work remotely. In addition to the ability to use off-line programming of the manipulator. The application is designed to simplify the controlling ways by displaying a complete virtual environment where the tridimensional model of the robotic arm can be visualized and programmed according to the real manipulator’s parameters and specifications. All the movements and parameters in the virtual environment can be synchronized with the real robot in an on-line or off-line path planning depending on the application or the task. The system integrates a set of virtual reality controllers and Leapmotion sensor as options to allow the user to control and see the robot and its parameters in the virtual environment. As a result of this research, the manipulator moves on the planned trajectory in a smooth way after applying some filtering techniques without losing its accuracy.

An Indoor Navigation System Based on Stereo Camera and Inertial Sensors with Points and Lines

An indoor navigation system based on stereo camera and inertial sensors with points and lines is proposed to further improve the accuracy and robustness of the navigation system in complex indoor environments. The point and line features, which are fast extracted by ORB method and line segment detector (LSD) method, are both employed in this system to improve its ability to adapt to complex environments. In addition, two different representations of lines are adopted to improve the efficiency of the system. Besides stereo camera, an inertial measurement unit (IMU) is also used in the system to further improve its accuracy and robustness. An estimator is designed to integrate the camera and IMU measurements in a tightly coupled approach. The experimental results show that the performance of the proposed navigation system is better than the point-only VINS and the vision-only navigation system with points and lines.

A study on ego-motion estimation based on stereo camera sensor and 2G1Y inertial sensor with considering vehicle dynamics

The vision sensor on a vehicle platform with dynamic motion is affected severely by disturbances that occur owing to vehicle dynamic movement, which results in the degradation of outputs from visio...

Hybrid obstacle avoidance system with vision and ultrasonic sensors for multi-rotor MAVs

PurposeThis paper aims to develop an obstacle avoidance system for a multi-rotor micro aerial vehicle (MAV) that flies in indoor environments which usually contain transparent, texture-less or moving objects.Design/methodology/approachThe system adopts a combination of a stereo camera and an ultrasonic sensor to detect obstacles and extracts three-dimensional (3D) point clouds. The obstacle map is built on a coarse global map and updated by local maps generated by the recent 3D point clouds. An efficient layered A* path planning algorithm is also proposed to address the path planning in 3D space for MAVs.FindingsThe authors conducted a lot of experiments in both static and dynamic scenes. The results show that the obstacle avoidance system works reliably even when transparent or texture-less obstacles are present. The layered A* path planning algorithm is much faster than the traditional 3D algorithm and makes the system response quickly when the obstacle map has been changed because of the moving objects.Research limitations/implicationsThe limited field of view of both stereo camera and ultrasonic sensor makes the system need to change heading first before moving side to side or moving backward. But this problem could be addressed when multiple systems are mounted toward different directions on the MAV.Practical implicationsThe developed approach could be valuable to applications in indoors.Originality/valueThis paper presents a robust obstacle avoidance system and a fast layered path planning algorithm that are easy to be implemented for practical systems.

Merging of Depth Image Between Stereo Camera and Structure Sensor on Robot “FloW” Vision

<p class='IJASEITAbtract'>Human can recognize an object just by looking at the environment, this capability is very useful for designing the reference of humanoid robot with the ability of adapting it on its environment. By knowing the field conditions that exist in such environments, robot can understand the obstacles or anything that can be passed. To do that, robot vision needs to have a knowledge to understanding an obstacles that exist around it. Because of these problems, this paper shows a method for reducing error rate of vacant space in the data depth by combining a stereo camera and structure sensor. Merging the stereo camera and structure sensor can extract depth information becomes dense. The proposed method has been successfully running the whole algorithm built and has a density of depth with an average error rate of vacant space is<span lang='EN-GB'> 18.10%.</span>

Merging of Depth Image Between Stereo Camera and Structure Sensor on Robot “FloW” Vision

Merging of Depth Image Between Stereo Camera and Structure Sensor on Robot “FloW” Vision

Neural Control for a Differential Drive Wheeled Mobile Robot Integrating Stereo Vision Feedback

This paper proposes a tracking control method for a differential drive wheeled mobile robot with nonholonomic constraints with an inverse optimal neural controller. It is based on two techniques: first, an identifier using a discrete-time recurrent high-order neural network (RHONN) trained with an extended Kalman filter (EKF) algorithm is employed; second, an inverse optimal control is used to avoid solving the Hamilton Jacobi Bellman (HJB) equation. The desired trajectory of the robot is computed during the navigation process using a stereo camera sensor.

Real-time pose estimation of a dynamic quadruped in GPS-denied environments for 24-hour operation

We present a real-time system that enables a highly capable dynamic quadruped robot to maintain an accurate six-degree-of-freedom pose estimate (within a 1.0% error of distance traveled) over long distances traversed through complex, dynamic outdoor terrain, during day and night, in the presence of camera occlusion and saturation, and occasional large external disturbances, such as slips or falls. The system fuses a stereo-camera sensor, inertial measurement unit, leg odometry, and optional intermittent GPS position updates with an extended Kalman filter to ensure robust, low-latency performance. To maintain a six-degree-of-freedom local positioning accuracy alongside the global positioning knowledge, two reference frames are used; a local reference frame and a global reference frame, with the former benefiting obstacle detection and mapping and the latter for operator-specified and autonomous way-point following. Extensive experimental results obtained from multiple field tests are presented to illustrate the performance and robustness of the system over hours of continuous runs and hundreds of kilometers of distance traveled in a wide variety of terrains and conditions.

Neural Control for Driving a Mobile Robot Integrating Stereo Vision Feedback

This paper proposes a neural control integrating stereo vision feedback for driving a mobile robot. The proposed approach consists in synthesizing a suitable inverse optimal control to avoid solving the Hamilton Jacobi Bellman equation associated to nonlinear system optimal control. The mobile robot dynamics is approximated by an identifier using a discrete-time recurrent high order neural network, trained with an extended Kalman filter algorithm. The desired trajectory of the robot is computed during navigation using a stereo camera sensor. Simulation and experimental result are presented to illustrate the effectiveness of the proposed control scheme.

Fusing Stereo Camera and Low-Cost Inertial Measurement Unit for Autonomous Navigation in a Tightly-Coupled Approach

Exact motion estimation is a major task in autonomous navigation. The integration of Inertial Navigation Systems (INS) and the Global Positioning System (GPS) can provide accurate location estimation, but cannot be used in a GPS denied environment. In this paper, we present a tight approach to integrate a stereo camera and low-cost inertial sensor. This approach takes advantage of the inertial sensor's fast response and visual sensor's slow drift. In contrast to previous approaches, features both near and far from the camera are simultaneously taken into consideration in the visual-inertial approach. The near features are parameterised in three dimensional (3D) Cartesian points which provide range and heading information, whereas the far features are initialised in Inverse Depth (ID) points which provide bearing information. In addition, the inertial sensor biases and a stationary alignment are taken into account. The algorithm employs an Iterative Extended Kalman Filter (IEKF) to estimate the motion of the system, the biases of the inertial sensors and the tracked features over time. An outdoor experiment is presented to validate the proposed algorithm and its accuracy.

Daytime fog detection and density estimation with entropy minimization

Abstract. Fog disturbs the proper image processing in many outdoor observation tools. For instance, fog reduces the visibility of obstacles in vehicle driving applications. Usually, the estimation of the amount of fog in the scene image allows to greatly improve the image processing, and thus to better perform the observation task. One possibility is to restore the visibility of the contrasts in the image from the foggy scene image before applying the usual image processing. Several algorithms were proposed in the recent years for defogging. Before to apply the defogging, it is necessary to detect the presence of fog, not to emphasis the contrasts due to noise. Surprisingly, few a reduced number of image processing algorithms were proposed for fog detection and characterization. Most are dedicated to static cameras and can not be used when the camera is moving. Daytime fog is characterized by its extinction coefficient, which is equivalent to the visibility distance. A visibility-meter can be used for fog detection and characterization, but this kind of sensor performs an estimation in a relatively small volume of air, and is thus sensitive to heterogeneous fog, and air turbulence with moving cameras. In this paper, we propose an original algorithm, based on entropy minimization, to detect fog and estimate its extinction coefficient by the processing of stereo pairs. This algorithm is fast, provides accurate results using low cost stereo camera sensor and, the more important, can work when the cameras are moving. The proposed algorithm is evaluated on synthetic and camera images with ground truth. Results show that the proposed method is accurate, and, combined with a fast stereo reconstruction algorithm, should provide a solution, close to real time, for fog detection and visibility estimation for moving sensors.

Validity, Reliability, and Sensitivity of a 3D Vision Sensor-based Upper Extremity Reachable Workspace Evaluation in Neuromuscular Diseases

Introduction: One of the major challenges in the neuromuscular field has been lack of upper extremity outcome measures that can be useful for clinical therapeutic efficacy studies. Using vision-based sensor system and customized software, 3-dimensional (3D) upper extremity motion analysis can reconstruct a reachable workspace as a valid, reliable and sensitive outcome measure in various neuromuscular conditions where proximal upper extremity range of motion and function is impaired. Methods: Using a stereo-camera sensor system, 3D reachable workspace envelope surface area normalized to an individual’s arm length (relative surface area: RSA) to allow comparison between subjects was determined for 20 healthy controls and 9 individuals with varying degrees of upper extremity dysfunction due to neuromuscular conditions. All study subjects were classified based on Brooke upper extremity function scale. Right and left upper extremity reachable workspaces were determined based on three repeated measures. The RSAs for each frontal hemi-sphere quadrant and total reachable workspaces were determined with and without loading condition (500 gram wrist weight). Data were analyzed for assessment of the developed system and validity, reliability, and sensitivity to change of the reachable workspace outcome. Results: The mean total RSAs of the reachable workspace for the healthy controls and individuals with NMD were significantly different (0.586 ± 0.085 and 0.299 ± 0.198 respectively; p<0.001). All quadrant RSAs were reduced for individuals with NMDs compared to the healthy controls and these reductions correlated with reduced upper limb function as measured by Brooke grade. The upper quadrants of reachable workspace (above the shoulder level) demonstrated greatest reductions in RSA among subjects with progressive severity in upper extremity impairment. Evaluation of the developed outcomes system with the Bland-Altman method demonstrated narrow 95% limits of agreement (LOA) around zero indicating high reliability. In addition, the intraclass correlation coefficient (ICC) was 0.97. Comparison of the reachable workspace with and without loading condition (wrist weight) showed significantly greater RSA reduction in the NMD group than the control group (p<0.012), with most of the workspace reduction occurring in the ipsilateral upper quadrant relative to the tested arm (p<0.001). Reduction in reachable workspace due to wrist weight was most notable in those subjects with NMD with marginal strength reserve and moderate degree of impairment (Brooke = 2) rather than individuals with mild upper extremity impairment (Brooke = 1) or individuals who were more severely impaired (Brooke =3). Discussion: The developed reachable workspace evaluation method using scalable 3D vision technology appears promising as an outcome measure system for clinical studies. A rationally-designed combination of upper extremity outcome measures including a region-specific global upper extremity outcome measure, such as the reachable workspace, complemented by targeted disease- or function-specific endpoints, may be optimal for future clinical efficacy trials.

Real-time 2D height mapping method for an unmanned vehicle using a stereo camera and laser sensor fusion

This paper proposes a method for mapping the height information on an area around a vehicle and of identifying a drivable area by fusing a stereo camera and a laser sensor. A SOM (Self Organizing Map) clustering algorithm obtained from the depth information of the stereo camera is used to analyze the front part area of a vehicle in forms of several candidate planes. In addition, an IMU indicating the current pose of a vehicle is applied to detect a drivable plane. A laser sensor installed on a vehicle’s roof scans the front part with a single line and informs a distance value. A drivable plane detected is utilized to calculate height value in the normal direction detected by the laser scan data. Additionally, when the height already mapped has a value higher than that of the threshold, it is regarded as an obstacle and the vehicle is prevented from coming into contact with it. Regarding the vehicle position estimation, a Kalman filter method is used for real-time mapping during driving. The moving location of the vehicle is dead reckoned based on steering angle and velocity, and this value is compensated using the position value received from the GPS. The vehicle’s position and mapping coordinates are converted into latitude and longitude values. This study demonstrates that it is possible to generate a precise 2D height map by conducting a test in a real road environment with various slope angles and obstacles.

Visual Odometry Based on Structural Matching of Local Invariant Features Using Stereo Camera Sensor

This paper describes a novel sensor system to estimate the motion of a stereo camera. Local invariant image features are matched between pairs of frames and linked into image trajectories at video rate, providing the so-called visual odometry, i.e., motion estimates from visual input alone. Our proposal conducts two matching sessions: the first one between sets of features associated to the images of the stereo pairs and the second one between sets of features associated to consecutive frames. With respect to previously proposed approaches, the main novelty of this proposal is that both matching algorithms are conducted by means of a fast matching algorithm which combines absolute and relative feature constraints. Finding the largest-valued set of mutually consistent matches is equivalent to finding the maximum-weighted clique on a graph. The stereo matching allows to represent the scene view as a graph which emerge from the features of the accepted clique. On the other hand, the frame-to-frame matching defines a graph whose vertices are features in 3D space. The efficiency of the approach is increased by minimizing the geometric and algebraic errors to estimate the final displacement of the stereo camera between consecutive acquired frames. The proposed approach has been tested for mobile robotics navigation purposes in real environments and using different features. Experimental results demonstrate the performance of the proposal, which could be applied in both industrial and service robot fields.