Surround-view Cameras Research Articles

Review and analysis of the properties of 360-degree surround view cameras in autonomous vehicles

Autonomous vehicles are becoming increasingly prevalent in today's world, and the demand for efficient and safe self-driving technology has never been higher. One key component of this technology is the 360-degree surround view camera, which provides a complete view of the vehicle's surroundings, allowing it to navigate safely and efficiently. In this paper, we conduct a comprehensive review and analysis of the properties of these cameras in autonomous vehicles. We begin by exploring the various types of 360-degree cameras available, including fisheye, parabolic, and mirror-based designs. We then delve into the technical specifications of these cameras, including resolution, frame rate, and field of view. We also examine the various image processing techniques used to enhance the quality of the images captured by these cameras, such as image stabilization and noise reduction. Next, we discuss the benefits and drawbacks of using 360-degree surround view cameras in autonomous vehicles. On the one hand, these cameras offer a complete view of the vehicle's surroundings, allowing for safe and efficient navigation. On the other hand, they can be expensive and may not always provide the necessary level of detail for certain driving scenarios. We also discuss the various challenges that must be overcome when using 360-degree cameras in autonomous vehicles, such as occlusions, lighting conditions, and the need for real-time processing. Finally, we conclude by discussing the future of 360-degree camera technology in autonomous vehicles and the potential for further advancements in this field. Overall, this paper provides a comprehensive review and analysis of the properties of 360-degree surround view cameras in autonomous vehicles, offering insights into their benefits, drawbacks, and technical specifications.

Semantic closed-loop based visual mapping algorithm for automated valet parking

Simultaneous localization and mapping (SLAM) is one of the core technologies to realize automated valet parking (AVP). Currently, advanced visual feature-based SLAM systems suffer from feature extraction failure and tracking loss due to the constraints of textureless scenes, unclear illumination, and dynamic conditions. To address these problems, this paper proposes a visual SLAM algorithm based on a semantic closed-loop detection algorithm using surround-view cameras and inertial measurement units (IMU) as sensors. The algorithm combines semantic features and the idea of inverse index to improve the traditional keyframes selection methods and the closed-loop detection algorithms, effectively avoiding the tedious and complicated feature point matching and improving the computational efficiency of the computer. Experiments show that the algorithm in this paper achieves better results in terms of precision and recall, absolute trajectory error (ATE), and relative pose error (RPE), and can meet the demand for SLAM and subsequent navigation in indoor parking lots.

SynWoodScape: Synthetic Surround-View Fisheye Camera Dataset for Autonomous Driving

Surround-view cameras are a primary sensor for automated driving, used for near-field perception. It is one of the most commonly used sensors in commercial vehicles primarily used for parking visualization and automated parking. Four fisheye cameras with a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$190^{\circ }$</tex-math></inline-formula> field of view cover the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> around the vehicle. Due to its high radial distortion, the standard algorithms do not extend easily. Previously, we released the first public fisheye surround-view dataset named WoodScape. In this work, we release a synthetic version of the surround-view dataset, covering many of its weaknesses and extending it. Firstly, it is not possible to obtain ground truth for pixel-wise optical flow and depth. Secondly, WoodScape did not have all four cameras annotated simultaneously in order to sample diverse frames. However, this means that multi-camera algorithms cannot be designed to obtain a unified output in birds-eye space, which is enabled in the new dataset. We implemented surround-view fisheye geometric projections in CARLA Simulator matching WoodScape’s configuration and created SynWoodScape. We release 80 k images from the synthetic dataset with annotations for 10+ tasks. We also release the baseline code and supporting scripts.

Online Correction of Camera Poses for the Surround-view System: A Sparse Direct Approach

The surround-view module is an indispensable component of a modern advanced driving assistance system. By calibrating the intrinsics and extrinsics of the surround-view cameras accurately, a top-down surround-view can be generated from raw fisheye images. However, poses of these cameras sometimes may change. At present, how to correct poses of cameras in a surround-view system online without re-calibration is still an open issue. To settle this problem, we introduce the sparse direct framework and propose a novel optimization scheme of a cascade structure. This scheme is actually composed of two levels of optimization and two corresponding photometric error based models are proposed. The model for the first-level optimization is called the ground model, as its photometric errors are measured on the ground plane. For the second level of the optimization, it’s based on the so-called ground-camera model, in which photometric errors are computed on the imaging planes. With these models, the pose correction task is formulated as a nonlinear least-squares problem to minimize photometric errors in overlapping regions of adjacent bird’s-eye-view images. With a cascade structure of these two levels of optimization, an appropriate balance between the speed and the accuracy can be achieved. Experiments show that our method can effectively eliminate the misalignment caused by cameras’ moderate pose changes in the surround-view system. Source code and test cases are available online at https://cslinzhang.github.io/CamPoseCorrection/ .

Visual SLAM in Long-Range Autonomous Parking Application Based on Instance-Aware Semantic Segmentation via Multi-Task Network Cascades and Metric Learning Scheme

&lt;div class="section abstract"&gt;&lt;div class="htmlview paragraph"&gt;Long-range Autonomous Parking is becoming an attractive application in terms of demands. The vehicle is capable of driving autonomously into the appointed parking slot when the driver leaves it at the drop-off spot. In this application, the ability of accurate localization has become a key issue, especially in GPS-denied environments. This paper proposes a method of localization and mapping for Long-range Autonomous Parking, which is achieved by Visual SLAM based on deep learning algorithms. Firstly, we propose an instance segmentation via multi-task network cascades, and even in a complex visual environment, the main roadway instances of interest in the parking lot IPM image can be detected, such as parking corners, speed bumps. Then we combine the information of wheel encoders to build a global semantic map of the parking lot. Vehicles can often rely on semantic map matching to achieve high-precision localization. However, without a good initial position, it is difficult to infer an accurate position by matching the semantic map, such as randomly selecting entrances to enter the parking lot. Therefore, we propose an area feature network based on metric learning to extract features that distinguish different areas and infer the approximate initial position of the vehicle. Specifically, we extract features from the images of the surround-view cameras, use the vehicle position as weak supervision, and finally construct an area feature map. In summary, our proposed method provides accurate vehicle localization and parking lot maps for Long-range Autonomous Parking.&lt;/div&gt;&lt;/div&gt;

OmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving

Surround View fisheye cameras are commonly deployed in automated driving for 360 $^\circ$ near-field sensing around the vehicle. This work presents a multi-task visual perception network on unrectified fisheye images to enable the vehicle to sense its surrounding environment. It consists of six primary tasks necessary for an autonomous driving system: depth estimation, visual odometry, semantic segmentation, motion segmentation, object detection, and lens soiling detection. We demonstrate that the jointly trained model performs better than the respective single task versions. Our multi-task model has a shared encoder providing a significant computational advantage and has synergized decoders where tasks support each other. We propose a novel camera geometry based adaptation mechanism to encode the fisheye distortion model both at training and inference. This was crucial to enable training on the WoodScape dataset, comprised of data from different parts of the world collected by 12 different cameras mounted on three different cars with different intrinsics and viewpoints. Given that bounding boxes is not a good representation for distorted fisheye images, we also extend object detection to use a polygon with non-uniformly sampled vertices. We additionally evaluate our model on standard automotive datasets, namely KITTI and Cityscapes. We obtain the state-of-the-art results on KITTI for depth estimation and pose estimation tasks and competitive performance on the other tasks. We perform extensive ablation studies on various architecture choices and task weighting methodologies. A short video at https://youtu.be/xbSjZ5OfPes provides qualitative results.

Stereo Vision by Combination of Machine-Learning Techniques for Pedestrian Detection at Intersections Utilizing Surround-View Cameras

The frequency of pedestrian traffic accidents continues to increase in Japan. Thus, a driver assistance system is expected to reduce the number of accidents. However, it is difficult for the current environmental recognition sensors to detect crossing pedestrians when turning at intersections, owing to the field of view and the cost. We propose a pedestrian detection system that utilizes surround-view fisheye cameras with a wide field of view. The system can be realized at low cost if the fisheye cameras are already equipped. It is necessary to detect the positioning of pedestrians accurately because more precise prediction of future collision points is required at intersections. Stereo vision is suitable for this purpose. However, there are some concerns regarding realizing stereo vision using fisheye cameras due to the distortion of the lens, asynchronous capturing, and fluctuating camera postures. As a countermeasure, we propose a novel method combining various machine-learning techniques. The D-Brief with histogram of oriented gradients and normalized cross-correlation are combined by a support-vector machine for stereo matching. A random forest was adopted to discriminate the pedestrians from noise in the 3D reconstructed point cloud. We evaluated this for images of crossing pedestrians at actual intersections. A tracking rate of 96.0% was achieved as the evaluation result. It was verified that this algorithm can accurately detect a pedestrian with an average position error of 0.17 m.

Lane-level localization system using surround-view cameras adaptive to different driving conditions

This article presents a lane-level localization system adaptive to different driving conditions, such as occlusions, complicated road structures, and lane-changing maneuvers. The system uses surround-view cameras, other low-cost sensors, and a lane-level road map which suits for mass deployment. A map-matching localizer is proposed to estimate the probabilistic lateral position. It consists of a sub-map extraction module, a perceptual model, and a matching model. A probabilistic lateral road feature is devised as a sub-map without limitations of road structures. The perceptual model is a deep learning network that processes raw images from surround-view cameras to extract a local probabilistic lateral road feature. Unlike conventional deep-learning-based methods, the perceptual model is trained by auto-generated labels from the lane-level map to reduce manual effort. The matching model computes the correlation between the sub-map and the local probabilistic lateral road feature to output the probabilistic lateral estimation. A particle-filter-based framework is developed to fuse the output of map-matching localizer with the measurements from wheel speed sensors and an inertial measurement unit. Experimental results demonstrate that the proposed system provides the localization results with submeter accuracy in different driving conditions.

Restricted Deformable Convolution-Based Road Scene Semantic Segmentation Using Surround View Cameras

Understanding the surrounding environment of the vehicle is still one of the challenges for autonomous driving. This paper addresses 360-degree road scene semantic segmentation using surround view cameras, which are widely equipped in existing production cars. First, in order to address large distortion problem in the fisheye images, Restricted Deformable Convolution (RDC) is proposed for semantic segmentation, which can effectively model geometric transformations by learning the shapes of convolutional filters conditioned on the input feature map. Second, in order to obtain a large-scale training set of surround view images, a novel method called zoom augmentation is proposed to transform conventional images to fisheye images. Finally, an RDC based semantic segmentation model is built; the model is trained for real-world surround view images through a multi-task learning architecture by combining real-world images with transformed images. Experiments demonstrate the effectiveness of the RDC to handle images with large distortions, and that the proposed approach shows a good performance using surround view cameras with the help of the transformed images.

INTRODUCING NOVEL GENERATION OF HIGH ACCURACY CAMERA OPTICAL-TESTING AND CALIBRATION TEST-STANDS FEASIBLE FOR SERIES PRODUCTION OF CAMERAS

Abstract. The recent advances in the field of computer-vision have opened the doors of many opportunities for taking advantage of these techniques and technologies in many fields and applications. Having a high demand for these systems in today and future vehicles implies a high production volume of video cameras. The above criterions imply that it is critical to design test systems which deliver fast and accurate calibration and optical-testing capabilities. In this paper we introduce new generation of test-stands delivering high calibration quality in single-shot calibration of fisheye surround-view cameras. This incorporates important geometric features from bundle-block calibration, delivers very high (sub-pixel) calibration accuracy, makes possible a very fast calibration procedure (few seconds), and realizes autonomous calibration via machines. We have used the geometrical shape of a Spherical Helix (Type: 3D Spherical Spiral) with special geometrical characteristics, having a uniform radius which corresponds to the uniform motion. This geometrical feature was mechanically realized using three dimensional truncated icosahedrons which practically allow the implementation of a spherical helix on multiple surfaces. Furthermore the test-stand enables us to perform many other important optical tests such as stray-light testing, enabling us to evaluate the certain qualities of the camera optical module.