Eye View Map Research Articles

BEV Semantic Map Reconstruction for Self-Driving Cars with the Multi-Head Attention Mechanism

Environmental perception is crucial for safe autonomous driving, enabling accurate analysis of the vehicle’s surroundings. While 3D LiDAR is traditionally used for 3D environment reconstruction, its high cost and complexity present challenges. In contrast, camera-based cross-view frameworks can offer a cost-effective alternative. Hence, this manuscript proposes a new cross-view model to extract mapping features from camera images and then transfer them to a Bird’s-Eye View (BEV) map. Particularly, a multi-head attention mechanism in the decoder architecture generates the final semantic map. Each camera learns embedding information corresponding to its position and angle within the BEV map. Cross-view attention fuses information from different perspectives to predict top-down map features enriched with spatial information. The multi-head attention mechanism then globally performs dependency matches, enhancing long-range information and capturing latent relationships between features. Transposed convolution replaces traditional upsampling methods, avoiding high similarities of local features and facilitating semantic segmentation inference of the BEV map. Finally, we conduct numerous simulation experiments to verify the performance of our cross-view model.

Addressing Diverging Training Costs Using BEVRestore for High-Resolution Bird's Eye View Map Construction

Addressing Diverging Training Costs Using BEVRestore for High-Resolution Bird's Eye View Map Construction

Dynamic Scene Representation for Docking in Urban Waters Using a Stereo Camera

Abstract To navigate and dock safely in urban waters, an autonomous ferry must assess its safe navigation paths by identifying navigable free space and potential obstacles. A deep understanding of its surroundings is crucial for situational awareness. Knowing which objects are static and dynamic is an indispensable ingredient for path planning algorithms and collision avoidance systems, and is especially useful for docking operations, as moving obstacles can interfere with the optimal path to the dock. To handle all types of obstacles, we present a novel dynamic scene representation for docking in urban waters using a short baseline stereo camera. Obstacles protruding from the water surface are represented with vertically oriented rectangles, known as stixels. The stixels are tracked over consecutive frames with dense optical flow. Together with stereo correspondences, their 3D motion is estimated. Stixels representing the same object are grouped together into line segments which are classified as either dynamic or static based on the motion of its associated stixels. Our dynamic representation presents objects in the 3D camera coordinate system as a bird’s eye view map. Additionally, stereo depth uncertainty can significantly impact depth estimates and is considered using a Kalman filter, providing smooth position and velocity estimates. We test our dynamic scene representation on a real docking scenario recorded with an autonomous ferry using a stereo camera. We evaluate the accuracy of the reconstructed 3D scene points and the velocity estimates of the dynamic objects using LiDAR and GNSS.

Famous places along the Keifuku Electric Railway line in the prewar period as depicted in Hatsusaburo Yoshida's style bird's eye view map

Famous places along the Keifuku Electric Railway line in the prewar period as depicted in Hatsusaburo Yoshida's style bird's eye view map

CI3D: Context Interaction for Dynamic Objects and Static Map Elements in 3D Driving Scenes.

Multi-view 3D visual perception including 3D object detection and Birds'-eye-view (BEV) map segmentation is essential for autonomous driving. However, there has been little discussion about 3D context attention between dynamic objects and static elements with multi-view camera inputs, due to the challenging nature of recovering the 3D spatial information from images and performing effective 3D context interaction. 3D context information is expected to provide more cues to enhance 3D visual perception for autonomous driving. We thus propose a new transformer-based framework named CI3D in an attempt to implicitly model 3D context interaction between dynamic objects and static map elements. To achieve this, we use dynamic object queries and static map queries to gather information from multi-view image features, which are represented sparsely in 3D space. Moreover, a dynamic 3D position encoder is utilized to precisely generate queries' positional embeddings. With accurate positional embeddings, the queries effectively aggregate 3D context information via a multi-head attention mechanism to model 3D context interaction. We further reveal that sparse supervision signals from the limited number of queries result in the issue of rough and vague image features. To overcome this challenge, we introduce a panoptic segmentation head as an auxiliary task and a 3D-to-2D deformable cross-attention module, greatly enhancing the robustness of spatial feature learning and sampling. Our approach has been extensively evaluated on two large-scale datasets, nuScenes and Waymo, and significantly outperforms the baseline method on both benchmarks.

Semantic interpretation of raw survey vehicle sensory data for lane-level HD map generation

High-definition (HD) maps provide a complementary source of information for Advanced Driver Assistance Systems (ADAS), allowing them to better understand the vehicle’s surroundings and make more informed decisions. HD maps are also largely employed in virtual testing phases to evaluate the behavior of ADAS components under simulated conditions. With the advent of autonomous sensorized vehicles, raw machine-oriented data will be increasingly available. The proposed pipeline aims to provide a high-level semantic interpretation of raw vehicle sensory data to derive, in an automated fashion, lane-oriented HD maps of the environment. We first present RoadStarNet, a deep learning architecture designed to extract and classify road line markings from imagery data. We show how to obtain a semantic Bird’s-Eye View (BEV) mapping of the extracted road line markings by exploiting frame-by-frame localization information. Then, we present how to progress to a graph-based representation that allows modeling complex road line markings’ structures practically, as this representation can be leveraged to produce a Lanelet2 format HD map. Lastly, we experimentally evaluate the proposed approach in real-world scenarios in terms of accuracy and coverage performance.

Generating evidential BEV maps in continuous driving space

Safety is critical for autonomous driving, and one aspect of improving safety is to accurately capture the uncertainties of the perception system, especially knowing the unknown. Different from only providing deterministic or probabilistic results, e.g., probabilistic object detection, that only provide partial information for the perception scenario, we propose a complete probabilistic model named GevBEV. It interprets the 2D driving space as a probabilistic Bird’s Eye View (BEV) map with point-based spatial Gaussian distributions, from which one can draw evidence as the parameters for the categorical Dirichlet distribution of any new sample point in the continuous driving space. The experimental results show that GevBEV not only provides more reliable uncertainty quantification but also outperforms the previous works on the benchmarks OPV2V and V2V4Real of BEV map interpretation for cooperative perception in simulated and real-world driving scenarios, respectively. A critical factor in cooperative perception is the data transmission size through the communication channels. GevBEV helps reduce communication overhead by selecting only the most important information to share from the learned uncertainty, reducing the average information communicated by 87% with only a slight performance drop. Our code is published at https://github.com/YuanYunshuang/GevBEV.

Cooperative Perception With V2V Communication for Autonomous Vehicles

Occlusion is a critical problem in the Autonomous Driving System. Solving this problem requires robust collaboration among autonomous vehicles traveling on the same roads. However, transferring the entirety of raw sensors' data among autonomous vehicles is expensive and can cause a delay in communication. This paper proposes a method called Realtime Collaborative Vehicular Communication based on Bird's-Eye-View (BEV) map. The BEV map holds the accurate depth information from the point cloud image while its 2D representation enables the method to use a novel and well-trained image-based backbone network. Most importantly, we encode the object detection results into the BEV representation to reduce the volume of data transmission and make real-time collaboration between autonomous vehicles possible. The output of this process, the BEV map, can also be used as direct input to most route planning modules. Numerical results show that this novel method can increase the accuracy of object detection by cross-verifying the results from multiple points of view. Thus, in the process, this new method also reduces the object detection challenges that stem from occlusion and partial occlusion. Additionally, different from many existing methods, this new method significantly reduces the data needed for transfer between vehicles, achieving a speed of 21.92 Hz for both the object detection process and the data transmission process, which is sufficiently fast for a real-time system.

Monocular Road Scene Bird’s Eye View Prediction via Big Kernel-Size Encoder and Spatial-Channel Transform Module

A detailed representation of the surrounding road scene is crucial for an autonomous driving system. yellow The camera-based Bird’s Eye View map has been a popular solution to present the surrounding information, due to its low cost and rich spatial context information. Most of the existing methods predict the BEV map based on the depth-estimation or the trivial homography method, which may cause the error propagation and the absence of content. To overcome these drawbacks, we propose a novel end-to-end framework that employs the front monocular image to predict the road layout and vehicle occupancy. In particular, to capture the long-range feature, we redesign a CNN encoder with a large kernel size to extract the image features. For reducing the big difference between the front image features and the top-down features, we propose a novel Spatial-Channel projection module to convert the front map into the top-down space. Additionally, concerning the correlation between front view and top-down view, we propose the Dual Cross-view Transformer module to refine the top-down view feature maps and strengthen the transformation. Extensive evaluations on the KITTI and Argoverse datasets present that the proposed model achieves the state-of-the-art results for both datasets. Furthermore, the proposed model runs in 37 FPS on a single GPU, demonstrating the generation of a real-time BEV map. The code will be published at https://github.com/raozhongyu/BEV_LKA.

GAF-RCNN: Grid attention fusion 3D object detection from point cloud

Background: Due to the refinement of region of the interests (RoIs), two-stage 3D detection algorithms can usually obtain better performance compared with most single-stage detectors. However, most two-stage methods adopt feature connection, to aggregate the grid point features using multi-scale RoI pooling in the second stage. This connection mode does not consider the correlation between multi-scale grid features. Methods: In the first stage, we employ 3D sparse convolution and 2D convolution to fully extract rich semantic features. Then, a small number of coarse RoIs are predicted based region proposal network (RPN) on generated bird’s eye view (BEV) map. After that, we adopt voxel RoI-pooling strategy to aggregate the neighborhood nonempty voxel features of each grid point in RoI in the last two layers of 3D sparse convolution. In this way, we obtain two aggregated features from 3D sparse voxel space for each grid point. Next, we design an attention feature fusion module. This module includes a local and a global attention layer, which can fully integrate the grid point features from different voxel layers. Results: We carry out relevant experiments on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset. The average precisions of our proposed method are 88.21%, 81.51%, 77.07% on three difficulty levels (easy, moderate, and hard, respectively) for 3D detection, and 92.30%, 90.19%, 86.00% on three difficulty levels (easy, moderate, and hard, respectively) for BEV detection. Conclusions: In this paper, we propose a novel two-stage 3D detection algorithm named Grid Attention Fusion Region-based Convolutional Neural Network (GAF-RCNN) from point cloud. Because we integrate multi-scale RoI grid features with attention mechanism in the refinement stage, different multi-scale features can be better correlated, achieving a competitive level compared with other well tested detection algorithms. This 3D object detection has important implications for robot and cobot technology.

Bird’s-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

Bird’s-Eye-View (BEV) maps have emerged as one of the most powerful representations for scene understanding due to their ability to provide rich spatial context while being easy to interpret and process. Such maps have found use in many real-world tasks that extensively rely on accurate scene segmentation as well as object instance identification in the BEV space for their operation. However, existing segmentation algorithms only predict the semantics in the BEV space, which limits their use in applications where the notion of object instances is also critical. In this work, we present the first BEV panoptic segmentation approach for directly predicting dense panoptic segmentation maps in the BEV, given a single monocular image in the frontal view (FV). Our architecture follows the top-down paradigm and incorporates a novel dense transformer module consisting of two distinct transformers that learn to independently map vertical and flat regions in the input image from the FV to the BEV. Additionally, we derive a mathematical formulation for the sensitivity of the FV-BEV transformation which allows us to intelligently weight pixels in the BEV space to account for the varying descriptiveness across the FV image. Extensive evaluations on the KITTI-360 and nuScenes datasets demonstrate that our approach exceeds the state-of-the-art in the PQ metric by 3.61 pp and 4.93 pp respectively.

SIEV-Net: A Structure-Information Enhanced Voxel Network for 3D Object Detection From LiDAR Point Clouds

As one of the fundamental tasks in scene understanding, 3D object detection from LiDAR point clouds has drawn extensive attention in the past few years. Although the existing voxel-based methods have achieved remarkable performance, how to effectively exploit geometric structure information of the point clouds to boost the detection performance remains to be explored. In this paper, we propose a novel structure-information enhanced voxel network (SIEV-Net) for 3D object detection from LiDAR point clouds. The proposed SIEV-Net learns feature representations of 3D objects by jointly considering uneven spatial distribution and height information of the point clouds. Specifically, considering the uneven spatial distribution characteristics of point clouds, a hierarchical-voxel feature encoding module is proposed to effectively extract features of voxels in both sparse and dense regions. Besides, by utilizing the Bird’s Eye View (BEV) map of point clouds, a height information complement module is designed to minimize the height information lost in the process of point feature aggregation in a voxel network. Experimental results on the widely used KITTI benchmark dataset have demonstrated the efficacy of the proposed SIEV-Net.

Efficient LiDAR Odometry for Autonomous Driving

LiDAR odometry plays an important role in self-localization and mapping for autonomous navigation, which is usually treated as a scan registration problem. Although having achieved promising performance on the KITTI odometry benchmark, the conventional tree-based neighbor search still has the difficulty in dealing with the large-scale point cloud efficiently. The recent spherical range image-based method enjoys the merits of fast nearest neighbor search by spherical mapping. However, it is not very effective to deal with the ground points nearly parallel to LiDAR beams. To address these issues, we propose a novel efficient LiDAR odometry approach by taking advantage of both non-ground spherical range images and bird's-eye-view maps for ground points. Moreover, a range adaptive method is introduced to robustly estimate the local surface normal. Additionally, a very fast and memory-efficient model update scheme is proposed to fuse the points and their corresponding normals at different time-stamps. We have conducted extensive experiments on the KITTI odometry benchmark and UrbanLoco dataset, whose promising results demonstrate that our proposed approach is effective.

Hatsusaburo Yoshida’s Bird’s-Eye View Map of Colonial-Era Busan and the Representation of Other Places - What is Depicted and What is Not Depicted -

본 연구는 근대 식민지도시 부산을 그린 요시다 하츠사부로(吉田初三郞)의 조감도를 분석대상으로 한다. 조감도의 제작배경과 구도를 살펴보고, 조감도 지리정보의 데이터베이스화 및 묘사내용 분석을 통하여 타자/타소(他所)표상에 대하여 고찰한 것이다. 이 조감도는 1929년 조선박람회 개최에 맞추어 선전용으로 제작된 것이다. 조선총독부의 조선박람회 개최목적은 일본 내지인들에게 조선을 선전하는 것으로, ‘조선에 대한 올바른 이해’ 를 위한 것이었다. 조선 각 지방 일본인 사회는 조선박람회를 지역 진흥과 관광객 유치를 위한 절호의 기회로 삼았다. 이러한 배경을 바탕으로 하츠사부로 조감도에 그려진 식민지도시 부산의 경관은 어디까지나 제국일본의 ‘시선’에 의해 선택된 요소로만 구성되어 있다. 조선 사람들의 시간의 흐름과 삶의 터전은 불평등하고 불균형한 역학관계 속에서 은폐, 망각, 배제된 채 주변화되어 있다. 거기에는 보는 측과 보이는 측의 사이에 메워지지 않는 ‘거리’ 가 있었다

Multi-objective Optimization Based Deep Reinforcement Learning for Autonomous Driving Policy

End-to-end autonomous driving approach seeks to solve the problems of perception, decision and control in an integrated way, which can better adapt to the new traffic scene. Due to the diversity of traffic scenes and the uncertainty of the interaction among surrounding vehicles, the design of autonomous driving policy is challenging. Many current methods manually design the corresponding driving policy for different traffic scene, resulting in suboptimal solutions and the maintaining is hard. Most of the existing deep reinforcement learning (DRL) methods can’t work well in the complex urban traffic scenes because of the sensing and simple driving policy. In this paper, to extend the adaptability of the SAC-based method, we proposed to take the multiple sensor data as input, and a VAE network was used to enhance the quality of training data for SAC-based DRL method. A multi-constraints reward function for SAC-based driving policy training is designed, which account for the errors of transverse distance, longitudinal distance, heading, velocity and the possibility of collision. The multiple sensor data include the original RGB image captured by forward-view camera, a 3D LiDAR and the bird’s-eye view map resulted from the perception processing, the mixed inputs could enrich the capability of scene perception. The proposed approach is validated with the multi-vehicle traffic simulation built with CARLA[1]. The results showed that the simulated vehicle could adapt to more challenging traffic scenes, like passing intersections and turning in crowded urban scenes, etc with the driving policy generated by the proposed method, and its performances are obviously outperformed against the existing similar methods.

Adaptive and azimuth-aware fusion network of multimodal local features for 3D object detection

This paper focuses on the construction of strong local features and the effective fusion of image and LiDAR data for 3D object detection. We adopt different modalities of LiDAR data to generate rich features and present an adaptive and azimuth-aware network to aggregate local features from image, bird’s eye view maps and point cloud. Our network mainly consists of three subnetworks: ground plane estimation network, region proposal network and adaptive fusion network. The ground plane estimation network extracts features of point cloud and predicts the parameters of a plane which are used for generating abundant 3D anchors. The region proposal network generates features of image and bird’s eye view maps to output region proposals. To integrate heterogeneous image and point cloud features, the adaptive fusion network explicitly adjusts the intensity of multiple local features and achieves the orientation consistency between image and LiDAR data by introducing an azimuth-aware fusion module. Experiments are conducted on KITTI dataset and the results validate the advantages of our aggregation of multimodal local features and the adaptive fusion network.

Historical coastal maps: importance and challenges in their use in studying coastal geomorphology

Historical maritime maps such as navigation charts, bird’s-eye view maps and plan maps preserve useful coastal information of the time. Through the archival survey of such documents, a large number of charts and maps (particularly of large and medium scale) have been found that were published from sixteenth-century CE onwards. These maps– depict manmade structures such as forts and other buildings visible to sailors approaching the coast; and mark geomorphic features such as spit, bar, shoal, delta; with appropriate shape, relative position and size. Understanding the geomorphic features present in these maps and analyzing them using remote sensing imagery, historical text and recent studies can immensely enhance knowledge of coastal geomorphic changes in the interim period. However, there are challenges in analyzing these maps, such as scale variability within or across the maps, date of publication versus survey, originality of map content (copied versus original), the importance of the place to the cartographer, the purpose of making the map, cartographic skills, style and methods, etc. The present study discusses the untapped potential of historical maritime maps and charts to understand coastal geomorphology along with the challenges one can confront while studying them and suggests ways to overcome them.

Feature Deep Continuous Aggregation for 3D Vehicle Detection

3D object detection has recently become a research hotspot in the field of autonomous driving. Although great progress has been made, it still needs to be further improved. Therefore, this paper presents FDCA, a feature deep continuous aggregation network using multi-sensors for 3D vehicle detection. The proposed network adopts a two-stage structure with the bird’s-eye view (BEV) map and the RGB image as an input. In the first stage, two feature extractors were used to generate feature maps with the high-resolution and representational ability for each input view. These feature maps were then fused and fed to a 3D proposal generator to obtain the reliable 3D vehicle proposals. In the second stage, the refinement network aggregated the features of the proposal regions further and performed classifications, a 3D bounding boxes regression, and orientation estimations to predict the location and heading of vehicles in 3D space. The FDCA network proposed was trained and evaluated on the KITTI 3D object detection benchmark. The experimental results of the validation set illustrated that compared with other fusion-based methods, the 3D average precision (AP) could achieve 76.82% on a moderate setting while having real-time capability, which was higher than that of the second-best performing method by 2.38%. Meanwhile, the results of ablation experiments show that the convergence rate of FDCA was much faster and the stability was also much better, making it a candidate for application in autonomous driving.

Monocular Semantic Occupancy Grid Mapping With Convolutional Variational Encoder–Decoder Networks

In this work, we research and evaluate end-to-end learning of monocular semantic-metric occupancy grid mapping from weak binocular ground truth. The network learns to predict four classes, as well as a camera to bird's eye view mapping. At the core, it utilizes a variational encoder-decoder network that encodes the front-view visual information of the driving scene and subsequently decodes it into a 2-D top-view Cartesian coordinate system. The evaluations on Cityscapes show that the end-to-end learning of semantic-metric occupancy grids outperforms the deterministic mapping approach with flat-plane assumption by more than 12% mean IoU. Furthermore, we show that the variational sampling with a relatively small embedding vector brings robustness against vehicle dynamic perturbations, and generalizability for unseen KITTI data. Our network achieves real-time inference rates of approx. 35 Hz for an input image with a resolution of 256x512 pixels and an output map with 64x64 occupancy grid cells using a Titan V GPU.

Adaptive Vision Based Condition Monitoring and Fault Detection Method for Multi Robots at Production Lines in Industrial Systems

Continuity of production is a highly important in the days that manufacturing is becoming bigger and serial. The mistakes done while producing process cause fail on products and it may bring about even big losses for the facility. Furthermore, hitches on robots at production line may also cause crucial damages that may give rise to high repair costs and discontinuance of production. In this study, it is aimed to obtain alive bird's eye view map of production lines, which are big and impossible to be monitored with only one camera, by using multi cameras and stitching algorithms. Finding the similar scenes of input images, estimation of homography, warping and blending operations, which are the steps used in feature based image-stitching algorithms, are applied respectively on images that are taken by cameras. The assignment of second nearest neighbor distance rate adaptively makes the results more qualified. After obtaining single stitched image movement detection is actualized by using the difference of sequential frames, and anomaly movements are determined. As a result, the robots at the long production lines can be monitored in one screen, and with processing the obtained image, faults on robots that may cause damage at non-cheap machines can be handled before time.