Bird's Eye View Image Research Articles

LiDAR‐based place recognition for mobile robots in ground/water surface multiple scenes

AbstractLiDAR‐based 3D place recognition is an essential component of simultaneous localization and mapping systems in multi‐scene robotic applications. However, extracting discriminative and generalizable global descriptors of point clouds is still an open issue due to the insufficient use of the information contained in the LiDAR scans in existing approaches. In this paper, we propose a novel spatial‐temporal point cloud encoding network for multiple scenes, dubbed STM‐Net, to fully fuse the multi‐view spatial information and temporal information of LiDAR point clouds. Specifically, we first develop a spatial feature encoding module consisting of the single‐view transformer and multi‐view transformer. The module learns the correlation both within a single view and between two views by utilizing the multi‐layer range images generated by spherical projection and multi‐layer bird's eye view images generated by top‐down projection. Then in the temporal feature encoding module, we exploit the temporal transformer to mine the temporal information in the sequential point clouds, and a NetVLAD layer is applied to aggregate features and generate sub‐descriptors. Furthermore, we use a GeM pooling layer to fuse more information along the time dimension for the final global descriptors. Extensive experiments conducted on unmanned ground/surface vehicles with different LiDAR configurations indicate that our method (1) achieves superior place recognition performance than state‐of‐the‐art algorithms, (2) generalizes well to diverse sceneries, (3) is robust to viewpoint changes, (4) can operate in real‐time, demonstrating the effectiveness and satisfactory capability of the proposed approach and highlighting its promising applications in multi‐scene place recognition tasks.

Local Climate Zone Classification Using YOLOV8 Modeling in Instance Segmentation Method

Local climate zones play a crucial role in understanding the microclimates within urban areas, contributing to urban planning, environmental sustainability, and human comfort. Istanbul, as a transcontinental city straddling Europe and Asia, exhibits a rich blend of historical and modern architecture, varying land use patterns, and diverse microclimates. In this study, using high-resolution Google Earth imagery for explores the classification, utilizing a cutting-edge deep learning architecture YOLOv8 model, of Local Climate Zones (LCZ) in Istanbul, a city known for its diverse and dynamic urban landscape. The latest cutting-edge YOLO model, YOLOv8, is designed for tasks such as object detection, image classification, and instance segmentation, showcasing its versatility in computer vision applications. Labeled data was created according to WUDAPT's sharing the things to consider when "create LCZ training areas" from google earth images. The model is trained on high-resolution, bird's-eye-view images of Istanbul obtained from Google Earth, meticulously labeled with LCZ categories. The results obtained from the test images demonstrate the model's efficacy in accurately classifying and segmenting LCZ categories, providing valuable insights into the local climate variations within Istanbul. This research contributes to the field of urban climate studies by offering a robust and scalable approach to LCZ classification using advanced deep learning techniques. The outcomes hold implications for urban planning, environmental sustainability, and informed decision-making in the context of Istanbul's unique and diverse urban environment.

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints.

Autonomous driving has the potential to revolutionize transportation, but developing safe and reliable systems remains a significant challenge. Reinforcement learning (RL) has emerged as a promising approach for learning optimal control policies in complex driving environments. However, existing RL-based methods often suffer from low sample efficiency and lack explicit safety constraints, leading to unsafe behaviors. In this paper, we propose a novel framework for safe reinforcement learning in autonomous driving that addresses these limitations. Our approach incorporates a latent dynamic model that learns the underlying dynamics of the environment from bird's-eye view images, enabling efficient learning and reducing the risk of safety violations by generating synthetic data. Furthermore, we introduce state-wise safety constraints through a barrier function, ensuring safety at each state by encoding constraints directly into the learning process. Experimental results in the CARLA simulator demonstrate that our framework significantly outperforms baseline methods in terms of both driving performance and safety. Our work advances the development of safe and efficient autonomous driving systems by leveraging the power of reinforcement learning with explicit safety considerations.

A method for estimating ship berthing angle based on 2D bird's eye view point cloud

The accurate estimation of berthing angle is essential for the parallel berthing of inshore ships. Herein, a new method for calculating the ship berthing angle using a shore-based LiDAR system is proposed. First, the LiDAR-scanned point cloud data is preprocessed, including coordinate conversion, noise filtering, and data downsampling. Then, the processed point cloud data is projected onto a two-dimensional (2D) plane, and the height value is used as the pixel value to generate bird's eye view (BEV) images. Finally, an improved Hough transform is proposed to detect the straight line representing the ship's nearshore profile in the BEV images and replace the ship midline with the detected line for calculating the berthing angle. The performance of the proposed method is verified by applying it on simulated and real ship data. Simulation experiment results show that the maximum absolute error of our method is only 0.151°. Real ship experiments were conducted at Dalian Port (China) to verify the reliability and stability of the method compared with other berthing angle estimation methods and the compass equipment. Overall, the proposed method can effectively obtain the berthing angle of ships, assist in parallel berthing, and improve berthing efficiency.

DPCN++: Differentiable Phase Correlation Network for Versatile Pose Registration.

Pose registration is critical in vision and robotics. This paper focuses on the challenging task of initialization-free pose registration up to 7DoF for homogeneous and heterogeneous measurements. While recent learning-based methods show promise using differentiable solvers, they either rely on heuristically defined correspondences or require initialization. Phase correlation seeks solutions in the spectral domain and is correspondence-free and initialization-free. Following this, we propose a differentiable solver and combine it with simple feature extraction networks, namely DPCN++. It can perform registration for homo/hetero inputs and generalizes well on unseen objects. Specifically, the feature extraction networks first learn dense feature grids from a pair of homogeneous/heterogeneous measurements. These feature grids are then transformed into a translation and scale invariant spectrum representation based on Fourier transform and spherical radial aggregation, decoupling translation and scale from rotation. Next, the rotation, scale, and translation are independently and efficiently estimated in the spectrum step-by-step. The entire pipeline is differentiable and trained end-to-end. We evaluate DCPN++ on a wide range of tasks taking different input modalities, including 2D bird's-eye view images, 3D object and scene measurements, and medical images. Experimental results demonstrate that DCPN++ outperforms both classical and learning-based baselines, especially on partially observed and heterogeneous measurements.

The Software Design Overview by Processing The Recording From Bird's-Eye View Images to Determine The Crop Detection and Functionality of The Various Land Types

To meet the demand for food, humanity has begun to use the latest technologies such as artificial intelligence, the internet of things, and drones, in addition to advanced tractors, seeds, and crop planting methods. Precision agriculture has been achieved by using the latest technology in this field, especially in recent years, the use of drones for agricultural land spraying has gained great interest. In this study, a Python programming language was used to process the video footage of a certain height taken from agricultural land with the help of a drone, using individual photo frames obtained from the footage. Each pixel was separated into different color contrast values, and certain colors were distinguished by counting. The proportional distribution of different types of surfaces on the land was determined. The software enabled the determination of various geophysical properties such as the productivity of crops, crop development status, and identification of areas where the crop does not grow.

Exploiting Multi-Modal Fusion for Urban Autonomous Driving Using Latent Deep Reinforcement Learning

Human driving decisions are the leading cause of road fatalities. Autonomous driving naturally eliminates such incompetent decisions and thus can improve traffic safety and efficiency. Deep reinforcement learning (DRL) has shown great potential in learning complex tasks. Recently, researchers investigated various DRL-based approaches for autonomous driving. However, exploiting multi-modal fusion to generate perception and motion prediction and then leveraging these predictions to train a latent DRL has not been targeted yet. To that end, we propose enhancing urban autonomous driving using multi-modal fusion with latent DRL. A single LIDAR sensor is used to extract bird's-eye view (BEV), range view (RV), and residual input images. These images are passed into LiCaNext, a real-time multi-modal fusion network, to produce accurate joint perception and motion prediction. Next, predictions are fed with another simple BEV image into the latent DRL to learn a complex end-to-end driving policy ensuring safety, efficiency, and comfort. A sequential latent model is deployed to learn more compact representations from inputs, leading to improved sampling efficiency for reinforcement learning. Our experiments are simulated on CARLA and evaluated against state-of-the-art DRL models. Results manifest that our method learns a better driving policy that outperforms other prevailing models. Further experiments are conducted to reveal the effectiveness of our proposed approach under different environments and varying weather conditions.

Multi‐feature subspace representation network for person re‐identification via bird's‐eye view image

AbstractPerson re‐identification (Re‐ID) is one of the most remarkable research topics that widely applied in our daily lives. For person Re‐ID in bird's eye scenes, traditional computer vision‐based methods used multiple features, for example, texture and color, of a pedestrian's head and shoulders. Those methods are difficult to cope with environments of variety and the change the appearance of different people due to the instability of feature detection. On the other hand, although recent advanced deep learning‐based methods are powerful to extract discriminative features, the requirement of a large amount of annotated training data restricts the appliable tasks. To overcome this problem, in this article, we propose a novel method fusing multiple heterogeneous features through a multi‐feature subspace representation network (MFSRN) to maximize the classification performance while keeping the disparity among features as small as possible, that is, common‐subspace constraints. We conducted comparative experiments with state‐of‐the‐art models on the bird's‐eye view person dataset, and extensive experimental results demonstrated that our proposed MFSRN could achieve better recognition performance. Furthermore, the validity and stability of the method are confirmed.

Recognition of field roads based on improved U-Net++ Network

Unmanned driving of agricultural machinery has garnered significant attention in recent years, especially with the development of precision farming and sensor technologies. To achieve high performance and low cost, perception tasks are of great importance. In this study, a low-cost and high-safety method was proposed for field road recognition in unmanned agricultural machinery. The approach of this study utilized point clouds, with low-resolution lidar point clouds as inputs, generating high-resolution point clouds and Bird's Eye View (BEV) images that were encoded with several basic statistics. Using a BEV representation, road detection was reduced to a single-scale problem that could be addressed with an improved U-Net++ neural network. Three enhancements were proposed for U-Net++: 1) replacing the convolutional kernel in the original U-Net++ with an Asymmetric Convolution Block (ACBlock); 2) adding a multi-branch Asymmetric Dilated Convolutional Block (MADC) in the highest semantic information layer; 3) adding an Attention Gate (AG) model to the long-skip-connection in the decoding stage. The results of experiments of this study showed that our algorithm achieved a Mean Intersection Over Union of 96.54% on the 16-channel point clouds, which was 7.35 percentage points higher than U-Net++. Furthermore, the average processing time of the model was about 70 ms, meeting the time requirements of unmanned driving in agricultural machinery. The proposed method of this study can be applied to enhance the perception ability of unmanned agricultural machinery thereby increasing the safety of field road driving. Keywords: image segmentation, unmanned agricultural machinery, field roads, point cloud super-resolution, point cloud bird's eye view DOI: 10.25165/j.ijabe.20231602.7941 Citation: Yang L L, Li Y B, Chang M S, Xu Y Y, Hu B B, Wang X X, et al. Recognition of field roads based on improved U-Net++ Network. Int J Agric & Biol Eng, 2023; 16(2): 171-178.

TRIMMING AND ROAD ORTHO IMAGING FOR NIGHT IMAGES BY ONBOARD HIGH SENSITIVITY CONSUMER GRADE DIGITAL CAMERAS

Abstract. Various kinds of cameras have been utilizing as the onboard cameras in the construction of Intelligent Transport Systems. In recent years, utilization of the high sensitivity consumer grade digital cameras at night is attracting attention from the viewpoint of avoiding the effects of sunlight and congestion of people and cars. However, due to the image taken by the onboard cameras is a perspective projection image, the image is projected small at the far from car and the effect of the lens distortion will be greater at points far from the image center. In order to avoid these issues, the lower part of the projection image or a bird's-eye view image is used, but the imaging of the bonnet part caused by the car models and tilts of the cameras becomes a new issue. Furthermore, a bird's-eye view image at night has to be trimmed to coincide with the irradiation range because the irradiation distance and range of the headlights are limited. On the other hand, feature quantities such as vanishing points and feature points on the lane have been used for projective transformation from a perspective projection image to a bird's-eye view image, but the projective transformation based on feature quantities is an ill-posed problem.Therefore, this paper discusses a quantitative trimming method based on projective transformation that does not depend on feature quantities and coincide with the irradiation range of the headlights.

Vehicle Trajectory Estimation based on Drive Recorder Data

In the present study, a method for estimating the self-position of a vehicle using data recorded by a drive recorder is investigated. In order to confirm the performance of the self-position estimation method, an experiment was conducted using a simulator. The self-position of a vehicle was estimated using image registration based on brightness between a bird's-eye view image made from a vehicle's front image and a 2D map image of the GNSS positioning location. As a result, if road boundary features were found in the bird's-eye view image, the proposed method is able to estimate the vehicle's self-position more accurately than GNSS, regardless of the existence of road markings. In addition, on straight roads with road boundary features in only one direction, the vehicle's self-position estimation was accurate enough to identify the lane in which the vehicle is currently located relative to the direction of the brightness gradient on a 2D map.

Quantitative Trimming for Ortho Imaging at Night by Onboard High Sensitivity Consumer Grade Digital Cameras

Various kinds of cameras have been utilizing as onboard cameras in the construction of Intelligent Transport Systems. In recently, utilization of the high sensitivity consumer grade digital cameras at night is attracting attention from the viewpoint of avoiding the effects of sunlight and congestion of people and cars. However, due to the image taken by the onboard cameras is a perspective projection image, the image is projected small at the far from car and the greater the influence of the lens distortion the farther from the image center. In order to avoid the issues, lower part of projection image or a bird's-eye view image is used, but the imaging of the bonnet part due to the vehicle model and the tilt of the camera becomes a new issue. Furthermore, a bird's-eye view image at night has to be trimmed to coincide with irradiation range since the irradiation distance and range for headlight are limited. On the other hand, feature quantities such as vanishing points and feature points on the lane have been used for projective transformation from a perspective projection image to a bird's-eye view image, but the projective transformation based on the feature quantity is an ill-posed problem.

포인트 클라우드 보간을 이용한 카메라-라이다의 센서 퓨전 기반 객체 검출

This paper aims to detect objects from BEV (Bird-Eye View) images by utilizing multi-source data from three-dimensional light detection and ranging (3D LiDAR) and a camera. Object detection via the camera is implemented by using a single-stage detector, YOLO-v3, and the distance of the detected objects is estimated by performing camera calibration with the 3D LiDAR. BEV images for sensor fusion are generated using LiDAR data. Sensor fusion estimates the distance of objects detected by the camera and enhances the features in BEV images using point interpolation based on point cloud data obtained by the 3D LiDAR. Object detection is performed by applying the proposed deep neural network on enhanced BEV images. Evaluation is performed using the KITTI validation dataset and a privately collected dataset, which includes images obtained on actual roads.

CrossFusion net: Deep 3D object detection based on RGB images and point clouds in autonomous driving

In recent years, accurate 3D detection plays an important role in a lot of applications. Autonomous driving, for instance, is one of typical representatives. This paper aims to design an accurate 3D detector that takes both Li-DAR point clouds and RGB images as inputs according to the fact that both LiDAR and camera have their own merits. A deep novel end-to-end two-stream learnable architecture, CrossFusion Net, is designed to exploit features from both LiDAR point clouds as well as RGB images through a hierarchical fusion structure. Specifically, CrossFusion Net utilizes bird's eye view (BEV) of point clouds through projection. Besides, these two feature maps of different streams are fused through the newly introduced CrossFusion(CF) layer. The proposed CF layer transforms feature maps of one stream to another based on the spatial relationship between the BEV and RGB images. Additionally, we apply attention mechanism on the transformed feature map and the original one to automatically decide the importance of the two feature maps from the two sensors. Experiments on the challenging KITTI car 3D detection benchmark and BEV detection benchmark show that the presented approach outperforms the other state-of-the-art methods in average precision(AP), specifically, as well as outperforms UberATG-ContFuse [3] of 8% AP in moderate 3D car detection. Furthermore, the proposed network learns an effective representation in perception of circumstances via RGB feature maps and BEV feature maps.

Vision-based Bed Detection for Hospital Patient Monitoring System.

In recent years, as a way to prevent patient fall-down, studies have been conducted using patient room cameras to detect the patient behavior of leaving the bed. It is very important to specify the patient bed location in the process of detecting patient behavior using camera images. In this study, we propose a method to specify the patient bed location using a monocular camera. In this proposal, we convert a camera image viewpoint into a bird's-eye view image as a preprocessing step. By using planer perspective transformation, it is possible to display the bed as a rectangular shape with a fixed ratio, even if the bed location or camera position is changed. Therefore, it is possible to detect the bed location with a high degree of accuracy by means of machine learning. The simulation experiment results confirm that the average error and standard deviation of the bed coordinates are 7.9 and 5.0 pixels, respectively; in the practical scene, we confirm that the average error and standard deviation of the bed coordinates are 12.1 and 8.2 pixels, respectively.

Support on Berthing Manoeuver by Using Image Processing

When berthing or un-berthing a large ship, a pilot is expected to improve the safety by reducing his stress by using both numerical information obtained at the bridge and the visual information at the wing. In our previous study, the usefulness of the support system by using the image processing technology was presented. However, that system did not consider the trim effect, and the bird's-eye view image was partial.

ロボット遠隔操作のためのLiDARを用いた全方位3次元測距による俯瞰映像上での障害物提示

Bird's-eye view system is one of image presentation systems for teleoperation. Bird's-eye view image helps an operator to visually recognize surrounding environments because of its high visibility. However, when there are obstacles that are higher than the floor or the ground, there is a problem that their positional appearance is incorrectly shown on bird's-eye view image because of the image distortion. This paper presents a new method to visualize correctly obstacles' positional appearance on the bird's-eye view image by omnidirectional 3D ranging. Specifically, the information of the distance to obstacles measured by LiDAR is superimposed onto the floor or the ground as points with colors obtained by fish-eye cameras. As shown in the experimental results, the image generated by this proposed method can improve the safety and visibility in teleoperation.

Generic Dynamic Environment Perception Using Smart Mobile Devices.

The driving environment is complex and dynamic, and the attention of the driver is continuously challenged, therefore computer based assistance achieved by processing image and sensor data may increase traffic safety. While active sensors and stereovision have the advantage of obtaining 3D data directly, monocular vision is easy to set up, and can benefit from the increasing computational power of smart mobile devices, and from the fact that almost all of them come with an embedded camera. Several driving assistance application are available for mobile devices, but they are mostly targeted for simple scenarios and a limited range of obstacle shapes and poses. This paper presents a technique for generic, shape independent real-time obstacle detection for mobile devices, based on a dynamic, free form 3D representation of the environment: the particle based occupancy grid. Images acquired in real time from the smart mobile device’s camera are processed by removing the perspective effect and segmenting the resulted bird-eye view image to identify candidate obstacle areas, which are then used to update the occupancy grid. The occupancy grid tracked cells are grouped into obstacles depicted as cuboids having position, size, orientation and speed. The easy to set up system is able to reliably detect most obstacles in urban traffic, and its measurement accuracy is comparable to a stereovision system.

Accurate Mobile Urban Mapping via Digital Map-Based SLAM.

This paper presents accurate urban map generation using digital map-based Simultaneous Localization and Mapping (SLAM). Throughout this work, our main objective is generating a 3D and lane map aiming for sub-meter accuracy. In conventional mapping approaches, achieving extremely high accuracy was performed by either (i) exploiting costly airborne sensors or (ii) surveying with a static mapping system in a stationary platform. Mobile scanning systems recently have gathered popularity but are mostly limited by the availability of the Global Positioning System (GPS). We focus on the fact that the availability of GPS and urban structures are both sporadic but complementary. By modeling both GPS and digital map data as measurements and integrating them with other sensor measurements, we leverage SLAM for an accurate mobile mapping system. Our proposed algorithm generates an efficient graph SLAM and achieves a framework running in real-time and targeting sub-meter accuracy with a mobile platform. Integrated with the SLAM framework, we implement a motion-adaptive model for the Inverse Perspective Mapping (IPM). Using motion estimation derived from SLAM, the experimental results show that the proposed approaches provide stable bird’s-eye view images, even with significant motion during the drive. Our real-time map generation framework is validated via a long-distance urban test and evaluated at randomly sampled points using Real-Time Kinematic (RTK)-GPS.

A Low-Cost Solution for Automatic Lane-Level Map Generation Using Conventional In-Car Sensors

Lane-level digital maps are crucial to advanced driver assistance systems (ADAS) and autonomous driving, since they can simplify driving tasks and enhance system performance by providing strong priors about the driving environment. However, the high cost of current map generation systems prevents their benefits to normal commercial cars as they usually depend on specialized sensors and need great manual postprocessing. In this paper, a low-cost solution is proposed for automatic generation of a precise lane-level map by using conventional sensors that have been already installed in contemporary cars. It mainly consists of two modules, i.e., road orthographic image generation and lane graph construction. First, the global map is divided into fixed local segments based on the road network topology. With the reference of the local map segments, the bird's eye view images of the road surface are accumulated by fusing GPS, INS, and visual odometry and subsequently integrated into synthetic orthographic images. Next, the driving lane information is extracted from the road orthographic images and a large amount of vehicle trajectories. Such information is then used to construct a lane graph of the map based on the sophisticated lane models we proposed without manual processing. Experiments show promising results of the automatic map generation of the real-world roads, which substantiated the effectiveness of the proposed approach. Such a system can offer increased value and promote the automation level for today's commercial cars without being supplemented additional sensors.