Monocular 3D Object Detection Research Articles

Depth-enhancement network for monocular 3D object detection

Abstract To accurately obtain 3D information, the correct use of depth data is crucial. Compared with radar-based methods, detecting objects in 3D space in a single image is extremely challenging due to the lack of depth cues. However, monocular 3D object detection provides a more economical solution. Traditional monocular 3D object detection methods often rely on geometric constraints, such as key points, object shape relationships and 3D to 2D optimization, to address the inherent lack of depth information. However, these methods still make it challenging to extract rich information directly from depth estimation for fusion. To fundamentally enhance the ability of monocular 3D object detection, we propose a monocular 3D object detection network based on depth information enhancement. The network learns object detection and depth estimation tasks simultaneously through a unified framework, integrates depth features as auxiliary information into the detection branch, and then constrains and enhances them to obtain better spatial representation. To this end, we introduce a new cross-modal fusion strategy, which realizes a more reasonable fusion of cross-modal information by exploring redundant, complementary information and their interactions between RGB features and depth features. Extensive experiments on the KITTI dataset show that our method can significantly improve the performance of monocular 3D object detection.

Monocular 3D object detection for distant objects

Monocular 3D object detection for distant objects

Geometry-Guided Domain Generalization for Monocular 3D Object Detection

Monocular 3D object detection (M3OD) is important for autonomous driving. However, existing deep learning-based methods easily suffer from performance degradation in real-world scenarios due to the substantial domain gap between training and testing. M3OD's domain gaps are complex, including camera intrinsic parameters, extrinsic parameters, image appearance, etc. Existing works primarily focus on the domain gaps of camera intrinsic parameters, ignoring other key factors. Moreover, at the feature level, conventional domain invariant learning methods generally cause the negative transfer issue, due to the ignorance of dependency between geometry tasks and domains. To tackle these issues, in this paper, we propose MonoGDG, a geometry-guided domain generalization framework for M3OD, which effectively addresses the domain gap at both camera and feature levels. Specifically, MonoGDG consists of two major components. One is geometry-based image reprojection, which mitigates the impact of camera discrepancy by unifying intrinsic parameters, randomizing camera orientations, and unifying the field of view range. The other is geometry-dependent feature disentanglement, which overcomes the negative transfer problems by incorporating domain-shared and domain-specific features. Additionally, we leverage a depth-disentangled domain discriminator and a domain-aware geometry regression attention mechanism to account for the geometry-domain dependency. Extensive experiments on multiple autonomous driving benchmarks demonstrate that our method achieves state-of-the-art performance in domain generalization for M3OD.

FD3D: Exploiting Foreground Depth Map for Feature-Supervised Monocular 3D Object Detection

Monocular 3D object detection usually adopts direct or hierarchical label supervision. Recently, the distillation supervision transfers the spatial knowledge from LiDAR- or stereo-based teacher networks to monocular detectors, but remaining the domain gap. To mitigate this issue and pursue adequate label manipulation, we exploit Foreground Depth map for feature-supervised monocular 3D object detection named FD3D, which develops the high-quality instructive intermediate features to conduct desirable auxiliary feature supervision with only the original image and annotation foreground object-wise depth map (AFOD) as input. Furthermore, we build up our instructive feature generation network to create instructive spatial features based on the sufficient correlation between image features and pre-processed AFOD, where AFOD provides the attention focus only on foreground objects to achieve clearer guidance in the detection task. Moreover, we apply the auxiliary feature supervision from the pixel and distribution level to achieve comprehensive spatial knowledge guidance. Extensive experiments demonstrate that our method achieves state-of-the-art performance on both the KITTI and nuScenes datasets, with no external data and no extra inference computational cost. We also conduct quantitative and qualitative studies to reveal the effectiveness of our designs.

Depth-Guided Vision Transformer With Normalizing Flows for Monocular 3D Object Detection

Depth-Guided Vision Transformer With Normalizing Flows for Monocular 3D Object Detection

Monocular 3D object detection with thermodynamic loss and decoupled instance depth

Monocular 3D detection is to obtain the 3D information of the object from the image. The mainstream methods mainly use L1 loss or L1-like loss to control the instance depth prediction. However, these methods have not achieved satisfactory results. One of the main reasons is that L1 loss or L1-like loss does not accurately reflect the fit between the predicted instance depth and the corresponding ground truth. Another of the main reason is that the instance depth on the RGB image hard to be directly learned in the network. In order to solve the above problems, a novel thermodynamic loss based on the principle of free energy minimisation and a novel depth decoupling method are proposed in this paper. The proposed method is called the monocular 3D object detection network with thermodynamic loss and decoupled instance depth (TDN). In TDN, the optimisation of the instance depth prediction is regarded as the thermodynamic process. Therefore, the thermodynamic loss is designed according to the principle of free energy minimisation. TDN decouples the instance depth into three different depths. By combining the thermodynamic loss and the different types of depths, we can obtain the final instance depth.

GRAMO: geometric resampling augmentation for monocular 3D object detection

Data augmentation is widely recognized as an effective means of bolstering model robustness. However, when applied to monocular 3D object detection, non-geometric image augmentation neglects the critical link between the image and physical space, resulting in the semantic collapse of the extended scene. To address this issue, we propose two geometric-level data augmentation operators named Geometric-Copy-Paste (Geo-CP) and Geometric-Crop-Shrink (Geo-CS). Both operators introduce geometric consistency based on the principle of perspective projection, complementing the options available for data augmentation in monocular 3D. Specifically, Geo-CP replicates local patches by reordering object depths to mitigate perspective occlusion conflicts, and Geo-CS re-crops local patches for simultaneous scaling of distance and scale to unify appearance and annotation. These operations ameliorate the problem of class imbalance in the monocular paradigm by increasing the quantity and distribution of geometrically consistent samples. Experiments demonstrate that our geometric-level augmentation operators effectively improve robustness and performance in the KITTI and Waymo monocular 3D detection benchmarks.

Asymmetric Convolution: An Efficient and Generalized Method to Fuse Feature Maps in Multiple Vision Tasks.

Fusing features from different sources is a critical aspect of many computer vision tasks. Existing approaches can be roughly categorized as parameter-free or learnable operations. However, parameter-free modules are limited in their ability to benefit from offline learning, leading to poor performance in some challenging situations. Learnable fusing methods are often space-consuming and timeconsuming, particularly when fusing features with different shapes. To address these shortcomings, we conducted an in-depth analysis of the limitations associated with both fusion methods. Based on our findings, we propose a generalized module named Asymmetric Convolution Module (ACM). This module can learn to encode effective priors during offline training and efficiently fuse feature maps with different shapes in specific tasks. Specifically, we propose a mathematically equivalent method for replacing costly convolutions on concatenated features. This method can be widely applied to fuse feature maps across different shapes. Furthermore, distinguished from parameter-free operations that can only fuse two features of the same type, our ACM is general, flexible, and can fuse multiple features of different types. To demonstrate the generality and efficiency of ACM, we integrate it into several state-of-the-art models on three representative vision tasks: visual object tracking, referring video object segmentation, and monocular 3D object detection. Extensive experimental results on three tasks and several datasets demonstrate that our new module can bring significant improvements and noteworthy efficiency.

MonoAux: Fully Exploiting Auxiliary Information and Uncertainty for Monocular 3D Object Detection.

Monocular 3D object detection plays a pivotal role in autonomous driving, presenting a formidable challenge by requiring the precise localization of 3D objects within a single image, devoid of depth information. Most existing methods in this domain fall short of harnessing the limited information available in monocular 3D detection tasks. They typically provide only a single detection outcome, omitting essential uncertainty analysis and result post-processing during model inference, thus limiting overall model performance. In this paper, we propose a comprehensive framework that maximizes information extraction from monocular images while encompassing diverse depth estimation and incorporating uncertainty analysis. Specifically, we mine additional information intrinsic to the monocular 3D detection task to augment supervision, thereby addressing the information scarcity challenge. Moreover, our framework handles depth estimation by recovering multiple sets of depth values from calculated visual heights. The final depth estimate and 3D confidence are determined through an uncertainty fusion process, effectively reducing inference errors. Furthermore, to address task weight allocation in multi-task training, we present a versatile training strategy tailored to monocular 3D detection. This approach leverages measurement indicators to monitor task progress, adaptively adjusting loss weights for different tasks. Experimental results on the KITTI and Waymo dataset confirm the effectiveness of our approach. The proposed method consistently provides enhanced performance across various difficulty levels compared to the original framework while maintaining real-time efficiency.

Selective Transfer Learning of Cross-Modality Distillation for Monocular 3D Object Detection

Selective Transfer Learning of Cross-Modality Distillation for Monocular 3D Object Detection

Monocular 3D object detection for construction scene analysis

AbstractThree‐dimensional (3D) object detection, that is, localizing and classifying all critical objects in a 3D space, is essential for downstream construction scene analysis tasks. However, accurate instance segmentation, few 2D object segmentation and 3D object detection data sets, high‐quality feature representations for depth estimation, and limited 3D cues from a single red‐green‐blue (RGB) image pose significant challenges to 3D object detection and severely hinder its practical applications. In response to these challenges, an improved cascade‐based network with a transformer backbone and a boundary‐patch‐refinement method is proposed to build hierarchical features and refine object boundaries, resulting in better results in 2D object detection and instance segmentation. Furthermore, a novel self‐supervised monocular depth learning method is proposed to extract better feature representations for depth estimation from construction site video data with unknown camera parameters. Additionally, a pseudo‐LiDAR point cloud method and a 3D object detection method with a density‐based clustering algorithm are proposed to detect 3D objects in a construction scene without help from 3D labels, which will serve as a good foundation for other downstream 3D tasks. Finally, the proposed model is evaluated for object instance segmentation and depth estimation on the moving objects in construction sites (MOCS) and construction scene data sets. It brings a 9.16% gain in terms of mean average precision (mAP) for object detection and a 4.92% gain in mask mAP for object instance segmentation. The average order accuracy and relative mean error for depth estimation are improved by 0.94% and 60.56%, respectively. This study aims to overcome the challenges and limitations of 3D object detection and facilitate practical applications in construction scene analysis.

Mix-Teaching: A Simple, Unified and Effective Semi-Supervised Learning Framework for Monocular 3D Object Detection

Semi-supervised learning (SSL) has promising potential for improving model performance using both labelled and unlabelled data. Since recovering 3D information from 2D images is an ill-posed problem, the current state-of-the-art methods of monocular 3D object detection (Mono3D) have relatively low precision and recall, making semi-supervised learning for Mono3D tasks challenging and understudied. In this work, we propose a unified and effective semi-supervised learning framework called Mix-Teaching that can be applied to most monocular 3D object detectors. Based on the idea of decomposition and recombination, unlabelled samples are firstly decomposed into collections of image patches with high-quality predictions and collections of background images containing no objects. The student model is then trained on the mixed images containing dense instances with high-quality pseudo-labels generated by the recombination operation. In addition, we propose an uncertainty-based filter to distinguish high-quality pseudo-labels from noisy predictions during the decomposition process. As results in KITTI and nuScenes benchmarks, Mix-Teaching consistently improves MonoFlex and GUPNet by significant margins under various labeling ratios. Our method achieves around +6.34% <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AP</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3<i>D</i></sub> improvement against the GUPNet on the validation set when using only 10% labelled data. Using the full training set and the additional 38K raw images from KITTI, it can further improve the MonoFlex by +4.65% absolute improvement on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AP</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3<i>D</i></sub> for car detection, reaching 18.54% <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AP</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3<i>D</i></sub> , which ranks the 1st place among all monocular based methods on the KITTI test leaderboard.

GPro3D: Deriving 3D BBox from ground plane in monocular 3D object detection

Considering the inherent ill-posed nature, monocular 3D object detection (M3OD) is extremely challenging. The ground plane prior is a highly informative geometry clue in M3OD. However, it has been neglected by most mainstream methods. This paper introduces an original M3OD framework that leverages the ground plane to directly derive the object’s 3D Bounding Box (BBox) and 3D attributes geometrically. We identify and tackle three key factors that limit the applicability of the ground plane: the projection point localization issue, the ground plane tilt issue, and the lack of ground plane annotation issue. For the projection point localization issue, we propose leveraging the car’s explicit and salient wheel pixels, which are easier for the neural network to detect compared to the bottom vertices or the bottom center of the 3D BBox. To tackle the ground plane tilt problem, we propose a vertical-edge-enhanced horizon line detection algorithm to precisely deduce the ground plane equation. Moreover, using only M3OD labels, wheel pixel and horizon line pseudo-labels can be easily generated to train the network without extra data or annotation cost. Extensive experiments demonstrate the effectiveness and superiority of our framework over previous methods.

Pseudo-Mono for Monocular 3D Object Detection in Autonomous Driving

Current monocular 3D object detection algorithms generally suffer from inaccurate depth estimation, which leads to reduction of detection accuracy. The depth error from image-to-image generation for the stereo view is insignificant compared with the gap in single-image generation. Therefore, a novel pseudo-monocular 3D object detection framework is proposed, which is called Pseudo-Mono. Particularly, stereo images are brought into monocular 3D detection. Firstly, stereo images are taken as input, then a lightweight depth predictor is used to generate the depth map of input images. Secondly, the left input images obtained from stereo camera are used as subjects, which generate enhanced visual feature and multi-scale depth feature by depth indexing and feature matching probabilities, respectively. Finally, sparse anchors set by the foreground probability maps and the multi-scale feature maps are used as reference points to find the suitable initialization approach of object query. The encoded visual feature is adopted to enhance object query for enabling deep interaction between visual feature and depth feature. Compared with popular monocular 3D object detection methods, Pseudo-Mono is able to achieve richer fine-grained information without additional data input. Extensive experimental results on the datasets of KITTI, NuScenes, and MS-COCO demonstrate the generalizability and portability of the proposed method. The effectiveness and efficiency of Pseudo-Mono have been demonstrated by extensive ablation experiments. Experiments on a real vehicle platform have shown that the proposed method maintains high performance in complex real-world environments.

Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection

Monocular 3D object detection is a low-cost but challenging task, as it requires generating accurate 3D localization solely from a single image input. Recent developed depth-assisted methods show promising results by using explicit depth maps as intermediate features, which are either precomputed by monocular depth estimation networks or jointly evaluated with 3D object detection. However, inevitable errors from estimated depth priors may lead to misaligned semantic information and 3D localization, hence resulting in feature smearing and suboptimal predictions. To mitigate this issue, we propose ADD, an Attention-based Depth knowledge Distillation framework with 3D-aware positional encoding. Unlike previous knowledge distillation frameworks that adopt stereo- or LiDAR-based teachers, we build up our teacher with identical architecture as the student but with extra ground-truth depth as input. Credit to our teacher design, our framework is seamless, domain-gap free, easily implementable, and is compatible with object-wise ground-truth depth. Specifically, we leverage intermediate features and responses for knowledge distillation. Considering long-range 3D dependencies, we propose 3D-aware self-attention and target-aware cross-attention modules for student adaptation. Extensive experiments are performed to verify the effectiveness of our framework on the challenging KITTI 3D object detection benchmark. We implement our framework on three representative monocular detectors, and we achieve state-of-the-art performance with no additional inference computational cost relative to baseline models. Our code is available at https://github.com/rockywind/ADD.

A Simple Baseline for Multi-Camera 3D Object Detection

3D object detection with surrounding cameras has been a promising direction for autonomous driving. In this paper, we present SimMOD, a Simple baseline for Multi-camera Object Detection, to solve the problem. To incorporate multiview information as well as build upon previous efforts on monocular 3D object detection, the framework is built on sample-wise object proposals and designed to work in a twostage manner. First, we extract multi-scale features and generate the perspective object proposals on each monocular image. Second, the multi-view proposals are aggregated and then iteratively refined with multi-view and multi-scale visual features in the DETR3D-style. The refined proposals are endto-end decoded into the detection results. To further boost the performance, we incorporate the auxiliary branches alongside the proposal generation to enhance the feature learning. Also, we design the methods of target filtering and teacher forcing to promote the consistency of two-stage training. We conduct extensive experiments on the 3D object detection benchmark of nuScenes to demonstrate the effectiveness of SimMOD and achieve competitive performance. Code will be available at https://github.com/zhangyp15/SimMOD.

RADIANT: Radar-Image Association Network for 3D Object Detection

As a direct depth sensor, radar holds promise as a tool to improve monocular 3D object detection, which suffers from depth errors, due in part to the depth-scale ambiguity. On the other hand, leveraging radar depths is hampered by difficulties in precisely associating radar returns with 3D estimates from monocular methods, effectively erasing its benefits. This paper proposes a fusion network that addresses this radar-camera association challenge. We train our network to predict the 3D offsets between radar returns and object centers, enabling radar depths to enhance the accuracy of 3D monocular detection. By using parallel radar and camera backbones, our network fuses information at both the feature level and detection level, while at the same time leveraging a state-of-the-art monocular detection technique without retraining it. Experimental results show significant improvement in mean average precision and translation error on the nuScenes dataset over monocular counterparts. Our source code is available at https://github.com/longyunf/radiant.

Leveraging front and side cues for occlusion handling in monocular 3D object detection

Leveraging front and side cues for occlusion handling in monocular 3D object detection

ABC: Aligning binary centers for single-stage monocular 3D object detection

Precisely perceiving the environment through a 3D perspective is essential and challenging for autonomous navigation and driving. Existing techniques rely on depth from LiDAR data or disparity in stereo vision to solve the poorly presented problem of detecting far-off and occluded objects. This increases structure complexity and computation burden, especially for single-stage systems. We argue that existing well-established detectors have the intrinsic potential to detect full-scene objects, but the extrinsic capabilities are limited by the structure form and optimization. Hence, we propose a double-branch single-stage monocular 3D object detection framework that aligns binary centers of object. Structurally, we construct two symmetrical and independent detectors, respectively using different prediction manners for 3D box parameters. Functionally, two detection heads have different sensitivities for the same object due to disentangling alignment. During the training, the detection heads were trained separately to obtain specific ability and aligned to promote the convergence. At inference, predictions of two branches are filtered via depth-aware non-maximal suppression (NMS) to acquire comprehensive detection results. Extensive experiments demonstrate that the proposed method achieves the state-of-the-art performance in monocular 3D detection on the KITTI-3D benchmark.

Uncertainty Prediction for Monocular 3D Object Detection

For object detection, capturing the scale of uncertainty is as important as accurate localization. Without understanding uncertainties, self-driving vehicles cannot plan a safe path. Many studies have focused on improving object detection, but relatively little attention has been paid to uncertainty estimation. We present an uncertainty model to predict the standard deviation of bounding box parameters for a monocular 3D object detection model. The uncertainty model is a small, multi-layer perceptron (MLP) that is trained to predict uncertainty for each detected object. In addition, we observe that occlusion information helps predict uncertainty accurately. A new monocular detection model is designed to classify occlusion levels as well as to detect objects. An input vector to the uncertainty model contains bounding box parameters, class probabilities, and occlusion probabilities. To validate predicted uncertainties, actual uncertainties are estimated at the specific predicted uncertainties. The accuracy of the predicted values is evaluated using these estimated actual values. We find that the mean uncertainty error is reduced by 7.1% using the occlusion information. The uncertainty model directly estimates total uncertainty at the absolute scale, which is critical to self-driving systems. Our approach is validated through the KITTI object detection benchmark.