Multi-modal Object Detection Research Articles

Efficient multimodal object detection via coordinate attention fusion for adverse environmental conditions

Integrating complementary visual information from multimodal image pairs can significantly improve the robustness and accuracy of object detection algorithms, particularly in challenging environments. However, a key challenge lies in the effective fusion of modality-specific features within these algorithms. To address this, we propose a novel lightweight fusion module, termed the Coordinate Attention Fusion (CAF) module, built on the YOLOv5 object detection framework. The CAF module exploits differential amplification and coordinated attention mechanisms to selectively enhance distinctive cross-modal features, thereby preserving critical modality-specific information. To further optimize performance and reduce computational overhead, the two-stream backbone network has been refined, reducing the model's parameter count without compromising accuracy. Comprehensive experiments conducted on two benchmark multimodal datasets demonstrate that the proposed approach consistently surpasses conventional methods and outperforms existing state-of-the-art multimodal object detection algorithms. These findings underscore the potential of cross-modality fusion as a promising direction for improving object detection in adverse conditions.

Informative Data Selection With Uncertainty for Multimodal Object Detection.

Noise has always been nonnegligible trouble in object detection by creating confusion in model reasoning, thereby reducing the informativeness of the data. It can lead to inaccurate recognition due to the shift in the observed pattern, that requires a robust generalization of the models. To implement a general vision model, we need to develop deep learning models that can adaptively select valid information from multimodal data. This is mainly based on two reasons. Multimodal learning can break through the inherent defects of single-modal data, and adaptive information selection can reduce chaos in multimodal data. To tackle this problem, we propose a universal uncertainty-aware multimodal fusion model. It adopts a multipipeline loosely coupled architecture to combine the features and results from point clouds and images. To quantify the correlation in multimodal information, we model the uncertainty, as the inverse of data information, in different modalities and embed it in the bounding box generation. In this way, our model reduces the randomness in fusion and generates reliable output. Moreover, we conducted a completed investigation on the KITTI 2-D object detection dataset and its derived dirty data. Our fusion model is proven to resist severe noise interference like Gaussian, motion blur, and frost, with only slight degradation. The experiment results demonstrate the benefits of our adaptive fusion. Our analysis on the robustness of multimodal fusion will provide further insights for future research.

Snow-CLOCs: Camera-LiDAR Object Candidate Fusion for 3D Object Detection in Snowy Conditions.

Although existing 3D object-detection methods have achieved promising results on conventional datasets, it is still challenging to detect objects in data collected under adverse weather conditions. Data distortion from LiDAR and cameras in such conditions leads to poor performance of traditional single-sensor detection methods. Multi-modal data-fusion methods struggle with data distortion and low alignment accuracy, making accurate target detection difficult. To address this, we propose a multi-modal object-detection algorithm, Snow-CLOCs, specifically for snowy conditions. In image detection, we improved the YOLOv5 algorithm by integrating the InceptionNeXt network to enhance feature extraction and using the Wise-IoU algorithm to reduce dependency on high-quality data. For LiDAR point-cloud detection, we built upon the SECOND algorithm and employed the DROR filter to remove noise, enhancing detection accuracy. We combined the detection results from the camera and LiDAR into a unified detection set, represented using a sparse tensor, and extracted features through a 2D convolutional neural network to achieve object detection and localization. Snow-CLOCs achieved a detection accuracy of 86.61% for vehicle detection in snowy conditions.

Learnable fusion mechanisms for multimodal object detection in autonomous vehicles

AbstractPerception systems in autonomous vehicles need to accurately detect and classify objects within their surrounding environments. Numerous types of sensors are deployed on these vehicles, and the combination of such multimodal data streams can significantly boost performance. The authors introduce a novel sensor fusion framework using deep convolutional neural networks. The framework employs both camera and LiDAR sensors in a multimodal, multiview configuration. The authors leverage both data types by introducing two new innovative fusion mechanisms: element‐wise multiplication and multimodal factorised bilinear pooling. The methods improve the bird's eye view moderate average precision score by +4.97% and +8.35% on the KITTI dataset when compared to traditional fusion operators like element‐wise addition and feature map concatenation. An in‐depth analysis of key design choices impacting performance, such as data augmentation, multi‐task learning, and convolutional architecture design is offered. The study aims to pave the way for the development of more robust multimodal machine vision systems. The authors conclude the paper with qualitative results, discussing both successful and problematic cases, along with potential ways to mitigate the latter.

CrossFormer: Cross-guided attention for multi-modal object detection

Object detection is one of the essential tasks in a variety of real-world applications such as autonomous driving and robotics. In a real-world scenario, unfortunately, there are numerous challenges such as illumination changes, adverse weather conditions, and geographical changes, to name a few. To tackle the problem, we propose a novel multi-modal object detection model that is built upon a hierarchical transformer and cross-guidance between different modalities. The proposed hierarchical transformer consists of domain-specific feature extraction networks where intermediate features are connected by the proposed Cross-Guided Attention Module (CGAM) to enrich their representational power. Specifically, in the CGAM, one domain is regarded as a guide and the other is assigned to a base. After that, the cross-modal attention from the guide to the base is applied to the base feature. The CGAM works bidirectionally in parallel by exchanging roles between modalities to refine multi-modal features simultaneously. Experimental results on FLIR-aligned, LLVIP, and KAIST multispectral pedestrian datasets demonstrate that the proposed method is superior to previous multi-modal detection algorithms quantitatively and qualitatively.

SeaDATE: Remedy Dual-Attention Transformer with Semantic Alignment via Contrast Learning for Multimodal Object Detection

SeaDATE: Remedy Dual-Attention Transformer with Semantic Alignment via Contrast Learning for Multimodal Object Detection

Multimodal Object Detection of UAV Remote Sensing Based on Joint Representation Optimization and Specific Information Enhancement

Multimodal Object Detection of UAV Remote Sensing Based on Joint Representation Optimization and Specific Information Enhancement

Weakly Aligned Feature Fusion for Multimodal Object Detection.

To achieve accurate and robust object detection in the real-world scenario, various forms of images are incorporated, such as color, thermal, and depth. However, multimodal data often suffer from the position shift problem, i.e., the image pair is not strictly aligned, making one object has different positions in different modalities. For the deep learning method, this problem makes it difficult to fuse multimodal features and puzzles the convolutional neural network (CNN) training. In this article, we propose a general multimodal detector named aligned region CNN (AR-CNN) to tackle the position shift problem. First, a region feature (RF) alignment module with adjacent similarity constraint is designed to consistently predict the position shift between two modalities and adaptively align the cross-modal RFs. Second, we propose a novel region of interest (RoI) jitter strategy to improve the robustness to unexpected shift patterns. Third, we present a new multimodal feature fusion method that selects the more reliable feature and suppresses the less useful one via feature reweighting. In addition, by locating bounding boxes in both modalities and building their relationships, we provide novel multimodal labeling named KAIST-Paired. Extensive experiments on 2-D and 3-D object detection, RGB-T, and RGB-D datasets demonstrate the effectiveness and robustness of our method.

An improved YOLOv7 network using RGB-D multi-modal feature fusion for tea shoots detection

Due to the increasing scarcity of tea pickers, the implementation of intelligent harvesting for premium tea is a crucial prerequisite for the sustainable development of the premium tea industry. The initial step towards achieving intelligent and precise harvesting is the accurate detection of tender shoots, which consist of one bud and one leaf. However, accurately identifying tea shoots poses a challenging visual task due to their small size, variable shapes, as well as similar colors and backgrounds. The existing model, based on RGB images, can only detect partial targets. To address this issue and further enhance the detection of tea buds, this study proposes the utilization of multi-modal features encompassing red, green, blue, and depth (RGB-D) for identification. In addition, a unidirectional complementary multi-modal fusion method is introduced to minimize the adverse effects caused by low-quality depth information. Firstly, an RGB-D dataset comprising high-quality tea leaves is constructed, and the samples are carefully calibrated. Subsequently, an enhanced end-to-end RGB-D multi-modal object detection network, referred to as YOLO-RGBDtea, is developed based on You Only Look Once version 7 (YOLOv7). This model incorporates a parallel lightweight depth image feature extraction backbone network and incorporates a self-attention mechanism to prioritize contextual information. Lastly, a cross-modal spatial attention fusion module (CSFM) is devised to collaboratively integrate depth features with RGB features in a unidirectional manner. The experimental results reveal that YOLO-RGBDtea achieves an AP50 of 91.12% when confronting complex outdoor tea shoots, exhibiting significant performance improvements compared to YOLOv7, especially in scenarios involving small targets, overlapping target groups, and highly overexposed images. Notably, the parameter increment in YOLO-RGBDtea compared to the original YOLOv7 model is merely 17.8%, and the additional components can be seamlessly transferred to other models. Overall, this study introduces a straightforward yet effective multi-modal fusion method that bears theoretical and practical significance in advancing the detection of high-quality tea shoots in complex outdoor environments.

MLFFusion: Multi-level feature fusion network with region illumination retention for infrared and visible image fusion

Infrared image and visible light image fusion is widely used in night vision, surveillance, military and other fields. The focus of the fusion task is to integrate complementary information in visible and infrared light images and eliminate redundant information. In addition, most of the fusion tasks are performed in the harsh environment of low light, and it is worth studying how to maintain the lighting information of the fusion results. In order to solve the problems existing, firstly, we designs a multi-level feature module to fusion multi-source information. Different from the parallel layer fusion strategy of the traditional network, we proposed a fusion strategy combine parallel layers and depth layers. Secondly, we add attention computing to the feature extraction network to improve the performance of the feature extraction network. Thirdly, in order to make the fusion image have good illumination information, we design the area illumination retention module, improved the performance of the fusion algorithm in low-light environments. A large number of experiments show that the proposed method has excellent performance and will perform better in low light environment. In addition, the proposed algorithm also shows great potential in multi modal object detection.

Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection

RGB-Infrared multi-modal object detection utilizes diverse and complementary information, showing some advantages in intelligent transportation field. The main challenge of RGB-Infrared object detection is how to fuse the two modalities. The difficulty of fusion is reflected in two aspects: 1) large visual differences between modalities make it difficult to learn effective complementary features, 2) some misaligned RGB-Infrared images increase the difficulty of fusion. To this end, based on feature pyramid commonly used in object detection, we propose Multi-modal Feature Pyramid Transformer (MFPT) to fuse the two modalities. The proposed MFPT learns semantic and modal complementary information to enhance each modal features via intra-modal feature pyramid transformer and inter-modal feature pyramid transformer. The intra-modal feature pyramid transformer enables features to interact across space and scales, improving the semantic representations of features in each modality. The inter-modal feature pyramid transformer conducts feature interaction between modalities, enabling each modality to learn complementary features from other modalities. Meanwhile, the inter-modal feature pyramid transformer can also learn distance independent dependencies between modalities, which are not sensitive to misaligned images. Furthermore, a local attention mechanism is introduced within different windows into MFPT to achieve efficient correlation between regions of different scales or different modalities. Experimental results on two RGB-Infrared detection datasets demonstrate the proposed method is superior to state-of-the-art methods.

Lightweight Transformer for Multi-Modal Object Detection (Student Abstract)

It has become a common practice for many perceptual systems to integrate information from multiple sensors to improve the accuracy of object detection. For example, autonomous vehicles use visible light, and infrared (IR) information to ensure that the car can cope with complex weather conditions. However, the accuracy of the algorithm is usually a trade-off between the computational complexity and memory consumption. In this study, we evaluate the performance and complexity of different fusion operators in multi-modal object detection tasks. On top of that, a Poolformer-based fusion operator (PoolFuser) is proposed to enhance the accuracy of detecting targets without compromising the efficiency of the detection framework.

Low saliency crack detection based on improved multimodal object detection network: an example of wind turbine blade inner surface

Low saliency crack detection based on improved multimodal object detection network: an example of wind turbine blade inner surface

The registration of visible and thermal images through multi-objective optimization

Multimodal imaging with visible and thermal sensors attracts much attention due to its robustness under challenging illumination conditions. Due to spectral differences, the visible and thermal images are typically misaligned, where image registration is necessary before high-level vision tasks such as information fusion, multimodal object detection, etc. Contemporary registration methods measure either an angular distance or a linear distance as an objective function to solve affine parameters between visible and thermal images. It is noticeable that both distances have unique advantages in image alignment, which suggests that integrating these measurements may enhance the registration quality. Thus, this paper proposes a new algorithm, namely multi-objective optimization-based image registration (MOIR), to align visible and thermal images. The MOIR comprises a normalized gradient measurement (NGM) that quantifies and normalizes differences of image gradient in both angular and linear aspects, and a regularized stochastic gradient descent (RSGD) that smoothly solves affine parameters. Both quantitative and qualitative results validate the effectiveness and robustness of MOIR for the registration of visible and thermal images in both controlled and real-world scenarios. Some animated examples can be found in https://github.com/jb2849/MOIR.

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of “what to fuse”, “when to fuse”, and “how to fuse” remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.

Air-to-ground multimodal object detection algorithm based on feature association learning

Detecting objects on unmanned aerial vehicles is a hard task, due to the long visual distance and the subsequent small size and lack of view. Besides, the traditional ground observation manners based on visible light camera are sensitive to brightness. This article aims to improve the target detection accuracy in various weather conditions, by using both visible light camera and infrared camera simultaneously. In this article, an association network of multimodal feature maps on the same scene is used to design an object detection algorithm, which is the so-called feature association learning method. In addition, this article collects a new cross-modal detection data set and proposes a cross-modal object detection algorithm based on visible light and infrared observations. The experimental results show that the algorithm improves the detection accuracy of small objects in the air-to-ground view. The multimodal joint detection network can overcome the influence of illumination in different weather conditions, which provides a new detection means and ideas for the space-based unmanned platform to the small object detection task.

Multi-modal object detection and localization for high integrity driving assistance

Much work is currently devoted to increasing the reliability, completeness and precision of the data used by driving assistance systems, particularly in urban environments. Urban environments represent a particular challenge for the task of perception, since they are complex, dynamic and completely variable. This article examines a multi-modal perception approach for enhancing vehicle localization and the tracking of dynamic objects in a world-centric map. 3D ego-localization is achieved by merging stereo vision perception data and proprioceptive information from vehicle sensors. Mobile objects are detected using a multi-layer lidar that is simultaneously used to identify a zone of interest to reduce the complexity of the perception process. Object localization and tracking is then performed in a fixed frame which simplifies analysis and understanding of the scene. Finally, tracked objects are confirmed by vision using 3D dense reconstruction in focused regions of interest. Only confirmed objects can generate an alarm or an action on the vehicle. This is crucial to reduce false alarms that affect the trust that the driver places in the driving assistance system. Synchronization issues between the sensing modalities are solved using predictive filtering. Real experimental results are reported so that the performance of the multi-modal system may be evaluated.