Object Detector Research Articles

Using Data Augmentation to Improve the Generalization Capability of an Object Detector on Remote-Sensed Insect Trap Images.

Traditionally, monitoring insect populations involved the use of externally placed sticky paper traps, which were periodically inspected by a human operator. To automate this process, a specialized sensing device and an accurate model for detecting and counting insect pests are essential. Despite considerable progress in insect pest detector models, their practical application is hindered by the shortage of insect trap images. To attenuate the "lack of data" issue, the literature proposes data augmentation. However, our knowledge about data augmentation is still quite limited, especially in the field of insect pest detection. The aim of this experimental study was to investigate the effect of several widely used augmentation techniques and their combinations on remote-sensed trap images with the YOLOv5 (small) object detector model. This study was carried out systematically on two different datasets starting from the single geometric and photometric transformation toward their combinations. Our results show that the model's mean average precision value (mAP50) could be increased from 0.844 to 0.992 and from 0.421 to 0.727 on the two datasets using the appropriate augmentation methods combination. In addition, this study also points out that the integration of photometric image transformations into the mosaic augmentation can be more efficient than the native combination of augmentation techniques because this approach further improved the model's mAP50 values to 0.999 and 0.756 on the two test sets, respectively.

A Neuroinspired Contrast Mechanism enables Few-Shot Object Detection

Deep learning-based object detectors often demand abundant annotated data for training. However, in practice, only limited training data are available, making Few-Shot Object Detection (FSOD) an attractive research topic. Existing two-stage proposal-based Faster R-CNN detectors for FSOD struggle to match the performance of models trained on large datasets. We argue that detectors trained with limited samples cannot establish robust comparison-based relationships. Additionally, FSOD methods only explore these relationships during the training phase. To address these issues, we draw inspiration from neuroscience studies and propose Residual Contrast Faster R-CNN (RcFRCN). RcFRCN incorporates two novel customized contrast blocks: a Residual Spatial Contrast Block and a Residual Proposal Contrast Block. These blocks capture cross-spatial and cross-proposal contrast information, enhancing both training and testing phases. We conduct comprehensive experiments on two FSOD benchmarks: PASCAL VOC and MS-COCO. Our RcFRCN achieves a mAP of 21.9 under a 30-shot setting on MS-COCO. It also achieves AP scores of 69.1, 55.8, and 64.0 under 10-shot settings of different splits on PASCAL VOC, respectively. Moreover, we apply RcFRCN on remote sensing and use our contrast blocks for open-vocabulary detection. Experiment results on these tasks also demonstrate the robustness and generalization ability of our methods.

Faster R–CNN, RetinaNet and Single Shot Detector in different ResNet backbones for marine vessel detection using cross polarization C-band SAR imagery

Detection of marine vessels plays an important role in monitoring, managing and securing seas and oceans, and forms the foundation of Maritime Domain Awareness (MDA). Although marine vessel detection has remained an active area of research for many years, unlike other object detectors, techniques of detection have been left far behind and lack systematic robustness. Hence, this study compared the performance of Faster R–CNN, RetinaNet and Single Shot Detector (SSD) across different epochs and complexities of ResNet architectures using Sentinel-1 VH polarization in one of the busiest ports in the Philippines. In particular, the models were created from the training samples dataset derived from Sentinel-1 VH imagery captured on January 12, 2024 in ResNet-34, -50, and −101 backbones, and 20 and 100 epochs. In this study, a total of 18 different object detector models were created for the comparative analysis. The models were tested with respect to different dates but having the same imagery type to determine their applicability across other base maps. Faster R–CNN with the highest F1 score of 0.85 outperformed RetinaNet with a highest F1 score of 0.74 and SSD with the highest F1 score of 0.38. The fastest model created was SSD, with an average speed of 9 to 44 minutes, followed by RetinaNet with an average speed of 8 to 58 minutes; the slowest is Faster R–CNN with an average speed of 25 minutes to 1 hour and 3 minutes. The use of Sentinel-1 VH imagery for marine vessel detection is a viable alternative, but the choice of object detectors should be carefully considered. The presence of geospatial software with advance deep learning tools improves remote sensing applications and allows non-programmers to optimize their competence. This study highlights the potential utilization of other imagery with higher spatial resolution, testing of other deep learning algorithms, finetuning of parameters, and utilization of higher computing infrastructure. The findings of this study can be applied in other areas for MDA, particularly in regions where advanced remote sensing applications have yet to be extensively explored.

Dehazing & Reasoning YOLO: Prior knowledge-guided network for object detection in foggy weather

Fast and accurate object detection in foggy weather is crucial for visual tasks such as autonomous driving and video surveillance. Existing methods typically preprocess images with enhancement techniques before the object detector, so that the real-time performance of object detection decreases to some extent. Meanwhile, many popular object detection models rely solely on visual features for localization and classification. When fog is present, visual features would be so adversely impacted that the detection accuracy sharply decreases. Therefore, we propose an end-to-end prior knowledge-guided network called DR-YOLO for object detection in foggy weather. DR-YOLO integrates the atmospheric scattering model and the co-occurrence relation graph as prior knowledge into the entire training process of the detector. Firstly, Restoration Subnet Module (RSM) is designed to employ the atmospheric scattering model to guide the learning direction of the detector for dehazing features. Specifically, it is only adopted during the training process and does not increase the time cost of detection process. Secondly, for guiding the detector to pay more attention to potential co-occurring objects in the same scene, we introduce Relation Reasoning Attention Module (RRAM) that utilizes the co-occurrence relation graph to supplement deficient visual features in foggy weather. In addition, DR-YOLO employs Adaptive Feature Fusion Module (AFFM) to effectively merge the key features from the backbone and neck for the needs of RRAM and RSM. Finally, we conduct experiments on clear, synthetic and real-world foggy datasets to demonstrate the effectiveness of DR-YOLO. The source code is available at https://github.com/wenxinss/DR-YOLO.

Comparative Evaluation of Convolutional Neural Network Object Detection Algorithms for Vehicle Detection.

The domain of object detection was revolutionized with the introduction of Convolutional Neural Networks (CNNs) in the field of computer vision. This article aims to explore the architectural intricacies, methodological differences, and performance characteristics of three CNN-based object detection algorithms, namely Faster Region-Based Convolutional Network (R-CNN), You Only Look Once v3 (YOLO), and Single Shot MultiBox Detector (SSD) in the specific domain application of vehicle detection. The findings of this study indicate that the SSD object detection algorithm outperforms the other approaches in terms of both performance and processing speeds. The Faster R-CNN approach detected objects in images with an average speed of 5.1 s, achieving a mean average precision of 0.76 and an average loss of 0.467. YOLO v3 detected objects with an average speed of 1.16 s, achieving a mean average precision of 0.81 with an average loss of 1.183. In contrast, SSD detected objects with an average speed of 0.5 s, exhibiting the highest mean average precision of 0.92 despite having a higher average loss of 2.625. Notably, all three object detectors achieved an accuracy exceeding 99%.

Leveraging foreground–background cues for semantically-driven, training-free moving object detection

Deep learning algorithms and semantic cue-based methods are increasingly used to detect moving objects. The semantic label changes in pixel location over time are used for moving object detection in semantic cue-based studies. In this approach, traditional background subtraction is employed only when there is a lack of semantic background. However, the change in semantic information in each pixel is insufficient for accurate detection of moving objects due to similar labels assignments for different objects and limited use of pixel intensity data. Moreover, challenges arise in detecting moving objects, such as a lack of diversity in labeled objects, an uneven distribution of labels or classes, and an insufficient amount of unseen datasets. In this study, we propose the Moving Semantic Object Detector (MSemOD), a train-less moving object detection method based on a traditional method enhanced with semantic information provided by pre-trained semantic segmentation models. The proposed method includes a region-based decision step, utilizing the traditional methods and semantic information as spatial–temporal cues. Threshold values are applied in the method to determine the temporal and spatial movements of each segment in the image. To improve the consistency of the semantic information, the proposed method converts the semantic segmentation output into a semantic probability map by enhancing the temporal and spatial state. The performance of the proposed method is evaluated on the LASIESTA, CDNet2014, and the BMC datasets and the results are compared with the state-of-the-art algorithms. The results show that using region-based semantic information can significantly reduce false positives and enhance foreground–background segmentation in moving object detection.

Language-aware multiple datasets detection pretraining for DETRs

Pretraining on large-scale datasets can boost the performance of object detectors while the annotated datasets for object detection are hard to scale up due to the high labor cost. What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly pretrain models across aggregation of datasets to enhance data volume and diversity. In this paper, we propose a strong framework for utilizing Multiple datasets to pretrain DETR-like detectors, termed METR, without the need for manual label spaces integration. It converts the typical multi-classification in object detection into binary classification by introducing a pre-trained language model. Specifically, we design a category extraction module for extracting potential categories involved in an image and assign these categories into different queries by language embeddings. Each query is only responsible for predicting a class-specific object. Besides, to adapt our novel detection paradigm, we propose a Class-wise Bipartite Matching strategy that limits the ground truths to match queries assigned to the same category. Extensive experiments demonstrate that METR achieves extraordinary results on either multi-task joint training or the pretrain & finetune paradigm. Notably, our pre-trained models have high flexible transferability and increase the performance upon various DETR-like detectors on COCO val2017 benchmark. Our code is publicly available at: https://github.com/isbrycee/METR.

Artificial immune systems for data augmentation

We study object detection models and observe that their respective architectures are vulnerable to image distortions such as noise, compression, blur, or snow. We propose alleviating this problem by training the models with antibodies generated using Artificial Immune Systems (AIS) from original training samples (antigens). These antibodies are AIS-distorted antigens at the pixel level through cycles of “select, clone, mutate, select” until an affinity to the antigen is achieved. We then add the antibodies to the antigens, train the models, validate and test them under 15 distortions, and show that our data augmentation approach (AISbod) significantly improved their accuracy without altering their architecture or inference speed. For example, the DINO object detector under the COCO dataset improves by 4% under clean samples, by 6.50% on average over all 15 distortions, by 2.15% under snow, and by 27.60% under impulse noise. Our simulations show that our method performs better under distortions and clean samples than related defense methods and is more consistent across datasets and object detection models. For instance, our method is, on average, 70% better than the closest related method across 15 distortions for the evaluated models under COCO. Moreover, we show that our approach to image classification and object tracking models significantly improves accuracy under distortions. We provide the code of our method and the DINO model trained using our method at https://github.com/moforio/AISbod.

Detection Based on Semantics and a Detail Infusion Feature Pyramid Network and a Coordinate Adaptive Spatial Feature Fusion Mechanism Remote Sensing Small Object Detector

In response to the challenges of remote sensing imagery, such as unmanned aerial vehicle (UAV) aerial imagery, including differences in target dimensions, the dominance of small targets, and dense clutter and occlusion in complex environments, this paper optimizes the YOLOv8n model and proposes an innovative small-object-detection model called DDSC-YOLO. First, a DualC2f structure is introduced to improve the feature-extraction capabilities of the model. This structure uses dual-convolutions and group convolution techniques to effectively address the issues of cross-channel communication and preserving information in the original input feature mappings. Next, a new attention mechanism, DCNv3LKA, was developed. This mechanism uses adaptive and fine-grained information-extraction methods to simulate receptive fields similar to self-attention, allowing adaptation to a wide range of target size variations. To address the problem of false and missed detection of small targets in aerial photography, we designed a Semantics and Detail Infusion Feature Pyramid Network (SDI-FPN) and added a dedicated detection scale specifically for small targets, effectively mitigating the loss of contextual information in the model. In addition, the coordinate adaptive spatial feature fusion (CASFF) mechanism is used to optimize the original detection head, effectively overcoming multi-scale information conflicts while significantly improving small target localization accuracy and long-range dependency perception. Testing on the VisDrone2019 dataset shows that the DDSC-YOLO model improves the mAP0.5 by 9.3% over YOLOv8n, and its performance on the SSDD and RSOD datasets also confirms its superior generalization capabilities. These results confirm the effectiveness and significant progress of our novel approach to small target detection.

Clustering-based mask recovery for image captioning

Region features play a huge role in image captioning. However, obtaining region features requires pre-training an object detector by using a large number of object detection datasets. It may be impossible for the end-to-end training. And if there is a large distribution difference between the object detection datasets and the image captioning datasets, the object detector may not be able to extract accurate region features. This makes it limited in application. In this paper, we propose a clustering-based mask recovery for image captioning. In the encoder, the pseudo-region features are obtained by clustering the grid features, which are extracted using Swin Transformer. Then we input the grid features together with the pseudo-region features into the decoder, and make the model to dynamically learns the weights of the two features in the decoding process to minimize the effect of errors caused by clustering. By using a clustering method to generate pseudo-region features for images, not only does the training process become end-to-end, but there is no need to introduce additional object detection datasets to train the object detector. In addition, the Transformer decoder has a misplaced problem in the decoding process. This means that the positional information used by the model when generating a word is not the same as the positional information used when it continues to use the word to reason. This may have some negative impact on the position encoding of the model. Therefore, we changed the original decoding method to mask recovery. Furthermore, a masked multi-head attention module with relative position is proposed in the decoder to integrate the information in the fusion features, and reconstruct the relative position relationship between words. We conduct experiments on MSCOCO 2014 dataset. The experiment results show that our model obtains 144.3% (single model) and 147.0% (ensemble of 4 models) CIDEr scores on ‘Karpathy’ offline test split, and 143.2% (c40) CIDEr scores on the official online test server.

Low-latency and high-accuracy hybrid image- and event-based object detector

Low-latency and high-accuracy hybrid image- and event-based object detector

A Defect Detection Method Based on YOLOv7 for Automated Remanufacturing

Remanufacturing of mechanical parts has recently gained much attention due to the rapid development of green technologies and sustainability. Recent efforts to automate the inspection step in the remanufacturing process using artificial intelligence are noticeable. In this step, a visual inspection of the end-of-life (EOL) parts is carried out to detect defective regions for restoration. This operation relates to the object detection process, a typical computer vision task. Many researchers have adopted well-known deep-learning models for the detection of damage. A common technique in the object detection field is transfer learning, where general object detectors are adopted for specific tasks such as metal surface defect detection. One open-sourced model, YOLOv7, is known for real-time object detection, high accuracy, and optimal scaling. In this work, an investigation into the YOLOv7 behavior on various public metal surface defect datasets, including NEU-DET, NRSD, and KolektorSDD2, is conducted. A case study validation is also included to demonstrate the model’s application in an industrial setting. The tiny variant of the YOLOv7 model showed the best performance on the NEU-DET dataset with a 73.9% mAP (mean average precision) and 103 FPS (frames per second) in inference. For the NRSD dataset, the model’s base variant resulted in 88.5% for object detection and semantic segmentation inferences. In addition, the model achieved 65% accuracy when testing on the KolektorSDD2 dataset. Further, the results are studied and compared with some of the existing defect detection models. Moreover, the segmentation performance of the model was also reported.

FSH-DETR: An Efficient End-to-End Fire Smoke and Human Detection Based on a Deformable DEtection TRansformer (DETR).

Fire is a significant security threat that can lead to casualties, property damage, and environmental damage. Despite the availability of object-detection algorithms, challenges persist in detecting fires, smoke, and humans. These challenges include poor performance in detecting small fires and smoke, as well as a high computational cost, which limits deployments. In this paper, we propose an end-to-end object detector for fire, smoke, and human detection based on Deformable DETR (DEtection TRansformer) called FSH-DETR. To effectively process multi-scale fire and smoke features, we propose a novel Mixed Encoder, which integrates SSFI (Separate Single-scale Feature Interaction Module) and CCFM (CNN-based Cross-scale Feature Fusion Module) for multi-scale fire, smoke, and human feature fusion. Furthermore, we enhance the convergence speed of FSH-DETR by incorporating a bounding box loss function called PIoUv2 (Powerful Intersection of Union), which improves the precision of fire, smoke, and human detection. Extensive experiments on the public dataset demonstrate that the proposed method surpasses state-of-the-art methods in terms of the mAP (mean Average Precision), with mAP and mAP50 reaching 66.7% and 84.2%, respectively.

DIB-UAP: enhancing the transferability of universal adversarial perturbation via deep information bottleneck

Significant structural differences in DNN-based object detectors hinders the transferability of adversarial attacks. Studies show that intermediate features extracted by the detector contain more model-independent information, and disrupting these features can enhance attack transferability across different detectors. However, the challenge lies in selecting crucial features that impact detection from redundant intermediate features. To address this issue, we introduce the Deep information bottleneck universal adversarial perturbation (DIB-UAP). DIB-UAP utilizes the deep information bottleneck to establish a link between intermediate features and model output, extracting crucial intermediate features and disrupting them to generate UAP with strong attack transferability. Additionally, we propose a data augmentation method, Scale & Tile, which effectively enhances the attack performance of UAP on medium and large-scale objects. Testing on two benchmark datasets with eight comparative methods across four black-box mainstream detectors has confirmed the attack transferability of DIB-UAP. Furthermore, practical utility validation of DIB-UAP has been conducted on a commercial object detection platform.

X-Detect: explainable adversarial patch detection for object detectors in retail

Object detection models, which are widely used in various domains (such as retail), have been shown to be vulnerable to adversarial attacks. Existing methods for detecting adversarial attacks on object detectors have had difficulty detecting new real-life attacks. We present X-Detect, a novel adversarial patch detector that can: (1) detect adversarial samples in real time, allowing the defender to take preventive action; (2) provide explanations for the alerts raised to support the defender’s decision-making process, and (3) handle unfamiliar threats in the form of new attacks. Given a new scene, X-Detect uses an ensemble of explainable-by-design detectors that utilize object extraction, scene manipulation, and feature transformation techniques to determine whether an alert needs to be raised. X-Detect was evaluated in both the physical and digital space using five different attack scenarios (including adaptive attacks) and the benchmark COCO dataset and our new Superstore dataset. The physical evaluation was performed using a smart shopping cart setup in real-world settings and included 17 adversarial patch attacks recorded in 1700 adversarial videos. The results showed that X-Detect outperforms the state-of-the-art methods in distinguishing between benign and adversarial scenes for all attack scenarios while maintaining a 0% FPR (no false alarms) and providing actionable explanations for the alerts raised. A demo is available.

Head motion-corrected eye gaze tracking with the da Vinci surgical system.

To facilitate the integration of point of gaze (POG) as an input modality for robot-assisted surgery, we introduce a robust head movement compensation gaze tracking system for the da Vinci Surgical System. Previous surgical eye gaze trackers require multiple recalibrations and suffer from accuracy loss when users move from the calibrated position. We investigate whether eye corner detection can reduce gaze estimation error in a robotic surgery context. A polynomial regressor is first used to estimate POG after an 8-point calibration, and then, using another regressor, the POG error from head movement is estimated from the shift in 2D eye corner location. Eye corners are computed by first detecting regions of interest using the You Only Look Once (YOLO) object detector trained on 1600 annotated eye images (open dataset included). Contours are then extracted from the bounding boxes and a derivative-based curvature detector refines the eye corner. Through a user study (n = 24), our corner-contingent head compensation algorithm showed an error reduction in degrees of visual angle of 1.20 (p = 0.037) for the left eye and 1.26 (p = 0.079) for the right compared to the previous gold-standard POG error correction method. In addition, the eye corner pipeline showed a root-mean-squared error of 3.57 (SD = 1.92) pixels in detecting eye corners over 201 annotated frames. We introduce an effective method of using eye corners to correct for eye gaze estimation, enabling the practical acquisition of POG in robotic surgery.

Few-Shot Object Detection in Remote Sensing Images via Data Clearing and Stationary Meta-Learning

Nowadays, the focus on few-shot object detection (FSOD) is fueled by limited remote sensing data availability. In view of various challenges posed by remote sensing images (RSIs) and FSOD, we propose a meta-learning-based Balanced Few-Shot Object Detector (B-FSDet), built upon YOLOv9 (GELAN-C version). Firstly, addressing the problem of incompletely annotated objects that potentially breaks the balance of the few-shot principle, we propose a straightforward yet efficient data clearing strategy, which ensures balanced input of each category. Additionally, considering the significant variance fluctuations in output feature vectors from the support set that lead to reduced effectiveness in accurately representing object information for each class, we propose a stationary feature extraction module and corresponding stationary and fast prediction method, forming a stationary meta-learning mode. In the end, in consideration of the issue of minimal inter-class differences in RSIs, we propose inter-class discrimination support loss based on the stationary meta-learning mode to ensure the information provided for each class from the support set is balanced and easier to distinguish. Our proposed detector’s performance is evaluated on the DIOR and NWPU VHR-10.v2 datasets, and comparative analysis with state-of-the-art detectors reveals promising performance.

EUAVDet: An Efficient and Lightweight Object Detector for UAV Aerial Images with an Edge-Based Computing Platform

Crafting an edge-based real-time object detector for unmanned aerial vehicle (UAV) aerial images is challenging because of the limited computational resources and the small size of detected objects. Existing lightweight object detectors often prioritize speed over detecting extremely small targets. To better balance this trade-off, this paper proposes an efficient and low-complexity object detector for edge computing platforms deployed on UAVs, termed EUAVDet (Edge-based UAV Object Detector). Specifically, an efficient feature downsampling module and a novel multi-kernel aggregation block are first introduced into the backbone network to retain more feature details and capture richer spatial information. Subsequently, an improved feature pyramid network with a faster ghost module is incorporated into the neck network to fuse multi-scale features with fewer parameters. Experimental evaluations on the VisDrone, SeaDronesSeeV2, and UAVDT datasets demonstrate the effectiveness and plug-and-play capability of our proposed modules. Compared with the state-of-the-art YOLOv8 detector, the proposed EUAVDet achieves better performance in nearly all the metrics, including parameters, FLOPs, mAP, and FPS. The smallest version of EUAVDet (EUAVDet-n) contains only 1.34 M parameters and achieves over 20 fps on the Jetson Nano. Our algorithm strikes a better balance between detection accuracy and inference speed, making it suitable for edge-based UAV applications.

Efficient flexible voxel-based two-stage network for 3D object detection in autonomous driving

3D object detection from the LiDAR point cloud plays an important role in autonomous driving. It is difficult to balance inference speed and detection accuracy when performing 3D point cloud object detection due to the large size of point cloud data and its unstructured storage, which makes it difficult to represent its features. To address the challenge, we propose a two-stage point cloud object detector, AFV-RCNN. In stage-1, the attention flexible voxel feature encoding layer is introduced, which utilizes flexible voxels to enhance feature encoding speed and focuses on foreground points to be detected through voxel attention. In stage-2, the multi-level and grid-based multi-scale RoI (Region of Interest) feature fusion module is designed. It directly extracts complete 3D structures from 3D region proposals and focuses on both local and global features through multi-scale partitioning. In the training stage, GHM-C Loss is applied to address the challenges associated with imbalanced target categories and the imbalance between difficult and easy samples in the classification task. We evaluate the model on the public KITTI Dataset and Waymo Open Dataset. The mAP in KITTI for 3D detection is 73.41% and inference on a single GPU reaches 30.0 fps. Compared with other state-of-the-art methods, AFV-RCNN achieves both the inference speed of a single-stage detector and the detection accuracy of a two-stage detector. It ensures higher detection accuracy while efficiently processing the point cloud.

Line segment detectors with deformable attention

The object detectors based on Transformers are advancing rapidly. On the contrary, the development of line segment detectors is relatively slow. It is noteworthy that the object and line segments are both 2D targets. In this work, we design a line segment detection algorithm based on deformable attention. Leveraging this algorithm and the line segment loss function, we transform the object detectors, Deformable DETR and ViDT, into end-to-end line segment detectors named Deformable LETR and ViDTLE, respectively. In order to adapt the idea of sparse modeling for line segment detection, we propose a new attention mechanism named line segment deformable attention (LSDA). This mechanism focuses on the valuable positions under the guidance of reference line to refine line segments. We design an auxiliary algorithm named line segment iterative refinement for LSDA. With as few modifications as possible, we transform two object detection detectors, namely SMCA DETR and PnP DETR into competitive line segment detectors named SMCA LETR and PnP LETR, respectively. The experimental results show that the performances of the proposed methods are efficient.