Features For Object Detection Research Articles

Diverse Image Captioning via Panoptic Segmentation and Sequential Conditional Variational Transformer

Recently, transformer-based image captioning models have achieved significant performance improvement. However, due to the limitations of region visual features and deterministic projections between image space and caption space, existing methods still suffer from disentangled visual features and rigid sentences. To address these issues, we first introduce panoptic segmentation to extract the segmentation region features, which can effectively alleviate the visual confusion caused by the widely-adopted region visual features. Then, we propose a panoptic segmentation based sequential conditional variational transformer (PS-SCVT) framework for diverse image captioning, which not only accurately extracts the image visual representations by fusing the segmentation region features and object detection features, but has the ability of learning one-to-many mappings from image space to caption space. The experimental results demonstrate that our approach achieves better interpretability and generalization performance compared with the state-of-the-art diverse image captioning models.

Privacy‐protected object detection through trustworthy image fusion

AbstractThe neural network‐based technologies have emerged as a potent method for image fusion, object detection, and other computer vision tasks as the rapid development of deep learning. Multi‐band infrared images, in particular, capture a more extensive range of radiation details and information compared to conventional single‐band infrared images. Consequently, the fusion of multi‐band infrared images can provide more features for object detection. However, it is crucial to consider that infrared images may contain sensitive information, potentially leading to privacy concerns. Ensuring datasets privacy protection plays a crucial role in the fusion and tracking process. To address both the need for improved detection performance and the necessity for privacy protection in the infrared environment, we proposed a procedure for object detection based on multi‐band infrared image datasets and utilized the transfer learning technique to migrate knowledge learned from external infrared data to internal infrared data, thereby training the infrared image fusion model and detection model. The procedure consists of several steps: (1) data preprocessing of multi‐band infrared images, (2) multi‐band infrared image fusion, and (3) object detection. Standard evaluation metrics for image fusion and object detection ensure the authenticity of the experiments. The comprehensive validation experiments demonstrate the effectiveness of the proposed procedure in object detection tasks. Furthermore, the transfer learning can train our datasets and update the model without exposing the original data. This aspect of transfer learning is particularly beneficial for maintaining the privacy of multi‐band infrared images during the fusion and detection processes.

Object detection in drone video based on recurrent motion attention

In recent years, object detection in drone videos has garnered widespread attention. Conventional image-based detectors have encountered a performance bottleneck for drone images due to the severe appearance deterioration of targets and numerous background clutters. Video object detection (VOD) is a promising paradigm for such problems as it can effectively enhance target appearance by aggregating features from nearby frames in drone videos. However, struggling with the feature alignment problem which is particularly important for drone images where the background varies frequently. Following the VOD philosophy, we propose the Recurrent Motion Attention Network (RMA-Net) to extract spatial-temporal image features for object detection in drone vision. A recurrent fashion is designed to effectively and efficiently learn motion features in long sequences, only requiring a few frames of optical flows for initialization. Experiments demonstrate that the proposed method could easily promote existing object detectors to achieve high performance on drone videos. Specifically, our RMA adapted YOLOv7 outperforms the state-of-the-arts by a large margin on the challenging VisDrone2019-VID dataset, not only boosts the mean average precision by 6.99 % but also reaches a batch-1 inference speed of 31 frames per second.

Putting ChatGPT vision (GPT-4V) to the test: risk perception in traffic images.

Vision-language models are of interest in various domains, including automated driving, where computer vision techniques can accurately detect road users, but where the vehicle sometimes fails to understand context. This study examined the effectiveness of GPT-4V in predicting the level of 'risk' in traffic images as assessed by humans. We used 210 static images taken from a moving vehicle, each previously rated by approximately 650 people. Based on psychometric construct theory and using insights from the self-consistency prompting method, we formulated three hypotheses: (i) repeating the prompt under effectively identical conditions increases validity, (ii) varying the prompt text and extracting a total score increases validity compared to using a single prompt, and (iii) in a multiple regression analysis, the incorporation of object detection features, alongside the GPT-4V-based risk rating, significantly contributes to improving the model's validity. Validity was quantified by the correlation coefficient with human risk scores, across the 210 images. The results confirmed the three hypotheses. The eventual validity coefficient was r = 0.83, indicating that population-level human risk can be predicted using AI with a high degree of accuracy. The findings suggest that GPT-4V must be prompted in a way equivalent to how humans fill out a multi-item questionnaire.

A Convolution with Transformer Attention Module Integrating Local and Global Features for Object Detection in Remote Sensing Based on YOLOv8n

Object detection in remote sensing scenarios plays an indispensable and significant role in civilian, commercial, and military areas, leveraging the power of convolutional neural networks (CNNs). Remote sensing images, captured by crafts and satellites, exhibit unique characteristics including complicated backgrounds, limited features, distinct density, and varied scales. The contextual and comprehensive information in an image can make a detector precisely localize and classify targets, which is extremely valuable for object detection in remote sensing scenarios. However, CNNs, restricted by the essence of the convolution operation, possess local receptive fields and scarce contextual information, even in large models. To address this limitation and improve detection performance by extracting global contextual information, we propose a novel plug-and-play attention module, named Convolution with Transformer Attention Module (CTAM). CTAM is composed of a convolutional bottleneck block and a simplified Transformer layer, which can facilitate the integration of local features and position information with long-range dependency. YOLOv8n, a superior and faster variant of the YOLO series, is selected as the baseline. To demonstrate the effectiveness and efficiency of CTAM, we incorporated CTAM into YOLOv8n and conducted extensive experiments on the DIOR dataset. YOLOv8n-CTAM achieves an impressive 54.2 mAP@50-95, surpassing YOLOv8n (51.4) by a large margin. Notably, it outperforms the baseline by 2.7 mAP@70 and 4.4 mAP@90, showcasing its superiority with stricter IoU thresholds. Furthermore, the experiments conducted on the TGRS-HRRSD dataset validate the excellent generalization ability of CTAM.

Deeply Unsupervised Patch Re-Identification for Pre-Training Object Detectors.

Unsupervised pre-training aims at learning transferable features that are beneficial for downstream tasks. However, most state-of-the-art unsupervised methods concentrate on learning global representations for image-level classification tasks instead of discriminative local region representations, which limits their transferability to region-level downstream tasks, such as object detection. To improve the transferability of pre-trained features to object detection, we present Deeply Unsupervised Patch Re-ID (DUPR), a simple yet effective method for unsupervised visual representation learning. The patch Re-ID task treats individual patch as a pseudo-identity and contrastively learns its correspondence in two views, enabling us to obtain discriminative local features for object detection. Then the proposed patch Re-ID is performed in a deeply unsupervised manner, appealing to object detection, which usually requires multi-level feature maps. Extensive experiments demonstrate that DUPR outperforms state-of-the-art unsupervised pre-trainings and even the ImageNet supervised pre-training on various downstream tasks related to object detection.

HYDRO-3D: Hybrid Object Detection and Tracking for Cooperative Perception Using 3D LiDAR

3D-LiDAR-based cooperative perception has been generating significant interest for its ability to tackle challenges such as occlusion, sparse point clouds, and out-of-range issues that can be problematic for single-vehicle perception. Despite its effectiveness in overcoming various challenges, cooperative per-ception's performance can still be affected by the aforementioned issues when Connected Automated Vehicles (CAVs) operate at the edges of their sensing range. Our proposed approach called HYDRO-3D aims to improve object detection performance by explicitly incorporating historical object tracking information. Specifically, HYDRO-3D combines object detection features from a state-of-the-art object detection algorithm (V2X-ViT) with historical information from the object tracking algorithm to infer objects. Afterward, a novel spatial-temporal 3D neural network performing global and local manipulations of object-tracking historical data is applied to generate the feature map to enhance object detection. The proposed HYDRO-3D method is comprehensively evaluated on the state-of-the-art V2XSet. The qualitative and quantitative experiment results demonstrate that the HYDRO-3D can effectively utilize the object tracking information and achieve robust object detection performance. It outperforms the SOTA V2X-ViT by 3.7% in AP@0.7 of object detection for CAVs and can also be generalized to single-vehicle object detection with 4.5% improvement in AP@0.7.

CB-FPN: object detection feature pyramid network based on context information and bidirectional efficient fusion

CB-FPN: object detection feature pyramid network based on context information and bidirectional efficient fusion

Object Detection and Monitor System for Building Security Based on Internet of Things (IoT) Using Illumination Invariant Face Recognition

Theft and intrusion are crimes that often occur in neighborhoods when there is opportunity or negligence by owners and security personnel. Many studies have been carried out to improve environmental security by applying cameras as a surveillance medium. However, the camera is not optimal in detecting objects when the lighting conditions are lacking. Therefore, in this study, a monitoring and object detection system was built by applying the Illumination Invariant model. This model is used to improve the appearance of the image from light and shadow reflections. The process of detecting and identifying objects is done by using human facial features (face detection) captured by the camera. The camera used is a Logitec C270 Webcam 720p which is connected via a USB port on the Raspberry Pi 4. The Raspberry Pi 4 processes human face image data and sends the processing results to a MySQL database using the HTTP protocol. Data transmission is done using the Python Flask web framework. The system was successfully run 100% by using black box testing of all functional requirements. Tests on the object detection feature were carried out based on different lighting conditions 15 times by comparing the original image and the results of the Illumination Invariant implementation. Based on the test results obtained object detection accuracy of 86.7%.

Illumination-Guided RGBT Object Detection With Inter- and Intra-Modality Fusion

Robust object detection is hindered by various illumination conditions in real-world applications. Common practice introduces thermal modality to augment the detection capability of RGB images in poor illumination conditions. However, a major challenge for such work is how to leverage the complementary information of RGB and thermal images effectively. In this paper, we tackle this by combining the strengths of illumination-guided inter- and intra-modality information in both modalities. Specifically, we propose an effective object detection network dubbed TINet to adaptively fuse the complementary features extracted from RGB and thermal images. Firstly, we design an illumination-guided feature weighting module to guide the network in leaning towards a reliable modality. Then, an inter-modality attention module is developed to amplify and complement differential features between these two modalities. We further design an intra-modality attention module using object heatmap prediction to enhance the foreground features of each modality. Finally, we fuse the inter- and intra-modality features for object detection with illumination-guided feature weights. Experimental results demonstrate the effectiveness of the proposed method on the FLIR-aligned dataset and the KAIST multispectral pedestrian detection dataset.

Dynamic activation and enhanced image contour features for object detection

Object detection technology is a popular research direction which is widely used in areas such as autonomous driving and medical diagnosis. At this stage mobile devices often have limited storage resources to deploy large object detection networks and need to meet real-time requirements. This paper proposes a lightweight and efficient object detection model based on YOLOv4, first using the lightweight network GhostNet to extract image features and reduce the number of parameters and computation of the backbone structure; then combining AFmodule and Meta-ACON activation function to enhance the feature extraction capability of the backbone network, which strengthen the mode’s ability to capture image spatial feature information; this paper also designs the RL-PAFPN feature fusion structure is with the Reslayer module to further improve the model’s ability to extract and fuse image features. By comparing other mainstream object detection models, the YOLOv4-Ghost-AMR network in this paper has less computation and fewer parameters, and the accuracy of the model reaches 86.83%, which is suitable for deployment in mobile devices with limited storage. The model proposed in this paper can be applied to medical, traffic and fault detection fields, which changes the traditional manual detection method and saves manpower and time costs, achieving high precision real-time object detection.

Improvement in Error Recognition of Real-Time Football Images by an Object-Augmented AI Model for Similar Objects

In this paper, we analyze the recognition error of the general AI recognition model and propose the structure-modified and object-augmented AI recognition model. This model has object detection features that distinguish specific target objects in target areas where players with similar shapes and characteristics overlapped in real-time football images. We implemented the AI recognition model by reinforcing the training dataset and augmenting the object class. In addition, it is shown that the recognition rate increased by modifying the model structure based on the recognition errors analysis of general AI recognition models. Recognition errors were decreased by applying the modules of HSV processing and differentiated classes (overlapped player groups) learning to the general AI recognition model. We experimented in order to compare the recognition error reducing performance with the general AI model and the proposed AI model by the same real-time football images. Through this study, we have confirmed that the proposed model can reduce errors. It showed that the proposed AI model structure to recognize similar objects in real-time and in various environments could be used to analyze football games.

Spiral Search Grasshopper Features Selection with VGG19-ResNet50 for Remote Sensing Object Detection

Remote sensing object detection plays a major role in satellite imaging and is required in various scenarios such as transportation, forestry, and the ocean. Deep learning techniques provide efficient performance in remote sensing object detection. The existing techniques have the limitations of data imbalance, overfitting, and lower efficiency in detecting small objects. This research proposes the spiral search grasshopper (SSG) optimization technique to increase the exploitation in feature selection. Augmentation is applied in input images to generate more images, and this helps to train the model and reduces data imbalance problems. The VGG-19 and ResNet50 model is applied for feature extraction, and this helps to extract deep features to represent objects. The SSG feature selection technique increases the exploitation and select unique features for object detection that helps to overcome the data imbalance and overfitting problem. The SSG feature selection model helps to balance the exploration and exploitation that escape from the local optima trap. The SSG model has 82.45% mAP, the SSD model has 52.6% mAP, and the MPFP-Net model has 80.43% mAP.

Learning Balance Feature for Object Detection

In the field of studying scale variation, the Feature Pyramid Network (FPN) replaces the image pyramid and has become one of the most popular object detection methods for detecting multi-scale objects. State-of-the-art methods have FPN inserted into a pipeline between the backbone and the detection head to enable shallow features with more semantic information. However, FPN is insufficient for object detection on various scales, especially for small-scale object detection. One of the reasons is that the features are extracted at different network depths, which introduces gaps between features. That is, as the network becomes deeper and deeper, the high-level features have more semantics but less content description. This paper proposes a new method that includes a multi-scale receptive fields extraction module, a feature constructor module, and an attention module to improve the detection efficiency of FPN for objects of various scales and to bridge the gap in content description and semantics between different layers. Together, these three modules make the detector capable of selecting the most suitable feature for objects. Especially for the attention module, this paper chooses to use a parallel structure to simultaneously extract channel and spatial attention from the same features. When we use Adopting Adaptive Training Sample Selection (ATSS) and FreeAnchor as the baseline and ResNet50 as the backbone, the experimental results on the MS COCO dataset show that our algorithm can enhance the mean average precision (mAP) by 3.7% and 2.4% compared to FPN, respectively.

Adaptive feedback connection with a single‐level feature for object detection

From the perspective of detector optimisation, detecting objects using only a one-level feature cannot provide good performance for a wide range of scales. Various complex feature pyramidal structures address this problem using the divide-and-conquer strategy and multi-scale feature fusion. However, this requires adding too many additional convolutional layers and fusion operations. To address the issue, a simple detection part is proposed, which includes three components, namely a one-level feature map for detection, the encoder structure with feedback connection, and a decoupled head. The redesigned encoder and decoupled head can successfully address the performance decline caused by the one-level feature-based detection. Moreover, the proposed method can accelerate the convergence of the detector and achieve a faster inference time. Based on the optimised detection part, an adaptive feedback connection with a single-level feature (AFS) is proposed for object detection. The experiments conducted on the MS COCO 2017 benchmark show that the proposed method can achieve comparable results with its multi-scale pyramid counterpart, You Only Look Once v4 (YOLOv4). In addition, AFS can help the YOLOv4 achieve 44.9 mAP at 27 frame per second and converging 82 epochs earlier under the image size of 608×608, which represents a 42.1% improvements in the convergence speed.

HollowBox: An anchor‐free UAV detection method

Unmanned aerial vehicles (UAV) “black flight” incidents cause serious security risks and economic losses to airports. Additionally, existing UAV detection methods are mainly radar technology at airports. Whereas, it is unable to correctly identify the number of UAVs and visualise their size, which undoubtedly poses a serious security risk. Accordingly, an anchor-free UAV detection method HollowBox is proposed to supplement radar detection equipment for the airport “black flight” problem. It is inspired by the FoveaBox object detection method, the object detection feature layers are reset and the allocation ratio of positive and negative samples in the training phase are redefined, the HollowBox UAV detection is proposed according to the multi-size characteristics. Extensive experiments show that, the approach achieves 90.1% AP, 6% false detection rate and 17.2 FPS inference speed, which accomplished a satisfactory performance, as verified by the real-shot data collected of an airport in Tianjin. This work is of great significance for the application in airport UAV detection.

GF-Detection: Fusion with GAN of Infrared and Visible Images for Vehicle Detection at Nighttime

Vehicles are important targets in the remote sensing applications and nighttime vehicle detection has been a hot study topic in recent years. Vehicles in the visible images at nighttime have inadequate features for object detection. Infrared images retain the contours of vehicles while they lose the color information. Thus, it is valuable to fuse infrared and visible images to improve the vehicle detection performance at nighttime. However, it is still a challenge to design effective fusion models due to the complexity of visible and infrared images. In order to improve vehicle detection performance at nighttime, this paper proposes a fusion model of infrared and visible images with Generative Adversarial Networks (GAN) for vehicle detection named GF-detection. GAN is utilized in the image reconstruction and introduced in the image fusion recently. To be specific, to exploit more features for the fusion, GAN is utilized to fuse the infrared and visible images via the image reconstruction. The generator fuses the image features and detection features, and then generates the reconstructed images for the discriminator to classify. Two branches, visible and infrared branches, are designed in the GF-detection model. Different feature extraction strategies are conducted according to the variance of the visible and infrared images. Detection features and self-attention mechanism are added to the fusion model aiming to build a detection task-driven fusion model of infrared and visible images. Extensive experiments based on nighttime images are conducted to demonstrate the effectiveness of the proposed fusion model in night vehicle detection.

CE-FPN: enhancing channel information for object detection

Feature pyramid network (FPN) has been an efficient framework to extract multi-scale features in object detection. However, current FPN-based methods mostly suffer from the intrinsic flaw of channel reduction, which brings about the loss of semantical information. And the miscellaneous feature maps may cause serious aliasing effects. In this paper, we present a novel channel enhancement feature pyramid network (CE-FPN) to alleviate these problems. Specifically, inspired by sub-pixel convolution, we propose sub-pixel skip fusion (SSF) to perform both channel enhancement and upsampling. Instead of the original 1 × 1 convolution and linear upsampling, it mitigates the information loss due to channel reduction. Then we propose sub-pixel context enhancement (SCE) for extracting stronger feature representations, which is superior to other context methods due to the utilization of rich channel information by sub-pixel convolution. Furthermore, we introduce a channel attention guided module (CAG) to optimize the final integrated features on each level. It alleviates the aliasing effect only with a few computational burdens. We evaluate our approaches on Pascal VOC and MS COCO benchmark. Extensive experiments show that CE-FPN achieves competitive performance and is more lightweight compared to state-of-the-art FPN-based detectors.

Image captioning model using attention and object features to mimic human image understanding

Image captioning spans the fields of computer vision and natural language processing. The image captioning task generalizes object detection where the descriptions are a single word. Recently, most research on image captioning has focused on deep learning techniques, especially Encoder-Decoder models with Convolutional Neural Network (CNN) feature extraction. However, few works have tried using object detection features to increase the quality of the generated captions. This paper presents an attention-based, Encoder-Decoder deep architecture that makes use of convolutional features extracted from a CNN model pre-trained on ImageNet (Xception), together with object features extracted from the YOLOv4 model, pre-trained on MS COCO. This paper also introduces a new positional encoding scheme for object features, the “importance factor”. Our model was tested on the MS COCO and Flickr30k datasets, and the performance is compared to performance in similar works. Our new feature extraction scheme raises the CIDEr score by 15.04%. The code is available at: https://github.com/abdelhadie-almalla/image_captioning

CFC-Net: A Critical Feature Capturing Network for Arbitrary-Oriented Object Detection in Remote-Sensing Images

Object detection in optical remote sensing images is an important and challenging task. In recent years, the methods based on convolutional neural networks have made good progress. However, due to the large variation in object scale, aspect ratio, and arbitrary orientation, the detection performance is difficult to be further improved. In this paper, we discuss the role of discriminative features in object detection, and then propose a Critical Feature Capturing Network (CFC-Net) to improve detection accuracy from three aspects: building powerful feature representation, refining preset anchors, and optimizing label assignment. Specifically, we first decouple the classification and regression features, and then construct robust critical features adapted to the respective tasks through the Polarization Attention Module (PAM). With the extracted discriminative regression features, the Rotation Anchor Refinement Module (R-ARM) performs localization refinement on preset horizontal anchors to obtain superior rotation anchors. Next, the Dynamic Anchor Learning (DAL) strategy is given to adaptively select high-quality anchors based on their ability to capture critical features. The proposed framework creates more powerful semantic representations for objects in remote sensing images and achieves high-performance real-time object detection. Experimental results on three remote sensing datasets including HRSC2016, DOTA, and UCAS-AOD show that our method achieves superior detection performance compared with many state-of-the-art approaches. Code and models are available at https://github.com/ming71/CFC-Net.