Extract Object Features Research Articles

Development of a multi-level feature fusion model for basketball player trajectory tracking

To solve the problems of low matching degree, long tracking time, and low accuracy of multi-target tracking in the process of athlete motion trajectory tracking using deep learning technology, a new athlete motion trajectory tracking model was proposed in this study. The study first optimized the current object detection algorithm in basketball, utilized a hybrid attention mechanism to extract object features, and improved the non-maximum suppression strategy. Then, a hybrid branch network was introduced to improve the residual network and a new athlete identity recognition model was proposed. Finally, a new trajectory tracking model was designed by combining the object detection model and the athlete identity recognition model. The research results indicated that in the object detection experiment, the detection time of the proposed object detection algorithm was always below 0.4 s, and its average accuracy reached up to 0.63. In trajectory tracking testing, the final built tracking model had a multi-target tracking accuracy of up to 0.98, and its tracking overlap rate was as low as 0.02. This study has the following two contributions. Firstly, a new model of athlete trajectory tracking is proposed, which improves the accuracy and efficiency of multi-target tracking by optimizing object detection algorithm and introducing hybrid branch network. Second, the model has excellent performance in both object detection and track tracking, which can not only provide a new solution for athletes' motion trajectory tracking, but also significantly improve the effect of motion tracking.

T-UNet: triplet UNet for change detection in high-resolution remote sensing images

ABSTRACT Remote sensing image change detection aims to identify differences between images acquired at different times in the same area, crucial for land management, environmental monitoring, and disaster assessment. Current change detection methods mainly use Siamese or early fusion structures. Siamese networks focus on object features at different times but lack attention to change information, leading to false alarms and missed detections. Early fusion structures focus on fused features but neglect temporal object features, hindering accurate change detection. To address these issues, we propose a novel network, Triplet UNet (T-UNet), employing a triplet encoder to simultaneously extract object features and change features between the pre- and post-time-phase images. To effectively interact features extracted from the three branches of triplet encoder, we propose a multi-branch spatial-spectral cross-attention module. In the decoder, we employ channel attention and spatial attention mechanisms to fully mine and integrate detailed texture and semantic localization information. The proposed T-UNet surpasses seven other state-of-the-art methods on three publicly available datasets. Extensive experiments verify the effectiveness of the proposed structure and modules as well as the superiority of the proposed T-UNet. The source code for the proposed T-UNet is accessible at https://github.com/Pl-2000/T-UNet.

A feature enhancement FCOS algorithm for dynamic traffic object detection

The development of object detection plays an important role in the realisation of fully autonomous driving, and the feature extraction is the key step for object detection. There has been significant difference and scale variation of object features for different road traffic participants (RTPs), meanwhile traditional Convolutional Neural Networks (CNNs) was difficult to extract object features efficiently for small targets. In order to improve the ability of feature extraction, a RTP object detection method combining dynamic convolution and feature enhancement was proposed. The Fully Convolutional One-Stage (FCOS) object detection algorithm was used as baseline. First, the dynamic convolution module was designed in the backbone network to identify different object features to the maximum extent. Second, a dual attention module was designed to filter object feature information while reducing the amount of computation. Finally, in the detection part, the feature expression ability of shallow network was further enhanced by multi-scale feature fusion module, and the effectiveness of the proposed algorithm was verified using Cityscapes dataset. The experimental result indicated that mAP increased by 2.3% compared with baseline. This study can improve the efficiency of RTP detection and contribute to the industrialisation of intelligent connected vehicles.

Composite Backbone Small Object Detection Based on Context and Multi-Scale Information with Attention Mechanism

Object detection has gained widespread application across various domains; nevertheless, small object detection still presents numerous challenges due to the inherent limitations of small objects, such as their limited resolution and susceptibility to interference from neighboring elements. To improve detection accuracy of small objects, this study presents a novel method that integrates context information, attention mechanism, and multi-scale information. First, to realize feature augmentation, a composite backbone network is employed which can jointly extract object features. On this basis, to efficiently incorporate context information and focus on key features, the composite dilated convolution and attention module (CDAM) is designed, consisting of a composite dilated convolution module (CDM) and convolutional block attention module (CBAM). Then, a feature elimination module (FEM) is introduced to reduce the feature proportion of medium and large objects on feature layers; the impact of neighboring objects on small object detection can thereby be mitigated. Experiments conducted on MS COCO validate the superior performance of the method compared with baseline detectors, while it yields an average enhancement of 0.8% in overall detection accuracy, with a notable enhancement of 2.7% in small object detection.

LPST-Det: Local-Perception-Enhanced Swin Transformer for SAR Ship Detection

Convolutional neural networks (CNNs) and transformers have boosted the rapid growth of object detection in synthetic aperture radar (SAR) images. However, it is still a challenging task because SAR images usually have the characteristics of unclear contour, sidelobe interference, speckle noise, multiple scales, complex inshore background, etc. More effective feature extraction by the backbone and augmentation in the neck will bring a promising performance increment. In response, we make full use of the advantage of CNNs in extracting local features and the advantage of transformers in capturing long-range dependencies to propose a Swin Transformer-based detector for arbitrary-oriented SAR ship detection. Firstly, we incorporate a convolution-based local perception unit (CLPU) into the transformer structure to establish a powerful backbone. The local-perception-enhanced Swin Transformer (LP-Swin) backbone combines the local information perception ability of CNNs and the global feature extraction ability of transformers to enhance representation learning, which can extract object features more effectively and boost the detection performance. Then, we devise a cross-scale bidirectional feature pyramid network (CS-BiFPN) by strengthening the propagation and integration of both location and semantic information. It allows for more effective utilization of the feature extracted by the backbone and mitigates the problem of multi-scale ships. Moreover, we design a one-stage framework integrated with LP-Swin, CS-BiFPN, and the detection head of R3Det for arbitrary-oriented object detection, which can provide more precise locations for inclined objects and introduce less background information. On the SAR Ship Detection Dataset (SSDD), ablation studies are implemented to verify the effectiveness of each component, and competing experiments illustrate that our detector attains 93.31% in mean average precision (mAP), which is a comparable detection performance with other advanced detectors.

Few-Shot Air Object Detection Network

Focusing on the problem of low detection precision caused by the few-shot and multi-scale characteristics of air objects, we propose a few-shot air object detection network (FADNet). We first use a transformer as the backbone network of the model and then build a multi-scale attention mechanism (MAM) to deeply fuse the W- and H-dimension features extracted from the channel dimension and the local and global features extracted from the spatial dimension with the object features to improve the network’s performance when detecting air objects. Second, the neck network is innovated based on the path aggregation network (PANet), resulting in an improved path aggregation network (IPANet). Our proposed network reduces the information lost during feature transfer by introducing a jump connection, utilizes sparse connection convolution, strengthens feature extraction abilities at all scales, and improves the discriminative properties of air object features at all scales. Finally, we propose a multi-scale regional proposal network (MRPN) that can establish multiple RPNs based on the scale types of the output features, utilizing adaptive convolutions to effectively extract object features at each scale and enhancing the ability to process multi-scale information. The experimental results showed that our proposed method exhibits good performance and generalization, especially in the 1-, 2-, 3-, 5-, and 10-shot experiments, with average accuracies of 33.2%, 36.8%, 43.3%, 47.2%, and 60.4%, respectively. The FADNet solves the problems posed by the few-shot characteristics and multi-scale characteristics of air objects, as well as improving the detection capabilities of the air object detection model.

Multi-Target Tracking Based on a Combined Attention Mechanism and Occlusion Sensing in a Behavior-Analysis System.

Multi-object tracking (MOT) is a topic of great interest in the field of computer vision, which is essential in smart behavior-analysis systems for healthcare, such as human-flow monitoring, crime analysis, and behavior warnings. Most MOT methods achieve stability by combining object-detection and re-identification networks. However, MOT requires high efficiency and accuracy in complex environments with occlusions and interference. This often increases the algorithm's complexity, affects the speed of tracking calculations, and reduces real-time performance. In this paper, we present an improved MOT method combining an attention mechanism and occlusion sensing as a solution. A convolutional block attention module (CBAM) calculates the weights of space and channel attention from the feature map. The attention weights are used to fuse the feature maps to extract adaptively robust object representations. An occlusion-sensing module detects an object's occlusion, and the appearance characteristics of an occluded object are not updated. This can enhance the model's ability to extract object features and improve appearance feature pollution caused by the short-term occlusion of an object. Experiments on public datasets demonstrate the competitive performance of the proposed method compared with the state-of-the-art MOT methods. The experimental results show that our method has powerful data association capability, e.g., 73.2% MOTA and 73.9% IDF1 on the MOT17 dataset.

SOCDet: A Lightweight and Accurate Oriented Object Detection Network for Satellite On-Orbit Computing

In recent years, the performance of the deep learning based object detection models for remote sensing images improves with the increase of the hyperparameter scale. However, the state of the art of detection models are generally too cumbersome to adapt to the resource-constrained satellite platforms, and even general lightweight detection models cannot satisfy performance requirements. To solve those problems, a lightweight and accurate oriented object detection network for satellite on-orbit computing (SOCDet) is proposed from three levels. At the computing unit level, we propose an efficient computing unit Single-kernel Omni-dimensional Dynamic Convolution to make SOCDet feature extraction more efficient. At the network module level, we design a structural reparameterization block based on composite structure reparameterization to improve inference accuracy. A guided attention module for guiding SOCDet is designed to extract object features. We design a lightweight and concise one-stage detection architecture at the network architecture level to accommodate satellite platforms with extremely constrained computing and storage resources. We conduct ablation experiments on the ground server and compare experiments with the state-of-the-art methods on embedded GPU, using DOTA, HRSC2016 and FAIR1M. The experiment results show that SOCDet can achieve 2.36 times faster than the baseline in inference speed with 10.78 more mAP, 72.6% fewer Params and 92.56% fewer floating point operations. Compared with state-of-the-art methods, SOCDet improves the inference speed and derives competitive mAP results with affordable computing capability.

Improved Traffic Sign Detection Algorithm Based on Faster R-CNN

The traffic sign detection algorithm based on Faster Region-Based Convolutional Neural Network (R-CNN) has been applied to various intelligent-vehicles driving scenarios. However, the model of the current detection algorithm has certain shortcomings, which include the influence of weather and light, the detection of distance traffic signs, and the detection of similar traffic signs. To solve these problems, this paper proposes an improved traffic sign detection method based on Faster R-CNN. First, we propose a fusion method that fuses the feature pyramid into the Faster R-CNN algorithm. This fusion method can extract object features with precision and decrease the influence of weather and light. Second, a deformable convolution (DCN) which can train the algorithm to identify traffic signs with precision and make similar signs more distinguishable, and in particular make it work better with distorted images, is added to the backbone network. Lastly, we apply ROI align to replace the ROI pooling, which can avoid the distant traffic sign detail loss caused by pooling and increase the detection precision of distant traffic signs. The experimental results on both the TT100k dataset and real intelligent vehicle tests demonstrate that the algorithm is superior to the original Faster R-CNN algorithm and four other state-of-the-art methods in traffic sign detection, specifically in small-target traffic sign detection and low-intensity environments such as sunset time and rainy days. Therefore, the proposed method is helpful to improve the traffic sign detection performance in extreme environments (low-light intensity or rainy weather).

Deep Morphological Neural Networks

Mathematical morphology intends to extract object features such as geometric and topological structures in digital images. Given a set of target images and original images, it is cumbersome and time-consuming to determine the suitable morphological operations and structuring elements. In this paper, we propose deep morphological neural networks, which include a nonlinear feature extraction layer to learn the structuring element correctly and an adaptive layer to select appropriate morphological operations automatically. We demonstrate the applications of object recognition, including hand-written digits, geometric shapes, traffic signs, and brain tumor. Experimental results show the higher computational efficiency and higher accuracy of our developed model as compared against existing convolutional neural network models.

Exploring the Spatio‐Temporal Aware Graph for video captioning

Describing video content automatically in natural language sentences is a long-standing challenge in computer vision. Although existing methods that capture relational information among objects have made significant strides in the past years, the detailed geometrical and temporal information of objects remains to be further explored. To address this problem, a novel Spatio-Temporal Aware Graph is proposed to capture more elaborate visual representations, which are able to exploit the detailed spatio-temporal clues of the extracted object features. By performing graph-structured aggregation, the proposed model is capable of capturing not only the interactions among objects but also the detailed spatio-temporal relations. Meanwhile, a Frame Similarity Graph is constructed on frame features to learn comprehensive representations, which aim to extract the global information that the object feature lacks. Moreover, to capture rich video semantics from different perspectives, multiple video representations, that is appearance and motion information, are utilised to learn discriminative representations. Experiments on the prevalent benchmarks: Microsoft Video Description Corpus and Microsoft Research Video to Text demonstrate that the proposed approach achieves state-of-the-art performance in several widely used evaluation metrics: BLEU-4, METEOR, ROUGE, and CIDEr.

LMO-YOLO: A Ship Detection Model for Low-Resolution Optical Satellite Imagery

It has been observed that the existing convolutional neural network (CNN)-based ship detection models often result in high false detection rate in low-resolution optical satellite images. This problem arises from the following factors: 1) the current 8-b rescaling schemes make the images lose some important information about ships in low-resolution imagery; 2) the effective features of ships at low resolution are far fewer than those of ships at high resolution; and 3) the detection of low-resolution ships is more sensitive to object-background contrast variation. To solve these problems, a low-resolution marine object (LMO) detection YOLO model, called LMO-YOLO, is proposed in this article. First, a multiple linear rescaling scheme is developed to quantize the original satellite images into 8-b images; second, dilated convolutions are included in a YOLO network to extract object features and object-background features; finally, an adaptive weighting scheme is designed to balance the loss between easy-to-detect ships and hard-to-detect ships. The proposed method was validated by level 1 product images captured by the wide-field-of-view sensor on the GaoFen-1 satellite. The experimental results demonstrated that our method accurately detected ships from low-resolution images and outperformed state-of-the-art methods.

R²IPoints: Pursuing Rotation-Insensitive Point Representation for Aerial Object Detection

Anchor-free aerial object detection methods have recently attracted much attention due to their simplicity and efficiency. However, the performance is still unsatisfactory due to the following two main limitations. On the one hand, the anchor-free detector employs ordinary convolution layers with axis-aligned receptive fields to extract object features, resulting in lacking internal mechanisms to handle the rotation variance. On the other hand, the detector sacrifices much semantic information to achieve faster detection, leading to the inability to deal with objects’ high inter-class similarity and intra-class diversity. To address these issues, in this paper, we present a unique anchor-free detector, termed Rotation-Insensitive Point Representation (R2IPoints), of which a set of category-aware points are employed to encode the spatial and semantic information of the arbitrary-oriented objects. Specifically, we first devise a Stacked Rotation convolution Module (SRM) to encourage the learning of rotation-insensitive point representation by adaptively modelling orientation-agnostic interdependencies over stochastically rotated features. Meanwhile, we further introduce a Class-specific Semantic enhancement Module (CSM). It performs category-aware semantic activation to recalibrate features, thus enabling the point representation to be aware of object categories. Through jointly optimizing the two proposed modules in an end-to-end manner, R2IPoints could simultaneously generate rotation-insensitive and category-aware point representation. Extensive experiments on the challenging DIOR and DOTA datasets demonstrate the superiority of the proposed method. We achieve 72.7% mAP on DIOR and 74.34% mAP on DOTA, surpassing the baseline method of +2.4% mAP and +2.49% mAP, respectively. The code is available at https://github.com/shnew/R2IPoints.

DGANet: Dynamic Gradient Adjustment Anchor-Free Object Detection in Optical Remote Sensing Images

Remote sensing image object detection has been studied by many researchers in recent years using deep neural networks. However, optical remote sensing images contain many scenes with small and dense objects, resulting in a high rate of misrecognition. Firstly, in this work we selected a deep layer aggregation network with updated deformable convolution layers as the backbone to extract object features. The detection and classification of objects was based on the center-point network without non-maximum suppression. Secondly, the dynamic gradient adjustment embedded into the classification loss function was put forward to harmonize the quantity imbalance between easy and hard examples, as well as between positive and negative examples. Furthermore, the complete intersection over union (CIoU) loss function was selected as the objective function of bounding box regression, which achieves better convergence speed and accuracy. Finally, in order to validate the effectiveness and precision of the dynamic gradient adjustment network (DGANet), we conducted a series of experiments in remote sensing public datasets UCAS-AOD and LEVIR. The comparison experiments demonstrate that the DGANet achieves a more accurate detection result in optical remote sensing images.

Detection of level crossing barriers using the histogram of oriented gradients method and support vector machine

Railroad crossing is a place where the railroad lines intersect with other roads, such as a highway. Referring to the Regulation of Minister of Transportation No. 36/2011, level crossing must be equipped with signs, markers and traffic signaling devices and crossing gate guards. However, 4600 of the 5800 level crossing points are without railroad keeper so that they are prone to traffic accidents. In addition, hazard information (danger signs) from the railroad keeper to the PUSDALOP and machinists sometimes cannot be seen at night and in a foggy situation. Therefore, this research aims to detect obstacles (cars) at a railroad crossing using the Histogram of Oriented gradient (HOG) method and the Support Vector Machine (SVM) classifier. HOG functions to extract object features (cars), while SVM is responsible for classifying car objects whether they fit the criteria of car features or not. The results show that an accuracy rate of car objects was 85%, 73% for empty train tracks and 91% for detection of passing trains.

Learning Scene Attribute for Scene Recognition

Scene recognition has been a challenging task in the field of computer vision and multimedia for a long time. The current scene recognition works often extract object features and scene features through CNN, and combine these two types of features to obtain complementary and discriminative scene representations. However, when the scene categories are visually similar, the object features might lack of discriminations. Therefore, it may be debatable to consider only object features. In contrast to the existing works, in this paper, we discuss the discrimination of scene attributes in local regions and utilize scene attributes as the complementary features of object and scene features. We extract these visual features from two individual CNN branches, one extracting the global features of the image while the other extracting the features of local regions. Through contextual modeling framework, we aggregate these features and generate more discriminative scene representations, which achieve better performance than the feature aggregation of object and scene. Moreover, we achieve the new state-of-the-art performance on both standard scene recognition benchmarks by aggregating more complementary visual features: MIT67 (88.06%) and SUN397 (74.12%).

Visual Content Recognition by Exploiting Semantic Feature Map with Attention and Multi-task Learning

Recent studies have shown that spatial relationships among objects are very important for visual recognition, since they can provide rich clues on object contexts within the images. In this article, we introduce a novel method to learn the Semantic Feature Map (SFM) with attention-based deep neural networks for image and video classification in an end-to-end manner, aiming to explicitly model the spatial object contexts within the images. In particular, we explicitly apply the designed gate units to the extracted object features for important objects selection and noise removal. These selected object features are then organized into the proposed SFM, which is a compact and discriminative representation with the spatial information among objects preserved. Finally, we employ either Fully Convolutional Networks (FCN) or Long-Short Term Memory (LSTM) as the classifiers on top of the SFM for content recognition. A novel multi-task learning framework with image classification loss, object localization loss, and grid labeling loss are also introduced to help better learn the model parameters. We conduct extensive evaluations and comparative studies to verify the effectiveness of the proposed approach on Pascal VOC 2007/2012 and MS-COCO benchmarks for image classification. In addition, the experimental results also show that the SFMs learned from the image domain can be successfully transferred to CCV and FCVID benchmarks for video classification.

Representation of object's shape by multiple electric images in electrolocation.

Weakly electric fish generate an electric field by discharging an electric organ located on the tail region. An object near the fish modulates the self-generated electric field. The modulated field enables the fish to perceive objects even in complete darkness. The ability to perceive objects is provided by the electrosensory system of the fish. Electroreceptors distributed on the fish's skin surface can sense the modulated field, on the basis of transdermal voltage across the skin surface, called electric images. The fish can extract object's features such as lateral distance, size, shape, and electric property from an electric image. Although previous studies have demonstrated the relationship between electric-image features and object's distance and size, it remains unclear what features of an electric image represent the object's shape. We make here a hypothesis that shape information is not represented by a single image but by multiple images caused by the object's rotation or fish movement around the object. To test the hypothesis, we develop a computational model that can predict electric images produced by the rotation of differently shaped objects. We used five different shapes of resistive objects: a circle, a square, an equilateral triangle, a rectangle, and an ellipsoid. We show that differently shaped objects of a fixed arrangement generate similar Gaussian electric images, irrespective of their shapes. We also show that the features of an electric image such as the peak amplitude, half-maximum width, and peak position exhibit the angle-dependent variations characteristic to object rotation, depending on object shapes and lateral distances. Furthermore, we demonstrate that an integration effect of the peak amplitude and half-maximum width could be an invariant measure of object shape. These results suggest that the fish could perceive an object shape by combining those image features produced during exploratory behaviors around the object.

Robust Similar Image Retrieval Based on Extracted Object Features

Robust Similar Image Retrieval Based on Extracted Object Features

Digital passband processing of wideband-modulated optical signals for enhanced underwater imaging.

Radar modulation, demodulation, and signal processing techniques have been merged with laser imaging to enhance visibility in murky underwater environments. The modulation provides a way to reject multiple scattered light that would otherwise reduce image contrast and resolution. Recent work has focused on the use of wideband modulation schemes and digital passband processing to resolve range details of an underwater scene. Use of the CLEAN algorithm has also been investigated to extract object features that are obscured by scattered light. Results from controlled laboratory experiments show an improvement in the range resolution and accuracy of underwater imagery relative to data collected with a conventional short pulse system.