Multi-scale Object Research Articles

Leveraging Multiscale Adaptive Object Detection and Contrastive Feature Learning for Customer Behavior Analysis in Retail Settings

Multiscale adaptive object detection is a powerful computer vision technique that holds great potential for customer behavior analysis in various domains. By accurately detecting and tracking objects of interest, such as customers or products, at different scales, this approach enables detailed analysis of customer behavior. It allows businesses to track customer movements, interactions with products, and dwell times, providing valuable insights into shopping patterns and preferences. The application of multiscale adaptive object detection in customer behavior analysis offers businesses the opportunity to optimize store layouts, product placements, and marketing strategies, leading to enhanced customer experiences and improved business performance. In this paper, we introduce an innovative technique for object detection that leverages contrastive feature learning to augment the efficacy of multiscale object detection. Our methodology incorporates a contrastive loss function to extract discriminative features that exhibit resilience to scale and perspective disparities. This empowers our model to precisely detect objects across a broad range of sizes and viewpoints, even in arduous scenarios encompassing partial occlusion or low contrast against the background. Through comprehensive experiments conducted on benchmark datasets, we demonstrate that our approach surpasses state-of-the-art methodologies in terms of both accuracy and efficiency.

A Traffic Parameter Extraction Model Using Small Vehicle Detection and Tracking in Low-Brightness Aerial Images

It is still a challenge to detect small-size vehicles from a drone perspective, particularly under low-brightness conditions. In this context, a YOLOX-IM-DeepSort model was proposed, which improved the object detection performance in low-brightness conditions accurately and efficiently. At the stage of object detection, this model incorporates the data enhancement algorithm as well as an ultra-lightweight subspace attention module, and optimizes the number of detection heads and the loss function. Then, the ablation experiment was conducted and the analysis results showed that the YOLOX-IM model has better mAP than the baseline model YOLOX-s for multi-scale object detection. At the stage of object tracking, the DeepSort object-tracking algorithm is connected to the YOLOX-IM model, which can extract vehicle classification data, vehicle trajectory, and vehicle speed. Then, the VisDrone2021 dataset was adopted to verify the object-detection and tracking performance of the proposed model, and comparison experiment results showed that the average vehicle detection accuracy is 85.00% and the average vehicle tracking accuracy is 71.30% at various brightness levels, both of which are better than those of CenterNet, YOLOv3, FasterR-CNN, and CascadeR-CNN. Next, a field experiment using an in-vehicle global navigation satellite system and a DJI Phantom 4 RTK drone was conducted in Tianjin, China, and 12 control experimental scenarios with different drone flight heights and vehicle speeds were designed to analyze the effect of drone flight altitude on speed extraction accuracy. Finally, the conclusions and discussions were presented.

STO2Vec: A Multiscale Spatio-Temporal Object Representation Method for Association Analysis

Spatio-temporal association analysis has attracted attention in various fields, such as urban computing and crime analysis. The proliferation of positioning technology and location-based services has facilitated the expansion of association analysis across spatio-temporal scales. However, existing methods inadequately consider the scale differences among spatio-temporal objects during analysis, leading to suboptimal precision in association analysis results. To remedy this issue, we propose a multiscale spatio-temporal object representation method, STO2Vec, for association analysis. This method comprises of two parts: graph construction and embedding. For graph construction, we introduce an adaptive hierarchical discretization method to distinguish the varying scales of local features. Then, we merge the embedding method for spatio-temporal objects with that for discrete units, establishing a heterogeneous graph. For embedding, to enhance embedding quality for homogeneous and heterogeneous data, we use biased sampling and unsupervised models to capture the association strengths between spatio-temporal objects. Empirical results using real-world open-source datasets show that STO2Vec outperforms other models, improving accuracy by 16.25% on average across diverse applications. Further case studies indicate STO2Vec effectively detects association relationships between spatio-temporal objects in a range of scenarios and is applicable to tasks such as moving object behavior pattern mining and trajectory semantic annotation.

DMS-YOLOv5: A Decoupled Multi-Scale YOLOv5 Method for Small Object Detection

Small objects detection is a challenging task in computer vision due to the limited semantic information that can be extracted and the susceptibility to background interference. In this paper, we propose a decoupled multi-scale small object detection algorithm named DMS-YOLOv5. The algorithm incorporates a receptive field module into the feature extraction network for better focus on low-resolution small objects. The coordinate attention mechanism, which combines spatial and channel attention information, is introduced to reduce interference from background information and enhance the network’s attention to object information. A detection layer tailored to small-sized objects is added to compensate for the loss of small object information in multiple downsampling operations, greatly improving the detection capability of small objects. Next, The decoupled network is introduced into the detection head network for branch processing of classification and bounding box regression tasks. Finally, the bounding box loss function is improved to alleviate missed detection problems caused by the concentration of small objects and mutual occlusion between objects. The improved method achieved a mean average precision improvement of 12.1% on VisDrone2019-DET dataset compared to the original method. In comparison experiments with similar methods, our proposed method also demonstrated good performance, validating its effectiveness.

Multi-Head-Self-Attention based YOLOv5X-transformer for multi-scale object detection

Multi-Head-Self-Attention based YOLOv5X-transformer for multi-scale object detection

Estimation of Bamboo Forest Aboveground Carbon Using the RGLM Model Based on Object-Based Multiscale Segmentation of SPOT-6 Imagery

Remote sensing is an important tool for the quantitative estimation of forest carbon stock. This study presents a multiscale, object-based method for the estimation of aboveground carbon stock in Moso bamboo forests. The method differs from conventional pixel-based approaches and is more suitable for Chinese forest management inventory. This research indicates that the construction of a SPOT-6 multiscale hierarchy with the 30 scale as the optimal segmentation scale achieves accurate information extraction for Moso bamboo forests. The producer’s and user’s accuracy are 88.89% and 86.96%, respectively. A random generalized linear model (RGLM), constructed using the multiscale hierarchy, can accurately estimate carbon storage of the bamboo forest in the study area, with a fitting and test accuracy (R2) of 0.74 and 0.64, respectively. In contrast, pixel-based methods using the RGLM model have a fitting and prediction accuracy of 0.24 and 0.01, respectively; thus, the object-based RGLM is a major improvement. The multiscale object hierarchy correctly analyzed the multiscale correlation and responses of bamboo forest elements to carbon storage. Objects at the 30 scale responded to the microstructure of the bamboo forest and had the strongest correlation between estimated carbon storage and measured values. Objects at the 60 scale did not directly inherit the forest information, so the response to the measured carbon storage of the bamboo forest was the smallest. Objects at the 90 scale serve as super-objects containing the forest feature information and have a significant correlation with the measured carbon storage. Therefore, in this study, a carbon storage estimation model was constructed based on the multiscale characteristics of the bamboo forest so as to analyze correlations and greatly improve the fitting and prediction accuracy of carbon storage.

IDD-Net: Industrial defect detection method based on Deep-Learning

Detecting defects in industrial products is one of the most widespread applications of industrial automation. Various product defects, large similarities, and drastic changes in scale in industrial scenarios pose challenges to existing industrial inspection networks. This paper proposes a deep learning-based industrial defect detection method (IDD-Net) to address the above challenges. Specifically, IDD-Net has three distinct features. First, for the defects of diversity and similarity (rolled-in_scale, crazing in steel defects), IDD-Net designed a novel local–global backbone feature network (LGB-Net). Second, IDD-Net proposes a novel Three-Layer Feature Aggregation network (TFLA-Net) to solve the problem of drastic scale changes. TFLA-Net adopts a novel three-layer descending method to aggregate semantic and fine-grained features effectively. At the same time, the dense connection of adjacent nodes of TFLA-Net ensures the efficient fusion of features of different scales in the network. In particular, this paper proposes a novel IoU loss (Defect-IoU loss) for the problem of object loss deviation at different scales. The novelty of Defect-IoU Loss is that the loss value is scaled by the difference in the area of different scale objects, which is more conducive to the balance of multi-scale object loss. The experimental results show that the calculation amount of IDD-Net is only 24.9 Gflops, and the mAP@.5 of 79.66%, 99.5%, and 95.9% in the steel defect, aluminium defect, and PCB defect datasets were respectively obtained, surpassing all comparison models. In addition, the test in the actual industrial scene also demonstrates the feasibility of the application of IDD-Net.

Multi-Modal and Multi-Scale Fusion 3D Object Detection of 4D Radar and LiDAR for Autonomous Driving

Multi-modal fusion plays a critical role in 3D object detection, overcoming the inherent limitations of single-sensor perception in autonomous driving. Most fusion methods require data from high-resolution cameras and LiDAR sensors, which are less robust and the detection accuracy drops drastically with the increase of range as the point cloud density decreases. Alternatively, the fusion of Radar and LiDAR alleviates these issues but is still a developing field, especially for 4D Radar with a more robust and broader detection range. Nevertheless, different data characteristics and noise distributions between two sensors hinder performance improvement when directly integrating them. Therefore, we are the first to propose a novel fusion method termed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$M^{2}$</tex-math></inline-formula> -Fusion for 4D Radar and LiDAR, based on Multi-modal and Multi-scale fusion. To better integrate two sensors, we propose an Interaction-based Multi-Modal Fusion (IMMF) method utilizing a self-attention mechanism to learn features from each modality and exchange intermediate layer information. Specific to the current single-resolution voxel division's precision and efficiency balance problem, we also put forward a Center-based Multi-Scale Fusion (CMSF) method to first regress the center points of objects and then extract features in multiple resolutions. Furthermore, we present a data preprocessing method based on Gaussian distribution that effectively decreases data noise to reduce errors caused by point cloud divergence of 4D Radar data in the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$x$</tex-math></inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$z$</tex-math></inline-formula> plane. To evaluate the proposed fusion method, a series of experiments were conducted using the Astyx HiRes 2019 dataset, including the calibrated 4D Radar and 16-line LiDAR data. The results demonstrated that our fusion method compared favorably with state-of-the-art algorithms. When compared to PointPillars, our method achieves mAP (mean average precision) increases of 5.64 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> and 13.57 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> for 3D and BEV (bird's eye view) detection of the car class at a moderate level, respectively.

ConvNeXt-based anchor-free object detection model for infrared image of power equipment

Infrared thermography has been widely used in the monitoring and diagnosis of power equipment to ensure the reliable functioning of power systems because of its non-contact and rapid capabilities. With the promotion of intelligent detection schemes in daily patrol inspection of power equipment by utilities, the amount of infrared images of power equipment (IIOPE) is growing geometrically. The research community has begun using deep learning technology to automatically detect objects in the IIOPE. However, the performance of existing detection methods is still not satisfactory because of the inherent attributes of power equipment infrared images, such as low contrast, complex background, and multi-scale object distribution. This research proposes an anchor-free detection model based on ConvNeXt to address these issues. An enhanced feature pyramid network (FPN) is presented to address the issue of multi-scale object distribution in the IIOPE. In addition, a dynamic soft label assignment strategy is proposed for precise object localization. Extensive experimental results on the self-built IIOPE dataset and the PASCAL VOC 2007 dataset demonstrate that the proposed method yields positive results. When compared to the current anchor-based as well as anchor-free infrared image detectors for power equipment, the proposed model offers improved accuracy while ensuring the detection speed that may satisfy the practical requirements.

Human object interaction detection based on feature optimization and key human-object enhancement

Aiming at the problem of unclear or missing human object interaction behavior objects in complex background, we propose a human object interaction detection algorithm based on feature optimization and key human-object enhancement. In order to solve the problem of missing human behavior objects, we propose Feature Optimized Faster Region Convolutional Neural Network (FOFR-CNN). FOFR-CNN is an object detection network optimized by multi-scale feature optimization algorithm, taking into account both image semantics and image structure. In order to reduce the interference of complex background, we propose a Key Human-Object Enhancement Network. The network uses an instance-based method to enhance the features of interactive objects. In order to enrich the interaction information, we use the graph convolutional network. Experimental results on HICO-DET, V-COCO and HOI-A datasets show that the proposed algorithm has significantly improved accuracy and multi-scale object detection ability compared with other human object interaction algorithms.

A Multi-Feature Fusion and Attention Network for Multi-Scale Object Detection in Remote Sensing Images

Accurate multi-scale object detection in remote sensing images poses a challenge due to the complexity of transferring deep features to shallow features among multi-scale objects. Therefore, this study developed a multi-feature fusion and attention network (MFANet) based on YOLOX. By reparameterizing the backbone, fusing multi-branch convolution and attention mechanisms, and optimizing the loss function, the MFANet strengthened the feature extraction of objects at different sizes and increased the detection accuracy. The ablation experiment was carried out on the NWPU VHR-10 dataset. Our results showed that the overall performance of the improved network was around 2.94% higher than the average performance of every single module. Based on the comparison experiments, the improved MFANet demonstrated a high mean average precision of 98.78% for 9 classes of objects in the NWPU VHR-10 10-class detection dataset and 94.91% for 11 classes in the DIOR 20-class detection dataset. Overall, MFANet achieved an mAP of 96.63% and 87.88% acting on the NWPU VHR-10 and DIOR datasets, respectively. This method can promote the development of multi-scale object detection in remote sensing images and has the potential to serve and expand intelligent system research in related fields such as object tracking, semantic segmentation, and scene understanding.

Multi-Scale Ship Detection Algorithm Based on YOLOv7 for Complex Scene SAR Images

Recently, deep learning techniques have been extensively used to detect ships in synthetic aperture radar (SAR) images. The majority of modern algorithms can achieve successful ship detection outcomes when working with multiple-scale ships on a large sea surface. However, there are still issues, such as missed detection and incorrect identification when performing multi-scale ship object detection operations in SAR images of complex scenes. To solve these problems, this paper proposes a complex scenes multi-scale ship detection model, according to YOLOv7, called CSD-YOLO. First, this paper suggests an SAS-FPN module that combines atrous spatial pyramid pooling and shuffle attention, allowing the model to focus on important information and ignore irrelevant information, reduce the feature loss of small ships, and simultaneously fuse the feature maps of ship targets on various SAR image scales, thereby improving detection accuracy and the model’s capacity to detect objects at several scales. The model’s optimization is then improved with the aid of the SIoU loss function. Finally, thorough tests on the HRSID and SSDD datasets are presented to support our methodology. CSD-YOLO achieves better detection performance than the baseline YOLOv7, with a 98.01% detection accuracy, a 96.18% recall, and a mean average precision (mAP) of 98.60% on SSDD. In addition, in comparative experiments with other deep learning-based methods, in terms of overall performance, CSD-YOLO still performs better.

CF-YOLOX: An Autonomous Driving Detection Model for Multi-Scale Object Detection.

In self-driving cars, object detection algorithms are becoming increasingly important, and the accurate and fast recognition of objects is critical to realize autonomous driving. The existing detection algorithms are not ideal for the detection of small objects. This paper proposes a YOLOX-based network model for multi-scale object detection tasks in complex scenes. This method adds a CBAM-G module to the backbone of the original network, which performs grouping operations on CBAM. It changes the height and width of the convolution kernel of the spatial attention module to 7 × 1 to improve the ability of the model to extract prominent features. We proposed an object-contextual feature fusion module, which can provide more semantic information and improve the perception of multi-scale objects. Finally, we considered the problem of fewer samples and less loss of small objects and introduced a scaling factor that could increase the loss of small objects to improve the detection ability of small objects. We validated the effectiveness of the proposed method on the KITTI dataset, and the mAP value was 2.46% higher than the original model. Experimental comparisons showed that our model achieved superior detection performance compared to other models.

Residual attention mechanism and weighted feature fusion for multi-scale object detection

Residual attention mechanism and weighted feature fusion for multi-scale object detection

DenseSPH-YOLOv5: An automated damage detection model based on DenseNet and Swin-Transformer prediction head-enabled YOLOv5 with attention mechanism

Objective.Computer vision-based up-to-date accurate damage classification and localization are of decisive importance for infrastructure monitoring, safety, and the serviceability of civil infrastructure. Current state-of-the-art deep learning (DL)-based damage detection models, however, often lack superior feature extraction capability in complex and noisy environments, limiting the development of accurate and reliable object distinction. Method.To this end, we present DenseSPH-YOLOv5, a real-time DL-based high-performance damage detection model where DenseNet blocks have been integrated with the backbone to improve in preserving and reusing critical feature information. Additionally, convolutional block attention modules (CBAM) have been implemented to improve attention performance mechanisms for strong and discriminating deep spatial feature extraction that results in superior detection under various challenging environments. Moreover, an additional feature fusion layers and a Swin-Transformer Prediction Head (SPH) have been added leveraging advanced self-attention mechanism for more efficient detection of multiscale object sizes and simultaneously reducing the computational complexity. Results.Evaluating the model performance in large-scale Road Damage Dataset (RDD-2018), at a detection rate of 62.4 FPS, DenseSPH-YOLOv5 obtains a mean average precision (mAP) value of 85.25%, F1-score of 81.18%, and precision (P) value of 89.51% outperforming current state-of-the-art models. Significance.The present research provides an effective and efficient damage localization model addressing the shortcoming of existing DL-based damage detection models by providing highly accurate localized bounding box prediction. Current work constitutes a step towards an accurate and robust automated damage detection system in real-time in-field applications.

Field-matching attention network for object detection

Feature pyramid network (FPN) is widely used in object detection in order to divide and conquer objects of different scales and to fuse high and low-level features, and it has achieved encouraging achievements in multi-scale object processing. However, due to the mismatch between receptive fields at different stages, the direct fusion of the two features from different receptive fields may be unable to achieve satisfactory results. Moreover, simple lateral connections in FPN may lead to loss of spatial relationships and details. To alleviate these problems, in this paper we propose a field-matching attention network (FMANet) for object detection. Particularly, we first propose a receptive field dilated module (RFDM), which is used to normalize receptive fields between features at different stages to the same scale. Furthermore, to capture the spatial informations and details, we build a dual attention module (DAM) by employing the spatial attention and channel attention. Utilizing both spatial and channel attention mechanisms simultaneously improves performance while maintaining speed. Finally, experimental results reveal that our proposed FMANet with DSPDarkNet-53 as backbone achieves a competitive detection performance.

FFR-SSD: feature fusion and reconstruction single shot detector for multi-scale object detection

Object detection is one of the most fundamental tasks toward image content understanding. Although numerous algorithms have been proposed, implementing effective and efficient object detection is still very challenging for now. To solve the challenges of small and multi-scale objects, we propose a hierarchical feature fusion and reconstruction method called feature fusion and reconstruction single shot detector (FFR-SSD). We first present a multi-scale visual attention model, which incorporates the channel-wise and space-wise information into the multi-branch feature enhancement module to improve the feature representation capacities. Second, a hierarchical feature map weighing mechanism is developed to fuse multi-layer feature maps, which contributes to describing the intact objects for the subsequent module. Third, we present an effective feature map reconstruction module to encourage the model to focus on pivotal information. Therefore, the overall contour information is preserved for the shallow enhanced response map and the semantic feature information is enriched for the deep enhanced feature map. Numerous experiments on two public benchmark datasets show that the proposed method achieves significant improvements over the state of the arts.

MS-YOLOv7:YOLOv7 Based on Multi-Scale for Object Detection on UAV Aerial Photography

A multi-scale UAV aerial image object detection model MS-YOLOv7 based on YOLOv7 was proposed to address the issues of a large number of objects and a high proportion of small objects that commonly exist in the Unmanned Aerial Vehicle (UAV) aerial image. The new network is developed with a multiple detection head and a CBAM convolutional attention module to extract features at different scales. To solve the problem of high-density object detection, a YOLOv7 network architecture combined with the Swin Transformer units is proposed, and a new pyramidal pooling module, SPPFS is incorporated into the network. Finally, we incorporate the SoftNMS and the Mish activation function to improve the network’s ability to identify overlapping and occlusion objects. Various experiments on the open-source dataset VisDrone2019 reveal that our new model brings a significant performance boost compared to other state-of-the-art (SOTA) models. Compared with the YOLOv7 object detection algorithm of the baseline network, the mAP0.5 of MS-YOLOv7 increased by 6.0%, the mAP0.95 increased by 4.9%. Ablation experiments show that the designed modules can improve detection accuracy and visually display the detection effect in different scenarios. This experiment demonstrates the applicability of the MS-YOLOv7 for UAV aerial photograph object detection.

Context-Driven Feature-Focusing Network for Semantic Segmentation of High-Resolution Remote Sensing Images

High-resolution remote sensing images (HRRSIs) cover a broad range of geographic regions and contain a wide variety of artificial objects and natural elements at various scales that comprise different image contexts. In semantic segmentation tasks based on deep convolutional neural networks (DCNNs), different resolution features are not equally effective for extracting ground objects with different scales. In this article, we propose a novel context-driven feature-focusing network (CFFNet) aimed at focusing on the multi-scale ground object in fused features of different resolutions. The CFFNet consists of three components: a depth-residual encoder, a context-driven feature-focusing (CFF) decoder, and a classifier. First, features with different resolutions are extracted using the depth-residual encoder to construct a feature pyramid. The multi-scale information in the fused features is then extracted using the feature-focusing (FF) module in the CFF decoder, followed by computing the focus weights of different scale features adaptively using the context-focusing (CF) module to obtain the weighted multi-scale fused feature representation. Finally, the final results are obtained using the classifier. The experiments are conducted on the public LoveDA and GID datasets. Quantitative and qualitative analyses of state-of-the-art (SOTA) segmentation benchmarks demonstrate the rationality and effectiveness of the proposed approach.

A Multi-Scale Traffic Object Detection Algorithm for Road Scenes Based on Improved YOLOv5

Object detection in road scenes is a task that has recently become popular and it is also an important part of intelligent transportation systems. Due to the different locations of cameras in the road scenes, the size of the traffic objects captured varies greatly, which imposes a burden on the network optimization. In addition, in some dense traffic scenes, the size of the traffic objects captured is extremely small and it is easy to miss detection and to encounter false detection. In this paper, we propose an improved multi-scale YOLOv5s algorithm based on the YOLOv5s algorithm. In detail, we add a detection head for extremely small objects to the original YOLOv5s model, which significantly improves the accuracy in detecting extremely small traffic objects. A content-aware reassembly of features (CARAFE) module is introduced in the feature fusion part to enhance the feature fusion. A new SPD-Conv CNN Module is introduced instead of the original convolutional structure to enhance the overall computational efficiency of the model. Finally, the normalization-based attention module (NAM) is introduced, allowing the model to focus on more useful information during training and significantly improving detection accuracy. The experimental results demonstrate that compared with the original YOLOv5s algorithm, the detection accuracy of the multi-scale YOLOv5s model proposed in this paper is improved by 7.1% on the constructed diverse traffic scene datasets. The improved multi-scale YOLOv5s algorithm also maintains the highest detection accuracy among the current mainstream object detection algorithms and is superior in accomplishing the task of detecting traffic objects in complex road scenes.