Multi-scale Feature Extraction Research Articles

Multi-scale Lesion Feature Fusion and Location-Aware for Chest Multi-disease Detection.

Accurately identifying and locating lesions in chest X-rays has the potential to significantly enhance diagnostic efficiency, quality, and interpretability. However, current methods primarily focus on detecting of specific diseases in chest X-rays, disregarding the presence of multiple diseases in a single chest X-ray scan. Moreover, the diversity in lesion locations and attributes introduces complexity in accurately discerning specific traits for each lesion, leading to diminished accuracy when detecting multiple diseases. To address these issues, we propose a novel detection framework that enhances multi-scale lesion feature extraction and fusion, improving lesion position perception and subsequently boosting chest multi-disease detection performance. Initially, we construct a multi-scale lesion feature extraction network to tackle the uniqueness of various lesion features and locations, strengthening the global semantic correlation between lesion features and their positions. Following this, we introduce an instance-aware semantic enhancement network that dynamically amalgamates instance-specific features with high-level semantic representations across various scales. This adaptive integration effectively mitigates the loss of detailed information within lesion regions. Additionally, we perform lesion region feature mapping using candidate boxes to preserve crucial positional information, enhancing the accuracy of chest disease detection across multiple scales. Experimental results on the VinDr-CXR dataset reveal a 6% increment in mean average precision (mAP) and an 8.4% improvement in mean recall (mR) when compared to state-of-the-art baselines. This demonstrates the effectiveness of the model in accurately detecting multiple chest diseases by capturing specific features and location information.

Learning-Based Non-Intrusive Electric Load Monitoring for Smart Energy Management

State-of-the-art smart cities have been calling for economic but efficient energy management over a large-scale network, especially for the electric power system. It is a critical issue to monitor, analyze, and control electric loads of all users in the system. In this study, a non-intrusive load monitoring method was designed for smart power management using computer vision techniques popular in artificial intelligence. First of all, one-dimensional current signals are mapped onto two-dimensional color feature images using signal transforms (including the wavelet transform and discrete Fourier transform) and Gramian Angular Field (GAF) methods. Second, a deep neural network with multi-scale feature extraction and attention mechanism is proposed to recognize all electrical loads from the color feature images. Third, a cloud-based approach was designed for the non-intrusive monitoring of all users, thereby saving energy costs during power system control. Experimental results on both public and private datasets demonstrate that the method achieves superior performances compared to its peers, and thus supports efficient energy management over a large-scale Internet of Things network.

FocusDet: an efficient object detector for small object

The object scale of a small object scene changes greatly, and the object is easily disturbed by a complex background. Generic object detectors do not perform well on small object detection tasks. In this paper, we focus on small object detection based on FocusDet. FocusDet refers to the small object detector proposed in this paper. It consists of three parts: backbone, feature fusion structure, and detection head. STCF-EANet was used as the backbone for feature extraction, the Bottom Focus-PAN for feature fusion, and the detection head for object localization and recognition.To maintain sufficient global context information and extract multi-scale features, the STCF-EANet network backbone is used as the feature extraction network.PAN is a feature fusion module used in general object detectors. It is used to perform feature fusion on the extracted feature maps to supplement feature information.In the feature fusion network, FocusDet uses Bottom Focus-PAN to capture a wider range of locations and lower-level feature information of small objects.SIOU-SoftNMS is the proposed algorithm for removing redundant prediction boxes in the post-processing stage. SIOU multi-dimension accurately locates the prediction box, and SoftNMS uses the Gaussian algorithm to remove redundant prediction boxes. FocusDet uses SIOU-SoftNMS to address the missed detection problem common in dense tiny objects.The VisDrone2021-DET and CCTSDB2021 object detection datasets are used as benchmarks, and tests are carried out on VisDrone2021-det-test-dev and CCTSDB-val datasets. Experimental results show that FocusDet improves mAP@.5% from 33.6% to 46.7% on the VisDrone dataset. mAP@.5% on the CCTSDB2021 dataset is improved from 81.6% to 87.8%. It is shown that the model has good performance for small object detection, and the research is innovative.

A Multi-scale features-based cloud detection method for Suomi-NPP VIIRS day and night imagery

Abstract. Cloud detection is a necessary step before the application of remote sensing images. However, most methods focus on cloud detection in daytime remote sensing images. The ignored nighttime remote sensing images play more and more important role in many fields such as urban monitoring, population estimation and disaster assessment. The radiation intensity similarity between artificial lights and clouds is higher in nighttime remote sensing images than in daytime remote sensing images, which makes it difficult to distinguish artificial lights from clouds. Therefore, this paper proposes a deep learning-based method (MFFCD-Net) to detect clouds for day and nighttime remote sensing images. MFFCD-Net is designed based on the encoder-decoder structure. The encoder adopts Resnet-50 as the backbone network for better feature extraction, and a dilated residual up-sampling module (DR-UP) is designed in the decoder for up-sampling feature maps while enlarging the receptive field. A multi-scale feature extraction fusion module (MFEF) is designed to enhance the ability of the MFFCD-Net to distinguish regular textures of artificial lights and random textures of clouds. An Global Feature Recovery Fusion Module (GFRF Module) is designed to select and fuse the feature in the encoding stage and the feature in the decoding stage, thus to achieve better cloud detection accuracy. This is the first time that a deep learning-based method is designed for cloud detection both in day and nighttime remote sensing images. The experimental results on Suomi-NPP VIIRS DNB images show that MFFCD-Net achieves higher accuracy than baseline methods both in day and nighttime remote sensing images. Results on daytime remote sensing images indicate that MFFCD-Net can obtain better balance on commission and omission rates than baseline methods (92.3% versus 90.5% on F1-score). Although artificial lights introduced strong interference in cloud detection in nighttime remote sensing images, the accuracy values of MFFCD-Net on OA, Precision, Recall, and F1-score are still higher than 90%. This demonstrates that MFFCD-Net can better distinguish artificial lights from clouds than baseline methods in nighttime remote sensing images. The effectiveness of MFFCD-Net proves that it is very promising for cloud detection both in day and nighttime remote sensing images.

Smartphone-based straw incorporation: An improved convolutional neural network

In conservation tillage, the effective incorporation of straw into soil is a critical factor for optimizing decomposition rates and addressing challenges related to straw accumulation. Consequently, this study introduces an innovative straw semantic segmentation model to extract straw mixed with soil from images, and proposes a method to measure the quality of straw and soil mixing based on this model. We conducted field experiments based on this method, analyzed the quality of straw incorporation at different depths, and finally integrated the model into a mobile application to facilitate the automatic, accurate, and rapid assessment of the quality of straw and soil mixing. The methodology of this study as an improved convolutional neural network entailed the fusion of deep separable convolution with a multi-scale feature extraction module, utilizing convolution kernels of varying sizes to capture the diverse features of straw. Further refinement included halving the first convolution channel and implementing a direct connection structure. Attention gates were strategically deployed to enhance salient features, resulting in superior accuracy compared to the original U-shaped network, fully convolutional network, and Otsu algorithm. Notably, the proposed model achieved enhanced performance while reducing the number of parameters and model complexity to one-third and one-half of the original U-shaped network, respectively. For a quantitative description of the rate and uniformity of straw incorporation, this study employed image processing technology and a grid counting method. The synergistic integration of smartphone imagery and deep learning expedited the delivery of rapid and reliable results, promising practical applicability in future straw incorporation practices. A demonstration video of the application is accessible at: https://doi.org/10.6084/m9.figshare.14480745.v3. The findings of this research contribute significantly to the advancement of conservation tillage by establishing a robust framework for the assessment and enhancement of straw incorporation operations’ efficiency.

Micro-expression recognition using a multi-scale feature extraction network with attention mechanisms

Micro-expression recognition using a multi-scale feature extraction network with attention mechanisms

EEG classification model for virtual reality motion sickness based on multi-scale CNN feature correlation

BackgroundVirtual reality motion sickness (VRMS) is a key issue hindering the development of virtual reality technology, and accurate detection of its occurrence is the first prerequisite for solving the issue. ObjectiveIn this paper, a convolutional neural network (CNN) EEG detection model based on multi-scale feature correlation is proposed for detecting VRMS. MethodsThe model uses multi-scale 1D convolutional layers to extract multi-scale temporal features from the multi-lead EEG data, and then calculates the feature correlations of the extracted multi-scale features among all the leads to form the feature adjacent matrixes, which converts the time-domain features to correlation-based brain network features, thus strengthen the feature representation. Finally, the correlation features of each layer are fused. The fused features are then fed into the channel attention module to filter the channels and classify them using a fully connected network. Finally, we recruit subjects to experience 6 different modes of virtual roller coaster scenes, and collect resting EEG data before and after the task to verify the model. Results: The results show that the accuracy, precision, recall and F1-score of this model for the recognition of VRMS are 98.66 %, 98.65 %, 98.68 %, and 98.66 %, respectively. The proposed model outperforms the current classic and advanced EEG recognition models. SignificanceIt shows that this model can be used for the recognition of VRMS based on the resting state EEG.

Implementation of hand‐object pose estimation using SSD and YOLOV5 model for object grasping by SCARA robot

AbstractEnforcement of advanced deep learning methods in hand‐object pose estimation is an imperative method for grasping the objects safely during the human–robot collaborative tasks. The position and orientation of a hand‐object from a two‐dimensional image is still a crucial problem under various circumstances like occlusion, critical lighting, and salient region detection and blur images. In this paper, the proposed method uses an enhanced MobileNetV3 with single shot detection (SSD) and YOLOv5 to ensure the improvement in accuracy and without compromising the latency in the detection of hand‐object pose and its orientation. To overcome the limitations of higher computation cost, latency and accuracy, the Network Architecture Search and NetAdapt Algorithm is used in MobileNetV3 that perform the network search for parameter tuning and adaptive learning for multiscale feature extraction and anchor box offset adjustment due to auto‐variance of weight in the level of each layers. The squeeze‐and‐excitation block reduces the computation and latency of the model. Hard‐swish activation function and feature pyramid networks are used to prevent over fitting the data and stabilizing the training. Based on the comparative analysis of MobileNetV3 with its predecessor and YOLOV5 are carried out, the obtained results are 92.8% and 89.7% of precision value, recall value of 93.1% and 90.2%, mAP value of 93.3% and 89.2%, respectively. The proposed methods ensure better grasping for robots by providing the pose estimation and orientation of hand‐objects with tolerance of −1.9 to 2.15 mm along x, −1.55 to 2.21 mm along y, −0.833 to 1.51 mm along z axis and −0.233° to 0.273° along z‐axis.

Dual-path multi-scale attention residual network for fault diagnosis of rolling bearings under complex operating conditions

Rolling bearing faults inevitably occur during the long-term continuous operation of rotating machinery. Therefore, fault diagnosis is greatly important for ensuring the normal and safe operation of rolling bearings. However, the complexity and diversity of working conditions of rolling bearings present a significant challenge in extracting fault characteristics accurately, which further affects the ultimate fault diagnosis results. In this article, we propose a new model, called dual-path multi-scale attention residual network (DPMARN), for diagnosing bearing faults under complex operating conditions. DPMARN can effectively capture the feature-feature correlation information at different scales, which is more beneficial for fusing fault features at different scales to improve the model’s performance. The main contributions of this work are summarized as follows: (1) the designed dual-path network model which incorporates parallel multi-scale branches of convolutional kernels and serially connects skip-layer multi-scale branches can integrate both low-frequency and high-frequency information and enhance the multi-scale feature extraction and complex data representation abilities. (2) The squeeze-and-excitation attention mechanism is embedded into the residual blocks to improve the ability of learning feature correlations and utilizing feature information effectively, which is helpful for extracting important fault characteristics. Extensive experiments conducted on two public bearing datasets demonstrate the superior performance of the DPMARN model for addressing the complex fault diagnosis problem. These results indicate that our proposed approach provides an effective solution for fault diagnosis of rolling bearings under complex operating conditions.

Swin Transformer and the Unet Architecture to Correct Motion Artifacts in Magnetic Resonance Image Reconstruction.

We present a deep learning-based method that corrects motion artifacts and thus accelerates data acquisition and reconstruction of magnetic resonance images. The novel model, the Motion Artifact Correction by Swin Network (MACS-Net), uses a Swin transformer layer as the fundamental block and the Unet architecture as the neural network backbone. We employ a hierarchical transformer with shifted windows to extract multiscale contextual features during encoding. A new dual upsampling technique is employed to enhance the spatial resolutions of feature maps in the Swin transformer-based decoder layer. A raw magnetic resonance imaging dataset is used for network training and testing; the data contain various motion artifacts with ground truth images of the same subjects. The results were compared to six state-of-the-art MRI image motion correction methods using two types of motions. When motions were brief (within 5 s), the method reduced the average normalized root mean square error (NRMSE) from 45.25% to 17.51%, increased the mean structural similarity index measure (SSIM) from 79.43% to 91.72%, and increased the peak signal-to-noise ratio (PSNR) from 18.24 to 26.57 dB. Similarly, when motions were extended from 5 to 10 s, our approach decreased the average NRMSE from 60.30% to 21.04%, improved the mean SSIM from 33.86% to 90.33%, and increased the PSNR from 15.64 to 24.99 dB. The anatomical structures of the corrected images and the motion-free brain data were similar.

SCFNet: Lightweight Steel Defect Detection Network Based on Spatial Channel Reorganization and Weighted Jump Fusion

The goal of steel defect detection is to enhance the recognition accuracy and accelerate the detection speed with fewer parameters. However, challenges arise in steel sample detection due to issues such as feature ambiguity, low contrast, and similarity among inter-class features. Moreover, limited computing capability makes it difficult for small and medium-sized enterprises to deploy and utilize networks effectively. Therefore, we propose a novel lightweight steel detection network (SCFNet), which is based on spatial channel reconstruction and deep feature fusion. The network adopts a lightweight and efficient feature extraction module (LEM) for multi-scale feature extraction, enhancing the capability to extract blurry features. Simultaneously, we adopt spatial and channel reconstruction convolution (ScConv) to reconstruct the spatial and channel features of the feature maps, enhancing the spatial localization and semantic representation of defects. Additionally, we adopt the Weighted Bidirectional Feature Pyramid Network (BiFPN) for defect feature fusion, thereby enhancing the capability of the model in detecting low-contrast defects. Finally, we discuss the impact of different data augmentation methods on the model accuracy. Extensive experiments are conducted on the NEU-DET dataset, resulting in a final model achieving an mAP of 81.2%. Remarkably, this model only required 2.01 M parameters and 5.9 GFLOPs of computation. Compared to state-of-the-art object detection algorithms, our approach achieves a higher detection accuracy while requiring fewer computational resources, effectively balancing the model size and detection accuracy.

SCPMan: Shape context and prior constrained multi-scale attention network for pancreatic segmentation

Due to the poor prognosis of Pancreatic cancer, accurate early detection and segmentation are critical for improving treatment outcomes. However, pancreatic segmentation is challenged by blurred boundaries, high shape variability, and class imbalance. To tackle these problems, we propose a multiscale attention network with shape context and prior constraint for robust pancreas segmentation. Specifically, we proposed a Multi-scale Feature Extraction Module (MFE) and a Mixed-scale Attention Integration Module (MAI) to address unclear pancreas boundaries. Furthermore, a Shape Context Memory (SCM) module is introduced to jointly model semantics across scales and pancreatic shape. Active Shape Model (ASM) is further used to model the shape priors. Experiments on NIH and MSD datasets demonstrate the efficiency of our model, which improves the state-of-the-art Dice Score for 1.01% and 1.03% respectively. Our architecture provides robust segmentation performance, against the blurry boundaries, and variations in scale and shape of pancreas.

DFSA-DAN: dynamic fusion of statistical metric and adversarial learning for domain adaptation network based intelligent fault diagnosis

The advancement of deep transfer learning has motivated research into the realization of intelligent fault diagnosis schemes for rolling bearing. Nevertheless, existing research rarely provides further insight into the importance of statistical distance metric-based methods and adversarial learning-based methods in domain adaptation, and the commonly used feature extractors are more difficult to extract features suitable for domain transformation. In this paper, a dynamic fusion of statistical metric and adversarial learning for domain adaptation network is proposed to achieve a dynamic measure of the importance of different domain adaptation methods. This new model utilizes a local maximum mean discrepancy metric to adjust the conditional distribution and adversarial training to adjust the marginal distribution between domains. Meanwhile, to assess the importance of the two distributions, a dynamic adaptation factor is introduced for dynamic evaluation. In addition, to extract features that are more suitable for domain transformation, the model incorporates a dual depth convolutional path with an attention mechanism as a feature extractor, enabling multi-scale feature extraction. Experimental results demonstrate the model’s superior generalization capability and robustness, enabling effective cross-domain fault diagnosis in diverse scenarios.

A multi-step loss meta-learning method based on multi-scale feature extraction for few-shot fault diagnosis

Existing deep learning (DL) algorithms are based on a large amount of training data and they face challenges in effectively extracting fault features when dealing with few-shot fault diagnoses. Model-agnostic meta-learning (MAML) also faces some challenges, including the limited capability of the basic convolutional neural network (CNN) with a single convolutional kernel to extract fault features comprehensively, as well as the instability of model training due to the inner and outer double-layer loops. To address these issues, this paper presents a multi-step loss meta-learning method based on multi-scale feature extraction (MFEML). Firstly, an improved multi-scale feature extraction module (IMFEM) is designed to solve the problem of the insufficient feature extraction capability of the CNN. Secondly, the multi-step loss is used to reconstruct the meta-loss to address the issue of MAML training instability. Finally, the experimental results of two datasets demonstrate the effectiveness of the MFEML.

Image fusion using Y-net-based extractor and global-local discriminator

Although some deep learning-based image fusion approaches have realized promising results, how to extract information-rich features from different source images while preserving them in the fused image with less distortions remains challenging issue that needs to be addressed. Here, we propose a well worked-out GAN-based scheme with multi-scale feature extractor and global-local discriminator for infrared and visible image fusion. We use Y-Net as the backbone architecture to design the generator network, and introduce the residual dense block (RDblock) to yield more realistic fused images for infrared and visible images by learning discriminative multi-scale representations that are closer to the essence of different modal images. During feature reconstruction, the cross-modality shortcuts with contextual attention (CMSCA) are employed to selectively aggregate features at different scales and different levels to construct information-rich fused images with better visual effect. To ameliorate the information content of the fused image, we not only constrain the structure and contrast information using structural similarity index, but also evaluate the intensity and gradient similarities at both feature and image levels. Two global-local discriminators that combine global GAN with PatchGAN as a unified architecture help to dig for finer differences between the generated image and reference images, which force the generator to learn both the local radiation information and pervasive global details in two source images. It is worth mentioning that image fusion is achieved during confrontation without fusion rules. Lots of assessment tests demonstrate that the reported fusion scheme achieves superior performance against state-of-the-art works in meaningful information preservation.

A feature matching and compensation method based on importance weighting for occluded human pose estimation

Human pose estimation is a hot research in the field of computer vision. Existing algorithms perform well in normal scenarios but struggle in occlusion environments. In order to address occlusion problems, we firstly design a multi-scale feature reinforcement backbone (MFRI) to obtain accurate features, which contains three sub-modules. The first sub-module reinforces detailed keypoint features and eliminates redundant background, The second sub-module assigns different weights to different scale information based on importance. The fourth sub-module integrates different scale feature information using a cascaded method. In addition, we design an occlusion handling method. To begin with, we design a multi-scale feature extractor (MSFE) to construct human topological feature, and propose a feature similarity matching method between keypoint detail features and topological features to identify occluded keypoints and eliminate obstacle features. Based on similarity, we design a feature compensation method to extract common attributes among all keypoints and unique features of each keypoint from multiple aspects to compensate for occluded features. Finally, we propose an adjacency matrix improvement method to enhance its ability for describing node relationships. We conduct comparative experiments on the COCO2017 dataset, COCO-Wholebody dataset, and CrowdPose dataset, achieving accuracy of 78.8%, 66.3%, and 77.8%, respectively. Additionally, we design a series of ablation experiments and visualizations. The results show that our method has better performance in handling occlusions.

Improved object detection method for unmanned driving based on Transformers.

The object detection method serves as the core technology within the unmanned driving perception module, extensively employed for detecting vehicles, pedestrians, traffic signs, and various objects. However, existing object detection methods still encounter three challenges in intricate unmanned driving scenarios: unsatisfactory performance in multi-scale object detection, inadequate accuracy in detecting small objects, and occurrences of false positives and missed detections in densely occluded environments. Therefore, this study proposes an improved object detection method for unmanned driving, leveraging Transformer architecture to address these challenges. First, a multi-scale Transformer feature extraction method integrated with channel attention is used to enhance the network's capability in extracting features across different scales. Second, a training method incorporating Query Denoising with Gaussian decay was employed to enhance the network's proficiency in learning representations of small objects. Third, a hybrid matching method combining Optimal Transport and Hungarian algorithms was used to facilitate the matching process between predicted and actual values, thereby enriching the network with more informative positive sample features. Experimental evaluations conducted on datasets including KITTI demonstrate that the proposed method achieves 3% higher mean Average Precision (mAP) than that of the existing methodologies.

GMSN: An efficient multi-scale feature extraction network for gait recognition

GMSN: An efficient multi-scale feature extraction network for gait recognition

Modality specific infrared and visible image fusion based on multi-scale rich feature representation under low-light environment

Infrared (IR) and Visible (VIS) image fusion is a critical task in computer vision, which aims to produce high-quality fused images by combining two modal images with complementary information. However, current image fusion techniques neglect the severe degradation of texture and details in low-light environments and do not account for human visual perception. In this paper, a multi-scale image fusion algorithm adapted to the low-light environment is proposed. Firstly, the light distribution of the low-light image is estimated and adjusted based on the grey value probability distribution, resulting in an enhanced VIS image with rich details. Simultaneously, considering the imaging characteristics of the two modal images, a superpixel method is utilized to obtain a saliency map for the IR image, and a multi-scale feature extraction scheme is employed to establish a VIS detail map. Finally, we employ entropy to measure the amount of information in the saliency and detail maps to guide the fusion process. Experimental results from qualitative and quantitative evaluations demonstrate that the fused image conforms to human visual perception, maintains a uniform distribution of pixel intensity, and preserves rich texture features at the edge details.

MASSNet: Multiscale Attention for Single-Stage Ship Instance Segmentation

Maritime surveillance is essential in understanding, predicting, and ensuring the security of events in the complex marine environment. In this context, we have used instance segmentation techniques, which provides an accurate and efficient method for segmenting objects (Ships) in maritime surveillance. However, prevalent two-stage algorithms have limitations, including complex models, extended training time, and high memory consumption, making them impractical for real-world application. To address these challenges, we present an efficient solution called Multiscale Attention for Single-Stage Ship Instance Segmentation, or MASSNet. MASSNet uses the power of attention mechanisms to enhance multiscale feature extraction across various dimensions, resulting in a more refined and contextually-aware representation. This approach significantly improves segmentation accuracy and overall performance. In our extensive experiments, we evaluate the effectiveness of MASSNet on three challenging datasets: MariboatS, ShipInsSeg, and ShipSG. Our proposed model achieves mask Average Precision (mask AP) scores of 55.4%, 55.5%, and 74.1% on MariboatS, ShipInsSeg, and ShipSG datasets and outperforms other models such as YOLACT, SOLO and SOLOv2 architectures. MASSNet offers a robust and efficient solution for Ship Instance Segmentation, making significant improvement in the capabilities of maritime surveillance.