Multi-scale Deep Features Research Articles

Enhanced Image Retrieval Using Multiscale Deep Feature Fusion in Supervised Hashing.

In recent years, deep-network-based hashing has gained prominence in image retrieval for its ability to generate compact and efficient binary representations. However, most existing methods predominantly focus on high-level semantic features extracted from the final layers of networks, often neglecting structural details that are crucial for capturing spatial relationships within images. Achieving a balance between preserving structural information and maximizing retrieval accuracy is the key to effective image hashing and retrieval. To address this challenge, we introduce Multiscale Deep Feature Fusion for Supervised Hashing (MDFF-SH), a novel approach that integrates multiscale feature fusion into the hashing process. The hallmark of MDFF-SH lies in its ability to combine low-level structural features with high-level semantic context, synthesizing robust and compact hash codes. By leveraging multiscale features from multiple convolutional layers, MDFF-SH ensures the preservation of fine-grained image details while maintaining global semantic integrity, achieving a harmonious balance that enhances retrieval precision and recall. Our approach demonstrated a superior performance on benchmark datasets, achieving significant gains in the Mean Average Precision (MAP) compared with the state-of-the-art methods: 9.5% on CIFAR-10, 5% on NUS-WIDE, and 11.5% on MS-COCO. These results highlight the effectiveness of MDFF-SH in bridging structural and semantic information, setting a new standard for high-precision image retrieval through multiscale feature fusion.

A Student Facial Expression Recognition Model Based on Multi-Scale and Deep Fine-Grained Feature Attention Enhancement.

In smart classroom environments, accurately recognizing students' facial expressions is crucial for teachers to efficiently assess students' learning states, timely adjust teaching strategies, and enhance teaching quality and effectiveness. In this paper, we propose a student facial expression recognition model based on multi-scale and deep fine-grained feature attention enhancement (SFER-MDFAE) to address the issues of inaccurate facial feature extraction and poor robustness of facial expression recognition in smart classroom scenarios. Firstly, we construct a novel multi-scale dual-pooling feature aggregation module to capture and fuse facial information at different scales, thereby obtaining a comprehensive representation of key facial features; secondly, we design a key region-oriented attention mechanism to focus more on the nuances of facial expressions, further enhancing the representation of multi-scale deep fine-grained feature; finally, the fusion of multi-scale and deep fine-grained attention-enhanced features is used to obtain richer and more accurate facial key information and realize accurate facial expression recognition. The experimental results demonstrate that the proposed SFER-MDFAE outperforms the existing state-of-the-art methods, achieving an accuracy of 76.18% on FER2013, 92.75% on FERPlus, 92.93% on RAF-DB, 67.86% on AffectNet, and 93.74% on the real smart classroom facial expression dataset (SCFED). These results validate the effectiveness of the proposed method.

Visible light infrared matching algorithm based on improved SuperPoint and linear converter

In order to solve the problems of high difficulty and mismatch rate in heterogeneous image matching of visible light and infrared images, a deep learning matching algorithm based on improved SuperPoint and linear transformer is proposed. The algorithm first introduces the idea of feature pyramid to construct a feature description branch based on the SuperPoint network structure, and trains it based on the hinge loss function, so as to better learn the multi-scale deep features of visible light and infrared images and increase the similarity of image point pair descriptors with the same name; in the feature matching module, the linear transformer is used to improve the SuperGlue matching algorithm and aggregate features to improve matching performance. The proposed algorithm is experimentally verified on multiple data sets. The results show that compared with the existing algorithms, the algorithm achieves better matching results and improves the matching accuracy.

CPINet: Towards A Novel Cross-Polarimetric Interaction Network for Dual-Polarized SAR Ship Classification

With the rapid development of the modern world, it is imperative to achieve effective and efficient monitoring for territories of interest, especially for the broad ocean area. For surveillance of ship targets at sea, a common and powerful approach is to take advantage of satellite synthetic aperture radar (SAR) systems. Currently, using satellite SAR images for ship classification is a challenging issue due to complex sea situations and the imaging variances of ships. Fortunately, the emergence of advanced satellite SAR sensors has shed much light on the SAR ship automatic target recognition (ATR) task, e.g., utilizing dual-polarization (dual-pol) information to boost the performance of SAR ship classification. Therefore, in this paper we have developed a novel cross-polarimetric interaction network (CPINet) to explore the abundant polarization information of dual-pol SAR images with the help of deep learning strategies, leading to an effective solution for high-performance ship classification. First, we establish a novel multiscale deep feature extraction framework to fully mine the characteristics of dual-pol SAR images in a coarse-to-fine manner. Second, to further leverage the complementary information of dual-pol SAR images, we propose a mixed-order squeeze–excitation (MO-SE) attention mechanism, in which the first- and second-order statistics of the deep features from one single-polarized SAR image are extracted to guide the learning of another polarized one. Then, the intermediate multiscale fused and MO-SE augmented dual-polarized deep feature maps are respectively aggregated by the factorized bilinear coding (FBC) pooling method. Meanwhile, the last multiscale fused deep feature maps for each single-polarized SAR image are also individually aggregated by the FBC. Finally, four kinds of highly discriminative deep representations are obtained for loss computation and category prediction. For better network training, the gradient normalization (GradNorm) method for multitask networks is extended to adaptively balance the contribution of each loss component. Extensive experiments on the three- and five-category dual-pol SAR ship classification dataset collected from the open and free OpenSARShip database demonstrate the superiority and robustness of CPINet compared with state-of-the-art methods for the dual-polarized SAR ship classification task.

A Multi-Scale Deep Learning Algorithm for Enhanced Forest Fire Danger Prediction Using Remote Sensing Images

Forest fire danger prediction models often face challenges due to spatial and temporal limitations, as well as a lack of universality caused by regional inconsistencies in fire danger features. To address these issues, we propose a novel algorithm, squeeze-excitation spatial multi-scale transformer learning (SESMTML), which is designed to extract multi-scale fire danger features from remote sensing images. SESMTML includes several key modules: the multi-scale deep feature extraction module (MSDFEM) captures global visual and multi-scale convolutional features, the multi-scale fire danger perception module (MFDPM) explores contextual relationships, the multi-scale information aggregation module (MIAM) aggregates correlations of multi-level fire danger features, and the fire danger level fusion module (FDLFM) integrates the contributions of global and multi-level features for predicting forest fire danger. Experimental results demonstrate the model’s significant superiority, achieving an accuracy of 83.18%, representing a 22.58% improvement over previous models and outperforming many widely used deep learning methods. Additionally, a detailed forest fire danger prediction map was generated using a test study area at the junction of the Miyun and Pinggu districts in Beijing, further confirming the model’s effectiveness. SESMTML shows strong potential for practical application in forest fire danger prediction and offers new insights for future research utilizing remote sensing images.

Intelligent mechanical fault diagnosis using multiscale residual network and multisensor fusion

Abstract Mechanical faults in manufacturing systems need to be diagnosed accurately to ensure safety and cost savings. With the development of sensor technologies, data from multiple sensors is frequently utilized to assess the health of intricate industrial systems. In such cases, it is necessary to study the multisensor data based intelligent mechanical fault diagnosis method. First, the multisensor data is converted into grey images and then fused into a three-channel red-green-blue (RGB) image. Then, a multiscale with residual convolution module is proposed, which can extract multiscale deep features of the complex raw signal. Additionally, an attention module for channel and spatial attention is introduced to adaptively adjust the feature response values of each scale. Two datasets and a specific engineering application are used to validate the superiority of the network. The results show that the multisensor multiscale residual network outperforms other fault diagnosis networks in terms of fault identification accuracy, diagnostic efficiency, and applicability.

Multi-Branch Attention Fusion Network for Cloud and Cloud Shadow Segmentation

In remote sensing image processing, the segmentation of clouds and their shadows is a fundamental and vital task. For cloud images, traditional deep learning methods often have weak generalization capabilities and are prone to interference from ground objects and noise, which not only results in poor boundary segmentation but also causes false and missed detections of small targets. To address these issues, we proposed a multi-branch attention fusion network (MAFNet). In the encoder section, the dual branches of ResNet50 and the Swin transformer extract features together. A multi-branch attention fusion module (MAFM) uses positional encoding to add position information. Additionally, multi-branch aggregation attention (MAA) in the MAFM fully fuses the same level of deep features extracted by ResNet50 and the Swin transformer, which enhances the boundary segmentation ability and small target detection capability. To address the challenge of detecting small cloud and shadow targets, an information deep aggregation module (IDAM) was introduced to perform multi-scale deep feature aggregation, which supplements high semantic information, improving small target detection. For the problem of rough segmentation boundaries, a recovery guided module (RGM) was designed in the decoder section, which enables the model to effectively allocate attention to complex boundary information, enhancing the network’s focus on boundary information. Experimental results on the Cloud and Cloud Shadow dataset, HRC-WHU dataset, and SPARCS dataset indicate that MAFNet surpasses existing advanced semantic segmentation techniques.

Identification of leaf diseases in field crops based on improved ShuffleNetV2.

Rapid and accurate identification and timely protection of crop disease is of great importance for ensuring crop yields. Aiming at the problems of large model parameters of existing crop disease recognition methods and low recognition accuracy in the complex background of the field, we propose a lightweight crop leaf disease recognition model based on improved ShuffleNetV2. First, the repetition number and the number of output channels of the basic module of the ShuffleNetV2 model are redesigned to reduce the model parameters to make the model more lightweight while ensuring the accuracy of the model. Second, the residual structure is introduced in the basic feature extraction module to solve the gradient vanishing problem and enable the model to learn more complex feature representations. Then, parallel paths were added to the mechanism of the efficient channel attention (ECA) module, and the weights of different paths were adaptively updated by learnable parameters, and then the efficient dual channel attention (EDCA) module was proposed, which was embedded into the ShuffleNetV2 to improve the cross-channel interaction capability of the model. Finally, a multi-scale shallow feature extraction module and a multi-scale deep feature extraction module were introduced to improve the model's ability to extract lesions at different scales. Based on the above improvements, a lightweight crop leaf disease recognition model REM-ShuffleNetV2 was proposed. Experiments results show that the accuracy and F1 score of the REM-ShuffleNetV2 model on the self-constructed field crop leaf disease dataset are 96.72% and 96.62%, which are 3.88% and 4.37% higher than that of the ShuffleNetV2 model; and the number of model parameters is 4.40M, which is 9.65% less than that of the original model. Compared with classic networks such as DenseNet121, EfficientNet, and MobileNetV3, the REM-ShuffleNetV2 model not only has higher recognition accuracy but also has fewer model parameters. The REM-ShuffleNetV2 model proposed in this study can achieve accurate identification of crop leaf disease in complex field backgrounds, and the model is small, which is convenient to deploy to the mobile end, and provides a reference for intelligent diagnosis of crop leaf disease.

Deep Supervised Hashing by Fusing Multiscale Deep Features for Image Retrieval

Deep network-based hashing has gained significant popularity in recent years, particularly in the field of image retrieval. However, most existing methods only focus on extracting semantic information from the final layer, disregarding valuable structural information that contains important semantic details, which are crucial for effective hash learning. On the one hand, structural information is important for capturing the spatial relationships between objects in an image. On the other hand, image retrieval tasks often require a more holistic representation of the image, which can be achieved by focusing on the semantic content. The trade-off between structural information and image retrieval accuracy in the context of image hashing and retrieval is a crucial consideration. Balancing these aspects is essential to ensure both accurate retrieval results and meaningful representation of the underlying image structure. To address this limitation and improve image retrieval accuracy, we propose a novel deep hashing method called Deep Supervised Hashing by Fusing Multiscale Deep Features (DSHFMDF). Our approach involves extracting multiscale features from multiple convolutional layers and fusing them to generate more robust representations for efficient image retrieval. The experimental results demonstrate that our method surpasses the performance of state-of-the-art hashing techniques, with absolute increases of 11.1% and 8.3% in Mean Average Precision (MAP) on the CIFAR-10 and NUS-WIDE datasets, respectively.

EMNet: Edge-guided multi-level network for salient object detection in low-light images

Salient object detection (SOD) has achieved remarkable performance in well-lit scenes. However, when generalized to low-light scenes, the performance of SOD shows significant decrease owing to more challenging conditions such as weak brightness, low contrast, and poor signal-to-noise ratio. To address this issue, we propose a novel edge-guided and multi-level network (EMNet) for SOD in low light images, which learns robust multiscale region features by optimizing the boundaries of salient objects and employing a multi-stage cascaded strategy. To be more specific, the proposed Edge Feature Highlight (EFH) module can establish mapping relationships at different scales and fuse the outputs obtained from pairs of different branches for extracting accurate boundary information. Secondly, Multi-layer Feature Fusion (MFF) module is proposed for combining multi-scale deep features with salient edge cues, using stepwise fusion for effective integration of deep features. Finally, we employ a coarse-to-fine way for iterative prediction to generate high-quality saliency maps. We conducted comprehensive experiments on LLI dataset, and the results demonstrate that the proposed method achieves the best performance in terms of five evaluation metrics, where an average improvement of max Fβ, ωFβ, Em, Sm, and MAE outperforms state-of-the-art method by 7.20%, 12.06%, 4.62%, 5.00%, and 36.34%, respectively.

Deep Multi-Dictionary Learning for Survival Prediction With Multi-Zoom Histopathological Whole Slide Images.

Survival prediction based on histopathological whole slide images (WSIs) is of great significance for risk-benefit assessment and clinical decision. However, complex microenvironments and heterogeneous tissue structures in WSIs bring challenges to learning informative prognosis-related representations. Additionally, previous studies mainly focus on modeling using mono-scale WSIs, which commonly ignore useful subtle differences existed in multi-zoom WSIs. To this end, we propose a deep multi-dictionary learning framework for cancer survival prediction with multi-zoom histopathological WSIs. The framework can recognize and learn discriminative clusters (i.e., microenvironments) based on multi-scale deep representations for survival analysis. Specifically, we learn multi-scale features based on multi-zoom tiles from WSIs via stacked deep autoencoders network followed by grouping different microenvironments by cluster algorithm. Based on multi-scale deep features of clusters, a multi-dictionary learning method with a post-pruning strategy is devised to learn discriminative representations from selected prognosis-related clusters in a task-driven manner. Finally, a survival model (i.e., EN-Cox) is constructed to estimate the risk index of an individual patient. The proposed model is evaluated on three datasets derived from The Cancer Genome Atlas (TCGA), and the experimental results demonstrate that it outperforms several state-of-the-art survival analysis approaches.

PDCA-FORMER: PRIOR-DIAGONAL CROSS ATTENTION-GUIDED TRANSFORMER FOR FLOOD MAPPING FROM SAR IMAGERY: A CASE IN KHARTOUM

Abstract. Floods are considered one of the most serious crises, with severe consequences such as loss of life, destruction of infrastructure, and economic disruption. In recent years, deep learning has gained popularity for fast and accurate flood mapping from synthetic aperture radar (SAR) for damage assessment and proactive mitigation. However, due to the complex characteristics of SAR images, accurate flood mapping remains challenging. In this study, we propose a novel Prior-Diagonal Cross Attention-guided transformer (PDCA-Former) network for flood mapping from SAR images. Specifically, PDCA-Former adopts Prior Siamese Feature Extraction (PSFE) to extract multi-scale deep features from the input SAR images. Additionally, we propose a novel Diagonal Cross-Attention Module (DCAM) to capture relational information of all pixel positions on the entire image. DCAM is integrated into the Transformer to acquire contextual tokens with spatio-temporal information from prior features, resulting in immersed maps. To investigate the potential of SAR images and the proposed PDCA-Former for effective flood detection and estimation of the extent of damaged farmland around the confluence of two rivers, this study chose the Sudanese city of Khartoum as an experimental study area. The experimental results show that PDCA-Former outperforms the latest comparator methods in terms of F1 by 88.9% and IoU by 85.7%. We conclude that PDCA-Former offers a promising solution for accurate and efficient flood mapping from SAR imagery that can be quickly generalized to other regions. Therefore, it can significantly aid disaster management efforts on vulnerable communities.

DSCA-Net: Double-stage Codec Attention Network for automatic nuclear segmentation

The rapid and precise segmentation of cell nuclei from hematoxylin and eosin-stained tissue images is an essential clinical undertaking with significant implications for various clinical applications. The segmentation of cell nuclei poses specific challenges due to the inherent instability of nuclear morphology and the complexity of the segmentation environments. Furthermore, previous studies have primarily relied on small-scale and limited-diverse datasets, potentially hindering their applicability to clinical tasks. This study introduces a novel approach, the Double-stage Codec Attention Network, designed to automatically and accurately segment nuclei. Specifically, we present a hierarchical feature extraction module, which maximizes the utilization of cell nuclei’s morphological characteristics in the tissue, thereby providing critical semantic information for nucleus segmentation. Furthermore, the feature selection units are employed to enhance relevant features and suppress interfering ones, thereby enhancing the overall expressive capacity of the information. The multi-scale deep feature fusion module utilizes interrelated encoder–decoder connections to jointly optimize and integrate this information, generating a robust hierarchical feature pyramid. Finally, the feature attention fusion mechanism captures spatial and directional information, aiding the model in the accurate localization and recognition of cell nuclei. We rigorously evaluated our proposed method using the PanNuke dataset, the largest comprehensive histology dataset of cancer tissues. In terms of the average F1-score across all segmentation classes in the PanNuke dataset, DSCA-Net outperforms other state-of-the-art models such as DeepLabV3+, TransUNet, Triple U-net, and TransNuSeg by 1.38, 1.44, 2.64, and 1.02, respectively. Additionally, DSCA-Net shows excellent efficiency in generating predictive images, outperforming all comparative models.

Cataract detection and visualization based on multi-scale deep features by RINet tuned with cyclic learning rate hyperparameter

The most common reason worldwide for blindness is cataract, the clouding of the lens where vision becomes increasingly blurry. Early detection of cataract is crucial for a better prognosis. This can be achieved with improved automated cataract detection and localization of the affected regions in the fundus images. This work proposes a supervised miniature U-Net (SMi-UNet) for feature extraction integrated with a convolutional neural network (RINet) explicitly designed for fundus images. The dataset used has been self-curated by collecting cataract images from multiple repositories.The proposed SMi-UNet allows a deeper network with significantly reduced parameters than conventional U-Net. The proposed RINet utilizes the features extracted by SMi-UNet for discriminating between a cataract and a normal fundus image. Further, the performance of the RINet is optimized using a cyclic learning rate (CLR) hyperparameter. CLR eliminates the need to find the best value of the learning rate and improves accuracy in a minimum number of epochs, making it suitable for edge devices. Further, to localize the prominent regions in the cataract images, colored heatmap techniques are applied at the last convolutional layer. These maps help visualize the affected areas with a hotter color.The exceptional performance of the proposed technique in cataract detection and its localizationhas been established by quantitative and qualitative data, demonstrating that it can be a valuabletool for early cataract detection. The obtained results were validated by an expert. The proposed RINet attains a classification accuracy of 96% and 93% with and without CLR respectively.

Multiscale deep feature selection fusion network for referring image segmentation

Multiscale deep feature selection fusion network for referring image segmentation

STDC-Net: A spatial-temporal deformable convolution network for conference video frame interpolation

Video conference communication can be seriously affected by dropped frames or reduced frame rates due to network or hardware restrictions. Video frame interpolation techniques can interpolate the dropped frames and generate smoother videos. However, existing methods can not generate plausible results in video conferences due to the large motions of the eyes, mouth and head. To address this issue, we propose a Spatial-Temporal Deformable Convolution Network (STDC-Net) for conference video frame interpolation. The STDC-Net first extracts shallow spatial-temporal features by an embedding layer. Secondly, it extracts multi-scale deep spatial-temporal features through Spatial-Temporal Representation Learning (STRL) module, which contains several Spatial-Temporal Feature Extracting (STFE) blocks and downsample layers. To extract the temporal features, each STFE block splits feature maps along the temporal pathway and processes them with Multi-Layer Perceptron (MLP). Similarly, the STFE block splits the temporal features along horizontal and vertical pathways and processes them by another two MLPs to get spatial features. By splitting the feature maps into segments of varying lengths in different scales, the STDC-Net can extract both local details and global spatial features, allowing it to effectively handle large motions. Finally, Frame Synthesis (FS) module predicts weights, offsets and masks using the spatial-temporal features, which are used in deformable convolution to generate the intermediate frames. Experimental results demonstrate the STDC-Net outperforms state-of-the-art methods in both quantitative and qualitative evaluations. Compared to the baseline, the proposed method achieved a PSNR improvement of 0.13 dB and 0.17 dB on the Voxceleb2 and HDTF datasets, respectively.

PathologicMyopia detection and visualization based on multi-scale deep features by PMnet tuned with cyclic learning rate hyperparameter

PathologicMyopia detection and visualization based on multi-scale deep features by PMnet tuned with cyclic learning rate hyperparameter

Multi-scale deep feature fusion based sparse dictionary selection for video summarization

The explosive growth of video data constitutes a series of new challenges in computer vision, and the function of video summarization (VS) is becoming more and more prominent. Recent works have shown the effectiveness of sparse dictionary selection (SDS) based VS, which selects a representative frame set to sufficiently reconstruct a given video. Existing SDS based VS methods use conventional handcrafted features or single-scale deep features, which could diminish their summarization performance due to the underutilization of frame feature representation. Deep learning techniques based on convolutional neural networks (CNNs) exhibit powerful capabilities among various vision tasks, as the CNN provides excellent feature representation. Therefore, in this paper, a multi-scale deep feature fusion based sparse dictionary selection (MSDFF-SDS) is proposed for VS. Specifically, multi-scale features include the directly extracted features from the last fully connected layer and the global average pooling (GAP) processed features from intermediate layers, then VS is formulated as a problem of minimizing the reconstruction error using the multi-scale deep feature fusion. In our formulation, the contribution of each scale of features can be adjusted by a balance parameter, and the row-sparsity consistency of the simultaneous reconstruction coefficient is used to select as few keyframes as possible. The resulting MSDFF-SDS model is solved by using an efficient greedy pursuit algorithm. Experimental results on two benchmark datasets demonstrate that the proposed MSDFF-SDS improves the F-score of keyframe based summarization more than 3% compared with the existing SDS methods, and performs better than most deep-learning methods for skimming based summarization.

Graphic image classification method based on an attention mechanism and fusion of multilevel and multiscale deep features

The paper explores the issue of privacy breaches caused by sharing and publishing images on social media platforms, which is becoming increasingly serious. While existing deep learning-based methods have achieved remarkable results in private image classification, they only consider single-level and single-scale features and neglect the problem of multilevel and multiscale features. To address this limitation, the paper proposes a novel graphic image classification method that fuses multilevel and multiscale deep features. The proposed method employs a multibranch convolutional neural network to extract multilevel features, which are then processed by a multilevel pooling layer to obtain multiscale features. Additionally, two feature fusion methods based on attention mechanism, Privacy-MSML (Privacy Multi-Scale Multi-Level) and Privacy-MLMS (Privacy Multi-Level Multi-Scale), are designed to fuse the multilevel and multiscale features for image classification. Both methods utilize Bi-LSTM and a self-attention mechanism to capture feature dependencies. The experimental results on public datasets demonstrate that the proposed methods effectively fuse multilevel and multiscale features, leading to a significant improvement in classification performance, which highlights the innovative contribution of the paper in addressing the issue of multilevel and multiscale features in private image classification.

ADINet: Attention-guided dual-branch integration network for atypical visual saliency prediction

Compared with typically developed people, patients with autism spectrum disorder (ASD) have many differences, including visual attention. Recently, researchers have employed saliency prediction to conduct early diagnosis for patients with ASD by observing whether the fixation maps of the patients are consistent with prediction results. Though numerous saliency models have been designed, which promotes the performance of saliency prediction significantly, the efforts major in atypical attention prediction are fewer. Therefore, we propose a novel atypical saliency model, namely attention-guided dual-branch integration network, to perform atypical visual saliency prediction. Generally, our model is a U-shaped architecture including an encoder and a decoder. In the encoder, we first deploy the dual-branch fusion module to extract multiscale deep features, where the dual branches consist of a coarse branch and a fine branch. Particularly, the multiscale features of each branch are integrated in a bilateral way. Then, we employ the multiattention module consisting of dilated convolution, channel attention, and spatial attention to further elevate the high-level semantic cues. Particularly, the attention units are constructed in a cascaded way. Besides, to give an effective representation for the differences between atypical saliency estimation and the typical saliency estimation, we design a new discriminative mask to compute the loss function. Extensive experiments are performed on challenging datasets, and the results show the superiority and effectiveness of our atypical saliency model when compared with the state-of-the-art saliency prediction models.