Channel-wise Attention Research Articles

Abnormal event detection in surveillance videos based on multi-scale feature and channel-wise attention mechanism

Abnormal event detection in surveillance videos based on multi-scale feature and channel-wise attention mechanism

Automatic zonal segmentation of the prostate from 2D and 3D T2-weighted MRI and evaluation for clinical use.

Purpose: An accurate zonal segmentation of the prostate is required for prostate cancer (PCa) management with MRI. Approach: The aim of this work is to present UFNet, a deep learning-based method for automatic zonal segmentation of the prostate from T2-weighted (T2w) MRI. It takes into account the image anisotropy, includes both spatial and channelwise attention mechanisms and uses loss functions to enforce prostate partition. The method was applied on a private multicentric three-dimensional T2w MRI dataset and on the public two-dimensional T2w MRI dataset ProstateX. To assess the model performance, the structures segmented by the algorithm on the private dataset were compared with those obtained by seven radiologists of various experience levels. Results: On the private dataset, we obtained a Dice score (DSC) of for the whole gland (WG), for the transition zone (TZ), and for the peripheral zone (PZ). Results were significantly better than other compared networks' ( ). On ProstateX, we obtained a DSC of for WG, for TZ, and for PZ. These results are similar to state-of-the art results and, on the private dataset, are coherent with those obtained by radiologists. Zonal locations and sectorial positions of lesions annotated by radiologists were also preserved. Conclusions: Deep learning-based methods can provide an accurate zonal segmentation of the prostate leading to a consistent zonal location and sectorial position of lesions, and therefore can be used as a helping tool for PCa diagnosis.

A Hybrid Multivariate Deep Learning Network for Multistep Ahead Sea Level Anomaly Forecasting

Abstract The accumulated remote sensing data of altimeters and scatterometers have provided new opportunities for ocean state forecasting and have improved our knowledge of ocean–atmosphere exchanges. Studies on multivariate, multistep, spatiotemporal sequence forecasts of sea level anomalies (SLA) for different modalities, however, remain problematic. In this paper, we present a novel hybrid and multivariate deep neural network, named HMnet3, which can be used for SLA forecasting in the South China Sea (SCS). First, a spatiotemporal sequence forecasting network is trained by an improved convolutional long short-term memory (ConvLSTM) network using a channelwise attention mechanism and multivariate data from 1993 to 2015. Then a time series forecasting network is trained by an improved long short-term memory (LSTM) network, which is realized by ensemble empirical mode decomposition (EEMD). Finally, the two networks are combined by a successive correction method to produce SLA forecasts for lead times of up to 15 days, with a special focus on the open sea and coastal regions of the SCS. During the testing period of 2016–18, the performance of HMnet3 with sea surface temperature anomaly (SSTA), wind speed anomaly (SPDA), and SLA data is much better than those of state-of-the-art dynamic and statistical (ConvLSTM, persistence, and climatology) forecast models. Stricter testbeds for trial simulation experiments with real-time datasets are investigated, where the eddy classification metrics of HMnet3 are favorable for all properties, especially for those of small-scale eddies.

DISTEMA: distance map-based estimation of single protein model accuracy with attentive 2D convolutional neural network

BackgroundEstimation of the accuracy (quality) of protein structural models is important for both prediction and use of protein structural models. Deep learning methods have been used to integrate protein structure features to predict the quality of protein models. Inter-residue distances are key information for predicting protein’s tertiary structures and therefore have good potentials to predict the quality of protein structural models. However, few methods have been developed to fully take advantage of predicted inter-residue distance maps to estimate the accuracy of a single protein structural model.ResultWe developed an attentive 2D convolutional neural network (CNN) with channel-wise attention to take only a raw difference map between the inter-residue distance map calculated from a single protein model and the distance map predicted from the protein sequence as input to predict the quality of the model. The network comprises multiple convolutional layers, batch normalization layers, dense layers, and Squeeze-and-Excitation blocks with attention to automatically extract features relevant to protein model quality from the raw input without using any expert-curated features. We evaluated DISTEMA’s capability of selecting the best models for CASP13 targets in terms of ranking loss of GDT-TS score. The ranking loss of DISTEMA is 0.079, lower than several state-of-the-art single-model quality assessment methods.ConclusionThis work demonstrates that using raw inter-residue distance information with deep learning can predict the quality of protein structural models reasonably well. DISTEMA is freely at https://github.com/jianlin-cheng/DISTEMA

Local information fusion network for 3D shape classification and retrieval

Due to the wide applications of 3D shapes, exploring effective 3D shape recognition algorithms has attracted much research attention. Various approaches have been proposed in recent years, within which the multi-view based methods show their promising performances. Previous multi-view based methods mainly extract global features of views by employing the well-established CNN and then explore the correlations between the view-level descriptors for 3D shape representation. However, these approaches ignore the local characteristics extraction of view images. Besides, they also lack the consideration for the relationships between image regions and feature map channels, which could provide more detailed and descriptive information to improve the discrimination of shape descriptors. To address these issues, we propose a novel Local Information Fusion Network (LIFN) for local characteristics extraction and relationships exploration based on the feature maps during the convolution process. Concretely, the Region Organization Module (ROM) is introduced for feature map reorganization and region-wise feature extraction. Besides, the Region-wise Attention (RWA) and Channel-wise Attention (CWA) are designed for region-wise and channel-wise interaction exploration, respectively. Extensive experiments on the ModelNet40 database demonstrate the superiority of our proposed network against the state-of-the-art approaches.

MLFC-net: A multi-level feature combination attention model for remote sensing scene classification

The image labeling task of remote sensing image scene classification (RSSC) is based on the semantic content of remote sensing images. The semantic information within remote sensing photographs has become more complicated and difficult to detect as remote sensing technology has progressed. As a result, extracting more important semantic elements could aid in the completion of the RSSC assignment. Thus, in this research, we offer MLFC-Net, a multi-level semantic feature clustering attention model based on deep convolution neural networks (DCNNs) that extracts more accurate feature information. The concept of MLFC-Net stems from the utilization of rich spatial information found in remote sensing photos, but few approaches in the RSSC application considered merging general semantic feature information with clustered semantic feature information. By rearranging the weight of corresponding information, such as feature maps and tensor blocks of the feature map, we implemented the attention mechanism. To build a model with minimal computational cost and good portability, we use a channel-wise attention mechanism and an ensemble structure. We were able to improve the representation of several critical semantic aspects using the MLFC model. In the EuroSAT, UCM, and NWPU-RESISC45 RSSC datasets, the MLFC model's performance is demonstrated. And, on average, the MLFC model enhanced accuracy by 2.56 percent, 1.25 percent and 2.00 percent, respectively, producing results that were equivalent to the state-of-the-art.

Deep metric attention learning for skin lesion classification in dermoscopy images

Currently, convolutional neural networks (CNNs) have made remarkable achievements in skin lesion classification because of their end-to-end feature representation abilities. However, precise skin lesion classification is still challenging because of the following three issues: (1) insufficient training samples, (2) inter-class similarities and intra-class variations, and (3) lack of the ability to focus on discriminative skin lesion parts. To address these issues, we propose a deep metric attention learning CNN (DeMAL-CNN) for skin lesion classification. In DeMAL-CNN, a triplet-based network (TPN) is first designed based on deep metric learning, which consists of three weight-shared embedding extraction networks. TPN adopts a triplet of samples as input and uses the triplet loss to optimize the embeddings, which can not only increase the number of training samples, but also learn the embeddings robust to inter-class similarities and intra-class variations. In addition, a mixed attention mechanism considering both the spatial-wise and channel-wise attention information is designed and integrated into the construction of each embedding extraction network, which can further strengthen the skin lesion localization ability of DeMAL-CNN. After extracting the embeddings, three weight-shared classification layers are used to generate the final predictions. In the training procedure, we combine the triplet loss with the classification loss as a hybrid loss to train DeMAL-CNN. We compare DeMAL-CNN with the baseline method, attention methods, advanced challenge methods, and state-of-the-art skin lesion classification methods on the ISIC 2016 and ISIC 2017 datasets, and test its generalization ability on the PH2 dataset. The results demonstrate its effectiveness.

ShaTure: Shape and Texture Deformation for Human Pose and Attribute Transfer.

In this paper, we present a novel end-to-end pose transfer framework to transform a source person image to an arbitrary pose with controllable attributes. Due to the spatial misalignment caused by occlusions and multi-viewpoints, maintaining high-quality shape and texture appearance is still a challenging problem for pose-guided person image synthesis. Without considering the deformation of shape and texture, existing solutions on controllable pose transfer still cannot generate high-fidelity texture for the target image. To solve this problem, we design a new image reconstruction decoder - ShaTure which formulates shape and texture in a braiding manner. It can interchange discriminative features in both feature-level space and pixel-level space so that the shape and texture can be mutually fine-tuned. In addition, we develop a new bottleneck module - Adaptive Style Selector (AdaSS) Module which can enhance the multi-scale feature extraction capability by self-recalibration of the feature map through channel-wise attention. Both quantitative and qualitative results show that the proposed framework has superiority compared with the state-of-the-art human pose and attribute transfer methods. Detailed ablation studies report the effectiveness of each contribution, which proves the robustness and efficacy of the proposed framework.

Face Reconstruction Algorithm based on Lightweight Convolutional Neural Networks and Channel-wise Attention

Face Reconstruction Algorithm based on Lightweight Convolutional Neural Networks and Channel-wise Attention

Seismic Impedance Inversion Based on Residual Attention Network

Deep learning has achieved promising results for impedance inversion via seismic data. Generally, these networks, composed of convolution layers and residual blocks, tend to deliver good results with deep architectures. Nevertheless, deep networks accompany a large number of parameters and longer training time. The volume of seismic data, especially 3D scenarios, is very large. Therefore, it is particularly important to improve the accuracy while ensuring the model efficiency for practical implementation. With the flourishing new modules and techniques, deep learning has set the state-of-the-art in many applications across wide range of scientific and engineering disciplines. In this paper, we present Residual Attention Network (ResANet), a CNN incorporating with residual modules and two attention mechanisms: channel-wise attention and feature-map attention, for seismic impedance inversion. The proposed network can fuse multi-scale channel information and recalibrate channel-wise feature responses as well as receptive fields adaptively. At the same time, ResANet adopts grouped convolution, dilated convolution and dropout techniques to improve the computation efficiency and stability. Marmousi2 synthetic model and field data test results show that the proposed network outperforms several comparable neural networks in accuracy and generalization ability while ensuring efficiency for seismic data impedance inversion. For the field data test, transfer learning is also evoked to further improve the performance. ResANet tends to predict impedance with high resolution and strong lateral continuity compare with three closely related networks. The accuracy of ResANet is improved by 1 to 2 orders of magnitude on the 6 well logs provided in field dataset tests compare with commercial software (InverTrace Plus module in Jason) using Constrained Sparse Spike Inversion (CSSI) method.

Continuous Facial Motion Deblurring

We introduce a novel framework for continuous facial motion deblurring that restores the continuous sharp moment latent in a single motion-blurred face image via a moment control factor. Although a motion-blurred image is the accumulated signal of continuous sharp moments during the exposure time, most existing single image deblurring approaches aim to restore a fixed number of frames using multiple networks and training stages. To address this problem, we propose a continuous facial motion deblurring network based on GAN (CFMD-GAN), which is a novel framework for restoring the continuous moment latent in a single motion-blurred face image with a single network and a single training stage. To stabilize the network training, we train the generator to restore continuous moments in the order determined by our facial motion-based reordering process (FMR) utilizing domain-specific knowledge of the face. Moreover, we propose an auxiliary regressor that helps our generator produce more accurate images by estimating continuous sharp moments. Furthermore, we introduce a control-adaptive (ContAda) block that performs spatially deformable convolution and channel-wise attention as a function of the control factor. Extensive experiments on the 300VW datasets demonstrate that the proposed framework generates a various number of continuous output frames by varying the moment control factor. Compared with the recent single-to-single image deblurring networks trained with the same 300VW training set, the proposed method show the superior performance in restoring the central sharp frame in terms of perceptual metrics, including LPIPS, FID and Arcface identity distance. The proposed method outperforms the existing single-to-video deblurring method for both qualitative and quantitative comparisons. In our experiments on the 300VW test set, the proposed framework reached 33.14 dB and 0.93 for recovery of 7 sharp frames in PSNR and SSIM, respectively.

MA-GANet: A Multi-Attention Generative Adversarial Network for Defocus Blur Detection.

Background clutters pose challenges to defocus blur detection. Existing approaches often produce artifact predictions in background areas with clutter and relatively low confident predictions in boundary areas. In this work, we tackle the above issues from two perspectives. Firstly, inspired by the recent success of self-attention mechanism, we introduce channel-wise and spatial-wise attention modules to attentively aggregate features at different channels and spatial locations to obtain more discriminative features. Secondly, we propose a generative adversarial training strategy to suppress spurious and low reliable predictions. This is achieved by utilizing a discriminator to identify predicted defocus map from ground-truth ones. As such, the defocus network (generator) needs to produce 'realistic' defocus map to minimize discriminator loss. We further demonstrate that the generative adversarial training allows exploiting additional unlabeled data to improve performance, a.k.a. semi-supervised learning, and we provide the first benchmark on semi-supervised defocus detection. Finally, we demonstrate that the existing evaluation metrics for defocus detection generally fail to quantify the robustness with respect to thresholding. For a fair and practical evaluation, we introduce an effective yet efficient AUFβ metric. Extensive experiments on three public datasets verify the superiority of the proposed methods compared against state-of-the-art approaches.

FSC-Set: Counting, Localization of Football Supporters Crowd in the Stadiums

Counting the number of people in a crowd has gained attention in the last decade. Due to its benefit to many applications such as crowd behavior analysis, crowd management, and video surveillance systems, etc. Counting crowded scenes, like stadiums, represents a challenging task due to the inherent occlusions and density of the crowd inside and outside the stadiums. Finding a pattern to control thousands of people and counting them is a challenging task. With the introduction of Convolutional Neural Networks (CNN), enables performing this task with acceptable performance. The accuracy of a CNN-based method is related to the size of data used for training. The availability of the dataset is sparse. In particular, there is no dataset in the literature that can be used for training applications for crowd scene. This paper proposes two main contributions including a new dataset for crowd counting, and a CNN-based method for counting the number of people and generating the crowd density maps. The proposed dataset for Football Supporters Crowd (FSC-Set) is composed of 6000 annotated images (manually) of different types of scenes that contain thousands of people gathering in or around the stadiums. FSC-Set contains more than 1.5 Million individuals. The collected images are captured under varying Fields of Views (FOV), illuminations, resolutions, and scales. The proposed dataset can also be utilized for other applications, such as individual’s localization and face detection as well as team recognition from supporter images. Further, we propose a CNN-based method named FSCNet for crowd counting exploiting context-aware attention, spatial-wise attention, and channel-wise attention modules. The proposed method is evaluated on our established FSC-Set and other existing datasets then compared to state-of-the-art methods. The obtained results show satisfactory performances on all the datasets. The dataset is made publicly available and can be requested using the following link: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://sites.google.com/view/fscrowd-dataset/</uri>

Channel-wise attention mechanism-relevant multi-branches convolutional network-based depressive disorder recognition

Channel-wise attention mechanism-relevant multi-branches convolutional network-based depressive disorder recognition

Attention-Driven Graph Neural Network for Deep Face Super-Resolution.

With the help of convolutional neural networks (CNNs), deep learning-based methods have achieved remarkable performance in face super-resolution (FSR) task. Despite their success, most of the existing methods neglect non-local correlations of face images, leaving much room for improvement. In this paper, we introduce a novel end-to-end trainable attention-driven graph neural network (AD-GNN) for more discriminative feature extraction and feature relation modeling. This is achieved by two major components. The first component is a cross-scale dynamic graph (CDG) block. The CDG block considers cross-scale relationships of patches in distant areas and employs two dynamic graphs to construct enhanced features. The second component is a series of channel attention and spatial dynamic graph (CASDG) blocks. A CASDG block has a channel-wise attention unit and a spatial-aware dynamic graph (SDG) unit. The SDG unit extracts informative features by exploring spatial non-local self-similarity information of the patches using dynamic graph convolution. Using these two components, facial details can be effectively reconstructed with the help of information supplemented by similar but spatially remote patches and structural information of faces. Extensive experiments on two public benchmarks demonstrate the superiority of AD-GNN over the state-of-the-art FSR methods.

CU-Net: A New Improved Multi-Input Color U-Net Model for Skin Lesion Semantic Segmentation

Melanoma is considered one of the most dangerous skin cancer diseases that threaten human health and life. Early diagnosis of melanoma is a big challenge, especially with the presence of color variations across similar lesion types. Automatic skin lesion segmentation is an essential step to build a successful skin disease classification system. Recent deep learning architectures significantly improve the skin lesion segmentation results. Especially, U-Net deep convolutional neural network (CNN) is considered one of the state-of-the-art models with promising performance. Most deep CNNs and particularly U-Net model utilize a single input RGB color image for skin lesion semantic segmentation. However, RGB color space is not usually the best choice to represent the invariant characteristics of skin lesion chromatic information. The selection of the optimal color space significantly affects the performance of segmentation results. In this paper, three novel variants of U-Net model with single, dual, and triple inputs, namely, Single Input Color U-Net (SICU-Net), Dual Input Color U-Net (DICU-Net) and Triple Input Color U-Net (TICU-Net) are proposed. The structure of SICU-Net, DICU-Net, and TICU-Net contains single, dual, and triple encoder sub-networks connected with only a single decoder path. Each encoder sub-network is fed with different color space of the input image. A channel-wise attention module is utilized to fuse the contribution of the learned feature maps from each encoder sub-network which is fed to the decoder sub-network to generate segmented image map. Moreover, a composite loss function is designed to improve the performance of the proposed CU-Net models. Three public benchmark datasets, namely, International Skin Imaging Collaboration (ISIC 2017, ISIC 2018) and PH2 datasets, are utilized to evaluate the performance of the proposed models. Experimental results reveal that the proposed models significantly improve the performance of the original U-Net model and achieve comparable performance with other state-of-the-art methods.

Locality-Aware Channel-Wise Dropout for Occluded Face Recognition.

Face recognition remains a challenging task in unconstrained scenarios, especially when faces are partially occluded. To improve the robustness against occlusion, augmenting the training images with artificial occlusions has been proved as a useful approach. However, these artificial occlusions are commonly generated by adding a black rectangle or several object templates including sunglasses, scarfs and phones, which cannot well simulate the realistic occlusions. In this paper, based on the argument that the occlusion essentially damages a group of neurons, we propose a novel and elegant occlusion-simulation method via dropping the activations of a group of neurons in some elaborately selected channel. Specifically, we first employ a spatial regularization to encourage each feature channel to respond to local and different face regions. Then, the locality-aware channel-wise dropout (LCD) is designed to simulate occlusions by dropping out a few feature channels. The proposed LCD can encourage its succeeding layers to minimize the intra-class feature variance caused by occlusions, thus leading to improved robustness against occlusion. In addition, we design an auxiliary spatial attention module by learning a channel-wise attention vector to reweight the feature channels, which improves the contributions of non-occluded regions. Extensive experiments on various benchmarks show that the proposed method outperforms state-of-the-art methods with a remarkable improvement.

DC-tCNN: A Deep Model for EEG-Based Detection of Dim Targets.

Dim target detection in remote sensing images is a significant and challenging problem. In this work, we seek to explore event-related brain responses of dim target detection tasks and extend the brain-computer interface (BCI) systems to this task for efficiency enhancement. We develop a BCI paradigm named Asynchronous Visual Evoked Paradigm (AVEP), in which subjects are required to search the dim targets within satellite images when their scalp electroencephalography (EEG) signals are simultaneously recorded. In the paradigm, stimulus onset time and target onset time are asynchronous because subjects need enough time to confirm whether there are targets of interest in the presented serial images. We further propose a Domain adaptive and Channel-wise attention-based Time-domain Convolutional Neural Network (DC-tCNN) to solve the single-trial EEG classification problem for the AVEP task. In this model, we design a multi-scale CNN module combined with a channel-wise attention module to effectively extract event-related brain responses underlying EEG signals. Meanwhile, domain adaptation is proposed to mitigate cross-subject distribution discrepancy. The results demonstrate the superior performance and better generalizability of this model in classifying the single-trial EEG data of AVEP task in contrast to typical EEG deep learning networks. Visualization analyses of spatiotemporal features also illustrate the effectiveness and interpretability of our proposed paradigm and learning model. The proposed paradigm and model can effectively explore ambiguous event-related brain responses on EEG-based dim target detection tasks. Our work can provide a valuable reference for BCI-based image detection of dim targets.

SRUNet: Stacked Reversed U-Shape Network for Lightweight Single Image Super-Resolution

In recent years, for relieving the heavy computational cost, lightweight models have been successfully applied to single image super-resolution (SISR) task. However, most lightweight models adopt low-resolution images as input and apply transposed convolution or subpixel convolution only in the tail of the model to reconstruct the super-resolution image which neglect to fully utilize the multi-scale features. To resolve this problem, we propose a stacked reversed U-shape network (SRUNet) to further extract and utilize the multi-scale features at different resolutions. In detail, SRUNet consists of shallow feature extraction (SFE), stacked reversed U-shape module (SRUM), multi-scale backward fusion module (MSBFM) and feature refinement module (FRM). Instead of upsampling the feature map at the tail of the model, we perform the upsampling and downsampling operation progressively and iteratively by the stacked reversed U-shape module to extract richer multi-scale features. Furthermore, for archiving better use of multi-scale features, a scale-wise dense connection with residual channel-wise attention and multi-scale backward fusion is added to the network. The fused super-resolved features are refined by the feature refinement module and reconstructed to the image. Extensive experiments demonstrate that our model can achieve competitive performance compared with the state-of-the-art methods. When scaling factor is 4, SRUNet achieves the highest SSIM performance in all benchmarks and it takes 29.2ms to process per image in Urban100 dataset.

EMCA: Efficient Multiscale Channel Attention Module

Attention mechanisms have been explored with CNNs across the spatial and channel dimensions. However, all the existing methods devote the attention modules to capture local interactions from a uni-scale. This paper tackles the following question: can one consolidate multi-scale aggregation while learning channel attention more efficiently? To this end, we avail channel-wise attention over multiple feature scales, which empirically shows its aptitude to replace the limited local and uni-scale attention modules. EMCA is lightweight and can efficiently model the global context further; it is easily integrated into any feed-forward CNN architectures and trained in an end-to-end fashion. We validate our novel architecture through comprehensive experiments on image classification, object detection, and instance segmentation with different backbones. Our experiments show consistent gains in performances against their counterparts, where our proposed module, named EMCA, outperforms other channel attention techniques in accuracy and latency trade-off. More specifically, compared to SENet, we boost the accuracy by 0.8 %, 0.6 %, and 1 % on ImageNet benchmark for ResNet-18, 34, and 50, respectively. For detection and segmentation tasks, MS-COCO are for benchmarking, Our EMCA module boost the accuracy by 0.5 % and 0.3 %, respectively. We also conduct experiments that probe the robustness of the learned representations. Our code will be published once the paper is accepted.