Multi-scale Feature Learning Research Articles

Self-Attention-Based Multiscale Feature Learning Optical Flow With Occlusion Feature Map Prediction

Even though optical flow approaches based on convolutional neural networks have achieved remarkable performance with respect to both accuracy and efficiency, large displacements and motion occlusions remain challenges for most existing learning-based models. To address the abovementioned issues, we propose in this paper a self-attention-based multiscale feature learning optical flow computation method with occlusion feature map prediction. First, we exploit a self-attention mechanism-based multiscale feature learning module to compensate for large displacement optical flows, and the presented module is able to capture long-range dependencies from the input frames. Second, we design a simple but effective self-learning module to acquire an occlusion feature map, in which the predicted occlusion map is utilized to correct the optical flow estimation in occluded areas. Third, we explore a hybrid loss function that integrates the photometric and smoothness losses into the classical endpoint error (EPE)-based loss to ensure the accuracy and robustness of the presented network. Finally, we compare the proposed method with some state-of-the-art approaches using the MPI-Sintel and KITTI test databases. The experimental results demonstrate that the proposed method achieved competitive performance with respect to both accuracy and robustness, and it produced the better results compared to other methods under large displacements and motion occlusions.

SC-EADNet: A Self-Supervised Contrastive Efficient Asymmetric Dilated Network for Hyperspectral Image Classification

Unsupervised and semisupervised feature learning has recently emerged as an effective way to reduce the reliance on expensive data collection and annotation for hyperspectral image (HSI) analysis. Existing unsupervised and semisupervised convolutional neural network (CNN)-based HSI classification works still face two challenges: underutilization of pixel-wise multiscale contextual information for feature learning and expensive computational cost, for example, large floating-point operations per seconds (FLOPs), due to the lack of lightweight design. To utilize the unlabeled pixels in the HSIs more efficiently, we propose a self-supervised contrastive efficient asymmetric dilated network (SC-EADNet) for HSI classification. There are two novelties in the SC-EADNet. First, a self-supervised multiscale pixel-wise contextual feature learning model is proposed, which generates multiple patches around each hyperspectral pixel and develops a contrastive learning framework to learn from these patches for HSI classification. Second, a lightweight feature extraction network EADNet, composed of multiple plug-and-play efficient asymmetric dilated convolution (EADC) blocks, is designed and inserted into the contrastive learning framework. The EADC block adopts different dilation rates to capture the spatial information of objects with varying shapes and sizes. Compared with other unsupervised, semisupervised, and supervised learning methods, our SC-EADNet provides competitive classification performance on four hyperspectral datasets, including Indian Pines, Pavia University, Salinas, and Houston 2013, but few FLOPs and fast computational speed.

Global–Local Transformer Network for HSI and LiDAR Data Joint Classification

Hyperspectral images (HSI) contain rich spatial and spectral detail information, while light detection and ranging (LiDAR) data can provide the elevation information. Thus, the fusion of HSI and LiDAR data can help for more accurate image classification, which becomes a hot research topic. However, it is difficult to capture complex local and global spatial-spectral associations, meanwhile, how to build an effective interaction between multi-modal data is another important issue. To this end, a novel global-local transformer network (GLT-Net) is proposed for the joint classification of HSI and LiDAR data, in this paper. The main idea is to fully exploit the advantage of the convolution operator in characterizing locally correlated features and the promising capability of transformer architecture in learning long-range dependencies. Moreover, multi-scale feature fusion and probabilistic decision fusion strategies are also designed in one framework, in order to further improve classification performance. Here, the proposed GLT-Net mainly consists of multi-scale local spatial feature learning, global spectral feature learning, and global-local feature fusion classification. In specific, multi-modal image cubes of different sizes are firstly extracted and sent into convolutional neural networks (CNNs) to learn local spatial features, which is followed by multi-modal information propagation and spatial-attention guided multi-scale feature fusion. Afterwards, by considering spectral feature channels from a sequential perspective, vision transformers are introduced to model the global spectral dependencies. Finally, multiple class estimations based on local and global features are integrated via a probabilistic decision fusion strategy. In this way, complementary information of multi-modal data as well as local/global spectral-spatial information can be fully mined and jointly utilized. Extensive experiments on three popular HSI and LiDAR datasets demonstrate that the proposed method performs superiority over state-of-the-art methods. The source code of the proposed method will be made publicly available at https://github.com/Ding-Kexin/GLT-Net.

Unsupervised Spatial-Spectral CNN-Based Feature Learning for Hyperspectral Image Classification

The rapid development of remote sensing sensors makes the acquisition, analysis, and application of hyperspectral images (HSIs) more and more extensive. However, the limited sample sets, high-dimensional features, highly correlated bands, and mixing spectral information make the classification of HSIs a great challenge. In this article, an unsupervised multiscale and diverse feature learning (UMsDFL) method is proposed for HSI classification, which deeply considers the spatial–spectral features via convolutional neural networks (CNNs). Specifically, after employing the simple noniterative clustering (SNIC) algorithm with the heuristic calculation of superpixel size, the HSIs are segmented into superpixels for feature learning. The unsupervised network is designed with the convolutional encoder and decoder, the additional clustering branch, and the multilayer feature fusion to enhance the distinguishability of feature learning and the reusability of feature maps. Then, the spatial relationships and object attributes in large- and small-scale contexts are learned collaboratively through the unsupervised network to utilize the complementary multiscale characteristics. Moreover, the diverse features of hyperspectral information and nonsubsampled contourlet transform (NSCT) textures are learned simultaneously via the unsupervised network to alleviate the insufficiency of geometric representation. Finally, the random forest (RF) is adopted as the comprehensive classifier for land cover mapping based on the UMsDFL, and superpixel regularization is adopted to optimize the classification results. A series of experiments are performed on three real-world HSI datasets to demonstrate the effectiveness of our UMsDFL approach. The experimental results show that the proposed UMsDFL can achieve the overall accuracy of 79.23%, 96.49%, and 77.26% for Houston, Pavia, and Dioni datasets, respectively, when there are only five samples per class for training.

Class-Level Prototype Guided Multiscale Feature Learning for Remote Sensing Scene Classification With Limited Labels

Remote sensing scene classification (RSSC) is an open and challenging research topic in the remote sensing (RS) community. It aims to define semantic labels to RS scenes according to their contents. Recently, with the development of deep convolutional neural networks (DCNNs), the results of RSSC have been enhanced to a large extent. However, the cracking performance of these DCNN-based models depends on a large number of labeled data. Once the volume of the labeled data is decreased, their behavior would be weakened dramatically. In this paper, we propose a new training algorithm that can work smoothly with a few labeled samples to address this limitation. Along with the introduced DCNN, the presented methods perform satisfactorily. In particular, we first construct a dual-branch network (DBNet) to mine the multi-scale and multi-angle information from RS scenes. Thus, the abundant land covers with diverse sizes, directions, and shapes can be captured simultaneously. Then, to train DBNet using scarce semantic labels, a class-level prototype guided learning (CPGL) algorithm is developed based on the mate-learning paradigm. Besides the usual episode training manner, a prototype refinement module (PRM) and a prototype discrimination module (PDM) are designed with the help of metric learning theory to ensure the effectiveness of our CPGL. The comprehensive experiments are conducted on four public RS scene data sets, and the encouraging results imply that our DBNet and CPGL can copy with RSSC tasks with small labeled data. Our source codes are available at https://github.com/TangXu-Group/Remote-Sensing-Images-Classification/tree/main/CPGL.

An Attention-Enhanced End-to-End Discriminative Network With Multiscale Feature Learning for Remote Sensing Image Retrieval

The discriminative ability of image features plays a decisive role in content-based remote sensing image retrieval (CBRSIR). However, the widely-used convolutional neural networks cannot focus on the discriminative features of important scenes, resulting in unsatisfactory retrieval performance in complex contexts. In this paper, an attention-enhanced end-to-end discriminative network with multiscale learning for CBRSIR is proposed to solve this issue. First, a multiscale dilated convolution module is embedded into some of ResNet50's residual blocks to increase the perceptual field and capture the multiscale features of remote sensing image scenes. Then, a lightweight and efficient triplet attention module is added behind each residual block to capture the salient features of remote sensing images and establish the inter-dimensional dependencies using residual transform. In addition, the end-to-end training approach is performed using an online label smoothing loss to reduce the intra-class variance of features and enhance inter-class differentiability. Experimental results on four publicly available remote sensing image datasets show that our network achieves state-of-the-art or competitive performance, especially on complex scene dataset UCMD with an average retrieval precision improvement of 3.23% to 29.35% compared to other new methods.

Deep Multi-Scale Feature Learning for Defocus Blur Estimation

This paper presents an edge-based defocus blur estimation method from a single defocused image. We first distinguish edges that lie at depth discontinuities (called depth edges, for which the blur estimate is ambiguous) from edges that lie at approximately constant depth regions (called pattern edges, for which the blur estimate is well-defined). Then, we estimate the defocus blur amount at pattern edges only, and explore an interpolation scheme based on guided filters that prevents data propagation across the detected depth edges to obtain a dense blur map with well-defined object boundaries. Both tasks (edge classification and blur estimation) are performed by deep convolutional neural networks (CNNs) that share weights to learn meaningful local features from multi-scale patches centered at edge locations. Experiments on naturally defocused images show that the proposed method presents qualitative and quantitative results that outperform state-of-the-art (SOTA) methods, with a good compromise between running time and accuracy.

Transferring Transformer-Based Models for Cross-Area Building Extraction From Remote Sensing Images

Extracting buildings from remote sensing images is an important task with a variety of applications. Considerable attention has focused on achieving new SOTA accuracy with more and more advanced deep learning models. However, the developed models still hardly generalize across geographical areas, hindering the practical use of SOTA approaches. To attack this problem, we established a baseline for model cross-area generalization ability using available datasets for BE. In addition to two popular FCN-based models, we first adapted two novel transformer-based models, Swin Transformer and SegFormer, which are all able to output SOTA accuracy with no big difference when tested within one area. However, experimental results show that all models fail to generalize to a different area. We then propose to fine-tune pre-trained models from one area on a small subset of an unseen area, the effectiveness of which depends on the model choice and the data size for tuning. By jointly taking advantage of the transfer learning idea and the multiscale feature learning ability of SegFormer, a distinct improvement has been achieved compared to results from Swin Transformer and FCN-based models trained on the same amount of data. Commonly used metric, IoU, can be increased from 38.97% to 70.86%, and from 48.36% to 74.51%, when using 10% and 30% subset of the targeting area, respectively. The influence of model choice and data size for tuning has also been investigated. Our work contributes to complementing the algorithm development and within-area model evaluation in the hot field of BE from RS images.

Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images

Semantic segmentation using fine-resolution remotely sensed images plays a critical role in many practical applications, such as urban planning, environmental protection, natural and anthropogenic landscape monitoring, etc. However, the automation of semantic segmentation, i.e., automatic categorization/labeling and segmentation is still a challenging task, particularly for fine-resolution images with huge spatial and spectral complexity. Addressing such a problem represents an exciting research field, which paves the way for scene-level landscape pattern analysis and decision making. In this paper, we propose an approach for automatic land segmentation based on the Feature Pyramid Network (FPN). As a classic architecture, FPN can build a feature pyramid with high-level semantics throughout. However, intrinsic defects in feature extraction and fusion hinder FPN from further aggregating more discriminative features. Hence, we propose an Attention Aggregation Module (AAM) to enhance multi-scale feature learning through attention-guided feature aggregation. Based on FPN and AAM, a novel framework named Attention Aggregation Feature Pyramid Network (A2-FPN) is developed for semantic segmentation of fine-resolution remotely sensed images. Extensive experiments conducted on three datasets demonstrate the effectiveness of our A2 -FPN in segmentation accuracy. Code is available at https://github.com/lironui/A2-FPN.

MSegnet, a Practical Network for Building Detection from High Spatial Resolution Images

Building detection in big earth data by remote sensing is crucial for urban development. However, improving its accuracy remains challenging due to complicated background objects and different viewing angles from various remotely sensed images. The hereto proposed methods predominantly focus on multi-scale feature learning, which omits features in multiple aspect ratios. Moreover, postprocessing is required to refine the segmentation performance. We propose modified semantic segmentation (MSegnet), a single-shot semantic segmentation model based on a matrix of convolution layers to extract features in multiple scales and aspect ratios. MSegnet consists of two modules: backbone feature learning and matrix convolution to conduct vertical and horizontal learning. The matrix convolution comprises a set of convolution operations with different aspect ratios. MSegnet is applied to a public building data set that is widely used for evaluation and shown to achieve satisfactory accuracy, compared with the published single-shot methods.

Efficient infrared imaging–blur kernel estimation with multi-scale feature learning for online learning video source

To enhance the infrared images resolution, a novel infrared imaging–blur kernel estimation model is proposed, which is based on multiscale deconvolution and total variation regularization. The blur kernel can be refined by the deconvolution algorithm with increasing scale level. Furthermore, the split Bregman iteration algorithm is used to optimize the proposed cost functions. The major novelty of the study is that the proposed model can recover the high-resolution image and an accurate blur kernel from a degraded image simultaneously. Qualitative and quantitative results on simulated and actual images demonstrate that the proposed method can estimate the infrared blur function accurately and produce high-resolution image. The enhanced infrared images can be applied to the application for the learning video quality improvement.

PEMCNet: An Efficient Multi-Scale Point Feature Fusion Network for 3D LiDAR Point Cloud Classification

Point cloud classification plays a significant role in Light Detection and Ranging (LiDAR) applications. However, most available multi-scale feature learning networks for large-scale 3D LiDAR point cloud classification tasks are time-consuming. In this paper, an efficient deep neural architecture denoted as Point Expanded Multi-scale Convolutional Network (PEMCNet) is developed to accurately classify the 3D LiDAR point cloud. Different from traditional networks for point cloud processing, PEMCNet includes successive Point Expanded Grouping (PEG) units and Absolute and Relative Spatial Embedding (ARSE) units for representative point feature learning. The PEG unit enables us to progressively increase the receptive field for each observed point and aggregate the feature of a point cloud at different scales but without increasing computation. The ARSE unit following the PEG unit furthermore realizes representative encoding of points relationship, which effectively preserves the geometric details between points. We evaluate our method on both public datasets (the Urban Semantic 3D (US3D) dataset and Semantic3D benchmark dataset) and our new collected Unmanned Aerial Vehicle (UAV) based LiDAR point cloud data of the campus of Guangdong University of Technology. In comparison with four available state-of-the-art methods, our methods ranked first place regarding both efficiency and accuracy. It was observed on the public datasets that with a 2% increase in classification accuracy, over 26% improvement of efficiency was achieved at the same time compared to the second efficient method. Its potential value is also tested on the newly collected point cloud data with over 91% of classification accuracy and 154 ms of processing time.

Multi‐scale feature learning and temporal probing strategy for one‐stage temporal action localization

The aim of temporal action localization (TAL) is to determine the start and end frames of an action in a video. In recent years, TAL has attracted considerable attention because of its increasing applications in video understanding and retrieval. However, precisely estimating the duration of an action in the temporal dimension is still a challenging problem. In this paper, we propose an effective one-stage TAL method based on a self-defined motion data structure, called a dense joint motion matrix (DJMM), and a novel temporal detection strategy. Our method provides three main contributions. First, compared with mainstream motion images, DJMMs can preserve more pre-processed motion features and provides more precise detail representations. Furthermore, DJMMs perfectly solve the temporal information loss problem caused by motion trajectory overlaps within a certain time period. Second, a spatial pyramid pooling (SPP) layer, which is widely used in the object detection and tracking fields, is innovatively incorporated into the proposed method for multi-scale feature learning. Moreover, the SPP layer enables the backbone convolutional neural network (CNN) to receive DJMMs of any size in the temporal dimension. Third, a large-scale-first temporal detection strategy inspired by a well-developed Chinese text segmentation algorithm is proposed to address long-duration videos. Our method is evaluated on two benchmark data sets and one self-collected data set: Florence-3D, UTKinect-Action3D and HanYue-3D. The experimental results show that our method achieves competitive action recognition accuracy and high TAL precision, and its time efficiency and few-shot learning capabilities enable it to be utilized for real-time surveillance.

Broad-UNet: Multi-scale feature learning for nowcasting tasks

Weather nowcasting consists of predicting meteorological components in the short term at high spatial resolutions. Due to its influence in many human activities, accurate nowcasting has recently gained plenty of attention. In this paper, we treat the nowcasting problem as an image-to-image translation problem using satellite imagery. We introduce Broad-UNet, a novel architecture based on the core UNet model, to efficiently address this problem. In particular, the proposed Broad-UNet is equipped with asymmetric parallel convolutions as well as Atrous Spatial Pyramid Pooling (ASPP) module. In this way, the Broad-UNet model learns more complex patterns by combining multi-scale features while using fewer parameters than the core UNet model. The proposed model is applied on two different nowcasting tasks, i.e. precipitation maps and cloud cover nowcasting. The obtained numerical results show that the introduced Broad-UNet model performs more accurate predictions compared to the other examined architectures.

MSF-Net: Multi-Scale Feature Learning Network for Classification of Surface Defects of Multifarious Sizes.

In the field of surface defect detection, the scale difference of product surface defects is often huge. The existing defect detection methods based on Convolutional Neural Networks (CNNs) are more inclined to express macro and abstract features, and the ability to express local and small defects is insufficient, resulting in an imbalance of feature expression capabilities. In this paper, a Multi-Scale Feature Learning Network (MSF-Net) based on Dual Module Feature (DMF) extractor is proposed. DMF extractor is mainly composed of optimized Concatenated Rectified Linear Units (CReLUs) and optimized Inception feature extraction modules, which increases the diversity of feature receptive fields while reducing the amount of calculation; the feature maps of the middle layer with different sizes of receptive fields are merged to increase the richness of the receptive fields of the last layer of feature maps; the residual shortcut connections, batch normalization layer and average pooling layer are used to replace the fully connected layer to improve training efficiency, and make the multi-scale feature learning ability more balanced at the same time. Two representative multi-scale defect data sets are used for experiments, and the experimental results verify the advancement and effectiveness of the proposed MSF-Net in the detection of surface defects with multi-scale features.

Person re-identification based on multi-scale feature learning

Extracting discriminative pedestrian features is an effective method in person re-identification. Most person re-identification works focus on extracting abstract features from the high-layer of the network, but ignore the middle-layer features, thus reducing the identity accuracy. To solve this problem, we construct a Smooth Aggregation Module (SAM) to extract, align, and fuse the feature maps in the middle-layer of the network to make up for the lack of detailed information in the high-level network features, and propose an Omni-Scale Feature Aggregation method (OSFA)11Source codes are available at https://github.com/lyy973/OSFA.git. to jointly learn the abstract features and local detail features. Considering that the intra-class distance in person re-identification should be less than the inter-class distance, we combine multiple losses to constrain the model. We evaluate the performance of our method on three standard benchmark datasets: Market-1501, CUHK03 (both detected and labeled) and DukeMTMC-reID, and experimental results show that our method is superior to the state-of-the-art approaches.

Multiscale Feature Learning Based on Enhanced Feature Pyramid for Vehicle Detection

Vehicle detection is a crucial task in autonomous driving systems. Due to large variance of scales and heavy occlusion of vehicle in an image, this task is still a challenging problem. Recent vehicle detection methods typically exploit feature pyramid to detect vehicles at different scales. However, the drawbacks in the design prevent the multiscale features from being completely exploited. This paper introduces a feature pyramid architecture to address this problem. In the proposed architecture, an improving region proposal network is designed to generate intermediate feature maps which are then used to add more discriminative representations to feature maps generated by the backbone network, as well as improving the computational cost of the network. To generate more discriminative feature representations, this paper introduces multilayer enhancement module to reweight feature representations of feature maps generated by the backbone network to increase the discrimination of foreground objects and background regions in each feature map. In addition, an adaptive RoI pooling module is proposed to pool features from all pyramid levels for each proposal and fuse them for the detection network. Experimental results on the KITTI vehicle detection benchmark and the PASCAL VOC 2007 car dataset show that the proposed approach obtains better detection performance compared with recent methods on vehicle detection.

AKRNet: A novel convolutional neural network with attentive kernel residual learning for feature learning of gearbox vibration signals

Vibration signals have been widely used for machine health monitoring and fault diagnosis. However, due to the complex working conditions, vibration signals collected from gearbox are generally nonlinear and non-stationary, which may contain multiple time scales and much noise. Considering these physical characteristics of vibration signals, in this paper, a novel deep neural network (DNN), attentive kernel residual network (AKRNet), is proposed for multi-scale feature learning from vibration signals. Firstly, multiple branches with different kernel widths are used to extract multi-scale features from vibration signals. Secondly, an attentive kernel selection is proposed to fuse the multiple branches features, where dynamic selection is developed to adaptively highlight the informative feature maps, while suppress the useless feature maps. Thirdly, an attentive residual block is developed to improve the feature learning performance, which not only can alleviate gradient vanishing, but also further enhances the impulsive features in feature maps. Finally, the effectiveness of AKRNet for feature learning of vibration signals is verified on two gearbox test rigs. The experimental results show that AKRNet has good capacity of feature learning on vibration signals. It performs better on gearbox fault diagnosis than other typical DNNs, e.g., stacked auto-encoders (SAE), one-dimensional CNN (1-D CNN) and residual network (ResNet).

Multimodal Infant Brain Segmentation by Fuzzy-Informed Deep Learning

Magnetic resonance imaging (MRI) is a prevailing method of modal infant brain tissue analysis that precisely segments brain tissue and is vitally important for diagnosis, remediation, and analysis of early brain development. To achieve such segmentation is challenging, particularly for the brain of a six-month-old, owing to several factors: poor image quality; isointense contrast between white and gray matter and the simple incomplete volume consequence of a tiny brain size; and discrepancies in brain tissues, illumination settings, and the vagarious region. This article addresses these challenges with a fuzzy-informed deep learning segmentation network that takes T1- and T2-weighted MRIs as inputs. First, a fuzzy logic layer encodes input to the fuzzy domain. Second, a volumetric fuzzy pooling (VFP) layer models the local fuzziness of the volumetric convolutional maps by applying fuzzification, accumulation, and defuzzification on the adjacency feature map neighborhoods. Third, the VFP layer is employed to design the fuzzy-enabled multiscale feature learning module to enable the extraction of brain features in different receptive fields. Finally, we redesign the Project & Excite module using the VPF layer to enable modeling uncertainty during feature recalibration, and a comprehensive training paradigm is used to learn the ideal parameters of every building block. Extensive experimental comparative studies substantiate the efficiency and accuracy of the proposed model in terms of different evaluation metrics to solve multimodal infant brain segmentation problems on the iSeg-2017 dataset.

Open clusters identifying by multi-scale density feature learning

Open clusters (OCs) are important objects in exploring the structure and history of the Milky Way. Large amount of sky survey data can be used to detect OCs. However, analyzing these data manually has become a bottleneck of OC identification. This study proposes a multi-scale density feature learning (MSDFL), which includes the open cluster kernel density map to visualize the features of OCs; and open cluster identifying network, which is a deep learning model used to perform identifying with the maps. A test set and experimental region are utilized to evaluate the effectiveness of our method. For OCs that stand out as significant overdensities, experimental results show that the MSDFL method can achieve the accuracy of 94%. Lastly, the proposed method can successfully identify real OCs in the experimental sky region. The code is available at: https://gitee.com/colab_worker/cluster_search .