Multi-scale Feature Extraction Module Research Articles

Efficient video quality assessment with deeper spatiotemporal feature extraction and integration

The challenge of video quality assessment (VQA) modeling for user-generated content (UGC) (i.e., UGC-VQA) is how to accurately extract discriminative features and elaborately quantify interfeature interactions by following the behavior patterns of human eye-brain vision perception. To address this issue, we propose the Deeper Spatial–Temporal Scoring Network (DSTS-Net) to give a precise VQA. Concretely, we first deploy the multiscale feature extraction module to characterize content-aware features accounting for nonlinear reverse hierarchy theory in video perception process, which is not fully considered in the reported UGC-VQA models. Hierarchical handcraft and semantic features are simultaneously considered using content adaptive weighting. Second, we develop a feature integration structure, i.e., deeper gated recurrent unit (DGRU), to fully imitate the interfeature interactions in visionary perception, including feedforward and feedback processes. Third, the dual DGRU structure is employed to further account for interframe interactions of hierarchical features, imitating the nonlinearity of perception as much as possible. Finally, improved pooling is achieved in the local adaptive smoothing module accounting for the temporal hysteresis. Holistic validation of the proposed method on four public challenging UGC-VQA datasets presents a comparable performance over the state-of-the-art no-reference VQA methods, especially our method can give an accurate prediction of the low quality videos with weak temporal correlation. To promote reproducible research and public evaluation, an implementation of our method has been made available online: https://github.com/liu0527aa/DSTS-Net.

A novel ResNet101 model based on dense dilated convolution for image classification

Image classification plays an important role in computer vision. The existing convolutional neural network methods have some problems during image classification process, such as low accuracy of tumor classification and poor ability of feature expression and feature extraction. Therefore, we propose a novel ResNet101 model based on dense dilated convolution for medical liver tumors classification. The multi-scale feature extraction module is used to extract multi-scale features of images, and the receptive field of the network is increased. The depth feature extraction module is used to reduce background noise information and focus on effective features of the focal region. To obtain broader and deeper semantic information, a dense dilated convolution module is deployed in the network. This module combines the advantages of Inception, residual structure, and multi-scale dilated convolution to obtain a deeper level of feature information without causing gradient explosion and gradient disappearance. To solve the common feature loss problems in the classification network, the up- down-sampling module in the network is improved, and multiple convolution kernels with different scales are cascaded to widen the network, which can effectively avoid feature loss. Finally, experiments are carried out on the proposed method. Compared with the existing mainstream classification networks, the proposed method can improve the classification performance, and finally achieve accurate classification of liver tumors. The effectiveness of the proposed method is further verified by ablation experiments.HighlightsThe multi-scale feature extraction module is introduced to extract multi-scale features of images, it can extract deep context information of the lesion region and surrounding tissues to enhance the feature extraction ability of the network.The depth feature extraction module is used to focus on the local features of the lesion region from both channel and space, weaken the influence of irrelevant information, and strengthen the recognition ability of the lesion region.The feature extraction module is enhanced by the parallel structure of dense dilated convolution, and the deeper feature information is obtained without losing the image feature information to improve the classification accuracy.

Hyperspectral Image Classification Based on SE-Res2Net and Multi-Scale Spatial Spectral Fusion Attention Mechanism

<p indent=0mm>In order to extract more discriminative features for hyperspectral image and prevent the network from degrading caused by deepening, a novel multi-scale feature extraction module SE-Res2Net based on the new dimensional residual network (Res2Net) and squeeze and exception network (SENet), and a multi-scale spectral-spatial fusion attention module is developed for hyperspectral image classification. In order to overcome the degradation problem caused by network deepening, the SE-Res2Net module uses channel grouping to extract fine-grained multi-scale features of hyperspectral images, and gets multiple receptive fields of different granularity. Then, the channel optimization module is employed to quantify the importance of the feature maps at the channel level. In order to optimize the features from spatial and spectral dimensions simultaneously, a multi-scale spectral-spatial fusion attention module is designed to mine the relationship between different spatial positions and different spectral dimensions at different scales using asymmetric convolution, which can not only reduce the computation, but also effectively extract the discriminative spectral-spatial fusion features, and further improve the accuracy of hyperspectral image classification. Comparison experiments on three public datasets, Indian Pines, University of Pavia and Grss_dfc_2013 show that the proposed method has higher overall accuracy (OA), average accuracy (AA) and Kappa coefficient compared to other state-of-the-art deep networks.

A Deep and Supervised Atrous Convolutional Model for Multi-Focus Image Fusion

Due to the limitations of equipment and technology, few images captured by a single sensor can describe all the information in a scene. Multi-focus image fusion techniques provide an effective means of integrating images acquired from different sensors to produce a single fused image that is richer in scene information. To obtain better fusion results, we propose a novel deep learning structure for multi-focus image fusion to achieve end-to-end network structure without post-processing procedures. Specifically, an innovative Siamese multi-scale feature extraction module was introduced to extract multi-scale image features. Besides, aiming to enhance the quality of the fused image, we further designed a new adaptive fusion strategy by using the attention mechanism instead of following the previous manual design. Extensive experimental results confirm that the proposed algorithm is more effective, which surpasses the existing state-of-the-art multi-focus image fusion methods both quantitatively and qualitatively. Furthermore, additional expanded experiment conducted on the infrared and visible image dataset demonstrates that the proposed fusion framework is also capable of performing other fusion tasks with excellent fusion performance. The code of the proposed fusion method is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/govenda/</uri> MSPA-Fuse.

A new end-to-end image dehazing algorithm based on residual attention mechanism

Traditional image dehazing algorithms based on prior knowledge and deep learning rely on the atmospheric scattering model and are easy to cause color distortion and incomplete dehazing. To solve these problems, an end-to-end image dehazing algorithm based on residual attention mechanism is proposed in this paper. The network includes four modules: encoder, multi-scale feature extraction, feature fusion and decoder. The encoder module encodes the input haze image into feature map, which is convenient for subsequent feature extraction and reduces memory consumption; the multi-scale feature extraction module includes residual smoothed dilated convolution module, residual block and efficient channel attention, which can expand the receptive field and extract different scale features by filtering and weighting; the feature fusion module with efficient channel attention adjusts the channel weight dynamically, acquires rich context information and suppresses redundant information so as to enhance the ability to extract haze density image of the network; finally, the encoder module maps the fused feature nonlinearly to obtain the haze density image and then restores the haze free image. The qualitative and quantitative tests based on SOTS test set and natural haze images show good objective and subjective evaluation results. This algorithm improves the problems of color distortion and incomplete dehazing effectively.

ARMS Net: Overlapping chromosome segmentation based on Adaptive Receptive field Multi-Scale network

Karyotype analysis has become the crux for diagnosis of genetic diseases, which requires automatic precise identification of quantitative and structural abnormalities. Thus, segmentation of chromosomes from microscope captured photos by image processing technique can help promote the automatic identification of chromosomes. However, chromosomes are often curled and overlapped together randomly in different images, which makes overlapping chromosome segmentation the hot topic in karyotype analysis. Herein an Adaptive Receptive field Multi-Scale network (ARMS Net) based on UNet architecture is proposed. The number of pooling operations is optimized to balance the requirements of deep semantic information extraction and high precision segmentation. The adaptive multi-scale feature extraction module is designed to replace the standard convolution at the bottom of UNet, such that the receptive fields can adaptively match the size of feature map. Besides, an adaptive smooth weighted cross entropy loss function is defined to resolve category imbalance issue. Experimental results show that the Intersection of Union (IoU) score of ARMS Net segmented overlapping area is 99.45%, which is 3.2% higher than that achieved by UNet (96.38%), and 10.1% higher than that achieved by CE-Net (90.35%). In a word, ARMS Net is expected to be used as the backbone network for chromosome instance segmentation in its end-to-end identification.

Automatic mapping of urban green spaces using a geospatial neural network

ABSTRACT Detailed and precise urban green spaces (UGS) maps provide essential data for the sustainable urban development and related studies (e.g. heatwave events, heat related health risk, urban flooding, urban biodiversity and ecosystem services). However, remote sensing of mapping UGS is challenging due to the existence of mixed pixels and the cost and difficulty of collecting quality training data. This study presents a neural network-based automatic mapping method of UGS that integrates the use of Sentinel-2A satellite images and crowdsourced geospatial big data. The proposed neural network consists of three parts: (i) a multi-scale feature extraction module; (ii) a multi-modal information fuse module; and (iii) and a boundary enhancement module. The results showed that the proposed method achieved a high overall classification accuracy of 94.6%, which presents a clear UGS structure of a large scale. This study provides a fresh insight into how remote-sensing and crowdsourced geospatial big data can be integrated to improve urban mapping of green spaces through neural network.

Global context-aware multi-scale features aggregative network for salient object detection

Deep convolutional neural networks have gained aggressive success in salient object detection. This paper uses the Multi-Scale Feature Extraction Module (MFEM) for each backbone level to get multi-scale contextual knowledge. We propose the Cross Feature Aggregation Modules(CFAM) to integrate the various features from adjacent levels, which comparatively propagate less noise due to small up-/down sampling rates. To further refine individual-level integrated features, we design Self Interactive Modules (SIRM) at each decoder stage. The SIRM utilizes the spatial- and channel-wise attention to suppress the non-salient regions while assigning more weights to the foreground salient object to visualize the submissive regions (i.e., some salient regions looking like non-salient regions) of the salient objects. Our network can enhance size-varying objects’ illustration proficiency by adopting the multi-scale feature extraction capability in each module. Besides, we develop the Global Context Flow Module (GCFM) to get the global context knowledge at different points in the decoder, which aims to acquire the association among different salient regions and mitigate the dilution of high-level features. Our proposed model (i.e., GCMANet) follows a supervised way to generate the saliency maps. The results produced over publicly available datasets verify that our model outperforms its counterparts in quantitative and qualitative measurements.

Convolutional Neural Network for Crowd Counting on Metro Platforms

Owing to the increased use of urban rail transit, the flow of passengers on metro platforms tends to increase sharply during peak periods. Monitoring passenger flow in such areas is important for security-related reasons. In this paper, in order to solve the problem of metro platform passenger flow detection, we propose a CNN (convolutional neural network)-based network called the MP (metro platform)-CNN to accurately count people on metro platforms. The proposed method is composed of three major components: a group of convolutional neural networks is used on the front end to extract image features, a multiscale feature extraction module is used to enhance multiscale features, and transposed convolution is used for upsampling to generate a high-quality density map. Currently, existing crowd-counting datasets do not adequately cover all of the challenging situations considered in this study. Therefore, we collected images from surveillance videos of a metro platform to form a dataset containing 627 images, with 9243 annotated heads. The results of the extensive experiments showed that our method performed well on the self-built dataset and the estimation error was minimum. Moreover, the proposed method could compete with other methods on four standard crowd-counting datasets.

End-to-end multispectral image compression framework based on adaptive multiscale feature extraction

Multispectral image compression can considerably reduce the volume of data and promote their application. However, conventional single-scale compression schemes, such as JPEG2000 and three-dimensional set partitioning in hierarchical tree (3D-SPIHT), do not accurately preserve the features of images due to the complex features of multispectral images. A compression framework based on adaptive multiscale feature extraction with a convolutional neural network is proposed. First, an adaptive multiscale feature extraction module, which is the basic component of the compression framework, is designed to extract the multiscale spatial–spectral features of the multispectral images and adaptively adjust the weights of the features according to the content of the images. Second, the encoder, which is composed of multiscale feature extraction modules, extracts the multiscale spatial–spectral features of the multispectral images, and the extracted features are quantized and encoded by the quantizer and the entropy coder to generate a compressed bitstream. Third, the decoder, which is structurally similar to the encoder, is utilized to recover the images. The rate-distortion optimizer is embedded in the encoder to control the trade-off between the rate loss and the distortion. The results of these experiments on multispectral images of the Landsat 8 satellite and the WorldView-3 satellite validate the better performance of our compression framework compared with the performances of conventional schemes, including JPEG2000 and 3D-SPIHT. In order to further verify the effectiveness of multiscale features, the framework is compared with a single-scale compression algorithm based on deep learning, the experimental results validate that the performance of the single-scale compression algorithm superior to the conventional schemes but inferior to our multiscale algorithm, which indicates that the multiscale features can significantly improve the performance of the compression algorithm.

MMMNet: An End-to-End Multi-Task Deep Convolution Neural Network With Multi-Scale and Multi-Hierarchy Fusion for Blind Image Quality Assessment

As the evaluation of image quality depends on the human visual system (HVS), many existing image quality assessment (IQA) methods focus on modeling the HVS to account for subjective perception. The visual attention of the HVS makes humans more sensitive to distortion on the attended regions than on regions which are not the focus of attention. Therefore, we propose an end-to-end multi-task deep convolution neural network with multi-scale and multi-hierarchy fusion (MMMNet), in which the IQA and saliency subtasks are jointly optimized to improve saliency-guided IQA performance. Particularly, the incorporation of saliency information is achieved by fusing saliency features with IQA features hierarchically to progressively improve the IQA features over network depth. A multi-scale feature extraction module (MSFE) is proposed to provide effective saliency features for the IQA network. Based on the saliency fusion, MMMNet introduces an auxiliary saliency task, achieving the multi-task learning to improve the generalization of the IQA task. Experimental results show that MMMNet achieves state-of-the-art performance and strong generalization ability on IQA databases.

Light-field image super-resolution based on multi-scale feature fusion

As a new generation of the imaging device, light-field camera can simultaneously capture the spatial position and incident angle of light rays. However, the recorded light-field has a trade-off between spatial resolution and angular resolution. Especially the application range of light-field cameras is restricted by the limited spatial resolution of sub-aperture images. Therefore, a light-field super-resolution neural network that fuses multi-scale features to obtain super-resolved light-field is proposed in this paper. The deep-learning-based network framework contains three major modules: multi-scale feature extraction, global feature fusion, and up-sampling. Firstly, inherent structural features in the 4D light-field are learned through the multi-scale feature extraction module, and then the fusion module is exploited for feature fusion and enhancement. Finally, the up-sampling module is used to achieve light-field super-resolution. The experimental results on the synthetic light-field dataset and real-world light-field dataset showed that this method outperforms other state-of-the-art methods in both visual and numerical evaluations. In addition, the super-resolved light-field images were applied to depth estimation in this paper, the results illustrated that the disparity map was enhanced through the light-field spatial super-resolution.

A SIFT-FREAK Based Framework for Coastline Image Stitching

The quality of the images collected by the coastal zone video surveillance equipment is seriously degraded due to the sea fog, which directly affects the analysis of the image. Therefore, the study of the costal image dehazing method is of great significance to the related research of the coastal zone. Costal image has the characteristics of large sky area and monotonous color. The traditional method based on atmospheric scattering physics model is not suitable for this kind of image for block effect and color distortion. In this paper, we introduce the generative adversarial mechanism into sea fog image defogging, and propose a coastal image dehazing network based on it. The proposed model includes a generative network and a discriminative model, and is trained by adversarial mechanism. The generative model is composed of multi-scale feature extraction module and residual connection module. The discriminative network consists of two sub-networks of receptive field of different sizes.

A solanaceae disease recognition model based on SE-Inception

Aiming at the diseases of tomato and eggplant, we present a solanaceae disease recognition model based on SE-Inception. Our model uses batch normalization layer (BN) to accelerate network convergence. Besides, SE-Inception structure and multi-scale feature extraction module is adopted to improve accuracy of this model. Our sample data set consists of 4 disease categories including whitefly, powdery mildew, yellow smut, cotton blight. We also add healthy leaves into it. In order to reduce overfitting, the data set is expanded by the data enhancement method of translation, rotation and flip. Experiments show that the average recognition accuracy of this model is 98.29% and the model size is 14.68 MB on our constructed dataset. In addition, in order to verify the robustness of this model, it was also verified on the public data set of PlantVillage, and the top-1, top-5 accuracy and the size of our proposed model is 99.27%, 99.99% and 14.8 MB respectively. Moreover, we implemented a solanaceae disease image recognition system using this model based on the Android. The accuracy of average recognition and the recognition time of a single photo are 95.09% and 227 ms, respectively. Our constructed model has a small number of parameters with maintaining high accuracy, which can meet the needs of automatic recognition of disease images on mobile devices. Data and code are available at https://github.com/Jujube-sun/diseaseRecognition.

Enhanced Capsule Network for Medical image classification.

Nowadays, cancer has become a major threat to people's lives and health. Convolutional neural network (CNN) has been used for cancer early identification, which cannot achieve the desired results in some cases, such as images with affine transformation. Due to robustness to rotation and affine transformation, capsule network can effectively solve this problem of CNN and achieve the expected performance with less training data, which are very important for medical image analysis. In this paper, an enhanced capsule network is proposed for medical image classification. For the proposed capsule network, the feature decomposition module and multi-scale feature extraction module are introduced into the basic capsule network. The feature decomposition module is presented to extract richer features, which reduces the amount of calculation and speeds up the network convergence. The multi-scale feature extraction module is used to extract important information in the low-level capsules, which guarantees the extracted features to be transmitted to the high-level capsules. The proposed capsule network was applied on PatchCamelyon (PCam) dataset. Experimental results show that it can obtain good performance for medical image classification task, which provides good inspiration for other image classification tasks.

CAGNet: Content-Aware Guidance for Salient Object Detection

Beneficial from Fully Convolutional Neural Networks (FCNs), saliency detection methods have achieved promising results. However, it is still challenging to learn effective features for detecting salient objects in complicated scenarios, in which i) non-salient regions may have “salient-like” appearance; ii) the salient objects may have different-looking regions. To handle these complex scenarios, we propose a Feature Guide Network which exploits the nature of low-level and high-level features to i) make foreground and background regions more distinct and suppress the non-salient regions which have “salient-like” appearance; ii) assign foreground label to different-looking salient regions. Furthermore, we utilize a Multi-scale Feature Extraction Module (MFEM) for each level of abstraction to obtain multi-scale contextual information. Finally, we design a loss function which outperforms the widely used Cross-entropy loss. By adopting four different pre-trained models as the backbone, we prove that our method is very general with respect to the choice of the backbone model. Experiments on six challenging datasets demonstrate that our method achieves the state-of-the-art performance in terms of different evaluation metrics. Additionally, our approach contains fewer parameters than the existing ones, does not need any post-processing, and runs fast at a real-time speed of 28 FPS when processing a 480 × 480 image.

Saliency detection using a deep conditional random field network

Saliency detection has made remarkable progress along with the development of deep learning. While how to integrate the low-level intrinsic context with high-level semantic information to keep the object boundary sharp and restrain the background noise is still a challenging problem. Many attempts on network structures and refinement strategies have been explored, such as using Conditional Random Field (CRF) to improve the accuracy of saliency map, but it is independent from the deep network and cannot be trained end-to-end. To tackle this issue, we propose a novel Deep Conditional Random Field network (DCRF) to take both deep feature and neighbor information into consideration. First, Multi-scale Feature Extraction Module (MFEM) is adopted to capture the low level texture and high level semantic features, multi-stacks of deconvolution layers are employed to improve the spatial resolution of deep layers. Then we employ Backward Optimization Module (BOM) to guide shallower layers by high-level location and shape information derived from deeper layers, which intrinsically enhance the representational capacity of low-level features. Finally, a Deep Conditional Random Field Module (DCRFM) with unary and pairwise potentials is designed to concentrate on spatial neighbor relations to obtain a compact and uniformed saliency map. Extensive experimental results on 5 datasets in terms of 6 evaluation metrics demonstrate that the proposed method achieves state-of-the-art performance.

Multi-stream attentive generative adversarial network for dynamic scene deblurring

Image deblurring methods based on deep learning have been proved to be effective for dynamic scene deblurring. Nevertheless, the most existing methods usually fail to deal with severe and complex dynamic scene blurs. In this paper, we propose a novel multi-stream attentive generative adversarial network (MSA-GAN) for solving the problems. Firstly, we design a novel attentive guidance module for generating an attention map which can guide to pay more attention on the severe blur regions or objects and their edge or texture structures in the feature extraction and deblurring process. Secondly, we propose the multi-stream and multi-scale feature extraction strategy and design an attentive and multi-scale feature extraction module, in which we design a multi-scale residual block for extracting the multi-scale features and capturing more information of complex blurs. Thirdly, we propose a multi-scale feature fusion strategy and the strategy is implemented in the multi-stream and multi-scale feature fusion module through the designed gate structure, which can adaptively learn and fuse multi-scale features from different streams. Finally, The extensive experiments are performed on the GoPro dynamic scene dataset and real image data, and the experimental results both quantitatively and qualitatively have demonstrated that the proposed method outperform the recent state-of-the-art methods.

Sample generation of semi‐automatic pavement crack labelling and robustness in detection of pavement diseases

Recent convolutional neural networks have made significant advancements in the detection of road cracks. However, the lack of accurate crack training data reduces the generalisation ability of the deep model. In this Letter, a semi-automatic pavement crack labelling algorithm is proposed to solve the problem of insufficient training data. First, the modified C-V model is used to obtain the preliminary segmentation results. Second, the direction of the initial segmentation area is calculated by the ellipse fitting method, and the preliminary segmentation results are used as samples for accurate labelling. Finally, a multi-scale feature extraction module is proposed for learning rich deep convolutional features, which allows the acquired crack features under a complex background to be more discriminant. The experimental results were compared with the manual marking method, and this method can achieve accurate marking of crack images with a low amount of interaction, thereby significantly reducing the cost of ground-truth making. The results of the validation and comparison experiments on test data sets indicate that the proposed method can not only effectively identify cracks, but also overcome the interference of many factors in the environment.

Identification of tea leaf diseases by using an improved deep convolutional neural network

Accurate and rapid identification of tea leaf diseases is beneficial to their prevention and control. This study proposes a method based on an improved deep convolutional neural network (CNN) for tea leaf disease identification. A multiscale feature extraction module is added into the improved deep CNN of a CIFAR10-quick model to improve the ability to automatically extract image features of different tea leaf diseases. The depthwise separable convolution is used to reduce the number of model parameters and accelerate the calculation of the model. Experimental results show that the average identification accuracy of the proposed method is 92.5%, which is higher than that of traditional machine learning methods and classical deep learning methods. The number of parameters and the convergence iteration times of the improved model are significantly lower than those of VGG16 and AlexNet deep learning network models.