Down-sampling Operations Research Articles

Learning U-Net Based Multi-Scale Features in Encoding-Decoding for MR Image Brain Tissue Segmentation.

Accurate brain tissue segmentation of MRI is vital to diagnosis aiding, treatment planning, and neurologic condition monitoring. As an excellent convolutional neural network (CNN), U-Net is widely used in MR image segmentation as it usually generates high-precision features. However, the performance of U-Net is considerably restricted due to the variable shapes of the segmented targets in MRI and the information loss of down-sampling and up-sampling operations. Therefore, we propose a novel network by introducing spatial and channel dimensions-based multi-scale feature information extractors into its encoding-decoding framework, which is helpful in extracting rich multi-scale features while highlighting the details of higher-level features in the encoding part, and recovering the corresponding localization to a higher resolution layer in the decoding part. Concretely, we propose two information extractors, multi-branch pooling, called MP, in the encoding part, and multi-branch dense prediction, called MDP, in the decoding part, to extract multi-scale features. Additionally, we designed a new multi-branch output structure with MDP in the decoding part to form more accurate edge-preserving predicting maps by integrating the dense adjacent prediction features at different scales. Finally, the proposed method is tested on datasets MRbrainS13, IBSR18, and ISeg2017. We find that the proposed network performs higher accuracy in segmenting MRI brain tissues and it is better than the leading method of 2018 at the segmentation of GM and CSF. Therefore, it can be a useful tool for diagnostic applications, such as brain MRI segmentation and diagnosing.

ThriftyNets: Convolutional Neural Networks with Tiny Parameter Budget

Deep Neural Networks are state-of-the-art in a large number of challenges in machine learning. However, to reach the best performance they require a huge pool of parameters. Indeed, typical deep convolutional architectures present an increasing number of feature maps as we go deeper in the network, whereas spatial resolution of inputs is decreased through downsampling operations. This means that most of the parameters lay in the final layers, while a large portion of the computations are performed by a small fraction of the total parameters in the first layers. In an effort to use every parameter of a network at its maximum, we propose a new convolutional neural network architecture, called ThriftyNet. In ThriftyNet, only one convolutional layer is defined and used recursively, leading to a maximal parameter factorization. In complement, normalization, non-linearities, downsamplings and shortcut ensure sufficient expressivity of the model. ThriftyNet achieves competitive performance on a tiny parameters budget, exceeding 91% accuracy on CIFAR-10 with less than 40 k parameters in total, 74.3% on CIFAR-100 with less than 600 k parameters, and 67.1% On ImageNet ILSVRC 2012 with no more than 4.15 M parameters. However, the proposed method typically requires more computations than existing counterparts.

An automated zizania quality grading method based on deep classification model

Zizania, one of aquatic vegetables, needs to be graded before entering market for assuring the product quality. However, it is time-consuming, tedious, labor-intensive, inaccurate and expensive to assess qualitatively and grade zizanias manually. This paper gives an effective solution to automatically classify fresh zizania into two categories, high quality and defective quality, by using the deep learnt features from the appearances. A new architecture of convolutional neural network, called LightNet, has been proposed and described. Specifically, it is composed of many compressed blocks, which is designed to reduce the computation complexity mainly by converting serial down-sampling and convolution operation to parallel structure. We evaluate the proposed architecture on the zizania image dataset collected by ourselves and integrate the algorithm in the automatic grading device. The experimental results show that the accuracy rate achieves 95.62% and the speed of inference quality is around 47 ms per zizania image. The proposed LightNet for automate classification has less parameters and lower computation complexity than popular networks, while maintaining the comparable accuracy in the task of grading zizanias. It obtains 99.31% accuracy in the task of grading apples. The result proves that it can be extended to other tasks about classification.

MFANet: A Multi-Level Feature Aggregation Network for Semantic Segmentation of Land Cover

Detailed information regarding land utilization/cover is a valuable resource in various fields. In recent years, remote sensing images, especially aerial images, have become higher in resolution and larger span in time and space, and the phenomenon that the objects in an identical category may yield a different spectrum would lead to the fact that relying on spectral features only is often insufficient to accurately segment the target objects. In convolutional neural networks, down-sampling operations are usually used to extract abstract semantic features, which leads to loss of details and fuzzy edges. To solve these problems, the paper proposes a Multi-level Feature Aggregation Network (MFANet), which is improved in two aspects: deep feature extraction and up-sampling feature fusion. Firstly, the proposed Channel Feature Compression module extracts the deep features and filters the redundant channel information from the backbone to optimize the learned context. Secondly, the proposed Multi-level Feature Aggregation Upsample module nestedly uses the idea that high-level features provide guidance information for low-level features, which is of great significance for positioning the restoration of high-resolution remote sensing images. Finally, the proposed Channel Ladder Refinement module is used to refine the restored high-resolution feature maps. Experimental results show that the proposed method achieves state-of-the-art performance 86.45% mean IOU on LandCover dataset.

Multi-Scale Context-Guided Deep Network for Automated Lesion Segmentation With Endoscopy Images of Gastrointestinal Tract.

Accurate lesion segmentation based on endoscopy images is a fundamental task for the automated diagnosis of gastrointestinal tract (GI Tract) diseases. Previous studies usually use hand-crafted features for representing endoscopy images, while feature definition and lesion segmentation are treated as two standalone tasks. Due to the possible heterogeneity between features and segmentation models, these methods often result in sub-optimal performance. Several fully convolutional networks have been recently developed to jointly perform feature learning and model training for GI Tract disease diagnosis. However, they generally ignore local spatial details of endoscopy images, as down-sampling operations (e.g., pooling and convolutional striding) may result in irreversible loss of image spatial information. To this end, we propose a multi-scale context-guided deep network (MCNet) for end-to-end lesion segmentation of endoscopy images in GI Tract, where both global and local contexts are captured as guidance for model training. Specifically, one global subnetwork is designed to extract the global structure and high-level semantic context of each input image. Then we further design two cascaded local subnetworks based on output feature maps of the global subnetwork, aiming to capture both local appearance information and relatively high-level semantic information in a multi-scale manner. Those feature maps learned by three subnetworks are further fused for the subsequent task of lesion segmentation. We have evaluated the proposed MCNet on 1,310 endoscopy images from the public EndoVis-Ab and CVC-ClinicDB datasets for abnormal segmentation and polyp segmentation, respectively. Experimental results demonstrate that MCNet achieves [Formula: see text] and [Formula: see text] mean intersection over union (mIoU) on two datasets, respectively, outperforming several state-of-the-art approaches in automated lesion segmentation with endoscopy images of GI Tract.

A Serial-Parallel Self-Attention Network Joint With Multi-Scale Dilated Convolution

Semantic segmentation is a high-level task in the field of computer vision, which paves the way for the realization of a complete understanding of the scene, and has been widely used in automatic driving, human-computer interaction, virtual reality, and other aspects. Recently, the semantic segmentation method of convolutional neural networks with deep structure has been more accurate and efficient than other methods. However, there are some problems in these methods, such as the loss of information caused by the down-sampling operation, the lack of usage of image context information, and the neglect of the relationship between spatial features and channel features. To solve these problems, a novel self-attention network based on the series-parallel structure is proposed in the paper. Firstly, a multi-scale dilated convolution backbone network is constructed by combining the dilated convolution and the residual network, which makes up for the information loss caused by the restriction of the receptor field in the ordinary network and improves the richness of extracted features. Secondly, the self-attention modules are stacked with serial and parallel structures, which can effectively extract the contextual information of space, channel, and space-channel and fully integrate them. Finally, the proposed algorithm is tested extensively and compared with the existing classical algorithms. The experimental results show that the proposed algorithm achieves state-of-the-art performance on the public dataset.

AlignSeg: Feature-Aligned Segmentation Networks.

Aggregating features in terms of different convolutional blocks or contextual embeddings has been proven to be an effective way to strengthen feature representations for semantic segmentation. However, most of the current popular network architectures tend to ignore the misalignment issues during the feature aggregation process caused by step-by-step downsampling operations and indiscriminate contextual information fusion. In this paper, we explore the principles in addressing such feature misalignment issues and inventively propose Feature-Aligned Segmentation Networks (AlignSeg). AlignSeg consists of two primary modules, i.e., the Aligned Feature Aggregation (AlignFA) module and the Aligned Context Modeling (AlignCM) module. First, AlignFA adopts a simple learnable interpolation strategy to learn transformation offsets of pixels, which can effectively relieve the feature misalignment issue caused by multi-resolution feature aggregation. Second, with the contextual embeddings in hand, AlignCM enables each pixel to choose private custom contextual information adaptively, making the contextual embeddings be better aligned. We validate the effectiveness of our AlignSeg network with extensive experiments on Cityscapes and ADE20K, achieving new state-of-the-art mIoU scores of 82.6 and 45.95 percent, respectively. Our source code is available at https://github.com/speedinghzl/AlignSeg.

Hierarchical Multi-View Graph Pooling with Structure Learning

Graph Neural Networks (GNNs), which generalize deep neural networks to graph-structured data, have drawn considerable attention and achieved state-of-the-art performance in numerous graph related tasks. However, existing GNN models mainly focus on designing graph convolution operations. The graph pooling (or downsampling) operations, that play an important role in learning hierarchical representations, are usually overlooked. In this paper, we proposed a novel multi-view graph pooling operator dubbed as MVPool, which ranks nodes across different views with different contextual graph information. Meanwhile, attention mechanism is utilized to promote the collaboration of different views for generating robust node rankings. Then the pooling operation adaptively selects a subset of nodes to form an induced subgraph based on the ranking list. To preserve the underlying graph topological information, we further introduce a structure learning mechanism to learn a refined graph structure for the pooled graph at each layer. The proposed MVPool operator is a general strategy that can be integrated into various graph neural network architectures. By combining MVPool operator with graph neural networks, we perform hierarchical representation learning for both node and graph level classification as well as clustering tasks. Experimental results on nine widely used benchmarks demonstrate the effectiveness of our proposed model.

Block-Attentive Subpixel Prediction Networks for Computationally Efficient Image Restoration

Image restoration based on the Deep Convolutional Neural Network (CNN) based image restoration has demonstrated promising results in many sub-tasks, such as image super-resolution, compression artifacts removal, image denoising, and image enhancement. Compared to many CNN-based high-level vision tasks that predict sparse probabilities of each class, the CNN for image restoration requires dense pixel-level predictions with precise intensity-level values. Therefore, a minimum number of spatial pooling (or down-sampling) operations is required to maintain the image details. Therefore, designing a fast or lightweight model for image restoration is a difficult problem and is even more critical when the spatial resolution of an image becomes larger. In this paper, we propose a family of networks called Subpixel Prediction Networks (SPNs) that predict reshaped and spatially down-sampled block-wise tensors instead of raw images with full resolution. Under this scheme, spatial downsampling decreases the restoration performance less while making the network faster. We propose a novel Subpixel Block Attention (SBA) module that re-calibrates blockwise features to diminish blockwise discontinuity to increase the performance further. The experimental results reveal that these networks demonstrate good trade-offs between speed (number of computations) and restoration performance in the three image restoration tasks: image compression artifacts removal, color image denoising, and image enhancement.

Tracking Based Semi-Automatic Annotation for Scene Text Videos

Recently, video scene text detection has received increasing attention due to its comprehensive applications. However, the lack of annotated scene text video datasets has become one of the most important problems, which hinders the development of video scene text detection. The existing scene text video datasets are not large-scale due to the expensive cost caused by manual labeling. In addition, the text instances in these datasets are too clear to be a challenge. To address the above issues, we propose a tracking based semi-automatic labeling strategy for scene text videos in this paper. We get semi-automatic scene text annotation by labeling manually for the first frame and tracking automatically for the subsequent frames, which avoid the huge cost of manual labeling. Moreover, a paired low-quality scene text video dataset named Text-RBL is proposed, consisting of raw videos, blurry videos, and low-resolution videos, labeled by the proposed convenient semi-automatic labeling strategy. Through an averaging operation and bicubic down-sampling operation over the raw videos, we can efficiently obtain blurry videos and low-resolution videos paired with raw videos separately. To verify the effectiveness of Text-RBL, we propose a baseline model combined with the text detector and tracker for video scene text detection. Moreover, a failure detection scheme is designed to alleviate the baseline model drift issue caused by complex scenes. Extensive experiments demonstrate that Text-RBL with paired low-quality videos labeled by the semi-automatic method can significantly improve the performance of the text detector in low-quality scenes.

DA-RoadNet: A Dual-Attention Network for Road Extraction From High Resolution Satellite Imagery

Recent advances in deep-learning methods have shown extraordinary performance in road extraction from high resolution satellite imagery. However, most existing deep-learning network models yield discontinuous and incomplete results because of shadows and occlusions. To address this problem, a dual-attention road extraction network (DA-RoadNet) with a certain semantic reasoning ability is proposed. First, DA-RoadNet is designed based on a shallow encoder-to-decoder network with densely connected blocks, which can minimize the loss of road structure information caused by multiple down-sampling operations. Moreover, by constructing a novel attention mechanism module, the proposed network is able to explore and integrate the invisible correlations among road features with their global dependency in spatial and channel dimension respectively. Finally, considering that the proportion of road samples is small in the satellite imagery, a hybrid loss function is appended to handle class imbalance, which enables the network model to train stablely and avoid local optimum. The validation experiments using two open road datasets demonstrate that the proposed DA-RoadNet can effectively solve discontinuous problems and preserve integrity of the extracted roads, thus resulting in a higher accuracy of road extraction compared with other developed state-of-the-arts. The considerable performance on the two challenging benchmarks also proves the powerful generation ability of our method.

Robust Single-Image Super-Resolution via CNNs and TV-TV Minimization.

Single-image super-resolution is the process of increasing the resolution of an image, obtaining a high-resolution (HR) image from a low-resolution (LR) one. By leveraging large training datasets, convolutional neural networks (CNNs) currently achieve the state-of-the-art performance in this task. Yet, during testing/deployment, they fail to enforce consistency between the HR and LR images: if we downsample the output HR image, it never matches its LR input. Based on this observation, we propose to post-process the CNN outputs with an optimization problem that we call TV-TV minimization, which enforces consistency. As our extensive experiments show, such post-processing not only improves the quality of the images, in terms of PSNR and SSIM, but also makes the super-resolution task robust to operator mismatch, i.e., when the true downsampling operator is different from the one used to create the training dataset.

Single depth map super-resolution via a deep feedback network

Existing depth map-based super-resolution (SR) methods cannot achieve satisfactory results in depth map detail restoration. For example, boundaries of the depth map are always difficult to reconstruct effectively from the low-resolution (LR) guided depth map particularly at big magnification factors. In this paper, we present a novel super-resolution method for single depth map by introducing a deep feedback network (DFN), which can effectively enhance the feature representations at depth boundaries that utilize iterative up-sampling and down-sampling operations, building a deep feedback mechanism by projecting high-resolution (HR) representations to low-resolution spatial domain and then back-projecting to high-resolution spatial domain. The deep feedback (DF) block imitates the process of image degradation and reconstruction iteratively. The rich intermediate high-resolution features effectively tackle the problem of depth boundary ambiguity in depth map super-resolution. Extensive experimental results on the benchmark datasets show that our proposed DFN outperforms the state-of-the-art methods.

Multi‐scale feature fusion network for person re‐identification

Recently, it is becoming a challenging work for person re-identification due to the problems of occlusion, blurring and posture. The key of effective person re-identification is to capture sufficient detailed features of a person's appearance in images. Different from previous methods, our method mainly focuses on fusing different visual clues only depending on the features of different levels and scales without additional assistance. The major contributions of our paper are the mixed pooling strategy with different kernels and the mixed loss function. Firstly, we adopt ResNet50 as our backbone. We have slightly modified the backbone, which does not use the down-sampling operation at the beginning of stage 4. Inspired by pyramid pooling structure, we pass the outputs of Res4 and Res5 through the average pooling layer and max pooling layer with different kernels and strides separately. Secondly, we combine the averaged triplet losses and the averaged softmax losses as the final loss of the whole network. Extensive experiments on three datasets (CUHK3, Market1501, DukeMTMC-reID) show that compared with many state-of-the-art methods in recent years, our model achieve higher accuracy.

Hierarchical and Interactive Refinement Network for Edge-Preserving Salient Object Detection.

Salient object detection has undergone a very rapid development with the blooming of Deep Neural Network (DNN), which is usually taken as an important preprocessing procedure in various computer vision tasks. However, the down-sampling operations, such as pooling and striding, always make the final predictions blurred at edges, which has seriously degenerated the performance of salient object detection. In this paper, we propose a simple yet effective approach, i.e., Hierarchical and Interactive Refinement Network (HIRN), to preserve the edge structures in detecting salient objects. In particular, a novel multi-stage and dual-path network structure is designed to estimate the salient edges and regions from the low-level and high-level feature maps, respectively. As a result, the predicted regions will become more accurate by enhancing the weak responses at edges, while the predicted edges will become more semantic by suppressing the false positives in background. Once the salient maps of edges and regions are obtained at the output layers, a novel edge-guided inference algorithm is introduced to further filter the resulting regions along the predicted edges. Extensive experiments on several benchmark datasets have been conducted, in which the results show that our method significantly outperforms a variety of state-of-the-art approaches.

FMRI volume classification using a 3D convolutional neural network robust to shifted and scaled neuronal activations

Deep-learning methods based on deep neural networks (DNNs) have recently been successfully utilized in the analysis of neuroimaging data. A convolutional neural network (CNN) is a type of DNN that employs a convolution kernel that covers a local area of the input sample and moves across the sample to provide a feature map for the subsequent layers. In our study, we hypothesized that a 3D-CNN model with down-sampling operations such as pooling and/or stride would have the ability to extract robust feature maps from the shifted and scaled neuronal activations in a single functional MRI (fMRI) volume for the classification of task information associated with that volume. Thus, the 3D-CNN model would be able to ameliorate the potential misalignment of neuronal activations and over-/under-activation in local brain regions caused by imperfections in spatial alignment algorithms, confounded by variability in blood-oxygenation-level-dependent (BOLD) responses across sessions and/or subjects. To this end, the fMRI volumes acquired from four sensorimotor tasks (left-hand clenching, right-hand clenching, auditory attention, and visual stimulation) were used as input for our 3D-CNN model to classify task information using a single fMRI volume. The classification performance of the 3D-CNN was systematically evaluated using fMRI volumes obtained from various minimal preprocessing scenarios applied to raw fMRI volumes that excluded spatial normalization to a template and those obtained from full preprocessing that included spatial normalization. Alternative classifier models such as the 1D fully connected DNN (1D-fcDNN) and support vector machine (SVM) were also used for comparison. The classification performance was also assessed for several k-fold cross-validation (CV) schemes, including leave-one-subject-out CV (LOOCV). Overall, the classification results of the 3D-CNN model were superior to that of the 1D-fcDNN and SVM models. When using the fully-processed fMRI volumes with LOOCV, the mean error rates (± the standard error of the mean) for the 3D-CNN, 1D-fcDNN, and SVM models were 2.1% (± 0.9), 3.1% (± 1.2), and 4.1% (± 1.5), respectively (p = 0.041 from a one-way ANOVA). The error rates for 3-fold CV were higher (2.4% ± 1.0, 4.2% ± 1.3, and 10.1% ± 2.0; p < 0.0003 from a one-way ANOVA). The mean error rates also increased considerably using the raw fMRI 3D volume data without preprocessing (26.2% for the 3D-CNN, 75.0% for the 1D-fcDNN, and 75.0% for the SVM). Furthermore, the ability of the pre-trained 3D-CNN model to handle shifted and scaled neuronal activations was demonstrated in an online scenario for five-class classification (i.e., four sensorimotor tasks and the resting state) using the real-time fMRI of three participants. The resulting classification accuracy was 78.5% (± 1.4), 26.7% (± 5.9), and 21.5% (± 3.1) for the 3D-CNN, 1D-fcDNN, and SVM models, respectively. The superior performance of the 3D-CNN compared to the 1D-fcDNN was verified by analyzing the resulting feature maps and convolution filters that handled the shifted and scaled neuronal activations and by utilizing an independent public dataset from the Human Connectome Project.

Multi-path connected network for medical image segmentation

In recent years, deep learning has been successfully applied to medical image segmentation. However, as the network extends deeper, the consecutive downsampling operations will lead to more loss of spatial information. In addition, the limited data and diverse targets increase the difficulty for medical image segmentation. To address these issues, we propose a multi-path connected network (MCNet) for medical segmentation problems. It integrates multiple paths generated by pyramid pooling into the encoding phase to preserve semantic information and spatial details. We utilize multi-scale feature extractor block (MFE block) in the encoder to obtain large and multi-scale receptive fields. We evaluated MCNet on three medical datasets with different image modalities. The experimental results show that our method achieves better performance than the state-of-the-art approaches. Our model has strong feature learning ability and is robust to capture different scale targets. It can achieve satisfactory results while using only 0.98 million (M) parameters.

A single video super-resolution GAN for multiple downsampling operators based on pseudo-inverse image formation models

The popularity of high and ultra-high definition displays has led to the need for methods to improve the quality of videos already obtained at much lower resolutions. A large amount of current CNN-based Video Super-Resolution methods are designed and trained to handle a specific degradation operator (e.g., bicubic downsampling) and are not robust to mismatch between training and testing degradation models. This causes their performance to deteriorate in real-life applications. Furthermore, many of them use the Mean-Squared-Error as the only loss during learning, causing the resulting images to be too smooth. In this work we propose a new Convolutional Neural Network for video super resolution which is robust to multiple degradation models. During training, which is performed on a large dataset of scenes with slow and fast motions, it uses the pseudo-inverse image formation model as part of the network architecture in conjunction with perceptual losses and a smoothness constraint that eliminates the artifacts originating from these perceptual losses. The experimental validation shows that our approach outperforms current state-of-the-art methods and is robust to multiple degradations.

Single Fundus Image Super-Resolution Via Cascaded Channel-Wise Attention Network.

Fundus image is commonly used in aiding the diagnosis of ophthalmic diseases. A high-resolution (HR) image is valuable to provide the anatomic information on the eye conditions. Recently, image super-resolution (SR) though learning model has been shown to be an economic yet effective way to satisfy the high demands in the clinical practice. However, the reported methods ignore the mutual dependencies of low-and high-resolution images and did not fully exploit the dependencies between channels. To tackle with the drawbacks, we propose a novel network for fundus image SR, named by Fundus Cascaded Channel-wise Attention Network (FC-CAN). The proposed FCCAN cascades channel attention module and dense module jointly to exploit the semantic interdependencies both frequency and domain information across channels. The channel attention module rescales channel maps in spatial domain, while the dense module preserves the HR components by up- and down-sampling operation. Experimental results demonstrate the superiority of our net-work in comparison with the six methods.

FRNet: an end-to-end feature refinement neural network for medical image segmentation

Medical image segmentation is a crucial but challenging task for computer-aided diagnosis. In recent years, fully convolutional network-based methods have been widely applied to medical image segmentation. U-shape-based approaches are one of the most successful structures in this medical field. However, the consecutive down-sampling operations in the encoder lead to the loss of spatial information, which is important for medical image segmentation. In this paper, we present a novel lightweight end-to-end feature refinement network (FRNet) to address this issue. The structure of our model is simple and efficient. Specifically, the network adopts an encoder-decoder network as backbone, where two additional paths, spatial refinement path and semantic refinement path, are applied on the encoder and decoder, respectively, to improve the detailed representation ability and discriminative ability of our model. In addition, we introduce a feature adaptive fusion block (FAF block) that effectively combines features of different depths. The proposed FRNet can be trained in an end-to-end way. We have evaluated our method on three different medical image segmentation tasks. Experimental results show that FRNet has better performance than the state-of-the-art approaches. It achieves a high average accuracy without any post-processing of 0.968 and 0.936 for blood vessel segmentation and skin lesion segmentation, respectively. We further demonstrate that our method can be easily applied to other network structures to improve their performance.