Multi-scale Deep Neural Network Research Articles

Single Image Dehazing via Multi-scale Convolutional Neural Networks with Holistic Edges

Single image dehazing has been a challenging problem which aims to recover clear images from hazy ones. The performance of existing image dehazing methods is limited by hand-designed features and priors. In this paper, we propose a multi-scale deep neural network for single image dehazing by learning the mapping between hazy images and their transmission maps. The proposed algorithm consists of a coarse-scale net which predicts a holistic transmission map based on the entire image, and a fine-scale net which refines dehazed results locally. To train the multi-scale deep network, we synthesize a dataset comprised of hazy images and corresponding transmission maps based on the NYU Depth dataset. In addition, we propose a holistic edge guided network to refine edges of the estimated transmission map. Extensive experiments demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods on both synthetic and real-world images in terms of quality and speed.

A fast and efficient 3D reflection symmetry detector based on neural networks

Determining the 3D reflection symmetry planes from 3D models is very difficult and time-consuming. In this paper, we formulate the symmetry detection as a per-point classification problem and present a deep neural network based method to solve it. During the training procedure, we firstly collect a lot of CAD mesh models with reflection symmetry as the training data, and then convert each mesh model into a dense point cloud with points located on the symmetry planes labeled as positive. Based on the PointNet++ architecture, we train a multi-scale deep neural network to capture the reflection symmetry property from the point cloud automatically. In addition, a novel weighted cross-entropy loss function is adopted to balance the positive and the negative samples. During the inference procedure, we firstly feed the down-sampled point cloud into the trained neural network. Then, the output per-point classification result is used to calculate an initial symmetry plane equation with RANSAC strategy and the least square method. Finally, iterative closest point algorithm is performed to optimize the fitted symmetry plane. Experimental results on both the synthetic and the real data demonstrate the efficiency, robustness and flexibility of our approach. Our method is pretty fast and generates comparable or better results than the existing methods.

Adaptive multi-scale deep neural networks with perceptual loss for panchromatic and multispectral images classification

Due to the redundancy of imaging systems, multispectral and panchromatic images are of higher spatial resolutions and characterized by different attributes, and are often fused together for accurate land-cover mapping. In this work, we propose a novel framework via adaptive multi-scale convolutional neural networks and perceptual loss function for multispectral and panchromatic images classification. In the proposed scheme, adaptive multi-scale convolutional neural networks are designed to capture the scale information of objects adaptively. And perceptual loss function is constructed via non-local spectral and structural similarities to suppress the interference of unbalanced illumination and speckle noises. A corresponding iteration optimization algorithm is presented to solve the proposed perceptual loss function. Experimental results conducted on three datasets indicate that the proposed framework performs better than the state-of-the-art methods.

Multi‐scale deep neural network for salient object detection

Salient object detection is a fundamental problem and has been received a great deal of attention in computer vision. Recently, deep learning model became a powerful tool for image feature extraction. In this study, the authors propose a multi-scale deep neural network (MSDNN) for salient object detection. The proposed model first extracts global high-level features and context information over the whole source image with the recurrent convolutional neural network. Then several stacked deconvolutional layers are adopted to get the multi-scale feature representation and obtain a series of saliency maps. Finally, the authors investigate a fusion convolution module to build a final pixel level saliency map. The proposed model is extensively evaluated on six salient object detection benchmark datasets. Results show that the authors' deep model significantly outperforms other 12 state-of-the-art approaches.

Multiscale Deep Alternative Neural Network for Large-Scale Video Classification

With the rapid increase in the amount of multimedia data, video classification has become a demanding and challenging research topic. Compared with image classification, video classification requires mapping a video that contains hundreds of frames to semantic tags, which poses many challenges to the direct use of advanced models originally designed for image-oriented tasks. On the other hand, continuous frames in a video also give us more visual clues that we can leverage to achieve better classification. One of the most important clues is the context in the spatiotemporal domain. In this paper, we introduce the multiscale deep alternative neural network (DANN), a novel architecture combining the strengths of both convolutional neural network and recurrent neural networks to achieve a deep network that can collect rich context hierarchies for video classification. In particular, the DANN is stacked with alternative layers, each of which consists of a volumetric convolutional layer followed by a recurrent layer. The former acts as a local feature learner, whereas the latter is used to collect contexts. Compared with popular deep feed-forward neural networks, the DANN learns local features and their contexts from the very beginning. This setting enables preserving context evolutions, which we show to be essential for improving the accuracy of video classification. To release the full potential of the DANN, we develop a deeper version with stochastic-layer skip-connections and construct a multiscale DANN to incorporate contexts at different scales. We show how to apply the multiscale DANN for video classification with carefully designed configurations in terms of both input–output settings and training–testing methods. The DANN is shown to be robust to not only human-centric videos, but also natural videos. As there are few large-scale natural disaster video datasets, we construct a new large-scale one and make it publicly available. Experiments on four datasets show the effectiveness of our method for both human actions and natural events.

Multimodal and Multiscale Deep Neural Networks for the Early Diagnosis of Alzheimer\u2019s Disease using structural MR and FDG-PET images

Alzheimer’s Disease (AD) is a progressive neurodegenerative disease where biomarkers for disease based on pathophysiology may be able to provide objective measures for disease diagnosis and staging. Neuroimaging scans acquired from MRI and metabolism images obtained by FDG-PET provide in-vivo measurements of structure and function (glucose metabolism) in a living brain. It is hypothesized that combining multiple different image modalities providing complementary information could help improve early diagnosis of AD. In this paper, we propose a novel deep-learning-based framework to discriminate individuals with AD utilizing a multimodal and multiscale deep neural network. Our method delivers 82.4% accuracy in identifying the individuals with mild cognitive impairment (MCI) who will convert to AD at 3 years prior to conversion (86.4% combined accuracy for conversion within 1–3 years), a 94.23% sensitivity in classifying individuals with clinical diagnosis of probable AD, and a 86.3% specificity in classifying non-demented controls improving upon results in published literature.

Multiscale deep neural network based analysis of FDG-PET images for the early diagnosis of Alzheimer’s disease

Alzheimer's disease (AD) is one of the most common neurodegenerative diseases with a commonly seen prodromal mild cognitive impairment (MCI) phase where memory loss is the main complaint progressively worsening with behavior issues and poor self-care. However, not all individuals clinically diagnosed with MCI progress to AD. A fraction of subjects with MCI either progress to non-AD dementia or remain stable at the MCI stage without progressing to dementia. Although a curative treatment of AD is currently unavailable, it is extremely important to correctly identify the individuals in the MCI phase that will go on to develop AD so that they may benefit from a curative treatment when one becomes available in the near future. At the same time, it would be highly desirable to also correctly identify those in the MCI phase that do not have AD pathology so they may be spared from unnecessary pharmocologic interventions that, at best, may provide them no benefit, and at worse, could further harm them with adverse side-effects. Additionally, it may be easier and simpler to identify the cause of the cognitive impairment in these non-AD cases, and hence proper identification of prodromal AD will be of benefit to these individuals as well. Fluorodeoxy glucose positron emission tomography (FDG-PET) captures the metabolic activity of the brain, and this imaging modality has been reported to identify changes related to AD prior to the onset of structural changes. Prior work on designing classifier using FDG-PET imaging has been promising. Since deep-learning has recently emerged as a powerful tool to mine features and use them for accurate labeling of the group membership of given images, we propose a novel deep-learning framework using FDG-PET metabolism imaging to identify subjects at the MCI stage with presymptomatic AD and discriminate them from other subjects with MCI (non-AD / non-progressive). Our multiscale deep neural network obtained 82.51% accuracy of classification just using measures from a single modality (FDG-PET metabolism data) outperforming other comparable FDG-PET classifiers published in the recent literature.

Transform a Simple Sketch to a Chinese Painting by a Multiscale Deep Neural Network

Recently, inspired by the power of deep learning, convolution neural networks can produce fantastic images at the pixel level. However, a significant limiting factor for previous approaches is that they focus on some simple datasets such as faces and bedrooms. In this paper, we propose a multiscale deep neural network to transform sketches into Chinese paintings. To synthesize more realistic imagery, we train the generative network by using both L1 loss and adversarial loss. Additionally, users can control the process of the synthesis since the generative network is feed-forward. This network can also be treated as neural style transfer by adding an edge detector. Furthermore, additional experiments on image colorization and image super-resolution demonstrate the universality of our proposed approach.

Structurally-sensitive Multi-scale Deep Neural Network for Low-Dose CT Denoising.

Computed tomography (CT) is a popular medical imaging modality and enjoys wide clinical applications. At the same time, the x-ray radiation dose associated with CT scannings raises a public concern due to its potential risks to the patients. Over the past years, major efforts have been dedicated to the development of Low-Dose CT (LDCT) methods. However, the radiation dose reduction compromises the signal-to-noise ratio (SNR), leading to strong noise and artifacts that downgrade CT image quality. In this paper, we propose a novel 3D noise reduction method, called Structurally-sensitive Multi-scale Generative Adversarial Net (SMGAN), to improve the LDCT image quality. Specifically, we incorporate three-dimensional (3D) volumetric information to improve the image quality. Also, different loss functions for training denoising models are investigated. Experiments show that the proposed method can effectively preserve structural and textural information in reference to normal-dose CT (NDCT) images, and significantly suppress noise and artifacts. Qualitative visual assessments by three experienced radiologists demonstrate that the proposed method retrieves more information, and outperforms competing methods.

构建多尺度深度卷积神经网络行为识别模型

构建多尺度深度卷积神经网络行为识别模型

Abdominal adipose tissues extraction using multi-scale deep neural network

Segmentation of abdominal adipose tissues (AAT) into subcutaneous adipose tissues (SAT) and visceral adipose tissues (VAT) is of crucial interest for managing the obesity. Previous methods with raw or hand-crafted features rarely work well on large-scale subject cohorts, because of the inhomogeneous image intensities, artifacts and the diverse distributions of VAT. In this paper, we propose a novel two-stage coarse-to-fine algorithm for AAT segmentation. In the first stage, we formulate the AAT segmentation task as a pixel-wise classification problem. First, three types of features, intensity, spatial and contextual features, are extracted. Second, a new type of deep neural network, named multi-scale deep neural network (MSDNN), is provided to extract high-level features. In the second stage, to improve the segmentation accuracy, we refine coarse segmentation results by determining the internal boundaries of SAT based on coarse segmentation results and the continuous of SAT internal boundaries. Finally, we demonstrate the efficacy of our algorithm for both 2D and 3D cases on a wide population range. Compared with other algorithms, our method is not only more suitable for large-scale dataset, but also achieves better segmentation results. Furthermore, our system takes about 2s to segment an abdominal image, which implies potential clinical applications.

Recognition of Chinese characters based on multi-scale gradient and deep neural network

The method to extract the gradient histogram feature of the Chinese characters with a multiscale sliding window and to recognize the printed Chinese characters with deep neural network was presented.In order to acquire the spatial information of the gradient histogram,a retractable sliding window technique was proposed for segmenting the images and getting the gradient feature information from different scales which can effectively combine all the global features and local block features of Chinese characters. The experiment was carried out by using a 5-layer deep neural network to classify 3 755 categories of printed Chinese characters. A Dropout technique was applied so as to prevent over-fitting training and to improve the generalization ability of the neural network. The accuracy of the experiment reaches 98. 292%,which has better recognition performance and demonstrates that the method of applying a multi-scale gradient feature and deep neural network model on the recognition of Chinese characters is effective.

Multi-Scale Deep Neural Network Microscopic Image Segmentation

With digital technology and fast imaging speed, huge amount of microscopic images are produced in biomedical research every day. A key procedure in processing and analyzing the images involves recognition and segmentation of features, patterns, or regions that are of interest to the researchers. Such work is often manually done by humans, which is very time-consuming and thus impractical when image data are massive. Traditional image segmentation algorithms have difficulty in identifying high-level features (e.g. a particular morphology of a cell type or an organelle). Plus, many microscopic images, especially super-resolution STED images, are highly noisy. The analysis of noisy images poses an even higher challenge to traditional algorithms. Compared with computer algorithms, humans are incredibly good at identifying high-level features. Machine learning, which let computer learn human behaviors, is thus a suitable solution. Deep learning is a set of emerging machine learning methods that is being successfully applied in many applications, such as face recognition and speech recognition. It has also been used to process specific types of microscopic images.We have designed a multi-scale convolution neural network for segmentizing noisy microscopic images. Images at original resolution and down-sampled images at multiple scales are feed to the network for information extraction. The purpose of adopting this multi-scale architecture is to obtain local fine resolution while maintain larger field of views, which gives network low classification error rate while maintain reasonable processing speed. The network has been tested on segmentation of mitochondria labeled with mitotracker. Network separate image pixels into two categories: “background” and “mitochondria”. Training and testing of the network is done with standard back-propagation algorithm, with human labeled segmentation data. Preliminary result shows the trained network well outperformed multiple traditional image segmentation methods.