Deep Residual Learning Research Articles

Deep Residual Learning based Enhanced JPEG Compression in the Internet of Things

With the development of big data and network technology, there are more use cases, such as edge computing, that require more secure and efficient multimedia big data transmission. Data compression methods can help achieving many tasks like providing data integrity, protection, as well as efficient transmission. Classical multimedia big data compression relies on methods like the spatial-frequency transformation for compressing with loss. Recent approaches use deep learning to further explore the limit of the data compression methods in communication constrained use cases like the Internet of Things (IoT). In this article, we propose a novel method to significantly enhance the transformation-based compression standards like JPEG by transmitting much fewer data of one image at the sender's end. At the receiver's end, we propose a two-step method by combining the state-of-the-art signal processing based recovery method with a deep residual learning model to recover the original data. Therefore, in the IoT use cases, the sender like edge device can transmit only 60% data of the original JPEG image without any additional calculation steps but the image quality can still be recovered at the receiver's end like cloud servers with peak signal-to-noise ratio over 31 dB.

FastDerainNet: A Deep Learning Algorithm for Single Image Deraining

Existing neural network-based methods for de-raining single images exhibit dissatisfactory results owing to the inefficient propagation of features when objects with sizes and shapes similar to those of rain streaks are present in images. Furthermore, existing methods do not consider that the abundant information included in rain streaked images could interfere with the training process. To overcome these limitations, in this paper, we propose a deep residual learning algorithm called FastDerainNet for removing rain streaks from single images. We design a deep convolutional neural network architecture, based on a deep residual network called the share-source residual module (SSRM), by substituting the origins of all shortcut connections for one point. To further improve the de-raining performance, we adopt the SSRM as the parameter layers in FastDerainNet and use image decomposition to modify the loss function. Finally, we train FastDerainNet on a synthetic dataset. By learning the residual mapping between rainy and clean image detail layers, it is able to reduce the mapping range and simplify the training process. Experiments on both synthetic and real-world images demonstrate that the proposed method achieves increased performance with regard to de-raining, in addition to preserving original details, in comparison with other state-of-the-art methods.

Deep learning model predicts water interaction sites on the surface of proteins using limited-resolution data.

We develop a residual deep learning model, hotWater (https://pypi.org/project/hotWater/), to identify key water interaction sites on proteins for binding models and drug discovery. This is tested on new crystal structures, as well as cryo-EM and NMR structures from the PDB and in crystallographic refinement with promising results.

Multi-Scale Deep Residual Learning-Based Single Image Haze Removal via Image Decomposition.

Images/videos captured from outdoor visual devices are usually degraded by turbid media, such as haze, smoke, fog, rain, and snow. Haze is the most common one in outdoor scenes due to the atmosphere conditions. In this paper, a novel deep learning-based architecture (denoted by MSRL-DehazeNet) for single image haze removal relying on multi-scale residual learning (MSRL) and image decomposition is proposed. Instead of learning an end-to-end mapping between each pair of hazy image and its corresponding haze-free one adopted by most existing learningbased approaches, we reformulate the problem as restoration of the image base component. Based on the decomposition of a hazy image into the base and the detail components, haze removal (or dehazing) can be achieved by both of our multi-scale deep residual learning and our simplified U-Net learning only for mapping between hazy and haze-free base components, while the detail component is further enhanced via the other learned convolutional neural network (CNN). Moreover, benefited by the basic building block of our deep residual CNN architecture and our simplified UNet structure, the feature maps (produced by extracting structural and statistical features), and each previous layer can be fully preserved and fed into the next layer. Therefore, possible color distortion in the recovered image would be avoided. As a result, the final haze-removed (or dehazed) image is obtained by integrating the haze-removed base and the enhanced detail image components. Experimental results have demonstrated good effectiveness of the proposed framework, compared with state-ofthe-art approaches.

A deep residual learning network for predicting lung adenocarcinoma manifesting as ground-glass nodule on CT images.

To develop a deep learning-based artificial intelligence (AI) scheme for predicting the likelihood of the ground-glass nodule (GGN) detected on CT images being invasive adenocarcinoma (IA) and also compare the accuracy of this AI scheme with that of two radiologists. First, we retrospectively collected 828 histopathologically confirmed GGNs of 644 patients from two centers. Among them, 209 GGNs are confirmed IA and 619 are non-IA, including 409 adenocarcinomas in situ and 210 minimally invasive adenocarcinomas. Second, we applied a series of pre-preprocessing techniques, such as image resampling, rescaling and cropping, and data augmentation, to process original CT images and generate new training and testing images. Third, we built an AI scheme based on a deep convolutional neural network by using a residual learning architecture and batch normalization technique. Finally, we conducted an observer study and compared the prediction performance of the AI scheme with that of two radiologists using an independent dataset with 102 GGNs. The new AI scheme yielded an area under the receiver operating characteristic curve (AUC) of 0.92 ± 0.03 in classifying between IA and non-IA GGNs, which is equivalent to the senior radiologist's performance (AUC 0.92 ± 0.03) and higher than the score of the junior radiologist (AUC 0.90 ± 0.03). The Kappa value of two sets of subjective prediction scores generated by two radiologists is 0.6. The study result demonstrates using an AI scheme to improve the performance in predicting IA, which can help improve the development of a more effective personalized cancer treatment paradigm. • The feasibility of using a deep learning method to predict the likelihood of the ground-glass nodule being invasive adenocarcinoma. • Residual learning-based CNN model improves the performance in classifying between IA and non-IA nodules. • Artificial intelligence (AI) scheme yields higher performance than radiologists in predicting invasive adenocarcinoma.

WeatherNet: Recognising Weather and Visual Conditions from Street-Level Images Using Deep Residual Learning

Extracting information related to weather and visual conditions at a given time and space is indispensable for scene awareness, which strongly impacts our behaviours, from simply walking in a city to riding a bike, driving a car, or autonomous drive-assistance. Despite the significance of this subject, it has still not been fully addressed by the machine intelligence relying on deep learning and computer vision to detect the multi-labels of weather and visual conditions with a unified method that can be easily used in practice. What has been achieved to-date are rather sectorial models that address a limited number of labels that do not cover the wide spectrum of weather and visual conditions. Nonetheless, weather and visual conditions are often addressed individually. In this paper, we introduce a novel framework to automatically extract this information from street-level images relying on deep learning and computer vision using a unified method without any pre-defined constraints in the processed images. A pipeline of four deep convolutional neural network (CNN) models, so-called WeatherNet, is trained, relying on residual learning using ResNet50 architecture, to extract various weather and visual conditions such as dawn/dusk, day and night for time detection, glare for lighting conditions, and clear, rainy, snowy, and foggy for weather conditions. WeatherNet shows strong performance in extracting this information from user-defined images or video streams that can be used but are not limited to autonomous vehicles and drive-assistance systems, tracking behaviours, safety-related research, or even for better understanding cities through images for policy-makers.

Stable polyp-scene classification via subsampling and residual learning from an imbalanced large dataset.

This Letter presents a stable polyp-scene classification method with low false positive (FP) detection. Precise automated polyp detection during colonoscopies is essential for preventing colon-cancer deaths. There is, therefore, a demand for a computer-assisted diagnosis (CAD) system for colonoscopies to assist colonoscopists. A high-performance CAD system with spatiotemporal feature extraction via a three-dimensional convolutional neural network (3D CNN) with a limited dataset achieved about 80% detection accuracy in actual colonoscopic videos. Consequently, further improvement of a 3D CNN with larger training data is feasible. However, the ratio between polyp and non-polyp scenes is quite imbalanced in a large colonoscopic video dataset. This imbalance leads to unstable polyp detection. To circumvent this, the authors propose an efficient and balanced learning technique for deep residual learning. The authors’ method randomly selects a subset of non-polyp scenes whose number is the same number of still images of polyp scenes at the beginning of each epoch of learning. Furthermore, they introduce post-processing for stable polyp-scene classification. This post-processing reduces the FPs that occur in the practical application of polyp-scene classification. They evaluate several residual networks with a large polyp-detection dataset consisting of 1027 colonoscopic videos. In the scene-level evaluation, their proposed method achieves stable polyp-scene classification with 0.86 sensitivity and 0.97 specificity.

Dilated Residual Learning With Skip Connections for Real-Time Denoising of Laser Speckle Imaging of Blood Flow in a Log-Transformed Domain.

Laser speckle contrast imaging (LSCI) is a wide-field and noncontact imaging technology for mapping blood flow. Although the denoising method based on block-matching and three-dimensional transform-domain collaborative filtering (BM3D) was proposed to improve its signal-to-noise ratio (SNR) significantly, the processing time makes it difficult to realize real-time denoising. Furthermore, it is still difficult to obtain an acceptable level of SNR with a few raw speckle images given the presence of significant noise and artifacts. A feed-forward denoising convolutional neural network (DnCNN) achieves state-of-the-art performance in denoising nature images and is efficiently accelerated by GPU. However, it performs poorly in learning with original speckle contrast images of LSCI owing to the inhomogeneous noise distribution. Therefore, we propose training DnCNN for LSCI in a log-transformed domain to improve training accuracy and it achieves an improvement of 5.13 dB in the peak signal-to-noise ratio (PSNR). To decrease the inference time and improve denoising performance, we further propose a dilated deep residual learning network with skip connections (DRSNet). The image-quality evaluations of DRSNet with five raw speckle images outperform that of spatially average denoising with 20 raw speckle images. DRSNet takes 35 ms (i.e., 28 frames per second) for denoising a blood flow image with 486×648 pixels on an NVIDIA 1070 GPU, which is approximately 2.5 times faster than DnCNN. In the test sets, DRSNet also improves 0.15 dB in the PSNR than that of DnCNN. The proposed network shows good potential in real-time monitoring of blood flow for biomedical applications.

Temporally downsampled cerebral CT perfusion image restoration using deep residual learning.

Acute ischemic stroke is one of the most causes of death all over the world. Onset to treatment time is critical in stroke diagnosis and treatment. Considering the time consumption and high price of MR imaging, CT perfusion (CTP) imaging is strongly recommended for acute stroke. However, too much CT radiation during CTP imaging may increase the risk of health problems. How to reduce CT radiation dose in CT perfusion imaging has drawn our great attention. In this study, the original 30-pass CTP images are downsampled to 15 passes in time sequence, which equals to 50% radiation dose reduction. Then, a residual deep convolutional neural network (DCNN) model is proposed to restore the downsampled 15-pass CTP images to 30 passes to calculate the parameters such as cerebral blood flow, cerebral blood volume, mean transit time, time to peak for stroke diagnosis and treatment. The deep restoration CNN is implemented simply and effectively with 16 successive convolutional layers which form a wide enough receptive field for input image data. 18 patients' CTP images are employed as training set and the other six patients' CTP images are treated as test dataset in this study. Experiments demonstrate that our CNN can restore high-quality CTP images in terms of structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR). The average SSIM and PSNR for test images are 0.981 and 56.25, and the SSIM and PSNR of regions of interest are 0.915 and 42.44, respectively, showing promising quantitative level. In addition, we compare the perfusion maps calculated from the restored images and from the original images, and the average perfusion results of them are extremely close. Areas of hypoperfusion of six test cases could be detected with comparable accuracy by radiologists. The trained model can restore the temporally downsampled 15-pass CTP to 30 passes very well. According to the contrast test, sufficient information cannot be restored with, e.g., simple interpolation method and deep convolutional generative adversarial network, but can be restored with the proposed CNN model. This method can be an optional way to reduce radiation dose during CTP imaging.

Deep Residual Learning for Boosting the Accuracy of Hyperspectral Pansharpening

Recently, deep learning (DL) has gained impressive achievements in the field of remote sensing image fusion. However, most of the previous DL-based fusion methods are originally designed for multispectral pansharpening, which cannot be readily employed to hyperspectral pansharpening due to the much wider spectral range and lower spatial resolution of a hyperspectral image (HSI). In this letter, a novel framework based on deep residual learning is proposed for hyperspectral pansharpening. The proposed framework consists mainly of two parts. First, the initialized HSI with the enhanced spatial resolution is generated through contrast limited adaptive histogram equalization (CLAHE) and guided filter. Then, a deep residual convolutional neural network (DRCNN) is introduced to map the residuals between the initialized HSI and the reference HSI for further boosting the fusion accuracy. Experimental results demonstrate that the proposed framework can achieve superior performance compared with the existing state-of-the-art pansharpening methods, especially in terms of edge details enhancement.

Urine Sediment Recognition Method Based on Multi-View Deep Residual Learning in Microscopic Image.

Urine sediment recognition is attracting growing interest in the field of computer vision. A multi-view urine cell recognition method based on multi-view deep residual learning is proposed to solve some existing problems, such as multi-view cell gray change and cell information loss in the natural state. Firstly, the convolutional network is designed to extract the urine sediment features from different perspectives based on the residual network, and the depth-wise separable convolution is introduced to reduce the network parameters. Secondly, Squeeze-and-Excitation block is embedded to learn feature weights, using feature re-calibration to improve network representation, and the robustness of the network is enhanced by adding spatial pyramid pooling. Finally, for further optimizing the recognition results, the Adam with weight decay optimization method is used to accelerate the convergence of the network model. Experiments on self-built urine microscopic image data-set show that our proposed method has state-of-the-art classification accuracy and reduces network computing time.

Energy-Efficient Power Control in Wireless Networks With Spatial Deep Neural Networks

The energy-efficient power control of interfering links in a large wireless network is a challenging task. In this paper, we propose a deep learning based power control scheme, termed PowerNet, that uses the devices' geographical location information (GLI). We show that it is possible to bypass the complex channel estimation process and directly perform power control with GLI when the channel state information (CSI) can be viewed as a function of distance dependent path-loss. The time consuming and complex channel estimation process can thus be avoided. Moreover, with a proper training, PowerNet transforms the on-line complexity to off-line training, and is amenable for real-time services. Different from conventional deep neural network (DNN) that adopts fully connected structure, the proposed PowerNet leverages convolutional layers to better capture the interference pattern across different links in large wireless networks and utilizes deep residual learning to further enhance its robustness. Simulation results demonstrate that PowerNet can achieve a near-optimal performance at a remarkably high speed without explicit channel estimation. PowerNet also exhibits a great generalization ability in terms of problem sizes and channel fading types.

A Study of Two CNN Demosaicking Algorithms

Most cameras capture the information of only one color for a given pixel. This results in a mosaicked image that must be interpolated to get three colors at each pixel. The step going from a mosaicked image to a regular RGB image is called demosaicking. This paper studies two recent demosaicking methods based on convolutional neural networks that achieve artifact-free state-of-the-art results: Deep joint demosaicking and denoising by Gharbi et al. and Color image demosaicking via deep residual learning by Tan et al. We show that these methods beat by almost two decibels the best human-crafted methods, while being faster by one order of magnitude. This, arguably, seals the destiny of human-crafted methods on this subject.

Coarse-to-Fine Image DeHashing Using Deep Pyramidal Residual Learning

Image dehashing refers to the process of inferring images by inverting image hashes. Recently, image dehashing from real-valued image retrieval hashes is shown feasible using deep convolutional neural networks. However, the perceptual quality of dehashed images is challenged when real-valued hashes are quantized to less bits. Besides, the scalability to larger or color image dehashing is limited in the previous dehashing network. To this end, we propose a pyramidal long-range residual-learning network (PyLRR-Net). PyLRR-Net is a pyramidal image reconstruction network to dehash images in a progressive manner. At each image scale, we design and insert a long-range residual block to refine the coarse image reconstruction leveraging deep residual learning. Experiments on both grayscale and color image datasets show that the proposed PyLRR-Net outperforms previous work in terms of image dehashing quality, scalability, and flexibility for large and color image dehashing problems.

Accuracy of machine learning for differentiation between optic neuropathies and pseudopapilledema

BackgroundThis study is to evaluate the accuracy of machine learning for differentiation between optic neuropathies, pseudopapilledema (PPE) and normals.MethodsTwo hundred and ninety-five images of optic neuropathies, 295 images of PPE, and 779 control images were used. Pseudopapilledema was defined as follows: cases with elevated optic nerve head and blurred disc margin, with normal visual acuity (> 0.8 Snellen visual acuity), visual field, color vision, and pupillary reflex. The optic neuropathy group included cases of ischemic optic neuropathy (177), optic neuritis (48), diabetic optic neuropathy (17), papilledema (22), and retinal disorders (31). We compared four machine learning classifiers (our model, GoogleNet Inception v3, 19-layer Very Deep Convolution Network from Visual Geometry group (VGG), and 50-layer Deep Residual Learning (ResNet)). Accuracy and area under receiver operating characteristic curve (AUROC) were analyzed.ResultsThe accuracy of machine learning classifiers ranged from 95.89 to 98.63% (our model: 95.89%, Inception V3: 96.45%, ResNet: 98.63%, and VGG: 96.80%). A high AUROC score was noted in both ResNet and VGG (0.999).ConclusionsMachine learning techniques can be combined with fundus photography as an effective approach to distinguish between PPE and elevated optic disc associated with optic neuropathies.

Fine-grained ship classification based on deep residual learning for high-resolution SAR images

ABSTRACTAs the resolution of Synthetic Aperture Radar (SAR) images increases, the fine-grained classification of ships has become a focus of the SAR field. In this paper, a ship classification framework based on deep residual network for high-resolution SAR images is proposed. In general, networks with more layers have higher classification accuracy. However, the training accuracy degradation and the limited dataset are major problems in the training process. To build deeper networks, residual modules are constructed and batch normalization is applied to keep the activation function output. Different fine tuning strategies are used to select the best training scheme. To take advantage of the proposed framework, a dataset including 835 ship slices is augmented by different multiples and then used to validate our method and other Convolutional Neural Network (CNN) models. The experimental results show that, the proposed framework can achieve a 99% overall accuracy on the augmented dataset under the optimal fine-tuning strategy, 3% higher than that in other models, which demonstrates the effectiveness of our proposed approach.

An AMP-Based Network With Deep Residual Learning for mmWave Beamspace Channel Estimation

Beamspace channel estimation in millimeter-wave (mmWave) massive MIMO system is a very challenging task, especially when the number of radio-frequency chains is limited. To address this problem, we present a novel approximate message passing (AMP)-based network with deep residual learning, referred to as LampResNet. It mainly consists of two components: 1) a learned AMP (LAMP) network and 2) a deep residual learning network (ResNet). The former utilizes the sparsity property of beamspace channel matrix and is employed to obtain a preliminary estimation result, while the latter is designed to reduce the impact of channel noise and further refine the coarse estimation obtained by the LAMP network. Simulation results validate the efficiency of the proposed network.

Semantic Segmentation of Urban Buildings from VHR Remote Sensing Imagery Using a Deep Convolutional Neural Network

Urban building segmentation is a prevalent research domain for very high resolution (VHR) remote sensing; however, various appearances and complicated background of VHR remote sensing imagery make accurate semantic segmentation of urban buildings a challenge in relevant applications. Following the basic architecture of U-Net, an end-to-end deep convolutional neural network (denoted as DeepResUnet) was proposed, which can effectively perform urban building segmentation at pixel scale from VHR imagery and generate accurate segmentation results. The method contains two sub-networks: One is a cascade down-sampling network for extracting feature maps of buildings from the VHR image, and the other is an up-sampling network for reconstructing those extracted feature maps back to the same size of the input VHR image. The deep residual learning approach was adopted to facilitate training in order to alleviate the degradation problem that often occurred in the model training process. The proposed DeepResUnet was tested with aerial images with a spatial resolution of 0.075 m and was compared in performance under the exact same conditions with six other state-of-the-art networks—FCN-8s, SegNet, DeconvNet, U-Net, ResUNet and DeepUNet. Results of extensive experiments indicated that the proposed DeepResUnet outperformed the other six existing networks in semantic segmentation of urban buildings in terms of visual and quantitative evaluation, especially in labeling irregular-shape and small-size buildings with higher accuracy and entirety. Compared with the U-Net, the F1 score, Kappa coefficient and overall accuracy of DeepResUnet were improved by 3.52%, 4.67% and 1.72%, respectively. Moreover, the proposed DeepResUnet required much fewer parameters than the U-Net, highlighting its significant improvement among U-Net applications. Nevertheless, the inference time of DeepResUnet is slightly longer than that of the U-Net, which is subject to further improvement.

MSSTResNet-TLD: A robust tracking method based on tracking-learning-detection framework by using multi-scale spatio-temporal residual network feature model

The performance of tracking task is directly dependent on the appearance features of target object, a robust approach for constructing appearance features is crucial for adaptation the appearance change. To construct an accurate and robust appearance model for visual object tracking, we modify original deep residual learning network architecture and name it Multi-Scale Residual Network (MSResNet). The first video frame image and its related information of the current input video sequence are used to learn a multi-scale appearance model of target object and a loss function is minimized over the appearance features. Meanwhile, spatial information of each video frame and temporal information between successive video frames effectively combine with MSResNet. And thus the features are generated by Multi-Scale Spatio-Temporal Residual Network, which is named MSSTResNet feature model, can adapt to scale variation, illumination variation, background clutters, severe deformation of the target object, and so on. We implement a robust tracking method based on tracking-learning-detection framework by using our proposed MSSTResNet feature model and name it MSSTResNet-TLD tracker. Unlike the previous tracking methods, the MSResNet architecture is not offline pre-trained on a large auxiliary datasets but is directly learned end-to-end with a multi-task loss by using the current input video sequence. Furthermore, the multi-task loss function utilizes the classification loss and regression loss that is more accurate for target localization. Our experimental results demonstrate that the proposed tracking method outperforms the current state-of-the-art tracking methods on Visual Object Tracking Benchmark (VOT-2016), Object Tracking Benchmark (OTB-2015), and Unmanned Aerial Vehicles (UAV20L) test datasets. Furthermore, our MSSTResNet-TLD tracker is faster than previous most trackers based on deep Convolutional Neural Network (ConvNet or CNN) and our tracker is extremely robust to the tiny target object. Our source code is available for download athttps://github.com/binger1225/MSSTResNet-TLD-Tracker.

Multimodal Data Fusion of Deep Learning and Dynamic Functional Connectivity Features to Predict Alzheimer's Disease Progression.

Early prediction of diseased brain conditions is critical for curing illness and preventing irreversible neuronal dysfunction and loss. Generically regarding the different neuroimaging modalities as filtered, complementary insights of brain's anatomical and functional organization, multimodal data fusion could be hypothesized to enhance the predictive power as compared to a unimodal prediction of disease progression. More recently, deep learning (DL) based methods on structural MRI (sMRI) data have outperformed classical machine learning approaches in several neuroimaging applications including diagnostic classification and prediction. Similarly, functional MRI (fMRI) features estimated using a dynamic (i.e. time-varying) functional connectivity (FC) approach have been found to be more discriminative and predictive of the clinical diagnosis than those based on the static FC approach. Motivated by this, we introduce a novel multimodal data fusion framework featuring deep residual learning of non-linear sMRI features and dynamic FC (dFC) based extraction of fMRI features to predict the subset of individuals with mild cognitive impairments who would progress to Alzheimer's disease within a time-period of three years from the baseline scanning sessions. Our cross-validated results from the developed multimodal (sMRI-fMRI) data fusion framework demonstrate a significant improvement in performance over the unimodal prediction analyses with the fMRI (p = 7.03 x 10-7) and sMRI (p = 6.72 x 10-4) modalities. As such, the findings in this work highlight the benefits of combining multiple neuroimaging data modalities via data fusion, corroborate the predictive value of the tested DL and dFC features and argue in favor of exploration of similar approaches to learn neuroanatomical and functional alterations in the neuroimaging data.