Deep Network Architecture Research Articles

A Trace-restricted Kronecker-Factored Approximation to Natural Gradient

Second-order optimization methods have the ability to accelerate convergence by modifying the gradient through the curvature matrix. There have been many attempts to use second-order optimization methods for training deep neural networks. In this work, inspired by diagonal approximations and factored approximations such as Kronecker-factored Approximate Curvature (KFAC), we propose a new approximation to the Fisher information matrix (FIM) called Trace-restricted Kronecker-factored Approximate Curvature (TKFAC), which can hold the certain trace relationship between the exact and the approximate FIM. In TKFAC, we decompose each block of the approximate FIM as a Kronecker product of two smaller matrices and scaled by a coefficient related to trace. We theoretically analyze TKFAC's approximation error and give an upper bound of it. We also propose a new damping technique for TKFAC on convolutional neural networks to maintain the superiority of second-order optimization methods during training. Experiments show that our method has better performance compared with several state-of-the-art algorithms on some deep network architectures.

FusionNet: A Deep Fully Residual Convolutional Neural Network for Image Segmentation in Connectomics

Cellular-resolution connectomics is an ambitious research direction with the goal of generating comprehensive brain connectivity maps using high-throughput, nano-scale electron microscopy. One of the main challenges in connectomics research is developing scalable image analysis algorithms that require minimal user intervention. Deep learning has provided exceptional performance in image classification tasks in computer vision, leading to a recent explosion in popularity. Similarly, its application to connectomic analyses holds great promise. Here, we introduce a deep neural network architecture, FusionNet, with a focus on its application to accomplish automatic segmentation of neuronal structures in connectomics data. FusionNet combines recent advances in machine learning, such as semantic segmentation and residual neural networks, with summation-based skip connections. This results in a much deeper network architecture and improves segmentation accuracy. We demonstrate the performance of the proposed method by comparing it with several other popular electron microscopy segmentation methods. We further illustrate its flexibility through segmentation results for two different tasks: cell membrane segmentation and cell nucleus segmentation.

Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits.

Synaptic plasticity is believed to be a key physiological mechanism for learning. It is well established that it depends on pre- and postsynaptic activity. However, models that rely solely on pre- and postsynaptic activity for synaptic changes have, so far, not been able to account for learning complex tasks that demand credit assignment in hierarchical networks. Here we show that if synaptic plasticity is regulated by high-frequency bursts of spikes, then pyramidal neurons higher in a hierarchical circuit can coordinate the plasticity of lower-level connections. Using simulations and mathematical analyses, we demonstrate that, when paired with short-term synaptic dynamics, regenerative activity in the apical dendrites and synaptic plasticity in feedback pathways, a burst-dependent learning rule can solve challenging tasks that require deep network architectures. Our results demonstrate that well-known properties of dendrites, synapses and synaptic plasticity are sufficient to enable sophisticated learning in hierarchical circuits.

Learning spatiotemporal features of DSA using 3D CNN and BiConvGRU for ischemic moyamoya disease detection

Background Moyamoya disease (MMD) is a serious intracranial cerebrovascular disease. Cerebral hemorrhage caused by MMD will bring life risk to patients. Therefore, MMD detection is of great significance in the prevention of cerebral hemorrhage. In order to improve the accuracy of digital subtraction angiography (DSA) in the diagnosis of ischemic MMD, in this paper, a deep network architecture combined with 3D convolutional neural network (3D CNN) and bidirectional convolutional gated recurrent unit (BiConvGRU) is proposed to learn the spatiotemporal features for ischemic MMD detection. Methods Firstly, 2D convolutional neural network (2D CNN) is utilized to extract spatial features for each frame of DSA. Secondly, the long-term spatiotemporal features of DSA sequence are extracted by BiConvGRU. Thirdly, the short-term spatiotemporal features of DSA are further extracted by 3D convolutional neural network (3D CNN). In addition, different features are extracted when gray images and optical flow images pass through the network, and multiple features are extracted by features fusion. Finally, the fused features are utilized to classify. Results The proposed method was quantitatively evaluated on a data sets of 630 cases. The experimental results showed a detection accuracy of 0.9788, sensitivity and specificity were 0.9780 and 0.9796, respectively, and area under curve (AUC) was 0.9856. Compared with other methods, we can get the highest accuracy and AUC. Conclusions The experimental results show that the proposed method is stable and reliable for ischemic MMD detection, which provides an option for doctors to accurately diagnose ischemic MMD.

Efficient residual attention network for single image super-resolution

The use of deep convolutional neural networks (CNNs) for image super-resolution (SR) from low-resolution (LR) input has achieved remarkable reconstruction performance with the utilization of residual structures and visual attention mechanisms. However, existing single image super-resolution (SISR) methods with deeper network architectures can encounter representational bottlenecks in CNN-based networks and neglect model efficiency in model statistical inference. To solve these issues, in this paper, we design a channel hourglass residual structure (CHRS) and explore an efficient channel attention (ECA) mechanism to extract more representative features and ease the computational burden. Specifically, our CHRS, consisting of several nested residual modules, is developed to learn more discriminative representations with fewer model parameters, and the ECA is presented to efficiently capture local cross-channel interaction by subtly applying 1D convolution. Finally, we propose an efficient residual attention network (ERAN), which not only fully learns more representative features but also pays special attention to network learning efficiency. Extensive experiments demonstrate that our ERAN achieves certain improvements in model performance and implementation efficiency compared to other previous state-of-the-art methods.

Deep Multiscale Detail Networks for Multiband Spectral Image Sharpening.

We introduce a new deep detail network architecture with grouped multiscale dilated convolutions to sharpen images contain multiband spectral information. Specifically, our end-to-end network directly fuses low-resolution multispectral and panchromatic inputs to produce high-resolution multispectral results, which is the same goal of the pansharpening in remote sensing. The proposed network architecture is designed by utilizing our domain knowledge and considering the two aims of the pansharpening: spectral and spatial preservations. For spectral preservation, the up-sampled multispectral images are directly added to the output for lossless spectral information propagation. For spatial preservation, we train the proposed network in the high-frequency domain instead of the commonly used image domain. Different from conventional network structures, we remove pooling and batch normalization layers to preserve spatial information and improve generalization to new satellites, respectively. To effectively and efficiently obtain multiscale contextual features at a fine-grained level, we propose a grouped multiscale dilated network structure to enlarge the receptive fields for each network layer. This structure allows the network to capture multiscale representations without increasing the parameter burden and network complexity. These representations are finally utilized to reconstruct the residual images which contain spatial details of PAN. Our trained network is able to generalize different satellite images without the need for parameter tuning. Moreover, our model is a general framework, which can be directly used for other kinds of multiband spectral image sharpening, e.g., hyperspectral image sharpening. Experiments show that our model performs favorably against compared methods in terms of both qualitative and quantitative qualities.

ScieNet: Deep learning with spike-assisted contextual information extraction

Spiking neural network (SNN) is a type of artificial neural network that uses biologically inspired neuron models and learning rules to develop artificial intelligence with capability parallel to human brain. Deep neural networks (DNNs), on the other hand, uses less biologically plausible neurons and training methods such as gradient descent, and has shown good accuracy in computer vision tasks. However, human brain can still outperform DNN in certain scenarios. For example, DNN experiences significant performance degradation when perturbation from various sources is present in the input, which makes DNN less reliable for systems interacting with physical world. In this paper, we present a hybrid deep network architecture with spike-assisted contextual information extraction (ScieNet) as a solution to the problem. ScieNet integrates a front-end SNN with a novel stochastic spike-timing-dependent plasticity (STDP) algorithm that extracts visual context from images. The back-end DNN is trained for classification given the contextual information. The integrated network demonstrates high resilience to input perturbations without relying on pre-training on perturbed inputs. We demonstrate ScieNet with various back-end DNNs for image classification using different datasets and considering both stochastic and structured input perturbations. Experimental results demonstrate significant improvement in accuracy on perturbed images, while maintaining state-of-the-art accuracy on clean images.

CORSegNet: Deep Neural Network for Core Object Segmentation on Medical Images

In recent decades, convolutional neural networks (CNNs) have delivered promising results in vision-related tasks across different domains. Previous studies have introduced deeper network architectures to further improve the performances of object classification, localization, and segmentation. However, this induces the complexity in mapping network’s layer to the processing elements in the ventral visual pathway. Although CORnet models are not precisely biomimetic, they are closer approximations to the anatomy of ventral visual pathway compared with other deep neural networks. The uniqueness of this architecture inspires us to extend it into a core object segmentation network, CORSegnet-Z. This architecture utilizes CORnet-Z building blocks as the encoding elements. We train and evaluate the proposed model using two large datasets. Our proposed model shows significant improvements on the segmentation metrics in delineating cartilage tissues from knee magnetic resonance (MR) images and segmenting lesion boundary from dermoscopic images.

Predicting Spatial Displacement Based on Intraocular Image Design Using Convolution Neural Network – Preliminary Findings

The main global cause for blindness is due to cataract. The common treatment for cataract is to have the cloudy natural lens to be removed and replace with an artificial intraocular lens (IOL). Success in the post cataract surgery depends on the design and quality of the IOL implanted on the eye. ISO11979-3 is the standard adhered to by many lens manufacturers, to test the mechanical stability of their lenses that were produced. This compression test experiments on the lab are very costly and time consuming. Alternatively, we propose to use the convolution neural network (CNN) to predict the spatial displacement response based on the intraocular image designs. Due to limited number of images in the datasets, data augmentation was performed to tranform these images and increase the sample size to 240. On top of this, the ResNet-50 deep learning network architecture was utilized to transfer the learning done on over millions of images. The final RMSE value for the training set, validation set and testing set were at 0.47mm, 2.93mm and 2.92mm respectively. The model predictabillity is well within the range recommended by the standard between 0.15 to 1.98 mm.

TSAL: Two steps Adversarial learning based domain adaptation

Since labeled data availability differs greatly across domains, Domain Adaptation focuses on learning in new and unfamiliar domains by reducing distribution divergence. Recent research suggests that the adversarial learning approach could be a promising way to achieve the domain adaptation objective. Adversarial learning is a strategy for learning domain-transferable features in robust deep networks. This paper introduces the TSAL paradigm, a two-step adversarial learning framework. It addresses the real-world problem of text classification, where source domain(s) has labeled data but target domain (s) has only unlabeled data. TSAL utilizes joint adversarial learning with class information and domain alignment deep network architecture to learn both domain-invariant and domain-specific features extractors. It consists of two training steps that are similar to the paradigm, in which pre-trained model weights are used as initialization for training with new data. TSAL’s two training phases, however, are based on the same data, not different data, as is the case with fine-tuning. Furthermore, TSAL only uses the learned domain-invariant feature extractor from the first training as an initialization for its peer in subsequent training. By doubling the training, TSAL can emphasize the leverage of the small unlabeled target domain and learn effectively what to share between various domains. A detailed analysis of many benchmark datasets reveals that our model consistently outperforms the prior art across a wide range of dataset distributions.

Multi-Scale Neural Network for EEG Representation Learning in BCI

Recent advances in deep learning have had a methodological and practical impact on brain-computer interface (BCI) research. Among the various deep network architectures, convolutional neural networks (CNNs) have been well suited for spatio-spectral-temporal electroencephalogram (EEG) signal representation learning. Most of the existing CNN-based methods described in the literature extract features at a sequential level of abstraction with repetitive nonlinear operations and involve densely connected layers for classification. However, studies in neurophysiology have revealed that EEG signals carry information in different ranges of frequency components. To better reflect these multi-frequency properties in EEGs, we propose a novel deep multi-scale neural network that discovers feature representations in multiple frequency/time ranges and extracts relationships among electrodes, i.e., spatial representations, for subject intention/condition identification. Furthermore, by completely representing EEG signals with spatio-spectral-temporal information, the proposed method can be utilized for diverse paradigms in both active and passive BCIs, contrary to existing methods that are primarily focused on single-paradigm BCIs. To demonstrate the validity of our proposed method, we conducted experiments on various paradigms of active/passive BCI datasets. Our experimental results demonstrated that the proposed method achieved performance improvements when judged against comparable state-of-the-art methods. Additionally, we analyzed the proposed method using different techniques, such as PSD curves and relevance score inspection to validate the multi-scale EEG signal information capturing ability, activation pattern maps for investigating the learned spatial filters, and t-SNE plotting for visualizing represented features. Finally, we also demonstrated our method's application to real-world problems. Based on our experimental results and analyses, we believe that the proposed multi-scale neural network can be useful for various BCI paradigms, as a starting model or as a backbone network in any new BCI experiments.

Automatic colonic polyp detection using integration of modified deep residual convolutional neural network and ensemble learning approaches

Background and ObjectiveThe increased incidence of colorectal cancer (CRC) and its mortality rate have attracted interest in the use of artificial intelligence (AI) based computer-aided diagnosis (CAD) tools to detect polyps at an early stage. Although these CAD tools have thus far achieved a good accuracy level to detect polyps, they still have room to improve further (e.g. sensitivity). Therefore, a new CAD tool is developed in this study to detect colonic polyps accurately. MethodsIn this paper, we propose a novel approach to distinguish colonic polyps by integrating several techniques, including a modified deep residual network, principal component analysis and AdaBoost ensemble learning. A powerful deep residual network architecture, ResNet-50, was investigated to reduce the computational time by altering its architecture. To keep the interference to a minimum, median filter, image thresholding, contrast enhancement, and normalisation techniques were exploited on the endoscopic images to train the classification model. Three publicly available datasets, i.e., Kvasir, ETIS-LaribPolypDB, and CVC-ClinicDB, were merged to train the model, which included images with and without polyps. ResultsThe proposed approach trained with a combination of three datasets achieved Matthews Correlation Coefficient (MCC) of 0.9819 with accuracy, sensitivity, precision, and specificity of 99.10%, 98.82%, 99.37%, and 99.38%, respectively. ConclusionsThese results show that our method could repeatedly classify endoscopic images automatically and could be used to effectively develop computer-aided diagnostic tools for early CRC detection.

Deep embedded clustering with multiple objectives on scRNA-seq data.

In recent years, single-cell RNA sequencing (scRNA-seq) technologies have been widely adopted to interrogate gene expression of individual cells; it brings opportunities to understand the underlying processes in a high-throughput manner. Deep embedded clustering (DEC) was demonstrated successful in high-dimensional sparse scRNA-seq data by joint feature learning and cluster assignment for identifying cell types simultaneously. However, the deep network architecture for embedding clustering is not trivial to optimize. Therefore, we propose an evolutionary multiobjective DEC by synergizing the multiobjective evolutionary optimization to simultaneously evolve the hyperparameters and architectures of DEC in an automatic manner. Firstly, a denoising autoencoder is integrated into the DEC to project the high-dimensional sparse scRNA-seq data into a low-dimensional space. After that, to guide the evolution, three objective functions are formulated to balance the model's generality and clustering performance for robustness. Meanwhile, migration and mutation operators are proposed to optimize the objective functions to select the suitable hyperparameters and architectures of DEC in the multiobjective framework. Multiple comparison analyses are conducted on twenty synthetic data and eight real data from different representative single-cell sequencing platforms to validate the effectiveness. The experimental results reveal that the proposed algorithm outperforms other state-of-the-art clustering methods under different metrics. Meanwhile, marker genes identification, gene ontology enrichment and pathology analysis are conducted to reveal novel insights into the cell type identification and characterization mechanisms.

Recognition of pulmonary diseases from lung sounds using convolutional neural networks and long short-term memory.

In this paper, a study is conducted to explore the ability of deep learning in recognizing pulmonary diseases from electronically recorded lung sounds. The selected data-set included a total of 103 patients obtained from locally recorded stethoscope lung sounds acquired at King Abdullah University Hospital, Jordan University of Science and Technology, Jordan. In addition, 110 patients data were added to the data-set from the Int. Conf. on Biomedical Health Informatics publicly available challenge database. Initially, all signals were checked to have a sampling frequency of 4 kHz and segmented into 5 s segments. Then, several preprocessing steps were undertaken to ensure smoother and less noisy signals. These steps included wavelet smoothing, displacement artifact removal, and z-score normalization. The deep learning network architecture consisted of two stages; convolutional neural networks and bidirectional long short-term memory units. The training of the model was evaluated based on a k-fold cross-validation scheme of tenfolds using several performance evaluation metrics including Cohen’s kappa, accuracy, sensitivity, specificity, precision, and F1-score. The developed algorithm achieved the highest average accuracy of 99.62% with a precision of 98.85% in classifying patients based on the pulmonary disease types using CNN + BDLSTM. Furthermore, a total agreement of 98.26% was obtained between the predictions and original classes within the training scheme. This study paves the way towards implementing deep learning models in clinical settings to assist clinicians in decision making related to the recognition of pulmonary diseases.Supplementary InformationThe online version supplementary material available at 10.1007/s12652-021-03184-y.

Automated image splicing detection using deep CNN-learned features and ANN-based classifier

With the present-day rapid evolution of digital technology, images have become one of the most important means of communication and information carrier in our society. Since the last decade, with the emergence of social networking sites like Facebook, Instagram, Twitter, etc., there has been a huge increase in the amount of information exchanged in the form of digital images, on a regular basis. While traditionally we might have had faith in the credibility of these images, today’s digital technology has begun to erode that faith. Before sharing an image over social networks, editing it with relevant software application has become one of the simplest things to do today. While not many people do this with any sinister intent behind, there has been a significant increase in cybercrimes related to malicious image manipulation and sharing. To this end, image splicing has emerged as one of the major forms of image manipulation attacks, among others today. This demands investigation of intrinsic differences between authentic and forged images and hence development of automated splicing detection tools. Here, we propose a blind image splicing detection technique that employs a deep convolutional residual network architecture as a backbone, followed by a fully connected classifier network, that classifies between authentic and spliced images. The classifier networks have been evaluated using the CASIA v2.0 dataset. Both are proven to yield accuracies more than 96% on an average, having surpassed the state-of-the-art results.

Transfer Learning Technique with VGG-16 for Near-Infrared Facial Expression Recognition

In this paper, we investigate a deep learning vgg-16 network architecture for facial expression recognition under active near-infrared illumination condition and background. In particular, we consider the concept of transfer learning whereby features learned from high resolution images of huge datasets can be used to train a model of relatively small dataset without loosing the generalization ability. The pre-trained vgg-16 network architecture with transfer learning technique has been trained and validated on the Oulu-CASIA NIR dataset comprising of six (6) distinct facial expressions, and average test accuracy of 98.11% was achieved. The validation on our test data using the confusion, the precision, and the recall matrix reveals that our method achieves better results in comparison with the other methods in the literature.

Robotic Instrument Segmentation With Image-to-Image Translation

The semantic segmentation of robotic surgery video and the delineation of robotic instruments are important for enabling automation. Despite major recent progresses, the majority of the latest deep learning models for instrument detection and segmentation rely on large datasets with ground truth labels. While demonstrating the capability, reliance on large labelled data is a problem for practical applications because systems would need to be re-trained on domain variations such as procedure type or instrument sets. In this letter, we propose to alleviate this problem by training deep learning models on datasets that are synthesised using image-to-image translation techniques and we investigate different methods to perform this process optimally. Experimentally, we demonstrate that the same deep network architecture for robotic instrument segmentation can be trained on both real data and on our proposed synthetic data without affecting the quality of the output models’ performance. We show this for several recent approaches and provide experimental support on publicly available datasets, which highlight the potential value of this approach.

Ensemble multifeatured deep learning models for air quality forecasting

As air pollution becomes increasingly serious, accurate forecasting of air quality has become an important issue. Many studies related to machine learning and deep learning methods have been proposed to predict air quality. These studies have also confirmed that deep learning has better predictive performance than traditional machine learning. Ensemble Learning has also been proven that can improve the classification efficacy of machine learning. In order to improve the accuracy of air quality forecasting, in this paper, we first exploit a GRU (Gated Recurrent Unit) deep learning network architecture to design various predictive models (GRU13d, GRUAW13d, GRUAW14d, GRUSS13d, and GRUST14d, etc.) that meet various spatial and temporal situations, and then propose an ensemble learning forecasting model, named MLEGRU (Multiple Linear Regression based GRU), based on multiple linear regression technique to integrate these deep learning predictive models. We utilized 67 monitoring stations of EPA (Environmental Protection Administration) in Taiwan to train and test the forecasting models with data collected from years 2013–2019, and use actual forecasted results to evaluate the performance of MLEGRU. The results are compared with other ensemble methods by using various factors, such as MAE (Mean Absolute Error), RMSE (Root Mean Square Error), and AEL3 (Absolute Error less than 3). The results reveal that the proposed MLEGRU model provides a better forecasting accuracy than other ensemble learning methods.

Application of deep learning techniques in supportting the diagnosis of pneumonia through X-ray images

In the context of the outbreak of the COVID-19 epidemic in Vietnam and around the world, an artificial intelligence application that accurately diagnoses pneumonia will help reduce time and human resources for medical examination and treatment. This helps patients receive timely treatment, reducing the risk of aggravation and death. This paper presents the characteristics of modern deep learning network architectures based on convolutional neural networks such as ResNet50, VGG16, Inception, DenseNet. Thereby performing a test to evaluate these models in the diagnosis of pneumonia using the Chest-Xray dataset. The test results show that the deep learning model using VGG16 deep learning network architecture gives the highest accuracy rate. This is the basis for proposing to build an application to support effective pneumonia diagnosis based on X-ray images.

Voice pathology detection by using the deep network architecture

Pathological voice disorders are among the conditions affecting negatively our daily life. The aim of this study is to introduce the new feature vector in the hybrid axis and multi-model in order to diagnose these disorders with more conventional methods. Two different databases are used, and the results are compared with the previous studies. Here, two types of fusion models (feature and decision level fusion) are used to increase the classification accuracy of the multi-model. The experimental results show that the proposed multi-model gives the highest classification accuracies with decision level fusion (DLF). Inspecting the results obtained from two databases, the highest accuracy rate (99.58%) is obtained with DLF. It is also seen from the experiments that the proposed feature vector helps to classify pathological data successfully, depending on their pathological conditions. Together with the proposed multi-model, both LSTM and CNN are found to be similarly successful in the classification of data in multi-model architecture.