QuFeX: quantum feature extraction module for hybrid quantum-classical deep neural networks

In modern deep learning, there is a recent and growing literature on the interplay between large-width asymptotics for deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed weights, and classes of Gaussian stochastic processes (SPs). Such an interplay has proved to be critical in several contexts of practical interest, e.g. Bayesian inference under Gaussian SP priors, kernel regression for infinite-wide deep NNs trained via gradient descent, and information propagation within infinite-wide NNs. Motivated by empirical analysis, showing the potential of replacing Gaussian distributions with Stable distributions for the NN's weights, in this paper we investigate large-width asymptotics for (fully connected) feed-forward deep Stable NNs, i.e. deep NNs with Stable-distributed weights. First, we show that as the width goes to infinity jointly over the NN's layers, a suitable rescaled deep Stable NN converges weakly to a Stable SP whose distribution is characterized recursively through the NN's layers. Because of the non-triangular NN's structure, this is a non-standard asymptotic problem, to which we propose a novel and self-contained inductive approach, which may be of independent interest. Then, we establish sup-norm convergence rates of a deep Stable NN to a Stable SP, quantifying the critical difference between the settings of ``joint growth and ``sequential growth of the width over the NN's layers. Our work extends recent results on infinite-wide limits for deep Gaussian NNs to the more general deep Stable NNs, providing the first result on convergence rates for infinite-wide deep NNs.

Deep neural networks are a powerful tool for feature learning and extraction. One area worth exploring in feature extraction using deep neural networks is efficient neural connectivity formation for faster feature learning and extraction. Motivated by findings of stochastic synaptic connectivity formation in the brain as well as the brain's uncanny ability to efficiently represent information, we propose the efficient learning and extraction of features via StochasticNets, where sparsely-connected deep neural networks can be formed via stochastic connectivity between neurons. Experimental results show that features learned using deep convolutional StochasticNets, with fewer neural connections than conventional deep convolutional neural networks, can allow for better or comparable classification accuracy than conventional deep neural networks. Finally, it was also shown that significant gains in feature extraction speed can be achieved in embedded applications using StochasticNets.

Chapter 11 - Computational intelligence on medical imaging with artificial neural networks

The impressive gain in performance obtained using deep neural networks (DNN) for various tasks encouraged us to apply DNN for image classification task. We have used a variant of DNN called Deep convolutional Neural Networks (DCNN) for feature extraction and image classification. Neural networks can be used for classification as well as for feature extraction. Our whole work can be better seen as two different tasks. In the first task, DCNN is used for feature extraction and classification task. In the second task, features are extracted using DCNN and then SVM, a shallow classifier, is used to classify the extracted features. Performance of these tasks is compared. Various configurations ofDCNNare used for our experimental studies.Among different architectures that we have considered, the architecture with 3 levels of convolutional and pooling layers, followed by a fully connected output layer is used for feature extraction. In task 1 DCNN extracted features are fed to a 2 hidden layer neural network for classification. In task 2 SVM is used to classify the features extracted by DCNN. Experimental studies show that the performance of υ-SVM classification on DCNN features is slightly better than the results of neural network classification on DCNN extracted features.

Comprehensive Study for Breast Cancer Using Deep Learning and Traditional Machine Learning

Deep learning based on structured deep neural networks has provided powerful applications in various fields. The structures imposed on the deep neural networks are crucial, which makes deep learning essentially different from classical schemes based on fully connected neural networks. One of the commonly used deep neural network structures is generated by convolutions. The produced deep learning algorithms form the family of deep convolutional neural networks. Despite of their power in some practical domains, little is known about the mathematical foundation of deep convolutional neural networks such as universality of approximation. In this paper, we propose a family of new structured deep neural networks: deep distributed convolutional neural networks. We show that these deep neural networks have the same order of computational complexity as the deep convolutional neural networks, and we prove their universality of approximation. Some ideas of our analysis are from ridge approximation, wavelets, and learning theory.

Automatic speech recognition has experienced a breathtaking progress in the last few years, partially thanks to the introduction of deep neural networks into their approaches. This evolution in speech recognition systems has spread across related areas such as language and speaker recognition, where deep neural networks have noticeably improved their performance. In this PhD thesis, we have explored different approaches to the tasks of speaker and language recognition, focusing on systems where deep neural networks become part of traditional pipelines, replacing some stages or the whole system itself. Specifically, in the first experimental block, end-to-end language recognition systems based on deep neural networks are analyzed, where the neural network is used as classifier directly, without the use of any other backend but performing the language recognition task from the scores (posterior probabilities) provided by the network. Besides, these research works are focused on two architectures, convolutional neural networks and long short-term memory (LSTM) recurrent neural networks, which are less demanding in terms of computational resources due to the reduced amount of free parameters in comparison with other deep neural networks. Thus, these systems constitute an alternative to classical i-vectors, and achieve comparable results to them, especially when dealing with short utterances. In particular, we conducted experiments comparing a system based on convolutional neural networks with classical Factor Analysis GMM and i-vector reference systems, and evaluate them on two different tasks from the National Institute of Standards and Technology (NIST) Language Recognition Evaluation (LRE) 2009: one focused on language-pairs and the other, on multi-class language identification. Results shown comparable performance of the convolutional neural network based approaches and some improvements are achieved when fusing both classical and neural network approaches. We also present the experiments performed with LSTM recurrent neural networks, which have proven their ability to model time depending sequences. We evaluate our LSTM-based language recognition systems on different subsets of the NIST LRE 2009 and 2015, where LSTM systems are able to outperform the reference i-vector system, providing a model with less parameters, although more prone to overfitting and not able to generalize as well as i-vector in mismatched datasets. In the second experimental block of this Dissertation, we explore one of the most prominent applications of deep neural networks in speech processing, which is their use as feature extractors. In this kind of systems, deep neural networks are used to obtain a frame-by-frame representation of the speech signal, the so-called bottleneck feature vector, which is learned directly by the network and is then used instead of traditional acoustic features as input in language and speaker recognition systems based on i-vectors. This approach revolutionized these two fields, since they highly outperformed classical systems which had been state-of-the-art for many year (i-vector based on acoustic features). Our analysis focuses on how different configurations of the neural network used as bottleneck feature extractor, and which is trained for automatic speech recognition, influences performance of resulting features for language and speaker recognition. For the case of language recognition, we compare bottleneck features from networks that vary their depth in terms of number of hidden layers, the position of the bottleneck layer where it comprises the information and the number of units (size) of this layer, which would influence the representation obtained by the network. With the set of experiments performed on bottleneck features for speaker recognition, we analyzed the influence of the type of features used to feed the network, their pre-processing and, in general, the optimization of the network for the task of feature extraction for speaker recognition, which might not mean the optimal configuration for ASR. Finally, the third experimental block of this Thesis proposes a novel approach for language recognition, in which the neural network is used to extract a fixed-length utterance-level representation of speech segments known as embedding, able to replace the classical i-vector, and overcoming the variable length sequence of feature provided by the bottleneck features. This embedding based approach has recently shown promising results for speaker verification tasks, and our proposed system was able to outperform a strong state-of-the-art reference i-vector system on the last challenging language recognition evaluations organized by NIST in 2015 and 2017. Thus, we analyze language recognition systems based on embeddings, and explore different deep neural network architectures and data augmentation techniques to improve results of our system. In general, these embeddings are a fair competitor to the well-established i-vector pipeline which allows replacing the whole i-vector model by a deep neural network. Furthermore, the network is able to extract complementary information to the one contained in the i-vectors, even from the same input features. All this makes us consider that this contribution is an interesting research line to explore in other fields.

In this study, the deep multi-layered Group Method of Data Handling (GMDH)-type neural network algorithm using principal component-regression analysis is applied to recognition problems of the right and left kidney regions. The deep multi-layered GMDH-type neural network algorithm can automatically organize the deep neural network architectures which have many hidden layers and these deep neural networks can identify the characteristics of very complex nonlinear systems. The architecture of the deep neural network with many hidden layers is automatically organized using the heuristic self-organization method, so as to minimize the prediction error criterion defined as Akaike's information criterion (AIC) or Prediction Sum of Squares (PSS). The heuristic self-organization method is a type of the evolutional computation. In this deep GMDH-type neural network, principal component-regression analysis is used as the learning algorithm of the weights in the deep GMDH-type neural network, and multi-colinearity does not occur and stable and accurate prediction values are obtained. This new algorithm is applied to the medical image recognitions of the right and left kidney regions. The optimum neural network architectures, which fit the complexity of the right and left kidney regions, are automatically organized and the right and left kidney regions are automatically recognized and extracted by the organized deep GMDH-type neural networks. The recognition results are compared with the conventional sigmoid function neural network trained using back propagation method and it is shown that this deep GMDH-type neural networks are useful for the medical image recognition problems of the right and left kidney regions.

BackgroundOver 70% of Americans regularly experience stress. Chronic stress results in cancer, cardiovascular disease, depression, and diabetes, and thus is deeply detrimental to physiological health and psychological wellbeing. Developing robust methods for the rapid and accurate detection of human stress is of paramount importance.MethodsPrior research has shown that analyzing physiological signals is a reliable predictor of stress. Such signals are collected from sensors that are attached to the human body. Researchers have attempted to detect stress by using traditional machine learning methods to analyze physiological signals. Results, ranging between 50 and 90% accuracy, have been mixed. A limitation of traditional machine learning algorithms is the requirement for hand-crafted features. Accuracy decreases if features are misidentified. To address this deficiency, we developed two deep neural networks: a 1-dimensional (1D) convolutional neural network and a multilayer perceptron neural network. Deep neural networks do not require hand-crafted features but instead extract features from raw data through the layers of the neural networks. The deep neural networks analyzed physiological data collected from chest-worn and wrist-worn sensors to perform two tasks. We tailored each neural network to analyze data from either the chest-worn (1D convolutional neural network) or wrist-worn (multilayer perceptron neural network) sensors. The first task was binary classification for stress detection, in which the networks differentiated between stressed and non-stressed states. The second task was 3-class classification for emotion classification, in which the networks differentiated between baseline, stressed, and amused states. The networks were trained and tested on publicly available data collected in previous studies.ResultsThe deep convolutional neural network achieved 99.80% and 99.55% accuracy rates for binary and 3-class classification, respectively. The deep multilayer perceptron neural network achieved 99.65% and 98.38% accuracy rates for binary and 3-class classification, respectively. The networks’ performance exhibited a significant improvement over past methods that analyzed physiological signals for both binary stress detection and 3-class emotion classification.ConclusionsWe demonstrated the potential of deep neural networks for developing robust, continuous, and noninvasive methods for stress detection and emotion classification, with the end goal of improving the quality of life.

Deep artificial neural networks have become the go-to method for many machine learning tasks. In the field of computer vision, deep convolutional neural networks achieve state-of-the-art performance for tasks such as classification, object detection, or instance segmentation. As deep neural networks become more and more complex, their inner workings become more and more opaque, rendering them a “black box” whose decision making process is no longer comprehensible. In recent years, various methods have been presented that attempt to peek inside the black box and to visualize the inner workings of deep neural networks, with a focus on deep convolutional neural networks for computer vision. These methods can serve as a toolbox to facilitate the design and inspection of neural networks for computer vision and the interpretation of the decision making process of the network. Here, we present the new tool Interactive Feature Localization in Deep neural networks (IFeaLiD) which provides a novel visualization approach to convolutional neural network layers. The tool interprets neural network layers as multivariate feature maps and visualizes the similarity between the feature vectors of individual pixels of an input image in a heat map display. The similarity display can reveal how the input image is perceived by different layers of the network and how the perception of one particular image region compares to the perception of the remaining image. IFeaLiD runs interactively in a web browser and can process even high resolution feature maps in real time by using GPU acceleration with WebGL 2. We present examples from four computer vision datasets with feature maps from different layers of a pre-trained ResNet101. IFeaLiD is open source and available online at https://ifealid.cebitec.uni-bielefeld.de.

Adaptive deep dynamic programming for integrated frequency control of multi-area multi-microgrid systems

A convergence analysis of Nesterov’s accelerated gradient method in training deep linear neural networks

Deep artificial neural networks have become a good alternative to classical forecasting methods in solving forecasting problems. Popular deep neural networks classically use additive aggregation functions in their cell structures. It is available in the literature that the use of multiplicative aggregation functions in shallow artificial neural networks produces successful results for the forecasting problem. A type of high-order shallow artificial neural network that uses multiplicative aggregation functions is the dendritic neuron model artificial neural network, which has successful forecasting performance. In this study, the transformation of the dendritic neuron model turned into a multi-output architecture. A new dendritic cell based on the multi-output dendritic neuron model and a new deep artificial neural network is proposed. The training of this new deep dendritic artificial neural network is carried out with the differential evolution algorithm. The forecasting performance of the deep dendritic artificial neural network is compared with basic classical forecasting methods and some recent shallow and deep artificial neural networks over stock market time series. As a result, it has been observed that deep dendritic artificial neural network produces very successful forecasting results for the forecasting problem.

Reservoirs of natural gas and gas condensate have been proposed as a potential for providing affordable and cleaner energy sources to the global population growth and industrialization expansion simultaneously. This work evaluates reservoir simulation for production optimization using Deep Neural network - artificial neural network (DNN-ANN) model to predict the dew point pressure in gas condensate reservoirs from Field-X in the Niger Delta Region of Nigeria. The dew-point pressure (DPP) of gas condensate reservoirs was estimated as a function of gas composition, reservoir temperature, molecular weight and specific gravity of heptane plus percentage. Results obtained show that the mean relative error (MRE) and R-squared (R2) are 0.99965 and 3.35%, respectively, indicating that the model is excellent in predicting DPP values. The Deep Neural Network - Artificial Neural Network (DNN-ANN) model is also evaluated in comparison to earlier models created by previous authors. It was recommended that the DNN - ANN model developed in this study could be applied to reservoir simulation and modeling well performance analysis, reservoir engineering problems and production optimization.

In the medical domain, brain image classification is an extremely challenging field. Medical images play a vital role in making the doctor's precise diagnosis and in the surgery process. Adopting intelligent algorithms makes it feasible to detect the lesions of medical images quickly, and it is especially necessary to extract features from medical images. Several studies have integrated multiple algorithms toward medical images domain. Concerning feature extraction from the medical image, a vast amount of data is analyzed to achieve processing results, helping physicians deliver more precise case diagnoses. Image processing mechanism becomes extensive usage in medical science to advance the early detection and treatment aspects. In this aspect, this paper takes tumor, and healthy images as the research object and primarily performs image processing and data augmentation process to feed the dataset to the neural networks. Deep neural networks (DNN), to date, have shown outstanding achievement in classification and segmentation tasks. Carrying this concept into consideration, in this study, we adopted a pre-trained model Resnet_50 for image analysis. The paper proposed three diverse neural networks, particularly DNN, CNN, and ResNet-50. Finally, the splitting dataset is individually assigned to each simplified neural network. Once the image is classified as a tumor accurately, the OTSU segmentation is employed to extract the tumor alone. It can be examined from the experimental outcomes that the ResNet-50 algorithm shows high accuracy 0.996, precision 1.00 with best F1 score 1.0, and minimum test losses of 0.0269 in terms of Brain tumor classification. Extensive experiments prove our offered tumor detection segmentation efficiency and accuracy. To this end, our approach is comprehensive sufficient and only requires minimum pre-and post-processing, which allows its adoption in various medical image classification & segmentation tasks.