Deep Neural Nets Research Articles

Time Series Prediction of Wastewater Flow Rate by Bidirectional LSTM Deep Learning

This paper not only addresses a feasible strategy in predicting time series or sequences by using deep neural nets such as bi-LSTM (bidirectional Long Short-Term Memory), but also demonstrates fairly good results of forecasting wastewater flow rate for a municipal wastewater treatment plant in a practical sense. The basic procedures of time series prediction by deep learning are to collect the past information of all available states for deep learning and to utilize p-step ahead delays of a no-training interval with a sliding time window. Therefore, the sequence-to-point p-step prediction of sewage flow of Yangju wastewater treatment plant could be made possible by using bi-LSTM in accordance with this fundamental principle.

Methods for correcting inference based on outcomes predicted by machine learning

Many modern problems in medicine and public health leverage machine-learning methods to predict outcomes based on observable covariates. In a wide array of settings, predicted outcomes are used in subsequent statistical analysis, often without accounting for the distinction between observed and predicted outcomes. We call inference with predicted outcomes postprediction inference. In this paper, we develop methods for correcting statistical inference using outcomes predicted with arbitrarily complicated machine-learning models including random forests and deep neural nets. Rather than trying to derive the correction from first principles for each machine-learning algorithm, we observe that there is typically a low-dimensional and easily modeled representation of the relationship between the observed and predicted outcomes. We build an approach for postprediction inference that naturally fits into the standard machine-learning framework where the data are divided into training, testing, and validation sets. We train the prediction model in the training set, estimate the relationship between the observed and predicted outcomes in the testing set, and use that relationship to correct subsequent inference in the validation set. We show our postprediction inference (postpi) approach can correct bias and improve variance estimation and subsequent statistical inference with predicted outcomes. To show the broad range of applicability of our approach, we show postpi can improve inference in two distinct fields: modeling predicted phenotypes in repurposed gene expression data and modeling predicted causes of death in verbal autopsy data. Our method is available through an open-source R package: https://github.com/leekgroup/postpi.

Asynchronous parallel stochastic Quasi-Newton methods

Although first-order stochastic algorithms, such as stochastic gradient descent, have been the main force to scale up machine learning models, such as deep neural nets, the second-order quasi-Newton methods start to draw attention due to their effectiveness in dealing with ill-conditioned optimization problems. The L-BFGS method is one of the most widely used quasi-Newton methods. We propose an asynchronous parallel algorithm for stochastic quasi-Newton (AsySQN) method. Unlike prior attempts, which parallelize only the calculation for gradient or the two-loop recursion of L-BFGS, our algorithm is the first one that truly parallelizes L-BFGS with a convergence guarantee. Adopting the variance reduction technique, a prior stochastic L-BFGS, which has not been designed for parallel computing, reaches a linear convergence rate. We prove that our asynchronous parallel scheme maintains the same linear convergence rate but achieves significant speedup. Empirical evaluations in both simulations and benchmark datasets demonstrate the speedup in comparison with the non-parallel stochastic L-BFGS, as well as the better performance than first-order methods in solving ill-conditioned problems.

Super-resolution and denoising of 4D-Flow MRI using physics-Informed deep neural nets

Background and Objective: Time resolved three-dimensional phase contrast magnetic resonance imaging (4D-Flow MRI) has been used to non-invasively measure blood velocities in the human vascular system. However, issues such as low spatio-temporal resolution, acquisition noise, velocity aliasing, and phase-offset artifacts have hampered its clinical application. In this research, we developed a purely data-driven method for super-resolution and denoising of 4D-Flow MRI.Methods: The flow velocities, pressure, and the MRI image magnitude are modeled as a patient-specific deep neural net (DNN). For training, 4D-Flow MRI images in the complex Cartesian space are used to impose data-fidelity. Physics of fluid flow is imposed through regularization. Creative loss function terms have been introduced to handle noise and super-resolution. The trained patient-specific DNN can be sampled to generate noise-free high-resolution flow images. The proposed method has been implemented using the TensorFlow DNN library and tested on numerical phantoms and validated in-vitro using high-resolution particle image velocitmetry (PIV) and 4D-Flow MRI experiments on transparent models subjected to pulsatile flow conditions.Results: In case of numerical phantoms, we were able to increase spatial resolution by a factor of 100 and temporal resolution by a factor of 5 compared to the simulated 4D-Flow MRI. There is an order of magnitude reduction of velocity normalized root mean square error (vNRMSE). In case of the in-vitro validation tests with PIV as reference, there is similar improvement in spatio-temporal resolution. Although the vNRMSE is reduced by 50%, the method is unable to negate a systematic bias with respect to the reference PIV that is introduced by the 4D-Flow MRI measurement.Conclusions: This work has demonstrated the feasibility of using the readily available machinery of deep learning to enhance 4D-Flow MRI using a purely data-driven method. Unlike current state-of-the-art methods, the proposed method is agnostic to geometry and boundary conditions and therefore eliminates the need for tedious tasks such as accurate image segmentation for geometry, image registration, and estimation of boundary flow conditions. Arbitrary regions of interest can be selected for processing. This work will lead to user-friendly analysis tools that will enable quantitative hemodynamic analysis of vascular diseases in a clinical setting.

Incorporating physical constraints in a deep probabilistic machine learning framework for coarse-graining dynamical systems

Data-based discovery of effective, coarse-grained (CG) models of high-dimensional dynamical systems presents a unique challenge in computational physics and particularly in the context of multiscale problems. The present paper offers a data-based, probabilistic perspective that enables the quantification of predictive uncertainties. One of the outstanding problems has been the introduction of physical constraints in the probabilistic machine learning objectives. The primary utility of such constraints stems from the undisputed physical laws such as conservation of mass, energy etc. that they represent. Furthermore and apart from leading to physically realistic predictions, they can significantly reduce the requisite amount of training data which for high-dimensional, multiscale systems are expensive to obtain (Small Data regime). We formulate the coarse-graining process by employing a probabilistic state-space model and account for the aforementioned equality constraints as virtual observables in the associated densities. We demonstrate how deep neural nets in combination with probabilistic inference tools can be employed to identify the coarse-grained variables and their evolution model without ever needing to define a fine-to-coarse (restriction) projection and without needing time-derivatives of state variables.We advocate a sparse Bayesian learning perspective which avoids overfitting and reveals the most salient features in the CG evolution law. The formulation adopted enables the quantification of a crucial, and often neglected, component in the CG process, i.e. the predictive uncertainty due to information loss. Furthermore, it is capable of reconstructing the evolution of the full, fine-scale system and therefore the observables of interest need not be selected a priori. We demonstrate the efficacy of the proposed framework by applying it to systems of interacting particles and a series of images of a nonlinear pendulum. In both cases we identify the underlying coarse dynamics and can generate extrapolative predictions including the forming and propagation of a shock for the particle systems and a stable trajectory in the phase space for the pendulum.

���������� ������������ ������ ��������� ���� �����

The task of predicting the vessel kinematic parameters arises in the dead reckoning mode of its track calculation. This task is solved in the framework of the traditional approach based on using the ordinary differential equations. As a rule, while constructing these equations, the difficulties arise. The difficulties result from selection of the algorithms which are used for the certain forces calculation. These algorithms are not universal, and their working ability could not be guaranteed for all sailing conditions. The task of predicting the vessel kinematic parameters can be performed as the task of approximation of multiple variables functions. Neural networks are known as universal algorithms of such approximation. In the paper, a generalized model of the vessel dead reckoning on the basis of neural networks is performed. An algorithm of its operation is performed as well. The base of the model is formed with deep neural nets which are constructed through a cascade connection between two-layered (shallow) networks. There are two types of neural networks in the structure of the model: main and auxiliary nets. The main nets predict kinematic parameters using the information about force influences on the vessel. The auxiliary nets use the information about only the ship kinematic history. The model configuration allows two modes of operation. The first mode is used when dead reckoning is performed in the usual way. The second mode is used when the reiable information from a log or (and) gyrocompass is unavailable. In the case of both sensors failure the model reproduces the vessel dynamics in the horizontal plane (with three degree of freedom) thus substituting the system of corresponding differential equations. The input signals of the main nets are formed in accordance with the configuration vector of the model. The capabilities of learning the model and its using in practice are also analysed.

Human experts vs. machines in taxa recognition

The step of expert taxa recognition currently slows down the response time of many bioassessments. Shifting to quicker and cheaper state-of-the-art machine learning approaches is still met with expert scepticism towards the ability and logic of machines. In our study, we investigate both the differences in accuracy and in the identification logic of taxonomic experts and machines. We propose a systematic approach utilizing deep Convolutional Neural Nets and extensively evaluate it over a multi-pose taxonomic dataset with hierarchical labels specifically created for this comparison. We also study the prediction accuracy on different ranks of taxonomic hierarchy in detail. We compare the results of Convolutional Neural Networks to human experts and support vector machines. Our results revealed that human experts using actual specimens yield the lowest classification error (CE¯=6.1%). However, a much faster, automated approach using deep Convolutional Neural Nets comes close to human accuracy (CE¯=11.4%) when a typical flat classification approach is used. Contrary to previous findings in the literature, we find that for machines following a typical flat classification approach commonly used in machine learning performs better than forcing machines to adopt a hierarchical, local per parent node approach used by human taxonomic experts (CE¯=13.8%). Finally, we publicly share our unique dataset to serve as a public benchmark dataset in this field.

Foreign accent classification using deep neural nets

Foreign accent classification using deep neural nets

Road safety performance function analysis with visual feature importance of deep neural nets

Road safety performance function ( SPF ) analysis using data-driven and nonparametric methods, especially recent developed deep learning approaches, has gained increasing achievements. However, due to the learning mechanisms are hidden in a “ black box ” in deep learning, traffic features extraction and intelligent importance analysis are still unsolved and hard to generate. This paper focuses on this problem using a deciphered version of deep neural networks ( DNN ) , one of the most popular deep learning models. This approach builds on visualization, feature importance and sensitivity analysis, can evaluate the contributions of input variables on model ʼ s “ black box ” feature learning process and output decision. Firstly, a visual feature importance ( ViFI ) method that describes the importance of input features is proposed by adopting diagram and numerical-analysis. Secondly, by observing the change of weights using ViFI on unsupervised training and fine-tuning of DNN, the final contributions of input features are calculated according to importance equations for both steps that we proposed. Sequentially, a case study based on a road SPF analysis is demonstrated, using data collected from a major Canadian highway, Highway 401. The proposed method allows effective deciphering of the model ʼ s inner workings and allows the significant features to be identified and the bad features to be eliminated. Finally, the revised dataset is used in crash modeling and vehicle collision prediction, and the testing result verifies that the deciphered and revised model achieves state-of-the-art performance.

Positive sentiments as coping mechanisms and path to resilience: the case of Qatar blockade

ABSTRACT Existing research on coping accentuates the role of positive emotions as defensive mechanisms to cope with stressful situations and the ensuing negative emotions. The same literature justifies the long-term effects of positive emotions that help build lasting resilience. Grounded in theories of coping and resilience, this paper (1) identifies the emotions that people actuate to cope with adversaries and (2) evaluates the resulting long-lasting adaptation and resilience. To do this, we examined the emotions felt by Qatar residents due to a land, sea, and air blockade enforced by neighbouring counties. Accordingly, we analysed 160,000 Arabic tweets originating from Qatar between June-2017 and March-2018 using a novel machine-learning algorithm termed Weighted Conditional Probability. Our algorithm achieved state-of-the-art performance when compared with the often-used Support Vector Machine, Naïve Bayes and Deep Neural Nets algorithms. Results show that, while Qatar residents experienced an emotional roller coaster during the blockade, they used positive emotions like love and optimism to cope with adversities and accompanying emotions of fear and anger. Moreover, our analysis reveals that their adaptive resilient capacities gradually strengthened during the nine months of blockade. The study supports the renowned theory of positive emotions using an advanced methodology and a large-scale dataset.

Exploiting Bias for Cooperative Planning in Multi-Agent Tree Search

Graph search over states and actions is a valuable tool for robotic planning and navigation. However, the required computation is sensitive to the size of the state and action spaces, a fact which is further exacerbated in multi-agent planning by the number of agents and the presence of sparse reward signals dependent on the cooperation of agents. To tackle these problems, we introduce an algorithm that is pre-trained in a centralized fashion but implemented on robots in a distributed way at runtime. The centralized portion uses imitation learning to iteratively construct policies that help guide an individual agent‘s own runtime search as well as predict other agents’ future actions by exploiting previously discovered joint actions. Our algorithm includes a novel method of tree search based on a mixture of the individual and joint action space, which can be interpreted as a cascading effect where agents are biased by exploration of new actions, exploitation of previously profitable ones, and recommendation provided by deep neural nets. Simulations show the efficacy of the proposed method in cooperative scenarios with sparse rewards.

Predicting credit card fraud with Sarbanes‐Oxley assessments and Fama‐French risk factors

SummaryThis research developed and tested machine learning models to predict significant credit card fraud in corporate systems using Sarbanes‐Oxley (SOX) reports, news reports of breaches and Fama‐French risk factors (FF). Exploratory analysis found that SOX information predicted several types of security breaches, with the strongest performance in predicting credit card fraud. A systematic tuning of hyperparamters for a suite of machine learning models, starting with a random forest, an extremely‐randomized forest, a random grid of gradient boosting machines (GBMs), a random grid of deep neural nets, a fixed grid of general linear models where assembled into two trained stacked ensemble models optimized for F1 performance; an ensemble that contained all the models, and an ensemble containing just the best performing model from each algorithm class. Tuned GBMs performed best under all conditions. Without FF, models yielded an AUC of 99.3% and closeness of the training and validation matrices confirm that the model is robust. The most important predictors were firm specific, as would be expected, since control weaknesses vary at the firm level. Audit firm fees were the most important non‐firm‐specific predictors. Adding FF to the model rendered perfect prediction (100%) in the trained confusion matrix and AUC of 99.8%. The most important predictors of credit card fraud were the FF coefficient for the High book‐to‐market ratio Minus Low factor. The second most influential variable was the year of reporting, and third most important was the Fama‐French 3‐factor model R2 – together these described most of the variance in credit card fraud occurrence. In all cases the four major SOX specific opinions rendered by auditors and the signed SOX report had little predictive influence.

Gradient-driven unsupervised video segmentation using deep learning techniques

We propose a three-dimensional video segmentation method using deep learning convolutional neural nets. The algorithm utilizes the local gradient computed at each pixel location together with the global boundary map acquired through deep learning methods to generate initial pixel groups by traversing from low to high gradient regions. A local clustering method is then employed to refine these initial pixel groups. The refined subvolumes in the homogeneous regions of video are selected as initial seeds and iteratively combined with adjacent groups based on intensity similarities. The volume growth is terminated at the color boundaries of the video. The oversegments obtained from the above steps are then merged hierarchically by a multivariate approach yielding a final segmentation map for each frame. The results show that our proposed methodology compares favorably well, on a qualitative and quantitative level, in segmentation quality and computational efficiency, with the latest state-of-the-art techniques utilizing the video segmentation benchmark dataset.

On the Difference between the Information Bottleneck and the Deep Information Bottleneck.

Combining the information bottleneck model with deep learning by replacing mutual information terms with deep neural nets has proven successful in areas ranging from generative modelling to interpreting deep neural networks. In this paper, we revisit the deep variational information bottleneck and the assumptions needed for its derivation. The two assumed properties of the data, X and Y, and their latent representation T, take the form of two Markov chains and . Requiring both to hold during the optimisation process can be limiting for the set of potential joint distributions . We, therefore, show how to circumvent this limitation by optimising a lower bound for the mutual information between T and Y: , for which only the latter Markov chain has to be satisfied. The mutual information can be split into two non-negative parts. The first part is the lower bound for , which is optimised in deep variational information bottleneck (DVIB) and cognate models in practice. The second part consists of two terms that measure how much the former requirement is violated. Finally, we propose interpreting the family of information bottleneck models as directed graphical models, and show that in this framework, the original and deep information bottlenecks are special cases of a fundamental IB model.

Tensoring Volatility Calibration

Inspired by a series of remarkable papers in recent years, that use Deep Neural Nets to substantially speed up the calibration of pricing models, we investigate the use of Chebyshev Tensors instead of Deep Neural Nets. Given that Chebyshev Tensors can be, under certain circumstances, more efficient than Deep Neural Nets at exploring the input space of the function to be approximated – due to their exponential convergence – the problem of calibration of pricing models seems, a priori, a good case where Chebyshev Tensors can excel. In this piece of research, we built Chebyshev Tensors — either directly or with the help of the Tensor Extension Algorithms — to tackle the computational bottleneck associated with the calibration of the rough Bergomi volatility model. Results are encouraging as the accuracy of model calibration via Chebyshev Tensors is similar to that when using Deep Neural Nets, but with building efforts that range between 5 and 100 times more efficient in the experiments run. Our tests indicate that when using Chebyshev Tensors, the calibration of the rough Bergomi volatility model is around 40, 000 times more efficient than if calibrated via “brute-force” (using the pricing function).

Block Coordinate Regularization by Denoising

We consider the problem of reconstructing an image from its noisy measurements using a prior specified only with an image denoiser. Recent work on plug-and-play priors (PnP) and regularization by denoising (RED) has shown the state-of-the-art performance of image reconstruction algorithms under such priors in a range of imaging problems. In this work, we develop a new block coordinate RED algorithm that decomposes a large-scale estimation problem into a sequence of updates over a small subset of the unknown variables. We theoretically analyze the convergence of the algorithm and discuss its relationship to the traditional proximal optimization. Our analysis complements and extends recent theoretical results for RED-based estimation methods. We numerically validate our method using several denoising priors, including those based on deep neural nets.

Human activities recognition with a single writs IMU via a Variational Autoencoder and android deep recurrent neural nets

Human Activity Recognition (HAR) is an active research field because of its versatility towards various application areas such as healthcare and lifecare. In this study, a novel HAR system is proposed based on an autoencoder for denoising and Recurrent Neural Network (RNN) for classification with a single Inertial Measurement Unit (IMU) located on a dominant wrist. A Variational Autoencoder (VAE) is built to denoise IMU signals which improves HAR by Android Deep RNN. Evaluating our VAE and Android Deep RNN HAR system is done in two ways. First, the system is tested on a PC using discrete epochs of activities of daily living. Our results show that VAE improves Signal-to-Noise Ratio (SNR) of the IMU signal from 8.78 to 17.26 dB. In turn, HAR improves from 89.29% to 95.11% in F1-score and from 90.38% to 95.47% in accuracy. Secondly, the system is tested on an Android device (i.e., smartphone) using continuous activity signals. This is done by transferring the PC HAR system to an Android HAR App (i.e., Android Deep RNN). We have achieved 86.13% and 95.09% in accuracy without and with VAE respectively. Our results demonstrate that HAR can be achieved in real-time on a standalone smart device with a single IMU for lifelogging services.

Unsupervised Multi-Discriminator Generative Adversarial Network for Lung Nodule Malignancy Classification

Computer-aided diagnosis systems with deep learning frameworks have been used to identify benign and malignant pulmonary nodules in lung cancer diagnosis. It's commonly known that a premise of training complex deep neural nets is the large-scale labeled datasets. However, the abundance of labeled datasets is usually unavailable in many medical image domains. This factor can lead to the poor generalization performance of deep learning models. In this paper, we propose a novel multi-discriminator generative adversarial network model combined with an encoder for the classification of benign and malignant pulmonary nodules. To the best of our knowledge, we are the first to apply unsupervised learning to identify benign and malignant lung nodules. Firstly, we use a multi-discriminator generative adversarial network to build a generative model trained with unlabeled benign lung nodule images. Then an encoder is combined with the trained generative model to establish a mapping of benign pulmonary nodule images to the latent space. The benign and malignant lung nodules are scored by calculating the GAN discriminator feature loss and image reconstruction loss. The model yields high anomaly scores on malignant images and low anomaly scores on benign images. Experimental results show that our method with only a small number of unlabeled datasets could achieve more competitive results compared with other supervised deep learning approaches.

EnResNet: ResNets Ensemble via the Feynman--Kac Formalism for Adversarial Defense and Beyond

Empirical adversarial risk minimization is a widely used mathematical framework to robustly train deep neural nets that are resistant to adversarial attacks. However, both natural and robust accura...

Hamiltonian Markov chain Monte Carlo for partitioned sample spaces with application to Bayesian deep neural nets

Allocating computation over multiple chains to reduce sampling time in MCMC is crucial in making MCMC more applicable in the state of the art models such as deep neural networks. One of the parallelization schemes for MCMC is partitioning the sample space to run different MCMC chains in each component of the partition (VanDerwerken and Schmidler in Parallel Markov chain Monte Carlo. arXiv:1312.7479, 2013; Basse et al. in Artificial intelligence and statistics, pp 1318–1327, 2016). In this work, we take Basse et al. (2016)’s bridge sampling approach and apply constrained Hamiltonian Monte Carlo on partitioned sample spaces. We propose a random dimension partition scheme that combines well with the constrained HMC. We empirically show that this approach can expedite MCMC sampling for any unnormalized target distribution such as Bayesian neural network in a high dimensional setting. Furthermore, in the presence of multi-modality, this algorithm is expected to be more efficient in mixing MCMC chains when proper partition elements are chosen.