Hidden Units Research Articles

Dynamic Wide and Deep Neural Network for Hyperspectral Image Classification

Recently, deep learning has been successfully and widely used in hyperspectral image (HSI) classification. Considering the difficulty of acquiring HSIs, there are usually a small number of pixels used as the training instances. Therefore, it is hard to fully use the advantages of deep learning networks; for example, the very deep layers with a large number of parameters lead to overfitting. This paper proposed a dynamic wide and deep neural network (DWDNN) for HSI classification, which includes multiple efficient wide sliding window and subsampling (EWSWS) networks and can grow dynamically according to the complexity of the problems. The EWSWS network in the DWDNN was designed both in the wide and deep direction with transform kernels as hidden units. These multiple layers of kernels can extract features from the low to high level, and because they are extended in the wide direction, they can learn features more steadily and smoothly. The sliding windows with the stride and subsampling were designed to reduce the dimension of the features for each layer; therefore, the computational load was reduced. Finally, all the weights were only from the fully connected layer, and the iterative least squares method was used to compute them easily. The proposed DWDNN was tested with several HSI data including the Botswana, Pavia University, and Salinas remote sensing datasets with different numbers of instances (from small to big). The experimental results showed that the proposed method had the highest test accuracies compared to both the typical machine learning methods such as support vector machine (SVM), multilayer perceptron (MLP), radial basis function (RBF), and the recently proposed deep learning methods including the 2D convolutional neural network (CNN) and the 3D CNN designed for HSI classification.

Hospital Length of Stay Prediction based on Patient Examination Using General features

As of the year 2020, Indonesia has the fourth most populous country in the world. With Indonesia’s population expected to continuously grow, the increase in provision of healthcare needs to match its steady population growth. Hospitals are central in providing healthcare to the general masses, especially for patients requiring medical attention for an extended period of time. Length of Stay (LOS), or inpatient treatment, covers various treatments that are offered by hospitals, such as medical examination, diagnosis, treatment, and rehabilitation. Generally, hospitals determine the LOS by calculating the difference between the number of admissions and the number of discharges. However, this procedure is shown to be unproductive for some hospitals. A cost-effective way to improve the productivity of hospital is to utilize Information Technology (IT). In this paper, we create a system for predicting LOS using Neural Network (NN) using a sample of 3055 subjects, consisting of 30 input attributes and 1 output attribute. The NN default parameter experiment and parameter optimization with grid search as well as random search were carried out. Our results show that parameter optimization using the grid search technique give the highest performance results with an accuracy of 94.7403% on parameters with a value of Epoch 50, hidden unit 52, batch size 4000, Adam optimizer, and linear activation. Our designated system can be utilised by hospitals in improving their effectiveness and efficiency, owing to better prediction of LOS and better visualization of LOS done by web visualization.

Research on HR management based on RNN expectation theory

RNN convolutional neural network is an important part of deep learning algorithm. Compared with fully connected neural network, the biggest difference is that the hidden units are not independent of each other. Not only the hidden layer neurons are irrelated to each other, but also the state of the current hidden layer cells is affected by the historical input data. This feature enables it to extract the temporal relationship of temporal data structure. One of the current outputs of one sequence is also related to the previous output. The specific expression is that the network will memorize the information before and apply it to the calculation of current output, the nodes between hidden layers are no longer connected but connected. Moreover, the input of hidden layer includes not only the output of input layer but also the output of hidden layer at the last time. RNN neural network is widely used in natural language processing (NLP): machine translation, speech recognition, image description generation, text similarity calculation. The DAFDC-RNN model in this paper is verified by the financial and real estate industry data from 2019 to 2020, which shows that the model has excellent stability and sensitivity. Compared with CNN model, it increases by 34.3% and 24.9%. The advantages of data processing as follow: it can process a large number of time series data, the processing capacity of data set is large, it can process 1000 + data, and it has more data base for time series prediction. The disadvantage as follow: Compared with CNN model, the working time of CNN model is increased by 48%, but the model has strong stability and adaptability. Taking the data from 2019 to 2020 as the input of DAFDC-RNN time series forecasting model, the salary forecast of real estate and financial industry in 2021 is obtained. Based on this data, the industry expectation of Finance and real estate industry under human resource management under expectation theory is reduced.

Author identification in bibliographic data using deep neural networks

Author name disambiguation (AND) is a challenging task for scholars who mine bibliographic information for scientific knowledge. A constructive approach for resolving name ambiguity is to use computer algorithms to identify author names. Some algorithm-based disambiguation methods have been developed by computer and data scientists. Among them, supervised machine learning has been stated to produce decent to very accurate disambiguation results. This paper presents a combination of principal component analysis (PCA) as a feature reduction and deep neural networks (DNNs), as a supervised algorithm for classifying AND problems. The raw data is grouped into four classes, i.e., synonyms, homonyms, homonyms-synonyms, and non-homonyms-synonyms classification. We have taken into account several hyperparameters tuning, such as learning rate, batch size, number of the neuron and hidden units, and analyzed their impact on the accuracy of results. To the best of our knowledge, there are no previous studies with such a scheme. The proposed DNNs are validated with other ML techniques such as Naïve Bayes, random forest (RF), and support vector machine (SVM) to produce a good classifier. By exploring the result in all data, our proposed DNNs classifier has an outperformed other ML technique, with accuracy, precision, recall, and F1-score, which is 99.98%, 97.98%, 97.86%, and 99.99%, respectively. In the future, this approach can be easily extended to any dataset and any bibliographic records provider.

Uncertainty of Rules Extracted from Artificial Neural Networks

ABSTRACT Artificial neural networks evolve into deep learning recently and perform well in various fields, such as image and speech recognition and translation. However, there is a problem that it is difficult for a person to understand what exactly the trained knowledge of an artificial neural network. As one of the methods for solving the problem of the artificial neural network, rule extraction methods had been devised. In this study, rules are extracted from artificial neural networks using ordered-attribute search (OAS) algorithm, which is one of the methods of extracting rules from trained neural networks, and the rules are analyzed to improve the extracted rules. As a result, we found that when the output value of the hidden layer has an intermediate value that is not close to 0 or 1 after passing through the sigmoid function, the problem of rule uncertainty occurs and affects the accuracy of the rules. In order to solve the uncertainty problem of the rules, we applied the hidden unit clarification method and suggested that it is possible to extract the efficient rule by binarizing the hidden layer output value. In addition, we extracted CDRPs (critical data routing paths) from the trained neural networks and used CDRPs to prune the extracted rules, which showed that the uncertainty problem of rules can be improved.

Uncertainty quantification in fault detection using convolutional neural networks

Segmentation of faults based on seismic images is an important step in reservoir characterization. With the recent developments of deep-learning methods and the availability of massive computing power, automatic interpretation of seismic faults has become possible. The likelihood of occurrence for a fault can be quantified using a sigmoid function. Our goal is to quantify the fault model uncertainty that is generally not captured by deep-learning tools. We have used the dropout approach, a regularization technique to prevent overfitting and coadaptation in hidden units, to approximate the Bayesian inference and estimate the principled uncertainty over functions. Particularly, the variance of the learned model has been decomposed into aleatoric and epistemic parts. Our method is applied to a real data set from the Netherlands F3 block with two different dropout ratios in convolutional neural networks. The aleatoric uncertainty is irreducible because it relates to the stochastic dependency within the input observations. As the number of Monte Carlo realizations increases, the epistemic uncertainty asymptotically converges and the model standard deviation decreases because the variability of the model parameters is better simulated or explained with a larger sample size. This analysis can quantify the confidence to use fault predictions with less uncertainty. In addition, the analysis suggests where more training data are needed to reduce the uncertainty in low-confidence regions.

SDLER: stacked dedupe learning for entity resolution in big data era

In the Big Data Era, Entity Resolution (ER) faces many challenges such as high scalability, the coexistence of complex similarity metrics, tautonymy and synonym, and the requirement of Data Quality Evaluation. Moreover, despite more than seventy years of development efforts, there is still a high demand for democratizing ER to reduce human participation in tuning parameters, data labeling, defining blocking functions, and feature engineering. This study aimed to explore a novel Stacked Dedupe Learning ER system with high accuracy and efficiency. The study evaluated sophisticated composition methods, such as Bidirectional Recurrent Neural Networks (BiRNNs) and Long Short-Term Memory (LSTM) hidden units, to renovate each tuple to word representation distribution in a sense to capture similarities amidst tuples. Also, pre-trained words embedding where they were not available, ways to learn and tune Word Representation Distribution customized for ER tasks under different scenarios were considered. More so, the Locality Sensitive Hashing (LSH) based blocking approach, which considered the entire attributes of a tuple and produced slighter blocks, compared with traditional methods with few attributes, were assessed. The algorithm was tested on multiple datasets namely benchmarks, and multi-lingual data. The experimental results showed that Stacked Dedupe Learning achieves high quality and good performance, and scales well compared to the existing solutions.

Gamma Boltzmann Machine for Audio Modeling

This paper presents an energy-based probabilistic model that handles nonnegative data in consideration of both linear and logarithmic scales. In audio applications, magnitude of time-frequency representation, including spectrogram, is regarded as one of the most important features. Such magnitude-based features have been extensively utilized in learning-based audio processing. Since a logarithmic scale is important in terms of auditory perception, the features are usually computed with a logarithmic function. That is, a logarithmic function is applied within the computation of features so that a learning machine does not have to explicitly model the logarithmic scale. We think in a different way and propose a restricted Boltzmann machine (RBM) that simultaneously models linear- and log-magnitude spectra. RBM is a stochastic neural network that can discover data representations without supervision. To manage both linear and logarithmic scales, we define an energy function based on both scales. This energy function results in a conditional distribution (of the observable data, given hidden units) that is written as the gamma distribution, and hence the proposed RBM is termed gamma-Bernoulli RBM. The proposed gamma-Bernoulli RBM was compared to the ordinary Gaussian-Bernoulli RBM by speech representation experiments. Both objective and subjective evaluations illustrated the advantage of the proposed model.

On the locality of the natural gradient for learning in deep Bayesian networks

We study the natural gradient method for learning in deep Bayesian networks, including neural networks. There are two natural geometries associated with such learning systems consisting of visible and hidden units. One geometry is related to the full system, the other one to the visible sub-system. These two geometries imply different natural gradients. In a first step, we demonstrate a great simplification of the natural gradient with respect to the first geometry, due to locality properties of the Fisher information matrix. This simplification does not directly translate to a corresponding simplification with respect to the second geometry. We develop the theory for studying the relation between the two versions of the natural gradient and outline a method for the simplification of the natural gradient with respect to the second geometry based on the first one. This method suggests to incorporate a recognition model as an auxiliary model for the efficient application of the natural gradient method in deep networks.

Unit-Vise: Deep Shallow Unit-Vise Residual Neural Networks With Transition Layer For Expert Level Skin Cancer Classification.

Many modern neural network architectures with over parameterized regime have been used for identification of skin cancer. Recent work showed that network, where the hidden units are polynomially smaller in size, showed better performance than overparameterized models. Hence, in this paper, we present multistage unit-vise deep dense residual network with transition and additional supervision blocks that enforces the shorter connections resulting in better feature representation. Unlike ResNet, We divided the network into several stages, and each stage consists of several dense connected residual units that support residual learning with dense connectivity and limited the skip connectivity. Thus, each stage can consider the features from its earlier layers locally as well as less complicated in comparison to its counter network. Evaluation results on ISIC-2018 challenge consisting of 10,015 training images show considerable improvement over other approaches achieving 98.05 percent accuracy and improving on the best results achieved in comparison to state of the art methods. The code of Unit-vise network is publicly available.1.

Neural Cryptography Based on Complex-Valued Neural Network.

Neural cryptography is a public key exchange algorithm based on the principle of neural network synchronization. By using the learning algorithm of a neural network, the two neural networks update their own weight through exchanging output from each other. Once the synchronization is completed, the weights of the two neural networks are the same. The weights of the neural network can be used for the secret key. However, all the existing works are based on the real-valued neural network model. There are seldom works studying the neural cryptography based on a complex-valued neural network model. In this technical note, a neural cryptography based on the complex-valued tree parity machine network (CVTPM) is proposed. The input, output, and weights of CVTPM are a complex value, which can be considered as an extension of TPM. There are two advantages of the CVTPM: 1) the security of CVTPM is higher than that of TPM with the same hidden units, input neurons, and synaptic depths and 2) the two parties with the CVTPM can exchange two group keys in one neural synchronization process. A series of numerical simulation experiments is provided to verify our results.

Regularized Deep LSTM Autoencoder for Phonological Deviation Assessment

In this paper, the Deep Long-short term memory Autoencoder (DLAE), a regularized deep learning model, is proposed for the automatic severity assessment of phonological deviations which are crucial stuttering markers in children. This automatic noninvasive severity assessment plays a paramount role in prevenient diagnosis, progress inference, and post-care for the patients with specific speech disorder. The proposed model is an implementation of a multi-layered Autoencoder in the Encoder–Decoder architecture of the Long-Short Term Memory (LSTM) model with hierarchically appended hidden layers and hidden units. The DLAE has definite advantage over the baseline Autoencoders. During the training phase, the proposed DLAE reconstructs the phonological features in an unsupervised fashion and the latent bottleneck features are extracted from the Encoder. The trained and regularized DLAE model with drop out is then used to predict the severity of the phonological deviation with high precision and classification accuracy compared to the baseline models.

An EM Algorithm for Capsule Regression.

We investigate a latent variable model for multinomial classification inspired by recent capsule architectures for visual object recognition (Sabour, Frosst, & Hinton, 2017). Capsule architectures use vectors of hidden unit activities to encode the pose of visual objects in an image, and they use the lengths of these vectors to encode the probabilities that objects are present. Probabilities from different capsules can also be propagated through deep multilayer networks to model the part-whole relationships of more complex objects. Notwithstanding the promise of these networks, there still remains much to understand about capsules as primitive computing elements in their own right. In this letter, we study the problem of capsule regression-a higher-dimensional analog of logistic, probit, and softmax regression in which class probabilities are derived from vectors of competing magnitude. To start, we propose a simple capsule architecture for multinomial classification: the architecture has one capsule per class, and each capsule uses a weight matrix to compute the vector of hidden unit activities for patterns it seeks to recognize. Next, we show how to model these hidden unit activities as latent variables, and we use a squashing nonlinearity to convert their magnitudes as vectors into normalized probabilities for multinomial classification. When different capsules compete to recognize the same pattern, the squashing nonlinearity induces nongaussian terms in the posterior distribution over their latent variables. Nevertheless, we show that exact inference remains tractable and use an expectation-maximization procedure to derive least-squares updates for each capsule's weight matrix. We also present experimental results to demonstrate how these ideas work in practice.

Latent-space embedding of expression data identifies gene signatures from sputum samples of asthmatic patients

BackgroundThe pathogenesis of asthma is a complex process involving multiple genes and pathways. Identifying biomarkers from asthma datasets, especially those that include heterogeneous subpopulations, is challenging. Potentially, autoencoders provide ideal frameworks for such tasks as they can embed complex, noisy high-dimensional gene expression data into a low-dimensional latent space in an unsupervised fashion, enabling us to extract distinguishing features from expression data.ResultsHere, we developed a framework combining a denoising autoencoder and a supervised learning classifier to identify gene signatures related to asthma severity. Using the trained autoencoder with 50 hidden units, we found that hierarchical clustering on the low-dimensional embedding corresponds well with previously defined and clinically relevant clusters of patients. Moreover, each hidden unit has contributions from each of the genes, and pathway analysis of these contributions shows that the hidden units are significantly enriched in known asthma-related pathways. We then used genes that contribute most to the hidden units to develop a secondary random-forest classifier for directly predicting asthma severity. The feature importance metric from this classifier identified a signature based on 50 key genes, which are associated with severity. Furthermore, we can use these key genes to successfully estimate FEV1/FVC ratios across patients, via support-vector-machine regression.ConclusionWe found that the denoising autoencoder framework can extract meaningful patterns corresponding to functional gene groups and patient clusters from the gene expression of asthma patients.

Thermophysical Features of the Rümker Region in Northern Oceanus Procellarum: Insights from CE-2 CELMS Data

The Rümker region is located in the northern Oceanus Procellarum, which has been selected as the landing and sampling region for China’s Chang’e-5 (CE-5) mission. The thermophysical features of the mare units are studied in detail using the brightness temperature (TB) maps (TB, normalized TB, TB difference) derived from the CE-2 microwave radiometer data. The previously interpreted geological boundaries of the Rümker region are revisited in this study according to their TB behaviors: IR1, IR2, and IR3 Rümker plateau units are combined into one single unit (IR); and a hidden unit is found on the Mons Rümker; Mare basaltic units Im1 and Em1 are combined into Em1; and Em2 is more likely the extending of Im2. Each of the previous proposed landing sites and their scientific value are summarized and reevaluated. Based on this, four landing sites are recommended in order to maximize the scientific outcome of the CE-5 mission. We suggest that the Eratosthenian-aged Em4 and Em1 units as the top priority landing site for the CE-5 mission; the age-dating results will provide important clues concerning the thermal evolution of the Moon.

Factors controlling long-term carbon dioxide exchange between a Douglas-fir stand and the atmosphere identified using an artificial neural network approach

It is critical to have long-term carbon dioxide (CO2) flux observations in forest ecosystems to understand how changing climate can affect forest carbon (C) stocks and CO2 exchange between forests and the atmosphere. In this study, fifteen years (2002–2016) of continuous eddy-covariance flux and climate measurements in an intermediate-aged Douglas-fir stand on the east coast of Vancouver Island, Canada, were analyzed. First, the eddy covariance-measured CO2 fluxes were partitioned into gross primary production and ecosystem respiration using two artificial neural networks. Second, the responses of net ecosystem production, gross primary production and ecosystem respiration to interannual climate variability, including five El Niño-Southern Oscillation events, were determined. Three hyper-parameters (number of layers, hidden units, and batch size) of each artificial neural network were set by Bayesian optimization using sequential model-based optimization while the remaining hyper-parameters were taken from the literature. The first artificial neural network was fitted using only nighttime CO2 flux data and applied to estimate nighttime and daytime ecosystem respiration values, and the second one was used to gap-fill gross primary production values. In addition, a predictor analysis was done to investigate the most influential predictors (i.e., environmental variables) within seasons and years. When applied to half-hourly data, the ecosystem respiration model had an R2 of 0.43, whereas the gross primary production model had an R2 of 0.80. The stand was a moderate C sink (average net ecosystem production of 118 ± 404 g C m−2 year−1) during the entire study period, except for the years 2002–2006 when the stand was a moderate C source. The mean annual values of gross primary production and ecosystem respiration were 1649 ± 157 g C m−2 year−1 and 1531 ± 410 g C m−2 year−1, respectively. Our analysis showed that soil temperature was the most important predictor for the ecosystem respiration model, and photosynthetically active irradiance was the most important predictor for the gross primary production model. However, during dry periods in late summer, soil moisture became the most important predictor. Interannual variability of net ecosystem production was only slightly affected by annual total precipitation, mean soil temperature and mean air temperature. Instead, it depended on spring mean air temperature (start of the growing season), summer total precipitation (indicative of water deficiency) and mean summer air temperature. El Niño and La Niña events generally resulted in lower and higher annual net ecosystem production, respectively.

Extractive text summarization using F-RBM

The Intense growth of data over the internet and process extracting predominating details from it is a time-consuming task. A new application of making this task simplified is text-summarization. This paper presents the integration of Restricted Boltzmann Machine and Fuzzy Logic methods. Both of these processes have different ways of giving precision to a summary but together being part of the unsupervised world has surpassed their working alone to summarize the text. Integrated algorithm i.e. called FRBM (fuzzy Restricted Boltzmann Machine) provides enhanced depiction in the modeling of probability over visible and hidden units than the RBM.

PSO–LSTM for short term forecast of heterogeneous time series electricity price signals

Electricity price forecasting plays an important role in the power system network, in order to promote the decision-making process for power generation and consumption. Long term forecasting is not viable as there is an uncertainty in the forecast due to increasing the integration of renewable sources with the existing grids. Since the behavior of the electricity price time sequence signal is highly non-linear and seasonal, deep neural network is the best model for learning the non-linear behavior within the data and for the purpose of forecasting. Hence this paper proposes an enhanced particle swarm optimization based long short-term memory (LSTM) neural network model, which is used to forecast the closing price of the Indian Energy Exchange. Particle swarm optimization technique is used to optimize the LSTM network input weights, which in turn minimize the forecast error with reduced architecture. This paper discusses the statistical analysis for input data selection and investigates the performance analysis for the optimal selection of layers with hidden units’ combination. Finally, the analysis deploys the best-suited configuration for forecasting the market clearing price with the least mean absolute percentage error.

Matrix variate deep belief networks with CP decomposition algorithm and its application

Deep belief networks (DBNs) are used in many applications such as image processing and pattern recognition. But the data vectorized resulting in the loss of high-dimensional data and valuable spatial information. The classical DBNs model is based on restricted Boltzmann machines (RBMs) and full connectivity between the visible units and the hidden units. It requires a large number of parameters to be trained using an army of training samples. However, it is difficult to obtain so many training samples in reality. To solve this problem, this paper proposes a matrix-variate deep belief networks (MVDBNs) model which is created from matrix variate RBMs whose parameter is restricted as canonical polyadic (CP) decomposition. MVDBNs are composed of two or more matrix-variate restricted Boltzmann machines (MVRBMs) whose input and latent variables are in matrix form. The MVDBNs have much fewer model parameters and deeper layer with better features to avoid overfitting and more accurate model easier to be learned. We demonstrate the capacity of MVDBNs on handwritten digit classification and medical image processing. We also extend MVRBMs to a multimodal case for image super-resolution.

EMD and LSTM Hybrid Deep Learning Model for Predicting Sunspot Number Time Series with a Cyclic Pattern

The prediction of a time series such as climate indices and the sunspot number (SSN) with long-term oscillatory behaviors has been a challenging task due to the complex combination of oscillations. Frequency extraction algorithms have been developed to separate a time series into different oscillation components according to frequency, such as empirical model decomposition (EMD) and wavelet analysis. In the current study, the deep learning long short-term memory (LSTM) model was employed to predict the oscillation components extracted using EMD. The SSN series was modeled with the hybrid EMD-LSTM model. The simulation study results indicate that the LSTM model reproduces the smooth cyclic pattern of the sine function, and only a few hidden units are needed to model it. The EMD-LSTM model achieves better performance than does the LSTM model for mid-range SSN predictions while the LSTM achieves better performance within the first few time lags. However, the cyclic prediction of the SSN requires mid-range lags; thus, the superior performance of the EMD-LSTM model for these lags cannot be ignored. Furthermore, the remaining components from the significant EMD signals can be modeled to reveal the variability (or uncertainty) in the prediction. The summed residual is fitted by k-nearest neighbor resampling. The final SSN prediction results show that the EMD-LSTM model predicts a later and larger SSN for Solar Cycle 25 than does the LSTM model. Overall, the results lead to the conclusion that the EMD-LSTM model might be a suitable alternative for modeling complex sunspot time series with cyclic patterns.