Number Of Hidden Units Research Articles

A topological description of loss surfaces based on Betti Numbers

In the context of deep learning models, attention has recently been paid to studying the surface of the loss function in order to better understand training with methods based on gradient descent. This search for an appropriate description, both analytical and topological, has led to numerous efforts in identifying spurious minima and characterize gradient dynamics. Our work aims to contribute to this field by providing a topological measure for evaluating loss complexity in the case of multilayer neural networks. We compare deep and shallow architectures with common sigmoidal activation functions by deriving upper and lower bounds for the complexity of their respective loss functions and revealing how that complexity is influenced by the number of hidden units, training models, and the activation function used. Additionally, we found that certain variations in the loss function or model architecture, such as adding an ℓ2 regularization term or implementing skip connections in a feedforward network, do not affect loss topology in specific cases.

Optimisation and Calibration of Bayesian Neural Network for Probabilistic Prediction of Biogas Performance in an Anaerobic Lagoon.

This study aims to enhance diagnostic capabilities for optimising the performance of the anaerobic sewage treatment lagoon at Melbourne Water's Western Treatment Plant (WTP) through a novel machine learning (ML)-based monitoring strategy. This strategy employs ML to make accurate probabilistic predictions of biogas performance by leveraging diverse real-life operational and inspection sensor and other measurement data for asset management, decision making, and structural health monitoring (SHM). The paper commences with data analysis and preprocessing of complex irregular datasets to facilitate efficient learning in an artificial neural network. Subsequently, a Bayesian mixture density neural network model incorporating an attention-based mechanism in bidirectional long short-term memory (BiLSTM) was developed. This probabilistic approach uses a distribution output layer based on the Gaussian mixture model and Monte Carlo (MC) dropout technique in estimating data and model uncertainties, respectively. Furthermore, systematic hyperparameter optimisation revealed that the optimised model achieved a negative log-likelihood (NLL) of 0.074, significantly outperforming other configurations. It achieved an accuracy approximately 9 times greater than the average model performance (NLL = 0.753) and 22 times greater than the worst performing model (NLL = 1.677). Key factors influencing the model's accuracy, such as the input window size and the number of hidden units in the BiLSTM layer, were identified, while the number of neurons in the fully connected layer was found to have no significant impact on accuracy. Moreover, model calibration using the expected calibration error was performed to correct the model's predictive uncertainty. The findings suggest that the inherent data significantly contribute to the overall uncertainty of the model, highlighting the need for more high-quality data to enhance learning. This study lays the groundwork for applying ML in transforming high-value assets into intelligent structures and has broader implications for ML in asset management, SHM applications, and renewable energy sectors.

An efficient tool for Parkinson's disease detection and severity grading based on time-frequency and fuzzy features of cumulative gait signals through improved LSTM networks

Parkinson's disease (PD) is a widespread neurodegenerative condition that affects many individuals annually. Early identification and monitoring of disease progression are crucial to effectively managing symptoms and preventing motor complications. This research proposes an automated PD diagnosis and severity-grading model based on time-frequency and fuzzy features using improved uni-directional and bi-directional long short-term memory networks with sensitive hyperparameters optimization. We utilize vertical ground reaction force signals collected from Physionet's publicly available dataset recorded during regular and dual-task clinical trials of walking measurements. Only the cumulative signal of both feet was then utilized and segmented into 30-s windows without further pre-processing. Subsequently, we extracted only four key time-frequency and fuzzy features from each segment, effectively capturing the signal's inherent uncertainty. Bayesian optimization is employed in both detection and grading approaches to fine-tune the two critical hyperparameters: the initial learning rate and the number of hidden units in the network. The detection phase yields an exceptional accuracy of 99.19%, surpassing state-of-the-art studies with the same dataset. In the grading phase, classification based on the unified PD rating scale values achieves an accuracy of 92.28%. The proposed study delves into the potential of cumulative gait signals as a powerful diagnostic tool for PD, aiming to extract precise and intricate information by implementing straightforward and minimal processing endeavors. This method demonstrates significant efficiency in terms of complexity, cost, and energy consumption by utilizing a single-dimensional signal, eliminating the need for pre-processing steps, and limiting the features used for training.

Phase diagram of stochastic gradient descent in high-dimensional two-layer neural networks *

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow networks, which tend to get stuck in badly-generalizing local minima. Here we investigate the cross-over between these two regimes in the high-dimensional setting, and in particular investigate the connection between the so-called mean-field/hydrodynamic regime and the seminal approach of Saad & Solla. Focusing on the case of Gaussian data, we study the interplay between the learning rate, the time scale, and the number of hidden units in the high-dimensional dynamics of stochastic gradient descent (SGD). Our work builds on a deterministic description of SGD in high-dimensions from statistical physics, which we extend and for which we provide rigorous convergence rates.

Comparing feedforward neural networks using independent component analysis on hidden units.

Neural networks are widely used for classification and regression tasks, but they do not always perform well, nor explicitly inform us of the rationale for their predictions. In this study we propose a novel method of comparing a pair of different feedforward neural networks, which draws on independent components obtained by independent component analysis (ICA) on the hidden layers of these networks. It can compare different feedforward neural networks even when they have different structures, as well as feedforward neural networks that learned partially different datasets, yielding insights into their functionality or performance. We evaluate the proposed method by conducting three experiments with feedforward neural networks that have one hidden layer, and verify whether a pair of feedforward neural networks can be compared by the proposed method when the numbers of hidden units in the layer are different, when the datasets are partially different, and when activation functions are different. The results show that similar independent components are extracted from two feedforward neural networks, even when the three circumstances above are different. Our experiments also reveal that mere comparison of weights or activations does not lead to identifying similar relationships. Through the extraction of independent components, the proposed method can assess whether the internal processing of one neural network resembles that of another. This approach has the potential to help understand the performance of neural networks.

Asymptotic properties of neural network sieve estimators

Neural networks have become one of the most popularly used methods in machine learning and artificial intelligence. Due to the universal approximation theorem, a neural network with one hidden layer can approximate any continuous function on compact support as long as the number of hidden units is sufficiently large. Statistically, a neural network can be classified into a nonlinear regression framework. However, if we consider it parametrically, due to the unidentifiability of the parameters, it is difficult to derive its asymptotic properties. Instead, we consider the estimation problem in a nonparametric regression framework and use the results from sieve estimation to establish the consistency, the rates of convergence and the asymptotic normality of the neural network estimators. We also illustrate the validity of the theories via simulations.

Construction of personalized driver model for car-following behavior on highways using LSTM

In recent years, automatic driving and advanced driving support systems have become more widespread; however, control systems do not consider the driving characteristics of each driver. This study constructed a personalized driver model for following forward vehicles on highways using LSTM (Long Short-Term Memory). In particular, the setting parameters for training with LSTM was focused upon and consequently, the relationship between the setting parameters and model accuracy was evaluated. The LSTM parameters considered were the number of hidden units, learning rate, number of epochs, and number of LSTM layers. Further, the driving data were measured using a driving simulator. Moreover, the model was evaluated in terms of accuracy using the coefficient of determination, and 125 combinations of parameters were compared using the mean value of the coefficient of determination and its significant difference. The order of parameter comparisons was performed considering the accuracy and learning time of the model. Consequently, the parameter set that is likely to allow efficient building of a highly accurate driver model was determined. Applying the parameter evaluation in this study contributes to the development of personalized driver models with high accuracy.

Weighted Score Fusion Based LSTM Model for High-Speed Railway Propagation Scenario Identification

Propagation scenario identification is of vital significance for boosting the performance of future smart high-speed railway (HSR) communication networks. This paper investigates the HSR propagation scenario identification model, based on deep learning networks and feature fusion methods. With the assistance of railway long-term evolution (LTE) networks, we collected the channel impulse responses in four typical HSR scenarios including unobstructed viaduct, obstructed viaduct, station and suburban. Four channel characteristics involving power delay profile, root mean square (RMS) delay spread, RMS angular spread and Ricean K-factor form the datasets used for model training and testing. Then, a novel propagation scenario identification model is proposed by merging a weighted score based feature fusion method into the long short-term memory (LSTM) neural network. The hyper-parameters of the proposed model such as time window length and numbers of hidden units and layers are determined by autocorrelation analysis and cross-validation. Finally, the model performance is evaluated by focusing on the impact of feature selection, comparison of different feature fusion methods, and computational complexity. The evaluation results show that the proposed model has high identification accuracy but acceptable computational complexity.

Smooth Spatial Modeling of Extreme Mediterranean Precipitation

Extreme precipitation events can lead to disastrous floods, which are the most significant natural hazards in the Mediterranean regions. Therefore, a proper characterization of these events is crucial. Extreme events defined as annual maxima can be modeled with the generalized extreme value (GEV) distribution. Owing to spatial heterogeneity, the distribution of extremes is non-stationary in space. To take non-stationarity into account, the parameters of the GEV distribution can be viewed as functions of covariates that convey spatial information. Such functions may be implemented as a generalized linear model (GLM) or with a more flexible non-parametric non-linear model such as an artificial neural network (ANN). In this work, we evaluate several statistical models that combine the GEV distribution with a GLM or with an ANN for a spatial interpolation of the GEV parameters. Key issues are the proper selection of the complexity level of the ANN (i.e., the number of hidden units) and the proper selection of spatial covariates. Three sites are included in our study: a region in the French Mediterranean, the Cap Bon area in northeast Tunisia, and the Merguellil catchment in central Tunisia. The comparative analysis aim at assessing the genericity of state-of-the-art approaches to interpolate the distribution of extreme precipitation events.

Lightning Whistler Wave Speech Recognition Based on Grey Wolf Optimization Algorithm

The recognition algorithm of the lightning whistler wave, based on intelligent speech, is the key technology to break the bottleneck of massive data and study the temporal and spatial variation rules of the lightning whistler wave. However, its recognition effect depends on the hyperparameters determined by manual experiments repeatedly, which takes a great deal of time and cannot guarantee the best recognition effect of the model. Therefore, we proposed the lightning whistler wave recognition algorithm based on grey wolf optimization (GWO). In this paper, the GWO algorithm is used to automatically find the best value of hyperparameters of Long Short-Term Memory (LSTM) in their limited searching space. Here we consider the number of hidden units (hu) and learning rate (lr) as the hyperparameters to be optimized, and the spatial coordinate (hu, lr) as the grey wolf position. By the end of the GWO process, we obtain the position of the wolf king α with the optimal hu and lr searched by the GWO algorithm. Then we use the optimal hu and lr to configure LSTM and perform supervised learning on the train set to obtain the final lightning whistler wave speech recognition model. Through experimental verification, the recognition model based on the GWO not only overcomes the uncertainty of the traditional model relying on manual finetuning of parameters and realizes the mechanism of automatic search and acquisition of hyperparameters, but also its recognition effect improves by about 2% in accuracy, F1score, and other metrics compared with the model trained by manually setting hyperparameters.

Predicting Analyte Concentrations from Electrochemical Aptasensor Signals Using LSTM Recurrent Networks.

Nanomaterial-based aptasensors are useful devices capable of detecting small biological species. Determining suitable signal processing methods can improve the identification and quantification of target analytes detected by the biosensor and consequently improve the biosensor’s performance. In this work, we propose a data augmentation method to overcome the insufficient amount of available original data and long short-term memory (LSTM) to automatically predict the analyte concentration from part of a signal registered by three electrochemical aptasensors, with differences in bioreceptors, analytes, and the signals’ lengths for specific concentrations. To find the optimal network, we altered the following variables: the LSTM layer structure (unidirectional LSTM (LSTM) and bidirectional LSTM (BLSTM)), optimizers (Adam, RMSPROP, SGDM), number of hidden units, and amount of augmented data. Then, the evaluation of the networks revealed that the highest original data accuracy increased from 50% to 92% by exploiting the data augmentation method. In addition, the SGDM optimizer showed a lower performance prediction than that of the ADAM and RMSPROP algorithms, and the number of hidden units was ineffective in improving the networks’ performances. Moreover, the BLSTM nets showed more accurate predictions than those of the ULSTM nets on lengthier signals. These results demonstrate that this method can automatically detect the analyte concentration from the sensor signals.

Dissolved Gas Analysis of Insulating Oil for Power Transformer Fault Diagnosis with Bayesian Neural Network

Dissolved gas analysis is widely used for preventative maintenance techniques and fault diagnoses of oil-immersed power transformers. There are also various conventional methods of dissolved gas analysis for insulating oil in power transformers including methods of Doernenburg ratios, Rogers ratios and Duval’s triangle. The Bayesian techniques have been developed over many years and applied to a range of different fields including the problem of training in artificial neural networks. In particular, the Bayesian approach can solve the problem of over-fitting of artificial neural networks after being trained. The Bayesian framework can be also utilised to compare and rank different architectures and types of artificial neural networks. This research aims at deploying a detailed procedure of training artificial neural networks with the Bayesian inference, also known as Bayesian neural networks, to classify power transformer faults based on Doernenburg and Rogers gas ratios. In this research, the IEC TC 10 databases were used to form training and test data sets. The results obtained from the performance of trained Bayesian neural networks show that despite the limitation of the available dissolved gas analysis data, Bayesian neural networks with an appropriate number of hidden units can successfully classify power transformer faults with accuracy rates greater than 80%.

Improved genetic algorithm optimized LSTM model and its application in short-term traffic flow prediction.

Considering that the road short-term traffic flow has strong time series correlation characteristics, a new long-term and short-term memory neural network (LSTM)-based prediction model optimized by the improved genetic algorithm (IGA) is proposed to improve the prediction accuracy of road traffic flow. Firstly, an improved genetic algorithm (IGA) is proposed by dynamically adjusting the mutation rate and crossover rate of standard GA. Secondly, the parameters of the LSTM, such as the number of hidden units, training times, gradient threshold and learning rate, are optimized by the IGA. Therefore, the optimal parameters are obtained. In the analysis stage, 5-min short-term traffic flow data are used to demonstrate the superiority of the proposed method over the existing neural network algorithms. Finally, the results show that the Root Mean Square Error achieved by the proposed algorithm is lower than that achieved by the other neural network methods in both the weekday and weekend data sets. This verifies that the algorithm can adapt well to different kinds of data and achieve higher prediction accuracy.

Improved Human Activity Recognition Using Majority Combining of Reduced-Complexity Sensor Branch Classifiers

Human activity recognition (HAR) employs machine learning for the automated recognition of motion and has widespread applications across healthcare, daily-life and security spaces. High performances have especially been demonstrated using video cameras and intensive signal processing such as the convolutional neural network (CNN). However, lower complexity algorithms operating on low-rate inertial data is a promising approach for portable use-cases such as pairing with smart wearables. This work considers the performance benefits from combining HAR classification estimates from multiple sensors each with lower-complexity processing compared with a higher-complexity single-sensor classifier. We show that while the highest single-sensor classification accuracy of 91% can be achieved for seven activities with optimized number of hidden units and sample rate, the classification accuracy is reduced to 56% with a reduced-complexity 50-neuron classifier. However, by majority combining the predictions of three and four low-complexity classifiers, the average classification accuracy increased to 82.5% and 94.4%, respectively, demonstrating the efficacy of this approach.

Automatic electrocardiogram detection and classification using bidirectional long short-term memory network improved by Bayesian optimization

Electrocardiogram (ECG) signals contain a significant amount of subtle information that can be used to detect some types of heart dysfunction. The widespread availability of digital ECG and the algorithmic paradigm of the long short-term memory (LSTM) network present an opportunity to substantially improve the accuracy and scalability of automated ECG analysis. However, the number of hidden units and initial learning rate of an LSTM neural network for ECG classification are currently preset based on prior knowledge, which causes the model to reach a sub-optimal state. In this study, an automated ECG detection and classification method using a bidirectional LSTM (BiLSTM) network modified by Bayesian optimization is developed. Bayesian optimization is used to optimize the two hyperparameters of the BiLSTM network: the initial learning rate and the number of hidden layers. By classifying five ECG signals in the MIT-BIH arrhythmia database, the accuracy of the modified network reaches 99.00%, which is 0.86% higher than that before optimization. The results demonstrate that Bayesian optimization can be an effective approach to improving the quality of classifiers based on deep learning. The presented approach can also be considered for generalization to other quasi-periodical biometric signal-based classification tasks in future studies, which may have practical applications.

Developing a novel framework for forecasting groundwater level fluctuations using Bi-directional Long Short-Term Memory (BiLSTM) deep neural network

In this study, mean monthly groundwater level (GWL) fluctuations of four different monitoring piezometers are forecasted in an agricultural area by developing new proposed structures of BiLSTM based neural network models. For developing models, 186 monthly measured water table depth during time period (Sep 2001–Feb 2017) are employed. Due to a lack of meteorological variables records, the module of sequence-to-one is employed. By tuning several hyperparameters such as the number of hidden units (NHU), kind of state activation function (SAF), learning dropout rate (P-rate), and network topology pattern (NTP), the performance of designed models is improved. Based on results of modelling, in all designed models, the optimal P-rate is obtained 0.5, and the running time decrease by increasing P-rate in the same model. Since harsh fluctuations of GWL, the general Single-LSTM mode (1) and Single-BiLSTM model (2) perform too weak compared to other models. Nonetheless, by deepening models through adding the suitable hidden layers instead of many numbers of nodes, the performance of Single-BiLSTM model is improved strikingly. Based on statistical evaluation metrics, violin plot, and also TLP (Total Learnable Parameters) value, novel proposed model (5) (simple Double-BiLSTM model combined by a Multiplication layer (×)) is selected as the superior model. In the piezometer 4 with a range of 4.49 (m), the model (5) yielded in an R2 of 0.89 and an RMSE of 0.17 (m), while in the same physical characteristics, the simple Double-BiLSTM model (3) yielded in an R2 of 0.77 and an RMSE of 0.25 (m).

WaDGAN-AD를 이용한 전력 소비 패턴의 비지도 학습 기반 이상 탐지

Anomaly detection in time series is essential because it can detect outlying patterns such as a breakdown in machines and fraudulent customers. Among many anomaly detection domains, detecting abnormal patterns in energy consumption is used to detect technical breakdown in factories, general buildings, or energy theft in households. To overcome the limitations of previous studies, this paper suggests WaDGAN-AD, which combines generative adversarial network (GAN) and Long Short-Term Memory (LSTM) and applies two structural improvements. WaDGAN-AD has stacked discriminator LSTM layers to more precisely learn feature representations of time series data. Also, it has different numbers of hidden units in each hidden layer of LSTM to consider multiple cycles appearing in a single time-series data. Experimental results based on synthetic datasets and real datasets show that WaDGAN-AD can better detect abnormal energy consumption than benchmark methods.

Lung cancer medical images classification using hybrid CNN-SVM

Lung cancer is one of the leading causes of death worldwide. Early detection of this disease increases the chances of survival. Computer-Aided Detection (CAD) has been used to process CT images of the lung to determine whether an image has traces of cancer. This paper presents an image classification method based on the hybrid Convolutional Neural Network (CNN) algorithm and Support Vector Machine (SVM). This algorithm is capable of automatically classifying and analyzing each lung image to check if there is any presence of cancer cells or not. CNN is easier to train and has fewer parameters compared to a fully connected network with the same number of hidden units. Moreover, SVM has been utilized to eliminate useless information that affects accuracy negatively. In recent years, Convolutional Neural Networks (CNNs) have achieved excellent performance in many computer visions tasks. In this study, the performance of this algorithm is evaluated, and the results indicated that our proposed CNN-SVM algorithm has been succeed in classifying lung images with 97.91% accuracy. This has shown the method's merit and its ability to classify lung cancer in CT images accurately.

Deep Belief Network with FOA based Cooperative Spectrum Sensing in Cognitive Radio Network

The artificial neural network (ANN) is proposed to the cooperative spectrum sensing (CSS) at CR network. ANN has the drawback that the training of ANN with many hidden layers on large amount of data can affect the performance of the network and optimisation of ANN parameters is a challenging task. To overcome the above drawbacks, the deep belief network (DBN) and fruit fly optimisation algorithm (FOA) are employed. The DBN has four parameters on learning step: learning rate, weight decay, penalty parameter, number of hidden units. These parameters should be properly selected for the proper functioning of DBN. Tuning of these parameters is taken into an optimisation issue and it is addressed by FOA. The proposed method has three steps: training, validation and testing. The metrics used for the performance evaluation are accuracy, false alarm rate and loss detection.

Non-Factoid Answer Selection in Indonesian Science Question Answering System using Long Short-Term Memory (LSTM)

Understanding natural science has become vital since it has effects on various facets of daily necessity. However, the lack of comprehensive information sources has led to difficulties in finding information quickly and efficiently, emphasising the need to develop a question answering system. The type of question utilised for question answering systems that specifically deal with science topics is referred to as a non-factoid question. In this research, the question answering system utilised Long Short-Term Memory (LSTM) model for answer selection problems. Testing was done for the following LSTM hyper-parameters: dropouts, learning rates, number of hidden units, size of the answer pool, and margins. Data utilised in this study consisted of 400 pairs of questions and answers on science topics that were obtained from Wikipedia. The highest average values were 90.06% for Mean Reciprocal Rank (MRR) and 78.69% for Mean Average Precision (MAP), which were achieved when using a dropout value of 0.2, 50 hidden units, learning rate value of 0.05, a margin of 0.1, and an answer pool size of 20.