Large Number Of Hyperparameters Research Articles

The evaluation of convolutional neural network and genetic algorithm performance based on the number of hyperparameters for english handwritten recognition

<span>Convolutional neural network (CNN) has been widely applied to image recognition, especially Handwritten English Recognition. CNN's performance is good if the hyperparameter values are correct. However, the determination of precise hyperparameters is not a trivial task. This task is made more difficult when combined with a larger number of hyperparameters resulting in a high dimensionality of the search space. Usually, hyperparameter optimization uses a finite number. Previous studies have shown that a large number of hyperparameters can result in optimal CNN performance. However, the studies only apply to text mining datasets. This study offers two novelties. First, it applied 20 hyperparameters and their ranges to handwritten English. Second, this paper conducted seven experiments based on different hyperparameters and the number of hyperparameters. This paper also compares the existing methods, namely random and grid search. The experiment resulted in the proposed model being superior to the existing methods. EX3 is better than other experiments and a larger number of hyperparameters and layer-specific hyperparameter values are unimportant.</span>

A novel approach to minimal reservoir computing

Reservoir computers are powerful machine learning algorithms for predicting nonlinear systems. Unlike traditional feedforward neural networks, they work on small training data sets, operate with linear optimization, and therefore require minimal computational resources. However, the traditional reservoir computer uses random matrices to define the underlying recurrent neural network and has a large number of hyperparameters that need to be optimized. Recent approaches show that randomness can be taken out by running regressions on a large library of linear and nonlinear combinations constructed from the input data and their time lags and polynomials thereof. However, for high-dimensional and nonlinear data, the number of these combinations explodes. Here, we show that a few simple changes to the traditional reservoir computer architecture further minimizing computational resources lead to significant and robust improvements in short- and long-term predictive performances compared to similar models while requiring minimal sizes of training data sets.

MACLA-LSTM: A Novel Approach for Forecasting Water Demand

Sustainable and effective management of urban water supply is a key challenge for the well-being and security of current society. Urban water supply systems have to deal with a huge amount of data, and it is difficult to develop efficient intervention mechanisms by relying on the human experience. Deep learning methods make it possible to predict water demand in real-time; however, deep learning methods have a large number of hyperparameters, and the selection of hyperparameters can easily affect the accuracy of prediction. Within this context, a novel framework of short-term water demand forecast is proposed, in which a forecasting method clouded leopard algorithm based on multiple adaptive mechanisms—long short-term memory networks (MACLA-LSTM)—is developed to improve the accuracy of water demand predictions. Specifically, LSTM networks are used to predict water demand and the MACLA is utilized to optimize the input parameters of the LSTM. The MACLA-LSTM model is evaluated on a real dataset sampled from water distribution systems. In comparison with other methods, the MACLA-LSTM achieved MAE values of 1.12, 0.89, and 1.09; MSE values of 2.22, 1.21, and 2.38; and R2 values of 99.51%, 99.44%, and 99.01%. The results show the potential of the MACLA-LSTM model for water demand forecasting tasks and also demonstrate the positive effect of the MACLA on forecasting tasks by comparing results with LSTM variant models. The proposed MACLA-LSTM can provide a resilient, sustainable, and low-cost management strategy for water supply systems.

A Deep Learning Approach to Detect Microsleep Using Various Forms of EEG Signal

Electroencephalography (EEG) is a reliable method for identifying the onset of sleepiness behind the wheel. Using EEG technology for driving fatigue detection still presents challenges in extracting informative elements from noisy EEG signals. Due to their extensive computational parallelism, which is similar to how the brain processes information, neural networks have been explored as potential solutions for extracting relevant information from EEG data. The existing machine learning frameworks suffer from high computing costs and slow convergence, both of which contribute to low classification accuracy and efficiency due to the large number of hyper parameters that need to be improved. It is necessary to automate this micronap detection process before it can be used in real‐time scenarios. To distinguish between micronap and non‐micronap states, a deep neural network (DNN) framework is developed in this research using different EEG representations as input. Additional EEG representations utilized in this investigation include cleaned EEG as a time series, log‐power spectrum, 2D‐spatial map of log‐power spectrum, and raw EEG. Finally, traditional machine learning algorithms are evaluated for their effectiveness in detecting micronaps from these EEG inputs. The findings suggest that micronap detection can be greatly improved by combining cleaned EEG with DNN.

Evaluation of Class Distribution and Class Combinations on Semantic Segmentation of 3D Point Clouds With PointNet

Point clouds are generated by light imaging, detection and ranging (LIDAR) scanners or depth imaging cameras, which capture the geometry from the scanned objects with high accuracy. Unfortunately, these systems are unable to identify the semantics of the objects. Semantic 3D point clouds are an important basis for modeling the real world in digital applications. Manual semantic segmentation is a labor and cost intensive task. Automation of semantic segmentation using machine learning and deep learning (DL) approaches is therefore an interesting subject of research. In particular, point-based network architectures, such as PointNet, lead to a beneficial semantic segmentation in individual applications. For the application of DL methods, a large number of hyperparameters (HPs) have to be determined and these HPs influence the training success. In our work, the investigated HPs are the class distribution and the class combination. By means of seven combinations of classes following a hierarchical scheme and four methods to adapt the class sizes, these HPs are investigated in a detailed and structured manner. The investigated settings show an increased semantic segmentation performance, by an increase of 31% in recall for the class Erroneous points or that all classes have a recall of higher than 50%. However, based on our results the correct setting of only these HPs does not lead to a simple, universal and practical semantic segmentation procedure.

Lightweight Super-Resolution with Self-Calibrated Convolution for Panoramic Videos

Panoramic videos are shot by an omnidirectional camera or a collection of cameras, and can display a view in every direction. They can provide viewers with an immersive feeling. The study of super-resolution of panoramic videos has attracted much attention, and many methods have been proposed, especially deep learning-based methods. However, due to complex architectures of all the methods, they always result in a large number of hyperparameters. To address this issue, we propose the first lightweight super-resolution method with self-calibrated convolution for panoramic videos. A new deformable convolution module is designed first, with self-calibration convolution, which can learn more accurate offset and enhance feature alignment. Moreover, we present a new residual dense block for feature reconstruction, which can significantly reduce the parameters while maintaining performance. The performance of the proposed method is compared to those of the state-of-the-art methods, and is verified on the MiG panoramic video dataset.

CURIOUS: Efficient Neural Architecture Search Based on a Performance Predictor and Evolutionary Search

Neural networks (NNs) have been successfully deployed in various applications of artificial intelligence. However, architectural design of NNs is still a challenging problem. This is due to the need to navigate a search space based on a large number of hyperparameters. This forces the search space of possible architectures to grow exponentially. Using a trial-and-error design approach is very time consuming and leads to suboptimal architectures. In addition, approaches, such as neural architecture search based on reinforcement learning and differentiable gradient-based architecture search, often incur huge computational costs or significant memory requirements. To address these challenges, we propose the CURIOUS NN synthesis methodology. It uses a performance predictor to efficiently navigate the architectural search space with an evolutionary search process. The predictor is built using quasi Monte-Carlo sampling, boosted decision tree regression, and an intelligent iterative sampling method. It is designed to be sample efficient. CURIOUS starts from a base architecture and explores the architectural search space to obtain a variant of the base architecture with the highest performance. This search framework is general and covers all important NN architecture types, e.g., feedforward NNs (FFNNs), convolutional NNs (CNNs), recurrent NNs (RNNs), and transformers. We evaluate the performance of CURIOUS on various datasets and base architectures. Through these experiments, we demonstrate significant performance improvements over the baseline architectures. For the MNIST dataset, our CNN architecture achieves an error rate of 0.66%, with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.6\times $ </tex-math></inline-formula> fewer parameters compared to the LeNet-5 baseline. For the CIFAR-10 dataset, we use the ResNet architectures and residual networks with Shake-Shake regularization as the baselines. Our synthesized ResNet-18 has a 2.52% accuracy improvement over the original ResNet-18, 1.74% over ResNet-101, and 0.16% over ResNet-1001, while requiring comparable number of parameters and floating-point operations to the original ResNet-18. This result shows that instead of just increasing the number of layers to increase accuracy, an alternative is to use a better NN architecture with a small number of layers. In addition, CURIOUS achieves an error rate of just 2.69% with a variant of the residual architecture with Shake-Shake regularization. We also use the set of optimized hyperparameters found for ResNet-18 on the CIFAR-10 dataset to train and evaluate the model on the ImageNet dataset, and show 3.43% (1.83%) improvement in the top-1 (top-5) error rate compared to the original ResNet-18 model. CURIOUS also obtains the highest accuracy for various other FFNNs that are geared toward edge devices and IoT sensors. In addition, we use CURIOUS to search for deep RNN architectures for the SICK dataset for sentence similarity evaluation. It achieves a mean-squared error of only 0.2060, improving upon the base network performance, without the need to stack multiple long short-term memories. We also use CURIOUS to search for a better NN classifier for the sentiment analysis task on the Stanford sentiment treebank dataset using a pretrained BERT model and again demonstrate improvements in performance.

Deep reinforcement learning for self-tuning laser source of dissipative solitons

Increasing complexity of modern laser systems, mostly originated from the nonlinear dynamics of radiation, makes control of their operation more and more challenging, calling for development of new approaches in laser engineering. Machine learning methods, providing proven tools for identification, control, and data analytics of various complex systems, have been recently applied to mode-locked fiber lasers with the special focus on three key areas: self-starting, system optimization and characterization. However, the development of the machine learning algorithms for a particular laser system, while being an interesting research problem, is a demanding task requiring arduous efforts and tuning a large number of hyper-parameters in the laboratory arrangements. It is not obvious that this learning can be smoothly transferred to systems that differ from the specific laser used for the algorithm development by design or by varying environmental parameters. Here we demonstrate that a deep reinforcement learning (DRL) approach, based on trials and errors and sequential decisions, can be successfully used for control of the generation of dissipative solitons in mode-locked fiber laser system. We have shown the capability of deep Q-learning algorithm to generalize knowledge about the laser system in order to find conditions for stable pulse generation. Region of stable generation was transformed by changing the pumping power of the laser cavity, while tunable spectral filter was used as a control tool. Deep Q-learning algorithm is suited to learn the trajectory of adjusting spectral filter parameters to stable pulsed regime relying on the state of output radiation. Our results confirm the potential of deep reinforcement learning algorithm to control a nonlinear laser system with a feed-back. We also demonstrate that fiber mode-locked laser systems generating data at high speed present a fruitful photonic test-beds for various machine learning concepts based on large datasets.

Transient Voltage Stability Assessment Method based on gcForest

Aiming at the fact that the traditional method cannot quickly and accurately judge the power system transient voltage stability, and the deep neural network method needs many sample data sets and has a large number of hyperparameters, a power system transient voltage stability assessment method based on deep forest (gcForest) is proposed. gcForest effectively extracts features through ensemble learning, multi-layer feature transformation, and dynamically adjusting the number of model layers. Finally, this method is applied to transient voltage stability assessment of an electrolytic aluminium load-intensive area in Guangxi Power Grid. The results show that the method has high accuracy, the feasibility and effectiveness of this method are verified. This method can assist dispatchers to judge the transient voltage stability of the power grid.

Automatic Attention Learning Using Neural Architecture Search for Detection of Cardiac Abnormality in 12-Lead ECG

In this article, a new attention learning method based on neural architecture search (NAS) is introduced to detect cardiovascular diseases (CVDs). Cardiac abnormalities are a series of CVDs that threaten human health. Electrocardiography (ECG) is a useful tool to express the status of cardiac activity and detect cardiac abnormalities. In recent years, the convolutional neural network and attention mechanism have been applied in ECG classification with good feature-learning ability. However, there are a large number of hyperparameters that must be manually adjusted, so that they have limited ability to learn temporal and channel information from feature maps in the network. Therefore, in this article, a new attention learning method based on NAS is proposed, so that the network can better learn the temporal and channel information in the ECG and perform better than artificially designed architectures. The results were verified using the 12-lead ECG dataset of the China Physiological Signal Challenge (CPSC) 2018, which contains eight different CVDs and the proposed method performs better than state-of-the-art methods.

Implementing a Deep Learning Model for Intrusion Detection on Apache Spark Platform

Internet evolution produced a connected world with a massive amount of data. This connectivity advantage came with the price of more complex and advanced attacks. Intrusion Detection System (IDS) is an essential component for security in modern networks. The IDS methodology is either signature-based detection or anomaly behavior detection. Recently, researchers adopted Deep Learning (DL) because it has a better performance than traditional machine learning algorithms. The use of DL to produce a model for the IDS may take a long time because of computation complexity and a large number of hyperparameters. Different DL models for IDS on Apache Spark have been implemented in this article. This article uses the famous Network Security Lab - Knowledge Discovery and Data Mining (NSL-KDD) dataset and presents a computation delay comparison between Apache Spark and regular implementation. Moreover, an enhanced model is used to improve attack detection accuracy.

Uncertainty-optimized deep learning model for small-scale person re-identification

In recent years, deep learning has developed rapidly and is widely used in various fields, such as computer vision, speech recognition, and natural language processing. For end-to-end person re-identification, most deep learning methods rely on large-scale datasets. Relatively few methods work with small-scale datasets. Insufficient training samples will affect neural network accuracy significantly. This problem limits the practical application of person re-identification. For small-scale person re-identification, the uncertainty of person representation and the overfitting problem associated with deep learning remain to be solved. Quantifying the uncertainty is difficult owing to complex network structures and the large number of hyperparameters. In this study, we consider the uncertainty of pedestrian representation for small-scale person re-identification. To reduce the impact of uncertain person representations, we transform parameters into distributions and conduct multiple sampling by using multilevel dropout in a testing process. We design an improved Monte Carlo strategy that considers both the average distance and shortest distance for matching and ranking. When compared with state-of-the-art methods, the proposed method significantly improve accuracy on two small-scale person re-identification datasets and is robust on four large-scale datasets.

A Class of Conjugate Priors for Log‐Normal Claims Based on Conditional Specification

In this article, a new methodology for obtaining a premium based on a broad class of conjugate prior distributions, assuming lognormal claims, is presented. The new class of prior distributions arise in a natural way, using the conditional specification technique introduced by Arnold, Castillo, and Sarabia (1998, 1999). The new family of prior distributions is very flexible and contains, as particular cases, many other distributions proposed in the literature. Together with its flexibility, the main advantage of this distribution is that, due to its dependence on a large number of hyperparameters, it allows incorporating a wide amount of prior information. Several methods for hyperparameter elicitation are proposed. Finally, some examples with real and simulated data are given.