Random Initialization Research Articles

Low-temperature Ising dynamics with random initializations

Low-temperature Ising dynamics with random initializations

Cluster-Based Input Weight Initialization for Echo State Networks.

Echo state networks (ESNs) are a special type of recurrent neural networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. Despite the recent success of ESNs in various tasks of audio, image, and radar recognition, we postulate that a purely random initialization is not the ideal way of initializing ESNs. The aim of this work is to propose an unsupervised initialization of the input connections using the K-means algorithm on the training data. We show that for a large variety of datasets, this initialization performs equivalently or superior than a randomly initialized ESN while needing significantly less reservoir neurons. Furthermore, we discuss that this approach provides the opportunity to estimate a suitable size of the reservoir based on prior knowledge about the data.

A classification method for high‐dimensional imbalanced multi‐classification data

AbstractHigh‐dimensional imbalanced multi‐classification problems (HDIMCPs) occur frequently in engineering applications such as medical detection, item classification, and email classification. However, there is a paucity of research in the academic community on this topic. This paper proposes an evolutionary algorithm‐based classification method for HDIMCPs, named HIMALO (high‐dimensional imbalanced multi‐classification method based on ant lion optimizer). HIMALO proposes a new individual initialization strategy that replaces the random initialization of the ant lion optimizer with Fuch chaos. Then, it encodes individuals using concatenated sample features and base classifier weights, optimizes these features and weights concurrently during the iteration process. Additionally, a multi‐classification strategy, union one versus many, that combines one versus all and one‐against‐higher‐order is proposed. Numerous experiments are conducted to prove the superior classification performance and stability of HIMALO when compared with other algorithms.

Automatic DNN architecture design using CPSOTJUTT for power system inspection

To quickly and accurately automatically design more high-precision deep neural network models (DNNs), this paper proposes an automatic DNN architecture design ensemble model based on consensus particle swarm optimization-assisted trajectory unified and TRUST-TECH (CPSOTJUTT), called CPSOTJUTT-EM. The proposed model is a three-layer model, and its core is a three-stage method for addressing the sensitivity of the local solver to the initial point and enabling fast and robust training DNN, effectively avoiding missing high-quality DNN models in the process of automatic DNN architecture design. CPSOTJUTT has the following advantages: (1) high-quality local optimal solutions (LOSs) and (2) robust convergence against random initialization. CPSOTJUTT-EM consists of the bottom layer: stable and fast design high-quality DNN architectures, the middle layer: exploration for a diverse set of optimal DNN classification engines, and the top layer: ensemble model for higher performance. This paper tests the performance of CPSOTJUTT-EM on public datasets and three self-made power system inspection datasets. Experimental results show that the CPSOTJUTT-EM has excellent performance in automatic DNN architecture design, DNN model optimization. And the CPSOTJUTT-EM can automatically design high-quality DNN ensemble models, laying a solid foundation for the application of DNN in other fields.

Enhancing CNNs Performance on Object Recognition Tasks with Gabor Initialization

The use of Gabor filters in image processing has been well-established, and these filters are recognized for their exceptional feature extraction capabilities. These filters are usually applied through convolution. While convolutional neural networks (CNNs) are designed to learn optimal filters, little research exists regarding any advantages of initializing CNNs with Gabor filters. In this study, the performance of CNNs initialized with Gabor filters is compared to traditional CNNs with random initialization on six object recognition datasets. The results indicated that the Gabor-initialized CNNs outperformed the traditional CNNs in terms of accuracy, area under the curve, minimum loss, and convergence speed. A statistical analysis was performed to validate the performance of the classifiers, and the results showed that the Gabor classifiers outperformed the baseline classifiers. The findings of this study provide robust evidence in favor of using Gabor-based methods for initializing the receptive fields of CNN architectures.

Two-level deep learning ensemble model for forecasting hydroelectricity production

Due to its cheapest cost of production, flexibility, and very low indirect emissions, hydroelectricity is of utmost importance to the economy. Its production is challenging due to the stochastic nature of water and other environmental factors. Hence, forecasting its production is very important to the power planner. In this work, a novel two-level deep learning ensemble based on hyperparameter variations, random initialization, long short-term memory (LSTM), Convolutional one-dimensional neural network (Conv1D), and the dense neural network is proposed for short-term hydropower forecasting. In level, I, ensembles of LSTM, Conv1D, and dense network are used to forecast the components decomposed by Seasonal Trend Decomposition using Loess. These ensembles are built using multiple loss functions and making use of random initialization of the deep learning models. Integrating the median of these ensembles to obtain the final forecast constitutes level II. The model was validated with 61368 hourly univariate data sets from the largest hydropower plant in Cameroon, the Songloulou hydropower plant. The result reveals that the proposed model consistently outperforms all the other deep learning models base on evaluation criteria. The LSTM model was the best model among the individual deep models with a Mean square error (MSE), Mean absolute error (MAE), and Nash–Sutcliffe efficiency (NSE) of 533.557 (MWh)2, 15.690 MWh, and 0.646 respectively while the proposed two-level ensemble model had an MSE of 448.711 (MWh)2, MAE of 14.938 MWh, and NSE of 0.702 respectively.

Exploring unsupervised pre-training for echo state networks

Echo State Networks (ESNs) are a special type of Recurrent Neural Networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. However, recent publications have addressed the problem that a purely random initialization may not be ideal. Instead, a completely deterministic or data-driven initialized ESN structure was proposed. In this work, an unsupervised training methodology for the hidden components of an ESN is proposed. Motivated by traditional Hidden Markov Models (HMMs), which have been widely used for speech recognition for decades, we present an unsupervised pre-training method for the recurrent weights and bias weights of ESNs. This approach allows for using unlabeled data during the training procedure and shows superior results for continuous spoken phoneme recognition, as well as for a large variety of time-series classification datasets.

Research on Emotional Analysis of Online Book Reviews Based on Word2Vec Method

At present, emotional analysis in the field of natural language processing is gradually becoming a popular research direction on social media. With the rise of artificial intelligence technology and deep learning algorithms, word vector representation methods based on neural networks have emerged and emerged in the field of sentiment analysis. According to the current research results of experts, the Word2Vec model has become the mainstream word vector representation method. The algorithm model is based on continuous optimization and iteration of random initialization word vectors, ultimately obtaining a stable performance word vector representation. If the research goal is to obtain sentence vectors, weighting processing is also required. Therefore, after systematically summarizing and analyzing the representation techniques of initialized word vectors and solving sentence vectors through word vector weighting, this article proposes corresponding improvement methods. This article improves the initialization algorithm of its word vector based on the Word2Vec model. On the basis of the original randomly generated word vector, the correlation information of words is added to change the distribution characteristics of the initial word vector, thereby enhancing the performance of the word vector. In the comparative experiment of the user comment dataset, the improved model showed a 1% to 3% improvement in sentiment polarity classification accuracy compared to the other model.

HGSOXGB: Hunger-Games-Search-Optimization-Based Framework to Predict the Need for ICU Admission for COVID-19 Patients Using eXtreme Gradient Boosting

Millions of people died in the COVID-19 pandemic, which pressured hospitals and healthcare workers into keeping up with the speed and intensity of the outbreak, resulting in a scarcity of ICU beds for COVID-19 patients. Therefore, researchers have developed machine learning (ML) algorithms to assist in identifying patients at increased risk of requiring an ICU bed. However, many of these studies used state-of-the-art ML algorithms with arbitrary or default hyperparameters to control the learning process. Hyperparameter optimization is essential in enhancing the classification effectiveness and ensuring the optimal use of ML algorithms. Therefore, this study utilized an improved Hunger Games Search Optimization (HGSO) algorithm coupled with a robust extreme gradient boosting (XGB) classifier to predict a COVID-19 patient’s need for ICU transfer. To further mitigate the random initialization inherent in HGSO and facilitate an efficient convergence toward optimal solutions, the Metropolis–Hastings (MH) method is proposed for integration with HGSO. In addition, population diversity was reintroduced to effectively escape local optima. To evaluate the efficacy of the MH-based HGSO algorithm, the proposed method was compared with the original HGSO algorithm using the Congress on Evolutionary Computation benchmark function. The analysis revealed that the proposed algorithm converges better than the original method and exhibits statistical significance. Consequently, the proposed algorithm optimizes the XGB hyperparameters to further predict the need for ICU transfer for COVID-19 patients. Various evaluation metrics, including the receiver operating curve (ROC), precision–recall curve, bootstrap ROC, and recall vs. decision boundary, were used to estimate the effectiveness of the proposed HGSOXGB model. The model achieves the highest accuracy of 97.39% and an area under the ROC curve of 99.10% compared with other classifiers. Additionally, the important features that significantly affect the prediction of ICU transfer need using XGB were calculated.

Chaotic sparrow search algorithm with manta ray spiral foraging for engineering optimization

Targeting the problem of traditional sparrow search algorithms being prone to falling into local optima, a new algorithm called the Chaotic Sparrow Search Algorithm with Manta Ray Spiral Foraging (abbreviated as MSSA) is proposed. The Logistic-Sine-Cosine chaotic map and elite Reverse learning strategy are fused to initialize the population. It is experimentally demonstrated that this hybrid strategy outperforms the population after random initialization in reducing ineffective sparrow individuals. In the vigilante update stage, the spiral foraging behaviour of the manta ray population in the integrated manta ray optimization algorithm, the sparrows search around the best food source, which enhances the sparrow search algorithm's ability to explore the optimal solution. To enhance the stability of the algorithm to search for the optimum, a mixed Gaussian variational and logistic perturbation strategy is proposed to further improve the performance of the algorithm. Finally, using 12 commonly used benchmark test functions and the Wilcoxon rank sum test, MSSA was compared with other original algorithms and advanced improved algorithms, it is demonstrated that MSSA has higher accuracy and convergence performance, and the improved MSSA algorithm is applied to three types of engineering optimization problems with constraints, demonstrating its feasibility and effectiveness.

Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank

In deep learning, it is common to use more network parameters than training points. In such scenario of over-parameterization, there are usually multiple networks that achieve zero training error so that the training algorithm induces an implicit bias on the computed solution. In practice, (stochastic) gradient descent tends to prefer solutions which generalize well, which provides a possible explanation of the success of deep learning. In this paper we analyze the dynamics of gradient descent in the simplified setting of linear networks and of an estimation problem. Although we are not in an overparameterized scenario, our analysis nevertheless provides insights into the phenomenon of implicit bias. In fact, we derive a rigorous analysis of the dynamics of vanilla gradient descent, and characterize the dynamical convergence of the spectrum. We are able to accurately locate time intervals where the effective rank of the iterates is close to the effective rank of a low-rank projection of the ground-truth matrix. In practice, those intervals can be used as criteria for early stopping if a certain regularity is desired. We also provide empirical evidence for implicit bias in more general scenarios, such as matrix sensing and random initialization. This suggests that deep learning prefers trajectories whose complexity (measured in terms of effective rank) is monotonically increasing, which we believe is a fundamental concept for the theoretical understanding of deep learning.

Bi-objective optimization of the scheduling of risk-related resources for risk response

Risk response is a crucial component of project risk management, playing a key role in mitigating the adverse effects of risks. However, the implementation of such risk response relies on the utilization of appropriate risk-related resources. Given the scarcity of these resources, their proper scheduling and allocation are critical. To address this issue, this paper introduces a flow-based continuous-time bi-objective optimization model for risk-related resource planning. The proposed model is then applied to a case study, deriving the global optimal solution based on the tuned parameters. Moreover, we develop a tailored rule-based metaheuristic algorithm for the model. The algorithm incorporates improved random key-based population initialization and a rule-based genetic operator, facilitating the application of the proposed model to large-scale projects. The computational results of the case study and numerical experiments not only validate the effectiveness of the algorithm, but also emphasize the importance of the precautionary principle and redundant resources in project risk management.

Empowering Foot Health: Harnessing the Adaptive Weighted Sub-Gradient Convolutional Neural Network for Diabetic Foot Ulcer Classification.

In recent times, DFU (diabetic foot ulcer) has become a universal health problem that affects many diabetes patients severely. DFU requires immediate proper treatment to avert amputation. Clinical examination of DFU is a tedious process and complex in nature. Concurrently, DL (deep learning) methodologies can show prominent outcomes in the classification of DFU because of their efficient learning capacity. Though traditional systems have tried using DL-based models to procure better performance, there is room for enhancement in accuracy. Therefore, the present study uses the AWSg-CNN (Adaptive Weighted Sub-gradient Convolutional Neural Network) method to classify DFU. A DFUC dataset is considered, and several processes are involved in the present study. Initially, the proposed method starts with pre-processing, excluding inconsistent and missing data, to enhance dataset quality and accuracy. Further, for classification, the proposed method utilizes the process of RIW (random initialization of weights) and log softmax with the ASGO (Adaptive Sub-gradient Optimizer) for effective performance. In this process, RIW efficiently learns the shift of feature space between the convolutional layers. To evade the underflow of gradients, the log softmax function is used. When logging softmax with the ASGO is used for the activation function, the gradient steps are controlled. An adaptive modification of the proximal function simplifies the learning rate significantly, and optimal proximal functions are produced. Due to such merits, the proposed method can perform better classification. The predicted results are displayed on the webpage through the HTML, CSS, and Flask frameworks. The effectiveness of the proposed system is evaluated with accuracy, recall, F1-score, and precision to confirm its effectual performance.

Classification of Pulmonary Nodules in 2-[18F]FDG PET/CT Images with a 3D Convolutional Neural Network

Purpose2-[18F]FDG PET/CT plays an important role in the management of pulmonary nodules. Convolutional neural networks (CNNs) automatically learn features from images and have the potential to improve the discrimination between malignant and benign pulmonary nodules. The purpose of this study was to develop and validate a CNN model for classification of pulmonary nodules from 2-[18F]FDG PET images.MethodsOne hundred thirteen participants were retrospectively selected. One nodule per participant. The 2-[18F]FDG PET images were preprocessed and annotated with the reference standard. The deep learning experiment entailed random data splitting in five sets. A test set was held out for evaluation of the final model. Four-fold cross-validation was performed from the remaining sets for training and evaluating a set of candidate models and for selecting the final model. Models of three types of 3D CNNs architectures were trained from random weight initialization (Stacked 3D CNN, VGG-like and Inception-v2-like models) both in original and augmented datasets. Transfer learning, from ImageNet with ResNet-50, was also used.ResultsThe final model (Stacked 3D CNN model) obtained an area under the ROC curve of 0.8385 (95% CI: 0.6455–1.0000) in the test set. The model had a sensibility of 80.00%, a specificity of 69.23% and an accuracy of 73.91%, in the test set, for an optimised decision threshold that assigns a higher cost to false negatives.ConclusionA 3D CNN model was effective at distinguishing benign from malignant pulmonary nodules in 2-[18F]FDG PET images.

An intelligent hybrid approach for photovoltaic power forecasting using enhanced chaos game optimization algorithm and Locality sensitive hashing based Informer model

Highly accurate photovoltaic power forecasting is of great significance for increasing photovoltaic (PV) power generation on urban building rooftops. In this research context, a method based on variational mode decomposition (VMD), an enhanced chaos game optimization (ECGO) algorithm, and an improved Informer model is proposed for predicting PV power. First, the data are decomposed using VMD. Then the Informer model is improved using locality sensitive hashing (LSH) attention. In this study, an attempt is made to incorporate LSH to find the nearest few key vectors for query vectors, which enhances the accuracy of the model for prediction. The chaos game optimization (CGO) algorithm is then enhanced using the circle chaos initialization and golden sine algorithm. The Circle Chaos Initialization replaces the random initialization in the CGO algorithm, and the golden sine algorithm updates the position of the fourth seed in the CGO. The hyperparameters of the model proposed in this study were then optimized using an Enhanced Chaos Game Optimization (ECGO) algorithm. The experimental results show that the RMSE for April, May, and June data are improved by about 45.41%, 62.30%, and 50.52%, respectively, compared to Informer. In summary, the VMD-ECGO-LSH-Informer model can effectively improve the prediction accuracy of PV power generation, which provides important support and valuable contribution to improving the efficiency of rooftop PV power generation installations in urban buildings.

A feature-thresholds guided genetic algorithm based on a multi-objective feature scoring method for high-dimensional feature selection

The classical genetic algorithm utilizes random population initialization, an unguided crossover operator, and an unguided mutation operator for feature selection. However, this approach may be too stochastic and result in slow convergence. This paper proposes a hybrid feature selection algorithm named the Feature-Thresholds Guided Genetic Algorithm (FTGGA) to overcome this deficiency. FTGGA first employs ReliefF to filter out redundant features and retains crucial ones. Then, it generates a feature-thresholds set that contains all the feature thresholds. Each feature threshold represents the probability that the corresponding feature will be selected. The feature-thresholds set continuously updates to guide the iteration process of the genetic algorithm, accelerating its convergence. The experimental data demonstrates that FTGGA has a smaller feature subset and better classification accuracy compared to other algorithms.

Parameter estimation in non-linear chemical processes: an opposite point-based differential evolution (OPDE) approach

Abstract In recent years, evolutionary algorithms have been gaining popularity for finding optimal solutions to non-linear multimodal problems encountered in many engineering disciplines. Differential evolution (DE), an evolutionary algorithm, is a novel optimization method capable of handling nondifferentiable, non-linear, and multimodal objective functions. DE is an efficient, effective, and robust evolutionary optimization method. Still, DE takes large computational time to optimize the computationally expensive objective functions. Therefore, an attempt to speed up DE is considered necessary. This paper introduces a modification to the original DE that enhances the convergence rate without compromising solution quality. The proposed opposite point-based differential evolution (OPDE) algorithm utilizes opposite point-based population initialization, in addition to random initialization. Such an improvement reduces computational effort. The OPDE has been applied to benchmark test functions and high-dimensional non-linear chemical engineering problems. The proposed method of population initialization accelerates the convergence speed of DE, as indicated by the results obtained using benchmark test functions and non-linear chemical engineering problems.

Skip DETR: end-to-end Skip connection model for small object detection in forestry pest dataset.

Object detection has a wide range of applications in forestry pest control. However, forest pest detection faces the challenges of a lack of datasets and low accuracy of small target detection. DETR is an end-to-end object detection model based on the transformer, which has the advantages of simple structure and easy migration. However, the object query initialization of DETR is random, and random initialization will cause the model convergence to be slow and unstable. At the same time, the correlation between different network layers is not strong, resulting in DETR is not very ideal in small object training, optimization, and performance. In order to alleviate these problems, we propose Skip DETR, which improves the feature fusion between different network layers through skip connection and the introduction of spatial pyramid pooling layers so as to improve the detection results of small objects. We performed experiments on Forestry Pest Datasets, and the experimental results showed significant AP improvements in our method. When the value of IoU is 0.5, our method is 7.7% higher than the baseline and 6.1% higher than the detection result of small objects. Experimental results show that the application of skip connection and spatial pyramid pooling layer in the detection framework can effectively improve the effect of small-sample obiect detection.

Mirror Descent of Hopfield Model

Abstract Mirror descent is an elegant optimization technique that leverages a dual space of parametric models to perform gradient descent. While originally developed for convex optimization, it has increasingly been applied in the field of machine learning. In this study, we propose a novel approach for using mirror descent to initialize the parameters of neural networks. Specifically, we demonstrate that by using the Hopfield model as a prototype for neural networks, mirror descent can effectively train the model with significantly improved performance compared to traditional gradient descent methods that rely on random parameter initialization. Our findings highlight the potential of mirror descent as a promising initialization technique for enhancing the optimization of machine learning models.

Random projection forest initialization for graph convolutional networks

Graph convolutional networks (GCNs) were a great step towards extending deep learning to graphs. GCN uses the graph G and the feature matrix X as inputs. However, in most cases the graph G is missing and we are only provided with the feature matrix X. To solve this problem, classical graphs such as k-nearest neighbor (k-nn) are usually used to construct the graph G and initialize the GCN. Although it is computationally efficient to construct k-nn graphs, the constructed graph might not be very useful for learning. In a k-nn graph, points are restricted to have a fixed number of edges, and all edges in the graph have equal weights. Our contribution is Initializing GCN using a graph with varying weights on edges, which provides better performance compared to k-nn initialization. Our proposed method is based on random projection forest (rpForest). rpForest enables us to assign varying weights on edges indicating varying importance, which enhanced the learning. The number of trees is a hyperparameter in rpForest. We performed spectral analysis to help us setting this parameter in the right range. In the experiments, initializing the GCN using rpForest provides better results compared to k-nn initialization.•Constructing the graph G using rpForest sets varying weights on edges, which represents the similarity between a pair of samples.Unlike k-nearest neighbor graph where all weights are equal.•Using rpForest graph to initialize GCN provides better results compared to k-nn initialization. The varying weights in rpForest graph quantify the similarity between samples, which guided the GCN training to deliver better results.•The rpForest graph involves the tuning of the hyperparameter (number of trees T). We provided an informative way to set this hyperparameter through spectral analysis.