Set Of Hyperparameters Research Articles

A Review of Convex Clustering From Multiple Perspectives: Models, Optimizations, Statistical Properties, Applications, and Connections.

Traditional partition-based clustering is very sensitive to the initialized centroids, which are easily stuck in the local minimum due to their nonconvex objectives. To this end, convex clustering is proposed by relaxing K -means clustering or hierarchical clustering. As an emerging and excellent clustering technology, convex clustering can solve the instability problems of partition-based clustering methods. Generally, convex clustering objective consists of the fidelity and the shrinkage terms. The fidelity term encourages the cluster centroids to estimate the observations and the shrinkage term shrinks the cluster centroids matrix so that their observations share the same cluster centroid in the same category. Regularized by the lpn -norm ( pn ∈ {1,2,+∞} ), the convex objective guarantees the global optimal solution of the cluster centroids. This survey conducts a comprehensive review of convex clustering. It starts with the convex clustering as well as its nonconvex variants and then concentrates on the optimization algorithms and the hyperparameters setting. In particular, the statistical properties, the applications, and the connections of convex clustering with other methods are reviewed and discussed thoroughly for a better understanding the convex clustering. Finally, we briefly summarize the development of convex clustering and present some potential directions for future research.

Short term road network Macroscopic Fundamental Diagram parameters and traffic state prediction based on LSTM

Macroscopic Fundamental Diagram (MFD) is widely used in traffic state evaluation due to its description of the macro level of urban road network. This study focuses on the discrimination and short-term prediction of macro traffic states in urban road networks, using MFD combined with FCM clustering for state partitioning to characterize different macro states of the road network. To predict the MFD state, this paper builds two LSTMs to perform short-term predictions on two important parameters in MFD: road network weighted flow qw and road network weighted density kw. The parameters of the first three statistical intervals and the predicted time period are used as inputs to output the MFD parameters for the predicted time period. To ensure that the two LSTM structures and hyper-parameters settings can achieve the best prediction performance for MFD, the parameter optimization process of both should be included in the same search framework for hyper-parameters search. Therefore, this paper uses GA algorithm combined with multi-objective particle swarm optimization algorithm as the solving algorithm, with the accuracy of solving two MFD parameters and the accuracy of MFD point positioning as the hyper-parameters solving objectives. The study was validated using actual road network data from Hong Kong, and the results showed that the method proposed in this paper has an MRE prediction error of less than 7.8% for the two parameters of MFD, and can predict the future temporal trend of the two parameters, demonstrating the feasibility of MFD related predictions. The model's prediction of the overall shape and change trajectory trend of MFD is consistent with reality, and some test sets in MFD state prediction show high accuracy, the overall accuracy is 81.45%. To verify the effectiveness of the multi-objective search algorithm, typical LSTM models and RNN models were used for comparison. The experiment proved that the model used in this study performed better in error control and state prediction. This study explores and practices a short-term prediction method for road network MFD parameters, MFD status, and their changing trends. Provided path reference for urban road network prediction, traffic control status, and MFD related research.

An optimal goose lithium-ion batteries accurate and rapid RUL prediction method with automatic initial hyperparameters settings

Abstract Machine learning has emerged as a highly effective tool for addressing complex data problems, garnering significant attention in the field of equipment degradation and remaining service life prediction. Existing prediction models typically exhibit two primary shortcomings: on the one hand, the accuracy of life prediction reaches the desired level of precision while failing to achieve a sufficiently fast prediction speed, and on the other hand, generalization is not guaranteed while requiring the model to be robust. These two aspects present a significant challenge to the field of machine learning. In light of the aforementioned issues, we propose a prediction model based on the goose algorithm. Initially, we set the goose algorithm using adaptive initialization of the goose population to guarantee that the goose population is set at the appropriate interval, and we incorporate it into the extreme learning machine model through the improved goose algorithm. goose algorithm is used to predict the service life. Finally, we utilize different types of lithium batteries with varying operational conditions to conduct pertinent case studies to validate the proposed prediction model. The results demonstrated that the average accuracy was above 98% in all validated datasets. The shortest computation time was 0.19 s.

Unfolding multilevel agglomerative strategies for SVM classification: a case study in discriminating spectrally similar land covers

ABSTRACT In remote sensing applications, image classification algorithms normally require parameter optimization strategies to adapt to the complexities of the data and determine the model’s parameters for optimal performance. However, a unique set of hyper-parameters is generally chosen, without acknowledging the presence of subsets of classes that exhibit different levels of spectral similarity. A model used to distinguish between well-separable classes might not be optimal for delineating other classes. This study explores a classification method that extends the standard One-Against-One (OAO) and One-Against-All (OAA) strategies for Support Vector Machines (SVMs), based on agglomerative hierarchical clustering of spectrally-similar classes. The main objective is to provide additional classification scenarios where more detailed per-class analyses can be performed, rather than simple comparisons of global accuracies, particularly for hard-to-distinguish classes. Clusters of classes closely located in the feature space are initially classified using an OAA strategy, while each cluster undergoes further subdivision using an OAO approach. This strategy aims to identify the most effective sequence for incorporating subgroups of classes into the tuning process. The experimental results indicate that the proposed method can be efficiently applied to classify remote sensing data, producing varying levels of enhancement in per-class accuracies at each stage of the agglomerative process.

Metafeature Selection via Multivariate Sparse-Group Lasso Learning for Automatic Hyperparameter Configuration Recommendation.

The performance of classification algorithms is mainly governed by the hyperparameter settings deployed in applications, and the search for desirable hyperparameter configurations usually is quite challenging due to the complexity of datasets. Metafeatures are a group of measures that characterize the underlying dataset from various aspects, and the corresponding recommendation algorithm fully relies on the appropriate selection of metafeatures. Metalearning (MtL), aiming to improve the learning algorithm itself, requires development in integrating features, models, and algorithm learning to accomplish its goal. In this article, we develop a multivariate sparse-group Lasso (SGLasso) model embedded with MtL capacity in recommending suitable configurations via learning. The main idea is to select the principal metafeatures by removing those redundant or irregular ones, promoting both efficiency and performance in the hyperparameter configuration recommendation. To be specific, we first extract the metafeatures and classification performance of a set of configurations from the collection of historical datasets, and then, a metaregression task is established through SGLasso to capture the main characteristics of the underlying relationship between metafeatures and historical performance. For a new dataset, the classification performance of configurations can be estimated through the selected metafeatures so that the configuration with the highest predictive performance in terms of the new dataset can be generated. Furthermore, a general MtL architecture combined with our model is developed. Extensive experiments are conducted on 136 UCI datasets, demonstrating the effectiveness of the proposed approach. The empirical results on the well-known SVM show that our model can effectively recommend suitable configurations and outperform the existing MtL-based methods and the well-known search-based algorithms, such as random search, Bayesian optimization, and Hyperband.

Towards an open pipeline for the detection of critical infrastructure from satellite imagery—a case study on electrical substations in The Netherlands

Abstract Critical infrastructure (CI) are at risk of failure due to the increased frequency and magnitude of climate extremes related to climate change. It is thus essential to include them in a risk management framework to identify risk hotspots, develop risk management policies and support adaptation strategies to enhance their resilience. However, the lack of information on the exposure of CI to natural hazards prevents their incorporation in large-scale risk assessments. This study sets out to improve the representation of CI for risk assessment studies by building a neural network model to detect CI assets from optical remote sensing imagery. We present a pipeline that extracts CI from OpenStreetMap (OSM), processes the imagery and assets’ masks, and trains a Mask R-CNN model that allows for instance segmentation of CI at the asset level. This study provides an overview of the pipeline and tests it with the detection of electrical substations assets in the Netherlands. Several experiments are presented for different under-sampling percentages of the majority class (25 %, 50 % and 100 %) and hyperparameters settings (batch size and learning rate). The highest scoring experiment achieved an Average Precision at an Intersection over Union of 50 % of 30.93 and a tile F-score of 89.88 %. This allows us to confirm the feasibility of the method and invite disaster risk researchers to use this pipeline for other infrastructure types. We conclude by exploring the different avenues to improve the pipeline by addressing the class imbalance, Transfer Learning and Explainable Artificial Intelligence.

Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations.

Graph contrastive learning (GCL) is a promising direction toward alleviating the label dependence, poor generalization and weak robustness of graph neural networks, learning representations with invariance, and discriminability by solving pretasks. The pretasks are mainly built on mutual information estimation, which requires data augmentation to construct positive samples with similar semantics to learn invariant signals and negative samples with dissimilar semantics to empower representation discriminability. However, an appropriate data augmentation configuration depends heavily on lots of empirical trials such as choosing the compositions of data augmentation techniques and the corresponding hyperparameter settings. We propose an augmentation-free GCL method, invariant-discriminative GCL (iGCL), that does not intrinsically require negative samples. iGCL designs the invariant-discriminative loss (ID loss) to learn invariant and discriminative representations. On the one hand, ID loss learns invariant signals by directly minimizing the mean square error (MSE) between the target samples and positive samples in the representation space. On the other hand, ID loss ensures that the representations are discriminative by an orthonormal constraint forcing the different dimensions of representations to be independent of each other. This prevents representations from collapsing to a point or subspace. Our theoretical analysis explains the effectiveness of ID loss from the perspectives of the redundancy reduction criterion, canonical correlation analysis (CCA), and information bottleneck (IB) principle. The experimental results demonstrate that iGCL outperforms all baselines on five node classification benchmark datasets. iGCL also shows superior performance for different label ratios and is capable of resisting graph attacks, which indicates that iGCL has excellent generalization and robustness. The source code is available at https://github.com/lehaifeng/ T-GCN/tree/master/iGCL.

Chaos, overfitting and equilibrium: To what extent can machine learning beat the financial market?

In this paper, we applied 10 technical analysis indicators to predict stock price movement directions using support vector machines, investigating the effects of hyperparameter variations on the out-of-sample classification performance and the profitability of the resulting trading strategies. We collected daily data between January 1st, 2018, and March 31st, 2023 for the 30 firms that compose the Dow Jones Industrial Average (DJIA). Our results indicated that the out-of-sample accuracy converged to 50%, while a small percentage (13.63% for the pre-COVID period and 23.16% for the post-COVID period) of the hyperparameter combinations yielded gains above the buy-and-hold strategy; on the other hand, no clear patterns about the best-performing hyperparameter combinations emerged, as the behavior of the out-of-sample performance was found to exhibit high sensitive dependence to the hyperparameters settings in comparison to its in-sample counterpart. The outcomes of our empirical analysis are consistent with both classic results in the finance literature (such as the Efficient Market Hypothesis) and empirical setbacks commonly seen in machine learning experiments, notably the occurrence of overfitting under the incorporation of high-dimensional non-linear interactions.

Image-based facial emotion recognition using convolutional neural network on emognition dataset

Detecting emotions from facial images is difficult because facial expressions can vary significantly. Previous research on using deep learning models to classify emotions from facial images has been carried out on various datasets that contain a limited range of expressions. This study expands the use of deep learning for facial emotion recognition (FER) based on Emognition dataset that includes ten target emotions: amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, sadness, and neutral. A series of data preprocessing was carried out to convert video data into images and augment the data. This study proposes Convolutional Neural Network (CNN) models built through two approaches, which are transfer learning (fine-tuned) with pre-trained models of Inception-V3 and MobileNet-V2 and building from scratch using the Taguchi method to find robust combination of hyperparameters setting. The proposed model demonstrated favorable performance over a series of experimental processes with an accuracy and an average F1-score of 96% and 0.95, respectively, on the test data.

Utilization of XGBoost algorithm to predict dryout incipience quality for saturated flow boiling in mini/micro-channels

Critical heat flux (CHF) is arguably the most important design parameter for applications involving cooling of high heat flux devices. Despite prior limited successes, developing accurate predictive tools for CHF remains quite challenging, but key identifiable trigger mechanisms for CHF are intermittent liquid film dryout, complete liquid film dryout, and departure from nucleate boiling (DNB). For saturated flow boiling in mini/micro-channels, dryout incipience is a necessary condition for commencement of liquid film dryout, which is marked by a substantial deterioration in the heat transfer coefficient. The primary objective of this study is to predict system parameters for dryout incipience quality using machine learning, relying on a massive database for this parameter. Predictions are achieved using eXtreme Gradient Boosting (XGBoost), one of the supervised ensemble machine learning methods. This technique is applied to the Purdue University Boiling and Two-Phase Flow Laboratory (PU-BTPFL) consolidated database for dryout incipience quality for saturated flow boiling in mini/micro-channels. This database comprises 997 datapoints amassed from 26 sources and encompasses 13 different working fluids, hydraulic diameters from 0.51 to 6.0 mm, mass velocities from 29 to 2303 kg/m2s, liquid-only Reynolds numbers from 125 to 53,770, boiling numbers from 0.31×104 to 44.3×104, and reduced pressures from 0.005 to 0.78. Optuna, a supervised automated hyper-parameter optimization software, is used to set the best hyper-parameters in the learning process based on the consolidated database. A part of consolidated database is used to train the XGBoost algorithm, and the XGBoost machine, consisting of specific input parameters, is developed with appropriately determined hyper-parameter set after optimization. The trained XGBoost machine is shown to provide remarkable accuracy in predicting the dryout incipience quality, evidenced by a mean absolute error (MAE) of 2.45% and mean absolute deviation (MAD) of 3.57×10−2.

A stochastic gradient method with variance control and variable learning rate for Deep Learning

In this paper we study a stochastic gradient algorithm which rules the increase of the mini-batch size in a predefined fashion and automatically adjusts the learning rate by means of a monotone or non-monotone line search procedure. The mini-batch size is incremented at a suitable a priori rate throughout the iterative process in order that the variance of the stochastic gradients is progressively reduced. The a priori rate is not subject to restrictive assumptions, allowing for the possibility of a slow increase in the mini-batch size. On the other hand, the learning rate can vary non-monotonically throughout the iterations, as long as it is appropriately bounded. Convergence results for the proposed method are provided for both convex and non-convex objective functions. Moreover it can be proved that the algorithm enjoys a global linear rate of convergence on strongly convex functions. The low per-iteration cost, the limited memory requirements and the robustness against the hyperparameters setting make the suggested approach well-suited for implementation within the deep learning framework, also for GPGPU-equipped architectures. Numerical results on training deep neural networks for multi-class image classification show a promising behaviour of the proposed scheme with respect to similar state of the art competitors.

Extensive examination of hyper-parameters setting using neural-based methods for limited resources language: Nyishi-English

Extensive examination of hyper-parameters setting using neural-based methods for limited resources language: Nyishi-English

AI-based disease category prediction model using symptoms from low-resource Ethiopian language: Afaan Oromo text.

Automated disease diagnosis and prediction, powered by AI, play a crucial role in enabling medical professionals to deliver effective care to patients. While such predictive tools have been extensively explored in resource-rich languages like English, this manuscript focuses on predicting disease categories automatically from symptoms documented in the Afaan Oromo language, employing various classification algorithms. This study encompasses machine learning techniques such as support vector machines, random forests, logistic regression, and Naïve Bayes, as well as deep learning approaches including LSTM, GRU, and Bi-LSTM. Due to the unavailability of a standard corpus, we prepared three data sets with different numbers of patient symptoms arranged into 10 categories. The two feature representations, TF-IDF and word embedding, were employed. The performance of the proposed methodology has been evaluated using accuracy, recall, precision, and F1 score. The experimental results show that, among machine learning models, the SVM model using TF-IDF had the highest accuracy and F1 score of 94.7%, while the LSTM model using word2vec embedding showed an accuracy rate of 95.7% and F1 score of 96.0% from deep learning models. To enhance the optimal performance of each model, several hyper-parameter tuning settings were used. This study shows that the LSTM model verifies to be the best of all the other models over the entire dataset.

A Theoretical View of Linear Backpropagation and its Convergence.

Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs). Applied often along with stochastic gradient descent (SGD) or its variants, BP is considered as a de-facto choice in a variety of machine learning tasks including DNN training and adversarial attack/defense. Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks, by (Guo et al. 2020). Although it has been shown empirically effective in black-box attacks, theoretical studies and convergence analyses of such a method is lacking. This paper serves as a complement and somewhat an extension to Guo et al. (2020) paper, by providing theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training. We demonstrate that, somewhat surprisingly, LinBP can lead to faster convergence in these tasks in the same hyper-parameter settings, compared to BP. We confirm our theoretical results with extensive experiments.

Automated, quick, and precise building extraction from aerial images using ll-unet model

The subject matter of this article is the detection and semantic segmentation of buildings from high-resolution aerial images. It extracts building images from similar characteristics of roads and soil objects. It is used for various applications such as urban planning, infrastructure development, and disaster management. The goal of this study is to develop a fast, accurate, and automatic building detection model based on the semantic segmentation LL-UNET architecture that is optimized and tuned with proper hyper parameter settings. The tasks to be solved are as follows: collect remote sensing building dataset that is divided into three parts of training, validation, and testing; apply data augmentation on the training dataset by vertical flip, horizontal flip, and rotation methods; further pass into the bilateral filter to remove noises from the images; optimize LL-UNET model by various optimizer methods and tuned hyper parameter by proper selection value, the method is compared by the performance metrics recall, precision, and accuracy. The following results were obtained: the model is evaluated under the training loss curve and accuracy curve of different optimizers SGD, ASGD, ADAM, ADAMW, and RMSProp; it measures the training time, mean accuracy, and mean IOU parameters during the training phase; the testing phase is evaluated by precision and recall; the method is compared by visualizing the result of LL-UNET + different optimizers; and the proposed method is compared with the existing method by common evolution parameters metric. Conclusions. The scientific novelty of the results obtained is as follows: 1) the LL-UNET effectively segmented the building remotely sensed images in the limited number of training samples available; 2) the loss function of the model observed under hyper parameter selection of the optimizer, learning rate, batch size, and epochs; which makes an optimal model to extract the building in an accurate and fast manner from the complex background; 3) the proposed model results compared with a well-known model of the building extraction under the common evaluation metrics of F1 score.

Fine-tuning a pre-trained ResNet50 model to detect distributed denial of service attack

Distributed denial-of-service (DDoS) attacks pose a significant risk to the dependability and consistency of network services. The utilization of deep learning (DL) models has displayed encouraging outcomes in the identification of DDoS attacks. Nevertheless, crafting a precise DL model necessitates an extensive volume of labeled data and substantial computational capabilities. Within this piece, we introduce a technique to enhance a pre-trained DL model for the identification of DDoS attacks. Our strategy’s efficacy is showcased on an openly accessible dataset, revealing that the fine-tuned model we propose surpasses both the initial pre-trained model and other cutting-edge approaches in performance. The suggested fine-tuned model attained 95.1% accuracy, surpassing the initial pre-trained model as well as other leading-edge techniques. Please note that the specific evaluation metrics and their values may vary depending on the implementation, hyperparameter settings, number of datasets, or dataset characteristics. The proposed approach has several advantages, including reducing the amount of labeled data required and accelerating the training process. Initiating with a pre-existing ResNet50 model can also enhance the eventual model’s accuracy, given that the pre-trained model has already acquired the ability to extract significant features from unprocessed data.

Out-of-Sample Tuning for Causal Discovery.

Causal discovery is continually being enriched with new algorithms for learning causal graphical probabilistic models. Each one of them requires a set of hyperparameters, creating a great number of combinations. Given that the true graph is unknown and the learning task is unsupervised, the challenge to a practitioner is how to tune these choices. We propose out-of-sample causal tuning (OCT) that aims to select an optimal combination. The method treats a causal model as a set of predictive models and uses out-of-sample protocols for supervised methods. This approach can handle general settings like latent confounders and nonlinear relationships. The method uses an information-theoretic approach to be able to generalize to mixed data types and a penalty for dense graphs to penalize for complexity. To evaluate OCT, we introduce a causal-based simulation method to create datasets that mimic the properties of real-world problems. We evaluate OCT against two other tuning approaches, based on stability and in-sample fitting. We show that OCT performs well in many experimental settings and it is an effective tuning method for causal discovery.

Hybrid DE-Optimized GPR and NARX/SVR Models for Forecasting Gold Spot Prices: A Case Study of the Global Commodities Market

In this work, we highlight three different techniques for automatically constructing the dataset for a time-series study: the direct multi-step, the recursive multi-step, and the direct–recursive hybrid scheme. The nonlinear autoregressive with exogenous variable support vector regression (NARX SVR) and the Gaussian process regression (GPR), combined with the differential evolution (DE) for parameter tuning, are the two novel hybrid methods used in this study. The hyper-parameter settings used in the GPR and SVR training processes as part of this optimization technique DE significantly affect how accurate the regression is. The accuracy in the prediction of DE/GPR and DE/SVR, with or without NARX, is examined in this article using data on spot gold prices from the New York Commodities Exchange (COMEX) that have been made publicly available. According to RMSE statistics, the numerical results obtained demonstrate that NARX DE/SVR achieved the best results.

Glass design using machine learning property models with prediction uncertainties: Nuclear waste glass formulation

The United States Department of Energy is responsible for managing the legacy nuclear waste stored in underground tanks at the Hanford Site. The waste will be separately vitrified as low-activity waste and high-level waste fractions. Waste glass formulation algorithms have been traditionally developed using partial quadratic mixture property-composition models. Recently, machine learning (ML) techniques have been used to predict glass properties and discover new glass materials for nuclear waste vitrification, and these advancements can be utilized to improve waste glass composition design. In this proof-of-principle study, ML algorithms such as Gaussian process regression (GPR) were used to interpolate glass properties (e.g., viscosity, electrical conductivity, chemical durability). After selecting appropriate sets of GPR hyper-parameters for each property, an optimization program was developed to formulate glass compositions to maximize waste loading while simultaneously satisfying property constraints. The results of the ML-based waste loadings and glass compositions were compared to those obtained using the traditional methods. Comparing to the previous glass design framework, the ML-based optimization methods offer improved glass designs and a streamlined approach to generation of optimally designed data and near real-time updates.

Closed-loop high precision human tissue temperature measurements using a joint forward and inverse method

A closed-loop forward-inverse joint multi-layer temperature prediction method is introduced to overcome the limitations of non-contact microwave-based precise temperature measurements within human tissues. This approach merges an incoherent four-layer tissue forward model with a multi-frequency high-precision inversion algorithm. Random errors were incorporated into the forward model to construct a multi-layer human tissue dataset for performance validation and optimization of the inversion algorithm. The lack of precise temperature measurements in internal human tissues was addressed using an incoherent four-layer forward model that integrates the Pennes heat transfer equation with a fluid–solid coupling boundary modeling technique. Parameter differentiation analysis was conducted in the forward modeling step using incoherent electromagnetic transport equations. The proposed high-precision multi-frequency inversion process, added to the objective function, refines the XGBoost algorithm by assigning a penalty factor and adjusting for neighboring tissue temperature distributions. The Optuna framework was then utilized to optimize XGBoost hyper-parameter sets, resulting in the Opt-XGBoost inversion algorithm. This approach achieved a root mean square error of 0.033 °C and an average absolute error of 0.0256 °C in simulations involving forward modeling-generated data.