Hyperparameter Space Research Articles

Intelligent Generative Models for Quantum Neural Networks

AbstractThe potential of quantum neural networks (QNN) for applications in several fields has been widely demonstrated, but hyperparameter selection remains a technical challenge. The complexity of combinatorial optimization rises dramatically as the dimensionality of the hyperparameter space increases, making hyperparameter selection exceptionally complex. Traditional manual trial‐and‐error methods are both time‐consuming and expensive. To break through this bottleneck, the Ansatz structure set is proposed and the Hyperband algorithm is utilized to combine the structure set with the stacking number and algorithmic hyperparameters to propose the Hyperband‐QNN model. The model can intelligently generate QNN according to the actual data features, and in practical applications in many fields such as medicine and agriculture, it performs well compared with the traditional machine learning algorithms as well as the quantum algorithms QSVM and QKNN, and when it is used to deal with the large‐scale data, the efficiency advantage becomes more obvious with the increase of the number of samples. This not only proves the advantages of Hyperband‐QNN model in practical applications but also provides useful exploration and reference for the integration of traditional algorithms and quantum algorithms in practical problems and the play of their respective advantages.

A method for in silico exploration of potential glioblastoma multiforme attractors using single-cell RNA sequencing

We presented a method to find potential cancer attractors using single-cell RNA sequencing (scRNA-seq) data. We tested our method in a Glioblastoma Multiforme (GBM) dataset, an aggressive brain tumor presenting high heterogeneity. Using the cancer attractor concept, we argued that the GBM’s underlying dynamics could partially explain the observed heterogeneity, with the dataset covering a representative region around the attractor. Exploratory data analysis revealed promising GBM’s cellular clusters within a 3-dimensional marker space. We approximated the clusters’ centroid as stable states and each cluster covariance matrix as defining confidence regions. To investigate the presence of attractors inside the confidence regions, we constructed a GBM gene regulatory network, defined a model for the dynamics, and prepared a framework for parameter estimation. An exploration of hyperparameter space allowed us to sample time series intending to simulate myriad variations of the tumor microenvironment. We obtained different densities of stable states across gene expression space and parameters displaying multistability across different clusters. Although we used our methodological approach in studying GBM, we would like to highlight its generality to other types of cancer. Therefore, this report contributes to an advance in the simulation of cancer dynamics and opens avenues to investigate potential therapeutic targets.

MEDEA: A New Model for Emulating Radio Antenna Beam Patterns for 21 cm Cosmology and Antenna Design Studies

In 21 cm experimental cosmology, accurate characterization of a radio telescope’s antenna beam response is essential to measure the 21 cm signal. Computational electromagnetic (CEM) simulations estimate the antenna beam pattern and frequency response by subjecting the EM model to different dependencies, or beam hyperparameters, such as soil dielectric constant or orientation with the environment. However, it is computationally expensive to search all possible parameter spaces to optimize the antenna design or accurately represent the beam to the level required for use as a systematic model in 21 cm cosmology. We therefore present the Model for Emulating Directivities and Electric fields of Antennas (MEDEA), an emulator that rapidly and accurately generates far-field radiation patterns over a large hyperparameter space. MEDEA takes a subset of beams simulated by CEM software, spatially decomposes them into coefficients on a complete, linear basis, and then interpolates them to form new beams at arbitrary hyperparameters. We test MEDEA on an analytical dipole and two numerical beams motivated by upcoming lunar lander missions, and then employ MEDEA as a model to fit mock radio spectrometer data to extract covariances on the input beam hyperparameters. We find that the interpolated beams have rms relative errors of at most 10−2 using 20 input beams or less, and that fits to mock data are able to recover the input beam hyperparameters when the model and mock are derived from the same set of beams. When a systematic bias is introduced into the mock data, extracted beam hyperparameters exhibit bias, as expected. We propose several extensions to MEDEA to potentially account for such bias.

PSbBO-Net: A Hybrid Particle Swarm and Bayesian Optimization-based DenseNet for Lung Cancer Detection using Histopathological and CT Images

Lung cancer remains a substantial global fatality; early detection is imperative for successful intervention and treatment. Deep learning (DL) models have shown promise in predicting lung cancer from medical images, but optimizing their parameters remains a challenging task. To improve prediction capability, this study introduces an approach by merging Particle Swarm Optimization and Bayesian Optimization (PSbBO) to optimize deep learning parameters. PSO provides an effective way for exploring the hyperparameter space, while Bayesian optimization provides a probabilistic framework for the effective evaluation and refining of a DL network. The simulation study showcases the effectiveness of the proposed model, achieving notable metrics for histopathological images, including an accuracy of 99.5%, precision of 98.3%, recall of 99.2%, F1-score of 99.4%, and an error rate of 1.19%. Furthermore, when applied to lung CT images, the proposed PSbBO demonstrates an accuracy of 98.8%, precision of 97.4%, recall of 98.3%, F1-score of 98.6%, and an error rate of 1.21%.

A machine learning method based on TPE-XGBoost model for TRIP/TWIP near-β titanium alloy design

The traditional design method for TRIP/TWIP near-β titanium alloys is time-consuming and expensive. This paper proposes a machine-learning design method for a near-β titanium alloy that uses the Tree-structured Parzen Estimator (TPE) algorithm to optimize the hyperparameter space of XGBoost for improved model prediction accuracy in TRIP/TWIP. The model predicts the mechanical properties and plasticity mechanisms of the alloy based on its composition, key empirical parameters derived from the composition, and solution treatment temperature. The test results of the TPE-XGBoost model showed that when predicting the ultimate tensile strength (UTS), the adjusted determination coefficient (Adjusted R2), root mean square error (RMSE), and mean absolute error (MAE) values were 0.9264, 39.63 MPa and 44.83 MPa, respectively. When predicting elongation (El), the Adjusted R2, RMSE, and MAE values are 0.9134, 1.97 %, and 1.89 %, respectively. When predicting the plastic mechanisms of the alloy, the F1-score, Recall, Precision, Accuracy, and Area Under the ROC Curve (AUC) were 0.74, 0.75, 0.80, 0.78, and 0.83, respectively. A near-β titanium alloy, Ti-8V-4Mo-3.5Al-3Cr-2Zr (wt%), along with an associated heat treatment process, has been developed using this model. The experimental results have demonstrated that the model accurately predicts the strength, elongation, and plasticity mechanisms of the alloys. The TRIP/TWIP effects occur due to {332}<113> twinning and secondary α" martensite formation during tensile deformation. This study offers an alternative approach for designing TRIP/TWIP near-β titanium alloys.

(Invited) Forecasting Hybrid Perovskites’ Dynamic Response through Machine Learning

Machine learning (ML) can significantly accelerate the discovery of materials for clean energy. Hybrid perovskites (an emerging class of materials for photovoltaics and light-emitting diodes) encompass an enormous chemical composition space, resulting in a hyperparameter space that is nearly impossible to be interrogated via traditional trial-and-error methods. Thus, there has been a pressing need within the materials research community to identify ML models that can be implemented to inform the physical and chemical behavior of the perovskites. Here, we apply distinct ML models (mostly based on recurrent neural networks) to classify and forecast physical properties encompassing electrical conductivity, photoluminescence response, the power conversion efficiency of photovoltaic devices, etc. Specifically, we use automated, in situ optical measurements to predict the optical response of a series of perovskites for 50+ hours, upon materials’ exposure to moisture. We quantitatively compare linear regression, echo state network, and seasonal auto-regressive integrated moving average with eXogenous regressor algorithms and achieve accuracy of >90% for the latter. Demonstrating the ability of this model to capture non-linear features. Our high-throughput measurements and ML-assisted data analysis exemplifies the potential of ML to diagnose and forecast hybrid perovskites’ response with a variety of chemical compositions.

Efficient predictive modeling of resilient modulus in stabilized clayey soil using automated machine learning

Pavement subgrade design relies on the resilient modulus (Mr) to analyze structural response to vehicle-like loading. Adding stabilizers to the soil subgrade makes estimating Mr difficult and resource-intensive. This study uses an automated machine learning (ML) strategy to predict the Mr of stabilized clayey soil using recycled plastic waste. The proposed method automates model selection and hyperparameter tuning, making it a feasible alternative to tedious ML modeling and costly laboratory testing. From extensive laboratory investigation involving 3285 experimental data points, the automated ML model using Bayesian optimization evaluates ensembles, support vector machine (SVM), neural network (NET), decision trees, (TREE), and Gaussian process (GP) regression models, identifying the best model based on cross-validation mean squared error (MSE). Bayesian optimization explores hyperparameter spaces to find optimal configurations, enhancing the accuracy, scalability, and reliability of the prediction model. The optimization process yielded the best results for the ensemble least square boost (LSBoost) model with a cross-validation mean squared error (MSE) value of 6.723×10−29. The optimized ML model’s performance is measured using R2 and adjusted R2. The LSboost model’s R2 and adjusted R2 values of 0.9999 suggested overfitting, prompting further investigations using performance metrics like root mean squared error (RMSE), mean absolute error (MSE), and probability density function (PDF) for normalized absolute error (NAE) for training and testing datasets for the predictive ML model. The small RMSE and MAE (0.0049 and 0.0005) values and symmetrical NAE distribution of the proposed ML model demonstrate its high accuracy and generalization capabilities. The proposed model was subsequently tested on the new, unseen data and achieved predictions with an error rate of 0.24 %. This confirms the proposed ML model’s superiority over conventional deformation models, making it ideal for reliable geotechnical engineering applications.

Swarm intelligence for full Stokes dynamic imaging reconstruction of interferometric data

Context. In very long baseline interferometry (VLBI), the combination of multiple antennas permits the synthesis of a virtual telescope with a larger diameter and consequently higher resolution than the individual antennas. However, due to the sparse nature of the array, recovering an image from the observed data is a challenging ill-posed inverse problem. Aims. The VLBI community is interested in not only recovering an image in total intensity from interferometric data, but also in obtaining results in the polarimetric and the temporal domain. Only a few algorithms are able to work in all these domains simultaneously. In particular, the algorithms based on optimization that consider various penalty terms specific to static total intensity imaging, time-variability and polarimetry are restricted to grids in the domain of the objective function. In this work we present a novel algorithm, multiobjective particle swarm optimization (MO-PSO), that is able to recover the optimal weights without any space-gridding, and to obtain the marginal contribution of each of the playing terms. Methods. To this end, we utilized multiobjective optimization together with particle swarm metaheuristics. We let the swarm of weights converge to the best position. Results. We evaluate our algorithm with synthetic data sets that are representative for the main science targets and instrumental configuration of the Event Horizon Telescope Collaboration (EHTC) and its planned successors. We successfully recover the polarimetric, static, and time-dynamic signature of the ground truth movie' even with relative sparsity, and a set of realistic data corruptions. Conclusions. We have built a novel, fast, hyperparameter space gridding-free algorithm that successfully recovers static and dynamic polarimetric reconstructions. Compared to regularized maximum likelihood (RML) methods, it avoids the need for parameter surveys, and it is not limited to the number of pixels, unlike recently proposed multiobjective imaging algorithms. Hence, this technique is a novel useful alternative tool to characterize full Stokes time-(in)dependent signatures in a VLBI data set robustly with a minimal set of user-based choices.

Quantifying Predictive Uncertainty and Feature Selection in River Bed Load Estimation: A Multi-Model Machine Learning Approach with Particle Swarm Optimization

This study presents a comprehensive multi-model machine learning (ML) approach to predict river bed load, addressing the challenge of quantifying predictive uncertainty in fluvial geomorphology. Six ML models—random forest (RF), categorical boosting (CAT), extra tree regression (ETR), gradient boosting machine (GBM), Bayesian regression model (BRM), and K-nearest neighbors (KNNs)—were thoroughly evaluated across several performance metrics like root mean square error (RMSE), and correlation coefficient (R). To enhance model training and optimize performance, particle swarm optimization (PSO) was employed for hyperparameter tuning across all the models, leveraging its capability to efficiently explore complex hyperparameter spaces. Our findings indicated that RF, GBM, CAT, and ETR demonstrate superior predictive performance (R score &gt; 0.936), benefiting significantly from PSO. In contrast, BRM displayed lower performance (0.838), indicating challenges with Bayesian approaches. The feature importance analysis, including permutation feature and SHAP values, highlighted the non-linear interdependencies between the variables, with river discharge (Q), bed slope (S), and flow width (W) being the most influential. This study also examined the specific impact of individual variables on model performance by adding and excluding individual variables, which is particularly meaningful when choosing input variables for the model, especially in limited data conditions. Uncertainty quantification through Monte Carlo simulations highlighted the enhanced predictability and reliability of models with larger datasets. The correlation between increased training data and improved model precision was evident in the consistent rise in mean R scores and reduction in standard deviations as the sample size increased. This research underscored the potential of advanced ensemble methods and PSO to mitigate the limitations of single-predictor models and exploit collective model strengths, thereby improving the reliability of predictions in river bed load estimation. The insights from this study provide a valuable framework for future research directions focused on optimizing ensemble configurations for hydro-dynamic modeling.

Data-driven Parameterizable Generative Adversarial Networks for Synthetic Data Augmentation of Guided Ultrasonic Wave Sensor Signals

Detecting and characterizing hidden damages in composite materials like Fibre-Metal Laminates (FML) remains a challenge. Guided Ultrasonic Waves (GUW) or X-ray imaging are commonly used to detect these damages, but their interpretation remains limited. Data-driven predictor models can detect damages in structures using GUW time-dependent signals, but the training data lacks variance, quality, and sufficient coverage of the hyper-parameter space. Synthetic data augmentation is often the only solution to create robust and generalized damage predictor models. Synthetic sensor data can be generated using model-based, model-assisted, or model-free methods. However, computed GUW signals show poor reality conformance due to too much constraints and simplifications. Recent developments in Generative Adversarial (Neural) Networks (GAN), commonly driven by a random generation process [1], include deterministic style vectors to generate specific signal data, determining constraints such as damage size, position, transducer positions, material and environmental properties. These new architectures aim to reduce the impact of environmental changes on data-driven damage predictor models by using parameterizable synthetic data generation. We will elaborate these new architectures and discuss the possibilities and challenges of using such GAN models for data generation of US signals in GUW-based SHM applications, as originally proposed in [2]. We will have a focus on low-quality real measuring data used for training and composite materials (e.g., FML) with hidden damages, and show some preliminary results. [1] Karras et al. "A style-based generator architecture for generative adversarial networks." Proceedings of the IEEE/CVF 2019 [2] Virupakshappa et al. "Using Generative Adversarial Networks to Generate Ultrasonic Signals." 2020 IEEE International Ultrasonics Symposium (IUS). IEEE, 2020

Democratizing protein language models with parameter-efficient fine-tuning

Proteomics has been revolutionized by large protein language models (PLMs), which learn unsupervised representations from large corpora of sequences. These models are typically fine-tuned in a supervised setting to adapt the model to specific downstream tasks. However, the computational and memory footprint of fine-tuning (FT) large PLMs presents a barrier for many research groups with limited computational resources. Natural language processing has seen a similar explosion in the size of models, where these challenges have been addressed by methods for parameter-efficient fine-tuning (PEFT). In this work, we introduce this paradigm to proteomics through leveraging the parameter-efficient method LoRA and training new models for two important tasks: predicting protein-protein interactions (PPIs) and predicting the symmetry of homooligomer quaternary structures. We show that these approaches are competitive with traditional FT while requiring reduced memory and substantially fewer parameters. We additionally show that for the PPI prediction task, training only the classification head also remains competitive with full FT, using five orders of magnitude fewer parameters, and that each of these methods outperform state-of-the-art PPI prediction methods with substantially reduced compute. We further perform a comprehensive evaluation of the hyperparameter space, demonstrate that PEFT of PLMs is robust to variations in these hyperparameters, and elucidate where best practices for PEFT in proteomics differ from those in natural language processing. All our model adaptation and evaluation code is available open-source at https://github.com/microsoft/peft_proteomics. Thus, we provide a blueprint to democratize the power of PLM adaptation to groups with limited computational resources.

Revised clusters of annotated unknown sounds in the Belgian part of the North sea

Acoustic signals, especially those of biological source, remain unexplored in the Belgian part of the North Sea (BPNS). The BPNS, although dominated by anthrophony (sounds from human activities), is expected to be acoustically diverse given the presence of biodiverse sandbanks, gravel beds and artificial hard structures. Under the framework of the LifeWatch Broadband Acoustic Network, sound data have been collected since the spring of 2020. These recordings, encompassing both biophony, geophony and anthrophony, have been listened to and annotated for unknown, acoustically salient sounds. To obtain the acoustic features of these annotations, we used two existing automatic feature extractions: the Animal Vocalization Encoder based on Self-Supervision (AVES) and a convolutional autoencoder network (CAE) retrained on the data from this study. An unsupervised density-based clustering algorithm (HDBSCAN) was applied to predict clusters. We coded a grid search function to reduce the dimensionality of the feature sets and to adjust the hyperparameters of HDBSCAN. We searched the hyperparameter space for the most optimized combination of parameter values based on two selected clustering evaluation measures: the homogeneity and the density-based clustering validation (DBCV) scores. Although both feature sets produced meaningful clusters, AVES feature sets resulted in more solid, homogeneous clusters with relatively lower intra-cluster distances, appearing to be more advantageous for the purpose and dataset of this study. The 26 final clusters we obtained were revised by a bioacoustics expert. We were able to name and describe 10 unique sounds, but only clusters named as ‘Jackhammer’ and ‘Tick’ can be interpreted as biological with certainty. Although unsupervised clustering is conventional in ecological research, we highlight its practical use in revising clusters of annotated unknown sounds. The revised clusters we detailed in this study already define a few groups of distinct and recurring sounds that could serve as a preliminary component of a valid annotated training dataset potentially feeding supervised machine learning and classifier models.

Reduced‐Order Probabilistic Emulation of Physics‐Based Ring Current Models: Application to RAM‐SCB Particle Flux

AbstractIn this work, we address the computational challenge of large‐scale physics‐based simulation models for the ring current. Reduced computational cost allows for significantly faster than real‐time forecasting, enhancing our ability to predict and respond to dynamic changes in the ring current, valuable for space weather monitoring and mitigation efforts. Additionally, it can also be used for a comprehensive investigation of the system. Thus, we aim to create an emulator for the Ring current‐Atmosphere interactions Model with Self‐Consistent magnetic field (RAM‐SCB) particle flux that not only improves efficiency but also facilitates forecasting with reliable estimates of prediction uncertainties. The probabilistic emulator is built upon the methodology developed by Licata and Mehta (2023), https://doi.org/10.1029/2022sw003345. A novel discrete sampling is used to identify 30 simulation periods over 20 years of solar and geomagnetic activity. Focusing on a subset of particle flux, we use Principal Component Analysis for dimensionality reduction and Long Short‐Term Memory (LSTM) neural networks to perform dynamic modeling. Hyperparameter space was explored extensively resulting in about 5% median symmetric accuracy across all data sets for one‐step dynamic prediction. Using a hierarchical ensemble of LSTMs, we have developed a reduced‐order probabilistic emulator (ROPE) tailored for time‐series forecasting of particle flux in the ring current. This ROPE offers accurate predictions of omnidirectional flux at a single energy with no pitch angle information, providing robust predictions on the test set with an error score below 11% and calibration scores under 8% with bias under 2% providing a significant speed up as compared to the full RAM‐SCB run.

Positive Effect of Super-Resolved Structural Magnetic Resonance Imaging for Mild Cognitive Impairment Detection.

This paper presents a novel approach to improving the detection of mild cognitive impairment (MCI) through the use of super-resolved structural magnetic resonance imaging (MRI) and optimized deep learning models. The study introduces enhancements to the perceptual quality of super-resolved 2D structural MRI images using advanced loss functions, modifications to the upscaler part of the generator, and experiments with various discriminators within a generative adversarial training setting. It empirically demonstrates the effectiveness of super-resolution in the MCI detection task, showcasing performance improvements across different state-of-the-art classification models. The paper also addresses the challenge of accurately capturing perceptual image quality, particularly when images contain checkerboard artifacts, and proposes a methodology that incorporates hyperparameter optimization through a Pareto optimal Markov blanket (POMB). This approach systematically explores the hyperparameter space, focusing on reducing overfitting and enhancing model generalizability. The research findings contribute to the field by demonstrating that super-resolution can significantly improve the quality of MRI images for MCI detection, highlighting the importance of choosing an adequate discriminator and the potential of super-resolution as a preprocessing step to boost classification model performance.

A sequence to sequence prediction model for remaining useful life of lithium-ion batteries with Bayesian optimisation process visualization

Accurate battery remaining useful life (RUL) prediction plays an important role in ensuring reliable operation of electric vehicles. In this paper, a hybrid model based on Bayesian optimization of deep convolutional neural network and long short-term memory neural network (BO-DCNN-LSTM) is proposed for battery RUL prediction. Feature extraction of raw charging characteristic curves is performed by the multilayer CNN and preliminary capacity prediction is performed by the multilayer LSTM. The model performance is explored with different training, validation and testing strategies and different prediction starting points. Validation using NASA battery aging data shows that the mean absolute error (MAE) and root mean square error (RMSE) of the RUL prediction are 0.0139 Ah and 0.0195 Ah, respectively, when the prediction starting point is the 50th cycle. In addition, this paper visualizes the process of how the Bayesian optimization (BO) algorithm searches for the global optimal combinations in the high-dimensional hyperparameter space and discusses the impact of these hyperparameters on the prediction, filling the gap in this part of the research.

Developing New Fully Connected Layers for Convolutional Neural Networks with Hyperparameter Optimization for Improved Multi-Label Image Classification

Chest X-ray evaluation is challenging due to its high demand and the complexity of diagnoses. In this study, we propose an optimized deep learning model for the multi-label classification of chest X-ray images. We leverage pretrained convolutional neural networks (CNNs) such as VGG16, ResNet 50, and DenseNet 121, modifying their output layers and fine-tuning the models. We employ a novel optimization strategy using the Hyperband algorithm to efficiently search the hyperparameter space while adjusting the fully connected layers of the CNNs. The effectiveness of our approach is evaluated on the basis of the Area Under the Receiver Operating Characteristic Curve (AUC-ROC) metric. Our proposed methodology could assist in automated chest radiograph interpretation, offering a valuable tool that can be used by clinicians in the future.

Automating hyperparameter optimization in geophysics with Optuna: A comparative study

AbstractDeep learning has gained attraction amongst geophysicists for solving complex longstanding problems. Nevertheless, proper hyperparameter optimization methodologies remain critically underexplored in geophysical deep learning research. This paper attempts to first highlight the importance of hyperparameter optimization and then showcase two geophysics‐related deep learning examples where a grid search and Optuna framework (an automated optimization approach) were used for hyperparameter optimization. We consider two geophysical problems related to denoising seismic traces and the inversion of seismic traces for velocity information. In both cases, models created based on Optuna hyperparameter optimization were able to perform better than those created through grid search. The most significant advantage of Optuna, however, is having quantifiable results to justify the choice of a neural network architecture, depth and other hyperparameters rather than relying on inefficient methods of exploring the hyperparameter space such as a trial‐and‐error or grid search. This study aims to stimulate further exploration and adoption of these frameworks, pushing the boundaries of current deep learning based geophysical problem‐solving methodologies towards full automation.

Prediction of diabetes progress based on machine learning approach

Uropathy is a serious chronic disease whose prevalence is increasing at an alarming rate. Early detection and prediction of diabetes in women is important because of the increased risk of diabetes-related complications during pregnancy. This study introduces machine learning models to assess the likelihood of diabetes in women. The importance of studying characteristics and improving prediction accuracy to understand the nuances of categorization. Specifically, for data preprocessing, experiments are conducted to solve the problem of missing values and outliers by replacing the zero values of certain features with the median values of the corresponding features. This step reduces the impact of less reliable data on model performance. As recognition models, Gaussian Naive Bayes (GNB), Support Vector Machine (SVM), and Random Forest (RF) are built. Performance analysis is performed along with a careful exploration of the hyperparameter space. Scores for Receiver Operating Characteristic - Area Under the Curve (ROC-AUC) are used to compare various models. Different features affect the classification to different degrees. The experimental findings indicate that the modified random forest model demonstrates superior prediction accuracy and robustness. These findings can assist physicians in predicting a patient's risk of developing diabetes earlier.

Transfer learning of hyperparameters for fast construction of anisotropic GPR models: design and application to the machine-learned force field FFLUX.

The polarisable machine-learned force field FFLUX requires pre-trained anisotropic Gaussian process regression (GPR) models of atomic energies and multipole moments to propagate unbiased molecular dynamics simulations. The outcome of FFLUX simulations is highly dependent on the predictive accuracy of the underlying models whose training entails determining the optimal set of model hyperparameters. Unfortunately, traditional direct learning (DL) procedures do not scale well on this task, especially when the hyperparameter search is initiated from a (set of) random guess solution(s). Additionally, the complexity of the hyperparameter space (HS) increases with the number of geometrical input features, at least for anisotropic kernels, making the optimization of hyperparameters even more challenging. In this study, we propose a transfer learning (TL) protocol that accelerates the training process of anisotropic GPR models by facilitating access to promising regions of the HS. The protocol is based on a seeding-relaxation mechanism in which an excellent guess solution is identified by rapidly building one or several small source models over a subset of the target training set before readjusting the previous guess over the entire set. We demonstrate the performance of this protocol by building and assessing the performance of DL and TL models of atomic energies and charges in various conformations of benzene, ethanol, formic acid dimer and the drug fomepizole. Our experiments suggest that TL models can be built one order of magnitude faster while preserving the quality of their DL analogs. Most importantly, when deployed in FFLUX simulations, TL models compete with or even outperform their DL analogs when it comes to performing FFLUX geometry optimization and computing harmonic vibrational modes.

Symbolic Regression on FPGAs for Fast Machine Learning Inference

The high-energy physics community is investigating the potential of deploying machine-learning-based solutions on Field-Programmable Gate Arrays (FPGAs) to enhance physics sensitivity while still meeting data processing time constraints. In this contribution, we introduce a novel end-to-end procedure that utilizes a machine learning technique called symbolic regression (SR). It searches the equation space to discover algebraic relations approximating a dataset. We use PySR (a software to uncover these expressions based on an evolutionary algorithm) and extend the functionality of hls4ml (a package for machine learning inference in FPGAs) to support PySR-generated expressions for resource-constrained production environments. Deep learning models often optimize the top metric by pinning the network size because the vast hyperparameter space prevents an extensive search for neural architecture. Conversely, SR selects a set of models on the Pareto front, which allows for optimizing the performance-resource trade-off directly. By embedding symbolic forms, our implementation can dramatically reduce the computational resources needed to perform critical tasks. We validate our method on a physics benchmark: the multiclass classification of jets produced in simulated proton-proton collisions at the CERN Large Hadron Collider. We show that our approach can approximate a 3-layer neural network using an inference model that achieves up to a 13-fold decrease in execution time, down to 5 ns, while still preserving more than 90% approximation accuracy.