Hyperparameter Search Research Articles

A protocol for trustworthy EEG decoding with neural networks

Deep learning solutions have rapidly emerged for EEG decoding, achieving state-of-the-art performance on a variety of decoding tasks. Despite their high performance, existing solutions do not fully address the challenge posed by the introduction of many hyperparameters, defining data pre-processing, network architecture, network training, and data augmentation. Automatic hyperparameter search is rarely performed and limited to network-related hyperparameters. Moreover, pipelines are highly sensitive to performance fluctuations due to random initialization, hindering their reliability. Here, we design a comprehensive protocol for EEG decoding that explores the hyperparameters characterizing the entire pipeline and that includes multi-seed initialization for providing robust performance estimates. Our protocol is validated on 9 datasets about motor imagery, P300, SSVEP, including 204 participants and 26 recording sessions, and on different deep learning models. We accompany our protocol with extensive experiments on the main aspects influencing it, such as the number of participants used for hyperparameter search, the split into sequential simpler searches (multi-step search), the use of informed vs. non-informed search algorithms, and the number of random seeds for obtaining stable performance. The best protocol included 2-step hyperparameter search via an informed search algorithm, with the final training and evaluation performed using 10 random initializations. The optimal trade-off between performance and computational time was achieved by using a subset of 3-5 participants for hyperparameter search. Our protocol consistently outperformed baseline state-of-the-art pipelines, widely across datasets and models, and could represent a standard approach for neuroscientists for decoding EEG in a trustworthy and reliable way.

Convolutional Neural Networks for Real Time Classification of Beehive Acoustic Patterns on Constrained Devices.

Recent research has demonstrated the effectiveness of convolutional neural networks (CNN) in assessing the health status of bee colonies by classifying acoustic patterns. However, developing a monitoring system using CNNs compared to conventional machine learning models can result in higher computation costs, greater energy demand, and longer inference times. This study examines the potential of CNN architectures in developing a monitoring system based on constrained hardware. The experimentation involved testing ten CNN architectures from the PyTorch and Torchvision libraries on single-board computers: an Nvidia Jetson Nano (NJN), a Raspberry Pi 5 (RPi5), and an Orange Pi 5 (OPi5). The CNN architectures were trained using four datasets containing spectrograms of acoustic samples of different durations (30, 10, 5, or 1 s) to analyze their impact on performance. The hyperparameter search was conducted using the Optuna framework, and the CNN models were validated using k-fold cross-validation. The inference time and power consumption were measured to compare the performance of the CNN models and the SBCs. The aim is to provide a basis for developing a monitoring system for precision applications in apiculture based on constrained devices and CNNs.

Automated fault diagnosis of rotating machinery using sub domain greedy Network Architecture search

Network Architecture Search (NAS) automates hyperparameter adjustments in deep learning models, offering a potent solution for building intelligent fault diagnosis models. Despite its potential, NAS faces challenges in this application. The primary issues include the excessive time required when searching across the full data domain and the inefficacy of the common random search strategy in yielding high-performance sub networks. This study addresses these challenges through several key initiatives: It defines a specific library of hyperparameters tailored for intelligent fault diagnosis tasks; constructs sub data domain model clusters to reduce the computational costs associated with NAS; and introduces a hyperparameter search strategy leveraging a greedy algorithm, which enhances the search efficiency for optimal diagnostic sub network and minimizes its prediction errors. When applied to datasets involving gear and bearing faults, the proposed method demonstrates a reduction in both the time cost of searches and the prediction errors of sub networks, compared to traditional Network Architecture Search. Moreover, it outperforms manually tuned deep learning models by simultaneously optimizing multiple sets of hyperparameter and automatically generating higher accuracy diagnostic sub networks.

A dual-template molecularly imprinted electrochemical sensor assisted with BO-XGBoost algorithm for simultaneous determination of dopamine and acetaminophen

The simultaneous detection of dopamine (DA) and acetaminophen (AP) is crucial for diagnosing and treating related mental disorders. However, accurately determining DA and AP in biological samples remains challenging due to the interference from other biomolecules, such as ascorbic acid (AA), uric acid (UA), epinephrine (EP), etc. Here, we present a dual-template molecularly imprinted electrochemical sensor for the selective recognition of DA and AP. Reduced graphene(rGO)-gold nanoparticles(AuNPs) composite were utilized to enhance the effective area and increase the electron transfer rate of glassy carbon electrode(GCE), thus the electrochemical response signals were amplified. This sensor combined the advantages of nanocomposites and molecularly imprinted polymer (MIP) to achieve highly sensitive and selective detection of DA and AP. However, decreases in the current response of some analytes and variations in relationships of the peak current versus analyte concentration will occur due to adsorption competition on active sites in multianalyte mixtures, rendering the detection range constrained and traditional linear fitting methods inaccurate for predicting analyte concentrations. Therefore, a concentration prediction model based on XGBoost was developed for the sensor to achieve the reliable multi-component determination of DA and AP within the identical measurement range as individual detection. In this model, nine characteristic parameters were extracted from the differential pulse voltammetry (DPV) response curve and Bayesian optimization (BO) was adopted for automatic hyperparameter search, thereby the prediction errors were reduced and the generalization ability was improved. Experiments indicated that the MIP/AuNPs/rGO/GCE exhibited a wide detection range of 2–240 μM and 3–240 μM for DA and AP, with detection limits down to 0.26 μM and 0.33 μM, respectively. In addition, the intelligent MIP/AuNPs/rGO/GCE sensor based on BO-XGBoost model provides more accurate prediction results for untrained concentration combinations obtained from fetal bovine serum samples, indicating that the proposed detection scheme holds potential in future clinical diagnosis.

EAR DISEASE CLASIFICATION USING DEEP LEARNING WITH XCEPTION AND MOBILENET-V2 ARCHITECTURE

Hearing loss is a significant global health problem, with a high prevalence in Indonesia. Limited access to ENT specialists, especially in remote areas, causes delays in diagnosis and treatment of ear diseases. This research aims to develop an early diagnosis system for ear diseases using deep learning. The proposed method applies Xception and MobileNet-V2 Convolutional Neural Network (CNN) architecture with hyperparameter optimization using Bayesian Optimizer. The dataset consists of 1,101 images covering 20 types of ear diseases, collected using an endoscope ear cleaning kit at Mataram University Hospital. The dataset was divided into 60% training data, 20% validation data, and 20% test data. Xception recorded the best performance with accuracy, precision, recall, and f1-score of 0.911, 0.166, 0.166, and 0.151, respectively. The best model performance was obtained on MobileNet-V2 with the application of Bayesian Optimizer, resulting in the best hyperparameters at Unit Dense 174, Dropout Rate 0.2, and LXceptionearning Rate 0.003. This scenario resulted in an increase in accuracy, precision, recall, and f1-score compared to the scenario without hyperparameter search of 0.004, 0.010, 0.018, and 0.012, respectively. This research demonstrates the potential of deep learning in improving early diagnosis of ear diseases.

Enhancing Deep Learning-Based City-Wide Traffic Prediction Pipelines Through Complexity Analysis

Deep learning models can effectively capture the non-linear spatiotemporal dynamics of city-wide traffic forecasting. Evidence of varying deep learning model performance between different cities, different prediction horizons, different scales, specific city regions, and during particular hours of the day abounds in the literature on deep learning-based traffic prediction, yet a unified metric to quantify the complexity of different prediction tasks does not exist. This paper proposes two metrics—model complexity (MC) and intrinsic complexity (IC). While MC quantifies the effective complexity of deep learning models for city-wide traffic prediction tasks, the IC quantifies the underlying complexity of the prediction task. Being an effective complexity metric, MC depends on the model and the data. The IC depends only on the data and is invariant to the model being used. Both metrics are validated through systematic experimentation using traffic volume data from three cities. Finally, we demonstrate how these metrics can improve the workflows for deep learning-based data-driven traffic prediction pipelines and deployment by reducing the hyperparameter search scope and comparing the effectiveness of different design pathways.

Hybrid Quantum or Purely Classical? Assessing the Utility of Quantum Feature Embeddings

Background As graph datasets—including social networks, supply chains, and bioinformatics data—grow in size and complexity, researchers are driven to search for solutions enhancing model efficiency and speed. One avenue that may provide a solution is Quantum Graph Learning (QGL), a subfield of Quantum Machine Learning (QML) that applies machine learning inspired or powered by quantum computing to graph learning tasks. Methods We reevaluate Quantum Feature Embeddings (QFE), a QGL methodology published by Xu et al. earlier this year. QFE uses Variational Quantum Circuits to preprocess node features and then sends them to a classical Graph Neural Network (GNN), with the goal of increasing performance and/or decreasing total model size. Xu et al. evaluated this methodology by comparing its performance with the performance of variously-sized classical models on the benchmark datasets PROTEINS and ENZYMES, and they report success. Our core methodology and learning task remain unchanged. However, we have made several changes to the experimental design that enhance the rigor of the study: 1) we include the testing of models with no embedder; 2) we conduct a thorough hyperparameter search using a state-of-the-art optimization algorithm; and 3) we conduct stratified five-fold cross-validation, which mitigates the bias produced by our small datasets and provides multiple test statistics from which we can calculate a confidence interval. Results We produce classical models that perform comparably to QFE and significantly outperform the small classical models used in Xu et al.’s comparison. Notably, many of our classical models achieve this using fewer parameters than the QFE models we trained. Xu et al. do not report their total model sizes. Conclusion Our study sheds doubt on the efficacy of QFE by demonstrating that small, well-tuned classical models can perform just as well as QFE, highlighting the importance of hyperparameter tuning and rigorous experimental design.

Democratizing cheminformatics: interpretable chemical grouping using an automated KNIME workflow

With the increased availability of chemical data in public databases, innovative techniques and algorithms have emerged for the analysis, exploration, visualization, and extraction of information from these data. One such technique is chemical grouping, where chemicals with common characteristics are categorized into distinct groups based on physicochemical properties, use, biological activity, or a combination. However, existing tools for chemical grouping often require specialized programming skills or the use of commercial software packages. To address these challenges, we developed a user-friendly chemical grouping workflow implemented in KNIME, a free, open-source, low/no-code, data analytics platform. The workflow serves as an all-encompassing tool, expertly incorporating a range of processes such as molecular descriptor calculation, feature selection, dimensionality reduction, hyperparameter search, and supervised and unsupervised machine learning methods, enabling effective chemical grouping and visualization of results. Furthermore, we implemented tools for interpretation, identifying key molecular descriptors for the chemical groups, and using natural language summaries to clarify the rationale behind these groupings. The workflow was designed to run seamlessly in both the KNIME local desktop version and KNIME Server WebPortal as a web application. It incorporates interactive interfaces and guides to assist users in a step-by-step manner. We demonstrate the utility of this workflow through a case study using an eye irritation and corrosion dataset.Scientific contributionsThis work presents a novel, comprehensive chemical grouping workflow in KNIME, enhancing accessibility by integrating a user-friendly graphical interface that eliminates the need for extensive programming skills. This workflow uniquely combines several features such as automated molecular descriptor calculation, feature selection, dimensionality reduction, and machine learning algorithms (both supervised and unsupervised), with hyperparameter optimization to refine chemical grouping accuracy. Moreover, we have introduced an innovative interpretative step and natural language summaries to elucidate the underlying reasons for chemical groupings, significantly advancing the usability of the tool and interpretability of the results.

Computationally Efficient IMplicit Training Strategy for UNrolled NEtworks (IMUNNE): A preliminary analysis using accelerated real-time cardiac cine MRI.

Highly-undersampled, dynamic MRI reconstruction, particularly in multi-coil scenarios, is a challenging inverse problem. Unrolled networks achieve state-of-the-art performance in MRI reconstruction but suffer from long training times and extensive GPU memory cost. In this work, we propose a novel training strategy for IMplicit UNrolled NEtworks (IMUNNE) for highly-undersampled, multi-coil dynamic MRI reconstruction. It formulates the MRI reconstruction problem as an implicit fixed-point equation and leverages gradient approximation for backpropagation, enabling training of deep architectures with fixed memory cost. This study represents the first application of implicit network theory in the context of real-time cine MRI. The proposed method is evaluated using a prospectively undersampled, real-time cine dataset using radial k-space sampling, comprising balanced steady-state free precession (b-SSFP) readouts. Experiments include a hyperparameter search, head-to-head comparisons with a complex U-Net (CU-Net) and an alternating unrolled network (Alt-UN), and an analysis of robustness under noise perturbations; peak signal-to-noise ratio, structural similarity index, normalized root mean-square error, spatio-temporal entropic difference, and a blur metric were used. IMUNNE produced significantly and slightly better image quality compared to CU-Net and Alt-UN, respectively. Compared with Alt-UN, IMUNNE significantly reduced training and inference times, making it a promising approach for highly-accelerated, multi-coil real-time cine MRI reconstruction. IMUNNE strategy successfully applies unrolled networks to image reconstruction of highly-accelerated, real-time radial cine MRI. Implicit training enables rapid, high-quality, and cost-effective CMR exams by reducing training and inference times and lowering memory cost associated with advanced reconstruction methods.

Hga-lstm: LSTM architecture and hyperparameter search by hybrid GA for air pollution prediction

Hga-lstm: LSTM architecture and hyperparameter search by hybrid GA for air pollution prediction

Neural Network Models for Ionospheric Electron Density Prediction at a Fixed Altitude Using Neural Architecture Search

AbstractSpecification and forecast of ionospheric parameters, such as ionospheric electron density (Ne), have been an important topic in space weather and ionospheric research. Neural networks (NNs) emerge as a powerful modeling tool for Ne prediction. However, heavy manual adjustments are time consuming to determine the optimal NN structures. In this work, we propose to use neural architecture search (NAS), an automatic machine learning method, to mitigate this problem. NAS aims to find the optimal network structure through the alternate optimization of the hyperparameters and the corresponding network parameters within a pre‐defined hyperparameter search space. A total of 16‐year data from Millstone Hill incoherent scatter radar (ISR) are used for the NN models. One single‐layer NN (SLNN) model and one deep NN (DNN) model are both trained with NAS, namely SLNN‐NAS and DNN‐NAS, for Ne prediction and compared with their manually tuned counterparts (SLNN and DNN) based on previous studies. Our results show that SLNN‐NAS and DNN‐NAS outperformed SLNN and DNN, respectively. These NN predictions of Ne daily variation patterns reveal a 27‐day mid‐latitude topside Ne variation, which cannot be reasonably represented by traditional empirical models developed using monthly averages. DNN‐NAS yields the best prediction accuracy measured by quantitative metrics and rankings of daily pattern prediction, especially with an improvement in mean absolute error more than 10% compared to the SLNN model. The limited improvement of NAS is likely due to the network complexity and the limitation of fully connected NN without the time histories of input parameters.

AutoCTS++: zero-shot joint neural architecture and hyperparameter search for correlated time series forecasting

AutoCTS++: zero-shot joint neural architecture and hyperparameter search for correlated time series forecasting

Cross-Modal Retrieval-enhanced code Summarization based on joint learning for retrieval and generation

Context:Code summarization refers to a task that automatically generates a natural language description of a code snippet to facilitate code comprehension. Existing methods have achieved satisfactory results by incorporating information retrieval into generative deep-learning models for reusing summaries of existing code. However, most of these existing methods employed non-learnable generic retrieval methods for content-based retrieval, resulting in a lack of diversity in the retrieved results during training, thereby making the model over-reliant on retrieved results and reducing the generative model’s ability to generalize to unknown samples. Objective:To address this issue, this paper introduces CMR-Sum: a novel Cross-Modal Retrieval-enhanced code Summarization framework based on joint learning for generation and retrieval tasks, where both two tasks are allowed to be optimized simultaneously. Method:Specifically, we use a cross-modal retrieval module to dynamically alter retrieval results during training, which enhances the diversity of the retrieved results and maintains a relative balance between the two tasks. Furthermore, in the summary generation phase, we employ a cross-attention mechanism to generate code summaries based on the alignment between retrieved and generated summaries. We conducted experiments on three real-world datasets, comparing the performance of our method with baseline models. Additionally, we performed extensive qualitative analysis. Result:Results from qualitative and quantitative experiments indicate that our approach effectively enhances the performance of code summarization. Our method outperforms both the generation-based and the retrieval-enhanced baselines. Further ablation experiments demonstrate the effectiveness of each component of our method. Results from sensitivity analysis experiments suggest that our approach achieves good performance without requiring extensive hyper-parameter search. Conclusion:The direction of utilizing retrieval-enhanced generation tasks shows great potential. It is essential to increase the diversity of retrieval results during the training process, which is crucial for improving the generality and the performance of the model.

A new risk level identification model for aviation safety

The aviation safety reporting system (ASRS) data, which are accident investigation reports issued by the national transportation safety board (NTSB), are often used for aviation risk level identification modeling. However, ASRS data suffer from sample imbalance and inaccurate encoding of textual information. To address these issues, this paper proposes a novel risk identification model. Initially, the pre-trained sentence bidirectional encoder representations from transformers (SBERT) model is fine-tuned using the textual information in the ASRS data, improving the accuracy of the text encoding. Subsequently, a generative adversarial network (GAN) is employed to generate samples for underrepresented categories in the ASRS data, thereby addressing the sample imbalance problem. Finally, a bayesian optimization algorithm is integrated to automatically search for the hyperparameters of the risk identification model, further enhancing the model's performance in risk level identification. Experimental results demonstrate that not only does the SBERT fine-tuning substantially improve the accuracy of text information encoding, but both data augmentation and automatic hyperparameter search also significantly contribute to the performance improvement of the risk level recognition model.

Kernel Bayesian logistic tensor decomposition with automatic rank determination for predicting multiple types of miRNA-disease associations.

Identifying the association and corresponding types of miRNAs and diseases is crucial for studying the molecular mechanisms of disease-related miRNAs. Compared to traditional biological experiments, computational models can not only save time and reduce costs, but also discover potential associations on a large scale. Although some computational models based on tensor decomposition have been proposed, these models usually require manual specification of numerous hyperparameters, leading to a decrease in computational efficiency and generalization ability. Additionally, these linear models struggle to analyze complex, higher-order nonlinear relationships. Based on this, we propose a novel framework, KBLTDARD, to identify potential multiple types of miRNA-disease associations. Firstly, KBLTDARD extracts information from biological networks and high-order association network, and then fuses them to obtain more precise similarities of miRNAs (diseases). Secondly, we combine logistic tensor decomposition and Bayesian methods to achieve automatic hyperparameter search by introducing sparse-induced priors of multiple latent variables, and incorporate auxiliary information to improve prediction capabilities. Finally, an efficient deterministic Bayesian inference algorithm is developed to ensure computational efficiency. Experimental results on two benchmark datasets show that KBLTDARD has better Top-1 precision, Top-1 recall, and Top-1 F1 for new type predictions, and higher AUPR, AUC, and F1 values for new triplet predictions, compared to other state-of-the-art methods. Furthermore, case studies demonstrate the efficiency of KBLTDARD in predicting multiple types of miRNA-disease associations.

ВИКОРИСТАННЯ ШТУЧНОГО ІНТЕЛЕКТУ ДЛЯ РОЗПІЗНАВАННЯ ТЕРОРИСТИЧНИХ ТА ВОРОЖИХ ВІЙСЬКОВИХ ОБ’ЄКТІВ

The development of artificial intelligence technologies and their application in solving military tasks is the main concept of scientific advancement in Ukraine. Despite the availability of existing solutions for classifying limited types of civilian objects, there is a need to develop new technologies to gain an advantage over the enemy. Existing artificial intelligence models available in the public domain are oriented towards diagnosing restricted types of civilian objects. This limitation prevents the use of these models for analyzing military enemy equipment. Automating the process of recognizing enemy military objects by constructing a deep learning classification model. The process of building a classification model for enemy military equipment using deep neural networks is considered. The Sequential model type is chosen as the basis, which includes batch normalization layers added after each convolutional layer before further processing. Accelerated hyperparameter search is provided by the Random Search tool. The quality of classification models was measured using classical accuracy metrics. The implementation of the proposed solutions is carried out using the Python programming language and the Python Imaging Library, TensorFlow, Keras Tuner, and Matplotlib libraries. A convolutional neural network classification model has been constructed capable of classifying enemy military tanks and vehicles. The model demonstrates the best accuracy metrics on training/testing datasets of 0.6/0.55 with errors of 0.658/0.705, respectively. A comparative analysis of classification models with different types of architectures showed the advantage of the model in terms of accuracy and error rate when using an architecture that includes batch normalization layers. The created classification model is recommended for further training on new data or for solving computer vision tasks.

Prediction of main engine power of oil tankers using artificial intelligence algorithms

In the preliminary ship design, the accurate determination of a vessel’s main engine power is one of the most critical aspects next to service speed, main particulars, and cargo capacity. However, this task can be quite intricate due to its reliance on an extremely great number of influencing factors. In the research that is presented in this paper dataset of 357 oil tankers was gathered and developed to research the idea in which genetic programming is applied to the mentioned dataset to obtain mathematical equations (MEs) that can estimate the ship’s main engine power with high accuracy. The highest estimation accuracy of MEs is achieved by tuning the GP hyperparameter values through the random hyperparameter search (RHS) method. The initial dataset was divided into train and test datasets in a 70:30 ratio. The train dataset was used to train GP in a 5-fold cross-validation process and after the process was done the obtained MEs were evaluated on the test dataset. To evaluate the GP training testing process several evaluation metrics were used i.e., coefficients of determination (R2), mean absolute error (MAE), root mean square error (RMSE), and length of obtained MEs. The conducted investigation showed that GP generated MEs that can estimate ship main engine power with high accuracy.

Predicting recurrent chat contact in a psychological intervention for the youth using natural language processing.

Chat-based counseling hotlines emerged as a promising low-threshold intervention for youth mental health. However, despite the resulting availability of large text corpora, little work has investigated Natural Language Processing (NLP) applications within this setting. Therefore, this preregistered approach (OSF: XA4PN) utilizes a sample of approximately 19,000 children and young adults that received a chat consultation from a 24/7 crisis service in Germany. Around 800,000 messages were used to predict whether chatters would contact the service again, as this would allow the provision of or redirection to additional treatment. We trained an XGBoost Classifier on the words of the anonymized conversations, using repeated cross-validation and bayesian optimization for hyperparameter search. The best model was able to achieve an AUROC score of 0.68 (p < 0.01) on the previously unseen 3942 newest consultations. A shapely-based explainability approach revealed that words indicating younger age or female gender and terms related to self-harm and suicidal thoughts were associated with a higher chance of recontacting. We conclude that NLP-based predictions of recurrent contact are a promising path toward personalized care at chat hotlines.

SpyderZ: An Efficient Support Vector Machine Library for Photometric Redshift Estimation and Redshift Probability Information

We present SpyderZ, a Python-based library for photometric redshift estimation using support vector machines (implemented with scikit-learn). Our approach discretizes redshift values into uniformly-sized bins and uses one-vs-one support vector classifiers with voting strategies to produce effective probability density functions (ePDFs) over redshift for each galaxy. These ePDFs, which are not constrained to be Gaussian or any other shape, allow for our model's predictions to be used quantitatively with uncertainty analysis methods, and have been shown to enable reliable catastrophic outlier detection. Adapted from the previous IDL package SpiderZ, SpyderZ offers training and evaluation speed optimizations on the order of 102, along with support for parallelization across CPU cores. Our library also offers in-built data sanity checks, result visualizations, metric calculations, cross validation, batch evaluations, and parallelized hyperparameter search (grid search and random search).

A novel approach for asphalt pavement invisible distress classification prediction by machine learning algorithms

ABSTRACT The detection and identification of road invisible distress (ID) has always been a key issue of concern for pavement maintenance and rehabilitation (M&R) optimisation, especially for asphalt pavement with semi-rigid base. This study is to build a supervised machine learning model based on the detected characteristic variable data of the pavement structure strength, roughness, cracking, permeability, traffic volume, etc. It is to fit the model with class labelled data on the training set and to train an algorithm model that can intelligently detect the degree of ID. In addition, K-fold cross-validation and hyperparameter search are employed to find the optimal machine learning suitable for road ID category prediction. The results show that Gradient Boosting machine (GBM) model, LightGBM model and Random Forest (RF) model are marketed out from the other six models, for their classification performance of the models is the best, with statistical indicators up to 96%. Additionally, the LightGBM model achieves the best results with shortest computation time and 99% the accuracy of the test set. Furthermore, it also indicates that the three main factors affecting the ID are permeability, alligator cracking and equivalent single axle load (ESAL).