Training Set Size Research Articles

A Multilabel Active Learning Framework for Microcontroller Performance Screening

In safety-critical applications, microcontrollers have to be tested to satisfy strict quality and performances constraints. It has been demonstrated that on-chip ring oscillators can be used as speed monitors to reliably predict the performances. However, any machine-learning model is likely to be inaccurate if trained on an inadequate dataset, and labeling data for training is quite a costly process. In this paper, we present a methodology based on active learning to select the best samples to be included in the training set, significantly reducing the time and cost required. Moreover, since different speed measurements are available, we designed a multi-label technique to take advantage of their correlations. Experimental results demonstrate that the approach halves the training-set size, with respect to a random-labelling, while it increases the predictive accuracy, with respect to standard single-label machine-learning models.

LEAF: Navigating Concept Drift in Cellular Networks

Operational networks commonly rely on machine learning models for many tasks, including detecting anomalies, inferring application performance, and forecasting demand. Yet, model accuracy can degrade due to concept drift, whereby the relationship between the features and the target to be predicted changes. Mitigating concept drift is an essential part of operationalizing machine learning models in general, but is of particular importance in networking's highly dynamic deployment environments. In this paper, we first characterize concept drift in a large cellular network for a major metropolitan area in the United States. We find that concept drift occurs across many important key performance indicators (KPIs), independently of the model, training set size, and time interval---thus necessitating practical approaches to detect, explain, and mitigate it. We then show that frequent model retraining with newly available data is not sufficient to mitigate concept drift, and can even degrade model accuracy further. Finally, we develop a new methodology for concept drift mitigation, Local Error Approximation of Features (LEAF). LEAF works by detecting drift; explaining the features and time intervals that contribute the most to drift; and mitigates it using forgetting and over-sampling. We evaluate LEAF against industry-standard mitigation approaches (notably, periodic retraining) with more than four years of cellular KPI data. Our initial tests with a major cellular provider in the US show that LEAF consistently outperforms periodic and triggered retraining on complex, real-world data while reducing costly retraining operations.

A hybrid finite element/neural network solver and its application to the Poisson problem

AbstractWe analyze a hybrid method that enriches coarse grid finite element solutions with fine scale fluctuations obtained from a neural network. The idea stems from the Deep Neural Network Multigrid Solver (DNN‐MG) which embeds a neural network into a multigrid hierarchy by solving coarse grid levels directly and predicting the corrections on fine grid levels locally (e.g., on small patches that consist of several cells) by a neural network. Such local designs are quite appealing, as they allow a very good generalizability. In this work, we formalize the method and describe main components of the a‐priori error analysis. Moreover, we numerically investigate how the size of training set affects the solution quality.

A Machine Learning Based Reliability Analysis of Negative Bias Temperature Instability (NBTI) Compliant Design for Ultra Large Scale Digital Integrated Circuit

NBTI is a key reliability challenge in nanoscale digital design, and it is vital to address it throughout the exploration of design space at high levels of abstraction in order to improve reliability. A prediction model of aging that is adequate for these levels ought to be faster. In addition to this, the model should be able to forecast the recently discovered stochastic consequences of growing older. The purpose of this study is to offer a model that is based on machine learning (ML) and can predict aging effects. After obtaining a training set of sufficient size using Synopsis HSPICE (MOSFET Reliability, MOSRA) in the beginning, the machine-learning-based model is then trained and built in order to forecast the aging statistical features. Evaluation is done on a number of machine learning techniques, including Adaptive Neuro-Fuzzy Inference System (ANFIS), K-Nearest Neighbor (KNN), Support Vector Machine (SVM) and Random Forest (RF). The findings indicate that ANFIS algorithms are very effective in the process of age prediction. The proposed technique shows that the aging prediction runtime is reduced by more than 99% when compared to the MOSRA-based approach, and accurate predictions of the statistical properties of aging are obtained with an accuracy of more than 99% on complementary metal oxide semiconductor (CMOS) and metal gate/high-K (MGK) circuits at the 22nm technology node.

A prediction model for blood-brain barrier penetrating peptides based on masked peptide transformers with dynamic routing.

Blood-brain barrier penetrating peptides (BBBPs) are short peptide sequences that possess the ability to traverse the selective blood-brain interface, making them valuable drug candidates or carriers for various payloads. However, the in vivo or in vitro validation of BBBPs is resource-intensive and time-consuming, driving the need for accurate in silico prediction methods. Unfortunately, the scarcity of experimentally validated BBBPs hinders the efficacy of current machine-learning approaches in generating reliable predictions. In this paper, we present DeepB3P3, a novel framework for BBBPs prediction. Our contribution encompasses four key aspects. Firstly, we propose a novel deep learning model consisting of a transformer encoder layer, a convolutional network backbone, and a capsule network classification head. This integrated architecture effectively learns representative features from peptide sequences. Secondly, we introduce masked peptides as a powerful data augmentation technique to compensate for small training set sizes in BBBP prediction. Thirdly, we develop a novel threshold-tuning method to handle imbalanced data by approximating the optimal decision threshold using the training set. Lastly, DeepB3P3 provides an accurate estimation of the uncertainty level associated with each prediction. Through extensive experiments, we demonstrate that DeepB3P3 achieves state-of-the-art accuracy of up to 98.31% on a benchmarking dataset, solidifying its potential as a promising computational tool for the prediction and discovery of BBBPs.

Block sparsity promoting algorithm for efficient construction of cluster expansion models for multicomponent alloys

Multicomponent alloys are gaining significance as drivers of technological breakthroughs especially in structural and energy storage materials. The vast configuration space of these materials prohibit computational modeling using first-principles based methods alone. The cluster expansion (CE) method is the most widely used tool for modeling configurational disorder in alloys. CE relies on machine learning algorithms to train Hamiltonians and uses first-principles calculated data as training sets. In this paper we present a new compressive sensing-based algorithm for the efficient construction of CE Hamiltonians of multicomponent alloys. Our algorithm constructs highly sparse and physically reasonable models from a carefully selected small training set of alloy structures. Compared to conventional fitting algorithms, the algorithm achieves more than 50% reduction in the training set size. The resultant sparse models can sample the configuration space at least 3 × faster. We demonstrate this algorithm on 4 different alloy systems, namely Ag–Au, Ag–Au–Cu, Ag–Au–Cu–Pd and (Ge,Sn)(S,Se,Te).The sparse CE models for these alloys can rapidly reproduce known ground state orderings and order-disorder transitions. Our method can truly enable high-throughput multicomponent alloy thermodynamics by reducing the cost associated with model construction and configuration sampling.

A frequency domain-based machine learning architecture for short-term wave height forecasting

Accurate and reliable short-term waves forecasting plays a crucial role in ensuring the safety of offshore operations. However, the non-stationary and high-noise characteristics of wave height time series in real-world environments present significant challenges for prediction. This paper introduces a novel lightweight machine learning forecasting architecture, named FreMixer, that leverages frequency domain information. Specifically designed for sequences with a low signal-to-noise ratio (SNR), FreMixer utilizes a Gaussian filter block for noise reduction and a Fourier block for frequency domain feature extraction. The Mixer block, which incorporates skip-connections, serves as the key component for feature learning. Towing tank experiments demonstrate the superior performance of FreMixer compared to benchmark models such as the autoregressive moving average (ARMA), long short-term memory (LSTM), and extreme gradient boosting (XGBoost) models. Due to its simple structure, FreMixer offers low deployment costs and high training efficiency, making it an attractive option for practical applications. Furthermore, the impact of training set size and historical window size on the performance is discussed.

Data-Driven Prediction of Configurational Stability of Molecule-Adsorbed Heterogeneous Catalysts.

The design of new heterogeneous catalysts that convert small molecules into valuable chemicals is a key challenge for constructing sustainable energy systems. Density functional theory (DFT)-based design frameworks based on the understanding of molecular adsorption on the catalytic surface have been widely proposed to accelerate experimental approaches to develop novel catalysts. In addition, a machine learning (ML)-combined design framework was recently proposed to further reduce the inherent time cost of DFT-based frameworks. However, because of the lack of prior information on chemical interactions between arbitrary surfaces and adsorbates, the efficacy of the computational screening approaches would be reduced by obtaining unexpected structural anomalies (i.e., abnormally converged surface-adsorbate geometries after the DFT calculations) during an exhaustive exploration of chemical space. To overcome this challenge, we propose an ML framework that directly predicts the configurational stability of a given initial surface-adsorbate geometry. Our benchmark experiments with the Open Catalysts 20 (OC20) dataset show promising performance on classifying stable geometry (i.e., F1-score of 0.922, the area under the receiver operating characteristics (AUROC) of 0.906, and Matthews correlation coefficient (MCC) of 0.633) with a high precision of 0.921 by utilizing an ensemble approach. We further interpret the generalizability and domain applicability of the trained model in terms of the chemical space of the OC20 dataset. Furthermore, from an experiment on the training set size dependence of model performance, we found that our ML model could be practically applicable to classify stable configurations even with a relatively small number of training data.

Can diffusion model conditionally generate astrophysical images?

ABSTRACT Generative adversarial networks (GANs) are frequently utilized in astronomy to construct an emulator of numerical simulations. Nevertheless, training GANs can prove to be a precarious task, as they are prone to instability and often lead to mode collapse problems. Conversely, the diffusion model also has the ability to generate high-quality data without adversarial training. It has shown superiority over GANs with regard to several natural image data sets. In this study, we undertake a quantitative comparison between the denoising diffusion probabilistic model (DDPM) and StyleGAN2 (one of the most robust types of GANs) via a set of robust summary statistics from scattering transform. In particular, we utilize both models to generate the images of 21 cm brightness temperature mapping, as a case study, conditionally based on astrophysical parameters that govern the process of cosmic reionization. Using our new Fréchet Scattering Distance (FSD) as the evaluation metric to quantitatively compare the sample distribution between generative models and simulations, we demonstrate that DDPM outperforms StyleGAN2 on varied sizes of training sets. Through Fisher forecasts, we demonstrate that on our data sets, StyleGAN2 exhibits mode collapses in varied ways, while DDPM yields a more robust generation. We also explore the role of classifier-free guidance in DDPM and show the preference for a non-zero guidance scale only when the training data is limited. Our findings indicate that the diffusion model presents a promising alternative to GANs in the generation of accurate images. These images can subsequently provide reliable parameter constraints, particularly in the realm of astrophysics.

A supervised approach for the detection of AM-FM signals’ interference regions in spectrogram images

Ridge curves retrieval in time–frequency (TF) domains is fundamental in many application fields as they convey most of information concerning the instantaneous frequencies of non-stationary signals. However, it represents a very hard task in the case of multicomponent signals having non-separable modes as they generate interference in TF domains. A preliminary detection of these interference regions may be then useful for the definition of effective strategies for ridge curve recovery. This paper introduces SIID-CNN (Spectrogram Image Interference Detection via CNN), that is a novel approach based on machine learning for the automatic detection of interference regions in spectrogram images. Each spectrogram sample is suitably classified as interference, single mode or background by accounting for its relative information. Some critical problems, such as the training set size and the type of examples to use for populating the training set, are dealt with. Experimental results show that a local linear model for spectrogram image and a small training set can guarantee good classification rates for a wide class of non-stationary signals, even in the presence of moderate noise.

Optimizing the architecture of Behler-Parrinello neural network potentials.

The architecture of neural network potentials is typically optimized at the beginning of the training process and remains unchanged throughout. Here, we investigate the accuracy of Behler-Parrinello neural network potentials for varying training set sizes. Using the QM9 and 3BPA datasets, we show that adjusting the network architecture according to the training set size improves the accuracy significantly. We demonstrate that both an insufficient and an excessive number of fitting parameters can have a detrimental impact on the accuracy of the neural network potential. Furthermore, we investigate the influences of descriptor complexity, neural network depth, and activation function on the model's performance. We find that for the neural network potentials studied here, two hidden layers yield the best accuracy and that unbounded activation functions outperform bounded ones.

Robust Audio Copy-Move Forgery Detection Using Constant Q Spectral Sketches and GA-SVM

Audio recordings used as evidence have become increasingly important to litigation. Before their admissibility as evidence, an audio forensic expert is often required to help determine whether the submitted audio recordings are altered or authentic. Within this field, the copy-move forgery detection (CMFD), which focuses on finding possible forgeries that are derived from the same audio recording, has been an urgent problem in blind audio forensics. However, most of the existing methods require idealistic pre-segmentation and artificial threshold selection to calculate the similarity between segments, which may result in serious misleading and misjudgment especially on high frequency words. In this work, we present a robust method for detecting and locating an audio copy-move forgery on the basis of constant Q spectral sketches (CQSS) and the integration of a customised genetic algorithm (GA) and support vector machine (SVM). Specifically, the CQSS features are first extracted by averaging the logarithm of the squared-magnitude constant Q transform. Then, the CQSS feature set is automatically optimised by a customised GA combined with SVM to obtain the best feature subset and classification model at the same time. Finally, the integrated method, named CQSS-GA-SVM, is evaluated against the state-of-the-art approaches to blind detection of copy-move forgeries on real-world copy-move datasets with read English and Chinese corpus, respectively. The experimental results demonstrate that the proposed CQSS-GA-SVM exhibits significantly high robustness against post-processing based anti-forensics attacks and adaptability to the changes of the duplicated segment duration, the training set size, the recording length, and the forgery type, which may be beneficial to improving the work efficiency of audio forensic experts.

Geographical origin identification of ginseng using near-infrared spectroscopy coupled with subspace-based ensemble classifiers

Ginseng is a well-known traditional herbal medicine and the ginseng available on the market may not actually be produced in a certain place as claimed. Traditional methods of identifying the geographical origin of Ginseng are subjective, time-consuming or destructive. A more efficient approach is desirable. The feasibility of combining near-infrared (NIR) spectroscopy with ensemble learning for discriminating ginseng producing area was explored. A total of 270 samples were collected and evenly partitioned into the training and test sets. Random subspace ensemble (RSE) that uses linear discriminant classifier (LDA) as weak learner (abbreviated RSE-LDA) was used to construct predictive models. Two parameters including the size of subspace and the number of learners in ensemble were optimized. Classic partial least algorithm (PLS) was applied to build the reference model. The sensitivity, specificity, and total accuracy of final RSE-LDA and PLS models were 97.8 %, 100 %, 99.3 %, and 93.3 %, 96.7 %, 95.6 %, respectively. In order to study the impact of training set composition on the results, the samples were randomly divided 200 times and the algorithm was run repeatedly to statistically analyze the sensitivity and specificity on the test set. Similar results were obtained. The effect of training set size was also investigated. It indicates that the combination of NIR spectroscopy with the RSE algorithm is a potential tool of discriminating the origin of Ginseng.

Homogenizing SOHO/EIT and SDO/AIA 171 Å Images: A Deep-learning Approach

Extreme-ultraviolet (EUV) images of the Sun are becoming an integral part of space weather prediction tasks. However, having different surveys requires the development of instrument-specific prediction algorithms. As an alternative, it is possible to combine multiple surveys to create a homogeneous data set. In this study, we utilize the temporal overlap of Solar and Heliospheric Observatory Extreme ultraviolet Imaging Telescope and Solar Dynamics Observatory Atmospheric Imaging Assembly 171 Å surveys to train an ensemble of deep-learning models for creating a single homogeneous survey of EUV images for two solar cycles. Prior applications of deep learning have focused on validating the homogeneity of the output while overlooking the systematic estimation of uncertainty. We use an approach called “approximate Bayesian ensembling” to generate an ensemble of models whose uncertainty mimics that of a fully Bayesian neural network at a fraction of the cost. We find that ensemble uncertainty goes down as the training set size increases. Additionally, we show that the model ensemble adds immense value to the prediction by showing higher uncertainty in test data that are not well represented in the training data.

Improving short text classification with augmented data using GPT-3

Abstract GPT-3 is a large-scale natural language model developed by OpenAI that can perform many different tasks, including topic classification. Although researchers claim that it requires only a small number of in-context examples to learn a task, in practice GPT-3 requires these training examples to be either of exceptional quality or a higher quantity than easily created by hand. To address this issue, this study teaches GPT-3 to classify whether a question is related to data science by augmenting a small training set with additional examples generated by GPT-3 itself. This study compares two augmented classifiers: the Classification Endpoint with an increased training set size and the Completion Endpoint with an augmented prompt optimized using a genetic algorithm. We find that data augmentation significantly increases the accuracy of both classifiers, and that the embedding-based Classification Endpoint achieves the best accuracy of about 76%, compared to human accuracy of 85%. In this way, giving large language models like GPT-3 the ability to propose their own training examples can improve short text classification performance.

Using pseudo-labeling to improve performance of deep neural networks for animal identification

Contemporary approaches for animal identification use deep learning techniques to recognize coat color patterns and identify individual animals in a herd. However, deep learning algorithms usually require a large number of labeled images to achieve satisfactory performance, which creates the need to manually label all images when automated methods are not available. In this study, we evaluated the potential of a semi-supervised learning technique called pseudo-labeling to improve the predictive performance of deep neural networks trained to identify Holstein cows using labeled training sets of varied sizes and a larger unlabeled dataset. By using such technique to automatically label previously unlabeled images, we observed an increase in accuracy of up to 20.4 percentage points compared to using only manually labeled images for training. Our final best model achieved an accuracy of 92.7% on an independent testing set to correctly identify individuals in a herd of 59 cows. These results indicate that it is possible to achieve better performing deep neural networks by using images that are automatically labeled based on a small dataset of manually labeled images using a relatively simple technique. Such strategy can save time and resources that would otherwise be used for labeling, and leverage well annotated small datasets.

Comparative evaluation of uncertainty estimation and decomposition methods on liver segmentation

PurposeDeep neural networks need to be able to indicate error likelihood via reliable estimates of their predictive uncertainty when used in high-risk scenarios, such as medical decision support. This work contributes a systematic overview of state-of-the-art approaches for decomposing predictive uncertainty into aleatoric and epistemic components, and a comprehensive comparison for Bayesian neural networks (BNNs) between mutual information decomposition and the explicit modelling of both uncertainty types via an additional loss-attenuating neuron.MethodsExperiments are performed in the context of liver segmentation in CT scans. The quality of the uncertainty decomposition in the resulting uncertainty maps is qualitatively evaluated, and quantitative behaviour of decomposed uncertainties is systematically compared for different experiment settings with varying training set sizes, label noise, and distribution shifts.ResultsOur results show the mutual information decomposition to robustly yield meaningful aleatoric and epistemic uncertainty estimates, while the activation of the loss-attenuating neuron appears noisier with non-trivial convergence properties. We found that the addition of a heteroscedastic neuron does not significantly improve segmentation performance or calibration, while slightly improving the quality of uncertainty estimates.ConclusionsMutual information decomposition is simple to implement, has mathematically pleasing properties, and yields meaningful uncertainty estimates that behave as expected under controlled changes to our data set. The additional extension of BNNs with loss-attenuating neurons provides no improvement in terms of segmentation performance or calibration in our setting, but marginal benefits regarding the quality of decomposed uncertainties.

Adversarial Coreset Selection for Efficient Robust Training

It has been shown that neural networks are vulnerable to adversarial attacks: adding well-crafted, imperceptible perturbations to their input can modify their output. Adversarial training is one of the most effective approaches to training robust models against such attacks. Unfortunately, this method is much slower than vanilla training of neural networks since it needs to construct adversarial examples for the entire training data at every iteration. By leveraging the theory of coreset selection, we show how selecting a small subset of training data provides a principled approach to reducing the time complexity of robust training. To this end, we first provide convergence guarantees for adversarial coreset selection. In particular, we show that the convergence bound is directly related to how well our coresets can approximate the gradient computed over the entire training data. Motivated by our theoretical analysis, we propose using this gradient approximation error as our adversarial coreset selection objective to reduce the training set size effectively. Once built, we run adversarial training over this subset of the training data. Unlike existing methods, our approach can be adapted to a wide variety of training objectives, including TRADES, ell _p-PGD, and Perceptual Adversarial Training. We conduct extensive experiments to demonstrate that our approach speeds up adversarial training by 2–3 times while experiencing a slight degradation in the clean and robust accuracy.

Modelling inland Arctic bathymetry from space using cloud-based machine learning and Sentinel-2

Lakes and ponds are extensive features throughout the circumpolar region, spanning a broad range of environmental conditions which controls their hydro-ecological processes and spatiotemporal distribution. The physical characteristics of these freshwater ecosystems, including their extent and depth, are particularly responsive to climatic conditions. Thus, having the ability to efficiently map and monitor these elements is crucial as the climate continues to warm, especially over large spatial extents. In this study, satellite derived bathymetry (SDB) methods were implemented to model regional inland Arctic water depths within the open-source and cloud-based Google Earth Engine (GEE) platform. High-resolution (10 m) spectral reflectance data from Sentinel-2 was used as covariates in non-parametric and non-linear machine learning (ML) models, namely random forest (RF), support vector regression (SVR), and classification and regression trees (CART). These three ML models were also compared to a more classical and parametric multiple linear regression (MLR) model. All models were calibrated using in situ bathymetric data collected near the Toolik Field Station, in the Alaskan Arctic tundra. With such a large sample repository available, the effects of training set size, noise, and multicollinearity on model performance was comprehensively investigated. Results clearly demonstrated the superior efficacy and robustness of the RF algorithm for predicting water depths, achieving a best coefficient of determination (R2) of 0.74, mean absolute error (MAE) of 2.12 m, and root mean square error (RMSE) of 3.04 m. Overall, this study highlights the potential of advanced and flexible ML algorithms within the GEE, and demonstrates the capabilities of the polar-orbiting Copernicus Sentinel-2 satellite for this Arctic application. In the future, this ML-driven methodology can be applied using GEE’s cloud infrastructure to produce updated, cost-effective, and accurate bathymetry maps over large expanses of the Arctic tundra.

A neural-mechanistic hybrid approach improving the predictive power of genome-scale metabolic models

Constraint-based metabolic models have been used for decades to predict the phenotype of microorganisms in different environments. However, quantitative predictions are limited unless labor-intensive measurements of media uptake fluxes are performed. We show how hybrid neural-mechanistic models can serve as an architecture for machine learning providing a way to improve phenotype predictions. We illustrate our hybrid models with growth rate predictions of Escherichia coli and Pseudomonas putida grown in different media and with phenotype predictions of gene knocked-out Escherichia coli mutants. Our neural-mechanistic models systematically outperform constraint-based models and require training set sizes orders of magnitude smaller than classical machine learning methods. Our hybrid approach opens a doorway to enhancing constraint-based modeling: instead of constraining mechanistic models with additional experimental measurements, our hybrid models grasp the power of machine learning while fulfilling mechanistic constrains, thus saving time and resources in typical systems biology or biological engineering projects.