Small Training Set Research Articles

Convolutional neural network for classifying cartoon images augmented by DCGAN

Convolutional Neural Network (CNN) tend to have better results on large data sets and poor performance on small data sets, so the data augmentation is crucial for a CNN to get better performance based on the dataset with limited size. In this paper, Deep Convolution Generative Adversarial Network (DCGAN) was used to augment data to make the AlexNet perform better on an image classification task with small data sets. AlexNet was trained on a small anime face training set with only 160 samples to determine whether the anime face was male or female, and then tested its accuracy on a test set with 240 sample. Then, a pre-trained DCGAN was transferred to train on the male and female training sets respectively. And 2 DCGANs were obtained, one could generate male cartoon faces and another could generate female cartoon faces. The images generated by DCGANs were put in train set, which was used to train AlexNet again and the result was recorded. Other data augmentation methods such as cutout, cutmix and Noise Injection were compared as well. Finally, it is found that AlexNet has the best performance when using the DCGAN augmentation method, which can significantly improve the verification accuracy of the model.

Fine-grained Species Recognition with Privileged Pooling: Better Sample Efficiency Through Supervised Attention.

We propose a scheme for supervised image classification that uses privileged information, in the form of keypoint annotations for the training data, to learn strong models from small and/or biased training sets. Our main motivation is the recognition of animal species for ecological applications such as biodiversity modelling, which is challenging because of long-tailed species distributions due to rare species, and strong dataset biases such as repetitive scene background in camera traps. To counteract these challenges, we propose a visual attention mechanism that is supervised via keypoint annotations that highlight important object parts. This privileged information, implemented as a novel privileged pooling operation, is only required during training and helps the model to focus on regions that are discriminative. In experiments with three different animal species datasets, we show that deep networks with privileged pooling can use small training sets more efficiently and generalize better.

Sample-Efficient Cardinality Estimation Using Geometric Deep Learning

In database systems, accurate cardinality estimation is a cornerstone of effective query optimization. In this context, estimators that use machine learning have shown significant promise. Despite their potential, the effectiveness of these learned estimators strongly depends on their ability to learn from small training sets. This paper presents a novel approach for learned cardinality estimation that addresses this issue by enhancing sample efficiency. We propose a neural network architecture informed by geometric deep learning principles that represents queries as join graphs. Furthermore, we introduce an innovative encoding for complex predicates, treating their encoding as a feature selection problem. Additionally, we devise a regularization term that employs equalities of the relational algebra and three-valued logic, augmenting the training process without requiring additional ground truth cardinalities. We rigorously evaluate our model across multiple benchmarks, examining q-errors, runtimes, and the impact of workload distribution shifts. Our results demonstrate that our model significantly improves the end-to-end runtimes of PostgreSQL, even with cardinalities gathered from as little as 100 query executions.

Analog RF Circuit Sizing by a Cascade of Shallow Neural Networks

A deep neural network architecture for the automatic sizing of analog circuit components is proposed, with a focus on radio frequency (RF) applications in the 2 to 5 GHz region. It addresses the challenges of the typically small number of examples for network training and the existence of multiple solutions, of which impractical values for integrated circuit implementation. We address these issues by restricting the learning to one component size at a time, thanks to a cascade of dedicated shallow neural networks (SNN), where each network constrains the prediction of the next ones. Moreover, the SNNs are individually tuned by a genetic algorithm for the prediction order and accuracy. This reduction of the solution space at each step allows the use of small training sets, and the introduced constraints between SNNs handle component interdependencies. The method is successfully validated on three different types of RF microcircuits: a low-noise amplifier (LNA), a voltage-controlled oscillator (VCO), and a mixer, using 180 nm and 130 nm CMOS implementations. All the predictions were within 5 % of the true values, both at the component and performance levels, and all the responses were obtained in less than 5 s, after 4 to 47 min. training on a regular PC station. The obtained results show that the proposed method is fast and applicable to arbitrary analog circuit topologies, with no need to retrain the developed neural network for each new set of desired circuit performances.

Efficient Exploration of Chemical Compound Space Using Active Learning for Prediction of Thermodynamic Properties of Alkane Molecules.

We introduce an exploratory active learning (AL) algorithm using Gaussian process regression and marginalized graph kernel (GPR-MGK) to sample chemical compound space (CCS) at minimal cost. Targeting 251,728 enumerated alkane molecules with 4-19 carbon atoms, we applied the AL algorithm to select a diverse and representative set of molecules and then conducted high-throughput molecular simulations on these selected molecules. To demonstrate the power of the AL algorithm, we built directed message-passing neural networks (D-MPNN) using simulation data as the training set to predict liquid densities, heat capacities, and vaporization enthalpies of the CCS. Validations show that D-MPNN models built on the smallest training set considered in this work, which consists of 313 molecules or 0.124% of the original CCS, predict the properties with R2 > 0.99 against the computational data and R2 > 0.94 against the experimental data. The advantage of the presented AL algorithm is that the predicted uncertainty of GPR depends on only the molecular structures, which renders it compatible with high-throughput data generation.

MahaEmoSen: Towards Emotion-aware Multimodal Marathi Sentiment Analysis

With the advent of the Internet, social media platforms have witnessed an enormous increase in user-generated textual and visual content. Microblogs on platforms such as Twitter are extremely useful for comprehending how individuals feel about a specific issue through their posted texts, images, and videos. Owing to the plethora of content generated, it is necessary to derive an insight of its emotional and sentimental inclination. Individuals express themselves in a variety of languages and, lately, the number of people preferring native languages has been consistently increasing. Marathi language is predominantly spoken in the Indian state of Maharashtra. However, sentiment analysis in Marathi has rarely been addressed. In light of the above, we propose an emotion-aware multimodal Marathi sentiment analysis method (MahaEmoSen). Unlike the existing studies, we leverage emotions embedded in tweets besides assimilating the content-based information from the textual and visual modalities of social media posts to perform a sentiment classification. We mitigate the problem of small training sets by implementing data augmentation techniques. A word-level attention mechanism is applied on the textual modality for contextual inference and filtering out noisy words from tweets. Experimental outcomes on real-world social media datasets demonstrate that our proposed method outperforms the existing methods for Marathi sentiment analysis in resource-constrained circumstances.

A prediction model for blood-brain barrier penetrating peptides based on masked peptide transformers with dynamic routing.

Blood-brain barrier penetrating peptides (BBBPs) are short peptide sequences that possess the ability to traverse the selective blood-brain interface, making them valuable drug candidates or carriers for various payloads. However, the in vivo or in vitro validation of BBBPs is resource-intensive and time-consuming, driving the need for accurate in silico prediction methods. Unfortunately, the scarcity of experimentally validated BBBPs hinders the efficacy of current machine-learning approaches in generating reliable predictions. In this paper, we present DeepB3P3, a novel framework for BBBPs prediction. Our contribution encompasses four key aspects. Firstly, we propose a novel deep learning model consisting of a transformer encoder layer, a convolutional network backbone, and a capsule network classification head. This integrated architecture effectively learns representative features from peptide sequences. Secondly, we introduce masked peptides as a powerful data augmentation technique to compensate for small training set sizes in BBBP prediction. Thirdly, we develop a novel threshold-tuning method to handle imbalanced data by approximating the optimal decision threshold using the training set. Lastly, DeepB3P3 provides an accurate estimation of the uncertainty level associated with each prediction. Through extensive experiments, we demonstrate that DeepB3P3 achieves state-of-the-art accuracy of up to 98.31% on a benchmarking dataset, solidifying its potential as a promising computational tool for the prediction and discovery of BBBPs.

Block sparsity promoting algorithm for efficient construction of cluster expansion models for multicomponent alloys

Multicomponent alloys are gaining significance as drivers of technological breakthroughs especially in structural and energy storage materials. The vast configuration space of these materials prohibit computational modeling using first-principles based methods alone. The cluster expansion (CE) method is the most widely used tool for modeling configurational disorder in alloys. CE relies on machine learning algorithms to train Hamiltonians and uses first-principles calculated data as training sets. In this paper we present a new compressive sensing-based algorithm for the efficient construction of CE Hamiltonians of multicomponent alloys. Our algorithm constructs highly sparse and physically reasonable models from a carefully selected small training set of alloy structures. Compared to conventional fitting algorithms, the algorithm achieves more than 50% reduction in the training set size. The resultant sparse models can sample the configuration space at least 3 × faster. We demonstrate this algorithm on 4 different alloy systems, namely Ag–Au, Ag–Au–Cu, Ag–Au–Cu–Pd and (Ge,Sn)(S,Se,Te).The sparse CE models for these alloys can rapidly reproduce known ground state orderings and order-disorder transitions. Our method can truly enable high-throughput multicomponent alloy thermodynamics by reducing the cost associated with model construction and configuration sampling.

Towards detection of cancer biomarkers in human exhaled air by transfer-learning-powered analysis of odor-evoked calcium activity in rat olfactory bulb

Detection of volatile organic compounds in exhaled air is a promising approach to non-invasive and scalable gastric cancer screening. This work proposes a new approach for the detection of volatile organic compounds by analyzing odor-evoked calcium responses in the rat olfactory bulb. We estimate the feasibility of gastric cancer biomarker detection added to the exhaled air of healthy participants. Our detector consists of a convolutional encoder and a similarity-based classifier over encoder outputs. To minimize overfitting on a small available training set, we involve a pre-training where the encoder is trained on synthetic data representing spatiotemporal patterns similar to real calcium responses in the olfactory bulb. We estimate the classification accuracy of exhaled air samples by matching their encodings with encodings of calibration samples of two classes: 1) exhaled air and 2) a mixture of exhaled air with the cancer biomarker. On our data, the accuracy increased from 0.68 on real data up to 0.74 if pre-training on synthetic data is involved. Our work is focused on proving the feasibility of proposed new approach rather than on comparing its efficiency with existing methods. Such detection is often performed with an electronic nose, but its output becomes unstable over time due to a sensor drift. In contrast to the electronic nose, rats can robustly detect low concentrations of biomarkers over lifetime. The feasibility of gastric cancer biomarker detection in exhaled air by bio-hybrid system is shown. Pre-training of neural models for images analysis increases the accuracy of detection.

Accelerating the characterization of dynamic DNA origami devices with deep neural networks

Mechanical characterization of dynamic DNA nanodevices is essential to facilitate their use in applications like molecular diagnostics, force sensing, and nanorobotics that rely on device reconfiguration and interactions with other materials. A common approach to evaluate the mechanical properties of dynamic DNA nanodevices is by quantifying conformational distributions, where the magnitude of fluctuations correlates to the stiffness. This is generally carried out through manual measurement from experimental images, which is a tedious process and a critical bottleneck in the characterization pipeline. While many tools support the analysis of static molecular structures, there is a need for tools to facilitate the rapid characterization of dynamic DNA devices that undergo large conformational fluctuations. Here, we develop a data processing pipeline based on Deep Neural Networks (DNNs) to address this problem. The YOLOv5 and Resnet50 network architecture were used for the two key subtasks: particle detection and pose (i.e. conformation) estimation. We demonstrate effective network performance (F1 score 0.85 in particle detection) and good agreement with experimental distributions with limited user input and small training sets (~ 5 to 10 images). We also demonstrate this pipeline can be applied to multiple nanodevices, providing a robust approach for the rapid characterization of dynamic DNA devices.

Oracle-based data generation for highly efficient digital twin network training

Recent advances in Graph Neural Networks (GNNs) has opened new capabilities to analyze complex communication systems. However, little work has been done to study the effects of limited data samples on the performance of GNN-based systems. In this paper, we present a novel solution to the problem of finding an optimal training set for efficient training of a RouteNet-Fermi GNN model. The proposed solution ensures good model generalization to large previously unseen networks under strict limitations on the training data budget and training topology sizes. Specifically, we generate an initial data set by emulating the flow distribution of large networks while using small networks. We then deploy a new clustering method that efficiently samples the above generated data set by analyzing the data embeddings from different Oracle models. This procedure provides a very small but information-rich training set. The above data embedding method translates highly heterogeneous network samples into a common embedding spac, wherein the samples can be easily related to each other. The proposed method outperforms state-of-the-art approaches, including the winning solutions of the 2022 Graph Neural Networking challenge.

A supervised approach for the detection of AM-FM signals’ interference regions in spectrogram images

Ridge curves retrieval in time–frequency (TF) domains is fundamental in many application fields as they convey most of information concerning the instantaneous frequencies of non-stationary signals. However, it represents a very hard task in the case of multicomponent signals having non-separable modes as they generate interference in TF domains. A preliminary detection of these interference regions may be then useful for the definition of effective strategies for ridge curve recovery. This paper introduces SIID-CNN (Spectrogram Image Interference Detection via CNN), that is a novel approach based on machine learning for the automatic detection of interference regions in spectrogram images. Each spectrogram sample is suitably classified as interference, single mode or background by accounting for its relative information. Some critical problems, such as the training set size and the type of examples to use for populating the training set, are dealt with. Experimental results show that a local linear model for spectrogram image and a small training set can guarantee good classification rates for a wide class of non-stationary signals, even in the presence of moderate noise.

A new computationally efficient method to tune BERT networks – transfer learning

BERT is a pre-trained language model that achieves state-of-the-art performance on natural language processing (NLP) tasks. Once it was published, it quickly became one of the most popular models in the NLP field. The official recommended method for applying BERT to downstream tasks is fine-tuning. However, we argue that transfer learning is also a very practical approach to applying BERT, and this approach has its own advantages compared to fine-tuning. In this paper, we explore the advantages of the transfer learning approach through the method that uses transfer learning and fine-tuning approaches separately to apply BERT to the same eight GLUE benchmark tasks and compare the training time spent and model performance scores, such as accuracy or F1 score, of the two approaches. Finally, this paper finds that among all eight GLUE tasks and on small training sets, transfer learning saves 30% to 50% of the time compared to a fine-tuning approach to train the models on the data of downstream tasks and can achieve very similar performance. Besides, for the same amount of time, transfer learning can obtain higher performance scores than fine-tuning on larger training sets. In conclusion, on small training sets, transfer learning has a huge advantage in time consumption under the premise of approximate performance, and it also performs better on large training sets under the same length of time, compared to fine-tuning.

Improving short text classification with augmented data using GPT-3

Abstract GPT-3 is a large-scale natural language model developed by OpenAI that can perform many different tasks, including topic classification. Although researchers claim that it requires only a small number of in-context examples to learn a task, in practice GPT-3 requires these training examples to be either of exceptional quality or a higher quantity than easily created by hand. To address this issue, this study teaches GPT-3 to classify whether a question is related to data science by augmenting a small training set with additional examples generated by GPT-3 itself. This study compares two augmented classifiers: the Classification Endpoint with an increased training set size and the Completion Endpoint with an augmented prompt optimized using a genetic algorithm. We find that data augmentation significantly increases the accuracy of both classifiers, and that the embedding-based Classification Endpoint achieves the best accuracy of about 76%, compared to human accuracy of 85%. In this way, giving large language models like GPT-3 the ability to propose their own training examples can improve short text classification performance.

Conjugated quantitative structure-property relationship models: Prediction of kinetic characteristics linked by the Arrhenius equation.

Conjugated QSPR models for reactions integrate fundamental chemical laws expressed by mathematical equations with machine learning algorithms. Herein we present a methodology for building conjugated QSPR models integrated with the Arrhenius equation. Conjugated QSPR models were used to predict kinetic characteristics of cycloaddition reactions related by the Arrhenius equation: rate constant , pre-exponential factor , and activation energy . They were benchmarked against single-task (individual and equation-based models) and multi-task models. In individual models, all characteristics were modeled separately, while in multi-task models , and were treated cooperatively. An equation-based model assessed using the Arrhenius equation and and values predicted by individual models. It has been demonstrated that the conjugated QSPR models can accurately predict the reaction rate constants at extreme temperatures, at which reaction rate constants hardly can be measured experimentally. Also, in the case of small training sets conjugated models are more robust than related single-task approaches.

Representative Data Selection for Efficient Medical Incremental Learning.

To train a deep neural network relies on a large amount of annotated data. In special scenarios like industry defect detection and medical imaging, it is hard to collect sufficient labeled data all at once. Newly annotated data may arrive incrementally. In practice, we also prefer our target model to improve its capability gradually as new data comes in by quick re-training. This work tackles this problem from a data selection prospective by constraining ourselves to always retrain the target model with a fix amount of data after new data comes in. A variational autoencoder (VAE) and an adversarial network are combined for data selection, achieving fast model retraining. This enables the target model to continually learn from a small training set while not losing the information learned from previous iterations, thus incrementally adapting itself to new-coming data. We validate our framework on the LGG Segmentation dataset for the semantic segmentation task.Clinical relevance- The proposed VAE-based data selection model combined with adversarial training can choose a representative and reliable subset of data for time-efficient medical incremental learning. Users can immediately see the improvement of the medical segmentation model whenever new annotated images are contributed (after a few minutes of model retraining).

The use of generative adversarial networks for multi-site one-class follicular lymphoma classification

Recent advances in digital technologies have lowered the costs and improved the quality of digital pathology Whole Slide Images (WSI), opening the door to apply Machine Learning (ML) techniques to assist in cancer diagnosis. ML, including Deep Learning (DL), has produced impressive results in diverse image classification tasks in pathology, such as predicting clinical outcomes in lung cancer and inferring regional gene expression signatures. Despite these promising results, the uptake of ML as a common diagnostic tool in pathology remains limited. A major obstacle is the insufficient labelled data for training neural networks and other classifiers, especially for new sites where models have not been established yet. Recently, image synthesis from small, labelled datasets using Generative Adversarial Networks (GAN) has been used successfully to create high-performing classification models. Considering the domain shift and complexity in annotating data, we investigated an approach based on GAN that minimized the differences in WSI between large public data archive sites and a much smaller data archives at the new sites. The proposed approach allows the tuning of a deep learning classification model for the class of interest to be improved using a small training set available at the new sites. This paper utilizes GAN with the one-class classification concept to model the class of interest data. This approach minimizes the need for large amounts of labelled data from the new site to train the network. The GAN generates synthesized one-class WSI images to jointly train the classifier with WSIs available from the new sites. We tested the proposed approach for follicular lymphoma data of a new site by utilizing the data archives from different sites. The synthetic images for the one-class data generated from the data obtained from different sites with minimum amount of data from the new site have resulted in a significant improvement of 15% for the Area Under the curve (AUC) for the new site that we want to establish a new follicular lymphoma classifier. The test results have shown that the classifier can perform well without the need to obtain more training data from the test site, by utilizing GAN to generate the synthetic data from all existing data in the archives from all the sites.

Application of a wavelength angle mapper for variable selection in iterative optimization technology predictions of drug content in pharmaceutical powder mixtures

Process analytical technology (PAT) is an essential tool within pharmaceutical manufacturing to ensure consistent quality and maintain process control. Near-infrared (NIR) spectroscopy is one of the most popular PAT techniques, particularly for monitoring active pharmaceutical ingredient (API) concentrations. To interpret the spectral outputs of NIR spectroscopy, advanced multivariate models are required. Calibration-free models such as iterative optimization technology (IOT) algorithms are increasingly of interest, due primarily to their reduced material and time burdens. Variable/wavelength selection is a common method to improve prediction performance and robustness for IOT by focusing on spectral regions with the most relevant information. However, currently proposed wavelength selection approaches rely on training sets for optimization, therefore reducing or removing the advantages of IOT over empirical calibration-dependent models. In this work, a true calibration-free wavelength selection method is proposed based on measuring the difference between individual wavelengths of a mixture spectra and the net analyte signals via a wavelength angle mapper (WAM). An extension of the WAM utilizing a spectral window of wavelength instead of individual wavelengths, called SWAM, was also developed. However, the SWAM method does require a small training set to optimize wavelength selection parameters. The WAM and SWAM methods showed similar prediction performance for API in pharmaceutical powder blends when compared against other calibration-dependent models and the base IOT algorithm.

Predicting Fetal Alcohol Spectrum Disorders Using Machine Learning Techniques: Multisite Retrospective Cohort Study.

Fetal alcohol syndrome (FAS) is a lifelong developmental disability that occurs among individuals with prenatal alcohol exposure (PAE). With improved prediction models, FAS can be diagnosed or treated early, if not completely prevented. In this study, we sought to compare different machine learning algorithms and their FAS predictive performance among women who consumed alcohol during pregnancy. We also aimed to identify which variables (eg, timing of exposure to alcohol during pregnancy and type of alcohol consumed) were most influential in generating an accurate model. Data from the collaborative initiative on fetal alcohol spectrum disorders from 2007 to 2017 were used to gather information about 595 women who consumed alcohol during pregnancy at 5 hospital sites around the United States. To obtain information about PAE, questionnaires or in-person interviews, as well as reviews of medical, legal, or social service records were used to gather information about alcohol consumption. Four different machine learning algorithms (logistic regression, XGBoost, light gradient-boosting machine, and CatBoost) were trained to predict the prevalence of FAS at birth, and model performance was measured by analyzing the area under the receiver operating characteristics curve (AUROC). Of the total cases, 80% were randomly selected for training, while 20% remained as test data sets for predicting FAS. Feature importance was also analyzed using Shapley values for the best-performing algorithm. Overall, there were 20 cases of FAS within a total population of 595 individuals with PAE. Most of the drinking occurred in the first trimester only (n=491) or throughout all 3 trimesters (n=95); however, there were also reports of drinking in the first and second trimesters only (n=8), and 1 case of drinking in the third trimester only (n=1). The CatBoost method delivered the best performance in terms of AUROC (0.92) and area under the precision-recall curve (AUPRC 0.51), followed by the logistic regression method (AUROC 0.90; AUPRC 0.59), the light gradient-boosting machine (AUROC 0.89; AUPRC 0.52), and XGBoost (AUROC 0.86; AURPC 0.45). Shapley values in the CatBoost model revealed that 12 variables were considered important in FAS prediction, with drinking throughout all 3 trimesters of pregnancy, maternal age, race, and type of alcoholic beverage consumed (eg, beer, wine, or liquor) scoring highly in overall feature importance. For most predictive measures, the best performance was obtained by the CatBoost algorithm, with an AUROC of 0.92, precision of 0.50, specificity of 0.29, F1 score of 0.29, and accuracy of 0.96. Machine learning algorithms were able to identify FAS risk with a prediction performance higher than that of previous models among pregnant drinkers. For small training sets, which are common with FAS, boosting mechanisms like CatBoost may help alleviate certain problems associated with data imbalances and difficulties in optimization or generalization.

CellSighter: a neural network to classify cells in highly multiplexed images

Multiplexed imaging enables measurement of multiple proteins in situ, offering an unprecedented opportunity to chart various cell types and states in tissues. However, cell classification, the task of identifying the type of individual cells, remains challenging, labor-intensive, and limiting to throughput. Here, we present CellSighter, a deep-learning based pipeline to accelerate cell classification in multiplexed images. Given a small training set of expert-labeled images, CellSighter outputs the label probabilities for all cells in new images. CellSighter achieves over 80% accuracy for major cell types across imaging platforms, which approaches inter-observer concordance. Ablation studies and simulations show that CellSighter is able to generalize its training data and learn features of protein expression levels, as well as spatial features such as subcellular expression patterns. CellSighter’s design reduces overfitting, and it can be trained with only thousands or even hundreds of labeled examples. CellSighter also outputs a prediction confidence, allowing downstream experts control over the results. Altogether, CellSighter drastically reduces hands-on time for cell classification in multiplexed images, while improving accuracy and consistency across datasets.