Number Of Training Examples Research Articles

Dynamic Analogical Association Algorithm Based on Manifold Matching for Few-Shot Learning

At present, deep learning has been well applied in many fields. However, due to the high complexity of hypothesis space, numerous training samples are usually required to ensure the reliability of minimizing experience risk. Therefore, training a classifier with a small number of training examples is a challenging task. From a biological point of view, based on the assumption that rich prior knowledge and analogical association should enable human beings to quickly distinguish novel things from a few or even one example, we proposed a dynamic analogical association algorithm to make the model use only a few labeled samples for classification. To be specific, the algorithm search for knowledge structures similar to existing tasks in prior knowledge based on manifold matching, and combine sampling distributions to generate offsets instead of two sample points, thereby ensuring high confidence and significant contribution to the classification. The comparative results on two common benchmark datasets substantiate the superiority of the proposed method compared to existing data generation approaches for few-shot learning, and the effectiveness of the algorithm has been proved through ablation experiments.

Comparing ensemble methods combined with different aggregating models using micrograph cell segmentation as an initial application example

Strategies such as ensemble learning and averaging techniques try to reduce the variance of single deep neural networks. The focus of this study is on ensemble averaging techniques, fusing the results of differently initialized and trained networks. Thereby, using micrograph cell segmentation as an application example, various ensembles have been initialized and formed during network training, whereby the following methods have been applied: (a) random seeds, (b) L1-norm pruning, (c) variable numbers of training examples, and (d) a combination of the latter 2 items. Furthermore, different averaging methods are in common use and were evaluated in this study. As averaging methods, the mean, the median, and the location parameter of an alpha-stable distribution, fit to the histograms of class membership probabilities (CMPs), as well as a majority vote of the members of an ensemble were considered. The performance of these methods is demonstrated and evaluated on a micrograph cell segmentation use case, employing a common state-of-the art deep convolutional neural network (DCNN) architecture exploiting the principle of the common VGG-architecture. The study demonstrates that for this data set, the choice of the ensemble averaging method only has a marginal influence on the evaluation metrics (accuracy and Dice coefficient) used to measure the segmentation performance. Nevertheless, for practical applications, a simple and fast estimate of the mean of the distribution is highly competitive with respect to the most sophisticated representation of the CMP distributions by an alpha-stable distribution, and hence seems the most proper ensemble averaging method to be used for this application.

Disentangling multiple scattering with deep learning: application to strain mapping from electron diffraction patterns

A fast, robust pipeline for strain mapping of crystalline materials is important for many technological applications. Scanning electron nanodiffraction allows us to calculate strain maps with high accuracy and spatial resolutions, but this technique is limited when the electron beam undergoes multiple scattering. Deep-learning methods have the potential to invert these complex signals, but require a large number of training examples. We implement a Fourier space, complex-valued deep-neural network, FCU-Net, to invert highly nonlinear electron diffraction patterns into the corresponding quantitative structure factor images. FCU-Net was trained using over 200,000 unique simulated dynamical diffraction patterns from different combinations of crystal structures, orientations, thicknesses, and microscope parameters, which are augmented with experimental artifacts. We evaluated FCU-Net against simulated and experimental datasets, where it substantially outperforms conventional analysis methods. Our code, models, and training library are open-source and may be adapted to different diffraction measurement problems.

On Supervised Class-Imbalanced Learning: An Updated Perspective and Some Key Challenges

The problem of class imbalance has always been considered as a significant challenge to traditional machine learning and the emerging deep learning research communities. A classification problem can be considered as class imbalanced if the training set does not contain an equal number of labeled examples from all the classes. A classifier trained on such an imbalanced training set is likely to favor those classes containing a larger number of training examples than the others. Unfortunately, the classes that contain a small number of labelled instances usually correspond to rare and significant events. Thus, poor classification accuracy on these classes may lead to severe consequences. In this article, we aim to provide a comprehensive summary of the rich pool of research works attempting to combat the adversarial effects of class imbalance efficiently. Specifically, following a formal definition of the problem of class imbalance, we explore the plethora of traditional machine learning approaches aiming to mitigate its adversarial effects. We further discuss the state-of-the-art deep-learning-based approaches for improving a classifier’s resilience against class imbalance and highlight the need for techniques tailored for such a paradigm. Moreover, we look at the emerging applications where class imbalance can be a major concern. Finally, we outline a few open problems along with the various challenges emerging with the advent of modern applications, deep learning paradigm, and new sources of data.

Fuzzy Active Learning to Detect OpenCL Kernel Heterogeneous Machines in Cyber Physical Systems

Cyber-physical systems (CPS) consist of a variety of multicore architectures, including central processing units (CPU) and graphical processing units (GPU). In general, programmers assign sequential programs to the CPU while parallel applications are assigned to the GPU. This article provides a method for mapping an OpenCL application to a heterogeneous multicore architecture using active fuzzy learning to determine the adequacy and processing capabilities of the application. During learning, subsamples are created by developing a machine learning-based device suitability classifier that predicts which processors would have excessive computational compatibility for running OpenCL programs. In addition, this study integrates an active learning model based on entropy with a fuzzification model to find nonoverlapping patterns. To minimize rule generation, the fuzzification-based weighted probabilistic technique is presented. The defuzzification process is optimized by using uncertainty values in conjunction with classification probability. In addition, 20 different features are proposed for extraction using the newly developed LLVM-based static analyzer. The correlation analysis approach is used to determine the optimal subset of features. The synthetic minority oversampling approach with and without feature selection is used to differentiate the class imbalance problem. Instead of manually modifying the machine learning classifier, a tree-based pipeline construction approach is used to determine the optimal classifier and associated hyperparameters. Experiments are then conducted on a set of benchmarks to verify the performance of the designed model. The results show that by increasing the number of training examples and including an entropy uncertainty measure, the proposed model is able to support and improve decision boundaries. We achieved a high F-measure of 0.77 and a ROC of 0.92 by optimizing and reducing the feature subsets.

On the proliferation of support vectors in high dimensions* *This article is an updated version of: Hsu D, Muthukumar V and Xu J L 2021 On the proliferation of support vectors in high dimensions Proc. 24th Int. Conf. Artificial Intelligence and Statistics vol 130 ed Banerjee A and Fukumizu K pp 91–9.

The support vector machine (SVM) is a well-established classification method whose name refers to the particular training examples, called support vectors, that determine the maximum margin separating hyperplane. The SVM classifier is known to enjoy good generalization properties when the number of support vectors is small compared to the number of training examples. However, recent research has shown that in sufficiently high-dimensional linear classification problems, the SVM can generalize well despite a proliferation of support vectors where all training examples are support vectors. In this paper, we identify new deterministic equivalences for this phenomenon of support vector proliferation, and use them to (1) substantially broaden the conditions under which the phenomenon occurs in high-dimensional settings, and (2) prove a nearly matching converse result.

C-arm positioning for standard projections during spinal implant placement.

Fluoroscopy-guided trauma and orthopedic surgeries involve the repeated acquisition of correct anatomy-specific standard projections for guidance, monitoring, and evaluating the surgical result. C-arm positioning is usually performed by hand, involving repeated or even continuous fluoroscopy at a cost of radiation exposure and time. We propose to automate this procedure and estimate the pose update for C-arm repositioning directly from a first X-ray without the need for a patient-specific computed tomography scan (CT) or additional technical equipment. Our method is trained on digitally reconstructed radiographs (DRRs) which uniquely provide ground truth labels for an arbitrary number of training examples. The simulated images are complemented with automatically generated segmentations, landmarks, and with simulated k-wires and screws. To successfully achieve a transfer from simulated to real X-rays, and also to increase the interpretability of results, the pipeline was designed to closely reflect the actual clinical decision-making process followed by spinal neurosurgeons. It explicitly incorporates steps such as region-of-interest (ROI) localization, detection of relevant and view-independent landmarks, and subsequent pose regression. The method was validated on a large human cadaver study simulating a real clinical scenario, including k-wires and screws. The proposed procedure obtained superior C-arm positioning accuracy of dθ=8.8°±4.2° average improvement (pt-test≪0.01), robustness, and generalization capabilities compared to the state-of-the-art direct pose regression framework.

Probabilistic learning for pulsar classification

In this work, we explore the possibility of using probabilistic learning to identify pulsar candidates. We make use of Deep Gaussian Process (DGP) and Deep Kernel Learning (DKL). Trained on a balanced training set in order to avoid the effect of class imbalance, the performance of the models, achieving relatively high probability of differentiating the positive class from the negative one (roc-auc ∼ 0.98), is very promising overall. We estimate the predictive entropy of each model predictions and find that DKL is more confident than DGP in its predictions and provides better uncertainty calibration. Upon investigating the effect of training with imbalanced dataset on the models, results show that each model performance decreases with an increasing number of the majority class in the training set. Interestingly, with a number of negative class 10× that of positive class, the models still provide reasonably well calibrated uncertainty, i.e. an expected Uncertainty Calibration Error (UCE) less than 6%. We also show in this study how, in the case of relatively small amount of training dataset, a convolutional neural network based classifier trained via Bayesian Active Learning by Disagreement (BALD) performs. We find that, with an optimized number of training examples, the model — being the most confident in its predictions — generalizes relatively well and produces the best uncertainty calibration which corresponds to UCE = 3.118%.

Brain inspired neuronal silencing mechanism to enable reliable sequence identification

Real-time sequence identification is a core use-case of artificial neural networks (ANNs), ranging from recognizing temporal events to identifying verification codes. Existing methods apply recurrent neural networks, which suffer from training difficulties; however, performing this function without feedback loops remains a challenge. Here, we present an experimental neuronal long-term plasticity mechanism for high-precision feedforward sequence identification networks (ID-nets) without feedback loops, wherein input objects have a given order and timing. This mechanism temporarily silences neurons following their recent spiking activity. Therefore, transitory objects act on different dynamically created feedforward sub-networks. ID-nets are demonstrated to reliably identify 10 handwritten digit sequences, and are generalized to deep convolutional ANNs with continuous activation nodes trained on image sequences. Counterintuitively, their classification performance, even with a limited number of training examples, is high for sequences but low for individual objects. ID-nets are also implemented for writer-dependent recognition, and suggested as a cryptographic tool for encrypted authentication. The presented mechanism opens new horizons for advanced ANN algorithms.

Multimodal CT Image Synthesis Using Unsupervised Deep Generative Adversarial Networks for Stroke Lesion Segmentation

Deep learning-based techniques can obtain high precision for multimodal stroke segmentation tasks. However, the performance often requires a large number of training examples. Additionally, existing data extension approaches for the segmentation are less efficient in creating much more realistic images. To overcome these limitations, an unsupervised adversarial data augmentation mechanism (UTC-GAN) is developed to synthesize multimodal computed tomography (CT) brain scans. In our approach, the CT samples generation and cross-modality translation differentiation are accomplished simultaneously by integrating a Siamesed auto-encoder architecture into the generative adversarial network. In addition, a Gaussian mixture translation module is further proposed, which incorporates a translation loss to learn an intrinsic mapping between the latent space and the multimodal translation function. Finally, qualitative and quantitative experiments show that UTC-GAN significantly improves the generation ability. The stroke dataset enriched by the proposed model also provides a superior improvement in segmentation accuracy, compared with the performance of current competing unsupervised models.

On improving the performance of glitch classification for gravitational wave detection by using Generative Adversarial Networks

ABSTRACT Spectrogram classification plays an important role in analysing gravitational wave data. In this paper, we propose a framework to improve the classification performance by using Generative Adversarial Networks (GANs). As substantial efforts and expertise are required to annotate spectrograms, the number of training examples is very limited. However, it is well known that deep networks can perform well only when the sample size of the training set is sufficiently large. Furthermore, the imbalanced sample sizes in different classes can also hamper the performance. In order to tackle these problems, we propose a GAN-based data augmentation framework. While standard data augmentation methods for conventional images cannot be applied on spectrograms, we found that a variant of GANs, ProGAN, is capable of generating high-resolution spectrograms which are consistent with the quality of the high-resolution original images and provide a desirable diversity. We have validated our framework by classifying glitches in the Gravity Spy data set with the GAN-generated spectrograms for training. We show that the proposed method can provide an alternative to transfer learning for the classification of spectrograms using deep networks, i.e. using a high-resolution GAN for data augmentation instead. Furthermore, fluctuations in classification performance with small sample sizes for training and evaluation can be greatly reduced. Using the trained network in our framework, we have also examined the spectrograms with label anomalies in Gravity Spy.

Accurate Molecular-Orbital-Based Machine Learning Energies via Unsupervised Clustering of Chemical Space.

We introduce an unsupervised clustering algorithm to improve training efficiency and accuracy in predicting energies using molecular-orbital-based machine learning (MOB-ML). This work determines clusters via the Gaussian mixture model (GMM) in an entirely automatic manner and simplifies an earlier supervised clustering approach [ J. Chem. Theory Comput. 2019, 15, 6668] by eliminating both the necessity for user-specified parameters and the training of an additional classifier. Unsupervised clustering results from GMM have the advantages of accurately reproducing chemically intuitive groupings of frontier molecular orbitals and exhibiting improved performance with an increasing number of training examples. The resulting clusters from supervised or unsupervised clustering are further combined with scalable Gaussian process regression (GPR) or linear regression (LR) to learn molecular energies accurately by generating a local regression model in each cluster. Among all four combinations of regressors and clustering methods, GMM combined with scalable exact GPR (GMM/GPR) is the most efficient training protocol for MOB-ML. The numerical tests of molecular energy learning on thermalized data sets of drug-like molecules demonstrate the improved accuracy, transferability, and learning efficiency of GMM/GPR over other training protocols for MOB-ML, i.e., supervised regression clustering combined with GPR (RC/GPR) and GPR without clustering. GMM/GPR also provides the best molecular energy predictions compared with ones from the literature on the same benchmark data sets. With a lower scaling, GMM/GPR has a 10.4-fold speedup in wall-clock training time compared with scalable exact GPR with a training size of 6500 QM7b-T molecules.

Estimating BOS Image Deformation With A Lightweight CNN

We introduce a Convolutional Neural Network (CNN) to post-process recordings obtained by means of Background Oriented Schlieren (BOS), a popular technique to visualize compressible/convective flows. To reconstruct BOS image deformation, we devised a lightweight network (LIMA) that has comparatively fewer parameters to train, allowing the deployment of the network on embedded GPU hardware. To train the CNN, we introduce a novel strategy based on the generation of synthetic images with random, irrotational displacement field that mimic those provided by real BOS recording. This allows us to generate a large number of training examples at minimal computational cost. To assess the accuracy of the reconstructed displacement, we consider test cases consisting of synthetic images with sinusoidal displacement as well as images obtained in a real experimental study of flow past and inside a heated hollow hemisphere. By comparing the prediction of the CNN with conventional post-processing techniques such as Direct Image Correlation (DIC) and Particle Image Velocimetry (PIV) cross-correlation, we show that LIMA gives more accurate and robust results for the synthetic example. When applied to the recordings from the real experiment, all methods provide consistent deformation fields. As they offer similar or better accuracy at a fraction of the computational costs, properly designed CNNs offer a valuable alternative to conventional post-processing techniques for BOS experiments.

A Generic Semi-Supervised and Active Learning Framework for Biomedical Text Classification.

Biomedical text classification requires having training examples labeled by clinical specialists, a process that can be costly. To address this problem, active learning incrementally selects a subset of the most informative unlabeled examples, samples that are then labeled and used to train a given classifier, seeking to reduce the number of labeled samples. Nonetheless, the other unlabeled examples are not used by active learning, but incorporating semi-supervised techniques that use unlabeled samples could improve the representativeness of the data and the discriminatory power of the classifiers. This work proposes a generic semi-supervised learning framework for improving active learning and reducing the number of labeled training examples in biomedical text classification. The proposed framework combines manually annotated training examples selected by active learning and pseudo-labels obtained from a trained classifier. To evaluate the proposed framework, three biomedical datasets with textual information on obesity and smoking habit were used across different classification algorithms. The classification results show that the proposed framework can reduce the number of training examples that are manually labeled by clinical specialists by a 10% without affecting the performance of the classifiers. This performance is attributable to the ability of the classifiers to correctly select and label the training examples. Clinical relevance- We demonstrate the effectiveness of the proposed semi-supervised learning framework to reduce manual labeling efforts of biomedical texts by clinical specialists for the training of classifiers.

Inferring Prototypes for Multi-Label Few-Shot Image Classification with Word Vector Guided Attention

Multi-label few-shot image classification (ML-FSIC) is the task of assigning descriptive labels to previously unseen images, based on a small number of training examples. A key feature of the multi-label setting is that images often have multiple labels, which typically refer to different regions of the image. When estimating prototypes, in a metric-based setting, it is thus important to determine which regions are relevant for which labels, but the limited amount of training data makes this highly challenging. As a solution, in this paper we propose to use word embeddings as a form of prior knowledge about the meaning of the labels. In particular, visual prototypes are obtained by aggregating the local feature maps of the support images, using an attention mechanism that relies on the label embeddings. As an important advantage, our model can infer prototypes for unseen labels without the need for fine-tuning any model parameters, which demonstrates its strong generalization abilities. Experiments on COCO and PASCAL VOC furthermore show that our model substantially improves the current state-of-the-art.

A tomographic workflow to enable deep learning for X-ray based foreign object detection

Detection of unwanted (‘foreign’) objects within products is a common procedure in many branches of industry for maintaining production quality. X-ray imaging is a fast, non-invasive and widely applicable method for foreign object detection. Deep learning has recently emerged as a powerful approach for recognizing patterns in radiographs (i.e., X-ray images), enabling automated X-ray based foreign object detection. However, these methods require a large number of training examples and manual annotation of these examples is a subjective and laborious task. In this work, we propose a Computed Tomography (CT) based method for producing training data for supervised learning of foreign object detection, with minimal labor requirements. In our approach, a few representative objects are CT scanned and reconstructed in 3D. The radiographs that are acquired as part of the CT-scan data serve as input for the machine learning method. High-quality ground truth locations of the foreign objects are obtained through accurate 3D reconstructions and segmentations. Using these segmented volumes, corresponding 2D segmentations are obtained by creating virtual projections. We outline the benefits of objectively and reproducibly generating training data in this way. In addition, we show how the accuracy depends on the number of objects used for the CT reconstructions. The results show that in this workflow generally only a relatively small number of representative objects (i.e., fewer than 10) are needed to achieve adequate detection performance in an industrial setting.

Embedded Temporal Convolutional Networks for Essential Climate Variables Forecasting.

Forecasting the values of essential climate variables like land surface temperature and soil moisture can play a paramount role in understanding and predicting the impact of climate change. This work concerns the development of a deep learning model for analyzing and predicting spatial time series, considering both satellite derived and model-based data assimilation processes. To that end, we propose the Embedded Temporal Convolutional Network (E-TCN) architecture, which integrates three different networks, namely an encoder network, a temporal convolutional network, and a decoder network. The model accepts as input satellite or assimilation model derived values, such as land surface temperature and soil moisture, with monthly periodicity, going back more than fifteen years. We use our model and compare its results with the state-of-the-art model for spatiotemporal data, the ConvLSTM model. To quantify performance, we explore different cases of spatial resolution, spatial region extension, number of training examples and prediction windows, among others. The proposed approach achieves better performance in terms of prediction accuracy, while using a smaller number of parameters compared to the ConvLSTM model. Although we focus on two specific environmental variables, the method can be readily applied to other variables of interest.

Thermoacoustic stability prediction using classification algorithms

Abstract Predicting the occurrence of thermoacoustic instabilities is of major interest in a variety of engineering applications such as aircraft propulsion, power generation, and industrial heating. Predictive methodologies based on a physical approach have been developed in the past decades, but have a moderate-to-high computational cost when exploring a large number of designs. In this study, the stability prediction capabilities and computational cost of four well-established classification algorithms—the K-Nearest Neighbors, Decision Tree (DT), Random Forest (RF), and Multilayer Perceptron (MLP) algorithms—are investigated. These algorithms are trained using an in-house physics-based low-order network model tool called OSCILOS. All four algorithms are able to predict which configurations are thermoacoustically unstable with a very high accuracy and a very low runtime. Furthermore, the frequency intervals containing unstable modes for a given configuration are also accurately predicted using multilabel classification. The RF algorithm correctly predicts the overall stability and finds all frequency intervals containing unstable modes for 99.6 and 98.3% of all configurations, respectively, which makes it the most accurate algorithm when a large number of training examples is available. For smaller training sets, the MLP algorithm becomes the most accurate algorithm. The DT algorithm is found to be slightly less accurate, but can be trained extremely quickly and runs about a million times faster than a traditional physics-based low-order network model tool. These findings could be used to devise a new generation of combustor optimization tools that would run much faster than existing codes while retaining a similar accuracy.

A review on remote sensing imagery augmentation using deep learning

The quantity of remote sensing pictures being captured is rising dramatically due to advancements in earth observation technology, bringing remote sensing in a long and distinct period of history of big data. Discovering how to successfully harvest this enormous intensity of remote sensing data will be a new challenge. With deep learning, remote sensing data analysis may take on a narrative tone. CNNs are a Deep Learning model capable of directly extracting features from large volumes of picture data and are effective at exploiting realistic imagery data characteristics. With regards to computer vision, CNNs have shown to be highly successful. CNNs are a powerful tool. There have been several studies employing CNNs for remote sensing picture classification in the last few years, and these studies show that CNNs can produce fast and accurate results. It is the goal of this study to offer an overview of the most current and advanced applications of deep learning based on CNNs for remote sensing picture enhancement. To begin, the concepts and features of convolutional neural networks (CNNs) will be briefly reviewed in this review. There are new innovations and structural enhancements to neural network models that make them more suited for RS picture enhancement, as well as data for remote sensing image categorization and augmentation approaches that may be obtained easily and quickly.. As a result of the survey, we expect remote sensing scientists will be better equipped to tackle categorization challenges utilizing cutting-edge deep learning algorithms and methodologies. To increase the number of training examples with a considerable reduction in model over-fitting, data augmentation is a well-known approach used in most ML & AI applications.

Распознавание микроскопических изображений пыльцевых зерен с помощью сверточной нейронной сети VGG-16

The article presents the result of an experiment on the application of transfer learning using the Visual Geometry Group with 16 layers (VGG-16) convolutional neural network in relation to the problem of recognizing pollen grains in images. An analysis of the information-theoretical base on the application of machine learning algorithms to the problem of classifying pollen grains over the past few years has shown the need to develop (apply) a new method for recognizing images of pollen grains obtained using an optical microscope. Currently, automatic classification for pollen identification is becoming a very active area of research. The article substantiates the task of automating the classification of pollen grains. The aim of the study is to analyze the efficiency and accuracy of classifying microscopic images of pollen grains using transfer learning of the VGG-16 convolutional neural network. Transfer learning was performed using the VGG-16 neural network, which has 13 convolutional layers grouped into 5 blocks with pooling and 3 smoothing layers at the output. Since transfer learning is used, the number of training epochs can be chosen to be small. For this network, only the smoothing output layers change, and the feature extraction remains the same as in the classical model. Therefore, it was chosen to use 10 training epochs. Other hyperparameters are as follows: Drop Out regularization with a probability of 50%, optimization method is ADAM, activation function is sigmoid, loss function is cross-entropy, batch size is 32 images. As a result, by adjusting the hyperparameters of the model and using augmentation, it was possible to achieve a share of correct recognitions of about 80%. At the same time, due to the different number of training examples, the particular characteristics of the classes differ somewhat. Thus, the maximum precision and recall reach 94% and 83%, respectively, for the Dandelion type. In the future, studies are planned to use augmentation as a preprocessing to create a balanced sample. By applying the VGG-16 convolutional neural network to the problem of pollen grain image recognition, high accuracy and efficiency of the method were achieved.