Small Training Data Research Articles

COVID-19 identification in ct images based on deep learning models: a comparative approach

People's lives could be in danger if a contagious disease spreads quickly, Corona-2019 virus disease (COVID-19) is one. The coronavirus epidemic rapidly spread over the world. The Corona virus has had a major impact on the health of populations and healthcare systems all over the world. RT-PCR (RT-Reverse transcription, PCR-polymerase chain reaction) testing can benefit from the use of computed tomography images. Most available methods use large training data, and the detection accuracy needs to be improved due to the inadequate border segment of symptom descriptions. This study proposes a robust and effective way for identifying normal and COVID-19 patients using small training data. Deep learning quickly creates accurate models. Data augmentation increases the training dataset to reduce over fitting and improve model generalisation. Using data augmentation, we evaluated Xception and VGG-19. The study showed that deep learning can detect COVID-19.

Boosting the Transformer with the BERT Supervision in Low-Resource Machine Translation

Previous works trained the Transformer and its variants end-to-end and achieved remarkable translation performance when there are huge parallel sentences available. However, these models suffer from the data scarcity problem in low-resource machine translation tasks. To deal with the mismatch problem between the big model capacity of the Transformer and the small parallel training data set, this paper adds the BERT supervision on the latent representation between the encoder and the decoder of the Transformer and designs a multi-step training algorithm to boost the Transformer on such a basis. The algorithm includes three stages: (1) encoder training, (2) decoder training, and (3) joint optimization. We introduce the BERT of the target language in the encoder and the decoder training and alleviate the data starvation problem of the Transformer. After the training stage, the BERT will not further attend the inference section explicitly. Another merit of our training algorithm is that it can further enhance the Transformer in the task where there are limited parallel sentence pairs but large amounts of monolingual corpus of the target language. The evaluation results on six low-resource translation tasks suggest that the Transformer trained by our algorithm significantly outperforms the baselines which were trained end-to-end in previous works.

A General Alternating-Direction Implicit Framework with Gaussian Process Regression Parameter Prediction for Large Sparse Linear Systems

This paper proposes an efficient general alternating-direction implicit (GADI) framework for solving large sparse linear systems. The convergence property of the GADI framework is discussed. Most of the existing ADI methods can be viewed as particular schemes of the developed framework. Meanwhile the GADI framework can derive new ADI methods. Moreover, as the algorithm efficiency is sensitive to the splitting parameters, we offer a data-driven approach, the Gaussian process regression (GPR) method based on the Bayesian inference, to predict the GADI framework's relatively optimal parameters. The GPR method requires a small training data set to learn the regression prediction mapping, which has sufficient accuracy and high generalization capability. It allows us to efficiently solve linear systems with a one-shot computation, and does not require any repeated computations to obtain relatively optimal splitting parameters. Finally, we use the three-dimensional convection-diffusion equation and continuous Sylvester matrix equation to examine the performance of our proposed methods. Numerical results demonstrate that the proposed framework is faster tens to thousands of times than the existing ADI methods, such as (inexact) Hermitian and skew-Hermitian splitting type methods in which the consumption of obtaining relatively optimal splitting parameters is ignored. Due to the efficiency of the developed methods, we can solve much larger linear systems which these existing ADI methods have been not reached.

Infusing Expert Knowledge Into a Deep Neural Network Using Attention Mechanism for Personalized Learning Environments.

Machine learning models are biased toward data seen during the training steps. The models will tend to give good results in classes where there are many examples and poor results in those with few examples. This problem generally occurs when the classes to predict are imbalanced and this is frequent in educational data where for example, there are skills that are very difficult or very easy to master. There will be less data on students that correctly answered questions related to difficult skills and who incorrectly answered those related to skills easy to master. In this paper, we tackled this problem by proposing a hybrid architecture combining Deep Neural Network architectures— especially Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNN)—with expert knowledge for user modeling. The proposed solution uses attention mechanism to infuse expert knowledge into the Deep Neural Network. It has been tested in two contexts: knowledge tracing in an intelligent tutoring system (ITS) called Logic-Muse and prediction of socio-moral reasoning in a serious game called MorALERT. The proposed solution is compared to state-of-the-art machine learning solutions and experiments show that the resulting model can accurately predict the current student's knowledge state (in Logic-Muse) and thus enable an accurate personalization of the learning process. Other experiments show that the model can also be used to predict the level of socio-moral reasoning skills (in MorALERT). Our findings suggest the need for hybrid neural networks that integrate prior expert knowledge (especially when it is necessary to compensate for the strong dependency—of deep learning methods—on data size or the possible unbalanced datasets). Many domains can benefit from such an approach to building models that allow generalization even when there are small training data.

Cross-Leg Prediction of Running Kinematics across Various Running Conditions and Drawing from a Minimal Data Set Using a Single Wearable Sensor

The feasibility of prediction of same-limb kinematics using a single inertial measurement unit attached to the same limb has been demonstrated using machine learning. This study was performed to see if a single inertial measurement unit attached to the tibia can predict the opposite leg’s kinematics (cross-leg prediction). It also investigated if there is a minimal or smaller data set in a convolutional neural network model to predict lower extremity running kinematics under other running conditions and with what accuracy for the intra- and inter-participant situations. Ten recreational runners completed running exercises under five conditions, including treadmill running at speeds of 2, 2.5, 3, and 3.5 m/s and level-ground running at their preferred speed. A one-predict-all scheme was adopted to determine which running condition could be used to best predict a participant’s overall running kinematics. Running kinematic predictions were performed for intra- and inter-participant scenarios. Among the tested running conditions, treadmill running at 3 m/s was found to be the optimal condition for accurately predicting running kinematics under other conditions, with R2 values ranging from 0.880 to 0.958 and 0.784 to 0.936 for intra- and inter-participant scenarios, respectively. The feasibility of cross-leg prediction was demonstrated but with significantly lower accuracy than the same leg. The treadmill running condition at 3 m/s showed the highest intra-participant cross-leg prediction accuracy. This study proposes a novel, deep-learning method for predicting running kinematics under different conditions on a small training data set.

Quantum Gaussian process state: A kernel-inspired state with quantum support data

We introduce the quantum Gaussian process state, motivated via a statistical inference for the wave function supported by a data set of unentangled product states. We show that this condenses down to a compact and expressive parametric form, with a variational flexibility shown to be competitive or surpassing established alternatives. The connections of the state to its roots as a Bayesian inference machine as well as matrix product states, also allow for efficient deterministic training of global states from small training data with enhanced generalization, including on application to frustrated spin physics.

Inducing robustness and plausibility in deep learning optical 3D printer models.

Optical 3D printer models characterize multimaterial 3D printers by predicting optical or visual quantities from material arrangements or tonal values. Their accuracy and robustness to noisy training data are crucial for 3D printed appearance reproduction. In our recent paper [Opt. Express29, 615 (2021)10.1364/OE.410796], we have proposed a pure deep learning (PDL) optical model and a training strategy achieving high accuracy with a moderate number of training samples. Since the PDL model is essentially a black-box without considering any physical grounding, it is sensitive to outliers or noise of the training data and tends to create physically-implausible tonal-to-optical relationships. In this paper, we propose a methodology to narrow down the degrees-of-freedom of deep-learning based optical printer models by inducing physically plausible constraints and smoothness. Our methodology does not need any additional printed samples for training. We use this approach to introduce the robust plausible deep learning (RPDL) optical printer model enhancing robustness to erroneous and noisy training data as well as physical plausibility of the PDL model for selected tonal-to-optical monotonicity relationships. Our experiments on four state-of-the-art multimaterial 3D printers show that the RPDL model not only almost always corrects implausible tonal-to-optical relationships, but also ensures significantly smoother predictions, without sacrificing accuracy. On small training data, it even outperforms the PDL model in accuracy by up to 8% indicating a better generalization ability.

Automated knee cartilage segmentation for heterogeneous clinical MRI using generative adversarial networks with transfer learning.

This study aimed to build a deep learning model to automatically segment heterogeneous clinical MRI scans by optimizing a pre-trained model built from a homogeneous research dataset with transfer learning. Conditional generative adversarial networks pretrained on the Osteoarthritis Initiative MR images was transferred to 30 sets of heterogenous MR images collected from clinical routines. Two trained radiologists manually segmented the 30 sets of clinical MR images for model training, validation and test. The model performance was compared to models trained from scratch with different datasets, as well as two radiologists. A 5-fold cross validation was performed. The transfer learning model obtained an overall averaged Dice coefficient of 0.819, an averaged 95 percentile Hausdorff distance of 1.463 mm, and an averaged average symmetric surface distance of 0.350 mm on the 5 random holdout test sets. A 5-fold cross validation had a mean Dice coefficient of 0.801, mean 95 percentile Hausdorff distance of 1.746 mm, and mean average symmetric surface distance of 0.364 mm. It outperformed other models and performed similarly as the radiologists. A transfer learning model was able to automatically segment knee cartilage, with performance comparable to human, using heterogeneous clinical MR images with a small training data size. In addition, the model proved robust when tested through cross validation and on images from a different vendor. We found it feasible to perform fully automated cartilage segmentation of clinical knee MR images, which would facilitate the clinical application of quantitative MRI techniques and other prediction models for improved patient treatment planning.

Predicting stress–strain curves using transfer learning: Knowledge transfer across polymer composites

The engineering stress–strain curve of a material allows one to determine mechanical properties such as elastic modulus, strength, and toughness. While machine learning has recently been used to predict stress–strain curves, large volumes of experimental data are required to achieve high prediction accuracy. More importantly, conventional machine learning models are not generalizable from one material to another. To address this issue, a novel transfer learning approach is introduced to predict stress–strain curves across different fiber reinforced polymer (FRP) composites fabricated via additive manufacturing. Optimal transport (OT) is integrated with the transfer learning approach by mapping the source label space to the target label space, which further improves knowledge transferability across different FRP composites. The stress–strain curves of the additively manufactured FRP composites are generated under flexural loading conditions. Experimental results show that the OT-integrated transfer learning approach can predict the complex stress–strain curves of FRP composites with small training data. The predictive model achieved a mean absolute percentage error (MAPE) of less than 10%. The extracted modulus and strength from the predicted stress–strain curves achieved MAPEs of less than 5% and 10%, respectively. This work demonstrates that transfer learning has the potential to transform how stress–strain curves are generated.

A Compression-Based Multiple Subword Segmentation for Neural Machine Translation

In this study, we propose a simple and effective preprocessing method for subword segmentation based on a data compression algorithm. Compression-based subword segmentation has recently attracted significant attention as a preprocessing method for training data in neural machine translation. Among them, BPE/BPE-dropout is one of the fastest and most effective methods compared to conventional approaches; however, compression-based approaches have a drawback in that generating multiple segmentations is difficult due to the determinism. To overcome this difficulty, we focus on a stochastic string algorithm, called locally consistent parsing (LCP), that has been applied to achieve optimum compression. Employing the stochastic parsing mechanism of LCP, we propose LCP-dropout for multiple subword segmentation that improves BPE/BPE-dropout, and we show that it outperforms various baselines in learning from especially small training data.

Reduced-cost two-level surrogate antenna modeling using domain confinement and response features

Electromagnetic (EM) simulation tools have become indispensable in the design of contemporary antennas. Still, the major setback of EM-driven design is the associated computational overhead. This is because a single full-wave simulation may take from dozens of seconds up to several hours, thus, the cost of solving design tasks that involve multiple EM analyses may turn unmanageable. This is where faster system representations (surrogates) come into play. Replacing expensive EM-based evaluations by cheap yet accurate metamodels seems to be an attractive solution. Still, in antenna design, application of surrogate models is hindered by the curse of dimensionality. A practical workaround has been offered by the recently reported reference-design-free constrained modeling techniques that restrict the metamodel domain to the parameter space region encompassing high-quality designs. Therein, the domain is established using only a handful of EM-simulations. This paper proposes a novel modeling technique, which incorporates the response feature technology into the constrained modeling framework. Our methodology allows for rendering accurate surrogates using exceptionally small training data sets, at the expense of reducing the generality of the modeling procedure to antennas that exhibit consistent shape of input characteristics. The proposed technique can be employed in other fields that employ costly simulation models (e.g., mechanical or aerospace engineering).

Parameter continuity in time-varying Gauss–Markov models for learning from small training data sets

The Linear time-invariant dynamic models are widely adopted in the industry. In the machine learning domain, such models are known as time-invariant continuous-state hidden Gauss–Markov models. Their super-class, the linear time-varying dynamic models, have relatively sparse applications as predictive models and classifiers of time series. This is typically due to the model complexity and the need for a significantly larger training set than time-invariant models. Without a large training set, a better modeling performance is counteracted by a less robust model. In this paper, we propose the continuity preference of the time-varying parameters of the model, which significantly reduces the required amount of training data while maintaining the modeling performance. We also derive a simple modification of the Expectation–Maximization algorithm incorporating continuity in parameters. The modified algorithm shows robust learning performance. The model performance is demonstrated by experiments on real 6-axis robotic manipulators in a laboratory, the Skoda Auto car producer body shop, and also on a public benchmark data set.

Scattering response modeling scheme based on combined neural network inspired by the equivalent scattering center.

A novel scheme is proposed in this paper to model the complex scattering pattern of radar target with a small training data set. By employing the ideal equivalent scattering center as transfer function, the frequency domain response can be represented by series of parameters so that the aspect and frequency domain dependency can be decoupled, and modeled, independently. In specific, neural network is employed to model the aspect dependency considering the complexity. To maintain the continuity of transformed parameters, a parameter extraction algorithm based on the Orthogonal Matching Pursuit is designed. With the same amount of training set, the proposed scheme exhibits a much better performance than the existing representative modeling techniques such as Geometrical Theory of Diffraction (GTD)-based model, the polynomial scattering center model and so on. At the same time, the training speed of the proposed model is also faster than those techniques.

Dual attention based network for skin lesion classification with auxiliary learning

Skin lesion varies greatly in appearance, and its classification task suffers from large inter-class similarity and intra-class variation, thus the subtle differences of local pattern contained in the skin lesion regions are critical for its classification. In this paper, we propose a dual attention based network for skin lesion classification with auxiliary learning. The dual attention mechanism includes the spatial attention (SA) and the channel attention (CA) modules. The SA module can focus on the skin lesion region feature with reduced irrelevant artifacts features. In the subsequent CA module, it first captures the non-local based global feature of the skin lesion region and then generates the feature channels reweighting vector, which is used to further refine the meaningful local pattern feature contained in the skin lesion region. Therefore, the performance and the interpretability of the proposed network are enhanced at the same time. The proposed auxiliary learning contains two auxiliary supervision branches and KL regularization. The KL regularization makes the two auxiliary supervision branches collaborate with each other during training through mutual knowledge transferring. The introduced strong regularization can guide the dual attention mechanism to focus on the meaningful local pattern features in a weakly supervised manner and make the network avoid overfitting on small training data. Without extra training data, our proposed network can outperform current competition winners on several datasets, regardless of binary- or multi-classification. The proposed network is robust enough and owns strong interpretability which promotes its clinical application.

Automatic electron hologram acquisition of catalyst nanoparticles using particle detection with image processing and machine learning

To enable better statistical analysis of catalyst nanoparticles by high-resolution electron holography, we improved the particle detection accuracy of our previously developed automated hologram acquisition system by using an image classifier trained with machine learning. The detection accuracy of 83% was achieved with the small training data of just 232 images showing nanoparticles by utilizing transfer learning based on VGG16 to train the image classifier. Although the construction of training data generally requires much effort, the time needed to select the training data candidates was significantly shortened by utilizing a pattern matching technique. Experimental results showed that the high-resolution hologram acquisition efficiency was improved by factors of about 100 and 6 compared to a scan method and a pattern-matching-only method, respectively.

Estimation of hypersonic vehicle weight using Physics-Informed neural network supported by knowledge based engineering

Design space exploration is usually required during the conceptual design of hypersonic vehicle. Artificial neural network (ANN) is widely used for issues of design space exploration. However, gaps such as the low prediction accuracy of ANN with small training data and inefficiency of data generation are still unaddressed. In this paper, for investigation of the variation of the maximum takeoff weight (MTOW) of hypersonic vehicles as the configuration changes, a method combing physics-informed neural network (PINN) and knowledge based engineering (KBE) is developed to study the issue. In this method, PINN is used as the predictor of MTOW, the prior knowledge of engineers is able to be embedded into the loss function and guide the training of PINN. Compared with that of traditional neural network, the prediction accuracy of PINN is higher especially with small training data. Moreover, the KBE is used to guide the design process of generating training data and the efficiency of data generation can be improved. In order to account for the validity of the proposed method, the MTOW prediction of hypersonic vehicles with different wing configurations has been investigated. The results showed that the prediction accuracy of neural network can be improved with small training data.

A Mutual Guide Framework for Training Hyperspectral Image Classifiers With Small Data

This article develops a general yet effective hyperspectral image (HSI) classification framework that is trained with small data. To this end, two identically structured but differently initialized classifiers, which are referred to as two base classifiers, are trained in an iterative manner. Each iteration consists of three steps, that is: 1) the two base classifiers that are trained separately on guide data; 2) unclassified data that are processed by the two trained base classifiers; and 3) the classification results with high confidence that are explored as new guide data. In the first iteration, the guide data comprising the original small training data are the same for the two base classifiers. From the second iteration, the guide data for the two base classifiers start becoming different. Specifically, in each iteration, the guide data for training one base classifier keep being augmented by high confidence classification results provided by the other base classifier. It is in such an iterative manner that the two classifiers continuously provide different new guide data for each other, and thus increasingly augment labeled data from the original small training set to a reasonably larger amount of samples in a HSI. We refer to such a training strategy as mutual guide. We develop a mutual guide implementation scheme by exploiting extreme learning machines (ELMs) as base classifiers. Extensive experiments on four public HSI datasets, i.e., Indian Pines (IP), Kennedy Space Center (KSC), University of Pavia (UP), and Salinas (SA), validate the classification effectiveness of our mutual guide framework with small training data.

Power Generation Forecasting of a Solar Photovoltaic Power Plant by a Novel Transfer Learning Technique with Small Solar Radiation and Power Generation Training Data Sets

Power Generation Forecasting of a Solar Photovoltaic Power Plant by a Novel Transfer Learning Technique with Small Solar Radiation and Power Generation Training Data Sets

Radar HRRP Target Recognition Model Based on a Stacked CNN–Bi-RNN With Attention Mechanism

The range resolution of high-resolution wideband radar is much smaller than the target size. Its echo signals tend to be diverse and sensitive to small changes of targets. Therefore, it is difficult to capture and distinguish the features in radar signals. In this article, we propose a radar target recognition pipeline based on a deep nested neural network. The framework consists of three parts: The translation sensitivity of the training data is first addressed in the preprocessing section. The second step is to obtain an embedded representation of the radar echo signals by the combination of the adjustment layer, convolutional neural network (CNN), and the squeeze and excitation (SE) block. Finally, the target is recognized through inputting embedded representation as a time sequence into the stacked bidirectional recurrent neural network (bi-RNN) based on an attention mechanism. Compared with the traditional methods, the proposed deep nested neural network extracts and takes advantage of the features of radar echo signals more effectively, including the envelope features and local physical structural features. The experimental results based on the test data indicate that the proposed method has a great advantage over other methods in the case of large data sets as well as small training data sets and is robust to the small translation of test samples and noises, exhibiting high engineering practical value.

Constructing an Surrogate Model for Pedestrian Protection Using Machine Learning

In order to shorten the design process of industrial products, an alternative evaluation methodology to CAE is needed. Machine learning methodology is one such method. The authors have applied convolutional neural networks (CNN) to regression problems. Although CNNs are highly capable of interpolating nonlinear physical phenomena, the design of input data should be carefully considered when training on less sensitive data with a small training data set. In this paper, CNNs are adapted to predict crash simulations of a finite element method model that mimics the pedestrian leg protection performance test prescribed by JNCAP.