Small Training Dataset Research Articles

A Study on Prediction Skills and Reading Efficiency in College English Based on Optimized BP Networks

In this paper, an in-depth study and analysis of college English prediction skills and reading efficiency are conducted using an optimized BP network algorithm, which is designed separately, and the accuracy of scoring results is verified and the experimental results are analyzed using two datasets; finally, under the guidance of writing feedback theory and data visualization design criteria, a reasonable visualizable writing feedback is created. The learning and training of the BP neural network model revealed that there was significant information loss in the similar nearest-neighbor user dataset used for training, and there were instances where some items were rated by the target user but not by similar nearest-neighbor users. As a result, such data is useless in training, and the neural network loses a significant amount of useful information when learning. To address this issue, this paper proposes using the singular value decomposition technique for filling, which alleviates the sparsity of the filled matrix data and improves the accuracy of recommendations even more. Both scoring models constructed using 1D CNN and LSTM networks belong to the second class of models, and this type of “end-to-end” scoring model does not require feature engineering. Finally, using 650 spoken recordings and their corresponding manual scoring data, the model is trained and tested. The experimental results show that, with a smaller training dataset, the BP network model achieves a better overall scoring performance.

Nonlinear Schrödinger Kernel for Hardware Acceleration of Machine Learning

Alternative machine learning approaches that have extremely low latency and can work with only a small training dataset are needed for applications where the insatiable demands of deep learning methods for computing power and large training data cannot be met. Here we report a new optical accelerator for AI that exploits femtosecond pulses for both data acquisition and computing enabling classification at short time scales for fast optical imaging, sensing, and metrology without increasing data dimensions. Modulation of data onto the spectrum of femtosecond optical pulses followed by projection into a new space using nonlinear optics reduces the latency in the nonlinear classification of certain data by orders of magnitude. This Femtocomputing approach is validated by the classification of various datasets, including brain intracranial pressure, cancer cell imaging, spoken digit recognition, and the classic exclusive OR benchmark for nonlinear operation. The concept is demonstrated by seeding the nonlinear effect that is responsible for many fascinating natural phenomena, such as optical rogue waves. Stimulation of nonlinear optical interactions with spectrally modulated data transforms the data such that a computationally-light linear algorithm can learn a nonlinear decision boundary that separates the data into the correct classes. Since the optical kernel is not trained, its performance is inevitably data-dependent. Quantitative comparison with a popular numerical kernel offers insights into how this physical technique accelerates inference. Single-shot operation is demonstrated using time stretch data acquisition.

A Synaptic Pruning-Based Spiking Neural Network for Hand-Written Digits Classification.

A spiking neural network model inspired by synaptic pruning is developed and trained to extract features of hand-written digits. The network is composed of three spiking neural layers and one output neuron whose firing rate is used for classification. The model detects and collects the geometric features of the images from the Modified National Institute of Standards and Technology database (MNIST). In this work, a novel learning rule is developed to train the network to detect features of different digit classes. For this purpose, randomly initialized synaptic weights between the first and second layers are updated using average firing rates of pre- and postsynaptic neurons. Then, using a neuroscience-inspired mechanism named, “synaptic pruning” and its predefined threshold values, some of the synapses are deleted. Hence, these sparse matrices named, “information channels” are constructed so that they show highly specific patterns for each digit class as connection matrices between the first and second layers. The “information channels” are used in the test phase to assign a digit class to each test image. In addition, the role of feed-back inhibition as well as the connectivity rates of the second and third neural layers are studied. Similar to the abilities of the humans to learn from small training trials, the developed spiking neural network needs a very small dataset for training, compared to the conventional deep learning methods that have shown a very good performance on the MNIST dataset. This work introduces a new class of brain-inspired spiking neural networks to extract the features of complex data images.

Parameter continuity in time-varying Gauss–Markov models for learning from small training data sets

The Linear time-invariant dynamic models are widely adopted in the industry. In the machine learning domain, such models are known as time-invariant continuous-state hidden Gauss–Markov models. Their super-class, the linear time-varying dynamic models, have relatively sparse applications as predictive models and classifiers of time series. This is typically due to the model complexity and the need for a significantly larger training set than time-invariant models. Without a large training set, a better modeling performance is counteracted by a less robust model. In this paper, we propose the continuity preference of the time-varying parameters of the model, which significantly reduces the required amount of training data while maintaining the modeling performance. We also derive a simple modification of the Expectation–Maximization algorithm incorporating continuity in parameters. The modified algorithm shows robust learning performance. The model performance is demonstrated by experiments on real 6-axis robotic manipulators in a laboratory, the Skoda Auto car producer body shop, and also on a public benchmark data set.

Scattering response modeling scheme based on combined neural network inspired by the equivalent scattering center.

A novel scheme is proposed in this paper to model the complex scattering pattern of radar target with a small training data set. By employing the ideal equivalent scattering center as transfer function, the frequency domain response can be represented by series of parameters so that the aspect and frequency domain dependency can be decoupled, and modeled, independently. In specific, neural network is employed to model the aspect dependency considering the complexity. To maintain the continuity of transformed parameters, a parameter extraction algorithm based on the Orthogonal Matching Pursuit is designed. With the same amount of training set, the proposed scheme exhibits a much better performance than the existing representative modeling techniques such as Geometrical Theory of Diffraction (GTD)-based model, the polynomial scattering center model and so on. At the same time, the training speed of the proposed model is also faster than those techniques.

Neural network training using ℓ1-regularization and bi-fidelity data

With the capability of accurately representing a functional relationship between the inputs of a physical system's model and output quantities of interest, neural networks have become popular for surrogate modeling in scientific applications. However, as these networks are over-parameterized, their training often requires a large amount of data. To prevent overfitting and improve generalization error, regularization based on, e.g., ℓ1- and ℓ2-norms of the parameters is applied. Similarly, multiple connections of the network may be pruned to increase sparsity in the network parameters. In this paper, we explore the effects of sparsity promoting ℓ1-regularization on training neural networks when only a small training dataset from a high-fidelity model is available. As opposed to standard ℓ1-regularization that is known to be inadequate, we consider two variants of ℓ1-regularization informed by the parameters of an identical network trained using data from lower-fidelity models of the problem at hand. These bi-fidelity strategies are generalizations of transfer learning of neural networks that uses the parameters learned from a large low-fidelity dataset to efficiently train networks for a small high-fidelity dataset. We also compare the bi-fidelity strategies with two ℓ1-regularization methods that only use the high-fidelity dataset but with similar computational cost for generating the training datasets. Three numerical examples for propagating uncertainty through physical systems are used to show that the proposed bi-fidelity ℓ1-regularization strategies produce errors that are one order of magnitude smaller than those of networks trained only using datasets from the high-fidelity models.

Computing With Networks of Chemical Oscillators and its Application for Schizophrenia Diagnosis.

Chemical reactions are responsible for information processing in living organisms, yet biomimetic computers are still at the early stage of development. The bottom-up design strategy commonly used to construct semiconductor information processing devices is not efficient for chemical computers because the lifetime of chemical logic gates is usually limited to hours. It has been demonstrated that chemical media can efficiently perform a specific function like labyrinth search or image processing if the medium operates in parallel. However, the number of parallel algorithms for chemical computers is very limited. Here we discuss top-down design of such algorithms for a network of chemical oscillators that are coupled by the exchange of reaction activators. The output information is extracted from the number of excitations observed on a selected oscillator. In our model of a computing network, we assume that there is an external factor that can suppress oscillations. This factor can be applied to control the nodes and introduce input information for processing by a network. We consider the relationship between the number of oscillation nodes and the network accuracy. Our analysis is based on computer simulations for a network of oscillators described by the Oregonator model of a chemical oscillator. As the example problem that can be solved with an oscillator network, we consider schizophrenia diagnosis on the basis of EEG signals recorded using electrodes located at the patient’s scalp. We demonstrated that a network formed of interacting chemical oscillators can process recorded signals and help to diagnose a patient. The parameters of considered networks were optimized using an evolutionary algorithm to achieve the best results on a small training dataset of EEG signals recorded from 45 ill and 39 healthy patients. For the optimized networks, we obtained over 82% accuracy of schizophrenia detection on the training dataset. The diagnostic accuracy can be increased to almost 87% if the majority rule is applied to answers of three networks with different number of nodes.

AraConv: Developing an Arabic Task-Oriented Dialogue System Using Multi-Lingual Transformer Model mT5

Task-oriented dialogue systems (DS) are designed to help users perform daily activities using natural language. Task-oriented DS for English language have demonstrated promising performance outcomes; however, developing such systems to support Arabic remains a challenge. This challenge is mainly due to the lack of Arabic dialogue datasets. This study introduces the first Arabic end-to-end generative model for task-oriented DS (AraConv), which uses the multi-lingual transformer model mT5 with different settings. We also present an Arabic dialogue dataset (Arabic-TOD) and used it to train and test the proposed AraConv model. The results obtained are reasonable compared to those reported in the studies of English and Chinese using the same mono-lingual settings. To avoid problems associated with a small training dataset and to improve the AraConv model’s results, we suggest joint-training, in which the model is jointly trained on Arabic dialogue data and data from one or two high-resource languages such as English and Chinese. The findings indicate the AraConv model performed better in the joint-training setting than in the mono-lingual setting. The results obtained from AraConv on the Arabic dialogue dataset provide a baseline for other researchers to build robust end-to-end Arabic task-oriented DS that can engage with complex scenarios.

Deep learning image transmission through a multimode fiber based on a small training dataset.

An improved deep neural network incorporating attention mechanism and DSSIM loss function (AM_U_Net) is used to recover input images with speckles transmitted through a multimode fiber (MMF). The network is trained on a relatively small dataset and demonstrates an optimal reconstruction ability and generalization ability. Furthermore, a bimodal fusion method is developed based on S polarization and P polarization speckles, greatly improving the recognition accuracy. These findings prove that AM_U_Net has remarkable capabilities for information recovery and transfer learning and good tolerance and robustness under different MMF transmission conditions, indicating its significant application potential in medical imaging and secure communication.

Sheared edge defect segmentation using a convolutional U-Net for quantified quality assessment of fine blanked workpieces

Fine blanking allows for the economic mass production of sheet metal parts of high dimensional accuracy. The smooth cut section at the sheared edge is an important quality characteristic that is, however, reduced by the formation of tearings and fractures in the cut-off zone. At present, the smooth cut section cannot be assessed immediately as no capable methods for a 100% inline quality control of sheared edges for fine blanking exist, causing high scrap rates. To address this deficit this work proposes U-Nets to segment tearings and measure the cut-off height at sheared edges. A systematic model search together with a transfer learning strategy is conducted based on a dataset of 80 images of sheared edges of fine blanked reference workpieces. After a hyperparameter optimization, trained models are able to measure the cut-off height on the test dataset with a mean relative error of 1.1% and a mean absolute error of <1 px corresponding to an average measurement error of 7.5 μm. Besides the natural ability to incorporate additional disturbances in a real-world application, investigations of the reproducibility and the impact of the dataset size indicate that the proposed U-Net model is easy to set up and achieves a comparable average performance of 1.14 ± 0.31 px (17.1 ± 3.7 μm) even for non-optimal model initialization and small training datasets. For this reason, the shown approach is a promising solution for an inline machine vision system.

Regularizing deep networks with label geometry for accurate object localization on small training datasets

Localization is a critical subtask in object detection, which is closely related to spatial information of objects. Most current detectors simply rely on the fitting ability of deep neural networks to regress towards numerical targets such as coordinates of object boxes. Training deep networks for sufficient fitting ability requires a large number of annotations that are expensive to obtain. In this work, we fully exploit limited annotations by extracting label geometry to improve localization performance on small datasets. We generate distance transform of bounding box edges according to localization labels, with which we supervise intermediate outputs of networks pixel by pixel to reconstruct object geometry for localization. Distance transform is sensitive to box edges and provides geometric gradients flowing into boundaries. We learn such gradients to enhance geometric-aware features through a coupled training with regression, and use it to refine regressed boxes in an evolutionary manner in inference. Extensive experiments are implemented to demonstrate the effectiveness of our method. Our method can be applied in applications that required human-machine interaction, such as the driver-assistance system in autonomous driving, by providing accurate detections to assist humans in making better decisions.

Tooth Instance Segmentation on Panoramic Dental Radiographs Using U-Nets and Morphological Processing

Automatic teeth segmentation in panoramic x-ray images is an important research subject of the image analysis in dentistry. In this study, we propose a post-processing stage to obtain a segmentation map in which the objects in the image are separated, and apply this technique to tooth instance segmentation with U-Net network. The post-processing consists of grayscale morphological and filtering operations, which are applied to the sigmoid output of the network before binarization. A dice overlap score of 95.4±0.3% is obtained in overall teeth segmentation. The proposed post-processing stages reduce the mean error of tooth count to 6.15%, whereas the error without post-processing is 26.81%. The performances of both segmentation and tooth counting are the highest in the literature, to our knowledge. Moreover, this is achieved by using a relatively small training dataset, which consists of 105 images. Although the aim in this study is to segment tooth instances, the presented method is applicable to similar problems in other domains, such as separating the cell instances.

Site-agnostic 3D dose distribution prediction with deep learning neural networks.

Typically, the current dose prediction models are limited to small amounts of data and require retraining for a specific site, often leading to suboptimal performance. We propose a site-agnostic, three-dimensional dose distribution prediction model using deep learning that can leverage data from any treatment site, thus increasing the total data available to train the model. Applying our proposed model to a new target treatment site requires only a brief fine-tuning of the model to the new data and involves no modifications to the model input channels or its parameters. Thus, it can be efficiently adapted to a different treatment site, even with a small training dataset. This study uses two separate datasets/treatment sites: data from patients with prostate cancer treated with intensity-modulated radiation therapy (source data), and data from patients with head-and-neck cancer treated with volumetric-modulated arc therapy (target data). We first developed a source model with 3D UNet architecture, trained from random initial weights on the source data. We evaluated the performance of this model on the source data. We then studied the generalizability of the model to the new target dataset via transfer learning. To do this, we built three more models, all with the same 3D UNet architecture: target model, adapted model, and combined model. The source and target models were trained on the source and target data from random initial weights, respectively. The adapted model fine-tuned the source model to the target domain by using the target data. Finally, the combined model was trained from random initial weights on a combined data pool consisting of both target and source datasets. We tested all four models on the target dataset and evaluated quantitative dose-volume histogram metrics for the planning target volume (PTV) and organs at risk (OARs). When tested on the source treatment site, the source model accurately predicted the dose distributions with average (mean, max) absolute dose errors of (0.32%±0.14, 2.37%±0.93) (PTV) relative to the prescription dose, and highest mean dose error of 1.68%±0.76, and highest max dose error of 5.47%± 3.31 for femoral head right. The error in PTV dose coverage prediction is 3.21%±1.51 for D98 , 3.04%±1.69 for D95 , and 1.83%±1.01 for D02 . Averaging across all OARs, the source model predicted the OAR mean dose within 1.38% and the OAR max dose within 3.64%. For the target treatment site, the target model average (mean, max) absolute dose errors relative to the prescription dose for the PTV were (1.08%±0.95, 2.90%±1.35). Left cochlea had the highest mean and max dose errors of 5.37%±5.82 and 8.33%±8.88, respectively. The errors in PTV dose coverage prediction for D98 and D95 were 2.88%±1.59 and 2.55%±1.28, respectively. The target model can predict the OAR mean dose within 2.43% and the OAR max dose within 4.33% on average across all OARs. We developed a site-agnostic model for three-dimensional dose prediction and tested its adaptability to a new target treatment site via transfer learning. Our proposed model can make accurate predictions with limited training data.

Automated Breast Cancer Detection Models Based on Transfer Learning.

Breast cancer is among the leading causes of mortality for females across the planet. It is essential for the well-being of women to develop early detection and diagnosis techniques. In mammography, focus has contributed to the use of deep learning (DL) models, which have been utilized by radiologists to enhance the needed processes to overcome the shortcomings of human observers. The transfer learning method is being used to distinguish malignant and benign breast cancer by fine-tuning multiple pre-trained models. In this study, we introduce a framework focused on the principle of transfer learning. In addition, a mixture of augmentation strategies were used to prevent overfitting and produce stable outcomes by increasing the number of mammographic images; including several rotation combinations, scaling, and shifting. On the Mammographic Image Analysis Society (MIAS) dataset, the proposed system was evaluated and achieved an accuracy of 89.5% using (residual network-50) ResNet50, and achieved an accuracy of 70% using the Nasnet-Mobile network. The proposed system demonstrated that pre-trained classification networks are significantly more effective and efficient, making them more acceptable for medical imaging, particularly for small training datasets.

A Multi-Domain Collaborative Transfer Learning Method with Multi-Scale Repeated Attention Mechanism for Underwater Side-Scan Sonar Image Classification

Due to the strong speckle noise caused by the seabed reverberation which makes it difficult to extract discriminating and noiseless features of a target, recognition and classification of underwater targets using side-scan sonar (SSS) images is a big challenge. Moreover, unlike classification of optical images which can use a large dataset to train the classifier, classification of SSS images usually has to exploit a very small dataset for training, which may cause classifier overfitting. Compared with traditional feature extraction methods using descriptors—such as Haar, SIFT, and LBP—deep learning-based methods are more powerful in capturing discriminating features. After training on a large optical dataset, e.g., ImageNet, direct fine-tuning method brings improvement to the sonar image classification using a small-size SSS image dataset. However, due to the different statistical characteristics between optical images and sonar images, transfer learning methods—e.g., fine-tuning—lack cross-domain adaptability, and therefore cannot achieve very satisfactory results. In this paper, a multi-domain collaborative transfer learning (MDCTL) method with multi-scale repeated attention mechanism (MSRAM) is proposed for improving the accuracy of underwater sonar image classification. In the MDCTL method, low-level characteristic similarity between SSS images and synthetic aperture radar (SAR) images, and high-level representation similarity between SSS images and optical images are used together to enhance the feature extraction ability of the deep learning model. Using different characteristics of multi-domain data to efficiently capture useful features for the sonar image classification, MDCTL offers a new way for transfer learning. MSRAM is used to effectively combine multi-scale features to make the proposed model pay more attention to the shape details of the target excluding the noise. Experimental results of classification show that, in using multi-domain data sets, the proposed method is more stable with an overall accuracy of 99.21%, bringing an improvement of 4.54% compared with the fine-tuned VGG19. Results given by diverse visualization methods also demonstrate that the method is more powerful in feature representation by using the MDCTL and MSRAM.

(Retracted) Estimation of human age by features of face and eyes based on multilevel feature convolutional neural network

Age estimation can be effectively used in security, human–computer interaction, entertainment, and audio-visual fields. However, current face-based age recognition algorithms are not highly accurate due to factors, such as face makeup and plastic surgery. Among the various feature points of a human face, the eyes are the most difficult part to be modified. And with the changes of age, the lens, cornea, and vitreous of eyes will also change accordingly. Therefore, we believe that paying attention to the local feature of the eye can improve the accuracy of age estimation to a certain extent. The multilevel feature convolutional neural network (MLFCNN) is proposed, which values eyes features and combines them with face features to jointly estimate human age. MLFCNN performs two rounds of estimation based on extracted features. First, the age range of the sample is estimated as the age group, and then on this basis, the fine age of the sample is further estimated. Field tests show that the mean absolute error of MLFCNN is 2.87, which is lower than other network models tested. When the MLFCNN estimated age did not differ more than 4 years from the actual age of the sample, the network predictions were considered correct (i.e., the tolerable age error was 4). The age estimation accuracy of MLFCNN under this condition was 91.14%. And 98.32% accuracy can be reached when the tolerable age error is 6. Ablation experiments verify that the fusion of ocular features and facial features can improve the performance of the age estimation network. In addition, it can still maintain a good performance under small training dataset.

Meta domain generalization for smart manufacturing: Tool wear prediction with small data

Conventional machine learning based predictive modeling methods require large volumes of training data; however, collecting large training data is very labor-intensive and expensive in real-world applications such as manufacturing. Therefore, small data is a common challenge for machine learning approaches. To address this issue, we introduce a meta domain generalization method and demonstrate its effectiveness for tool wear prediction when a small set of training data are collected under different operating conditions. Domain generalization aims to generalize a machine learning model in a source domain to a target domain where there are small or no data. Meta-learning aims to optimize the initial hyper-parameters of a model for improving prediction accuracy. In this paper, we employ meta-learning to improve model pre-training for domain generalization for tool wear prediction. The proposed meta domain generalization involves three phases: source data split, meta-learning pre-training, and model fine-tuning. First, a training dataset from source domain is divided into pre-training and fine-tuning subsets based on the prior knowledge about target domain, i.e. the cutting conditions, to address the domain shift problem. Then, meta-learning is used to optimize model pre-training to better adapt to fine-tuning subset and improve model fine-tuning. Finally, model fine-tuning is carried out to enhance model generalization to the target domain. We conduct extensive experiments and justify the approach. The results show that meta domain generalization can predict tool wear under different operating conditions accurately with small data. We also find that data split optimization significantly affects the performance of meta domain generalization.

Radar HRRP Target Recognition Model Based on a Stacked CNN–Bi-RNN With Attention Mechanism

The range resolution of high-resolution wideband radar is much smaller than the target size. Its echo signals tend to be diverse and sensitive to small changes of targets. Therefore, it is difficult to capture and distinguish the features in radar signals. In this article, we propose a radar target recognition pipeline based on a deep nested neural network. The framework consists of three parts: The translation sensitivity of the training data is first addressed in the preprocessing section. The second step is to obtain an embedded representation of the radar echo signals by the combination of the adjustment layer, convolutional neural network (CNN), and the squeeze and excitation (SE) block. Finally, the target is recognized through inputting embedded representation as a time sequence into the stacked bidirectional recurrent neural network (bi-RNN) based on an attention mechanism. Compared with the traditional methods, the proposed deep nested neural network extracts and takes advantage of the features of radar echo signals more effectively, including the envelope features and local physical structural features. The experimental results based on the test data indicate that the proposed method has a great advantage over other methods in the case of large data sets as well as small training data sets and is robust to the small translation of test samples and noises, exhibiting high engineering practical value.

Constructing an Surrogate Model for Pedestrian Protection Using Machine Learning

In order to shorten the design process of industrial products, an alternative evaluation methodology to CAE is needed. Machine learning methodology is one such method. The authors have applied convolutional neural networks (CNN) to regression problems. Although CNNs are highly capable of interpolating nonlinear physical phenomena, the design of input data should be carefully considered when training on less sensitive data with a small training data set. In this paper, CNNs are adapted to predict crash simulations of a finite element method model that mimics the pedestrian leg protection performance test prescribed by JNCAP.

Memory-Modulated Transformer Network for Heterogeneous Face Recognition

Heterogeneous face recognition (HFR) aims at matching face images across different domains. It is challenging due to the severe domain discrepancies and overfitting caused by small training datasets. Some researchers apply a &#x201C;recognition via generation&#x201D; strategy and propose to solve the problem by translating images from a given domain into the visual domain. However, in many HFR tasks such as near-infrablack HFR, there is no paiblack data, which makes it an unsupervised generation. Pose variations, background differences, and many other factors present challenges. Moreover, the generated results lack diversity since many previous works regard this image translation as a &#x201C;one-to-one&#x201D; generation task. Considering the information deficiency in the input images, we propose to formulate this image translation process as a &#x201C;one-to-many&#x201D; generation problem. Specifically, we introduce reference images to guide the generation process. We propose a memory module to explore the prototypical style patterns of the reference domain. After self-supervised updating, the memory items are attentively aggregated to represent the style information. Moreover, to subtly fuse the contents of input images with the style of reference images, we propose a novel style transformer module. Specifically, we crop the encoded input and reference feature maps into patches, and use the style transformer to establish long-range dependencies between the input and reference patches. Thus, the style of every input patch is transferblack based on those of the most relevant reference patches. Extensive experiments on multiple datasets for various HFR tasks, including NIR-VIS, thermal-VIS, sketch-photo, and gray-RGB, are conducted. The robustness and effectiveness of the proposed MMTN are demonstrated both quantitatively and qualitatively.