Small Amount Of Training Data Research Articles

Cloze-Style Data Augmentation for Few-Shot Intent Recognition

Intent recognition aims to identify users’ potential intents from their utterances, which is a key component in task-oriented dialog systems. A real challenge, however, is that the number of intent categories has grown faster than human-annotated data, resulting in only a small amount of data being available for many new intent categories. This lack of data leads to the overfitting of traditional deep neural networks on a small amount of training data, which seriously affects practical applications. Hence, some researchers have proposed few-shot learning should address the data-scarcity issue. One of the efficient methods is text augmentation, which always generates noisy or meaningless data. To address these issues, we propose leveraging the knowledge in pre-trained language models and constructed the cloze-style data augmentation (CDA) model. We employ unsupervised learning to force the augmented data to be semantically similar to the initial input sentences and contrastive learning to enhance the uniqueness of each category. Experimental results on CLINC-150 and BANKING-77 datasets show the effectiveness of our proposal by its beating of the competitive baselines. In addition, we conducted an ablation study to verify the function of each module in our models, and the results illustrate that the contrastive learning module plays the most important role in improving the recognition accuracy.

Dynamics-disentangled deep learning model for multi-cycle prediction of unsteady flow field

The prediction of an unsteady flow field inherently involving high-dimensional dynamics is challenging. The multi-cycle prediction is especially difficult due to the inevitably accumulated errors over time. A novel deep learning model is proposed in this paper to disentangle the high-dimensional dynamics into two separate attributes that, respectively, represent spatial and temporal dynamics. A continuous mapping of temporal dynamics is subsequently constructed, which alleviates the error accumulation and, thus, contributes to the long-term prediction of the unsteady flow field. The dynamics-disentangled deep learning model (D3LM) processes sequential image data of the unsteady flow field and is constituted by three sub-networks, an encoder introducing a stochastic latent variable to explicitly model the low-order temporal dynamics (called varying attribute herein) and extracting multi-level representations of spatial dynamics (called consistent attribute herein), a decoder integrating the disentangled attributes and generating a future flow field, and a discriminator improving the quality of the predicted flow field. The proposed model is evaluated by two simulated datasets of unsteady flows around a circular cylinder at divergent Reynolds numbers. Benefiting from modeling the continuous distribution of temporal dynamics with the stochastic latent variable, the proposal can give multi-cycle future predictions with high accuracy both spatially and temporally on the two datasets with a small amount of training data. Our work demonstrates the potential practicability of deep learning techniques for modeling the long-term nonlinear laws of unsteady flow.

Automatic depression score estimation with word embedding models

Depression is one of the most common mental health illnesses. The biggest obstacle lies in an efficient and early detection of the disorder. Self-report questionnaires are the instruments used by medical experts to elaborate a diagnosis. These questionnaires were designed by analyzing different depressive symptoms. However, factors such as social stigmas negatively affect the success of traditional methods. This paper presents a novel approach for automatically estimating the degree of depression in social media users. In this regard, we addressed the task Measuring the Severity of the Signs of Depression of eRisk 2020, an initiative in the CLEF Conference. We aimed to explore neural language models to exploit different aspects of the subject’s writings depending on the symptom to capture. We devised two distinct methods based on the symptoms’ sensitivity in terms of willingness on commenting about them publicly. The first exploits users’ general language based on their publications. The second seeks more direct evidence from publications that specifically mention the symptoms concerns. Both methods automatically estimate the Beck Depression Inventory (BDI-II) total score. For evaluating our proposals, we used benchmark Reddit data for depression severity estimation. Our findings showed that approaches based on neural language models are a feasible alternative for estimating depression rating scales, even when small amounts of training data are available.

FusionNet: A Convolution–Transformer Fusion Network for Hyperspectral Image Classification

In recent years, deep-learning-based hyperspectral image (HSI) classification networks have become one of the most dominant implementations in HSI classification tasks. Among these networks, convolutional neural networks (CNNs) and attention-based networks have prevailed over other HSI classification networks. While convolutional neural networks with perceptual fields can effectively extract local features in the spatial dimension of HSI, they are poor at capturing the global and sequential features of spectral–spatial information; networks based on attention mechanisms, for example, Transformer, usually have better ability to capture global features, but are relatively weak in discriminating local features. This paper proposes a fusion network of convolution and Transformer for HSI classification, known as FusionNet, in which convolution and Transformer are fused in both serial and parallel mechanisms to achieve the full utilization of HSI features. Experimental results demonstrate that the proposed network has superior classification results compared to previous similar networks, and performs relatively well even on a small amount of training data.

Identifying self-admitted technical debt in issue tracking systems using machine learning

Technical debt is a metaphor indicating sub-optimal solutions implemented for short-term benefits by sacrificing the long-term maintainability and evolvability of software. A special type of technical debt is explicitly admitted by software engineers (e.g. using a TODO comment); this is called Self-Admitted Technical Debt or SATD. Most work on automatically identifying SATD focuses on source code comments. In addition to source code comments, issue tracking systems have shown to be another rich source of SATD, but there are no approaches specifically for automatically identifying SATD in issues. In this paper, we first create a training dataset by collecting and manually analyzing 4,200 issues (that break down to 23,180 sections of issues) from seven open-source projects (i.e., Camel, Chromium, Gerrit, Hadoop, HBase, Impala, and Thrift) using two popular issue tracking systems (i.e., Jira and Google Monorail). We then propose and optimize an approach for automatically identifying SATD in issue tracking systems using machine learning. Our findings indicate that: 1) our approach outperforms baseline approaches by a wide margin with regard to the F1-score; 2) transferring knowledge from suitable datasets can improve the predictive performance of our approach; 3) extracted SATD keywords are intuitive and potentially indicating types and indicators of SATD; 4) projects using different issue tracking systems have less common SATD keywords compared to projects using the same issue tracking system; 5) a small amount of training data is needed to achieve good accuracy.

A novel few-shot classification framework for diabetic retinopathy detection and grading

Diabetes Retinopathy (DR) is a major microvascular complication of diabetes. Computer-Aided Diagnosis (CAD) tools for DR management are primarily developed using Artificial Intelligence (AI) methods, such as machine and deep learning algorithms. DR diagnostic tools have been developed in recent years using deep learning models. Thus, these models require large amounts of data for training. Consequently, these huge amounts of data are not balanced due to fewer cases in the dataset. To solve the problems associated with training models with small datasets, such as overfitting and poor approximation, this paper proposes a paradigm called Few-Shot Learning (FSL) which uses a relatively small amount of training data to train the models effectively. This paper proposes a novel prototype network, a type of FSL classification network capable of grading and detecting DR based on attention. The DRNet framework uses episodic learning to train its model on few-shot classification tasks. We developed a DRNet based on the APTOS2019 dataset for diabetic detection and grading. In the proposed network, aggregated transformations and gradient activations of classes are leveraged to design the attention mechanism to capture image representations. As a result, the system achieves 99.73 % accuracy, 99.82 % sensitivity, 99.63 % specificity in DR detection, 98.18 % accuracy, 97.41% sensitivity, and 99.55% specificity in DR grading. An analysis of objective performance metrics and model interpretation shows that the proposed model can detect DR more efficiently and grade the severity more accurately when using unseen fundus images than existing state-of-the-art methods.Therefore, this tool could help provide a second opinion to an ophthalmologist about the severity level of DR.

Evaluation of transfer learning models for predicting the lateral strength of reinforced concrete columns

Transfer learning aims to extract knowledge from one or more source tasks and apply the knowledge to a different task for more accurate predictions. The main purposes of this study are to investigate different knowledge transfer techniques, apply them to accurately predict the lateral strength of reinforced concrete columns with only a small amount of training data, and compare the transferability of each method. According to the various source and target domains, three different experiments are designed to directly compare the performance across section type, shear reinforcement area, and concrete compressive strength. In all cases in this study, knowledge transfer techniques show better prediction performance than the models trained without any knowledge transfer techniques. Therefore, we can conclude that transferring pre-trained knowledge from the source domain enables a model to better explain the response variable in the target domain. The performance improvement is particularly emphasized when the available data for the target domain is small. Thus, transfer learning can be one way to address the data scarcity problem in structural engineering. Furthermore, transferring the pre-trained knowledge is more associated with the underlying physical relationship between the source and the target domains and less associated with the discrepancy between the source and the target domain distributions.

GANs for generation of synthetic ultrasound images from small datasets

Abstract The task of medical image classification is increasingly supported by algorithms. Deep learning methods like convolutional neural networks (CNNs) show superior performance in medical image analysis but need a high-quality training dataset with a large number of annotated samples. Particularly in the medical domain, the availability of such datasets is rare due to data privacy or the lack of data sharing practices among institutes. Generative adversarial networks (GANs) are able to generate high quality synthetic images. This work investigates the capabilities of different state-of-the-art GAN architectures in generating realistic breast ultrasound images if only a small amount of training data is available. In a second step, these synthetic images are used to augment the real ultrasound image dataset utilized for training CNNs. The training of both GANs and CNNs is conducted with systematically reduced dataset sizes. The GAN architectures are capable of generating realistic ultrasound images. GANs using data augmentation techniques outperform the baseline Style- GAN2 with respect to the Frechet Inception distance by up to 64.2%. CNN models trained with additional synthetic data outperform the baseline CNN model using only real data for training by up to 15.3% with respect to the F1 score, especially for datasets containing less than 100 images. As a conclusion, GANs can successfully be used to generate synthetic ultrasound images of high quality and diversity, improve classification performance of CNNs and thus provide a benefit to computer-aided diagnostics.

Data-oriented description of texture-dependent anisotropic material behavior

In metallurgical processes, as for example cold rolling or deep drawing of sheet metal, it is frequently observed that the crystallographic texture, and with it the anisotropic mechanical properties of a material, evolve dynamically. Hence, to describe such processes, it is necessary to model the functional dependence of anisotropic material parameters on the texture, which itself can vary locally with the different plastic strain histories. In this work, we present a new data-oriented approach to parametrize the anisotropic yield function Barlat Yld2004-18p from micromechanical simulations for different textures. This is accomplished by applying supervised machine learning (ML) methods to express the relationship between different crystallographic textures and the material parameters of the yield function. The crystallographic textures are chosen to vary continuously between a random texture on the one hand side, and a unimodal Goss or Copper texture the other. These crystallographic textures are rather common in sheet metal forming. In this way, furthermore, the transition from isotropic plasticity to a rather severe case of anisotropy can be modeled, which is thought to mimic the dynamical evolution of the texture in a metallurgical process. It is found that a regularization strategy is necessary to circumvent the known non-uniqueness between Yld2004-18p parameters and the resulting plastic yield behavior. After this regularization, a unique relationship between the material parameters and the yield onset is established, making it possible to train different ML models with excellent accuracy and generalization properties to anisotropic plastic material behavior. The trained ML models are able to reliably predict the coefficients of unknown textures even with a small amount of training data and, thus, to correctly represent the yield behavior resulting from the various textures. The proposed method represents an efficient extension of the description of anisotropic plastic yielding as it establishes a data-oriented way to explicitly consider microstructural parameters in the material description, which opens new pathways to formulate material models that include the process history.

Indoor Localization Method of Personnel Movement Based on Non-Contact Electrostatic Potential Measurements.

The indoor localization of people is the key to realizing “smart city” applications, such as smart homes, elderly care, and an energy-saving grid. The localization method based on electrostatic information is a passive label-free localization technique with a better balance of localization accuracy, system power consumption, privacy protection, and environmental friendliness. However, the physical information of each actual application scenario is different, resulting in the transfer function from the human electrostatic potential to the sensor signal not being unique, thus limiting the generality of this method. Therefore, this study proposed an indoor localization method based on on-site measured electrostatic signals and symbolic regression machine learning algorithms. A remote, non-contact human electrostatic potential sensor was designed and implemented, and a prototype test system was built. Indoor localization of moving people was achieved in a 5 m × 5 m space with an 80% positioning accuracy and a median error absolute value range of 0.4–0.6 m. This method achieved on-site calibration without requiring physical information about the actual scene. It has the advantages of low computational complexity and only a small amount of training data is required.

A mechanism informed neural network for predicting machining deformation of annular parts

Controlling machining deformation of annular parts is crucial for ensuring the performance of high value products and equipment. For example, during manufacturing of critical parts in aircrafts and spacecrafts, accurate prediction of machining deformation is the basis for guiding the formulation of deformation control strategies. However, due to the complexity of the machining deformation of annular parts, existing methods still have limitations in accurate prediction. To this end, this paper proposes a mechanism informed neural network (MINN) to predict machining deformation of annular parts. MINN is realized by establishing the dual sub-networks structure and using enhanced loss functions with the consideration of the deformation mechanism model characteristics of annular parts. The deformation was decomposed into the axisymmetric portion and the non-axisymmetric portion according to the deformation superposition principle, and modeled separately based on the thin-shell theory and Fourier series. Experiment results showed that the proposed method could predict the machining deformation of annular parts more accurately and stably with a small amount of training data, compared with previous methods.

An Improved U-Net Segmentation Model That Integrates a Dual Attention Mechanism and a Residual Network for Transformer Oil Leakage Detection

Accurately detecting oil leakage from a power transformer is important to maintain its normal operation. Deep learning (DL) methods have achieved satisfactory performance in automatic oil detection, but challenges remain due to the small amount of training data and oil targets with large variations in position, shape, and scale. To manage these issues, we propose a dual attention residual U-net (DAttRes-Unet) within a U-net architecture that extensively uses a residual network as well as spatial and channel-wise attention modules. To overcome the vanishing gradient problem due to deeper layers and a small amount of training data, a residual module from ResNet18 is used to construct the encoder path in the U-net framework. Meanwhile, to overcome the issue of training difficulty for the network, inspired by the advantage of transfer learning, initial network parameters in the encoder are obtained from the pre-trained ResNet18 on the ImageNet dataset. Further, in the decoder path, spatial attention and channel attention are integrated to highlight oil-stained regions while suppressing the background or irrelevant parts/channels. To facilitate the acquisition of the fluorescence images of the transformer, we designed a portable acquisition device integrating an ultraviolet light source and a digital camera. The proposed network is trained on the amount of fluorescence images after data augmentation is used and tested on actual fluorescence images. The experiment results show that the proposed DAttRes-Unet network can recognize oil-stained regions with a high accuracy of 98.49% for various shapes and scales of oil leakage.

Automated segmentation of head CT scans for computer-assisted craniomaxillofacial surgery applying a hierarchical patch-based stack of convolutional neural networks

PurposeComputer-assisted techniques play an important role in craniomaxillofacial surgery. As segmentation of three-dimensional medical imaging represents a cornerstone for these procedures, the present study was aiming at investigating a deep learning approach for automated segmentation of head CT scans.MethodsThe deep learning approach of this study was based on the patchwork toolbox, using a multiscale stack of 3D convolutional neural networks. The images were split into nested patches using a fixed 3D matrix size with decreasing physical size in a pyramid format of four scale depths. Manual segmentation of 18 craniomaxillofacial structures was performed in 20 CT scans, of which 15 were used for the training of the deep learning network and five were used for validation of the results of automated segmentation. Segmentation accuracy was evaluated by Dice similarity coefficient (DSC), surface DSC, 95% Hausdorff distance (95HD) and average symmetric surface distance (ASSD).ResultsMean for DSC was 0.81 ± 0.13 (range: 0.61 [mental foramen] – 0.98 [mandible]). Mean Surface DSC was 0.94 ± 0.06 (range: 0.87 [mental foramen] – 0.99 [mandible]), with values > 0.9 for all structures but the mental foramen. Mean 95HD was 1.93 ± 2.05 mm (range: 1.00 [mandible] – 4.12 mm [maxillary sinus]) and for ASSD, a mean of 0.42 ± 0.44 mm (range: 0.09 [mandible] – 1.19 mm [mental foramen]) was found, with values < 1 mm for all structures but the mental foramen.ConclusionIn this study, high accuracy of automated segmentation of a variety of craniomaxillofacial structures could be demonstrated, suggesting this approach to be suitable for the incorporation into a computer-assisted craniomaxillofacial surgery workflow. The small amount of training data required and the flexibility of an open source-based network architecture enable a broad variety of clinical and research applications.

A comparison of few-shot and traditional named entity recognition models for medical text.

Many research problems involving medical texts have limited amounts of annotated data available (e.g., expressions of rare diseases). Traditional supervised machine learning algorithms, particularly those based on deep neural networks, require large volumes of annotated data, and they underperform when only small amounts of labeled data are available. Few-shot learning (FSL) is a category of machine learning models that are designed with the intent of solving problems that have small annotated datasets available. However, there is no current study that compares the performances of FSL models with traditional models (e.g., conditional random fields) for medical text at different training set sizes. In this paper, we attempted to fill this gap in research by comparing multiple FSL models with traditional models for the task of named entity recognition (NER) from medical texts. Using five health-related annotated NER datasets, we benchmarked three traditional NER models based on BERT-BERT-Linear Classifier (BLC), BERT-CRF (BC) and SANER; and three FSL NER models-StructShot & NNShot, Few-Shot Slot Tagging (FS-ST) and ProtoNER. Our benchmarking results show that almost all models, whether traditional or FSL, achieve significantly lower performances compared to the state-of-the-art with small amounts of training data. For the NER experiments we executed, the F1-scores were very low with small training sets, typically below 30%. FSL models that were reported to perform well on non-medical texts significantly underperformed, compared to their reported best, on medical texts. Our experiments also suggest that FSL methods tend to perform worse on data sets from noisy sources of medical texts, such as social media (which includes misspellings and colloquial expressions), compared to less noisy sources such as medical literature. Our experiments demonstrate that the current state-of-the-art FSL systems are not yet suitable for effective NER in medical natural language processing tasks, and further research needs to be carried out to improve their performances. Creation of specialized, standardized datasets replicating real-world scenarios may help to move this category of methods forward.

MKD: Mixup-Based Knowledge Distillation for Mandarin End-to-End Speech Recognition

Large-scale automatic speech recognition model has achieved impressive performance. However, huge computational resources and massive amount of data are required to train an ASR model. Knowledge distillation is a prevalent model compression method which transfers the knowledge from large model to small model. To improve the efficiency of knowledge distillation for end-to-end speech recognition especially in the low-resource setting, a Mixup-based Knowledge Distillation (MKD) method is proposed which combines Mixup, a data-agnostic data augmentation method, with softmax-level knowledge distillation. A loss-level mixture is presented to address the problem caused by the non-linearity of label in the KL-divergence when adopting Mixup to the teacher–student framework. It is mathematically shown that optimizing the mixture of loss function is equivalent to optimize an upper bound of the original knowledge distillation loss. The proposed MKD takes the advantage of Mixup and brings robustness to the model even with a small amount of training data. The experiments on Aishell-1 show that MKD obtains a 15.6% and 3.3% relative improvement on two student models with different parameter scales compared with the existing methods. Experiments on data efficiency demonstrate MKD achieves similar results with only half of the original dataset.

Pseudo loss active learning for deep visual tracking

In visual tracking tasks, the training data are commonly composed of a large number of video sequences and each frame in the sequences needs to be labeled manually, which is labor-intensive and time-consuming. In addition, considering the similarity among the consecutive frames in the same sequence, there is significant redundancy in the training data. To address these problems, a novel pseudo loss active learning (PLAL) method is developed in this paper. PLAL aims to select the most informative and least redundant data for training to reduce the cost of labeling and maintain competitive tracking results simultaneously. Firstly, the Gaussian distribution based pseudo label is generated for the unlabeled candidates based on the tracking model which is initially trained on a small amount of training data. Then, the pseudo loss based on cross entropy is designed to compute the difference between the pseudo label and the target response map. The pseudo loss measures the uncertainty of the target spatial context which is used as the informativeness criterion of the image frame for selection. Meanwhile, a sampling interval threshold and a temporal penalty are employed for frame selection to avoid drastic variation in target appearance and reduce the redundancy within the consecutive candidate frames. Only the selected frames are labeled by the oracle (human expert) and then added to the training data. Extensive experiments on public benchmarks (OTB2013, OTB2015, VOT2018, UAV123, GOT-10K, TrackingNet, LaSOT, OxUvA and TLP) demonstrate that PLAL method outperforms the baseline and other recent active learning approaches. With only 3% of labeled data from the training dataset, PLAL reaches competitive performance (98-100%) compared to the model trained on the entire training dataset.

Design and Implementation of a Multichannel Convolutional Neural Network for Hate Speech Detection in Social Networks

As the number of social media comments available online grows, the spread of hate speech has grown gradually. When someone uses hate speech as a weapon to injure, degrade, and humiliate others, their freedom, dignity, and personhood can be jeopardized. Deep neural network-based hate speech detection models, such as the conventional single channel convolutional neural network (SC-CNN), have recently demonstrated promising results. The success of the models, however, is dependent on the type of language they are trained on and the training data size. Even with a small amount of training data, the model's performance can be improved by using a multichannel convolutional neural network (MC-CNN) model. The study assesses and compares the performance of a multichannel convolutional neural network model to single channel convolutional neural network models using a support vector machine (SVM) as a baseline. The models' F1 score values are computed, and promising results are obtained. The MC-CNN model outperforms the SC-CNN models in all three hate speech datasets. The study's findings indicate that the proposed MC-CNN model could be used as a deep learning-based alternative for hate speech detection.

A Model-Agnostic Meta-Baseline Method for Few-Shot Fault Diagnosis of Wind Turbines.

The technology of fault diagnosis is helpful to improve the reliability of wind turbines, and further reduce the operation and maintenance cost at wind farms. However, in reality, wind turbines are not allowed to operate with faults, so few fault samples could be obtained. With a small amount of training data, traditional fault diagnosis models that need huge samples under a deep learning framework are difficult to maintain with high accuracy and effectiveness. Few-shot learning can effectively solve the problem of overfitting caused by fewer fault samples in model training. In view of model-agnostic meta-learning (MAML), this paper proposes a model for few-shot fault diagnosis of the wind turbines drivetrain, which is named model-agnostic meta-baseline (MAMB). The training data is input to the base classification model for pre-training, then, some data is randomly selected from the training set to form multiple meta-learning tasks that are utilized to train the MAML to finally fine-tune the later layers of the model at a smaller learning rate. The proposed model was analyzed by the small samples of the bearing data from Case Western Reserve University (CWRU) data, the generator bearings, and gearboxes vibration data in wind turbines under randomly changing operating conditions. The results verified that the proposed method was superior in one-shot, five-shot, and ten-shot tasks of wind turbines.

Expert System Detects Laptop Damage Using Naive Bayes Method

During the Covid-19 pandemic, teachers and students carried out the online teaching and learning process from home. Distance learning has several problems including the limitations of teachers and students in the world of information and communication technology, the facilities and infrastructure they have, and environmental conditions that are less supportive. The use of laptops and the internet every day are used by students and teachers as the main means of the online teaching and learning process. Continuous use without proper maintenance and lack of knowledge in overcoming the problem of damage to laptops makes teachers and students unable to identify the location of the damage and how to deal with it. Therefore, this expert system application was created to assist teachers and students in detecting the symptoms of laptop damage experienced and solutions to overcome the damage. In the development of this expert system using the Naive Bayes method, this method only requires a small amount of training data to determine parameter estimates during the classification process. The results of the application of the nave Bayes method produce appropriate calculations based on the symptoms of damage and a predetermined list of damage so that it can make it easier for users when analyzing the beginning by using existing symptoms with a system that has been built with very efficient time and has an accuracy rate of 100%.

NIR-Based Intelligent Sensing of Product Yield Stress for High-Value Bioresorbable Polymer Processing.

PLA (polylactide) is a bioresorbable polymer used in implantable medical and drug delivery devices. Like other bioresorbable polymers, PLA needs to be processed carefully to avoid degradation. In this work we combine in-process temperature, pressure, and NIR spectroscopy measurements with multivariate regression methods for prediction of the mechanical strength of an extruded PLA product. The potential to use such a method as an intelligent sensor for real-time quality analysis is evaluated based on regulatory guidelines for the medical device industry. It is shown that for the predictions to be robust to processing at different times and to slight changes in the processing conditions, the fusion of both NIR and conventional process sensor data is required. Partial least squares (PLS), which is the established ’soft sensing’ method in the industry, performs the best of the linear methods but demonstrates poor reliability over the full range of processing conditions. Conversely, both random forest (RF) and support vector regression (SVR) show excellent performance for all criteria when used with a prior principal component (PC) dimension reduction step. While linear methods currently dominate for soft sensing of mixture concentrations in highly conservative, regulated industries such as the medical device industry, this work indicates that nonlinear methods may outperform them in the prediction of mechanical properties from complex physicochemical sensor data. The nonlinear methods show the potential to meet industrial standards for robustness, despite the relatively small amount of training data typically available in high-value material processing.