Small Amount Of Training Data Research Articles

GeDa: Improving training data with large language models for Aspect Sentiment Triplet Extraction

Aspect Sentiment Triplet Extraction (ASTE) is a subtask of Aspect-based Sentiment Analysis (ABSA). Recently, ASTE methods have achieved promising results. However, the performance of ASTE models is restricted to both the quantity and the quality of training data. As such, challenges lie in collecting valuable data and selecting targeted data for diversified ASTE model architecture. To this end, we propose a novel General Data-Centric Framework (GeDa), which is capable of improving the training data for ASTE models accurately and efficiently. Specifically, two types of prompts are designed to guide large language models in synthetic candidates synthesizing for ASTE task. Then, the Characteristic-Driven Iterative Strategy is put forward to optimize the interaction between the model and the training data. The data is iteratively selected from the synthetic candidates, aiming to improve the quantity and the quality of training data. With multiple iterations, a targeted training set can be obtained to benefit ASTE model learning. Extensive experiments reveal that ASTE models with GeDa reach a more than 5% increment on average F1 by adding only a small amount of training data.

Performance Analysis of Random Forest Algorithm in Automatic Building Segmentation with Limited Data

Airborne laser technology produces point clouds that can be used to build 3D models of buildings. However, the work is a laborious process that could benefit from automation. Artificial intelligence (AI) has been widely used in automating building segmentation as one of the initial stages in the 3D modeling process. The algorithms with a high success rate using point clouds for automatic semantic segmentation are random forest (RF) and PointNet++, with each algorithm having its own advantages and disadvantages. However, the training and testing data to develop and test the model usually share similar characteristics. Moreover, producing a good automation model requires a lot of training data, which may become an issue for users with a small amount of training data (limited data). The aim of this research is to test the performance of the RF and PointNet++ models in different regions with limited training and testing data. We found that the RF model developed from a small amount data, in different regions between the training and testing data, performs well compared to PointNet++, yielding an OA score of 73.01% for the RF model. Furthermore, several scenarios have been used in this research to explore the capabilities of RF in several cases.

Development and Validation of an Artificial Intelligence Model for Detecting Rib Fractures on Chest Radiographs.

Background: Chest radiography is the standard method for detecting rib fractures. Our study aims to develop an artificial intelligence (AI) model that, with only a relatively small amount of training data, can identify rib fractures on chest radiographs and accurately mark their precise locations, thereby achieving a diagnostic accuracy comparable to that of medical professionals. Methods: For this retrospective study, we developed an AI model using 540 chest radiographs (270 normal and 270 with rib fractures) labeled for use with Detectron2 which incorporates a faster region-based convolutional neural network (R-CNN) enhanced with a feature pyramid network (FPN). The model's ability to classify radiographs and detect rib fractures was assessed. Furthermore, we compared the model's performance to that of 12 physicians, including six board-certified anesthesiologists and six residents, through an observer performance test. Results: Regarding the radiographic classification performance of the AI model, the sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC) were 0.87, 0.83, and 0.89, respectively. In terms of rib fracture detection performance, the sensitivity, false-positive rate, and free-response receiver operating characteristic (JAFROC) figure of merit (FOM) were 0.62, 0.3, and 0.76, respectively. The AI model showed no statistically significant difference in the observer performance test compared to 11 of 12 and 10 of 12 physicians, respectively. Conclusions: We developed an AI model trained on a limited dataset that demonstrated a rib fracture classification and detection performance comparable to that of an experienced physician.

Gaussian processes for Bayesian inverse problems associated with linear partial differential equations

This work is concerned with the use of Gaussian surrogate models for Bayesian inverse problems associated with linear partial differential equations. A particular focus is on the regime where only a small amount of training data is available. In this regime the type of Gaussian prior used is of critical importance with respect to how well the surrogate model will perform in terms of Bayesian inversion. We extend the framework of Raissi et. al. (2017) to construct PDE-informed Gaussian priors that we then use to construct different approximate posteriors. A number of different numerical experiments illustrate the superiority of the PDE-informed Gaussian priors over more traditional priors.

Meta-learning Achieves High Accuracy with a Small Amount of Training Data

Meta-learning Achieves High Accuracy with a Small Amount of Training Data

Cross-domain structural damage identification using transfer learning strategy

In damage identification of civil structures, training deep neural networks (DNNs) often requires a large volume of annotated training data. However, artificial damage to real-world structures is always forbidden, resulting in insufficient data for training. Transfer learning provides a solution that allows structural damage information in a data-rich domain to be transferred and shared as the prior knowledge to a data-scarce domain, thereby indirectly augmenting available training data in the latter domain. To this end, a multi-task transfer learning strategy has been adopted for the purpose of achieving cross-domain damage identification between a plane frame in the source domain and a three-dimensional frame in the target domain. The strategy can share the learning knowledge from the domain with sufficient training data to the target domain with insufficient data. Thereby, the DNN (named DNN#2) in the target domain can be trained on a small amount of training data. Owing to their ability in data feature extraction, stacked auto-encoders (SAEs) are used to construct the desirable DNN#1 and DNN#2 corresponding to different damage identification tasks, named Task#1 and Task#2, in the two domains. The two DNNs share the hidden layers in order to share damage feature information between the source and target domains. The training datasets of the two domains are first used to jointly pre-train the auto-encoders’ parameters in an unsupervised learning way. Afterwards, a supervised fine-tuning step is carried out to retraining the entire SAEs for better performance. By these means, Task#2 receives some prior knowledge from Task#1, and thus, it is accomplished on limited training data when Task#1 is synchronously achieved. The analysis results demonstrate that the proposed multi-task learning strategy requires only a single training session to simultaneously realize the cross-domain damage identification of two different steel frames.

Depth estimation and 3D reconstruction from UAV-borne imagery: Evaluation on the UseGeo dataset

Depth estimation and 3D model reconstruction from aerial imagery is an important task in photogrammetry, remote sensing, and computer vision. To compare the performance of different image-based approaches, this study presents a benchmark for UAV-based aerial imagery using the UseGeo dataset. The contributions include the release of various evaluation routines on GitHub, as well as a comprehensive comparison of baseline approaches, such as methods for offline multi-view 3D reconstruction resulting in point clouds and triangle meshes, online multi-view depth estimation, as well as single-image depth estimation using self-supervised deep learning. With the release of our evaluation routines, we aim to provide a universal protocol for the evaluation of depth estimation and 3D reconstruction methods on the UseGeo dataset. The conducted experiments and analyses show that each method excels in a different category: the depth estimation from COLMAP outperforms that of the other approaches, ACMMP achieves the lowest error and highest completeness for point clouds, while OpenMVS produces triangle meshes with the lowest error. Among the online methods for depth estimation, the approach from the Plane-Sweep Library outperforms the FaSS-MVS approach, while the latter achieves the lowest processing time. And even though the particularly challenging nature of the dataset and the small amount of training data leads to a significantly higher error in the results of the self-supervised single-image depth estimation approach, it outperforms all other approaches in terms of processing time and frame rate. In our evaluation, we have also considered modern learning-based approaches that can be used for image-based 3D reconstruction, such as NeRFs. However, due to the significantly lower quality of the resulting 3D models, we have only included a qualitative comparison between NeRF-based and conventional approaches in the scope of this work.

Interpretable Prediction of SARS-CoV-2 Epitope-Specific TCR Recognition Using a Pre-Trained Protein Language Model.

The emergence of the novel coronavirus, designated as severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), has posed a significant threat to public health worldwide. There has been progress in reducing hospitalizations and deaths due to SARS-CoV-2. However, challenges stem from the emergence of SARS-CoV-2 variants, which exhibit high transmission rates, increased disease severity, and the ability to evade humoral immunity. Epitope-specific T-cell receptor (TCR) recognition is key in determining the T-cell immunogenicity for SARS-CoV-2 epitopes. Although several data-driven methods for predicting epitope-specific TCR recognition have been proposed, they remain challenging due to the enormous diversity of TCRs and the lack of available training data. Self-supervised transfer learning has recently been proven useful for extracting information from unlabeled protein sequences, increasing the predictive performance of fine-tuned models, and using a relatively small amount of training data. This study presents a deep-learning model generated by fine-tuning pre-trained protein embeddings from a large corpus of protein sequences. The fine-tuned model showed markedly high predictive performance and outperformed the recent Gaussian process-based prediction model. The output attentions captured by the deep-learning model suggested critical amino acid positions in the SARS-CoV-2 epitope-specific TCRβ sequences that are highly associated with the viral escape of T-cell immune response.

Rician Noise Removal in Low-noise Condition via a Deep Unfolding Network

Nuclear magnetic resonance (NMR) signals as the key role of research on denoising, are mainly influenced by Rician noise. Restoring clean Rician medical images from signal-related Rician noise is a challenging task with great practical significance. In this paper, an energy function based on MAP estimation is proposed, mainly for conditions of low noise, where the noise distribution is approximately Gaussian. Then the mathematical model is embedded into the network structure to realize the knot of knowledge-driven and network learning. The experiment shows that the proposed model only uses simple and lightweight network modules and a small amount of training data, which has a better denoising effect and has achieved satisfactory results under the two evaluation indexes.

Data Augmentation for Sample Efficient and Robust Document Ranking

Contextual ranking models have delivered impressive performance improvements over classical models in the document ranking task. However, these highly over-parameterized models tend to be data-hungry and require large amounts of data even for fine-tuning. In this article, we propose data-augmentation methods for effective and robust ranking performance. One of the key benefits of using data augmentation is in achieving sample efficiency or learning effectively when we have only a small amount of training data. We propose supervised and unsupervised data augmentation schemes by creating training data using parts of the relevant documents in the query-document pairs. We then adapt a family of contrastive losses for the document ranking task that can exploit the augmented data to learn an effective ranking model. Our extensive experiments on subsets of the MS MARCO and TREC-DL test sets show that data augmentation, along with the ranking-adapted contrastive losses, results in performance improvements under most dataset sizes. Apart from sample efficiency, we conclusively show that data augmentation results in robust models when transferred to out-of-domain benchmarks. Our performance improvements in in-domain and more prominently in out-of-domain benchmarks show that augmentation regularizes the ranking model and improves its robustness and generalization capability.

UA-LLM: ADVANCING CONTEXT-BASED QUESTION ANSWERING IN UKRAINIAN THROUGH LARGE LANGUAGE MODELS

Context. Context-based question answering, a fundamental task in natural language processing, demands a deep understanding of the language’s nuances. While being a sophisticated task, it’s an essential part of modern search systems, intelligent assistants, chatbots, and the whole Conversational AI field. While English, Chinese, and other widely spoken languages have gathered an extensive number of datasets, algorithms, and benchmarks, the Ukrainian language, with its rich linguistic heritage and intricate syntax, has remained among low-resource languages in the NLP community, making the Question Answering problem even harder.
 Objective. The purpose of this work is to establish and benchmark a set of techniques, leveraging Large Language Models, combined in a single framework for solving the low-resource problem for Context-based question-answering task in Ukrainian.
 Method. A simple yet flexible framework for leveraging Large Language Models, developed as a part of this research work, enlights two key methods proposed and evaluated in this paper for dealing with a small amount of training data for context-based question-answering tasks. The first one utilizes Zero-shot and Few-shot learning – the two major subfields of N-shot learning, where N corresponds to the number of training samples, to build a bilingual instruction-based prompt strategy for language models inferencing in an extractive manner (find an answer span in context) instead of their natural generative behavior (summarize the context according to question). The second proposed method is based on the first one, but instead of just answering the question, the language model annotates the input context through the generation of question-answer pairs for the given paragraph. This synthetic data is used for extractive model training. This paper explores both augmentation-based training, when there is some annotated data already, and completely synthetic training, when no data is available. The key benefit of these two methods is the ability to obtain comparable prediction quality even without an expensive and long-term human annotation process.
 Results. Two proposed methods for solving the low-to-zero amount of training data problem for context-based questionanswering tasks in Ukrainian were implemented and combined into the flexible LLM experimentation framework.
 Conclusions. This research comprehensively studied OpenAI GPT-3.5, OpenAI GPT-4, Cohere Command, and Meta LLaMa-2 language understanding capabilities applied to context-based question answering in low-resource Ukrainian. The thorough evaluation of proposed methods on a diverse set of metrics proves their efficiency, unveiling the possibility of building components of search engines, chatbot applications, and standalone general-domain CBQA systems with Ukrainian language support while having almost zero annotated data. The prospect for further research is to extend the scope from the CBQA task evaluated in this paper to all major NLU tasks with the final goal of establishing a complete benchmark for LLMs’ capabilities evaluation in the Ukrainian language.

Capacity estimation of lithium-ion batteries with uncertainty quantification based on temporal convolutional network and Gaussian process regression

Reliable capacity estimation is crucial for safe operation of lithium-ion batteries (LIBs). This work combines the temporal convolutional network (TCN) and Gaussian process regression (GPR) to establish a novel probabilistic capacity estimation method. The proposed TCN-GPR method can not only provide accurate capacity estimation but also quantify the uncertainty of the estimation. Besides, the TCN-GPR method can automatically extract degradation features from partial charging segments, overcoming the limitations of manual experience. In addition, the TCN-GPR method can be applied to different types of LIBs through transfer learning using only a small amount of training data. For validation, the Oxford battery dataset is used to demonstrate the accuracy and robustness of the TCN-GPR method, where a mean absolute percentage error (MAPE) of less than 0.3% can be achieved with only a 15-min partial charging segment. Furthermore, our own experimental dataset is used to demonstrate the generalization ability of the TCN-GPR method through transfer learning, where a MAPE of less than 0.7% can be achieved by using only one battery cell as the training sample.

Multi-path residual attention network for cancer diagnosis robust to a small number of training data of microscopic hyperspectral pathological images

Duct cancer is a malignant disease with higher mortality rates in males than in females, emphasizing the need for early diagnosis to improve treatment outcomes. Although various imaging modalities such as magnetic resonance imaging (MRI) and computed tomography scan (CT-scan) have been used for pathological analysis, hyperspectral imaging stands out as a promising approach, especially when combined with deep learning techniques. Hyperspectral imaging provides detailed information on tissue composition and biochemical properties, enabling better distinction between cancerous and healthy tissues. Although previous research based on hyperspectral imaging shows high accuracy, no previous research has used a small amount of training data, despite this being the usual case in medical image applications. Therefore, we propose a multi-path residual attention network (MRA-Net) with chunked residual channel attention (CRCA), which is a novel deep learning model specifically designed to address the challenges posed by limited training data, with a particular focus on using hyperspectral images. By leveraging the unique spectral information provided by hyperspectral imaging, MRA-Net extracts distinctive features, enhancing its ability to differentiate between cancerous and healthy tissues.We conducted the training and validation of our model using a publicly accessible dataset, resulting in an accuracy of 84.31% and a weighted harmonic mean of precision and recall (F1 score) of 84.29%, demonstrating its state-of-the-art performance compared to existing methods.

Scientific Deep Machine Learning Concepts for the Prediction of Concentration Profiles and Chemical Reaction Kinetics: Consideration of Reaction Conditions.

Emerging concepts from scientific deep machine learning such as physics-informed neural networks (PINNs) enable a data-driven approach for the study of complex kinetic problems. We present an extended framework that combines the advantages of PINNs with the detailed consideration of experimental parameter variations for the simulation and prediction of chemical reaction kinetics. The approach is based on truncated Taylor series expansions for the underlying fundamental equations, whereby the external variations can be interpreted as perturbations of the kinetic parameters. Accordingly, our method allows for an efficient consideration of experimental parameter settings and their influence on the concentration profiles and reaction kinetics. A particular advantage of our approach, in addition to the consideration of univariate and multivariate parameter variations, is the robust model-based exploration of the parameter space to determine optimal reaction conditions in combination with advanced reaction insights. The benefits of this concept are demonstrated for higher-order chemical reactions including catalytic and oscillatory systems in combination with small amounts of training data. All predicted values show a high level of accuracy, demonstrating the broad applicability and flexibility of our approach.

A predicted-loss based active learning approach for robust cancer pathology image analysis in the workplace.

Convolutional neural network-based image processing research is actively being conducted for pathology image analysis. As a convolutional neural network model requires a large amount of image data for training, active learning (AL) has been developed to produce efficient learning with a small amount of training data. However, existing studies have not specifically considered the characteristics of pathological data collected from the workplace. For various reasons, noisy patches can be selected instead of clean patches during AL, thereby reducing its efficiency. This study proposes an effective AL method for cancer pathology that works robustly on noisy datasets. Our proposed method to develop a robust AL approach for noisy histopathology datasets consists of the following three steps: 1) training a loss prediction module, 2) collecting predicted loss values, and 3) sampling data for labeling. This proposed method calculates the amount of information in unlabeled data as predicted loss values and removes noisy data based on predicted loss values to reduce the rate at which noisy data are selected from the unlabeled dataset. We identified a suitable threshold for optimizing the efficiency of AL through sensitivity analysis. We compared the results obtained with the identified threshold with those of existing representative AL methods. In the final iteration, the proposed method achieved a performance of 91.7% on the noisy dataset and 92.4% on the clean dataset, resulting in a performance reduction of less than 1%. Concomitantly, the noise selection ratio averaged only 2.93% on each iteration. The proposed AL method showed robust performance on datasets containing noisy data by avoiding data selection in predictive loss intervals where noisy data are likely to be distributed. The proposed method contributes to medical image analysis by screening data and producing a robust and effective classification model tailored for cancer pathology image processing in the workplace.

Fast and accurate excited states predictions: machine learning and diabatization.

The efficiency of machine learning algorithms for electronically excited states is far behind ground-state applications. One of the underlying problems is the insufficient smoothness of the fitted potential energy surfaces and other properties in the vicinity of state crossings and conical intersections, which is a prerequisite for an efficient regression. Smooth surfaces can be obtained by switching to the diabatic basis. However, diabatization itself is still an outstanding problem. We overcome these limitations by solving both problems at once. We use a machine learning approach combining clustering and regression techniques to correct for the deficiencies of property-based diabatization which, in return, provides us with smooth surfaces that can be easily fitted. Our approach extends the applicability of property-based diabatization to multidimensional systems. We utilize the proposed diabatization scheme to achieve higher prediction accuracy for adiabatic states and we show its performance by reconstructing global potential energy surfaces of excited states of nitrosyl fluoride and formaldehyde. While the proposed methodology is independent of the specific property-based diabatization and regression algorithm, we show its performance for kernel ridge regression and a very simple diabatization based on transition multipoles. Compared to most other algorithms based on machine learning, our approach needs only a small amount of training data.

MHCSeqNet2-improved peptide-class I MHC binding prediction for alleles with low data.

The binding of a peptide antigen to a Class I major histocompatibility complex (MHC) protein is part of a key process that lets the immune system recognize an infected cell or a cancer cell. This mechanism enabled the development of peptide-based vaccines that can activate the patient's immune response to treat cancers. Hence, the ability of accurately predict peptide-MHC binding is an essential component for prioritizing the best peptides for each patient. However, peptide-MHC binding experimental data for many MHC alleles are still lacking, which limited the accuracy of existing prediction models. In this study, we presented an improved version of MHCSeqNet that utilized sub-word-level peptide features, a 3D structure embedding for MHC alleles, and an expanded training dataset to achieve better generalizability on MHC alleles with small amounts of data. Visualization of MHC allele embeddings confirms that the model was able to group alleles with similar binding specificity, including those with no peptide ligand in the training dataset. Furthermore, an external evaluation suggests that MHCSeqNet2 can improve the prioritization of T cell epitopes for MHC alleles with small amount of training data. The source code and installation instruction for MHCSeqNet2 are available at https://github.com/cmb-chula/MHCSeqNet2.

A Few-Shot-Based Model-Agnostic Meta-Learning for Intrusion Detection in Security of Internet of Things

Currently, the effective technologies used to protect the security of the Internet of Things include blockchain and intrusion detection systems. The traditional IoT intrusion detection systems based on supervised learning usually requires a large amount of data for training, with high costs, and most of them can only work in several specific types of attacks. Faced with the variability of current IoT attack methods and the complexity of the network environment, they are usually unable to play a role in a short time. Especially in dealing with new security problems such as Zero-day attacks, which have a small number of learnable samples. Therefore, this paper proposed an IoT intrusion detection model to deal with the situation where learnable samples are insufficient. The model adopts the idea of meta-learning and divides the training process into two layers based on model-agnostic meta-learning. Its purpose is to train a relatively general model using the malicious network flow data of various known attack modes. To this end, a few-shot IoT intrusion detection dataset, FSIDS-IoT, is constructed based on five datasets, namely CIC-DDoS2019, CIC-IDS2017, CSE-CIC-IDS2018, NSL-KDD, and UNSW-NB15. Experiments show that our proposed approach, using only a few gradient steps and a small amount of training data from the new attack type, can perform well to protect cyber security. In the test of identifying unknown attacks, the optimal accuracy under the 5-way 1-shot setting reached 78.26%, 90.09% under the 5-way 5-shot setting, and 92.19% under the 5-way 10-shot setting.

Stroke classification and treatment support system artificial intelligence for usefulness of stroke diagnosis.

It is important to diagnose cerebral infarction at an early stage and select an appropriate treatment method. The number of stroke-trained physicians is unevenly distributed; thus, a shortage of specialists is a major problem in some regions. In this retrospective design study, we tested whether an artificial intelligence (AI) we built using computer-aided detection/diagnosis may help medical physicians to classify stroke for the appropriate treatment. To build the Stroke Classification and Treatment Support System AI, the clinical data of 231 hospitalized patients with ischemic stroke from January 2016 to December 2017 were used for training the AI. To verify the diagnostic accuracy, 151 patients who were admitted for stroke between January 2018 and December 2018 were also enrolled. By utilizing multimodal data, such as DWI and ADC map images, as well as patient examination data, we were able to construct an AI that can explain the analysis results with a small amount of training data. Furthermore, the AI was able to classify with high accuracy (Cohort 1, evaluation data 88.7%; Cohort 2, validation data 86.1%). In recent years, the treatment options for cerebral infarction have increased in number and complexity, making it even more important to provide appropriate treatment according to the initial diagnosis. This system could be used for initial treatment to automatically diagnose and classify strokes in hospitals where stroke-trained physicians are not available and improve the prognosis of cerebral infarction.

Advantages of Hybrid Neural Network Architectures to Enhance Prediction of Tensile Properties in Laser Powder Bed Fusion

The properties of AlSi10Mg produced by Laser Powder Bed Fusion (PBF-LB) are defined by a multitude of different machine and laser parameters. This multi-parameter space presents the challenge of optimizing the material properties for a given application by the sheer amount of possible parameter combinations. Characterizing this multi-parameter space empirically is limited by time and resources and thus yields an incomplete picture of the process capabilities and local optima, respectively. To improve on this situation, machine learning to map the process parameters on the tensile properties of AlSi10Mg was used. The Hybrid Neural Network (HNN) used in this study consisted of a Convolutional Neural Network (CNN) to process the micrographs and a Dense Neural Network (DNN) to process the LPBF process parameters as well as the output of the CNN. The micrographs given to the CNN part of the network were printed with the same parameters given to the DNN part to include the information of the bulk microstructure as it strongly influences the tensile properties of the material. With the HNN, we observed good accuracy of the predicted tensile properties, given the small amount of training data. Furthermore, we explore which features of the micrographs were extracted by the CNN.