Small Amount Of Training Data Research Articles

Improved region proposal network for enhanced few-shot object detection

Despite significant success of deep learning in object detection tasks, the standard training of deep neural networks requires access to a substantial quantity of annotated images across all classes. Data annotation is an arduous and time-consuming endeavor, particularly when dealing with infrequent objects. Few-shot object detection (FSOD) methods have emerged as a solution to the limitations of classic object detection approaches based on deep learning. FSOD methods demonstrate remarkable performance by achieving robust object detection using a significantly smaller amount of training data. A challenge for FSOD is that instances from novel classes that do not belong to the fixed set of training classes appear in the background and the base model may pick them up as potential objects. These objects behave similarly to label noise because they are classified as one of the training dataset classes, leading to FSOD performance degradation. We develop a semi-supervised algorithm to detect and then utilize these unlabeled novel objects as positive samples during the FSOD training stage to improve FSOD performance. Specifically, we develop a hierarchical ternary classification region proposal network (HTRPN) to localize the potential unlabeled novel objects and assign them new objectness labels to distinguish these objects from the base training dataset classes. Our improved hierarchical sampling strategy for the region proposal network (RPN) also boosts the perception ability of the object detection model for large objects. We test our approach and COCO and PASCAL VOC baselines that are commonly used in FSOD literature. Our experimental results indicate that our method is effective and outperforms the existing state-of-the-art (SOTA) FSOD methods. Our implementation is provided as a supplement to support reproducibility of the results https://github.com/zshanggu/HTRPN.11Early partial results of this work is presented in the 2023 ICCV Workshop on Visual Continual Learning (Shangguan and Rostami, 2023).

Fine-Tuning SSL-Model to Enhance Detection of Cilioretinal Arteries on Colored Fundus Images.

Cilioretinal arteries are a common congenital anomaly of retinal blood supply. This paper presents a deep learning-based approach for the automated detection of a CRA from color fundus images. Leveraging the Vision Transformer architecture, a pre-trained model from RETFound was fine-tuned to transfer knowledge from a broader dataset to our specific task. An initial dataset of 85 was expanded to 170 images through data augmentation using self-supervised learning-driven techniques. To address the imbalance in the dataset and prevent overfitting, Focal Loss and Early Stopping were implemented. The model's performance was evaluated using a 70-30 split of the dataset for training and validation. The results showcase the potential of ophthalmic foundation models in enhancing detection of CRAs and reducing the effort required for labeling by retinal experts, as promising results could be achieved with only a small amount of training data through fine-tuning.

Hybrid AI framework for the predictions of film cooling effectiveness distribution with various surface curvatures and compound angles

Film cooling is essential for protecting high-temperature components in gas turbines. For efficient cooling design, it is vital to predict film cooling effectiveness based on flow and geometrical parameters accurately and rapidly. Artificial intelligence (AI) models have recently been popular for this task due to their strong fitting nature. However, standard image-based AI models require significant training data to cover different surface curvatures and compound angles. Moreover, incorporating curvature complicates the input format for these models. Therefore, the current study proposes a hybrid AI framework for the rapid prediction of the film cooling effectiveness distribution under the influence of surface curvature and compound angle, which requires a relatively small amount of training data by integrating film cooling knowledge and different neural networks (Back-propagation neural network and U-Net). The proposed model achieves high accuracy across diverse flow and geometrical parameters, with a mean absolute error of approximately 0.006 or 0.012 for the film holes without or with compound angles on curved surfaces, respectively. The hybrid AI framework offers an accurate and rapid prediction method, offering a valuable tool for turbine cooling design.

GeDa: Improving training data with large language models for Aspect Sentiment Triplet Extraction

Aspect Sentiment Triplet Extraction (ASTE) is a subtask of Aspect-based Sentiment Analysis (ABSA). Recently, ASTE methods have achieved promising results. However, the performance of ASTE models is restricted to both the quantity and the quality of training data. As such, challenges lie in collecting valuable data and selecting targeted data for diversified ASTE model architecture. To this end, we propose a novel General Data-Centric Framework (GeDa), which is capable of improving the training data for ASTE models accurately and efficiently. Specifically, two types of prompts are designed to guide large language models in synthetic candidates synthesizing for ASTE task. Then, the Characteristic-Driven Iterative Strategy is put forward to optimize the interaction between the model and the training data. The data is iteratively selected from the synthetic candidates, aiming to improve the quantity and the quality of training data. With multiple iterations, a targeted training set can be obtained to benefit ASTE model learning. Extensive experiments reveal that ASTE models with GeDa reach a more than 5% increment on average F1 by adding only a small amount of training data.

Performance Analysis of Random Forest Algorithm in Automatic Building Segmentation with Limited Data

Airborne laser technology produces point clouds that can be used to build 3D models of buildings. However, the work is a laborious process that could benefit from automation. Artificial intelligence (AI) has been widely used in automating building segmentation as one of the initial stages in the 3D modeling process. The algorithms with a high success rate using point clouds for automatic semantic segmentation are random forest (RF) and PointNet++, with each algorithm having its own advantages and disadvantages. However, the training and testing data to develop and test the model usually share similar characteristics. Moreover, producing a good automation model requires a lot of training data, which may become an issue for users with a small amount of training data (limited data). The aim of this research is to test the performance of the RF and PointNet++ models in different regions with limited training and testing data. We found that the RF model developed from a small amount data, in different regions between the training and testing data, performs well compared to PointNet++, yielding an OA score of 73.01% for the RF model. Furthermore, several scenarios have been used in this research to explore the capabilities of RF in several cases.

Development and Validation of an Artificial Intelligence Model for Detecting Rib Fractures on Chest Radiographs.

Background: Chest radiography is the standard method for detecting rib fractures. Our study aims to develop an artificial intelligence (AI) model that, with only a relatively small amount of training data, can identify rib fractures on chest radiographs and accurately mark their precise locations, thereby achieving a diagnostic accuracy comparable to that of medical professionals. Methods: For this retrospective study, we developed an AI model using 540 chest radiographs (270 normal and 270 with rib fractures) labeled for use with Detectron2 which incorporates a faster region-based convolutional neural network (R-CNN) enhanced with a feature pyramid network (FPN). The model's ability to classify radiographs and detect rib fractures was assessed. Furthermore, we compared the model's performance to that of 12 physicians, including six board-certified anesthesiologists and six residents, through an observer performance test. Results: Regarding the radiographic classification performance of the AI model, the sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC) were 0.87, 0.83, and 0.89, respectively. In terms of rib fracture detection performance, the sensitivity, false-positive rate, and free-response receiver operating characteristic (JAFROC) figure of merit (FOM) were 0.62, 0.3, and 0.76, respectively. The AI model showed no statistically significant difference in the observer performance test compared to 11 of 12 and 10 of 12 physicians, respectively. Conclusions: We developed an AI model trained on a limited dataset that demonstrated a rib fracture classification and detection performance comparable to that of an experienced physician.

Gaussian processes for Bayesian inverse problems associated with linear partial differential equations

This work is concerned with the use of Gaussian surrogate models for Bayesian inverse problems associated with linear partial differential equations. A particular focus is on the regime where only a small amount of training data is available. In this regime the type of Gaussian prior used is of critical importance with respect to how well the surrogate model will perform in terms of Bayesian inversion. We extend the framework of Raissi et. al. (2017) to construct PDE-informed Gaussian priors that we then use to construct different approximate posteriors. A number of different numerical experiments illustrate the superiority of the PDE-informed Gaussian priors over more traditional priors.

Meta-learning Achieves High Accuracy with a Small Amount of Training Data

Meta-learning Achieves High Accuracy with a Small Amount of Training Data

Cross-domain structural damage identification using transfer learning strategy

In damage identification of civil structures, training deep neural networks (DNNs) often requires a large volume of annotated training data. However, artificial damage to real-world structures is always forbidden, resulting in insufficient data for training. Transfer learning provides a solution that allows structural damage information in a data-rich domain to be transferred and shared as the prior knowledge to a data-scarce domain, thereby indirectly augmenting available training data in the latter domain. To this end, a multi-task transfer learning strategy has been adopted for the purpose of achieving cross-domain damage identification between a plane frame in the source domain and a three-dimensional frame in the target domain. The strategy can share the learning knowledge from the domain with sufficient training data to the target domain with insufficient data. Thereby, the DNN (named DNN#2) in the target domain can be trained on a small amount of training data. Owing to their ability in data feature extraction, stacked auto-encoders (SAEs) are used to construct the desirable DNN#1 and DNN#2 corresponding to different damage identification tasks, named Task#1 and Task#2, in the two domains. The two DNNs share the hidden layers in order to share damage feature information between the source and target domains. The training datasets of the two domains are first used to jointly pre-train the auto-encoders’ parameters in an unsupervised learning way. Afterwards, a supervised fine-tuning step is carried out to retraining the entire SAEs for better performance. By these means, Task#2 receives some prior knowledge from Task#1, and thus, it is accomplished on limited training data when Task#1 is synchronously achieved. The analysis results demonstrate that the proposed multi-task learning strategy requires only a single training session to simultaneously realize the cross-domain damage identification of two different steel frames.

Real-Time Microgrid Energy Scheduling Using Meta-Reinforcement Learning

With the rapid development of renewable energy and the increasing maturity of energy storage technology, microgrids are quickly becoming popular worldwide. The stochastic scheduling problem of microgrids can increase operational costs and resource wastage. In order to reduce operational costs and optimize resource utilization efficiency, the real-time scheduling of microgrids becomes particularly important. After collecting extensive data, reinforcement learning (RL) can provide good strategies. However, it cannot make quick and rational decisions in different environments. As a method with generalization ability, meta-learning can compensate for this deficiency. Therefore, this paper introduces a microgrid scheduling strategy based on RL and meta-learning. This method can quickly adapt to different environments with a small amount of training data, enabling rapid energy scheduling policy generation in the early stages of microgrid operation. This paper first establishes a microgrid model, including components such as energy storage, load, and distributed generation (DG). Then, we use a meta-reinforcement learning framework to train the initial scheduling strategy, considering the various operational constraints of the microgrid. The experimental results show that the MAML-based RL strategy has advantages in improving energy utilization and reducing operational costs in the early stages of microgrid operation. This research provides a new intelligent solution for microgrids’ efficient, stable, and economical operation in their initial stages.

Depth estimation and 3D reconstruction from UAV-borne imagery: Evaluation on the UseGeo dataset

Depth estimation and 3D model reconstruction from aerial imagery is an important task in photogrammetry, remote sensing, and computer vision. To compare the performance of different image-based approaches, this study presents a benchmark for UAV-based aerial imagery using the UseGeo dataset. The contributions include the release of various evaluation routines on GitHub, as well as a comprehensive comparison of baseline approaches, such as methods for offline multi-view 3D reconstruction resulting in point clouds and triangle meshes, online multi-view depth estimation, as well as single-image depth estimation using self-supervised deep learning. With the release of our evaluation routines, we aim to provide a universal protocol for the evaluation of depth estimation and 3D reconstruction methods on the UseGeo dataset. The conducted experiments and analyses show that each method excels in a different category: the depth estimation from COLMAP outperforms that of the other approaches, ACMMP achieves the lowest error and highest completeness for point clouds, while OpenMVS produces triangle meshes with the lowest error. Among the online methods for depth estimation, the approach from the Plane-Sweep Library outperforms the FaSS-MVS approach, while the latter achieves the lowest processing time. And even though the particularly challenging nature of the dataset and the small amount of training data leads to a significantly higher error in the results of the self-supervised single-image depth estimation approach, it outperforms all other approaches in terms of processing time and frame rate. In our evaluation, we have also considered modern learning-based approaches that can be used for image-based 3D reconstruction, such as NeRFs. However, due to the significantly lower quality of the resulting 3D models, we have only included a qualitative comparison between NeRF-based and conventional approaches in the scope of this work.

Interpretable Prediction of SARS-CoV-2 Epitope-Specific TCR Recognition Using a Pre-Trained Protein Language Model.

The emergence of the novel coronavirus, designated as severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), has posed a significant threat to public health worldwide. There has been progress in reducing hospitalizations and deaths due to SARS-CoV-2. However, challenges stem from the emergence of SARS-CoV-2 variants, which exhibit high transmission rates, increased disease severity, and the ability to evade humoral immunity. Epitope-specific T-cell receptor (TCR) recognition is key in determining the T-cell immunogenicity for SARS-CoV-2 epitopes. Although several data-driven methods for predicting epitope-specific TCR recognition have been proposed, they remain challenging due to the enormous diversity of TCRs and the lack of available training data. Self-supervised transfer learning has recently been proven useful for extracting information from unlabeled protein sequences, increasing the predictive performance of fine-tuned models, and using a relatively small amount of training data. This study presents a deep-learning model generated by fine-tuning pre-trained protein embeddings from a large corpus of protein sequences. The fine-tuned model showed markedly high predictive performance and outperformed the recent Gaussian process-based prediction model. The output attentions captured by the deep-learning model suggested critical amino acid positions in the SARS-CoV-2 epitope-specific TCRβ sequences that are highly associated with the viral escape of T-cell immune response.

Rician Noise Removal in Low-noise Condition via a Deep Unfolding Network

Nuclear magnetic resonance (NMR) signals as the key role of research on denoising, are mainly influenced by Rician noise. Restoring clean Rician medical images from signal-related Rician noise is a challenging task with great practical significance. In this paper, an energy function based on MAP estimation is proposed, mainly for conditions of low noise, where the noise distribution is approximately Gaussian. Then the mathematical model is embedded into the network structure to realize the knot of knowledge-driven and network learning. The experiment shows that the proposed model only uses simple and lightweight network modules and a small amount of training data, which has a better denoising effect and has achieved satisfactory results under the two evaluation indexes.

Data Augmentation for Sample Efficient and Robust Document Ranking

Contextual ranking models have delivered impressive performance improvements over classical models in the document ranking task. However, these highly over-parameterized models tend to be data-hungry and require large amounts of data even for fine-tuning. In this article, we propose data-augmentation methods for effective and robust ranking performance. One of the key benefits of using data augmentation is in achieving sample efficiency or learning effectively when we have only a small amount of training data. We propose supervised and unsupervised data augmentation schemes by creating training data using parts of the relevant documents in the query-document pairs. We then adapt a family of contrastive losses for the document ranking task that can exploit the augmented data to learn an effective ranking model. Our extensive experiments on subsets of the MS MARCO and TREC-DL test sets show that data augmentation, along with the ranking-adapted contrastive losses, results in performance improvements under most dataset sizes. Apart from sample efficiency, we conclusively show that data augmentation results in robust models when transferred to out-of-domain benchmarks. Our performance improvements in in-domain and more prominently in out-of-domain benchmarks show that augmentation regularizes the ranking model and improves its robustness and generalization capability.

UA-LLM: ADVANCING CONTEXT-BASED QUESTION ANSWERING IN UKRAINIAN THROUGH LARGE LANGUAGE MODELS

Context. Context-based question answering, a fundamental task in natural language processing, demands a deep understanding of the language’s nuances. While being a sophisticated task, it’s an essential part of modern search systems, intelligent assistants, chatbots, and the whole Conversational AI field. While English, Chinese, and other widely spoken languages have gathered an extensive number of datasets, algorithms, and benchmarks, the Ukrainian language, with its rich linguistic heritage and intricate syntax, has remained among low-resource languages in the NLP community, making the Question Answering problem even harder.
 Objective. The purpose of this work is to establish and benchmark a set of techniques, leveraging Large Language Models, combined in a single framework for solving the low-resource problem for Context-based question-answering task in Ukrainian.
 Method. A simple yet flexible framework for leveraging Large Language Models, developed as a part of this research work, enlights two key methods proposed and evaluated in this paper for dealing with a small amount of training data for context-based question-answering tasks. The first one utilizes Zero-shot and Few-shot learning – the two major subfields of N-shot learning, where N corresponds to the number of training samples, to build a bilingual instruction-based prompt strategy for language models inferencing in an extractive manner (find an answer span in context) instead of their natural generative behavior (summarize the context according to question). The second proposed method is based on the first one, but instead of just answering the question, the language model annotates the input context through the generation of question-answer pairs for the given paragraph. This synthetic data is used for extractive model training. This paper explores both augmentation-based training, when there is some annotated data already, and completely synthetic training, when no data is available. The key benefit of these two methods is the ability to obtain comparable prediction quality even without an expensive and long-term human annotation process.
 Results. Two proposed methods for solving the low-to-zero amount of training data problem for context-based questionanswering tasks in Ukrainian were implemented and combined into the flexible LLM experimentation framework.
 Conclusions. This research comprehensively studied OpenAI GPT-3.5, OpenAI GPT-4, Cohere Command, and Meta LLaMa-2 language understanding capabilities applied to context-based question answering in low-resource Ukrainian. The thorough evaluation of proposed methods on a diverse set of metrics proves their efficiency, unveiling the possibility of building components of search engines, chatbot applications, and standalone general-domain CBQA systems with Ukrainian language support while having almost zero annotated data. The prospect for further research is to extend the scope from the CBQA task evaluated in this paper to all major NLU tasks with the final goal of establishing a complete benchmark for LLMs’ capabilities evaluation in the Ukrainian language.

Capacity estimation of lithium-ion batteries with uncertainty quantification based on temporal convolutional network and Gaussian process regression

Reliable capacity estimation is crucial for safe operation of lithium-ion batteries (LIBs). This work combines the temporal convolutional network (TCN) and Gaussian process regression (GPR) to establish a novel probabilistic capacity estimation method. The proposed TCN-GPR method can not only provide accurate capacity estimation but also quantify the uncertainty of the estimation. Besides, the TCN-GPR method can automatically extract degradation features from partial charging segments, overcoming the limitations of manual experience. In addition, the TCN-GPR method can be applied to different types of LIBs through transfer learning using only a small amount of training data. For validation, the Oxford battery dataset is used to demonstrate the accuracy and robustness of the TCN-GPR method, where a mean absolute percentage error (MAPE) of less than 0.3% can be achieved with only a 15-min partial charging segment. Furthermore, our own experimental dataset is used to demonstrate the generalization ability of the TCN-GPR method through transfer learning, where a MAPE of less than 0.7% can be achieved by using only one battery cell as the training sample.

Multi-path residual attention network for cancer diagnosis robust to a small number of training data of microscopic hyperspectral pathological images

Duct cancer is a malignant disease with higher mortality rates in males than in females, emphasizing the need for early diagnosis to improve treatment outcomes. Although various imaging modalities such as magnetic resonance imaging (MRI) and computed tomography scan (CT-scan) have been used for pathological analysis, hyperspectral imaging stands out as a promising approach, especially when combined with deep learning techniques. Hyperspectral imaging provides detailed information on tissue composition and biochemical properties, enabling better distinction between cancerous and healthy tissues. Although previous research based on hyperspectral imaging shows high accuracy, no previous research has used a small amount of training data, despite this being the usual case in medical image applications. Therefore, we propose a multi-path residual attention network (MRA-Net) with chunked residual channel attention (CRCA), which is a novel deep learning model specifically designed to address the challenges posed by limited training data, with a particular focus on using hyperspectral images. By leveraging the unique spectral information provided by hyperspectral imaging, MRA-Net extracts distinctive features, enhancing its ability to differentiate between cancerous and healthy tissues.We conducted the training and validation of our model using a publicly accessible dataset, resulting in an accuracy of 84.31% and a weighted harmonic mean of precision and recall (F1 score) of 84.29%, demonstrating its state-of-the-art performance compared to existing methods.

Scientific Deep Machine Learning Concepts for the Prediction of Concentration Profiles and Chemical Reaction Kinetics: Consideration of Reaction Conditions.

Emerging concepts from scientific deep machine learning such as physics-informed neural networks (PINNs) enable a data-driven approach for the study of complex kinetic problems. We present an extended framework that combines the advantages of PINNs with the detailed consideration of experimental parameter variations for the simulation and prediction of chemical reaction kinetics. The approach is based on truncated Taylor series expansions for the underlying fundamental equations, whereby the external variations can be interpreted as perturbations of the kinetic parameters. Accordingly, our method allows for an efficient consideration of experimental parameter settings and their influence on the concentration profiles and reaction kinetics. A particular advantage of our approach, in addition to the consideration of univariate and multivariate parameter variations, is the robust model-based exploration of the parameter space to determine optimal reaction conditions in combination with advanced reaction insights. The benefits of this concept are demonstrated for higher-order chemical reactions including catalytic and oscillatory systems in combination with small amounts of training data. All predicted values show a high level of accuracy, demonstrating the broad applicability and flexibility of our approach.

A predicted-loss based active learning approach for robust cancer pathology image analysis in the workplace.

Convolutional neural network-based image processing research is actively being conducted for pathology image analysis. As a convolutional neural network model requires a large amount of image data for training, active learning (AL) has been developed to produce efficient learning with a small amount of training data. However, existing studies have not specifically considered the characteristics of pathological data collected from the workplace. For various reasons, noisy patches can be selected instead of clean patches during AL, thereby reducing its efficiency. This study proposes an effective AL method for cancer pathology that works robustly on noisy datasets. Our proposed method to develop a robust AL approach for noisy histopathology datasets consists of the following three steps: 1) training a loss prediction module, 2) collecting predicted loss values, and 3) sampling data for labeling. This proposed method calculates the amount of information in unlabeled data as predicted loss values and removes noisy data based on predicted loss values to reduce the rate at which noisy data are selected from the unlabeled dataset. We identified a suitable threshold for optimizing the efficiency of AL through sensitivity analysis. We compared the results obtained with the identified threshold with those of existing representative AL methods. In the final iteration, the proposed method achieved a performance of 91.7% on the noisy dataset and 92.4% on the clean dataset, resulting in a performance reduction of less than 1%. Concomitantly, the noise selection ratio averaged only 2.93% on each iteration. The proposed AL method showed robust performance on datasets containing noisy data by avoiding data selection in predictive loss intervals where noisy data are likely to be distributed. The proposed method contributes to medical image analysis by screening data and producing a robust and effective classification model tailored for cancer pathology image processing in the workplace.

Fast and accurate excited states predictions: machine learning and diabatization.

The efficiency of machine learning algorithms for electronically excited states is far behind ground-state applications. One of the underlying problems is the insufficient smoothness of the fitted potential energy surfaces and other properties in the vicinity of state crossings and conical intersections, which is a prerequisite for an efficient regression. Smooth surfaces can be obtained by switching to the diabatic basis. However, diabatization itself is still an outstanding problem. We overcome these limitations by solving both problems at once. We use a machine learning approach combining clustering and regression techniques to correct for the deficiencies of property-based diabatization which, in return, provides us with smooth surfaces that can be easily fitted. Our approach extends the applicability of property-based diabatization to multidimensional systems. We utilize the proposed diabatization scheme to achieve higher prediction accuracy for adiabatic states and we show its performance by reconstructing global potential energy surfaces of excited states of nitrosyl fluoride and formaldehyde. While the proposed methodology is independent of the specific property-based diabatization and regression algorithm, we show its performance for kernel ridge regression and a very simple diabatization based on transition multipoles. Compared to most other algorithms based on machine learning, our approach needs only a small amount of training data.