Small Amount Of Training Data Research Articles

Rician Noise Removal in Low-noise Condition via a Deep Unfolding Network

Nuclear magnetic resonance (NMR) signals as the key role of research on denoising, are mainly influenced by Rician noise. Restoring clean Rician medical images from signal-related Rician noise is a challenging task with great practical significance. In this paper, an energy function based on MAP estimation is proposed, mainly for conditions of low noise, where the noise distribution is approximately Gaussian. Then the mathematical model is embedded into the network structure to realize the knot of knowledge-driven and network learning. The experiment shows that the proposed model only uses simple and lightweight network modules and a small amount of training data, which has a better denoising effect and has achieved satisfactory results under the two evaluation indexes.

Depth estimation and 3D reconstruction from UAV-borne imagery: Evaluation on the UseGeo dataset

Depth estimation and 3D model reconstruction from aerial imagery is an important task in photogrammetry, remote sensing, and computer vision. To compare the performance of different image-based approaches, this study presents a benchmark for UAV-based aerial imagery using the UseGeo dataset. The contributions include the release of various evaluation routines on GitHub, as well as a comprehensive comparison of baseline approaches, such as methods for offline multi-view 3D reconstruction resulting in point clouds and triangle meshes, online multi-view depth estimation, as well as single-image depth estimation using self-supervised deep learning. With the release of our evaluation routines, we aim to provide a universal protocol for the evaluation of depth estimation and 3D reconstruction methods on the UseGeo dataset. The conducted experiments and analyses show that each method excels in a different category: the depth estimation from COLMAP outperforms that of the other approaches, ACMMP achieves the lowest error and highest completeness for point clouds, while OpenMVS produces triangle meshes with the lowest error. Among the online methods for depth estimation, the approach from the Plane-Sweep Library outperforms the FaSS-MVS approach, while the latter achieves the lowest processing time. And even though the particularly challenging nature of the dataset and the small amount of training data leads to a significantly higher error in the results of the self-supervised single-image depth estimation approach, it outperforms all other approaches in terms of processing time and frame rate. In our evaluation, we have also considered modern learning-based approaches that can be used for image-based 3D reconstruction, such as NeRFs. However, due to the significantly lower quality of the resulting 3D models, we have only included a qualitative comparison of such methods against conventional approaches in the scope of this work.

UA-LLM: ADVANCING CONTEXT-BASED QUESTION ANSWERING IN UKRAINIAN THROUGH LARGE LANGUAGE MODELS

Context. Context-based question answering, a fundamental task in natural language processing, demands a deep understanding of the language’s nuances. While being a sophisticated task, it’s an essential part of modern search systems, intelligent assistants, chatbots, and the whole Conversational AI field. While English, Chinese, and other widely spoken languages have gathered an extensive number of datasets, algorithms, and benchmarks, the Ukrainian language, with its rich linguistic heritage and intricate syntax, has remained among low-resource languages in the NLP community, making the Question Answering problem even harder.
 Objective. The purpose of this work is to establish and benchmark a set of techniques, leveraging Large Language Models, combined in a single framework for solving the low-resource problem for Context-based question-answering task in Ukrainian.
 Method. A simple yet flexible framework for leveraging Large Language Models, developed as a part of this research work, enlights two key methods proposed and evaluated in this paper for dealing with a small amount of training data for context-based question-answering tasks. The first one utilizes Zero-shot and Few-shot learning – the two major subfields of N-shot learning, where N corresponds to the number of training samples, to build a bilingual instruction-based prompt strategy for language models inferencing in an extractive manner (find an answer span in context) instead of their natural generative behavior (summarize the context according to question). The second proposed method is based on the first one, but instead of just answering the question, the language model annotates the input context through the generation of question-answer pairs for the given paragraph. This synthetic data is used for extractive model training. This paper explores both augmentation-based training, when there is some annotated data already, and completely synthetic training, when no data is available. The key benefit of these two methods is the ability to obtain comparable prediction quality even without an expensive and long-term human annotation process.
 Results. Two proposed methods for solving the low-to-zero amount of training data problem for context-based questionanswering tasks in Ukrainian were implemented and combined into the flexible LLM experimentation framework.
 Conclusions. This research comprehensively studied OpenAI GPT-3.5, OpenAI GPT-4, Cohere Command, and Meta LLaMa-2 language understanding capabilities applied to context-based question answering in low-resource Ukrainian. The thorough evaluation of proposed methods on a diverse set of metrics proves their efficiency, unveiling the possibility of building components of search engines, chatbot applications, and standalone general-domain CBQA systems with Ukrainian language support while having almost zero annotated data. The prospect for further research is to extend the scope from the CBQA task evaluated in this paper to all major NLU tasks with the final goal of establishing a complete benchmark for LLMs’ capabilities evaluation in the Ukrainian language.

Capacity estimation of lithium-ion batteries with uncertainty quantification based on temporal convolutional network and Gaussian process regression

Reliable capacity estimation is crucial for safe operation of lithium-ion batteries (LIBs). This work combines the temporal convolutional network (TCN) and Gaussian process regression (GPR) to establish a novel probabilistic capacity estimation method. The proposed TCN-GPR method can not only provide accurate capacity estimation but also quantify the uncertainty of the estimation. Besides, the TCN-GPR method can automatically extract degradation features from partial charging segments, overcoming the limitations of manual experience. In addition, the TCN-GPR method can be applied to different types of LIBs through transfer learning using only a small amount of training data. For validation, the Oxford battery dataset is used to demonstrate the accuracy and robustness of the TCN-GPR method, where a mean absolute percentage error (MAPE) of less than 0.3% can be achieved with only a 15-min partial charging segment. Furthermore, our own experimental dataset is used to demonstrate the generalization ability of the TCN-GPR method through transfer learning, where a MAPE of less than 0.7% can be achieved by using only one battery cell as the training sample.

Multi-path residual attention network for cancer diagnosis robust to a small number of training data of microscopic hyperspectral pathological images

Duct cancer is a malignant disease with higher mortality rates in males than in females, emphasizing the need for early diagnosis to improve treatment outcomes. Although various imaging modalities such as magnetic resonance imaging (MRI) and computed tomography scan (CT-scan) have been used for pathological analysis, hyperspectral imaging stands out as a promising approach, especially when combined with deep learning techniques. Hyperspectral imaging provides detailed information on tissue composition and biochemical properties, enabling better distinction between cancerous and healthy tissues. Although previous research based on hyperspectral imaging shows high accuracy, no previous research has used a small amount of training data, despite this being the usual case in medical image applications. Therefore, we propose a multi-path residual attention network (MRA-Net) with chunked residual channel attention (CRCA), which is a novel deep learning model specifically designed to address the challenges posed by limited training data, with a particular focus on using hyperspectral images. By leveraging the unique spectral information provided by hyperspectral imaging, MRA-Net extracts distinctive features, enhancing its ability to differentiate between cancerous and healthy tissues.We conducted the training and validation of our model using a publicly accessible dataset, resulting in an accuracy of 84.31% and a weighted harmonic mean of precision and recall (F1 score) of 84.29%, demonstrating its state-of-the-art performance compared to existing methods.

Scientific Deep Machine Learning Concepts for the Prediction of Concentration Profiles and Chemical Reaction Kinetics: Consideration of Reaction Conditions.

Emerging concepts from scientific deep machine learning such as physics-informed neural networks (PINNs) enable a data-driven approach for the study of complex kinetic problems. We present an extended framework that combines the advantages of PINNs with the detailed consideration of experimental parameter variations for the simulation and prediction of chemical reaction kinetics. The approach is based on truncated Taylor series expansions for the underlying fundamental equations, whereby the external variations can be interpreted as perturbations of the kinetic parameters. Accordingly, our method allows for an efficient consideration of experimental parameter settings and their influence on the concentration profiles and reaction kinetics. A particular advantage of our approach, in addition to the consideration of univariate and multivariate parameter variations, is the robust model-based exploration of the parameter space to determine optimal reaction conditions in combination with advanced reaction insights. The benefits of this concept are demonstrated for higher-order chemical reactions including catalytic and oscillatory systems in combination with small amounts of training data. All predicted values show a high level of accuracy, demonstrating the broad applicability and flexibility of our approach.

A predicted-loss based active learning approach for robust cancer pathology image analysis in the workplace.

Convolutional neural network-based image processing research is actively being conducted for pathology image analysis. As a convolutional neural network model requires a large amount of image data for training, active learning (AL) has been developed to produce efficient learning with a small amount of training data. However, existing studies have not specifically considered the characteristics of pathological data collected from the workplace. For various reasons, noisy patches can be selected instead of clean patches during AL, thereby reducing its efficiency. This study proposes an effective AL method for cancer pathology that works robustly on noisy datasets. Our proposed method to develop a robust AL approach for noisy histopathology datasets consists of the following three steps: 1) training a loss prediction module, 2) collecting predicted loss values, and 3) sampling data for labeling. This proposed method calculates the amount of information in unlabeled data as predicted loss values and removes noisy data based on predicted loss values to reduce the rate at which noisy data are selected from the unlabeled dataset. We identified a suitable threshold for optimizing the efficiency of AL through sensitivity analysis. We compared the results obtained with the identified threshold with those of existing representative AL methods. In the final iteration, the proposed method achieved a performance of 91.7% on the noisy dataset and 92.4% on the clean dataset, resulting in a performance reduction of less than 1%. Concomitantly, the noise selection ratio averaged only 2.93% on each iteration. The proposed AL method showed robust performance on datasets containing noisy data by avoiding data selection in predictive loss intervals where noisy data are likely to be distributed. The proposed method contributes to medical image analysis by screening data and producing a robust and effective classification model tailored for cancer pathology image processing in the workplace.

Interpretable Prediction of SARS-CoV-2 Epitope-specific TCR Recognition Using a Pre-Trained Protein Language Model.

The emergence of the novel coronavirus, designated as severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), has posed a significant threat to public health worldwide. There has been progress in reducing hospitalizations and deaths due to SARS-CoV-2. However, challenges stem from the emergence of SARS-CoV-2 variants, which exhibit high transmission rates, increased disease severity, and the ability to evade humoral immunity. Epitope-specific T-cell receptor (TCR) recognition is key in determining the T-cell immunogenicity for SARS-CoV-2 epitopes. Although several data-driven methods for predicting epitope-specific TCR recognition have been proposed, they remain challenging due to the enormous diversity of TCRs and the lack of available training data. Self-supervised transfer learning has recently been proven useful for extracting information from unlabeled protein sequences, increasing the predictive performance of fine-tuned models, and using a relatively small amount of training data. This study presents a deep-learning model generated by fine-tuning pre-trained protein embeddings from a large corpus of protein sequences. The fine-tuned model showed markedly high predictive performance and outperformed the recent Gaussian process-based prediction model. The output attentions captured by the deep-learning model suggested critical amino acid positions in the SARS-CoV-2 epitope-specific TCRβ sequences that are highly associated with the viral escape of T-cell immune response.

Fast and accurate excited states predictions: machine learning and diabatization.

The efficiency of machine learning algorithms for electronically excited states is far behind ground-state applications. One of the underlying problems is the insufficient smoothness of the fitted potential energy surfaces and other properties in the vicinity of state crossings and conical intersections, which is a prerequisite for an efficient regression. Smooth surfaces can be obtained by switching to the diabatic basis. However, diabatization itself is still an outstanding problem. We overcome these limitations by solving both problems at once. We use a machine learning approach combining clustering and regression techniques to correct for the deficiencies of property-based diabatization which, in return, provides us with smooth surfaces that can be easily fitted. Our approach extends the applicability of property-based diabatization to multidimensional systems. We utilize the proposed diabatization scheme to achieve higher prediction accuracy for adiabatic states and we show its performance by reconstructing global potential energy surfaces of excited states of nitrosyl fluoride and formaldehyde. While the proposed methodology is independent of the specific property-based diabatization and regression algorithm, we show its performance for kernel ridge regression and a very simple diabatization based on transition multipoles. Compared to most other algorithms based on machine learning, our approach needs only a small amount of training data.

MHCSeqNet2-improved peptide-class I MHC binding prediction for alleles with low data.

The binding of a peptide antigen to a Class I major histocompatibility complex (MHC) protein is part of a key process that lets the immune system recognize an infected cell or a cancer cell. This mechanism enabled the development of peptide-based vaccines that can activate the patient's immune response to treat cancers. Hence, the ability of accurately predict peptide-MHC binding is an essential component for prioritizing the best peptides for each patient. However, peptide-MHC binding experimental data for many MHC alleles are still lacking, which limited the accuracy of existing prediction models. In this study, we presented an improved version of MHCSeqNet that utilized sub-word-level peptide features, a 3D structure embedding for MHC alleles, and an expanded training dataset to achieve better generalizability on MHC alleles with small amounts of data. Visualization of MHC allele embeddings confirms that the model was able to group alleles with similar binding specificity, including those with no peptide ligand in the training dataset. Furthermore, an external evaluation suggests that MHCSeqNet2 can improve the prioritization of T cell epitopes for MHC alleles with small amount of training data. The source code and installation instruction for MHCSeqNet2 are available at https://github.com/cmb-chula/MHCSeqNet2.

A Few-Shot-Based Model-Agnostic Meta-Learning for Intrusion Detection in Security of Internet of Things

Currently, the effective technologies used to protect the security of the Internet of Things include blockchain and intrusion detection systems. The traditional IoT intrusion detection systems based on supervised learning usually requires a large amount of data for training, with high costs, and most of them can only work in several specific types of attacks. Faced with the variability of current IoT attack methods and the complexity of the network environment, they are usually unable to play a role in a short time. Especially in dealing with new security problems such as Zero-day attacks, which have a small number of learnable samples. Therefore, this paper proposed an IoT intrusion detection model to deal with the situation where learnable samples are insufficient. The model adopts the idea of meta-learning and divides the training process into two layers based on model-agnostic meta-learning. Its purpose is to train a relatively general model using the malicious network flow data of various known attack modes. To this end, a few-shot IoT intrusion detection dataset, FSIDS-IoT, is constructed based on five datasets, namely CIC-DDoS2019, CIC-IDS2017, CSE-CIC-IDS2018, NSL-KDD, and UNSW-NB15. Experiments show that our proposed approach, using only a few gradient steps and a small amount of training data from the new attack type, can perform well to protect cyber security. In the test of identifying unknown attacks, the optimal accuracy under the 5-way 1-shot setting reached 78.26%, 90.09% under the 5-way 5-shot setting, and 92.19% under the 5-way 10-shot setting.

Stroke classification and treatment support system artificial intelligence for usefulness of stroke diagnosis.

It is important to diagnose cerebral infarction at an early stage and select an appropriate treatment method. The number of stroke-trained physicians is unevenly distributed; thus, a shortage of specialists is a major problem in some regions. In this retrospective design study, we tested whether an artificial intelligence (AI) we built using computer-aided detection/diagnosis may help medical physicians to classify stroke for the appropriate treatment. To build the Stroke Classification and Treatment Support System AI, the clinical data of 231 hospitalized patients with ischemic stroke from January 2016 to December 2017 were used for training the AI. To verify the diagnostic accuracy, 151 patients who were admitted for stroke between January 2018 and December 2018 were also enrolled. By utilizing multimodal data, such as DWI and ADC map images, as well as patient examination data, we were able to construct an AI that can explain the analysis results with a small amount of training data. Furthermore, the AI was able to classify with high accuracy (Cohort 1, evaluation data 88.7%; Cohort 2, validation data 86.1%). In recent years, the treatment options for cerebral infarction have increased in number and complexity, making it even more important to provide appropriate treatment according to the initial diagnosis. This system could be used for initial treatment to automatically diagnose and classify strokes in hospitals where stroke-trained physicians are not available and improve the prognosis of cerebral infarction.

Data Augmentation for Sample Efficient and Robust Document Ranking

Contextual ranking models have delivered impressive performance improvements over classical models in the document ranking task. However, these highly over-parameterized models tend to be data-hungry and require large amounts of data even for fine-tuning. In this paper, we propose data-augmentation methods for effective and robust ranking performance. One of the key benefits of using data augmentation is in achieving sample efficiency or learning effectively when we have only a small amount of training data. We propose supervised and unsupervised data augmentation schemes by creating training data using parts of the relevant documents in the query-document pairs. We then adapt a family of contrastive losses for the document ranking task that can exploit the augmented data to learn an effective ranking model. Our extensive experiments on subsets of the MS MARCO and TREC-DL test sets show that data augmentation, along with the ranking-adapted contrastive losses, results in performance improvements under most dataset sizes. Apart from sample efficiency, we conclusively show that data augmentation results in robust models when transferred to out-of-domain benchmarks. Our performance improvements in in-domain and more prominently in out-of-domain benchmarks show that augmentation regularizes the ranking model and improves its robustness and generalization capability.

Advantages of Hybrid Neural Network Architectures to Enhance Prediction of Tensile Properties in Laser Powder Bed Fusion

The properties of AlSi10Mg produced by Laser Powder Bed Fusion (PBF-LB) are defined by a multitude of different machine and laser parameters. This multi-parameter space presents the challenge of optimizing the material properties for a given application by the sheer amount of possible parameter combinations. Characterizing this multi-parameter space empirically is limited by time and resources and thus yields an incomplete picture of the process capabilities and local optima, respectively. To improve on this situation, machine learning to map the process parameters on the tensile properties of AlSi10Mg was used. The Hybrid Neural Network (HNN) used in this study consisted of a Convolutional Neural Network (CNN) to process the micrographs and a Dense Neural Network (DNN) to process the LPBF process parameters as well as the output of the CNN. The micrographs given to the CNN part of the network were printed with the same parameters given to the DNN part to include the information of the bulk microstructure as it strongly influences the tensile properties of the material. With the HNN, we observed good accuracy of the predicted tensile properties, given the small amount of training data. Furthermore, we explore which features of the micrographs were extracted by the CNN.

Deep neural networks with adaptive solution space for inverse design of multilayer deep-etched grating

This article presents an inverse design technique of multilayer deep-etched grating (MDEG) using a deep neural network with adaptive solution space. MDEG is a key component in multispectral imaging, significantly affecting the quality of acquired multispectral images. Inverse design problems with one-to-many mappings from an optical response to multiple structures of MDEG have been addressed using machine learning techniques. Prior efforts, such as the tandem network and convolutional neural network (CNN), approached the problem from a modeling perspective. In this proposed method, a deep neural network is trained using cross-entropy loss for inverse design, predicting the probability distribution across the discretized space. The solution space is then formed by selecting the top K values with the highest probability, enabling evaluation of an optimal solution using the forward modeling network and avoiding the challenges of one-to-many mapping problems. Given the scarcity of optical training data, we explore training the network with reduced datasets of 30,000, 50,000 and 100,000 samples, demonstrating that the network performs well with a small amount of training data. Computational experiments conducted on the generated test dataset demonstrate that the proposed method improves in-band and out-of-band efficiencies by 26% and 12%, respectively, using only one-third of the required data.

Hybrid deep learning approach to improve classification of low-volume high-dimensional data

BackgroundThe performance of machine learning classification methods relies heavily on the choice of features. In many domains, feature generation can be labor-intensive and require domain knowledge, and feature selection methods do not scale well in high-dimensional datasets. Deep learning has shown success in feature generation but requires large datasets to achieve high classification accuracy. Biology domains typically exhibit these challenges with numerous handcrafted features (high-dimensional) and small amounts of training data (low volume).MethodA hybrid learning approach is proposed that first trains a deep network on the training data, extracts features from the deep network, and then uses these features to re-express the data for input to a non-deep learning method, which is trained to perform the final classification.ResultsThe approach is systematically evaluated to determine the best layer of the deep learning network from which to extract features and the threshold on training data volume that prefers this approach. Results from several domains show that this hybrid approach outperforms standalone deep and non-deep learning methods, especially on low-volume, high-dimensional datasets. The diverse collection of datasets further supports the robustness of the approach across different domains.ConclusionsThe hybrid approach combines the strengths of deep and non-deep learning paradigms to achieve high performance on high-dimensional, low volume learning tasks that are typical in biology domains.

Crosslingual Content Scoring in Five Languages Using Machine-Translation and Multilingual Transformer Models

AbstractThis paper investigates crosslingual content scoring, a scenario where scoring models trained on learner data in one language are applied to data in a different language. We analyze data in five different languages (Chinese, English, French, German and Spanish) collected for three prompts of the established English ASAP content scoring dataset. We cross the language barrier by means of both shallow and deep learning crosslingual classification models using both machine translation and multilingual transformer models. We find that a combination of machine translation and multilingual models outperforms each method individually - our best results are reached when combining the available data in different languages, i.e. first training a model on the large English ASAP dataset before fine-tuning on smaller amounts of training data in the target language.

Enhancing face recognition with latent space data augmentation and facial posture reconstruction

The small amount of training data for many state-of-the-art deep learning-based Face Recognition (FR) systems causes a marked deterioration in their performance. Although a considerable amount of research has addressed this issue by inventing new data augmentation techniques, using either input space transformations or Generative Adversarial Networks (GAN) for feature space augmentations, these techniques have yet to satisfy expectations. In this paper, we propose an approach named Face Representation Augmentation (FRA) for augmenting face datasets. To the best of our knowledge, FRA is the first method that shifts its focus towards manipulating the face embeddings generated by any face representation learning algorithm to create new embeddings representing the same identity and facial emotion but with an altered posture. Extensive experiments conducted in this study convince us of the efficacy of the methodology and its power to provide noiseless, completely new facial representations to improve the training procedure of any FR algorithm. Therefore, FRA can help the recent state-of-the-art FR methods by providing more data for training FR systems. The proposed method, using experiments conducted on the Karolinska Directed Emotional Faces (KDEF) dataset, improves the identity classification accuracies by 9.52 %, 10.04 %, and 16.60 %, in comparison with the base models of MagFace, ArcFace, and CosFace, respectively.

A comparison of methods for generating synthetic training data for domain adaption of deep learning models in ultrasonic non-destructive evaluation

This work provides a solution to the challenge of small amounts of training data in Non-Destructive Ultrasonic Testing for composite components. It was demonstrated that direct simulation alone is ineffective at producing training data that was representative of the experimental domain due to poor noise reconstruction. Therefore, four unique synthetic data generation methods were proposed which use semi-analytical simulated data as a foundation. Each method was evaluated for its performance in the classification of real experimental images when trained on a Convolutional Neural Network which underwent hyperparameter optimization using a genetic algorithm. The first method introduced task specific modifications to CycleGAN, a generative network for image-to-image translation, to learn the mapping from physics-based simulations of defect indications to experimental indications in resulting ultrasound images. The second method was based on combining real experimental defect free images with simulated defect responses. The final two methods fully simulated the noise responses at an image and signal level respectively. The purely simulated data produced a mean classification F1 score of 0.394. However, when trained on the new synthetic datasets, a significant improvement in classification performance on experimental data was realized, with mean classification F1 scores of 0.843, 0.688, 0.629, and 0.738 for the respective approaches.

A reduced order model for geometrically parameterized two-scale simulations of elasto-plastic microstructures under large deformations

In recent years, there has been a growing interest in understanding complex microstructures and their effect on macroscopic properties. In general, it is difficult to derive an effective constitutive law for such microstructures with reasonable accuracy and meaningful parameters. One numerical approach to bridge the scales is computational homogenization, in which a microscopic problem is solved at every macroscopic point, essentially replacing the effective constitutive model. Such approaches are, however, computationally expensive and typically infeasible in multi-query contexts such as optimization and material design. To render these analyses tractable, surrogate models that can accurately approximate and accelerate the microscopic problem over a large design space of shapes, material and loading parameters are required. In this work, we develop a reduced order model based on Proper Orthogonal Decomposition (POD), Empirical Cubature Method (ECM) and a geometrical transformation method with the following key features: (i) large shape variations of the microstructure are captured, (ii) only relatively small amounts of training data are necessary, and (iii) highly non-linear history-dependent behaviors are treated. The proposed framework is tested and examined in two numerical examples, involving two scales and large geometrical variations. In both cases, high speed-ups and accuracies are achieved while observing good extrapolation behavior.