Small-scale Training Research Articles

A CSI-Based Data-Driven Localization Framework Using Small-Scale Training Datasets in Single-Site MIMO Systems

A CSI-Based Data-Driven Localization Framework Using Small-Scale Training Datasets in Single-Site MIMO Systems

Clinical information extraction for lower-resource languages and domains with few-shot learning using pretrained language models and prompting

Abstract A vast amount of clinical data are still stored in unstructured text. Automatic extraction of medical information from these data poses several challenges: high costs of clinical expertise, restricted computational resources, strict privacy regulations, and limited interpretability of model predictions. Recent domain adaptation and prompting methods using lightweight masked language models showed promising results with minimal training data and allow for application of well-established interpretability methods. We are first to present a systematic evaluation of advanced domain-adaptation and prompting methods in a lower-resource medical domain task, performing multi-class section classification on German doctor’s letters. We evaluate a variety of models, model sizes (further-pre)training and task settings, and conduct extensive class-wise evaluations supported by Shapley values to validate the quality of small-scale training data and to ensure interpretability of model predictions. We show that in few-shot learning scenarios, a lightweight, domain-adapted pretrained language model, prompted with just 20 shots per section class, outperforms a traditional classification model, by increasing accuracy from $48.6\%$ to $79.1\%$ . By using Shapley values for model selection and training data optimization, we could further increase accuracy up to $84.3\%$ . Our analyses reveal that pretraining of masked language models on general-language data is important to support successful domain-transfer to medical language, so that further-pretraining of general-language models on domain-specific documents can outperform models pretrained on domain-specific data only. Our evaluations show that applying prompting based on general-language pretrained masked language models combined with further-pretraining on medical-domain data achieves significant improvements in accuracy beyond traditional models with minimal training data. Further performance improvements and interpretability of results can be achieved, using interpretability methods such as Shapley values. Our findings highlight the feasibility of deploying powerful machine learning methods in clinical settings and can serve as a process-oriented guideline for lower-resource languages and domains such as clinical information extraction projects.

Fast Hyperspectral Image Classification with Strong Noise Robustness Based on Minimum Noise Fraction

A fast hyperspectral image classification algorithm with strong noise robustness is proposed in this paper, aiming at the hyperspectral image classification problems under noise interference. Based on the Fast 3D Convolutional Neural Network (Fast-3DCNN), this algorithm enables the classification model to have good tolerance for various types of noise by using a Minimum Noise Fraction (MNF) as dimensionality reduction module for hyperspectral image input data. In addition, by introducing lightweight hybrid attention modules with the spatial and the channel information, the deep features extracted by the Convolutional Neural Network are further refined, ensuring that the model has high classification accuracy. Public dataset experiments have shown that compared to traditional methods, the MNF in this algorithm reduces the dimensionality of input spectral data, preserves information with higher signal-to-noise ratio(SNR) in the spectral bands, and aggregates spectral features into class feature vectors, greatly improving the noise robustness of the model. At the same time, based on a lightweight spectral–spatial hybrid attention mechanism, combined with fewer spectral dimensions, the model effectively avoids overfitting. With less loss in model training speed, it achieved better classification accuracy in small-scale training sample experiments, fully demonstrating the good generalization ability of this algorithm.

Enhancing generalization in genetic programming hyper-heuristics through mini-batch sampling strategies for dynamic workflow scheduling

Genetic Programming Hyper-heuristics (GPHH) have been successfully used to evolve scheduling rules for Dynamic Workflow Scheduling (DWS) as well as other challenging combinatorial optimization problems. The method of sampling training instances has a significant impact on the generalization ability of GPHH, yet they are rarely addressed in existing research. This article aims to fill this gap by proposing a GPHH algorithm with a sampling strategy to thoroughly investigate the impact of six instance sampling strategies on algorithmic generalization, including one rotation strategy, three mini-batch strategies, and two hybrid strategies. Experiments across four scenarios with varying settings reveal that: (1) mini-batch with random sampling can outperform rotation in generalizing to unseen workflow scheduling problems under the same computational cost; (2) employing a hybrid strategy that combines rotation and mini-batch further enhances the generalization ability of GPHH; and (3) mini-batch and hybrid strategies can effectively enable heuristics trained on small-scale training instances generalizing well to large-scale unseen ones. These findings highlight the potential of mini-batch strategies in GPHH, offering improved generalization performance while maintaining diversity and suggesting promising avenues for further exploration in GPHH domains.

A stacked deep multi-kernel learning framework for blast induced flyrock prediction

Blasting operations are widely and frequently used for rock excavation in Civil and Mining constructions. Flyrock is one of the most important issues induced by blasting operations in open pit mines, and therefore needs to be well predicted in order to identify the safety zone to prevent the potential injuries. For this purpose, 234 sets of blasting data were collected from Sungun Copper Mine site, and a stacked deep multi-kernel learning (SD-MKL) framework was proposed to estimate the blast induced flyrock with confidence accuracy. The proposed model uses the stacking-based representation learning framework (S-RL) to achieve deep learning on small-scale training sets. A multi-kernel learning model (MKL) is used as the base module of S-RL framework, which uses a multi-feature fusion strategy to generate multiple kernels with different kernel length in order to reduce the effort in tuning hyperparameters. In addition, this study further enhanced the predictive capability of SD-MKL by introducing the boosting method into the S-RL framework and hence proposed a boosted SD-MKL model. For comparison purpose, several existing machine learning models were implemented, i.e., kernel ridge regression (KRR), support vector machine (SVM), random forest (RF), gradient boosting decision tree (GBDT), ensemble deep random vector functional link (edRVFL), SD-KRR and SD-SVM. Our experimental results showed that the proposed boosted SD-MKL achieved the best overall performance, with the lowest RMSE of 0.21/1.73, MAE of 0.08/0.78, and the highest VAF of 99.98/99.24.

Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training

The goal of this paper is to alleviate the training cost for few-shot semantic segmentation (FSS) models. Despite that FSS in nature improves model generalization to new concepts using only a handful of test exemplars, it relies on strong supervision from a considerable amount of labeled training data for base classes. However, collecting pixel-level annotations is notoriously expensive and time-consuming, and small-scale training datasets convey low information density that limits test-time generalization. To resolve the issue, we take a pioneering step towards label-efficient training of FSS models from fully unlabeled training data, or additionally a few labeled samples to enhance the performance. This motivates an approach based on a novel unsupervised meta-training paradigm. In particular, the approach first distills pre-trained unsupervised pixel embedding into compact semantic clusters from which a massive number of pseudo meta-tasks is constructed. To mitigate the noise in the pseudo meta-tasks, we further advocate a robust Transformer-based FSS model with a novel prototype-based cross-attention design. Extensive experiments have been conducted on two standard benchmarks, i.e., PASCAL-5i and COCO-20i, and the results show that our method produces impressive performance without any annotations, and is comparable to fully supervised competitors even using only 20% of the annotations. Our code is available at: https://github.com/SSSKYue/UMTFSS.

BI-TST_YOLOv5: Ground Defect Recognition Algorithm Based on Improved YOLOv5 Model

Pavement defect detection technology stands as a pivotal component within intelligent driving systems, demanding heightened precision and rapid detection rates. Addressing the complexities arising from diverse defect types and intricate backgrounds in visual sensing, this study introduces an enhanced approach to augment the network structure and activation function within the foundational YOLOv5 algorithm. Initially, modifications to the YOLOv5′s architecture incorporate an adjustment to the Leaky ReLU activation function, thereby enhancing regression stability and accuracy. Subsequently, the integration of bi-level routing attention into the network’s head layer optimizes the attention mechanism, notably improving overall efficiency. Additionally, the replacement of the YOLOv5 backbone layer’s C3 module with the C3-TST module enhances initial convergence efficiency in target detection. Comparative analysis against the original YOLOv5s network reveals a 2% enhancement in map50 and a 1.8% improvement in F1, signifying an overall advancement in network performance. The initial convergence rate of the algorithm has been improved, and the accuracy and operational efficiency have also been greatly improved, especially on models with small-scale training sets.

A training program for improving the capacity of infection high-throughput sequencing and diagnosis in China

BackgroundInfectious diseases are a serious threat to human especially since the COVID-19 outbreak has proved the importance and urgency of their diagnosis and treatment again. Metagenomic next-generation sequencing (mNGS) has been widely used and recognized in clinical and carried out localized testing in hospitals. Increasing the training of mNGS detection technicians can enhance their professional quality and more effectively realize the application value of the hospital platform.MethodsBased on the initial theoretical understanding and practice of the mNGS platform for localization construction, we have designed a training program to enhance the ability of technicians to detect pathogens by utilizing mNGS, and hence to conduct training practices nationwide.ResultsUntil August 30, 2022, the page views of online classes have reached 51,500 times and 6 of offline small-scale training courses have been conducted. A total of 67 trainees from 67 hospitals have participated in the training with a qualified rate of 100%. After the training course, the localization platform of 1 participating hospital has been put into use, 2 have added the mNGS localization platform for admission, among which 3 have expressed strong intention of localization.ConclusionsThis study focuses on the training procedures and practical experience of the project which is the first systematic standardized program of mNGS in the world. It solves the training difficulties in the current industry, and effectively promotes the localization construction and application of mNGS in hospitals. It has great development potential in the future and is worth further promotion.

Double Consistency Regularization for Transformer Networks

The large-scale and deep-layer deep neural network based on the Transformer model is very powerful in sequence tasks, but it is prone to overfitting for small-scale training data. Moreover, the prediction result of the model with a small disturbance input is significantly lower than that without disturbance. In this work, we propose a double consistency regularization (DOCR) method for the end-to-end model structure, which separately constrains the output of the encoder and decoder during the training process to alleviate the above problems. Specifically, on the basis of the cross-entropy loss function, we build the mean model by integrating the model parameters of the previous rounds and measure the consistency between the models by calculating the KL divergence between the features of the encoder output and the probability distribution of the decoder output of the mean model and the base model so as to impose regularization constraints on the solution space of the model. We conducted extensive experiments on machine translation tasks, and the results show that the BLEU score increased by 2.60 on average, demonstrating the effectiveness of DOCR in improving model performance and its complementary impacts with other regularization techniques.

High-Impedance Fault Detection Methodology Using Time–Frequency Spectrum and Transfer Convolutional Neural Network in Distribution Networks

High-impedance fault (HIF) detection has always been difficult in distribution networks due to the lack of field data and the large difference between field and simulation waveforms. Based on the characteristics of zero-sequence currents, a novel HIF detection methodology is proposed, which combines time–frequency spectrum (TFS) and transfer convolutional neural network (TCNN). First, the TFSs are acquired by applying continuous wavelet transform (CWT) to the collected zero-sequence currents. Then, the TFSs of simulated zero-sequence currents are utilized for training source-domain convolutional neural network (SCNN). Next, the SCNN is transfer learned with very few TFSs of field zero-sequence currents to obtain TCNN. The performance of the proposed method is verified by simulation samples and field samples. The results show that the proposed method can effectively extract fault features from small-scale training samples under different fault circumstances. Besides, TCNN can adaptively extract the effective features of field HIF and detect field HIF more accurately than SCNN. Finally, this article provides a visualization scheme for interpretability of the neural network, which offers visual explanations for the decision-making basis of the neural network.

Poisson-based image editing for semi-supervised vitiligo lesion segmentation with limited annotations

Vitiligo lesion segmentation is crucial for the assessment and treatment of vitiligo. There are two significant challenges in this problem, namely, the availability of dense segmentation annotations and the collection of large amounts of vitiligo images, which are also major challenges in medical image analysis (MIA). However, most existing methods often heavily rely on the availability of large-scale labeled datasets and high-quality annotations. Consequently, the performance of these models may not be easily reproducible or transferable to those domains with limited data availability. As a result, there is a need to develop alternative approaches that can leverage unlabeled datasets for segmentation with a small-scale training set. In this paper, we propose a data augmentation strategy based on image editing, which can synthesize a large number of samples using a small number of annotated data. The synthesized examples are of high visual quality and enforce the segmentation performance without any cost. Besides, we also adapt the Mean-Teacher framework for reliable predictions mining from unlabeled samples to alleviate the demands of densely annotated segmentations. We obtain pseudo-labels for unlabeled samples by utilizing highly confident pixels. On the other hand, we proposed a new Bimodal Vitiligo Lesions Segmentation (BVLS) dataset containing fine-grain segmentation masks and bimodal images usually used for vitiligo diagnosis to mitigate the lack of a vitiligo segmentation dataset. Extensive experiments conducted on the BLVS dataset demonstrate that our approach can achieve significant improvements (+17.27%) compared with previous data augmentation methods on the UNet backbone. Furthermore, the semi-supervised framework can reach an IoU of 49.71% with only 10% annotated images. Our code and dataset are availabel at https://github.com/JcWang20/BLVS.

Disguised Face Recognition using Deep Learning

Abstract: The accuracy of facial recognition has significantly improved thanks to deep learning-based methods. However, several factors including facial ageing, disguises, and position fluctuations impair the efficiency of automatic face recognition algorithms. Disguises are commonly utilized to conceal one's identity or assume the identity of someone else by intentionally or unintentionally altering one's facial features. Here, we intend to make use of the Disguised Faces in the Wild (DFW) small-scale training data. Deep Convolutional Neural Networks(DCNNs) will be trained for general face recognition. The IIIIT-D testing data set will be used for model evaluation because it exhibits greater performance. The IIIT-D testing dataset is used to gauge the effectiveness of the model's performance when applied to faces that have been deliberately disguised. The performance of the results is encouraging and suggests that DCNNs have a chance of successfully recognizing disguised faces.

PreAugNet: improve data augmentation for industrial defect classification with small-scale training data

PreAugNet: improve data augmentation for industrial defect classification with small-scale training data

Accelerating magnetotelluric forward modeling with deep learning: Conv-BiLSTM and D-LinkNet

Magnetotelluric forward modeling is important for exploring underground electromagnetic anomalies. Although directly solving the electromagnetic wave equation has high accuracy, its computational cost is usually unaffordable for large-scale models. A neural network (NN) can increase the computation of the magnetotelluric forward modeling; however, its numerical accuracy is limited owing to the use of a simple network structure and small-scale training data sets. We have increased the computational efficiency of the magnetotelluric forward modeling by deep learning based on cyclic NN and full convolution NN models. First, we have extracted the basic characteristics of the 2D-magnetotelluric forward modeling, which were important for selecting and optimizing the network. Then, we have developed two forward network models: convolutional bidirectional long short-term memory (Conv-BiLSTM) and LinkNet with pretrained encoder and dilated convolution (D-LinkNet). Next, we have constructed data sets of large-scale multiple anomalies. Finally, we have tested our models using various examples. Existing methods only consider a single anomaly and have low accuracy; in contrast, our methods can handle multianomaly models because they have strong generalization, even though the training is based on models with two or three anomalies. Numerical experiments find that the average accuracy of the Conv-BiLSTM and D-LinkNet forward network models was 87.5% and 95.3%, respectively. Compared with D-LinkNet, the Conv-BiLSTM network model has lower accuracy but higher computational efficiency. Our deep-learning schemes can significantly reduce the computational burdens of the magnetotelluric forward modeling, and thus allow us to perform swift inversions of multianomaly models.

Facial Expression Guided Diagnosis of Parkinson's Disease via High-Quality Data Augmentation

Parkinson's disease (PD) is a neurodegenerative disease which is prevalent among the elder population and severely affects the life quality of patients and their families. Therefore, it is important to conduct an early diagnosis for potential patients with PD, so as to promote prompt treatment and avoid the aggravation of the disease. Recently, the in-vitro PD diagnosis based on facial expressions has received increasing attention because of its distinguishability (i.e., PD patients always possess the characteristics of “masked face”) and affordability. However, the performance of the existing facial expression-based PD diagnosis approaches is limited by: 1) the small-scale training data on PD patients' facial expressions, and 2) the weak prediction model. To address these two problems, we propose a new facial expression guided PD diagnosis method based on high-quality training data augmentation and deep neural network prediction. Specifically, the proposed method consists of three stages: Firstly, we synthesize virtual facial expression images with 6 basic emotions (i.e., anger, disgust, fear, happiness, sadness, and surprise) based on multi-domain adversarial learning to approximate the premorbid expressions of PD patients. Secondly, we introduce three facial image quality assessment (FIQA) criteria to measure the quality of these synthesized facial expression images and design a fusion screening strategy that shortlists the high-quality ones to augment the training data. Finally, we train a deep neural network prediction model based on the original and synthesized high-quality facial expression images for PD diagnosis. To show real-world impacts and evaluate the proposed method under different facial expressions, we also create a (currently largest) multiple facial expressions-based PD face dataset in collaboration with a hospital. Extensive experiments are performed to demonstrate the effectiveness of the multi-domain adversarial learning-based facial expression synthesis and the fusion screening strategy, particularly the superior performance of the proposed method for PD diagnosis.

Semi-Supervised Deep Learning Model for Efficient Computation of Optical Properties of Suspended-Core Fibers.

Suspended-core fibers (SCFs) are considered the best candidates for enhancing fiber nonlinearity in mid-infrared applications. Accurate modeling and optimization of its structure is a key part of the SCF structure design process. Due to the drawbacks of traditional numerical simulation methods, such as low speed and large errors, the deep learning-based inverse design of SCFs has become mainstream. However, the advantage of deep learning models over traditional optimization methods relies heavily on large-scale a priori datasets to train the models, a common bottleneck of data-driven methods. This paper presents a comprehensive deep learning model for the efficient inverse design of SCFs. A semi-supervised learning strategy is introduced to alleviate the burden of data acquisition. Taking SCF’s three key optical properties (effective mode area, nonlinear coefficient, and dispersion) as examples, we demonstrate that satisfactory computational results can be obtained based on small-scale training data. The proposed scheme can provide a new and effective platform for data-limited physical computing tasks.

Domain adaptive multi-task transformer for low-resource machine reading comprehension

In recent years, low-resource Machine Reading Comprehension (MRC) attracts increasing attention. Due to the difficulty in data collecting, current low-resource MRC approaches often suffer from poor generalizing capability: the model only learns limited task-aware and domain-aware knowledge from a small-scale training dataset. Previous works generally address such deficiency by learning the required knowledge from out-of-domain MRC datasets and in-domain self-supervised datasets. However, such approaches also introduce domain noise and task noise. This paper proposes a Domain Adaptive Multi-Task Transformer (DAMT2) to tackle these noises. For task noise, DAMT2 utilizes a well-designed Multi-Task Transformer (MT2) as the backbone to model the high-level features separately from different tasks. For domain noise, two kinds of domain adaptation approaches are incorporated into MT2 to learn domain-invariant representations. The experimental results show that our method outperforms several baselines on multiple datasets, and especially achieves a new SOTA on the RRC dataset. Moreover, using only 40%-60% training data, our work achieves comparable performance with the classic BERT model.

Multistage Deep Transfer Learning for EmIoT-Enabled Human–Computer Interaction

Emotional Internet of Things (EmIoT), which provides Internet of Things (IoT) devices cognitive and socialization capabilities, has been regarded as a future direction to improve users’ experiences. With the development of intelligent techniques, the requirement of EmIoT is not only sensing the users’ emotional states but also providing emotional feedbacks. Human–computer interaction has been studied to achieve speech interaction with IoT devices. The recent advances in neural text-to-speech (TTS) have made “human parity” synthesized speech possible for IoT-enabled human–computer interaction. Furthermore, emotion control can be achieved by using the emotional codes in a unified model, referred to as emotional TTS (or ETTS for short). Such ETTS models have achieved promising emotional expressiveness using large-scale emotion-annotated English data set; however, they are not practical in IoT environments with other mainstream languages, especially for Chinese. In fact, the limited available large-scale emotion-annotated data set is challenging the development of Chinese ETTS. To address that we propose a multistage deep transfer learning scheme to design a high-quality Chinese ETTS system under a small-scale training corpus to achieve EmIoT in Mandarin environments. In this scheme, the pretrained knowledge from the former stages corresponding to a large-scale neutral English and a medium-scale emotional English corpora is transferred to a Mandarin ETTS model. Thereby, the trained model can achieve high-quality emotional speech with limited available emotional corpus, which is able to serve various EmIoT-oriented applications. The experiments have been conducted to demonstrate the effectiveness and superiority of the proposed model as compared to other counterparts in terms of naturalness and emotional expressiveness. We refer readers to visit our demo Webpage <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref> enjoy the synthesized speech samples.

Fusion of Multidimensional CNN and Handcrafted Features for Small-Sample Hyperspectral Image Classification

Hyperspectral image (HSI) classification has attracted widespread concern in recent years. However, due to the complexity of the HSI gathering environment, it is difficult to obtain a great number of HSI labeled samples. Therefore, how to effectively extract the spatial–spectral feature with small-scale training samples is the crucial point of HSI classification. In this paper, a novel fusion framework for small-sample HSI classification is proposed to fully combine the advantages of multidimensional CNN and handcrafted features. Firstly, a 3D fuzzy histogram of oriented gradients (3D-FHOG) descriptor is proposed to fully extract the handcrafted spatial–spectral feature of HSI pixels, which is suggested to be more robust by overcoming the local spatial–spectral feature uncertainty. Secondly, a multidimensional Siamese network (MDSN), which is updated by minimizing both contrastive loss and classification loss, is designed to effectively exploit the CNN-based spatial–spectral features from multiple dimensions. Finally, the proposed MDSN combined with 3D-FHOG is utilized for small-sample HSI classification to verify the effectiveness of our proposed fusion framework. The experimental results on three public data sets indicate that the proposed MDSN combined with 3D-FHOG is significantly better than the representative handcrafted feature-based and CNN-based methods, which in turn demonstrates the superiority of the proposed fusion framework.

Determinants of Small-Scale Irrigation Use for Poverty Reduction: The Case of Offa Woreda, Wolaita Zone, Southern Ethiopia

Small-scale irrigation is one of the agricultural activities used by rural farmers to improve the overall livelihood of the rural community by increasing income, securing food, meeting social requirements, and reducing poverty. The main objective of this study was to look into the factors that influence small-scale irrigation for poverty reduction among small-holder farmers in the Offa Woreda, Wolaita Zone. Government and nongovernmental organizations supplied small-scale irrigation and training to small-holder farmers in the study area at various times; however, the key determinants impacting small-scale irrigation use were not fully identified in the study area. The study location was chosen for this study purpose because no prior in-depth research had been conducted. Simple random sampling was used to select the three kebeles for the study. A total of 134 people were included in the study sample frame. The sample was chosen using simple random sampling and the proportional probability-to-size (pps) method, which was proportional to the size and used multistage sampling procedure. The primary data were collected using the interview schedule. Both quantitative and qualitative data were analyzed using descriptive and econometric analyzes in SPSS version 21. Qualitative data were also collected through focus groups and key informant interviews. A factor that influences small-scale irrigation use was investigated using a binary logistic regression model. The binary logistic regression model revealed that both the irrigation participants (37.3%) and the nonparticipant (62.7%) families had a combination of personal and demographic, economic, institutional, and social influencing factors that hinder their likelihood of using irrigation. HH age, annual income of the household, distance from the closest farm sight, farm experience, participation in training, access to irrigation, and sex of HH were positively and significantly influencing factors in the model, while total ownership of livestock of households and health status of households were negatively significant variables. As a result, the Agricultural and Natural Resources Development Office, as well as the Water Mining and Energy Offices, should devote scientific attention to the significant factors that influence irrigation use to boost farm family production and productivity.