Training Deep Neural Networks Research Articles

An Online Generation Method of Terminal-Area Trajectories for Wave-Rider Using Deep Neural Networks

This paper presents a deep neural network-based online trajectory generation method for the aerodynamic characteristic description and terminal-area energy management of wave-rider aircrafts. First, the flight dynamics equations in the energy domain are linearized and discretized to generate numerous aircraft trajectory samples with sequential convex optimization (SCO) methods. Then, an optimization objective function is designed to promote the smoothness of the control variables and improve the trajectory similarity. Compared to the nonlinear programming (NLP), the proposed trajectory sample generation method is more suitable for the training of deep neural networks (DNNs). Finally, deep neural networks are formulated and trained for the control variables and state variables, using the generated obtained trajectory samples, so that the reference trajectories can be obtained online during the energy management process of the wave-rider’s terminal phase. Numerical simulations validate the high accuracy of the trajectories generated with the deep neural network. Meanwhile, this proposed method enables smaller storage usage, which is highly suitable for integration into on-board flight control systems.

Towards a robust, effective and resource efficient machine learning technique for IoT security monitoring

The application of Deep Neural Networks (DNNs) for monitoring cyberattacks in Internet of Things (IoT) systems has gained significant attention in recent years. However, achieving optimal detection performance through DNN training has posed challenges due to computational intensity and vulnerability to adversarial samples. To address these issues, this paper introduces an optimization method that combines regularization and simulated micro-batching. This approach enables the training of DNNs in a robust, efficient, and resource-friendly manner for IoT security monitoring. Experimental results demonstrate that the proposed DNN model, including its performance in Federated Learning (FL) settings, exhibits improved attack detection and resistance to adversarial perturbations compared to benchmark baseline models and conventional Machine Learning (ML) methods typically employed in IoT security monitoring. Notably, the proposed method achieves significant reductions of 79.54% and 21.91% in memory and time usage, respectively, when compared to the benchmark baseline in simulated virtual worker environments. Moreover, in realistic testbed scenarios, the proposed method reduces memory footprint by 6.05% and execution time by 15.84%, while maintaining accuracy levels that are superior or comparable to state-of-the-art methods. These findings validate the feasibility and effectiveness of the proposed optimization method for enhancing the efficiency and robustness of DNN-based IoT security monitoring.

Learning from algorithm-generated pseudo-annotations for detecting ants in videos

Deep learning (DL) based detection models are powerful tools for large-scale analysis of dynamic biological behaviors in video data. Supervised training of a DL detection model often requires a large amount of manually-labeled training data which are time-consuming and labor-intensive to acquire. In this paper, we propose LFAGPA (Learn From Algorithm-Generated Pseudo-Annotations) that utilizes (noisy) annotations which are automatically generated by algorithms to train DL models for ant detection in videos. Our method consists of two main steps: (1) generate foreground objects using a (set of) state-of-the-art foreground extraction algorithm(s); (2) treat the results from step (1) as pseudo-annotations and use them to train deep neural networks for ant detection. We tackle several challenges on how to make use of automatically generated noisy annotations, how to learn from multiple annotation resources, and how to combine algorithm-generated annotations with human-labeled annotations (when available) for this learning framework. In experiments, we evaluate our method using 82 videos (totally 20,348 image frames) captured under natural conditions in a tropical rain-forest for dynamic ant behavior study. Without any manual annotation cost but only algorithm-generated annotations, our method can achieve a decent detection performance (77% in F_1 score). Moreover, when using only 10% manual annotations, our method can train a DL model to perform as well as using the full human annotations (81% in F_1 score).

Interactive visual analytics of parallel training strategies for DNN models

Understanding and optimizing the parallel training strategies in the training of large-scale Deep Neural Network (DNN) models is crucial to enhance training efficiency. Existing works tried to demonstrate the layer-level information of the computational graph to support parallel training strategy selection. Whereas, the overall parallel execution logic is rarely considered by previous methods. In this paper, we proposed a novel visual analytics approach for parallel training strategies, demonstrating the execution logic of the distributed computing from up to bottom via explaining communication operators. Specifically, a computation-communication bipartite construction algorithm is designed for the computational graph visualization. Furthermore, a system is developed to help users easily access the proposed approach and explore the parallel training strategies interactively. With empirical evaluation through a quantitative user study and a qualitative expert interview, the practicality and superiority of the proposed approach is verified.

Malignant Mesothelioma subtyping via sampling driven multiple instance prediction on tissue image and cell morphology data

Malignant Mesothelioma is a difficult to diagnose and highly lethal cancer usually associated with asbestos exposure. It can be broadly classified into three subtypes: Epithelioid, Sarcomatoid, and a hybrid Biphasic subtype in which significant components of both of the previous subtypes are present. Early diagnosis and identification of the subtype informs treatment and can help improve patient outcome. However, the subtyping of malignant mesothelioma, and specifically the recognition of transitional features from routine histology slides has a high level of inter-observer variability.In this work, we propose an end-to-end multiple instance learning (MIL) approach for malignant mesothelioma subtyping. This uses an adaptive instance-based sampling scheme for training deep convolutional neural networks on bags of image patches that allows learning on a wider range of relevant instances compared to max or top-N based MIL approaches. We also investigate augmenting the instance representation to include aggregate cellular morphology features from cell segmentation. The proposed MIL approach enables identification of malignant mesothelial subtypes of specific tissue regions. From this a continuous characterisation of a sample according to predominance of sarcomatoid vs epithelioid regions is possible, thus avoiding the arbitrary and highly subjective categorisation by currently used subtypes. Instance scoring also enables studying tumor heterogeneity and identifying patterns associated with different subtypes. We have evaluated the proposed method on a dataset of 234 tissue micro-array cores with an AUROC of 0.89±0.05 for this task. The dataset and developed methodology is available for the community at: https://github.com/measty/PINS.

A Novel Residual Gated Recurrent Unit Framework for Runoff Forecasting

Runoff forecasting is the key to the rational use and protection of water resources by mankind. The large-scale application of machine learning and neural networks in hydrological models has made accurate and reliable short-term runoff forecasting possible. In this paper, a novel short-term runoff forecasting framework called ResGRU Plus is proposed with gated recurrent unit (GRU) as the backbone. GRU has the characteristics of long short-term memory (LSTM) that can selectively memorize and forget information while merging gating units to reduce the amount of parameters. Residual network (ResNet) is also deeply integrated with GRU, and its unique shortcut connection effectively solves the degradation problem of traditional neural networks, making it possible to train deep neural networks based on recurrent architecture. Moreover, a lightweight attention mechanism module: squeeze-and-excitation network (SENet) is embedded in the framework. SENet explicitly models the interdependence between feature dimensions through one global average pooling layer (GAP) and two fully-connected layers, and rescales the original features through the learned weights to adaptively amplify or suppress features. Snapshot ensemble method is also used to train ResGRU Plus, which can integrate multiple homogeneous weak learners through one training process to improve the performance of the model at a small cost. In this paper, the hourly runoff of the Columbia River is used as the dataset. The Nash-Sutcliffe coefficient (NSE) of Efficiency and coefficient of determination (R2), which are two common evaluation metrics for hydrological models, are used to measure the performance of the models. Multiple sets of ablation experiments show that the proposed ResGRU Plus, which combines ResNet and attention mechanism, is able to improve depth by a factor of over 4 and accuracy by nearly 18% compared to the vanilla GRU, which further fully validates the effectiveness of combining the residual structure and attention mechanism with the recurrent architecture-based neural network and the feasibility of applying it to runoff forecasting. In addition, several sets of comparative experiments have also demonstrated the state-of-the-art performance of ResGRU Plus with significant improvement in accuracy compared to mainstream time series forecasting models.

Lightweight and Elegant Data Reduction Strategies for Training Acceleration of Convolutional Neural Networks

Due to industrial demands to handle increasing amounts of training data, lower the cost of computing one model at a time, and lessen the ecological effects of intensive computing resource consumption, the job of speeding the training of deep neural networks becomes exceedingly challenging. Adaptive Online Importance Sampling and IDS are two brand-new methods for accelerating training that are presented in this research. On the one hand, Adaptive Online Importance Sampling accelerates neural network training by lowering the number of forward and backward steps depending on how poorly a model can identify a given data sample. On the other hand, Intellectual Data Selection accelerates training by removing semantic redundancies from the training dataset and subsequently lowering the number of training steps. The study reports average 1.9x training acceleration for ResNet50, ResNet18, MobileNet v2 and YOLO v5 on a variety of datasets: CIFAR-100, CIFAR-10, ImageNet 2012 and MS COCO 2017, where training data are reduced by up to five times. Application of Adaptive Online Importance Sampling to ResNet50 training on ImageNet 2012 results in 2.37 times quicker convergence to 71.7% top-1 accuracy, which is within 5% of the baseline. Total training time for the same number of epochs as the baseline is reduced by 1.82 times, with an accuracy drop of 2.45 p.p. The amount of time required to apply Intellectual Data Selection to ResNet50 training on ImageNet 2012 is decreased by 1.27 times with a corresponding decline in accuracy of 1.12 p.p. Applying both methods to ResNet50 training on ImageNet 2012 results in 2.31 speedup with an accuracy drop of 3.5 p.p.

Medical prediction from missing data with max-minus negative regularized dropout.

Missing data is a naturally common problem faced in medical research. Imputation is a widely used technique to alleviate this problem. Unfortunately, the inherent uncertainty of imputation would make the model overfit the observed data distribution, which has a negative impact on the model generalization performance. R-Drop is a powerful technique to regularize the training of deep neural networks. However, it fails to differentiate the positive and negative samples, which prevents the model from learning robust representations. To handle this problem, we propose a novel negative regularization enhanced R-Drop scheme to boost performance and generalization ability, particularly in the context of missing data. The negative regularization enhanced R-Drop additionally forces the output distributions of positive and negative samples to be inconsistent with each other. Especially, we design a new max-minus negative sampling technique that uses the maximum in-batch values to minus the mini-batch to yield the negative samples to provide sufficient diversity for the model. We test the resulting max-minus negative regularized dropout method on three real-world medical prediction datasets, including both missing and complete cases, to show the effectiveness of the proposed method.

Scalable K-FAC Training for Deep Neural Networks With Distributed Preconditioning

The second-order optimization methods, notably the D-KFAC (Distributed Kronecker Factored Approximate Curvature) algorithms, have gained traction on accelerating deep neural network (DNN) training on GPU clusters. However, existing D-KFAC algorithms require to compute and communicate a large volume of second-order information, i.e., Kronecker factors (KFs), before preconditioning gradients, resulting in large computation and communication overheads as well as a high memory footprint. In this paper, we propose DP-KFAC, a novel distributed preconditioning scheme that distributes the KF constructing tasks at different DNN layers to different workers. DP-KFAC not only retains the convergence property of the existing D-KFAC algorithms but also enables three benefits: reduced computation overhead in constructing KFs, no communication of KFs, and low memory footprint. Extensive experiments on a 64-GPU cluster show that DP-KFAC reduces the computation overhead by 1.55×-1.65×, the communication cost by 2.79×-3.15×, and the memory footprint by 1.14×-1.47× in each second-order update compared to the state-of-the-art D-KFAC methods. Our codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lzhangbv/kfac\_pytorch</uri> .

TRUST-TECH-Based Systematic Search for Multiple Local Optima in Deep Neural Nets.

Training deep neural networks (DNNs) rested heavily on efficient local solvers. Due to their local property, local solvers are sensitive to initialization and hyperparameters. In this article, a systematical method for finding multiple high-quality local optimal DNNs, based on the transformation under stability-retaining equilibria characterization (TRUST-TECH) method, is introduced. Our goal is to systematically search for multiple local optimal parameters for large models, such as DNNs, trained on large datasets. To achieve this, a dynamic searching path (DSP) method is proposed to provide improved search guidance used in TRUST-TECH. By integrating the DSP method with the TRUST-TECH (DSP-TT) method, multiple optimal training solutions with higher quality than randomly initialized ones can be obtained. To take advantage of these optimal solutions, a DSP-TT ensemble method is further developed. Experiments on various test cases show that the proposed DSP-TT method achieves considerable improvement over other ensemble methods developed for deep architectures. The DSP-TT ensemble method also shows diversity advantages over other ensemble methods.

Deep Perceptual Image Enhancement Network for Exposure Restoration.

Image restoration techniques process degraded images to highlight obscure details or enhance the scene with good contrast and vivid color for the best possible visibility. Poor illumination condition causes issues, such as high-level noise, unlikely color or texture distortions, nonuniform exposure, halo artifacts, and lack of sharpness in the images. This article presents a novel end-to-end trainable deep convolutional neural network called the deep perceptual image enhancement network (DPIENet) to address these challenges. The novel contributions of the proposed work are: 1) a framework to synthesize multiple exposures from a single image and utilizing the exposure variation to restore the image and 2) a loss function based on the approximation of the logarithmic response of the human eye. Extensive computer simulations on the benchmark MIT-Adobe FiveK and user studies performed using Google high dynamic range, DIV2K, and low light image datasets show that DPIENet has clear advantages over state-of-the-art techniques. It has the potential to be useful for many everyday applications such as modernizing traditional camera technologies that currently capture images/videos with under/overexposed regions due to their sensors limitations, to be used in consumer photography to help the users capture appealing images, or for a variety of intelligent systems, including automated driving and video surveillance applications.

"Detecting and Removing Clouds Affected Regions from Satellite Images Using Deep Learning"

Deep Learning is becoming a very popular tool for generating and reconstructing images. Research has shown that deep learning algorithms can perform cutting-edge restoration tasks for various types of images. The performance of these algorithms can be achieved by training Deep Convolutional Neural Networks (DCNNs) with data from a large sample size. The processing of high-resolution satellite imagery becomes difficult when there are only a few images in a dataset. An approach based on the intrinsic properties of Deep Convolutional Neural Networks (DCNNs) is presented in this paper for the detection and removal of clouds from remote sensing images without any prior training. Our results demonstrated that the algorithm we used performed well when compared to trained algorithms.

Autoencoder and restricted Boltzmann machine for transfer learning in functional magnetic resonance imaging task classification

Deep neural networks (DNNs) have been adopted widely as classifiers for functional magnetic resonance imaging (fMRI) data, advancing beyond traditional machine learning models. Consequently, transfer learning of the pre-trained DNN becomes crucial to enhance DNN classification performance, specifically by alleviating an overfitting issue that occurs when a substantial number of DNN parameters are fitted to a relatively small number of fMRI samples. In this study, we first systematically compared the two most popularly used, unsupervised pretraining models for resting-state fMRI (rfMRI) volume data to pre-train the DNNs, namely autoencoder (AE) and restricted Boltzmann machine (RBM). The group in-brain mask used when training AE and RBM displayed a sizable overlap ratio with Yeo's seven functional brain networks (FNs). The parcellated FNs obtained from the RBM were fine-grained compared to those from the AE. The pre-trained AE and RBM served as the weight parameters of the first of the two hidden DNN layers, and the DNN fulfilled the task classifier role for fMRI (tfMRI) data in the Human Connectome Project (HCP). We tested two transfer learning schemes: (1) fixing and (2) fine-tuning the DNN's pre-trained AE or RBM weights. The DNN with transfer learning was compared to a baseline DNN, trained using random initial weights. Overall, DNN classification performance from the transfer learning proved superior when the pre-trained RBM weights were fixed and when the pre-trained AE weights were fine-tuned (average error rates: 14.8% for fixed RBM, 15.1% fine-tuned AE, and 15.5% for the baseline model) compared to the alternative scenarios of DNN transfer learning schemes. Moreover, the optimal transfer learning scheme between the fixed RBM and fine-tuned AE varied according to seven task conditions in the HCP. Nonetheless, the computational load reduced substantially for the fixed-weight-based transfer learning compared to the fine-tuning-based transfer learning (e.g., the number of weight parameters for the fixed-weight-based DNN model reduced to 1.9% compared with a baseline/fine-tuned DNN model). Our findings suggest that weight initialization at the DNN's first layer using RBM-based pre-trained weights provides the most promising approach when the whole-brain fMRI volume supports associated task classification. We believe that our proposed scheme could be applied to a variety of task conditions to improve their classification performance and to utilize computational resources efficiently using our AE/RBM-based pre-trained weights compared to random initial weights for DNN training.

Implementation of Deep Neural Networks for performance prediction and optimization of a porous volumetric solar receiver considering mechanical safety

The volumetric solar receiver is a major component of concentrating solar thermal systems. Proper selection of design and operating parameters is necessary to obtain the best receiver performance. The current work models and optimizes the hydraulic, thermal, and mechanical performance of a porous volumetric solar receiver using the Deep Neural Network (DNN) technique. The dataset for DNN training is generated by numerical simulations and porosity and pore size of foam structure, inlet fluid velocity and absorber length are selected as the input parameters. The pressure drop, outlet fluid temperature and maximum failure index are selected as the output variables. The trained model demonstrates high accuracy in performance prediction for any combination of input variables in the trained range. Further, the present work proposes the integration of the DNN technique with Genetic Algorithm for the constrained multi-objective optimization of the absorber performance with a novel constraint of maximum failure index to select configurations preventing mechanical failure. Such integration is found to be nearly 524 times faster than the conventional direct approach, demonstrating a significant decrease in the time needed for optimization. Furthermore, the potential of the Technique for Order of Preference by Similarity to Ideal Solution in selecting the desired configuration from the Pareto optimal solutions as per individual requirements is also investigated. The optimization results indicate that the optimum values for porosity, pore size, and absorber length are in the range of 0.9–0.93, 2–4 mm, and 0.01–0.06 m, respectively, when all variables are optimized simultaneously. Furthermore, the optimal input velocity is observed to be close to the lower bound of 0.4 m/s while the corresponding maximum failure index varies from 0.346 to 0.836, indicating mechanically safe configurations. The high accuracy and faster optimization results obtained in the present work demonstrate the ability of the proposed approach in implementing quick and computationally cheap optimization studies for improved and mechanically safe absorber designs.

Beyond Correlations: Deep Learning for Seismic Interferometry.

Passive seismic interferometry is a vastly generalized blind deconvolution question, where different paths through the Earth correspond to different channels called Green's functions; the sources are completely incoherent and not shared by the channels, and the question is to estimate paths (channels) that are not present in the dataset. SI, turning noise to signal, has numerous applications, from monitoring industrial activities to crustal structure investigation. No standard method of signal processing will solve SI. Instead, domain scientists resort to a simple cross-correlation operation, a.k.a. correlogram, which can retrieve the Green's function directly, but only under restrictive assumptions of ergodicity (energy equipartitioning) of the random process generating the seismic source. However, in practice, correlograms are not equal to the empirical Green's function, because these assumptions are generally far from being satisfied in realistic situations. In the framework of supervised learning, we propose to train deep neural networks (NNs) to overcome two limitations of correlation-based SI: the temporal limitation of passive recordings and the spatial limitation of the random source distribution. Deep NNs are trained to implicitly find the relationship between the empirical Green's functions and the correlograms and then used to extract the correct Green's functions from ambient noise. The input of the network is correlograms (a virtual shot gather), and the desired output is the empirical Green's function (the active shot gather). The NN can often retrieve Green's functions from 5-min passive recordings with acceptable accuracy in our synthetic example. Although an exact estimation of the source locations may not be necessary, a prior knowledge of the source directionality (through a preliminary beamforming step) is helpful when training the NN to mitigate the challenges associated with inhomogeneous source distributions (directional wave fields). In this work, all the numerical examples are based on the retrieval of P-wave reflections in the exploration scale and are conducted on synthetic data. We use a modified ResNet in our numerical experiments.

Energy-Aware, Device-to-Device Assisted Federated Learning in Edge Computing

The surging of deep learning brings new vigor and vitality to shape the prospect of intelligent Internet of Things (IoT), and the rise of edge intelligence enables provisioning real-time deep neural network (DNN) inference services for mobile users. To perform efficient and effective DNN model training in edge computing environments while preserving training data security and privacy of IoT devices, federated learning has been envisioned as an ideal learning paradigm for this purpose. In this paper, we study energy-aware DNN model training in edge computing. We first formulate a novel energy-aware, Device-to-Device (D2D) assisted federated learning problem with the aim to minimize the global loss of a training DNN model, subject to bandwidth capacity on an edge server and energy capacity on each IoT device. We then devise a near-optimal learning algorithm for the problem when the training data follows the i.i.d. data distribution. The crux of the proposed algorithm is to explore using the energy of neighboring devices of each device for its local model uploading, by reducing the problem to a series of weighted maximum matching problems in corresponding auxiliary graphs. We also consider the problem without the assumption of the i.i.d. data distribution, for which we propose an efficient heuristic algorithm. We finally evaluate the performance of the proposed algorithms through experimental simulations. Experimental results show that the proposed algorithms are promising.

APPLICATION OF MACHINE LEARNING ALGORITMS FOR HEART DISEASE PREDICTION

This paper focuses on the application of machine learning algorithms to predict cardiovascular diseases (CVDs). Every year a large number of deaths are registered all over the world. According to the World Health Organisation, CVDs are the leading cause of high mortality in the world. One of the necessary preventive measures to reduce mortality from CVDs is the timely prediction of diseases in people at high risk of such diseases. Specially developed scales and machine learning algorithms are now being used for the timely prediction of CVDs.&#x0D; Background. To predict heart disease, algorithms are often used: naive Bayesian classifier (Gaussian Naïve Bayes Classificator, GNBC), k-nearest neighbours (K-Nearest Neghboors, KNN), decision tree (Decision Tree, DT). In domestic literature, there are known works devoted to the application of SWD prediction using Adam gradient algorithm in deep neural network training. One of the necessary conditions for increasing the predictive ability of a machine learning model (MLM) is the optimal selection of hyperparameters. The choice of the optimal hyperparameters is often made on the basis of empirical experience.&#x0D; Purpose. To explore the specific application of machine algorithms to the prediction of heart disease.&#x0D; Materials and methods. Scientific novelty of the work. In this research we analyse machine learning algorithms for predicting the risk of CVDs using the approach of automatic search for hyperparameters MMO. The following algorithms were used to construct MMOs: NBS, KNN, DT, Logistic Regression, Support Vector Machine (SVM), Random Foorest (RF), Complement Naïve Bayes Classificator (CNBC), Linear Discriminant Analysis (LDA), Radial Basic Function (RBF), Gradient Boost (XGBoost). To evaluate the accuracy of machine learning models we used the following indicators: mean absolute error (MAE), precision, completeness (recall), F-measure (F-beta), False Positive Rate (FPR), False Negative Rate (FNR). Additionally, visual analysis of ROC curve (receiver operating characteristic) and areas under the curve (areas under the curve, AUC) were used to analyse the results of MMO. Using AUC value allows to estimate prognostic ability of MLM.&#x0D; Results. The training results showed that RF and XGBoost algorithms are characterized by higher accuracy. With optimal selection of MMO parameters, the overall classification accuracy was 0.88 and 0.94 respectively.&#x0D; Conclusion. The application of machine learning algorithms allows predictive models to be built with high accuracy. This requires the construction of a machine learning model. The ensemble machine learning algorithms RF and XGBoost have higher accuracy rates than the following algorithms: decision trees, Bayesian classification methods, logistic regression, linear discriminant analysis.

Class-Independent Regularization for Learning with Noisy Labels

Training deep neural networks (DNNs) with noisy labels often leads to poorly generalized models as DNNs tend to memorize the noisy labels in training. Various strategies have been developed for improving sample selection precision and mitigating the noisy label memorization issue. However, most existing works adopt a class-dependent softmax classifier that is vulnerable to noisy labels by entangling the classification of multi-class features. This paper presents a class-independent regularization (CIR) method that can effectively alleviate the negative impact of noisy labels in DNN training. CIR regularizes the class-dependent softmax classifier by introducing multi-binary classifiers each of which takes care of one class only. Thanks to its class-independent nature, CIR is tolerant to noisy labels as misclassification by one binary classifier does not affect others. For effective training of CIR, we design a heterogeneous adaptive co-teaching strategy that forces the class-independent and class-dependent classifiers to focus on sample selection and image classification, respectively, in a cooperative manner. Extensive experiments show that CIR achieves superior performance consistently across multiple benchmarks with both synthetic and real images. Code is available at https://github.com/RumengYi/CIR.

Soft Target-Enhanced Matching Framework for Deep Entity Matching

Deep Entity Matching (EM) is one of the core research topics in data integration. Typical existing works construct EM models by training deep neural networks (DNNs) based on the training samples with onehot labels. However, these sharp supervision signals of onehot labels harm the generalization of EM models, causing them to overfit the training samples and perform badly in unseen datasets. To solve this problem, we first propose that the challenge of training a well-generalized EM model lies in achieving the compromise between fitting the training samples and imposing regularization, i.e., the bias-variance tradeoff. Then, we propose a novel Soft Target-EnhAnced Matching (Steam) framework, which exploits the automatically generated soft targets as label-wise regularizers to constrain the model training. Specifically, Steam regards the EM model trained in previous iteration as a virtual teacher and takes its softened output as the extra regularizer to train the EM model in the current iteration. As such, Steam effectively calibrates the obtained EM model, achieving the bias-variance tradeoff without any additional computational cost. We conduct extensive experiments over open datasets and the results show that our proposed Steam outperforms the state-of-the-art EM approaches in terms of effectiveness and label efficiency.

Neighborhood-Regularized Self-Training for Learning with Few Labels

Training deep neural networks (DNNs) with limited supervision has been a popular research topic as it can significantly alleviate the annotation burden. Self-training has been successfully applied in semi-supervised learning tasks, but one drawback of self-training is that it is vulnerable to the label noise from incorrect pseudo labels. Inspired by the fact that samples with similar labels tend to share similar representations, we develop a neighborhood-based sample selection approach to tackle the issue of noisy pseudo labels. We further stabilize self-training via aggregating the predictions from different rounds during sample selection. Experiments on eight tasks show that our proposed method outperforms the strongest self-training baseline with 1.83% and 2.51% performance gain for text and graph datasets on average. Our further analysis demonstrates that our proposed data selection strategy reduces the noise of pseudo labels by 36.8% and saves 57.3% of the time when compared with the best baseline. Our code and appendices will be uploaded to: https://github.com/ritaranx/NeST.