Large-scale Training Data Research Articles

CRS-CONT: A Well-Trained General Encoder for Facial Expression Analysis.

Existing facial expression recognition (FER) methods train encoders with different large-scale training data for specific FER applications. In this paper, we propose a new task in this field. This task aims to pre-train a general encoder to extract any facial expression representations without fine-tuning. To tackle this task, we extend the self-supervised contrastive learning to pre-train a general encoder for facial expression analysis. To be specific, given a batch of facial expressions, some positive and negative pairs are firstly constructed based on coarse-grained labels and a FER-specified data augmentation strategy. Secondly, we propose the coarse-contrastive (CRS-CONT) learning, where the features of positive pairs are pulled together, while pushed away from the features of negative pairs. Moreover, one key event is that the excessive constraint on the coarse-grained feature distribution will affect fine-grained FER applications. To address this, a weight vector is designed to control the optimization of the CRS-CONT learning. As a result, a well-trained general encoder with frozen weights could preferably adapt to different facial expressions and realize the linear evaluation on any target datasets. Extensive experiments on both in- the-wild and in- the-lab FER datasets show that our method provides superior or comparable performance against state-of-the-art FER methods, especially on unseen facial expressions and cross-dataset evaluation. We hope that this work will help to reduce the training burden and develop a new solution against the fully-supervised feature learning with fine-grained labels. Code and the general encoder will be publicly available at https://github.com/hangyu94/CRS-CONT.

Performance evaluation of selected cloud occlusion removal algorithms on remote sensing imagery

Removal of cloud cover from remote sensing satellite images is crucial to many optical remote sensing image data users because cloud cover can conceal important spatial information on an image data. This underscores the importance of making an informed choice in the selection of appropriate cloud cover detection and removal algorithms. For the purpose of large-scale training data, neural networks have been successful in many image processing tasks, but the use of neural networks to remove cloud occlusion in remote sensing imagery is still relatively evolving. The aim of this study is to evaluate the performance of two image restoration algorithms (The spatial attentive generative adversarial network and Convolutional autoencoder with symmetrical skip connection) used for the removal of cloud cover on remote sensing images. An open-source RICE dataset was used for the training and prediction of the models as each of them were implemented for the cloud cover removal. The evaluation metrics used to compare the models’ performance are the Structural similarity index ratio (SSIM), peak signal to noise ratio (PSNR), and the time taken for each model to complete its network training. After a successful completion of the network training using 80% of the data and the remaining 20% to test the networks, the spatial attentive generative adversarial network achieved the best performance on both the peak signal to noise ratio with a value of 26.3447 and the SSIM with a value of 0.8949 while the convolutional autoencoder generates a peak signal to noise ratio of 25.8257 and a SSIM of 0.6307. The result proves that SpaGAN is more effective for automatic cloud cover removal on remote sensing images and the improvement of the quality of the restored image when compared to CNN autoencoder.

Deeper Look at Image Salient Object Detection: Bi-Stream Network With a Small Training Dataset

Compared with the conventional hand-crafted approaches, the deep learning based ISOD (image salient object detection) models have achieved tremendous performance improvements by training exquisitely crafted fancy networks over large-scale training sets. However, do we really need large-scale training set for ISOD? In this article, we provide a deeper insight into the interrelationship between the ISOD performance and the training data. To alleviate the conventional demands for large-scale training data, we provide a feasible way to construct a novel small-scale training set, which only contains 4 K images. To take full advantage of this new set, we propose a novel bi-stream network consisting of two different feature backbones. Benefit from the proposed gate control unit, this bi-stream network is able to achieve complementary fusion status for its subbranches. To our best knowledge, this is the first attempt to use a small-scale training set to compete with other large-scale ones; nevertheless, our method can still achieve the leading SOTA performance on all tested benchmark datasets. Both the code and dataset are publicly available at <uri>https://github.com/wuzhenyubuaa/TSNet</uri>.

Combining Self-supervised Learning and Active Learning for Disfluency Detection

Spoken language is fundamentally different from the written language in that it contains frequent disfluencies or parts of an utterance that are corrected by the speaker. Disfluency detection (removing these disfluencies) is desirable to clean the input for use in downstream NLP tasks. Most existing approaches to disfluency detection heavily rely on human-annotated data, which is scarce and expensive to obtain in practice. To tackle the training data bottleneck, in this work, we investigate methods for combining self-supervised learning and active learning for disfluency detection. First, we construct large-scale pseudo training data by randomly adding or deleting words from unlabeled data and propose two self-supervised pre-training tasks: (i) a tagging task to detect the added noisy words and (ii) sentence classification to distinguish original sentences from grammatically incorrect sentences. We then combine these two tasks to jointly pre-train a neural network. The pre-trained neural network is then fine-tuned using human-annotated disfluency detection training data. The self-supervised learning method can capture task-special knowledge for disfluency detection and achieve better performance when fine-tuning on a small annotated dataset compared to other supervised methods. However, limited in that the pseudo training data are generated based on simple heuristics and cannot fully cover all the disfluency patterns, there is still a performance gap compared to the supervised models trained on the full training dataset. We further explore how to bridge the performance gap by integrating active learning during the fine-tuning process. Active learning strives to reduce annotation costs by choosing the most critical examples to label and can address the weakness of self-supervised learning with a small annotated dataset. We show that by combining self-supervised learning with active learning, our model is able to match state-of-the-art performance with just about 10% of the original training data on both the commonly used English Switchboard test set and a set of in-house annotated Chinese data.

Medical image segmentation with generative adversarial semi-supervised network

Recent medical image segmentation methods heavily rely on large-scale training data and high-quality annotations. However, these resources are hard to obtain due to the limitation of medical images and professional annotators. How to utilize limited annotations and maintain the performance is an essential yet challenging problem. In this paper, we try to tackle this problem in a self-learning manner by proposing a generative adversarial semi-supervised network. We use limited annotated images as main supervision signals, and the unlabeled images are manipulated as extra auxiliary information to improve the performance. More specifically, we modulate a segmentation network as a generator to produce pseudo labels for unlabeled images. To make the generator robust, we train an uncertainty discriminator with generative adversarial learning to determine the reliability of the pseudo labels. To further ensure dependability, we apply feature mapping loss to obtain statistic distribution consistency between the generated labels and the real labels. Then the verified pseudo labels are used to optimize the generator in a self-learning manner. We validate the effectiveness of the proposed method on right ventricle dataset, Sunnybrook dataset, STACOM, ISIC dataset, and Kaggle lung dataset. We obtain 0.8402–0.9121, 0.8103–0.9094, 0.9435–0.9724, 0.8635–0.886, and 0.9697–0.9885 dice coefficient with 1/8 to 1/2 proportion of densely annotated labels, respectively. The improvements are up to 28.6 points higher than the corresponding fully supervised baseline.

Adversarial Robustness of Deep Convolutional Neural Network-based Image Recognition Models: A Review

Deep convolutional neural networks have achieved great success in recent years. They have been widely used in various applications such as optical and SAR image scene classification, object detection and recognition, semantic segmentation, and change detection. However, deep neural networks rely on large-scale high-quality training data, and can only guarantee good performance when the training and test data are independently sampled from the same distribution. Deep convolutional neural networks are found to be vulnerable to subtle adversarial perturbations. This adversarial vulnerability prevents the deployment of deep neural networks in security-sensitive applications such as medical, surveillance, autonomous driving and military scenarios. This paper first presents a holistic view of security issues for deep convolutional neural network-based image recognition systems. The entire information processing chain is analyzed regarding safety and security risks. In particular, poisoning attacks and evasion attacks on deep convolutional neural networks are analyzed in detail. The root causes of adversarial vulnerabilities of deep recognition models are also discussed. Then, we give a formal definition of adversarial robustness and present a comprehensive review of adversarial attacks, adversarial defense, and adversarial robustness evaluation. Rather than listing existing research, we focus on the threat models for the adversarial attack and defense arms race. We perform a detailed analysis of several representative adversarial attacks on SAR image recognition models and provide an example of adversarial robustness evaluation. Finally, several open questions are discussed regarding recent research progress from our workgroup. This paper can be further used as a reference to develop more robust deep neural network-based image recognition models in dynamic adversarial scenarios.

A Robust Breast ultrasound segmentation method under noisy annotations

Background and Objective: A large-scale training data and accurate annotations are fundamental for current segmentation networks. However, the characteristic artifacts of ultrasound images always make the annotation task complicated, such as attenuation, speckle, shadows and signal dropout. Further complications arise as the contrast between the region of interest and background is often low. Without double-check from professionals, it is hard to guarantee that there is no noisy annotation in segmentation datasets. However, among the deep learning methods applied to ultrasound segmentation so far, no one can solve this problem.Method: Given a dataset with poorly labeled masks, including a certain amount of noises, we propose an end-to-end noisy annotation tolerance network (NAT-Net). NAT-Net can detect noise by the proposed noise index (NI) and dynamically correct noisy annotations in the training stage. Simultaneously, noise index is used to correct the noise along with the output of the learning model. This method does not need any auxiliary clean datasets or prior knowledge of noise distributions, so it is more general, robust and easier to apply than the existing methods. Results: NAT-Net outperforms previous state-of-the-art methods on synthesized data with different noise ratio. For real-world dataset with more complex noise types, the IoU of NAT-Net is higher than that of state-of-art approaches by nearly 6%. Experimental results show that our method can also achieve good results compared with the existing methods for clean dataset. Conclusion: The NAT-Net reduces manual interaction of data annotation, reduces dependence on medical personnel. After tumor segmentation, disease diagnosis efficiency is improved, which provides an auxiliary strategies for subsequent medical diagnosis systems based on ultrasound.

Decentralised Learning from Independent Multi-Domain Labels for Person Re-Identification

Deep learning has been successful for many computer vision tasks due to the availability of shared and centralised large-scale training data. However, increasing awareness of privacy concerns poses new challenges to deep learning, especially for human subject related recognition such as person re-identification (Re-ID). In this work, we solve the Re-ID problem by decentralised learning from non-shared private training data distributed at multiple user sites of independent multi-domain label spaces. We propose a novel paradigm called Federated Person Re-Identification (FedReID) to construct a generalisable global model (a central server) by simultaneously learning with multiple privacy-preserved local models (local clients). Specifically, each local client receives global model updates from the server and trains a local model using its local data independent from all the other clients. Then, the central server aggregates transferrable local model updates to construct a generalisable global feature embedding model without accessing local data so to preserve local privacy. This client-server collaborative learning process is iteratively performed under privacy control, enabling FedReID to realise decentralised learning without sharing distributed data nor collecting any centralised data. Extensive experiments on ten Re-ID benchmarks show that FedReID achieves compelling generalisation performance beyond any locally trained models without using shared training data, whilst inherently protects the privacy of each local client. This is uniquely advantageous over contemporary Re-ID methods.

An joint end-to-end framework for learning with noisy labels

Deep neural networks (DNNs) have achieved excellent performance in image classification research, part of which is due to the large-scale training data with accurate annotations. However, it is expensive and time-consuming to collect such clean data. In contrast, when collecting a dataset by crawling from websites, noisy labels are ubiquitous, which makes it easy for deep neural networks (DNNs) to overfit noisy labels and cause performance degradation. Most recent efforts have been focused on defending noisy labels by roughly ignoring some samples with high losses, which are treated as noise or reweight the training data in the loss function. Both strategies inevitably have priori conditions, such as a clean validation set or a ground-truth noise transition matrix, which are impractical in real-world datasets. In this paper, we propose a novel end-to-end framework for noise correction, called End-to-end Correction with Mixup and Balance terms (ECMB). ECMB can completely correct noisy labels to true labels and keep the number of each class more balanced. This framework uses a backbone network that is pre-trained by using an improved Mixup entropy instead of the traditional cross entropy, and does not need any extra conditions. In addition, we introduce a new balance term that can update noisy labels more accurately. Compared with other state-of-the-art methods, the experimental results on publicly available CIFAR-10, CIFAR-100 and Clothing1M datasets demonstrate that our method has superior performance.

Knowledge Transfer with Weighted Adversarial Network for Cold-Start Store Site Recommendation

Store site recommendation aims to predict the value of the store at candidate locations and then recommend the optimal location to the company for placing a new brick-and-mortar store. Most existing studies focus on learning machine learning or deep learning models based on large-scale training data of existing chain stores in the same city. However, the expansion of chain enterprises in new cities suffers from data scarcity issues, and these models do not work in the new city where no chain store has been placed (i.e., cold-start problem). In this article, we propose a unified approach for cold-start store site recommendation, Weighted Adversarial Network with Transferability weighting scheme (WANT), to transfer knowledge learned from a data-rich source city to a target city with no labeled data. In particular, to promote positive transfer, we develop a discriminator to diminish distribution discrepancy between source city and target city with different data distributions, which plays the minimax game with the feature extractor to learn transferable representations across cities by adversarial learning. In addition, to further reduce the risk of negative transfer, we design a transferability weighting scheme to quantify the transferability of examples in source city and reweight the contribution of relevant source examples to transfer useful knowledge. We validate WANT using a real-world dataset, and experimental results demonstrate the effectiveness of our proposed model over several state-of-the-art baseline models.

Boosting Semi-Supervised Face Recognition With Noise Robustness

Although deep face recognition benefits significantly from large-scale training data, a current bottleneck is the labelling cost. A feasible solution to this problem is semi-supervised learning, exploiting a small portion of labelled data and large amounts of unlabelled data. The major challenge, however, is the accumulated label errors through auto-labelling, compromising the training. In this paper, we present an effective solution to semi-supervised face recognition that is robust to the label noise aroused by the auto-labelling. Specifically, we introduce a multi-agent method, named GroupNet (GN), to endow our solution with the ability to identify the wrongly-labelled samples and preserve the clean samples. We show that GN alone achieves the leading accuracy in traditional supervised face recognition even when the noisy labels take over 50% of the training data. Further, we develop a semi-supervised face recognition solution, named Noise Robust Learning-Labelling (NRoLL), which is based on the robust training ability empowered by GN. It starts with a small amount of labelled data and consequently conducts high-confidence labelling on a large amount of unlabelled data to boost further training. The more data is labelled by NRoLL, the higher confidence is with the label in the dataset. To evaluate the competitiveness of our method, we run NRoLL with a rough condition that only one-fifth of the labelled MSCeleb is available and the rest is used as unlabelled data. On a wide range of benchmarks, our method compares favorably against the state-of-the-art methods.

Secure Cloud-Aided Object Recognition on Hyperspectral Remote Sensing Images

Object recognition of hyperspectral remote sensing images based on machine learning is widely applied in many industries. However, the efficiency of the training and recognizing process of object recognition on hyperspectral remote sensing images is a critical issue since it involves complex matrix operations and large scale training data sets, especially for resource-constrained devices. One solution is to outsource the heavy workload of object recognition on hyperspectral remote sensing images to a cloud server. Nonetheless, it may bring some security problems when the cloud server is untrustworthy. Therefore, how to enable resource-constrained devices to securely and efficiently accomplish the training and recognizing process of object recognition on hyperspectral remote sensing images is of significant importance. In this article, we propose a secure and efficient scheme to outsource the object recognition on hyperspectral remote sensing images to the untrustworthy cloud server. The proposed scheme can protect the privacy of the computation input and output. Also, we develop an effective verification approach in our scheme that can detect the misbehavior of cloud server with the optimal probability 1. The theoretical analysis and experimental results indicate that our proposed scheme is secure and efficient.

An Incremental Learning Based Convolutional Neural Network Model for Large-Scale and Short-Term Traffic Flow

Traffic flow prediction is very important for smooth road conditions in cities and convenient travel for residents. With the explosive growth of traffic flow data size, traditional machine learning algorithms cannot fit large-scale training data effectively and the deep learning algorithms do not work well because of the huge training and update costs, and the prediction accuracy may need to be further improved when an emergency affecting traffic occurs. In this study, an incremental learning based convolutional neural network model, TF-net, is proposed to achieve the efficient and accurate prediction of large-scale and short-term traffic flow. The key idea is to introduce the uncertainty features into the model without increasing the training cost to improve the prediction accuracy. Meanwhile, based on the idea of combining incremental learning with active learning, a certain percentage of typical samples in historical traffic flow data are sampled to fine-tune the prediction model, so as to further improve the prediction accuracy for special situations and ensure the real-time requirement. The experimental results show that the proposed traffic flow prediction model has better performance than the existing methods.

Meta-Learning Based Dynamic Computation Task Offloading for Mobile Edge Computing Networks

Deep learning-based algorithms provide a promising solution to efficiently generate offloading decisions in mobile edge computing (MEC) networks. However, considering dynamic MEC devices or offloading tasks, most of them require large-scale training data and long training time to retrain the deep neural networks (DNNs). In this letter, we propose a MEta-Learning-based computation Offloading (MELO) algorithm for dynamic computation tasks in MEC networks. Specifically, it learns from historical MEC task scenarios and adapts to a new MEC task scenario with a few training samples. Numerical results show that the proposed algorithm can adapt to a new MEC task scenario and achieve 99% accuracy via 1-step fine-tuning using only 10 training samples.

Grape Leaf Spot Identification Under Limited Samples by Fine Grained-GAN

In practice, early detection of disease is of high importance to practical value, corresponding measures can be taken at the early stage of plant disease. However, in the early stage of disease or when a rare disease occurs, there are limited training samples in practice, which makes machine learning especially deep learning models hardly work well while the stronger representative ability needs large-scale training data. To solve this problem, a fine grained-GAN based grape leaf spot identification method was proposed for local spot area image data augmentation to the generated local spot area images which were added and fed them into deep learning models for training to further strengthen the generalization ability of the classification models, which can effectively improve the accuracy and robustness of the prediction. Including 500 early-stage grape leaf spot images every category were fed into the proposed fine grained-GAN for local spot area data augmentation, 1000 local spot area sub images every category were generated in this study. The improved faster R-CNN was integrated in fine grained-GAN as local spot area detector. After that, the segmented sub-images and generated sub-images were mixed as training data input into the deep learning models, while the original segmented sub-images were used for testing. Experimental results showed that the proposed method had achieved higher identification accuracy on five state-of-art deep learning models; especially ResNet-50 had got 96.27% accuracy, which obtained significant improvements than other data augmentation methods and verified its satisfactory performance. This is of great practical significance for rare diseases or limited training samples.

CPM: A large-scale generative Chinese Pre-trained language model

Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3, with 175 billion parameters and 570 GB training data, drew a lot of attention due to the capacity of few-shot (even zero-shot) learning. However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus of GPT-3 is primarily English, and the parameters are not publicly available. In this technical report, we release the Chinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. To the best of our knowledge, CPM, with 2.6 billion parameters and 100 GB Chinese training data, is the largest Chinese pre-trained language model, which could facilitate several downstream Chinese NLP tasks, such as conversation, essay generation, cloze test, and language understanding. Extensive experiments demonstrate that CPM achieves strong performance on many NLP tasks in the settings of few-shot (even zero-shot) learning. The code and parameters are available at https://github.com/TsinghuaAI/CPM.

Combining Supervised Learning and Digital Twin for Autonomous Path-planning

Abstract Over the last decade, the evolution of autonomous automobiles based on artificial intelligence has increased rapidly with significant success. Naturally, this has caught the interest of the maritime industry and the development of autonomous vessels. However, unlike the highway, the ocean is considered a complex environment carrying unpredictable environmental forces, such as current, waves and wind-condition. For autonomous path-following and path-planning, particularly within the machine learning-field, Deep Reinforcement Learning (DRL) have generally been the favored approach. This follows from the fact that resulting models have demonstrated staggering performance. However, for practical implementations, Deep learning-based models are generally considered black box-solutions, and hence often introduce uncertainties in the operating domain. Therefore, in this paper an autonomous path-planner based on Supervised learning is proposed. Different Supervised learning models were investigated, and Gradient Boosting Regressor was found to be the most adequate model based on hyperparameter-tuning. The model was developed on constraints proposed by the class society DNV GL combined with International Regulations for Preventing Collision at Sea (COLREGs) rule 14 for collision-avoidance. Following this, the model was trained to design a suitable path based on parametrization of a cubic Bezier curve. To follow the parametrized path, a maneuvering-controller derived from the Maneuvering problem presented in Skjetne (2005) was applied. However, a drawback of Supervised learning is the necessity for large-scale training data. Hence, a digital twin of the own vessel was developed and utilized to generate sufficient training data. To demonstrate the performance of the autonomous path-planner, a number of simulation scenarios were introduced.

Dose-Conditioned Synthesis of Radiotherapy Dose With Auxiliary Classifier Generative Adversarial Network

In recent years, there are more and more researches on automatic radiotherapy planning based on artificial intelligence technology. Most of the work focuses on the dose prediction of radiotherapy planning, that is, the generation of radiation dose distribution image. Because of the small sample nature of radiotherapy planning data, it is difficult to obtain large-scale training data sets. In this paper, we propose a model of Dose-Conditioned Synthesis of Radiotherapy dose by using Auxiliary Classifier Generative Adversarial Network(ACGAN), and a method of customize and synthesis dose distribution images of specific tumor types and beam types is considered. This method can customize and generate dose distribution images of tumor types and beam types. The dose distribution images generated by our model are evaluated by MS-SSIM and PSNR, the results show that the image quality of dose distribution generated by ACGAN model was excellent, which was very close to the real data and shows high diversity, it can be used for data enhancement work of training data sets of dose prediction methods.

A Sentence-Level Joint Relation Classification Model Based on Reinforcement Learning.

Relation classification is an important semantic processing task in the field of natural language processing (NLP). Data sources generally adopt remote monitoring strategies to automatically generate large-scale training data, which inevitably causes label noise problems. At the same time, another challenge is that important information can appear at any place in the sentence. This paper presents a sentence-level joint relation classification model. The model has two modules: a reinforcement learning (RL) agent and a joint network model. In particular, we combine bidirectional long short-term memory (Bi-LSTM) and attention mechanism as a joint model to process the text features of sentences and classify the relation between two entities. At the same time, we introduce an attention mechanism to discover hidden information in sentences. The joint training of the two modules solves the noise problem in relation extraction, sentence-level information extraction, and relation classification. Experimental results demonstrate that the model can effectively deal with data noise and achieve better relation classification performance at the sentence level.

WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models

Using large-scale training data to build a pre-trained language model (PLM) with a larger volume of parameters can significantly improve downstream tasks. For example, OpenAI trained the GPT3 model with 175 billion parameters on 570 GB English training data, enabling downstream applications building with only a small number of samples. However, there is a lack of Chinese corpus to support large-scale PLMs. This paper introduces a super large-scale Chinese corpora WuDaoCorpora, containing about 3 TB training data and 1.08 trillion Chinese characters. We also release the base version of WuDaoCorpora, containing about 200 GB training data and 72 billion Chinese characters. As a baseline, we train a model transformer-XL with 3 billion parameters on the base version to test the corpora's effect. The results show that the models trained on this corpora can achieve excellent performance in Chinese. The data and model are available at https://data.wudaoai.cn and https://github.com/THUDM/Chinese-Transformer-XL, respectively.