Transfer Learning Tasks Research Articles

Iterative joint classifier and domain adaptation for visual transfer learning.

Current available supervised classifiers cannot generalize across various domains due to distribution mismatch among them. Domain adaptation and transfer learning algorithms are proposed to tackle domain shift problem that originates from different data collection conditions. In this paper, we propose a transfer learning framework called iterative joint classifier and domain adaptation for visual transfer learning (ICDAV), which utilizes the balanced maximum mean discrepancy to better transfer knowledge across domains. Also, for learning a robust classifier against domain shift, a set of graph manifold regularizer and modified joint probability maximum mean discrepancy are simultaneously exploited to capture the domain structures and adapt the distribution of projected samples during the model learning process. Variety of experiments on several public datasets indicates that our approach achieves remarkable performance on visual domain adaptation and transfer learning tasks.

Manifold Warping: Manifold Alignment over Time

Knowledge transfer is computationally challenging, due in part to the curse of dimensionality, compounded by source and target domains expressed using different features (e.g., documents written in different languages). Recent work on manifold learning has shown that data collected in real-world settings often have high-dimensional representations, but lie on low-dimensional manifolds. Furthermore, data sets collected from similar generating processes often present different high-dimensional views, even though their underlying manifolds are similar. The ability to align these data sets and extract this common structure is critical for many transfer learning tasks. In this paper, we present a novel framework for aligning two sequentially-ordered data sets, taking advantage of a shared low-dimensional manifold representation. Our approach combines traditional manifold alignment and dynamic time warping algorithms using alternating projections. We also show that the previously-proposed canonical time warping algorithm is a special case of our approach. We provide a theoretical formulation as well as experimental results on synthetic and real-world data, comparing manifold warping to other alignment methods.

Deep transfer Wasserstein adversarial network for wafer map defect recognition

Deep neural networks (DNNs) are capable of extracting effective features from data by using deep structure and multiple non-linear processing units. However, they dependent on large datasets from the same distribution. It is difficult to collect wafer maps with various defect patterns in semiconductor manufacturing process. A new deep transfer learning model, deep transfer Wasserstein adversarial network (DTWAN) is proposed to recognize wafer map defect. An adaptive transfer learning framework based on adversarial training is proposed for DTWAN, where multi-stage optimization based on the maximum mean discrepancy (MMD), cross entropy, and adversarial loss is performed. Finally, a generative adversarial algorithm is developed to guide the model to extract general features from source and target domain. DTWAN transfers the key knowledge of the source domain data to target domain, and effectively reduces cost of data collection and improve industrial applicability of the recognition model. The effectiveness of DTWAN is verified on the transfer learning tasks from simulation wafer maps to real wafer maps. The testing results indicate that DTWAN is superior to those typical transfer learning algorithms in WMDR. This study will provide effective guidance to transfer knowledge of simulation data that are easy to collect to real manufacturing processes.

Transfer learning-based strategies for fault diagnosis in building energy systems

Data-driven fault detection and diagnosis (FDD) in building energy systems is typically limited by the quantity and quality of training data. These methods can be only used for individual systems due to the insufficient extrapolation capabilities of most machine learning algorithms. A desirable solution is to utilize transfer learning, which can transfer the knowledge learned from data-rich building energy systems to FDD tasks in data-sparse systems. However, the potential of applying transfer learning to such FDD has not been systematically investigated. Accordingly, this paper proposes a transfer-learning-based methodology for fault diagnosis in building chillers. Experiments were conducted on two water-cooled screw chillers to collect both fault and fault-free data. Transfer-learning-based fault diagnosis experiments were implemented with consideration of different transfer learning tasks, training cases, learning scenarios, and transfer learning implementation strategies. The experimental results validate the value of transfer learning for FDD in building energy systems, especially when the experimental data available for model development are limited. The maximum accuracy improvements were 12.63% and 8.18% in the two learning tasks. The research outcomes provide practical guidelines for developing transfer-learning-based solutions for FDD in building energy systems.

A Transfer Model Based on Supervised Multi-Layer Dictionary Learning for Brain Tumor MRI Image Recognition.

Artificial intelligence (AI) is an effective technology for automatic brain tumor MRI image recognition. The training of an AI model requires a large number of labeled data, but medical data needs to be labeled by professional clinicians, which makes data collection complex and expensive. Moreover, a traditional AI model requires that the training data and test data must follow the independent and identically distributed. To solve this problem, we propose a transfer model based on supervised multi-layer dictionary learning (TSMDL) for brain tumor MRI image recognition in this paper. With the help of the knowledge learned from related domains, the goal of this model is to solve the task of transfer learning where the target domain has only a small number of labeled samples. Based on the framework of multi-layer dictionary learning, the proposed model learns the common shared dictionary of source and target domains in each layer to explore the intrinsic connections and shared information between different domains. At the same time, by making full use of the label information of samples, the Laplacian regularization term is introduced to make the dictionary coding of similar samples as close as possible and the dictionary coding of different class samples as different as possible. The recognition experiments on brain MRI image datasets REMBRANDT and Figshare show that the model performs better than competitive state of-the-art methods.

Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching

Knowledge distillation extracts general knowledge from a pretrained teacher network and provides guidance to a target student network. Most studies manually tie intermediate features of the teacher and student, and transfer knowledge through predefined links. However, manual selection often constructs ineffective links that limit the improvement from the distillation. There has been an attempt to address the problem, but it is still challenging to identify effective links under practical scenarios. In this paper, we introduce an effective and efficient feature distillation method utilizing all the feature levels of the teacher without manually selecting the links. Specifically, our method utilizes an attention based meta network that learns relative similarities between features, and applies identified similarities to control distillation intensities of all possible pairs. As a result, our method determines competent links more efficiently than the previous approach and provides better performance on model compression and transfer learning tasks. Further qualitative analyses and ablative studies describe how our method contributes to better distillation.

Harnessing Psycho-lingual and Crowd-Sourced Dictionaries for Predicting Taboos in Written Emotional Disclosure in Anonymous Confession Boards.

There have been many efforts in the last decade in the health informatics community to develop systems that can automatically recognize and predict disclosures on social media. However, a majority of such efforts have focused on simple topic prediction or sentiment classification. However, taboo disclosures on social media that people are not comfortable to talk with their friends represent an abstract theme dependent on context and background. Recent research has demonstrated the efficacy of injecting concept into the learning model to improve prediction. We present a vectorization scheme that combines corpus- and lexicon-based approaches for predicting taboo topics from anonymous social media datasets. The proposed vectorization scheme exploits two context-rich lexicons LIWC and Urban Dictionary. Our methodology achieves cross-validation accuracies of up to 78.1% for the supervised learning task on Facebook Confessions dataset, and 70.5% for the transfer learning task on the YikYak dataset. For both the tasks, supervised algorithms trained with features generated by the proposed vectorizer perform better than vanilla t f - i d f representation. This work presents a novel methodology for predicting taboos from anonymous emotional disclosures on confession boards.

Improve conditional adversarial domain adaptation using self‐training

Domain adaptation for image classification is one of the most fundamental transfer learning tasks and a promising solution to overcome the annotation burden. Existing deep adversarial domain adaptation approaches imply minimax optimization algorithms, matching the global features across domains. However, the information conveyed in unlabelled target samples is not fully exploited. Here, adversarial learning and self-training are unified in an objective function, where the neural network parameters and the pseudo-labels of target samples are jointly optimized. The model's predictions on unlabelled samples are leveraged to pseudo-label target samples. The training procedure consists of two alternating steps. The first one is to train the network, while the second is to generate pseudo-labels, and the loop continues. The proposed method achieves mean accuracy improvements of 2% on Office-31, 0.7% on ImageCLEF-DA, 1.8% on Office-Home, and 1.2% on Digits than the baseline, which is superior to most state-of-the-art approaches.

Improving speech recognition models with small samples for air traffic control systems

In the domain of air traffic control (ATC) systems, efforts to train a practical automatic speech recognition (ASR) model always faces the problem of small training samples since the collection and annotation of speech samples are expert- and domain-dependent task. In this work, a novel training approach based on pretraining and transfer learning is proposed to address this issue, and an improved end-to-end deep learning model is developed to address the specific challenges of ASR in the ATC domain. An unsupervised pretraining strategy is first proposed to learn speech representations from unlabeled samples for a certain dataset. Specifically, a masking strategy is applied to improve the diversity of the sample without losing their general patterns. Subsequently, transfer learning is applied to fine-tune a pretrained or other optimized baseline models to finally achieves the supervised ASR task. By virtue of the common terminology used in the ATC domain, the transfer learning task can be regarded as a sub-domain adaption task, in which the transferred model is optimized using a joint corpus consisting of baseline samples and new transcribed samples from the target dataset. This joint corpus construction strategy enriches the size and diversity of the training samples, which is important for addressing the issue of the small transcribed corpus. In addition, speed perturbation is applied to augment the new transcribed samples to further improve the quality of the speech corpus. Three real ATC datasets are used to validate the proposed ASR model and training strategies. The experimental results demonstrate that the ASR performance is significantly improved on all three datasets, with an absolute character error rate only one-third of that achieved through the supervised training. The applicability of the proposed strategies to other ASR approaches is also validated.

DASSI: differential architecture search for splice identification from DNA sequences

BackgroundThe data explosion caused by unprecedented advancements in the field of genomics is constantly challenging the conventional methods used in the interpretation of the human genome. The demand for robust algorithms over the recent years has brought huge success in the field of Deep Learning (DL) in solving many difficult tasks in image, speech and natural language processing by automating the manual process of architecture design. This has been fueled through the development of new DL architectures. Yet genomics possesses unique challenges that requires customization and development of new DL models.MethodsWe proposed a new model, DASSI, by adapting a differential architecture search method and applying it to the Splice Site (SS) recognition task on DNA sequences to discover new high-performance convolutional architectures in an automated manner. We evaluated the discovered model against state-of-the-art tools to classify true and false SS in Homo sapiens (Human), Arabidopsis thaliana (Plant), Caenorhabditis elegans (Worm) and Drosophila melanogaster (Fly).ResultsOur experimental evaluation demonstrated that the discovered architecture outperformed baseline models and fixed architectures and showed competitive results against state-of-the-art models used in classification of splice sites. The proposed model - DASSI has a compact architecture and showed very good results on a transfer learning task. The benchmarking experiments of execution time and precision on architecture search and evaluation process showed better performance on recently available GPUs making it feasible to adopt architecture search based methods on large datasets.ConclusionsWe proposed the use of differential architecture search method (DASSI) to perform SS classification on raw DNA sequences, and discovered new neural network models with low number of tunable parameters and competitive performance compared with manually engineered architectures. We have extensively benchmarked DASSI model with other state-of-the-art models and assessed its computational efficiency. The results have shown a high potential of using automated architecture search mechanism for solving various problems in the field of genomics.

The Power of Universal Contextualized Protein Embeddings in Cross-species Protein Function Prediction.

Computationally annotating proteins with a molecular function is a difficult problem that is made even harder due to the limited amount of available labeled protein training data. Unsupervised protein embeddings partly circumvent this limitation by learning a universal protein representation from many unlabeled sequences. Such embeddings incorporate contextual information of amino acids, thereby modeling the underlying principles of protein sequences insensitive to the context of species. We used an existing pre-trained protein embedding method and subjected its molecular function prediction performance to detailed characterization, first to advance the understanding of protein language models, and second to determine areas of improvement. Then, we applied the model in a transfer learning task by training a function predictor based on the embeddings of annotated protein sequences of one training species and making predictions on the proteins of several test species with varying evolutionary distance. We show that this approach successfully generalizes knowledge about protein function from one eukaryotic species to various other species, outperforming both an alignment-based and a supervised-learning-based baseline. This implies that such a method could be effective for molecular function prediction in inadequately annotated species from understudied taxonomic kingdoms.

Deep Transfer Network With Multi-Kernel Dynamic Distribution Adaptation for Cross-Machine Fault Diagnosis

Recently, various deep learning models, which are mainly based on data-driven algorithms, have received more and more attention in the field of intelligent fault diagnosis and prognostics. However, there are two major assumptions accepted by default in the existing studies: 1) The training (source domain) and testing (target domain) data sets obey the same feature distribution; 2) Sufficient labeled data with fault information is available for model training. In real industrial scenarios, especially for different machines, these assumptions are mostly invalid, which makes it a huge challenge to build reliable diagnostic model. Motivated by transfer learning, we present a novel intelligent method named deep transfer network (DTN) with multi-kernel dynamic distribution adaptation (MDDA) to address the problem of cross-machine fault diagnosis. In the proposed approach, the DTN has wide first-layer convolutional kernel and several small convolutional layers, which is utilized to extract transferable features across different machines and suppress high frequency noise. Then, the MDDA method constructs a weighted mixed kernel function to map different transferable features to a unified feature space, and the relative importance of the marginal and conditional distributions are also evaluated dynamically. The proposed method is verified by three transfer learning tasks of bearings, in which the health states of wind turbine bearings in real scenario are identified by using diagnosis knowledge from two different bearings in laboratories. The results show that the proposed method can achieve higher diagnosis accuracy and better transfer performance even under different noisy environment conditions than many other state-of-the-art methods. The presented framework offers a promising approach for cross-machine fault diagnosis.

Intelligent Diagnostics for Ball Screw Fault Through Indirect Sensing Using Deep Domain Adaptation

Intelligent data-driven fault diagnostics for rotating machinery is well established. However, ball screws pose a unique challenge of impractical sensor locations for long-term deployment due to their complex motion trajectory and sophisticated mechanical structure. To overcome this challenge, an indirect sensing method is proposed. While techniques are available for multiple transfer learning tasks, cross-sensor domain adaptation remains unexplored. Thus, a convolutional neural network-based domain adaptation method is proposed that minimizes the maximum mean discrepancy of high-level representations between domains and exploits novel parallel data to attain class-level alignment. Proposed method achieved a mean testing accuracy of 98.25% upon validation on 33 transfer tasks designed across five accelerometer locations on a ball screw testbed depicting nine health conditions through variations in preload levels and backlash. This convenience of transferability of the diagnostic model between sensor locations can go a long way in robust and reliable condition monitoring of critical assets.

Task Transfer Learning for EEG Classification in Motor Imagery-Based BCI System.

The motor-imagery brain-computer interface system (MI-BCI) has a board prospect for development. However, long calibration time and lack of enough MI commands limit its use in practice. In order to enlarge the command set, we add the combinations of traditional MI commands as new commands into the command set. We also design an algorithm based on transfer learning so as to decrease the calibration time for collecting EEG signal and training model. We create feature extractor based on data from traditional commands and transfer patterns through the data from new commands. Through the comparison of the average accuracy between our algorithm and traditional algorithms and the visualization of spatial patterns in our algorithm, we find that the accuracy of our algorithm is much higher than traditional algorithms, especially as for the low-quality datasets. Besides, the visualization of spatial patterns is meaningful. The algorithm based on transfer learning takes the advantage of the information from source data. We enlarge the command set while shortening the calibration time, which is of significant importance to the MI-BCI application.

Comparative analysis of feature representations and machine learning methods in Android family classification

In order to overcome the lasting increase of Android malware, malware family classification, which clusters malware with the same features into one family, has been proposed as an efficient way for malware analysis. Several machine learning based approaches have been proposed for such task of malware family classification. However, due to the adoption of very different features and learning methods in different approaches, it is still an open question to explore: which approach works better for malware family classification? In this paper, we conduct extensive experiments to answer this question. For three widely known Android malware datasets, we design five multi-classification methods for predicting Android malware family. Based on the survey of Android malware analysis literatures and the observation of a large number of Android malware, we construct a set of 250 common features shared by Android malware. And we also collect 16873 documentary features from Android Developer as a comparison. Furthermore, we investigate the effects of transfer learning for adapting the model on three malware datasets on different scales. Our empirical results show that (i) the classification methods perform very closely, with neural network model having marginally better performance (1% to 3% in F1-score), (ii) features contribute most for classification, especially to enhance API features on larger datasets, and (iii) it is model transferable across different malware datasets based on various transfer learning tasks.

The Transfer Learning Task as the Means of Metalearning Tasks Solution

The Transfer Learning Task as the Means of Metalearning Tasks Solution

A Developed Siamese CNN with 3D Adaptive Spatial-Spectral Pyramid Pooling for Hyperspectral Image Classification

Since hyperspectral images (HSI) captured by different sensors often contain different number of bands, but most of the convolutional neural networks (CNN) require a fixed-size input, the generalization capability of deep CNNs to use heterogeneous input to achieve better classification performance has become a research focus. For classification tasks with limited labeled samples, the training strategy of feeding CNNs with sample-pairs instead of single sample has proven to be an efficient approach. Following this strategy, we propose a Siamese CNN with three-dimensional (3D) adaptive spatial-spectral pyramid pooling (ASSP) layer, called ASSP-SCNN, that takes as input 3D sample-pair with varying size and can easily be transferred to another HSI dataset regardless of the number of spectral bands. The 3D ASSP layer can also extract different levels of 3D information to improve the classification performance of the equipped CNN. To evaluate the classification and generalization performance of ASSP-SCNN, our experiments consist of two parts: the experiments of ASSP-SCNN without pre-training and the experiments of ASSP-SCNN-based transfer learning framework. Experimental results on three HSI datasets demonstrate that both ASSP-SCNN without pre-training and transfer learning based on ASSP-SCNN achieve higher classification accuracies than several state-of-the-art CNN-based methods. Moreover, we also compare the performance of ASSP-SCNN on different transfer learning tasks, which further verifies that ASSP-SCNN has a strong generalization capability.

The Transfer Learning Task as the Means of Metaleaning Tasks Solution

The Transfer Learning Task as the Means of Metaleaning Tasks Solution

Smart Diagnosis

To effectively identify electroencephalogram (EEG) signals in multiple-source domains, a multiple-source transfer learning-based Takagi–Sugeno–Kang (TSK) fuzzy system (FS), called MST-TSK, is proposed, which combines multiple-source transfer learning and manifold regularization (MR) learning mechanisms together into the TSK-FS framework. Specifically, the advantages of MST-TSK include the following: (1) by evaluating the significance of each source domain (SD), a flexible domain entropy-weighting index is presented; (2) using the theory of sample transfer learning, a reweighting strategy is presented to weigh the prediction of unknown samples in the target domain (TD) and the output of the source prediction functions; (3) by taking into account the MR term, the manifold structure of the TD is effectively maintained in the proposed system; and (4) by inheriting the interpretability of TSK-FS, MST-TSK displays good interpretability in identifying EEG signals that are understandable by humans (domain experts). The effectiveness of the proposed FS is demonstrated in several EEG multiple-source transfer learning tasks.

Particle swarm optimization based parameter selection technique for unsupervised discriminant analysis in transfer learning framework

The purpose of transfer learning is to utilize the knowledge gained from the existing (source) domain to enhance the performance on a distinct but related (target) domain. Existing works on transfer learning are not capable of optimizing different quality measures (components) such as minimizing the marginal distribution, minimizing the conditional distribution, maximizing the target domain variance, modeling the manifold by utilizing the common geometric properties in the source as well as the target domain at the same time. Moreover, existing transfer learning methods use conventional approaches to determine the appropriate values of their parameters, which is very hectic and time-consuming. Therefore, in order to overcome the drawbacks of existing approaches, we propose a Particle Swarm Optimization based Parameter Selection Approach for Unsupervised Discriminant Analysis (UDATL-PSO) in transfer learning framework. In UDATL-PSO, all the quality measures are considered at the same time, as well as the PSO approach has been used to select the best values of their parameters. Extensive experiments on various transfer learning tasks show that the proposed method has a significant influence on state-of-the-art methods.