Proposed Transfer Learning Method Research Articles

Privacy-Preserving Federated Transfer Learning for Driver Drowsiness Detection

The drowsiness affects the driver’s sensory, cognitive, and psychomotor abilities, which are necessary for safe driving. Drowsiness detection is a critical technique to avoid traffic accidents. The federated learning (FL) can solve the problem of insufficient driver facial data by utilizing different industrial entities’ data. However, in the FL system, the privacy information of the drivers might be leaked. In addition, reducing the communication costs and maintaining the model performance is also a challenge in industrial scenarios. In this work, we propose a federated transfer learning method with privacy-preserving protocol for driver drowsiness detection, named PFTL-DDD. We use fine-tuning transfer learning on the initial model of the drowsiness detection FL system. Furthermore, a CKKS-based privacy-preserving protocol is applied to preserve the drivers’ privacy data by encrypting the exchanged parameters. The experimental results show that the PFTL-DDD method is superior in terms of accuracy and efficiency as compared to the conventional federated learning on the NTHU-DDD dataset and YAWDD dataset. The theoretical analysis demonstrates that the proposed transfer learning method can reduce the communication cost of the system, and the CKKS-based security protocol can protect the personal privacy.

Discussions of Different Deep Transfer Learning Models for Emotion Recognitions

In recent years, facial emotion recognition (FER) has been a popular topic in affective computing. However, FER still faces many challenges in automatic recognition for several reasons, including quality control of sample data, extraction of effective features, creation of models, and multi-feature fusion, which have not been thoroughly researched and therefore are still hot topics in computer visualization. In view of the mature development of deep learning, deep learning methods are increasingly being used in FER. However, because deep learning requires a large amount of data to achieve effective training, many studies have employed transfer learning to compensate for this drawback. Nevertheless, there has been no universal approach for transfer learning in FER. Accordingly, this study used the five classic models in FER (i.e., ResNet-50, Xception, EfficientNet-B0, Inception, and DenseNet121) to conduct a series of experiments: data preprocessing, training type, and the applicability of multi-stage pretraining. According to the results, class wight was the optimal technique for data balance. In addition, the freeze + fine-tuning training type can produce higher accuracy, regardless of the size of the dataset. Multi-stage training was also effective. Compared with the model accuracy in previous studies, the accuracy achieved in this study using the proposed transfer learning method was superior for both large and small datasets. Specifically, on AffectNet, the accuracy for the ResNet-50, Xception, EfficientNet-B0, Inception, and DenseNet-121 models increased by 8.37%, 10.45%, 10.45%, 8.55%, and 5.47%, respectively. On FER2013, the accuracy for these models increased by 5.72%, 2%, 10.45%, 5%, and 9%, respectively. These results proved the validity and advantages of the experiments in this study.

Syntax-based transfer learning for the task of biomedical relation extraction

BackgroundTransfer learning aims at enhancing machine learning performance on a problem by reusing labeled data originally designed for a related, but distinct problem. In particular, domain adaptation consists for a specific task, in reusing training data developedfor the same task but a distinct domain. This is particularly relevant to the applications of deep learning in Natural Language Processing, because they usually require large annotated corpora that may not exist for the targeted domain, but exist for side domains.ResultsIn this paper, we experiment with transfer learning for the task of relation extraction from biomedical texts, using the TreeLSTM model. We empirically show the impact of TreeLSTM alone and with domain adaptation by obtaining better performances than the state of the art on two biomedical relation extraction tasks and equal performances for two others, for which little annotated data are available. Furthermore, we propose an analysis of the role that syntactic features may play in transfer learning for relation extraction.ConclusionGiven the difficulty to manually annotate corpora in the biomedical domain, the proposed transfer learning method offers a promising alternative to achieve good relation extraction performances for domains associated with scarce resources. Also, our analysis illustrates the importance that syntax plays in transfer learning, underlying the importance in this domain to privilege approaches that embed syntactic features.

Identifying Patient-Ventilator Asynchrony on a Small Dataset Using Image-Based Transfer Learning.

Mechanical ventilation is an essential life-support treatment for patients who cannot breathe independently. Patient–ventilator asynchrony (PVA) occurs when ventilatory support does not match the needs of the patient and is associated with a series of adverse clinical outcomes. Deep learning methods have shown a strong discriminative ability for PVA detection, but they require a large number of annotated data for model training, which hampers their application to this task. We developed a transfer learning architecture based on pretrained convolutional neural networks (CNN) and used it for PVA recognition based on small datasets. The one-dimensional signal was converted to a two-dimensional image, and features were extracted by the CNN using pretrained weights for classification. A partial dropping cross-validation technique was developed to evaluate model performance on small datasets. When using large datasets, the performance of the proposed method was similar to that of non-transfer learning methods. However, when the amount of data was reduced to 1%, the accuracy of transfer learning was approximately 90%, whereas the accuracy of the non-transfer learning was less than 80%. The findings suggest that the proposed transfer learning method can obtain satisfactory accuracies for PVA detection when using small datasets. Such a method can promote the application of deep learning to detect more types of PVA under various ventilation modes.

Abstract 863: Transfer learning for gene expression prediction with deep neural networks

Abstract Conventional machine learning approaches try to solve problems in an isolated manner whereas humans utilize previous experiences between similar tasks. Transfer learning (TL) imitates this efficient approach and transfer knowledge that was learned in a source/domain task and employ that knowledge to improve learning in a related target task. We propose a deep convolutional neural networks (CNNs) structure to predict gene expression via epigenomic signals around the TSS. This structure allows to transfer knowledge that we learn in a cancer type to another one. We use H3K4me1, H3K4me3, H3K27Ac, H3K4me3, H3K27me3 histone modification marks for breast, lung, colon, glioma, ovarian, melanoma, and pancreas cancers as input and predict gene expression for each sample. Our model with transfer learning has improved prediction rates for various cancer types compared to the other established methods like Support Vector Machine (SVM), Random Forest (RF), and eXtreme Gradient Boosting (XGBoost). We also compared performances for different percentages of data with previously proposed DeepChrome algorithm. We checked normal datasets contribution for cancer samples as well. For this purpose, we train our model with normal samples from Roadmap Epigenome Project (REMC) repository first and passed the learned parameters to cancer datasets. It seems normal samples do not have common epigenomic patterns with cancer ones as we do not see any prediction of gene expression improvement. To our knowledge, there is no proposed transfer learning method in epigenetic data analysis. The proposed transfer learning mechanism will lead to avoid generating a significant amount of epigenomic data for different cancer types and will accelerate epigenomic studies in different cancer types. Citation Format: Emre Arslan, Kunal Rai. Transfer learning for gene expression prediction with deep neural networks [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 863.

Predictive model for water absorption in sublayers using a Joint Distribution Adaption based XGBoost transfer learning method

Calculating the water absorption in sublayers based on the on-site monitored water injection profile is the most accurate way to reflect the actual amount of water absorbed by subalyers. However, access to this profile data is scarce and limited due to its high cost of testing. As a result, some well blocks have tested some discontinuous profiles, but other well blocks do not have any profiles. Traditional machine learning can be applied to construct an intelligent surrogate model by training with historical injection profile. This well-trained model can be only used to make a good prediction for well blocks which contribute profile data, not applicable to well blocks without injection profile. In this study, a Joint Distribution Adaption based Extreme Gradient boosting transfer learning approach is presented to predict water absorption of sublayers in water injection well which do not have any injection profile. A handful of observations are obtained from source well block which has tested sufficient injection profiles. Joint Distribution Adaption is first conducted to transfer knowledge from source well block to the target well block which have no injection profile. The transferred dataset with new feature representation is used to constitute the training dataset for target well block. This transferred training dataset then can be feed into the Extreme Gradient Boosting model to construct a water absorption predictive model. The well-trained model can be applied to predicting water absorption of sublayers in injectors which do not have any water injection profile. The proposed approach is demonstrated by applying to two well blocks from SL oilfield, China. Demonstrated results imply that the proposed transfer learning method can be used to dividing water absorption of subalyers in injectors which have no water injection profile data.

Multiple-level biomedical event trigger recognition with transfer learning

BackgroundAutomatic extraction of biomedical events from literature is an important task in the understanding biological systems, allowing for faster update of the latest discoveries automatically. Detecting trigger words which indicate events is a critical step in the process of event extraction, because following steps depend on the recognized triggers. The task in this study is to identify event triggers from the literature across multiple levels of biological organization. In order to achieve high performances, the machine learning based approaches, such as neural networks, must be trained on a dataset with plentiful annotations. However, annotations might be difficult to obtain on the multiple levels, and annotated resources have so far mainly focused on the relations and processes at the molecular level. In this work, we aim to apply transfer learning for multiple-level trigger recognition, in which a source dataset with sufficient annotations on the molecular level is utilized to improve performance on a target domain with insufficient annotations and more trigger types.ResultsWe propose a generalized cross-domain neural network transfer learning architecture and approach, which can share as much knowledge as possible between the source and target domains, especially when their label sets overlap. In the experiments, MLEE corpus is used to train and test the proposed model to recognize the multiple-level triggers as a target dataset. Two different corpora having the varying degrees of overlapping labels with MLEE from the BioNLP’09 and BioNLP’11 Shared Tasks are used as source datasets, respectively. Regardless of the degree of overlap, our proposed approach achieves recognition improvement. Moreover, its performance exceeds previously reported results of other leading systems on the same MLEE corpus.ConclusionsThe proposed transfer learning method can further improve the performance compared with the traditional method, when the labels of the source and target datasets overlap. The most essential reason is that our approach has changed the way parameters are shared. The vertical sharing replaces the horizontal sharing, which brings more sharable parameters. Hence, these more shared parameters between networks improve the performance and generalization of the model on the target domain effectively.

Deep convolutional recurrent neural network with transfer learning for hyperspectral image classification

The deep learning methods have recently been successfully explored for hyperspectral image classification. However, it may not perform well when training samples are scarce. A deep transfer learning method is proposed to improve the hyperspectral image classification performance in the situation of limited training samples. First, a Siamese network composed of two convolutional neural networks is designed for local image descriptors extraction. Subsequently, the pretrained Siamese network model is reused to transfer knowledge to the hyperspectral image classification tasks by feeding deep features extracted from each band into a recurrent neural network. Indeed, a deep convolutional recurrent neural network is constructed for hyperspectral image classification by this way. Finally, the entire network is tuned by a small number of labeled samples. The important characteristic of the designed model is that the deep convolutional recurrent neural network provides a way of utilizing the spatial–spectral features without dimension reduction. Furthermore, the transfer learning method provides an opportunity to train such deep model with limited labeled samples. Experiments on three widely used hyperspectral datasets demonstrate that the proposed transfer learning method can improve the classification performance and competitive classification results can be achieved when compared with state-of-the-art methods.

Domain class consistency based transfer learning for image classification across domains

Distribution mismatch between the modeling data and the query data is a known domain adaptation issue in machine learning. To this end, in this paper, we propose a l2,1-norm based discriminative robust kernel transfer learning (DKTL) method for high-level recognition tasks. The key idea is to realize robust domain transfer by simultaneously integrating domain-class-consistency (DCC) metric based discriminative subspace learning, kernel learning in reproduced kernel Hilbert space, and representation learning between source and target domain. The DCC metric includes two properties: domain-consistency used to measure the between-domain distribution discrepancy and class-consistency used to measure the within-domain class separability. The essential objective of the proposed transfer learning method is to maximize the DCC metric, which is equivalently to minimize the domain-class-inconsistency (DCIC), such that domain distribution mismatch and class inseparability are well formulated and unified simultaneously. The merits of the proposed method include (1) the robust sparse coding selects a few valuable source data with noises (outliers) removed during knowledge transfer, and (2) the proposed DCC metric can pursue more discriminative subspaces of different domains. As a result, the maximum class-separability is also well guaranteed. Extensive experiments on a number of visual datasets demonstrate the superiority of the proposed method over other state-of-the-art domain adaptation and transfer learning methods.

Leveraging Auxiliary Data for Learning to Rank

In learning to rank, both the quality and quantity of the training data have significant impacts on the performance of the learned ranking functions. However, in many applications, there are usually not sufficient labeled training data for the construction of an accurate ranking model. It is therefore desirable to leverage existing training data from other tasks when learning the ranking function for a particular task, an important problem which we tackle in this article utilizing a boosting framework with transfer learning . In particular, we propose to adaptively learn transferable representations called super-features from the training data of both the target task and the auxiliary task. Those super-features and the coefficients for combining them are learned in an iterative stage-wise fashion. Unlike previous transfer learning methods, the super-features can be adaptively learned by weak learners from the data. Therefore, the proposed framework is sufficiently flexible to deal with complicated common structures among different learning tasks. We evaluate the performance of the proposed transfer learning method for two datasets from the Letor collection and one dataset collected from a commercial search engine, and we also compare our methods with several existing transfer learning methods. Our results demonstrate that the proposed method can enhance the ranking functions of the target tasks utilizing the training data from the auxiliary tasks.