Cross-domain Knowledge Transfer Research Articles

GTLNLP: A Mathematical Exploration of Cross-Domain Knowledge Transfer for Text Generation for Generative Transfer Learning in Natural Language Processing

In the field of Cross-Domain Knowledge Transfer generative transfer learning in Natural Language Processing (NLP) used for creating text using generative math. The paper aims to improve the usefulness of text creation models in a variety of areas by using cutting edge deep learning and neural network methods. We come up with a new system that makes it easier for information to move from one domain to another, even when the language and situation are different in each domain. Using domain adaptation techniques to match feature distributions and reduce domain gaps is what our method is based on mathematically. It is a complex version of transfer learning principles. We test our model's abilities on a wide range of tasks by doing a lot of careful experiments. We focus on how well it can share information and write text that makes sense and is relevant to the situation across different areas. This study not only adds to our theoretical understanding of cross-domain knowledge transfer, but it also gives us useful tips on how to make NLP models more flexible and useful in the real world. The results of our study could help improve the state of the art in generative transfer learning and make text generation systems that work better and more reliably in a variety of language settings.

Graph Disentangled Contrastive Learning with Personalized Transfer for Cross-Domain Recommendation

Cross-Domain Recommendation (CDR) has been proven to effectively alleviate the data sparsity problem in Recommender System (RS). Recent CDR methods often disentangle user features into domain-invariant and domain-specific features for efficient cross-domain knowledge transfer. Despite showcasing robust performance, three crucial aspects remain unexplored for existing disentangled CDR approaches: i) The significance nuances of the interaction behaviors are ignored in generating disentangled features; ii) The user features are disentangled irrelevant to the individual items to be recommended; iii) The general knowledge transfer overlooks the user's personality when interacting with diverse items. To this end, we propose a Graph Disentangled Contrastive framework for CDR (GDCCDR) with personalized transfer by meta-networks. An adaptive parameter-free filter is proposed to gauge the significance of diverse interactions, thereby facilitating more refined disentangled representations. In sight of the success of Contrastive Learning (CL) in RS, we propose two CL-based constraints for item-aware disentanglement. Proximate CL ensures the coherence of domain-invariant features between domains, while eliminatory CL strives to disentangle features within each domains using mutual information between users and items. Finally, for domain-invariant features, we adopt meta-networks to achieve personalized transfer. Experimental results on four real-world datasets demonstrate the superiority of GDCCDR over state-of-the-art methods.

Dual-Teacher Feature Distillation: A Transfer Learning Method for Insomniac PSG Staging.

Insomnia is the most common sleep disorder linked with adverse long-term medical and psychiatric outcomes. Automatic sleep staging plays a crucial role in aiding doctors to diagnose insomnia disorder. Only a few studies have been conducted to develop automatic sleep staging methods for insomniacs, and most of them have utilized transfer learning methods, which involve pre-training models on healthy individuals and then fine-tuning them on insomniacs. Unfortunately, significant differences in feature distribution between the two subject groups impede the transfer performance, highlighting the need to effectively integrate the features of healthy subjects and insomniacs. In this paper, we propose a dual-teacher cross-domain knowledge transfer method based on the feature-based knowledge distillation to improve the performance of sleep staging for insomniacs. Specifically, the insomnia teacher directly learns from insomniacs and feeds the corresponding domain-specific features into the student network, while the health domain teacher guide the student network to learn domain-generic features. During the training process, we adopt the OFD (Overhaul of Feature Distillation) method to build the health domain teacher. We conducted the experiments to validate the proposed method, using the Sleep-EDF database as the source domain and the CAP-Database as the target domain. The results demonstrate that our method surpasses advanced techniques, achieving an average sleep staging accuracy of 80.56% on the CAP-Database. Furthermore, our method exhibits promising performance on the private dataset.

Privacy-preserving Multi-source Cross-domain Recommendation Based on Knowledge Graph

The cross-domain recommender systems aim to alleviate the data sparsity problem in the target domain by transferring knowledge from the auxiliary domain. However, existing works ignore the fact that the data sparsity problem may also exist in the single auxiliary domain, and sharing user behavior data is restricted by the privacy policy. In addition, their cross-domain models lack interpretability. To address these concerns, we propose a novel multi-source cross-domain model based on knowledge graph. Specifically, to avoid the insufficiency of single auxiliary domain, we construct a knowledge graph comprehensively leveraging items from multiple auxiliary domains. To avoid the leakage of user privacy when user information is transferred to multiple domains, we construct graph for information transfer between items to effectively avoid the propagation of users’ private information between different domains. We implicitly integrate the user–item interaction by transferring the learned item embeddings. To improve the interpretability of cross-domain knowledge transfer, we propose a knowledge graph-based retrieval and fusion method to transfer knowledge derived from multiple auxiliary domains. An attention-based fusion network is designed to enhance the representation of the targeted user and items with the transferred item embedding. We perform extensive experiments on three real-world datasets, demonstrating that our model outperforms the states of the art.

Cross-domain Recommendation via Dual Adversarial Adaptation

Data scarcity is a perpetual challenge of recommendation systems, and researchers have proposed a variety of cross-domain recommendation methods to alleviate the problem of data scarcity in target domains. However, in many real-world cross-domain recommendation systems, the source domain and the target domain are sampled from different data distributions, which obstructs the cross-domain knowledge transfer. In this article, we propose to specifically align the data distributions between the source domain and the target domain to alleviate imbalanced sample distribution and thus challenge the data scarcity issue in the target domain. Technically, our proposed approach builds a dual adversarial adaptation (DAA) framework to adversarially train the target model together with a pre-trained source model. Two domain discriminators play the two-player minmax game with the target model and guide the target model to learn reliable domain-invariant features that can be transferred across domains. At the same time, the target model is calibrated to learn domain-specific information of the target domain. In addition, we formulate our approach as a plug-and-play module to boost existing recommendation systems. We apply the proposed method to address the issues of insufficient data and imbalanced sample distribution in real-world Click-through Rate/Conversion Rate predictions on two large-scale industrial datasets. We evaluate the proposed method in scenarios with and without overlapping users/items, and extensive experiments verify that the proposed method is able to significantly improve the prediction performance on the target domain. For instance, our method can boost PLE with a performance improvement of 15.4% in terms of Area Under Curve compared with single-domain PLE on our private game dataset. In addition, our method is able to surpass single-domain MMoE by 6.85% on the public datasets. Code: https://github.com/TL-UESTC/DAA .

Research on the application of the Sleep EEG Net model based on domain adaptation transfer in the detection of driving fatigue

Fatigue detection in driving faces challenges stemming from data scarcity and difficulty in data acquisition, which poses a significant challenge to traditional fatigue detection methods. To address this issue, this study introduces a Sleep EEG Net model based on domain adaptation transfer learning. This model was pre-trained using the publicly available Sleep-EDF dataset, and domain adaptation transfer training techniques were employed to train the feature extractor of the pre-trained model, enabling cross-domain knowledge transfer. As a result, the model has been successfully applied to the task of fatigue detection in driving with only a limited amount of fatigue driving data. Experimental results demonstrate that this approach achieves a recognition accuracy of 91.5% in fatigue detection tasks. Furthermore, the model exhibits strong generalization capabilities and robustness, achieving high recognition accuracy in both simulated and real driving environments, thereby validating its effectiveness in practical applications.

A novel continual reinforcement learning-based expert system for self-optimization of soft real-time systems

Virtual globes are soft real-time systems, which stream multi-resolution data sets and render world-scale landscapes in real-time. Such systems require an adaptation mechanism to fulfill user and quality of experience (QoE) requirements. Configuration and tuning of parameters to optimize system performance is challenging due to run-time uncertainties. In this work, we propose a continual reinforcement learning-based expert system for self-optimization of a soft real-time system to meet user and quality objectives. The proposed system is capable of learning continuously by utilizing the previously learned knowledge to fulfill adaptation requirements. The proposed managing system interacts with a managed system and evolves a knowledge base of adaptation rules for run-time optimization of the managed system to meet user and quality objectives. We devise a learning classifier system for the evolution, storage, and transfer of adaptation rules using a code fragments-based rich encoding scheme. We transform user goals, system states, and execution environment conditions into observable states. We validated the proposed system for the same and cross-domain knowledge transfer and reuse scenarios for different user goals and execution environments. The results of the proposed system are compared with baseline and state-of-the-art methods where the proposed system outperformed the methods in terms of adaptation accuracy.

Combining Graph Contrastive Embedding and Multi-head Cross-Attention Transfer for Cross-Domain Recommendation

Cross-domain recommendation (CDR) has become an important research direction in the field of recommender systems due to the increasing demand for personalized recommendations across different domains. However, CDR faces multiple challenges, including data sparsity, popularity bias, and long-tail problems. To address these challenges, we propose a novel framework that combines graph contrastive embedding and multi-head cross-attention transfer for cross-domain recommendation, called GCE-MCAT. Specifically, in the pre-training process, we generate more uniform user and item embeddings through contrastive learning, effectively solving the problem of inconsistent data embedding space distribution and recommendation popularity bias. Moreover, we propose a multi-head cross-attention transfer mechanism that allows the model to extract user common and specific domain features from multiple perspectives and perform cross-domain bidirectional knowledge transfer. Finally, we propose a cross-domain feature fusion mechanism that dynamically assigns weights to common user features and specific domain features. This enables the model to more effectively learn common user interests. We evaluate the proposed framework on three real-world CDR datasets and show that GCE-MCAT consistently and significantly improves recommendation performance compared to state-of-the-art methods. In particular, the proposed framework has demonstrated remarkable effectiveness in addressing long-tail distribution and enhancing recommendation novelty, providing users with more diversified recommendations and reducing popularity bias.

Multitasking evolutionary algorithm based on adaptive seed transfer for combinatorial problem

Evolutionary computing (EC) is widely used in dealing with combinatorial optimization problems (COP). Traditional EC methods can only solve a single task in a single run, while real-life scenarios often need to solve multiple COPs simultaneously. In recent years, evolutionary multitasking optimization (EMTO) has become an emerging topic in the EC community. And many methods have been designed to deal with multiple COPs concurrently through exchanging knowledge. However, many-task optimization, cross-domain knowledge transfer, and negative transfer are still significant challenges in this field. A new evolutionary multitasking algorithm based on adaptive seed transfer (MTEA-AST) is developed for multitasking COPs in this work. First, a dimension unification strategy is proposed to unify the dimensions of different tasks. And then, an adaptive task selection strategy is designed to capture the similarity between the target task and other online optimization tasks. The calculated similarity is exploited to select suitable source tasks for the target one and determine the transfer strength. Next, a task transfer strategy is established to select seeds from source tasks and correct unsuitable knowledge in seeds to suppress negative transfer. Finally, the experimental results indicate that MTEA-AST can adaptively transfer knowledge in both same-domain and cross-domain many-task environments. And the proposed method shows competitive performance compared to other state-of-the-art EMTOs in experiments consisting of four COPs.

Cross-domain knowledge transfer based parallel-cascaded multi-scale attention network for limited view reconstruction in projection magnetic particle imaging

Projection magnetic particle imaging (MPI) can significantly improve the temporal resolution of three-dimensional (3D) imaging compared to that using traditional point by point scanning. However, the dense view of projections required for tomographic reconstruction limits the scope of temporal resolution optimization. The solution to this problem in computed tomography (CT) is using limited view projections (sparse view or limited angle) for reconstruction, which can be divided into: completing the limited view sinogram and image post-processing for streaking artifacts caused by insufficient projections. Benefiting from large-scale CT datasets, both categories of deep learning-based methods have achieved tremendous progress; yet, there is a data scarcity limitation in MPI. We propose a cross-domain knowledge transfer learning strategy that can transfer the prior knowledge of the limited view learned by the model in CT to MPI, which can help reduce the network requirements for real MPI data. In addition, the size of the imaging target affects the scale of the streaking artifacts caused by insufficient projections. Therefore, we propose a parallel-cascaded multi-scale attention module that allows the network to adaptively identify streaking artifacts at different scales. The proposed method was evaluated on real phantom and in vivo mouse data, and it significantly outperformed several advanced limited view methods. The streaking artifacts caused by an insufficient number of projections can be overcome using the proposed method.

Cross-domain remote sensing image retrieval with Gabor-based CNN

ABSTRACT Domain adaptation is the ability to improve the learning efficiency of the target domain by using the prior knowledge of the source domain. When applied to new tasks in the target domain, the performance of domain adaptation trained in the source domain declines sharply. The purpose is to realize a new retrieval task in the target domain using only a small number of labelled data samples, with the aid of the prior knowledge learned in the source domain. This paper focuses on semi-supervised domain adaptation in remote sensing image retrieval. The contributions of this paper are threefold. First, we construct Gabor-based CNNs to facilitate the networks to effectively capture the texture information of images. Second, we propose a cross-domain knowledge transfer strategy based on dual Gabor neural network learning. Third, we propose an unsupervised random feature mapping method based on probability distance. A large number of experiments have been conducted on UCM, WHU-RS, RSSCN7, AID, and PatternNet datasets. The results show that this method greatly improves the retrieval accuracy on the target domain and obtains state-of-the-art retrieval accuracy. The source code is available at http://nave.vr3i.com.

Hybrid Multimodality Fusion with Cross-Domain Knowledge Transfer to Forecast Progression Trajectories in Cognitive Decline.

Magnetic resonance imaging (MRI) and positron emission tomography (PET) are increasingly used to forecast progression trajectories of cognitive decline caused by preclinical and prodromal Alzheimer's disease (AD). Many existing studies have explored the potential of these two distinct modalities with diverse machine and deep learning approaches. But successfully fusing MRI and PET can be complex due to their unique characteristics and missing modalities. To this end, we develop a hybrid multimodality fusion (HMF) framework with cross-domain knowledge transfer for joint MRI and PET representation learning, feature fusion, and cognitive decline progression forecasting. Our HMF consists of three modules: 1) a module to impute missing PET images, 2) a module to extract multimodality features from MRI and PET images, and 3) a module to fuse the extracted multimodality features. To address the issue of small sample sizes, we employ a cross-domain knowledge transfer strategy from the ADNI dataset, which includes 795 subjects, to independent small-scale AD-related cohorts, in order to leverage the rich knowledge present within the ADNI. The proposed HMF is extensively evaluated in three AD-related studies with 272 subjects across multiple disease stages, such as subjective cognitive decline and mild cognitive impairment. Experimental results demonstrate the superiority of our method over several state-of-the-art approaches in forecasting progression trajectories of AD-related cognitive decline.

A cross-domain knowledge transfer method for process discovery of urban community services with small datasets

PurposeProcess mining tools can help discover and improve the business processes of urban community services from historical service event records. However, for the community service domains with small datasets, the effects of process mining are generally limited due to process incompleteness and data noise. In this paper, a cross-domain knowledge transfer method is proposed to help service process discovery with small datasets by making use of rich knowledge in similar domains with large datasets.Design/methodology/approachFirst, ontology modeling is used to reduce the effects of cross-domain semantic ambiguity on knowledge transfer. Second, association rules (of the activities in the service processes) are extracted with Bayesian network. Third, applicable association rules are retrieved using an applicability assignment function. Further, the retrieved association rules in domains with large datasets are mapped to those with a small dataset using a linear programming method, with a heuristic miner being adopted to generate the process model.FindingsThe proposed method is verified based on the empirical data of 10 service domains from Beidaihe, China. Results show that process discovery performance of all 10 domains were improved with the overall robustness score, precision, recall and F1 score increased by 13%, 13%, 17% and 15%, respectively. For the domains with only small datasets, the cross-domain knowledge transfer method outperforms popular state-of-the art methods.Originality/valueThe limitations of sample sizes are greatly reduced. This scheme can be followed to establish business process management systems of community services with reasonable performance and limited sample sizes.

Offensive language detection in Tamil YouTube comments by adapters and cross-domain knowledge transfer

Over the past few years, researchers have been focusing on the identification of offensive language on social networks. In places where English is not the primary language, social media users tend to post/comment using a code-mixed form of text. This poses various hitches in identifying offensive texts, and when combined with the limited resources available for languages such as Tamil, the task becomes considerably more challenging. This study undertakes multiple tests in order to detect potentially offensive texts in YouTube comments, made available through the HASOC-Offensive Language Identification track in Dravidian Code-Mix FIRE 2021.11https://competitions.codalab.org/competitions/31146. To detect the offensive texts, models based on traditional machine learning techniques, namely Bernoulli Naïve Bayes, Support Vector Machine, Logistic Regression, and K-Nearest Neighbor, were created. In addition, pre-trained multilingual transformer-based natural language processing models such as mBERT, MuRIL (Base and Large), and XLM-RoBERTa (Base and Large) were also attempted. These models were used as fine-tuner and adapter transformers. In essence, adapters and fine-tuners accomplish the same goal, but adapters function by adding layers to the main pre-trained model and freezing their weights. This study shows that transformer-based models outperform machine learning approaches. Furthermore, in low-resource languages such as Tamil, adapter-based techniques surpass fine-tuned models in terms of both time and efficiency.Of all the adapter-based approaches, XLM-RoBERTa (Large) was found to have the highest accuracy of 88.5%. The study also demonstrates that, compared to fine-tuning the models, the adapter models require training of a fewer parameters. In addition, the tests revealed that the proposed models performed notably well against a cross-domain data set.

Cross-Domain Knowledge Transfer Using High Dynamic Range Imaging in Synthetic Datasets

Herein, we propose a framework for the generation of photorealistic synthetic datasets using High Dynamic Range Imaging (HDRI) that include all kinds of information computer vision algorithms need. Furthermore, by utilizing the proposed framework, we demonstrate cross-domain knowledge transfer in a semantic segmentation scenario. We found that deep neural networks trained with our synthetic images or with a mix of real and synthetic perform equal to or in cases better than those trained solely on real images. To our knowledge, this is the first work that uses HDRI to successfully transfer knowledge from the synthetic domain to the real world.

A Collaborative Alignment Framework of Transferable Knowledge Extraction for Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) aims to utilize knowledge from a label-rich source domain to understand a similar yet distinct unlabeled target domain. Notably, global distribution statistics across domains and local semantic characteristics across samples, are two essential factors of data analysis that should be fully explored. Most existing UDA approaches either harness only one of them or fail to closely associate them for efficient adaptation. In this work, we propose a unified framework, called Collaborative Alignment Framework (CAF), which simultaneously reduces the global domain discrepancy and preserves the local semantic consistency for cross-domain knowledge transfer in a collaborative manner. Specifically, for domain-oriented alignment, we utilize adversarial training or minimize the Wasserstein distance between the two distributions to learn domain-level invariant representations. For semantic-oriented matching, we capture the semantic discrepancy between the predictions of two diverse task-specific classifiers and enhance the features of target data to be near the support of the source data class-wisely, which promotes semantic consistency across domains effectively. These two adaptation processes can be deeply intertwined in CAF via collaborative training, thus CAF can learn domain-invariant and semantic-consistent feature representations. Extensive experiments on four popular benchmarks, including DomainNet, VisDA-2017, Office-31, and ImageCLEF, demonstrate the proposed methods significantly outperform the existing methods, especially on the large-scale dataset. The code is available at <uri>https://github.com/BIT-DA/CAF</uri>.

Weakly-supervised action localization via embedding-modeling iterative optimization

Action recognition and localization in untrimmed videos in weakly supervised scenario is a challenging problem of great application prospects. Limited by the information available in video-level labels, it is a promising attempt to fully leverage the instructive knowledge learned on trimmed videos to facilitate analysis of untrimmed videos, considering that there are abundant trimmed videos which are publicly available and well segmented with semantic descriptions. In order to enforce effective trimmed-untrimmed augmentation, this paper presents a novel framework of embedding-modeling iterative optimization network, referred to as IONet. In the proposed method, action classification modeling and shared subspace embedding are learned jointly in an iterative way, so that robust cross-domain knowledge transfer is achieved. With a carefully designed two-stage self-attentive representation learning workflow for untrimmed videos, irrelevant backgrounds are eliminated and fine-grained temporal relevance can be robustly explored. Extensive experiments are conducted on two benchmark datasets, i.e., THUMOS14 and ActivityNet1.3, and experimental results clearly corroborate the efficacy of our method. Source code is available on GitHub.22https://github.com/HCShi/IONet.

Domain-Adaptive Fall Detection Using Deep Adversarial Training

Fall detection (FD) systems are important assistive technologies for healthcare that can detect emergency fall events and alert caregivers. However, it is not easy to obtain large-scale annotated fall events with various specifications of sensors or sensor positions during the implementation of accurate FD systems. Moreover, the knowledge obtained through machine learning has been restricted to tasks in the same domain. The mismatch between different domains might hinder the performance of FD systems. Cross-domain knowledge transfer is very beneficial for machine-learning based FD systems to train a reliable FD model with well-labeled data in new environments. In this study, we propose domain-adaptive fall detection (DAFD) using deep adversarial training (DAT) to tackle cross-domain problems, such as cross-position and cross-configuration. The proposed DAFD can transfer knowledge from the source domain to the target domain by minimizing the domain discrepancy to avoid mismatch problems. The experimental results show that the average F1-score improvement when using DAFD ranges from 1.5% to 7% in the cross-position scenario, and from 3.5% to 12% in the cross-configuration scenario, compared to using the conventional FD model without domain adaptation training. The results demonstrate that the proposed DAFD successfully helps to deal with cross-domain problems and to achieve better detection performance.

A novel lifelong learning model based on cross domain knowledge extraction and transfer to classify underwater images

Artificial intelligence based autonomous systems interacting with dynamic environment are required to continuously learn, accumulate and improve the learned knowledge. Currently, most artificial intelligence based systems lack this ability and work in isolated learning paradigm. Human beings follow the continuous learning process by retaining and accumulating the learnt knowledge, and by using the learnt knowledge to solve the problem at hand. In this paper, we present a lifelong learning model, to solve challenging problem of real world underwater image classification. The proposed model is capable to learn from simple problems, accumulates the learnt knowledge by continual learning and uses the learnt knowledge to solve future complex problems of the same or related domain, in a similar way as humans do. In the proposed model, firstly, a deep classification convolutional autoencoder is presented to extract spatially localized features from images by utilizing convolution filters, then a code fragment based learning classifier system, with rich knowledge encoding scheme, is proposed for knowledge representation and transfer. In order to validate the model, experiments are conducted on two underwater images datasets and one in-air images dataset. Experiments results demonstrate that the proposed method outperforms base line method and state-of-the-art convolution neural network (CNN) methods.

Deep Learning Driven Venue Recommender for Event-Based Social Networks

Event-based online social platforms, such as Meetup and Plancast, have experienced increased popularity and rapid growth in recent years. In EBSN setup, selecting suitable venues for hosting events, which can attract a great turnout, is a key challenge. In this paper, we present a deep learning based venue recommendation system $DeepVenue$ D e e p V e n u e which provides context driven venue recommendations for the Meetup event-hosts to host their events. The crux of the proposed model relies on the notion of similarity between multiple Meetup entities such as events, venues, groups, etc. We develop deep learning techniques to compute a compact descriptor for each entity, such that two entities (say, venues) can be compared numerically. Notably, to mitigate the scarcity of venue related information in Meetup, we leverage on the cross domain knowledge transfer from popular LBSN service Yelp to extract rich venue related content. For hosting an event, the proposed $DeepVenue$ D e e p V e n u e model computes a success score for each candidate venue and ranks those venues according to the scores and finally recommend the top k venues. Our rigorous evaluation on the Meetup data collected for the city of Chicago shows that $DeepVenue$ D e e p V e n u e significantly outperforms the baselines algorithms. Precisely, for 84 percent of events, the correct hosting venue appears in the top 5 of the $DeepVenue$ D e e p V e n u e recommended list.