Domain Knowledge Transfer Research Articles

SDKT: Similar Domain Knowledge Transfer for Multivariate Time Series Classification Tasks

ABSTRACTMultivariate time series data classification has a wide range of applications in reality. With rapid development of deep learning, convolutional networks are widely used in this task and have achieved the current best performance. However, due to high difficulty and cost of collecting this type of data, labeled data is still scarce. In some tasks, the model shows overfitting, resulting in relatively poor classification performance. In order to improve the classification performance under such situation, we proposed a novel classification method based on transfer learning—similar domain knowledge transfer (call SDKT for short). Firstly, we designed a multivariate time series domain distance calculation method (call MTSDDC for short), which helped selecting the source domain that is most similar to target domain; Secondly, we used ResNet as a pre‐trained classifier, transferred the parameters of the similar domain network to the target domain network and continue to fine‐tune the parameters. To verify our method, we conducted experiments on several public datasets. Our study has also shown that the transfer effect from the source domain to the target domain is highly negatively correlated with the distance between them, with an average Pearson coefficient of −0.78. For the transfer of most similar source domain, compared to the ResNet model without transfer and the current best model, the average accuracy improvements on the datasets we used are 4.01% and 1.46% respectively.

Advancing language models through domain knowledge integration: a comprehensive approach to training, evaluation, and optimization of social scientific neural word embeddings

This article proposes a comprehensive strategy for training, evaluating, and optimizing domain-specific word2vec-based word embeddings, using social science literature as an example. Our primary objectives are: (1) to train the embeddings utilizing a corpus of social science text, (2) to test their performance against domain-unspecific embeddings using our developed intrinsic and extrinsic evaluation strategy, and (3) to enhance their performance even further by using domain knowledge. As an integral part of this approach, we present SociRel-461, a domain-knowledge dictionary designed for the intrinsic evaluation and subsequent refinement of social science word embeddings. Using a dataset of 100,000 full-text scientific articles in sociology, we train multiple vector space models, which we then benchmark against a larger, pre-trained general language embedding model as part of our extrinsic evaluation. Furthermore, we developed a transfer learning multi-label classification task for extrinsic evaluation. Our findings reveal that domain-specific embeddings outperform their domain-unspecific counterparts in both intrinsic and extrinsic evaluations. We also investigated the retrofitting post-processing method to enhance domain-unspecific embeddings with the domain knowledge embedded in SociRel-461. While retrofitting does not enhance our domain-specific vector space models, it significantly improves the performance of the domain-unspecific embeddings. This highlights the potential of retrofitting for the transfer of domain knowledge to domain-unspecific embeddings. Our results emphasize the importance of utilizing domain-specific word embeddings for better performance in domain specific transfer learning tasks, as they outperform conventional embeddings trained on everyday language.

Three Heads Are Better than One: Improving Cross-Domain NER with Progressive Decomposed Network

Cross-domain named entity recognition (NER) tasks encourage NER models to transfer knowledge from data-rich source domains to sparsely labeled target domains. Previous works adopt the paradigms of pre-training on the source domain followed by fine-tuning on the target domain. However, these works ignore that general labeled NER source domain data can be easily retrieved in the real world, and soliciting more source domains could bring more benefits. Unfortunately, previous paradigms cannot efficiently transfer knowledge from multiple source domains. In this work, to transfer multiple source domains' knowledge, we decouple the NER task into the pipeline tasks of mention detection and entity typing, where the mention detection unifies the training object across domains, thus providing the entity typing with higher-quality entity mentions. Additionally, we request multiple general source domain models to suggest the potential named entities for sentences in the target domain explicitly, and transfer their knowledge to the target domain models through the knowledge progressive networks implicitly. Furthermore, we propose two methods to analyze in which source domain knowledge transfer occurs, thus helping us judge which source domain brings the greatest benefit. In our experiment, we develop a Chinese cross-domain NER dataset. Our model improved the F1 score by an average of 12.50% across 8 Chinese and English datasets compared to models without source domain data.

Category re-balancing based domain adaptive semantic segmentation

The semantic segmentation task faces the bottleneck of high manual annotation costs. Domain adaptive learning provides an effective solution through inter domain knowledge transfer. However, existing domain adaptive semantic segmentation tasks face the situation of category asymmetry between scenes when dealing with complex scenes. Existing methods lack handling of class imbalance in the prediction process. In domain adaptive semantic segmentation tasks, class imbalance limits the performance of inter domain transfer. Our work focuses on the impact of category distribution on domain adaptive semantic segmentation tasks in image pairs. Inspired by long tail learning, we divided categories into head, middle, and tail categories, and designed effective strategies to enhance the contribution of tail categories in domain adaptation tasks. We introduce the concept of category rebalancing into domain adaptive image matching, and our method is lightweight and effective, proving its effectiveness on general datasets.

Gaussian Process-Based Transfer Kernel Learning for Unsupervised Domain Adaptation

The discriminability and transferability of models are two important factors for the success of domain adaptation methods. Recently, some domain adaptation methods have improved models by adding a discriminant information extraction module. However, these methods need to carefully balance the discriminability and transferability of a model. To address this problem, we propose a new deep domain adaptation method, Gaussian Process-based Transfer Kernel Learning (GPTKL), which can perform domain knowledge transfer and improve the discrimination ability of the model simultaneously. GPTKL uses the kernel similarity between all samples in the source and target domains as a priori information to establish a cross-domain Gaussian process. By maximizing its likelihood function, GPTKL reduces the domain discrepancy between the source and target domains, thereby enhancing generalization across domains. At the same time, GPTKL introduces the deep kernel learning strategy into the cross-domain Gaussian process to learn a transfer kernel function based on deep features. Through transfer kernel learning, GPTKL learns a deep feature space with both discriminability and transferability. In addition, GPTKL uses cross-entropy and mutual information to learn a classification model shared by the source and target domains. Experiments on four benchmarks show that GPTKL achieves superior classification performance over state-of-the-art methods.

Coronary heart disease prediction method fusing domain-adaptive transfer learning with graph convolutional networks (GCN)

Graph convolutional networks (GCNs) have achieved impressive results in many medical scenarios involving graph node classification tasks. However, there are difficulties in transfer learning for graph representation learning and graph network models. Most GNNs work only in a single domain and cannot transfer the learned knowledge to other domains. Coronary Heart Disease (CHD) is a high-mortality disease, and there are non-public and significant differences in CHD datasets for current research, which makes it difficult to perform unified transfer learning. Therefore, in this paper, we propose a novel adversarial domain-adaptive multichannel graph convolutional network (DAMGCN) that can perform graph transfer learning on cross-domain tasks to achieve cross-domain medical knowledge transfer on different CHD datasets. First, we use a two-channel GCN model for feature aggregation using local consistency and global consistency. Then, a uniform node representation is generated for different graphs using an attention mechanism. Finally, we provide a domain adversarial module to decrease the discrepancies between the source and target domain classifiers and optimize the three loss functions in order to accomplish source and target domain knowledge transfer. The experimental findings demonstrate that our model performs best on three CHD datasets, and its performance is greatly enhanced by graph transfer learning.

A Software Engineering Framework for Reusable Design of Personalized Serious Games for Health: Development Study.

The use of serious games in health care is on the rise, as these games motivate treatment adherence, reduce treatment costs, and educate patients and families. However, current serious games fail to offer personalized interventions, ignoring the need to abandon the one-size-fits-all approach. Moreover, these games, with a primary objective other than pure entertainment, are costly and complex to develop and require the constant involvement of a multidisciplinary team. No standardized approach exists on how serious games can be personalized, as existing literature focuses on specific use cases and scenarios. The serious game development domain fails to consider any transfer of domain knowledge, which means this labor-intensive process must be repeated for each serious game. We proposed a software engineering framework that aims to streamline the multidisciplinary design process of personalized serious games in health care and facilitates the reuse of domain knowledge and personalization algorithms. By focusing on the transfer of knowledge to new serious games by reusing components and personalization algorithms, the comparison and evaluation of different personalization strategies can be simplified and expedited. In doing so, the first steps are taken in advancing the state of the art of knowledge regarding personalized serious games in health care. The proposed framework aimed to answer 3 questions that need to be asked when designing personalized serious games: Why is the game personalized? What parameters can be used for personalization? and How is the personalization achieved? The 3 involved stakeholders, namely, the domain expert, the (game) developer, and the software engineer, were each assigned a question and then assigned responsibilities regarding the design of the personalized serious game. The (game) developer was responsible for all the game-related components; the domain expert was in charge of the modeling of the domain knowledge using simple or complex concepts (eg, ontologies); and the software engineer managed the personalization algorithms or models integrated into the system. The framework acted as an intermediate step between game conceptualization and implementation; it was illustrated by developing and evaluating a proof of concept. The proof of concept, a serious game for shoulder rehabilitation, was evaluated using simulations of heart rate and game scores to assess how personalization was achieved and whether the framework responded as expected. The simulations indicated the value of both real-time and offline personalization. The proof of concept illustrated how the interaction between different components worked and how the framework was used to simplify the design process. The proposed framework for personalized serious games in health care identifies the responsibilities of the involved stakeholders in the design process, using 3 key questions for personalization. The framework focuses on the transferability of knowledge and reusability of personalization algorithms to simplify the design process of personalized serious games.

VGGIN-Net: Deep Transfer Network for Imbalanced Breast Cancer Dataset.

In this paper, we have presented a novel deep neural network architecture involving transfer learning approach, formed by freezing and concatenating all the layers till block4 pool layer of VGG16 pre-trained model (at the lower level) with the layers of a randomly initialized naïve Inception block module (at the higher level). Further, we have added the batch normalization, flatten, dropout and dense layers in the proposed architecture. Our transfer network, called VGGIN-Net, facilitates the transfer of domain knowledge from the larger ImageNet object dataset to the smaller imbalanced breast cancer dataset. To improve the performance of the proposed model, regularization was used in the form of dropout and data augmentation. A detailed block-wise fine tuning has been conducted on the proposed deep transfer network for images of different magnification factors. The results of extensive experiments indicate a significant improvement of classification performance after the application of fine-tuning. The proposed deep learning architecture with transfer learning and fine-tuning yields the highest accuracies in comparison to other state-of-the-art approaches for the classification of BreakHis breast cancer dataset. The articulated architecture is designed in a way that it can be effectively transfer learned on other breast cancer datasets.

CNNs/ViTs-CNNs/ViTs: Mutual distillation for unsupervised domain adaptation

Unsupervised Domain Adaptation (UDA) is a popular machine learning technique to reduce the distribution discrepancy among domains. In previous UDA methods, only convolutional neural networks (CNNs) or vision transformers (ViTs) are used as the backbone. Therefore, these methods are subjected to inherent characteristics of single-CNNs or single-ViTs. In fact, CNNs are more biased towards local textures, while ViTs are inclined to learn the shape information of images. In this paper, we consider that the model combining the above two bias properties is closer to the human visual neural system and achieves more robust performance. A novel mutual distillation method for UDA without any assistant networks is proposed. Mutual distillation between two backbones with complementary properties (i.e. CNNs & ViTs) can promote each other, leading to better domain knowledge transfer. Additionally, traditional domain-mixup approaches can only mix limited cross-domain information through linear interpolation. To encourage more cross-domain information interaction between the two backbones and bridge the domain gap, we propose a patch-mixup method, building a train of mixed intermediate domains composed of augmented patches. Meanwhile, a cross-domain semantic alignment loss is proposed to align the semantic information of these domains. Extensive experiments show that our method achieves state-of-the-art results on several standard UDA benchmarks, such as Office-31, Office-Home, ImageCLEF-DA and VisDA-2017.

Holistic Fault Detection and Diagnosis System in Imbalanced, Scarce, Multi-Domain (ISMD) Data Setting for Component-Level Prognostics and Health Management (PHM)

In the current Industry 4.0 revolution, prognostics and health management (PHM) is an emerging field of research. The difficulty of obtaining data from electromechanical systems in an industrial setting increases proportionally with the scale and accessibility of the automated industry, resulting in a less interpolated PHM system. To put it another way, the development of an accurate PHM system for each industrial system necessitates a unique dataset acquired under specified conditions. In most circumstances, obtaining this one-of-a-kind dataset is difficult, and the resulting dataset has a significant imbalance, a lack of certain useful information, and contains multi-domain knowledge. To address those issues, this paper provides a fault detection and diagnosis system that evaluates and preprocesses imbalanced, scarce, multi-domain (ISMD) data acquired from an industrial robot, utilizing signal processing (SP) techniques and deep learning-based (DL) domain knowledge transfer. The domain knowledge transfer is used to produce a synthetic dataset with a high interpolation rate that contains all the useful information about each domain. For domain knowledge transfer and data generation, continuous wavelet transform (CWT) with a generative adversarial network (GAN) was used, as well as a convolutional neural network (CNN), to test the suggested methodology using transfer learning and categorize several faults. The proposed methodology was tested on a real experimental bench that included an industrial robot created by Hyundai Robotics. This test had a satisfactory outcome with a 99.7% (highest) classification accuracy achieved by transfer learning on several CNN benchmark models.

Unsupervised Black-Box Model Domain Adaptation for Brain Tumor Segmentation.

Unsupervised domain adaptation (UDA) is an emerging technique that enables the transfer of domain knowledge learned from a labeled source domain to unlabeled target domains, providing a way of coping with the difficulty of labeling in new domains. The majority of prior work has relied on both source and target domain data for adaptation. However, because of privacy concerns about potential leaks in sensitive information contained in patient data, it is often challenging to share the data and labels in the source domain and trained model parameters in cross-center collaborations. To address this issue, we propose a practical framework for UDA with a black-box segmentation model trained in the source domain only, without relying on source data or a white-box source model in which the network parameters are accessible. In particular, we propose a knowledge distillation scheme to gradually learn target-specific representations. Additionally, we regularize the confidence of the labels in the target domain via unsupervised entropy minimization, leading to performance gain over UDA without entropy minimization. We extensively validated our framework on a few datasets and deep learning backbones, demonstrating the potential for our framework to be applied in challenging yet realistic clinical settings.

Multi-source unsupervised domain adaptation for object detection

Domain adaptation for object detection has been extensively studied in recent years. Most existing approaches focus on single-source unsupervised domain adaptive object detection. However, a more practical scenario is that the labeled source data is collected from multiple domains with different feature distributions. The conventional approaches do not work very well since multiple domain gaps exist. We propose a Multi-source domain Knowledge Transfer (MKT) method to handle this situation. First, the low-level features from multiple domains are aligned by learning a shallow feature extraction network. Then, the high-level features from each pair of source and target domains are aligned by the followed multi-branch network. After that, we perform two parts of information fusion: (1) We train a detection network shared by all branches based on the transferability of each source sample feature. The transferability of a source sample feature means the indistinguishable degree to the target domain sample features. (2) For using our model, the target sample features output by the multi-branch network are fused based on the average transferability of each domain. Moreover, we leverage both image-level and instance-level attention to promote positive cross-domain transfer and suppress negative transfer. Our main contributions are the two-stage feature alignments and information fusion. Extensive experimental results on various transfer scenarios show that our method achieves the state-of-the-art performance.

Support mechanisms for research generation and application for postgraduate students in four universities in Uganda

BackgroundA large proportion of postgraduate students the world over complete a research thesis in partial fulfilment of their degree requirements. This study identified and evaluated support mechanisms for research generation and utilization for masters’ students in health institutions of higher learning in Uganda.MethodsThis was a self-administered cross-sectional survey using a modified self-assessment tool for research institutes (m-SATORI). Postgraduate students were randomly selected from four medical or public health Ugandan universities at Makerere, Mbarara, Nkozi and Mukono and asked to circle the most appropriate response on a Likert scale from 1, where the “situation was unfavourable and/or there was a need for an intervention”, to 5, where the “situation was good or needed no intervention”. These questions were asked under four domains: the research question; knowledge production, knowledge transfer and promoting use of evidence. Mean scores of individual questions and aggregate means under the four domains were computed and then compared to identify areas of strengths and gaps that required action.ResultsMost of the respondents returned their questionnaires, 185 of 258 (71.7%), and only 79 of these (42.7%) had their theses submitted for examination. The majority of the respondents were male (57.3%), married or cohabiting (58.4%), and were medical doctors (71.9%) from Makerere University (50.3%). The domain proposal development for postgraduate research project had the highest mean score of 3.53 out of the maximum 5. Three of the four domains scored below the mid-level domain score of 3, that is, the situation is neither favourable nor unfavourable. Areas requiring substantial improvements included priority-setting during research question identification, which had the lowest mean score of 2.12. This was followed by promoting use of postgraduate research products, tying at mean scores of 2.28 each. The domain knowledge transfer of postgraduate research products had an above-average mean score of 2.75.ConclusionsThis study reports that existing research support mechanisms for postgraduate students in Uganda encourage access to supervisors and mentors during proposal development. Postgraduate students’ engagement with research users was limited in priority-setting and knowledge transfer. Since supervisors and mentors views were not captured, future follow-on research could tackle this aspect.

Communication is a two-way street: Analyzing practices undertaken to systematically transfer audit research knowledge to policymakers

Audit academics and policymakers express ongoing concerns about limited knowledge transfer between audit research and policymaking. We use theory-based knowledge transfer norms to evaluate eight practices used by audit academics to transfer research knowledge to policymakers. The discontinued PCAOB-AAA Auditing Section “Research Synthesis Project” came closest to enacting these theory-based knowledge transfer norms. Hence, we examine why those involved in that project did not follow these norms. Interviews with project authors reveal that their review creation approach anchored on the traditional academic literature review, and insufficiently adjusted it to meet the goal of communicating audit research evidence to policymakers. However, interviews with PCAOB project liaisons indicate that policymakers were engaged with and found value in the review creation process. Thus, we analyze PCAOB rulemaking documents for evidence that policymakers valued and used the project’s reviews in their policymaking. Over the project’s duration, we find increasing citations of research to the reviews themselves and to a broader set of academic research. Our findings of knowledge transfer occurrence in this project warrant further research on the efficacy of mobilizing audit research via research syntheses. We also show how several current audit domain knowledge transfer practices can be combined with the research syntheses approach leading to systematic effective knowledge transfer to policymakers.

Towards effective deep transfer via attentive feature alignment

Training a deep convolutional network from scratch requires a large amount of labeled data, which however may not be available for many practical tasks. To alleviate the data burden, a practical approach is to adapt a pre-trained model learned on the large source domain to the target domain, but the performance can be limited when the source and target domain data distributions have large differences. Some recent works attempt to alleviate this issue by imposing feature alignment over the intermediate feature maps between the source and target networks. However, for a source model, many of the channels/spatial-features for each layer can be irrelevant to the target task. Thus, directly applying feature alignment may not achieve promising performance. In this paper, we propose an Attentive Feature Alignment (AFA) method for effective domain knowledge transfer by identifying and attending on the relevant channels and spatial features between two domains. To this end, we devise two learnable attentive modules at both the channel and spatial levels. We then sequentially perform attentive spatial- and channel-level feature alignments between the source and target networks, in which the target model and attentive module are learned simultaneously. Moreover, we theoretically analyze the generalization performance of our method, which confirms its superiority to existing methods. Extensive experiments on both image classification and face recognition demonstrate the effectiveness of our method. The source code and the pre-trained models are available at https://github.com/xiezheng-cs/AFAhttps://github.com/xiezheng-cs/AFA.

Deep Learning Driven Venue Recommender for Event-Based Social Networks

Event-based online social platforms, such as Meetup and Plancast, have experienced increased popularity and rapid growth in recent years. In EBSN setup, selecting suitable venues for hosting events, which can attract a great turnout, is a key challenge. In this paper, we present a deep learning based venue recommendation system $DeepVenue$ D e e p V e n u e which provides context driven venue recommendations for the Meetup event-hosts to host their events. The crux of the proposed model relies on the notion of similarity between multiple Meetup entities such as events, venues, groups, etc. We develop deep learning techniques to compute a compact descriptor for each entity, such that two entities (say, venues) can be compared numerically. Notably, to mitigate the scarcity of venue related information in Meetup, we leverage on the cross domain knowledge transfer from popular LBSN service Yelp to extract rich venue related content. For hosting an event, the proposed $DeepVenue$ D e e p V e n u e model computes a success score for each candidate venue and ranks those venues according to the scores and finally recommend the top k venues. Our rigorous evaluation on the Meetup data collected for the city of Chicago shows that $DeepVenue$ D e e p V e n u e significantly outperforms the baselines algorithms. Precisely, for 84 percent of events, the correct hosting venue appears in the top 5 of the $DeepVenue$ D e e p V e n u e recommended list.

Communication Is a Two-Way Street: Analyzing Approaches Undertaken to Enhance Audit Research Knowledge Transfer to Policymakers

Audit academics and policymakers express ongoing concerns about limited knowledge transfer between audit research and policymaking. We use theory-based knowledge transfer norms to evaluate eight practices used by audit academics to transfer research knowledge to policymakers. The discontinued PCAOB-AAA Auditing Section “Research Synthesis Project” came closest to enacting these theory-based knowledge transfer norms. Hence, we examine why those involved in that project did not follow these norms. Interviews with project authors reveal that their review creation approach anchored on the traditional academic literature review, and insufficiently adjusted it to meet the goal of communicating audit research evidence to policymakers. However, interviews with PCAOB project liaisons indicate that policymakers were engaged with and found value in the review creation process. Thus, we analyze PCAOB rulemaking documents for evidence that policymakers valued and used the project’s reviews in their policymaking. Over the project’s duration, we find increasing citations of research to the reviews themselves and to a broader set of academic research. Our findings of knowledge transfer occurrence in this project warrant further research on the efficacy of mobilizing audit research via research syntheses. We also show how several current audit domain knowledge transfer practices can be combined with the research syntheses approach leading to systematic effective knowledge transfer to policymakers.

Example-driven modeling: on effects of using examples on structural model comprehension, what makes them useful, and how to create them

We present a controlled experiment for the empirical evaluation of example-driven modeling (EDM), an approach that systematically uses examples for model comprehension and domain knowledge transfer. We conducted the experiment with 26 graduate (Masters and Ph.D. level) and undergraduate (Bachelor level) students from electrical and computer engineering, computer science, and software engineering programs at the University of Waterloo. The experiment involves a domain model, with UML class diagrams representing the domain abstractions and UML object diagrams representing examples of using these abstractions. The goal is to provide empirical evidence of the effects of suitable examples on model comprehension, compared to having model abstractions only, by having the participants perform model comprehension tasks. Our results show that EDM is superior to having model abstractions only, with an improvement of 39% for diagram completeness, 33% for questions completeness, 71% for efficiency, and a reduction in the number of mistakes by 80%. We provide qualitative results showing that participants receiving model abstractions augmented with examples experienced lower perceived difficulty in performing the comprehension tasks, higher perceived confidence in their tasks’ solutions, and asked 90% fewer clarifying domain questions. We also present participants’ feedback regarding the usefulness of the provided examples, their number and types, as well as the use of partial examples. We present a taxonomy of the different types of examples, explain their significance, and propose guidelines for manual and automatic creation of useful examples.

Autoencoder-based Unsupervised Domain Adaptation for Speech Emotion Recognition

With the availability of speech data obtained from different devices and varied acquisition conditions, we are often faced with scenarios, where the intrinsic discrepancy between the training and the test data has an adverse impact on affective speech analysis. To address this issue, this letter introduces an Adaptive Denoising Autoencoder based on an unsupervised domain adaptation method, where prior knowledge learned from a target set is used to regularize the training on a source set. Our goal is to achieve a matched feature space representation for the target and source sets while ensuring target domain knowledge transfer. The method has been successfully evaluated on the 2009 INTERSPEECH Emotion Challenge's FAU Aibo Emotion Corpus as target corpus and two other publicly available speech emotion corpora as sources. The experimental results show that our method significantly improves over the baseline performance and outperforms related feature domain adaptation methods.

Transfer of Domain Knowledge in Plan Generation: Learning Goal-dependent Annulling Conditions for Actions

In this paper we present an approach to avoid dead-ends during automated plan generation. A first-order logic formula can be learned that holds in a state if the application of a specific action will lead to a dead-end. Starting from small problems within a problem domain examples of states where the application of the action will lead to a dead-end will be collected. The states will be generalized using inductive logic programming to a first-order logic formula. We will show how different notions of goal-dependence could be integrated in this approach. The formula learned will be used to speed-up automated plan generation. Furthermore, it provides insight into the planning domain under consideration.