Multiple Source Domains Research Articles

Online transfer learning by leveraging multiple source domains

Transfer learning aims to enhance performance in a target domain by exploiting useful information from auxiliary or source domains when the labeled data in the target domain are insufficient or difficult to acquire. In some real-world applications, the data of source domain are provided in advance, but the data of target domain may arrive in a stream fashion. This kind of problem is known as online transfer learning. In practice, there can be several source domains that are related to the target domain. The performance of online transfer learning is highly associated with selected source domains, and simply combining the source domains may lead to unsatisfactory performance. In this paper, we seek to promote classification performance in a target domain by leveraging labeled data from multiple source domains in online setting. To achieve this, we propose a new online transfer learning algorithm that merges and leverages the classifiers of the source and target domain with an ensemble method. The mistake bound of the proposed algorithm is analyzed, and the comprehensive experiments on three real-world data sets illustrate that our algorithm outperforms the compared baseline algorithms.

Domain-specific sentiment classification via fusing sentiment knowledge from multiple sources

Analyzing the sentiments in massive user-generated online data, such as product reviews and microblogs, has become a hot research topic. It can help customers, companies and expert systems make more informed decisions. Sentiment analysis is widely known as a domain dependent problem. Different domains usually have different sentiment expressions and a general sentiment classifier is not suitable for all domains. A natural solution to this problem is to train a domain-specific sentiment classifier for each target domain. However, the labeled data in target domain is usually insufficient, and it is costly and time-consuming to annotate enough samples. In order to tackle this problem, we propose a novel approach to train domain-specific sentiment classifiers by fusing the sentiment knowledge from multiple sources. Sentiment information from four sources is extracted and fused in our approach. The first source is sentiment lexicons, which contain sentiment polarities of general sentiment words. The second source is the sentiment classifiers of multiple source domains. The third source is the unlabeled data in target domain, from which we extract domain-specific sentiment relations among words. The fourth source is the labeled data in target domain. We propose a unified framework to fuse these four kinds of sentiment knowledge and train domain-specific sentiment classifier for target domain. In addition, we present an efficient optimization algorithm to solve the model of our approach. Extensive experiments are conducted on both Amazon product review dataset and Twitter dataset. Experimental results show that by fusing the sentiment information extracted from multiple sources, our approach can effectively improve the performance of sentiment classification and reduce the dependence on labeled data. For instance, our approach can achieve an accuracy of 87.22% in Kitchen domain when only 200 samples in target domain are labeled. The performance improvements of our approach compared with purely supervised sentiment classifier are 8.98% and 7.92% on Amazon and Twitter datasets respectively.

Assessment of seismic liquefaction potential based on Bayesian network constructed from domain knowledge and history data

Prediction of seismic liquefaction is difficult due to the uncertainties and complexity of multiple related factors. Bayesian network is a just right effective tool to deal the problem because of merging multiple source information and domain knowledge in a consistent system, reflecting and analyzing the interdependent uncertain relationships between variables. This paper used two ways to construct generic Bayesian network models with twelve significant factors of seismic liquefaction, of which the first model is constructed only by interpretive structural modeling and causal mapping approach for incomplete data contained huge missing values. Another one is constructed by combining K2 algorithm and domain knowledge for complete data. Compared with artificial neural network and support vector machine using 5-fold cross-validation, the two Bayesian network models provided a better performance, and the second Bayesian network model is slightly better than the first one. This paper also offers a sensitivity analysis of the input factors. In the twelve variables, standard penetration test number, soil type, vertical effective stress, depth of soil deposit, and peak ground acceleration have more significant influences on seismic liquefaction than others. Our results suggest that the Bayesian network is useful for prediction of seismic liquefaction and is simple to perform in practice.

Robust multi-source adaptation visual classification using supervised low-rank representation

How to guarantee the robustness of multi-source adaptation visual classification is an important challenge in current visual learning community. To this end, we address in this paper the problem of robust visual classification with few labeled samples from the target domain of interest by leveraging multiple prior source models. Motivated by the recent success of low rank representation, we formulate this problem as a robust multi-source adaptation visual classification (RMAVC) model with supervised low rank representation by combining the strength of discriminative information from the target domain and the prior models from multiple source domains. Specifically, we propose a joint supervised low rank representation and multi-source adaptation visual classification framework, which achieves dual goals of finding the most discriminative low rank representation and multi-source adaptation classifier parameters for the target domain. While it is showed in this paper that the proposed RMAVC framework is effective and can produce high accuracy on several tasks of multi-source adaptation visual classification, this framework fails to consider the local geometrical structure of the target data and the heterogeneousness among multiple source domains. Hence, under this framework, we further present two effective extensions or variants, i.e., RMAVCK and RMAVC_FM, by exploiting multiple kernel trick and flexible manifold regularization, respectively. The proposed framework and its variants are robust for classifying visual objects accurately and the experimental results demonstrate the effectiveness of our methods on several types of image and video datasets.

Sample selection for visual domain adaptation via sparse coding

Visual domain adaptation has attracted much attention and has made great achievement in recent years. It deals with the problem of distribution divergence between source and target domains. Current methods mostly focus on transforming images from different domains into a common space to minimize the distribution divergence. However, there are many irrelevant source samples for target domain even after the transformation. In order to eliminate the irrelevant samples, we develop a sample selection algorithm using sparse coding theory. We do the sample selection in a common subspace of source and target data to find as many as relevant source samples. In the common subspace, data characteristics are preserved by using graph regularization. Therefore, we can select the most relevant samples for our target image classification task. Moreover, in order to build a discriminative classifier for the target domain, we use not only the common part of source and target domains learned in the common subspace but also the specific part of target domain. The algorithm can be extended to handle samples from multiple source domains. Experimental results show that our visual domain adaptation method on the image classification tasks can be very effective for the state-of-the-art datasets.

Action and Event Recognition in Videos by Learning From Heterogeneous Web Sources.

In this paper, we propose new approaches for action and event recognition by leveraging a large number of freely available Web videos (e.g., from Flickr video search engine) and Web images (e.g., from Bing and Google image search engines). We address this problem by formulating it as a new multi-domain adaptation problem, in which heterogeneous Web sources are provided. Specifically, we are given different types of visual features (e.g., the DeCAF features from Bing/Google images and the trajectory-based features from Flickr videos) from heterogeneous source domains and all types of visual features from the target domain. Considering the target domain is more relevant to some source domains, we propose a new approach named multi-domain adaptation with heterogeneous sources (MDA-HS) to effectively make use of the heterogeneous sources. In MDA-HS, we simultaneously seek for the optimal weights of multiple source domains, infer the labels of target domain samples, and learn an optimal target classifier. Moreover, as textual descriptions are often available for both Web videos and images, we propose a novel approach called MDA-HS using privileged information (MDA-HS+) to effectively incorporate the valuable textual information into our MDA-HS method, based on the recent learning using privileged information paradigm. MDA-HS+ can be further extended by using a new elastic-net-like regularization. We solve our MDA-HS and MDA-HS+ methods by using the cutting-plane algorithm, in which a multiple kernel learning problem is derived and solved. Extensive experiments on three benchmark data sets demonstrate that our proposed approaches are effective for action and event recognition without requiring any labeled samples from the target domain.

Multi-source adaptation learning with global and local regularization by exploiting joint kernel sparse representation

While the advantages of using information from multi-source domains for establishing an adaptation model have been widely recognized, how to boosting the robustness of the learning model has only recently received attention. For achieving a robust adaptation learning model for visual classification tasks, by exploiting multi-source adaptation regularization joint kernel sparse representation (ARJKSR), we propose a novel Multi-source Adaptation learning method with Global and Local Regularization (MAGLR), which aims to minimize a cost function that properly trades off the global and local learning costs so as to utilize both the local and global discriminative information of target domain. Specifically, we first learn the optimal ARJKSR of domain datasets, which jointly represents the datasets by a sparse linear combination of a self-expressive dictionary in some optimal reproduced kernel Hilbert space (RKHS) recovered by minimizing the inter-domain distribution discrepancy, whilst constraining the observations from domains to share their sparse representations. Thus, we simultaneously take into account correlations as well as coupling information among multiple source domains. Based on the ARJKSR, we then derive a robust label prediction matrix for unlabeled instances from the target domain based on the classical graph-based semi-supervised learning diagram, into which global and local regularization are incorporated. Furthermore, a multi-kernel regression way is exploited to extend MAGLR to out-of-sample data. The validity of our method is examined by several real-world visual classification tasks. Experimental results demonstrate the superiority of our method.

Linear combination of densities and its direct estimation framework with applications

In this paper, typical learning task including data condensation, binary classification, identification of the independence between random variables and conditional density estimation is described from a unified perspective of a linear combination of densities, and accordingly a direct estimation framework based on a linear combination of Gaussian components (i.e., Gaussian basis functions) under integrated square error criterion is proposed to solve these learning tasks. The proposed direct estimation framework has three advantages. Firstly, different from most of the existing state-of-the-art methods in which estimating each component's density in this linear combination of densities and then combining them linearly are required, it can directly estimate the linear combination of densities as a whole, and it has at least comparable to or even better approximation accuracy than the existing density estimation methods. Secondly, the time complexity of the proposed direct estimation framework is O(l3) in which l is the number of Gaussian components in this framework which are generally viewed as the Gaussian distributions of the clusters in a dataset, and hence l is generally much less than the size of the dataset, so it is very suitable for large datasets. Thirdly, this proposed framework can be typically used to develop alternative approaches to classification, data condensation, identification of the independence between random variables, conditional density estimation and the similarity identification between multiple source domains and a target domain. Our preliminary results about experiments on several typical applications indicate the power of the proposed direct estimation framework.

Multi-Source Domain Adaptation: A Causal View

This paper is concerned with the problem of domain adaptation with multiple sources from a causal point of view. In particular, we use causal models to represent the relationship between the features X and class label Y , and consider possible situations where different modules of the causal model change with the domain. In each situation, we investigate what knowledge is appropriate to transfer and find the optimal target-domain hypothesis. This gives an intuitive interpretation of the assumptions underlying certain previous methods and motivates new ones. We finally focus on the case where Y is the cause for X with changing PY and PX|Y , that is, PY and PX|Y change independently across domains. Under appropriate assumptions, the availability of multiple source domains allows a natural way to reconstruct the conditional distribution on the target domain; we propose to model PX|Y (the process to generate effect X from cause Y ) on the target domain as a linear mixture of those on source domains, and estimate all involved parameters by matching the target-domain feature distribution. Experimental results on both synthetic and real-world data verify our theoretical results.

Flowing on Riemannian manifold: domain adaptation by shifting covariance.

Domain adaptation has shown promising results in computer vision applications. In this paper, we propose a new unsupervised domain adaptation method called domain adaptation by shifting covariance (DASC) for object recognition without requiring any labeled samples from the target domain. By characterizing samples from each domain as one covariance matrix, the source and target domain are represented into two distinct points residing on a Riemannian manifold. Along the geodesic constructed from the two points, we then interpolate some intermediate points (i.e., covariance matrices), which are used to bridge the two domains. By utilizing the principal components of each covariance matrix, samples from each domain are further projected into intermediate feature spaces, which finally leads to domain-invariant features after the concatenation of these features from intermediate points. In the multiple source domain adaptation task, we also need to effectively integrate different types of features between each pair of source and target domains. We additionally propose an SVM based method to simultaneously learn the optimal target classifier as well as the optimal weights for different source domains. Extensive experiments demonstrate the effectiveness of our method for both single source and multiple source domain adaptation tasks.

Multi-source transfer learning based on label shared subspace

Multi-source transfer learning focuses on studying the scarcity of samples with labels in target domain, while neglecting the analysis about transferability relationship among multiple source domains. Thus, we propose a method that transforms samples in target domain into multi-label samples, with which it is able to analyze the correlations among predicted labels from different sources. We design a method that can extract the shared subspace among labels in multi-sources, and propose a novel multi-source transfer learning method based on multi-label shared subspace. This approach is required when knowledge about multiple sources are available but it is unknown which source is of more transferability. Experiments show that our proposed algorithm can improve the performance of transfer learning method and alleviate time complexity.

Video Annotation via Image Groups from the Web

Searching desirable events in uncontrolled videos is a challenging task. Current researches mainly focus on obtaining concepts from numerous labeled videos. But it is time consuming and labor expensive to collect a large amount of required labeled videos for training event models under various circumstances. To alleviate this problem, we propose to leverage abundant Web images for videos since Web images contain a rich source of information with many events roughly annotated and taken under various conditions. However, knowledge from the Web is noisy and diverse, brute force knowledge transfer of images may hurt the video annotation performance. Therefore, we propose a novel Group-based Domain Adaptation (GDA) learning framework to leverage different groups of knowledge (source domain) queried from the Web image search engine to consumer videos (target domain). Different from traditional methods using multiple source domains of images, our method organizes the Web images according to their intrinsic semantic relationships instead of their sources. Specifically, two different types of groups (i.e., event-specific groups and concept-specific groups) are exploited to respectively describe the event-level and concept-level semantic meanings of target-domain videos. Under this framework, we assign different weights to different image groups according to the relevances between the source groups and the target domain, and each group weight represents how contributive the corresponding source image group is to the knowledge transferred to the target video. In order to make the group weights and group classifiers mutually beneficial and reciprocal, a joint optimization algorithm is presented for simultaneously learning the weights and classifiers, using two novel data-dependent regularizers. Experimental results on three challenging video datasets (i.e., CCV, Kodak, and YouTube) demonstrate the effectiveness of leveraging grouped knowledge gained from Web images for video annotation.

A Survey of Using Weakly Supervised and Semi-Supervised for Cross-Domain Sentiment Classification

Supervised machine learning techniques can analyze sentiment very effectively. However, in many languages, there are few appropriate data for training sentiment classifiers. Thus, they need a large corpus of training data. In this paper, weakly-supervised techniques using a large collection of unlabeled text to determine sentiment is presented. The performance of this method maybe less depends on the domain, topic and time period represented by the testing data. In addition, semi-supervised classification using a sentiment-sensitive thesaurus is mentioned. It can be applicable when it does not have any labeled data for a target domain but have some labeled data for other multiple domains designated as the source domains. This method can learn efficiently from multiple source domains. The results show that the weakly-supervised techniques are suitable for applications requiring sentiment classification across some domains and semi-supervised techniques can learn efficiently from multiple source domains.

Instance transfer learning with multisource dynamic TrAdaBoost.

Since the transfer learning can employ knowledge in relative domains to help the learning tasks in current target domain, compared with the traditional learning it shows the advantages of reducing the learning cost and improving the learning efficiency. Focused on the situation that sample data from the transfer source domain and the target domain have similar distribution, an instance transfer learning method based on multisource dynamic TrAdaBoost is proposed in this paper. In this method, knowledge from multiple source domains is used well to avoid negative transfer; furthermore, the information that is conducive to target task learning is obtained to train candidate classifiers. The theoretical analysis suggests that the proposed algorithm improves the capability that weight entropy drifts from source to target instances by means of adding the dynamic factor, and the classification effectiveness is better than single source transfer. Finally, experimental results show that the proposed algorithm has higher classification accuracy.

Using space to talk and gesture about numbers

This paper examines naturally occurring gestures produced in descriptions of numbers and quantities in television newscasts. The results of our analysis show that gestures reveal the metaphorical and spatial nature of numerical thinking. That is, speakers’ hands mimic known spatial mappings between space and quantity, including horizontal mappings (smaller quantities left, larger quantities right), vertical mappings (smaller quantities down, larger quantities up) and size-based mappings (smaller quantities “small”, larger quantities “large”). Speakers frequently switch between these different spatial mappings, and they sometimes combine them within the same gesture. This points to the flexibility of how metaphors can become expressed in gesture, and how domains such as number and quantity can be conceptualized through multiple compatible source domains.

Transfer ordinal label learning.

Designing a classifier in the absence of labeled data is becoming a common encounter as the acquisition of informative labels is often difficult or expensive, particularly on new uncharted target domains. The feasibility of attaining a reliable classifier for the task of interest is embarked by some in transfer learning, where label information from relevant source domains is considered for complimenting the design process. The core challenge arising from such endeavors, however, is the induction of source sample selection bias, such that the trained classifier has the tendency of steering toward the distribution of the source domain. In addition, this bias is deemed to become more severe on data involving multiple classes. Considering this cue, our interest in this paper is to address such a challenge in the target domain, where ordinal labeled data are unavailable. In contrast to the previous works, we propose a transfer ordinal label learning paradigm to predict the ordinal labels of target unlabeled data by spanning the feasible solution space with ensemble of ordinal classifiers from the multiple relevant source domains. Specifically, the maximum margin criterion is considered here for the construction of the target classifier from an ensemble of source ordinal classifiers. Theoretical analysis and extensive empirical studies on real-world data sets are presented to study the benefits of the proposed method.

Transfer learning with one-class data

When training and testing data are drawn from different distributions, most statistical models need to be retrained using the newly collected data. Transfer learning is a family of algorithms that improves the classifier learning in a target domain of interest by transferring the knowledge from one or multiple source domains, where the data falls in a different distribution. In this paper, we consider a new scenario of transfer learning for two-class classification, where only data samples from one of the two classes (e.g., the negative class) are available in the target domain. We introduce a regression-based one-class transfer learning algorithm to tackle this new problem. In contrast to the traditional discriminative feature selection, which seeks the best classification performance in the training data, we propose a new framework to learn the most transferable discriminative features suitable for our transfer learning. The experiment demonstrates improved performance in the applications of facial expression recognition and facial landmark detection.

Cross-Domain Object Recognition Via Input-Output Kernel Analysis

It is of great importance to investigate the domain adaptation problem of image object recognition, because now image data is available from a variety of source domains. To understand the changes in data distributions across domains, we study both the input and output kernel spaces for cross-domain learning situations, where most labeled training images are from a source domain and testing images are from a different target domain. To address the feature distribution change issue in the reproducing kernel Hilbert space induced by vector-valued functions, we propose a domain adaptive input-output kernel learning (DA-IOKL) algorithm, which simultaneously learns both the input and output kernels with a discriminative vector-valued decision function by reducing the data mismatch and minimizing the structural error. We also extend the proposed method to the cases of having multiple source domains. We examine two cross-domain object recognition benchmark data sets, and the proposed method consistently outperforms the state-of-the-art domain adaptation and multiple kernel learning methods.

FIDOS: A generalized Fisher based feature extraction method for domain shift

Traditional pattern recognition techniques often assume that the data sets used for training and testing follow the same distribution. However, this assumption is usually not true for many real world problems as data from the same classes but different domains, e.g., data are collected under different conditions, may show different characteristics. We introduce FIDOS, a generalized FIsher based method for DOmain Shift problem, that aims at learning invariant features across domains in a supervised manner.Different from classical Fisher feature extraction, FIDOS aims to minimize not only the within-class scatter but also the difference in distributions between domains. Therefore, the subspace constructed by FIDOS reduces the drift in distributions among different domains and at the same time preserves the discriminants across classes. Another advantage of FIDOS over classical Fisher is that FIDOS extracts more features when multiple source domains are available in the training set; this is essential for a good classification especially when the number of classes is small. Experimental results on both artificial and real data and comparisons with other methods demonstrate the efficiency of our method in classifying objects under domain shift situations.

Erratum to "Mining Distinction and Commonality across Multiple Domains Using Generative Model for Text Classification"

The distribution difference among multiple domains has been exploited for cross-domain text categorization in recent years. Along this line, we show two new observations in this study. First, the data distribution difference is often due to the fact that different domains use different index words to express the same concept. Second, the association between the conceptual feature and the document class can be stable across domains. These two observations actually indicate the distinction and commonality across domains. Inspired by the above observations, we propose a generative statistical model, named Collaborative Dual-PLSA (CD-PLSA), to simultaneously capture both the domain distinction and commonality among multiple domains. Different from Probabilistic Latent Semantic Analysis (PLSA) with only one latent variable, the proposed model has two latent factors y and z, corresponding to word concept and document class, respectively. The shared commonality intertwines with the distinctions over multiple domains, and is also used as the bridge for knowledge transformation. An Expectation Maximization (EM) algorithm is developed to solve the CD-PLSA model, and further its distributed version is exploited to avoid uploading all the raw data to a centralized location and help to mitigate privacy concerns. After the training phase with all the data from multiple domains we propose to refine the immediate outputs using only the corresponding local data. In summary, we propose a two-phase method for cross-domain text classification, the first phase for collaborative training with all the data, and the second step for local refinement. Finally, we conduct extensive experiments over hundreds of classification tasks with multiple source domains and multiple target domains to validate the superiority of the proposed method over existing state-of-the-art methods of supervised and transfer learning. It is noted to mention that as shown by the experimental results CD-PLSA for the collaborative training is more tolerant of distribution differences, and the local refinement also gains significant improvement in terms of classification accuracy.