Multi-task Learning Techniques Research Articles

Combination of domain knowledge and deep learning for sentiment analysis of short and informal messages on social media

Sentiment analysis has been emerging recently as one of major natural language processing (NLP) tasks, with the increasing significance of social media channels for brands to observe user opinions about their products. In the previous work, we proposed to combine the typical deep learning techniques with domain knowledge. The combination is used for acquiring additional training data augmentation and a more reasonable loss function. However, there is a high volume of short and informal messages posted by users which makes the existing works suffer from many difficulties. In this work, we further improve our architecture, aiming to handle those problems, by various substantial enhancements, including negation-based data augmentation, transfer learning for word embeddings, combination of word-level embeddings and character-level embeddings, and using multitask learning technique for attaching domain knowledge rules in the learning process. Those enhancements help us to enjoy significant improvement in performance once experimenting on real datasets.

Combination of domain knowledge and deep learning for sentiment analysis of short and informal messages on social media

Sentiment analysis has been emerging recently as one of the major natural language processing (NLP) tasks in many applications. Especially, as social media channels (e.g. social networks or forums) have become significant sources for brands to observe user opinions about their products, this task is thus increasingly crucial. However, when applied with real data obtained from social media, we notice that there is a high volume of short and informal messages posted by users on those channels. This kind of data makes the existing works suffer from many difficulties to handle, especially ones using deep learning approaches. In this paper, we propose an approach to handle this problem. This work is extended from our previous work, in which we proposed to combine the typical deep learning technique of Convolutional Neural Networks with domain knowledge. The combination is used for acquiring additional training data augmentation and a more reasonable loss function. In this work, we further improve our architecture by various substantial enhancements, including negation-based data augmentation, transfer learning for word embeddings, the combination of word-level embeddings and character-level embeddings, and using multitask learning technique for attaching domain knowledge rules in the learning process. Those enhancements, specifically aiming to handle short and informal messages, help us to enjoy significant improvement in performance once experimenting on real datasets.

An Integrated Transfer Learning and Multitask Learning Approach for Pharmacokinetic Parameter Prediction.

Pharmacokinetic evaluation is one of the key processes in drug discovery and development. However, current absorption, distribution, metabolism, and excretion prediction models still have limited accuracy. This study aims to construct an integrated transfer learning and multitask learning approach for developing quantitative structure-activity relationship models to predict four human pharmacokinetic parameters. A pharmacokinetic data set included 1104 U.S. FDA approved small molecule drugs. The data set included four human pharmacokinetic parameter subsets (oral bioavailability, plasma protein binding rate, apparent volume of distribution at steady-state, and elimination half-life). The pretrained model was trained on over 30 million bioactivity data entries. An integrated transfer learning and multitask learning approach was established to enhance the model generalization. The pharmacokinetic data set was split into three parts (60:20:20) for training, validation, and testing by the improved maximum dissimilarity algorithm with the representative initial set selection algorithm and the weighted distance function. The multitask learning techniques enhanced the model predictive ability. The integrated transfer learning and multitask learning model demonstrated the best accuracies, because deep neural networks have the general feature extraction ability; transfer learning and multitask learning improve the model generalization. The integrated transfer learning and multitask learning approach with the improved data set splitting algorithm was first introduced to predict the pharmacokinetic parameters. This method can be further employed in drug discovery and development.

Joint Optical Performance Monitoring and Modulation Format/Bit-Rate Identification by CNN-Based Multi-Task Learning

A novel technique is proposed for joint multi-impairment optical performance monitoring (OPM) with bit-rate and modulation format identification (BR-MFI) in next-generation heterogeneous optic communication networks by convolution neural network (CNN)-based deep multi-task learning (MTL) on asynchronous delay-tap sampling phase portraits. Instead of treating the monitoring and identification tasks as separate problems, a novel MTL technique is used to joint optimization of them utilizing the ability of feature extraction and feature sharing. Compared with principal component analysis-based pattern recognition algorithm, CNN-based MTL achieves the better accuracies and has a shorter processing time (∼56 ms). The combination signals of three modulation formats and two bit rates under various impairments are used in numerical simulation. For OPM, the results show monitoring of optical signal-to-noise ratio, chromatic dispersion, and differential group delay with root-mean-square error of 0.73 dB, 1.34 ps/nm, and 0.47 ps, respectively. Similarly, for BR-MFI, even in the case of limited training data, 100% accuracies can be achieved. Additionally, the effects of training data size, task weights, and model structure on CNN-based MTL performance are comprehensively studied. The proposed technique can intelligently analyze the signals of future heterogeneous optic communication networks, and the analysis results are helpful for better management of optical networks.

Fast cascade face detection with pyramid network

Numerous recent face detectors based on convolutional neural networks (CNNs) have significantly improved the detection performance. However, CNNs usually have a huge number of parameters which lead to very low detection speeds. To address this issue, this paper proposes a fast CNNs cascade face detector with multi-task learning and network acceleration techniques. In particular, the first stage of the detector is an elaborately designed fully convolutional network with a novel pyramid architecture, which can generate multi-scale face proposals efficiently with no more than twice image resizing operations. Several network compression and acceleration techniques including multi-layer merging and knowledge distilling are adopted to further improve inference speed. In addition, online and offline hard sample mining are jointly utilized to further strengthen the power of networks. The experimental results on challenging FDDB show that the proposed face detector is comparable in performance with the-state-of-arts while the speed reaches astonishing 165 fps on Titan GPU.

ECG-Based Concentration Recognition With Multi-Task Regression.

Recognition of human activities and mental states using wearable sensors and smartphones has attracted considerable attention recently. In particular, prediction of the stress level of a subject using an electrocardiogram sensor has been studied extensively. In this paper, we attempt to predict the degree of concentration by using heart-rate features. However, due to strong diversity in individuals and high sampling costs, building an accurate prediction model is still highly challenging. To overcome these difficulties, we propose to use a multitask learning (MTL) technique for effectively sharing information among similar individuals. Through experiments with 18 healthy subjects performing daily office works, such as writing reports, we demonstrate that the proposed method significantly improves the accuracy of concentration prediction in small sample situations. The performance of the MTL method is shown to be stable across different subjects, which is an important advantage over conventional models. This improvement has significant impact in real-world concentration recognition because the data collection burden of each user can be drastically mitigated.

Electricity Demand Forecasting by Multi-Task Learning

We explore the application of kernel-based multi-task learning techniques to forecast the demand of electricity measured on multiple lines of a distribution network. We show that recently developed output kernel learning techniques are particularly well suited to solve this problem, as they allow to flexibly model the complex seasonal effects that characterize electricity demand data, while learning and exploiting correlations between multiple demand profiles. We also demonstrate that kernels with a multiplicative structure yield superior predictive performance with respect to the widely adopted (generalized) additive models. This paper is based on residential and industrial smart meter data provided by the Irish Commission for Energy Regulation.

Transfer and Multi-task Learning in QSAR Modeling: Advances and Challenges.

Medicinal chemistry projects involve some steps aiming to develop a new drug, such as the analysis of biological targets related to a given disease, the discovery and the development of drug candidates for these targets, performing parallel biological tests to validate the drug effectiveness and side effects. Approaches as quantitative study of activity-structure relationships (QSAR) involve the construction of predictive models that relate a set of descriptors of a chemical compound series and its biological activities with respect to one or more targets in the human body. Datasets used to perform QSAR analyses are generally characterized by a small number of samples and this makes them more complex to build accurate predictive models. In this context, transfer and multi-task learning techniques are very suitable since they take information from other QSAR models to the same biological target, reducing efforts and costs for generating new chemical compounds. Therefore, this review will present the main features of transfer and multi-task learning studies, as well as some applications and its potentiality in drug design projects.

Personalized Multitask Learning for Predicting Tomorrow's Mood, Stress, and Health.

While accurately predicting mood and wellbeing could have a number of important clinical benefits, traditional machine learning (ML) methods frequently yield low performance in this domain. We posit that this is because a one-size-fits-all machine learning model is inherently ill-suited to predicting outcomes like mood and stress, which vary greatly due to individual differences. Therefore, we employ Multitask Learning (MTL) techniques to train personalized ML models which are customized to the needs of each individual, but still leverage data from across the population. Three formulations of MTL are compared: i) MTL deep neural networks, which share several hidden layers but have final layers unique to each task; ii) Multi-task Multi-Kernel learning, which feeds information across tasks through kernel weights on feature types; and iii) a Hierarchical Bayesian model in which tasks share a common Dirichlet Process prior. We offer the code for this work in open source. These techniques are investigated in the context of predicting future mood, stress, and health using data collected from surveys, wearable sensors, smartphone logs, and the weather. Empirical results demonstrate that using MTL to account for individual differences provides large performance improvements over traditional machine learning methods and provides personalized, actionable insights.

Independent Encoding Joint Sparse Representation and Multitask Learning for Hyperspectral Target Detection

Target detection is playing an important role in hyperspectral image (HSI) processing. Many traditional detection methods utilize the discriminative information within all the single-band images to distinguish the target and the background. The critical challenge with these methods is simultaneously reducing spectral redundancy and preserving the discriminative information. The multitask learning (MTL) technique has the potential to solve the aforementioned challenge, since it can further explore the inherent spectral similarity between the adjacent single-band images. This letter proposes an independent encoding joint sparse representation and an MTL method. This approach has the following capabilities: 1) explores the inherent spectral similarity to construct multiple sub-HSIs in order to reduce spectral redundancy for each sub-HSI; 2) takes full advantage of the prior class label information to construct reasonable joint sparse representation and MTL models for the target and the background; 3) explores the great difference between the target dictionary and background dictionary with different regularization strategies in order to better encode the task relatedness for two joint sparse representation and MTL models; and 4) makes the detection decision by comparing the reconstruction residuals under different prior class labels. Experiments on two HSIs illustrated the effectiveness of the proposed method.

Joint Sparse Representation and Multitask Learning for Hyperspectral Target Detection

With the high spectral resolution, hyperspectral images (HSIs) provide great potential for target detection, which is playing an increasingly important role in HSI processing. Many target detection methods uniformly utilize all the spectral information or employ reduced spectral information to distinguish the targets and background. Simultaneously reducing spectral redundancy and preserving the discriminative information is a challenging problem in hyperspectral target detection. The multitask learning (MTL) technique may have the potential to solve the above problem, since it can explore the redundancy knowledge to construct multiple sub-HSIs and integrate them without any information loss. This paper proposes the joint sparse representation and MTL (JSR-MTL) method for hyperspectral target detection. This approach: 1) explores the HSIs similarity by a band cross-grouping strategy to construct multiple sub-HSIs; 2) takes full advantage of the MTL technique to integrate the sparse representation models for the multiple related sub-HSIs; and 3) applies the total reconstruction error difference accumulated over all the tasks to detect the targets. Extensive experiments were carried out on three HSIs, and it was founded that JSR-MTL generally shows a better detection performance than the other target detection methods.

Healthcare Predictive Analytics for Risk Profiling in Chronic Care: A Bayesian Multitask Learning Approach

Clinical intelligence about a patient's risk of future adverse health events can support clinical decision making in personalized and preventive care. Healthcare predictive analytics using electronic health records offers a promising direction to address the challenging tasks of risk profiling. Patients with chronic diseases often face risks of not just one, but an array of adverse health events. However, existing risk models typically focus on one specific event and do not predict multiple outcomes. To attain enhanced risk profiling, we adopt the design science paradigm and propose a principled approach called Bayesian multitask learning (BMTL). Considering the model development for an event as a single task, our BMTL approach is to coordinate a set of baseline models—one for each event—and communicate training information across the models. The BMTL approach allows healthcare providers to achieve multifaceted risk profiling and model an arbitrary number of events simultaneously. Our experimental evaluations demonstrate that the BMTL approach attains an improved predictive performance when compared with the alternatives that model multiple events separately. We also find that, in most cases, the BMTL approach significantly outperforms existing multitask learning techniques. More importantly, our analysis shows that the BMTL approach can create significant potential impacts on clinical practice in reducing the failures and delays in preventive interventions. We discuss several implications of this study for health IT, big data and predictive analytics, and design science research.

Cell tracking using deep neural networks with multi-task learning

Cell tracking plays crucial role in biomedical and computer vision areas. As cells generally have frequent deformation activities and small sizes in microscope image, tracking the non-rigid and non-significant cells is quite difficult in practice. Traditional visual tracking methods have good performances on tracking rigid and significant visual objects, however, they are not suitable for cell tracking problem. In this paper, a novel cell tracking method is proposed by using Convolutional Neural Networks (CNNs) as well as multi-task learning (MTL) techniques. The CNNs learn robust cell features and MTL improves the generalization performance of the tracking. The proposed cell tracking method consists of a particle filter motion model, a multi-task learning observation model, and an optimized model update strategy. In the training procedure, the cell tracking is divided into an online tracking task and an accompanying classification task using the MTL technique. The observation model is trained by building a CNN to learn robust cell features. The tracking procedure is started by assigning the cell position in the first frame of a microscope image sequence. Then, the particle filter model is applied to produce a set of candidate bounding boxes in the subsequent frames. The trained observation model provides the confidence probabilities corresponding to all of the candidates and selects the candidate with the highest probability as the final prediction. Finally, an optimized model update strategy is proposed to enable the multi-task observation model for the variation of the tracked cell over the entire tracking procedure. The performance and robustness of the proposed method are analyzed by comparing with other commonly-used methods. Experimental results demonstrate that the proposed method has good performance to the cell tracking problem.

An MIMLSVM algorithm based on ECC

In the multi-instance multi-label learning framework, an example is described by multiple instances and associated with multiple class labels at the same time. An idea of tackling with multi-instance multi-label problems is to identify its equivalence in the traditional supervised learning framework. However, some useful information such as the correlation between labels may be lost in the process of degeneration, which will influence the classification performance. In E-MIMLSVM+ algorithm, multi-task learning techniques are utilized to incorporate label correlations, while it is time consuming as well as memory consuming. Therefore, we propose an improved algorithm. In our algorithm, the classifier chains method is applied in E-MIMLSVM+ to incorporate label correlations instead of multi-task learning techniques. The experimental results show that the proposed algorithm can reduce time complexity and improve the predictive performance.

Multitask learning for host–pathogen protein interactions

Motivation: An important aspect of infectious disease research involves understanding the differences and commonalities in the infection mechanisms underlying various diseases. Systems biology-based approaches study infectious diseases by analyzing the interactions between the host species and the pathogen organisms. This work aims to combine the knowledge from experimental studies of host–pathogen interactions in several diseases to build stronger predictive models. Our approach is based on a formalism from machine learning called ‘multitask learning’, which considers the problem of building models across tasks that are related to each other. A ‘task’ in our scenario is the set of host–pathogen protein interactions involved in one disease. To integrate interactions from several tasks (i.e. diseases), our method exploits the similarity in the infection process across the diseases. In particular, we use the biological hypothesis that similar pathogens target the same critical biological processes in the host, in defining a common structure across the tasks.Results: Our current work on host–pathogen protein interaction prediction focuses on human as the host, and four bacterial species as pathogens. The multitask learning technique we develop uses a task-based regularization approach. We find that the resulting optimization problem is a difference of convex (DC) functions. To optimize, we implement a Convex–Concave procedure-based algorithm. We compare our integrative approach to baseline methods that build models on a single host–pathogen protein interaction dataset. Our results show that our approach outperforms the baselines on the training data. We further analyze the protein interaction predictions generated by the models, and find some interesting insights.Availability: The predictions and code are available at: http://www.cs.cmu.edu/∼mkshirsa/ismb2013_paper320.htmlContact: j.klein-seetharaman@warwick.ac.ukSupplementary information: Supplementary data are available at Bioinformatics online.

Learning output kernels for multi-task problems

Simultaneously solving multiple related learning tasks is beneficial under a variety of circumstances, but the prior knowledge necessary to properly model task relationships is rarely available in practice. In this paper, we develop a novel kernel-based multi-task learning technique that automatically reveals structural inter-task relationships. Building over the framework of output kernel learning (OKL), we introduce a method that jointly learns multiple functions and a low-rank multi-task kernel by solving a non-convex regularization problem. Optimization is carried out via a block coordinate descent strategy, where each subproblem is solved using suitable conjugate gradient (CG) type iterative methods for linear operator equations. The effectiveness of the proposed approach is demonstrated on pharmacological and collaborative filtering data.

Structured feature selection and task relationship inference for multi-task learning

Multi-task learning (MTL) aims to enhance the generalization performance of supervised regression or classification by learning multiple related tasks simultaneously. In this paper, we aim to extend the current MTL techniques to high dimensional data sets with structured input and structured output (SISO), where the SI means the input features are structured and the SO means the tasks are structured. We investigate a completely ignored problem in MTL with SISO data: the interplay of structured feature selection and task relationship modeling. We hypothesize that combining the structure information of features and task relationship inference enables us to build more accurate MTL models. Based on the hypothesis, we have designed an efficient learning algorithm, in which we utilize a task covariance matrix related to the model parameters to capture the task relationship. In addition, we design a regularization formulation for incorporating the structured input features in MTL. We have developed an efficient iterative optimization algorithm to solve the corresponding optimization problem. Our algorithm is based on the accelerated first order gradient method in conjunction with the projected gradient scheme. Using two real-world data sets, we demonstrate the utility of the proposed learning methods.

Modeling Disease Progression via Fused Sparse Group Lasso.

Alzheimer's Disease (AD) is the most common neurodegenerative disorder associated with aging. Understanding how the disease progresses and identifying related pathological biomarkers for the progression is of primary importance in the clinical diagnosis and prognosis of Alzheimer's disease. In this paper, we develop novel multi-task learning techniques to predict the disease progression measured by cognitive scores and select biomarkers predictive of the progression. In multi-task learning, the prediction of cognitive scores at each time point is considered as a task, and multiple prediction tasks at different time points are performed simultaneously to capture the temporal smoothness of the prediction models across different time points. Specifically, we propose a novel convex fused sparse group Lasso (cFSGL) formulation that allows the simultaneous selection of a common set of biomarkers for multiple time points and specific sets of biomarkers for different time points using the sparse group Lasso penalty and in the meantime incorporates the temporal smoothness using the fused Lasso penalty. The proposed formulation is challenging to solve due to the use of several non-smooth penalties. One of the main technical contributions of this paper is to show that the proximal operator associated with the proposed formulation exhibits a certain decomposition property and can be computed efficiently; thus cFSGL can be solved efficiently using the accelerated gradient method. To further improve the model, we propose two non-convex formulations to reduce the shrinkage bias inherent in the convex formulation. We employ the difference of convex (DC) programming technique to solve the non-convex formulations. We have performed extensive experiments using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). Results demonstrate the effectiveness of the proposed progression models in comparison with existing methods for disease progression. We also perform longitudinal stability selection to identify and analyze the temporal patterns of biomarkers in disease progression.

Intelligent Prediction of Turbulent Flow over Backward-Facing Step using Direct Numerical Simulation Data

In turbulent flows, there are very complicated nonlinear behaviors that are influenced by many indeterminate factors. Currently, Direct Numerical Simulation (DNS) plays a significant role in turbulent flow simulation modeling due to the improvement of computer technology, but it can still be too expensive and time-consuming. In this study, a multivariate Levenberg-Marquardt (LM) neural network is developed to predict the turbulent flow over backward-facing step using information from a DNS. A multi-task learning technique is used to improve generalization performance and the prediction accuracy of flow field properties with low computational cost. A novel approach is applied to determine the optimum number of hidden units in order to prevent overfitting. Radial Basis Function (RBF) networking is employed to evaluate the effectiveness of the proposed approach in predicting hidden units. The excellent performance of the Levenberg-Marquardt Artificial Neural Network is shown based on the estimation of the flow fields with a large amount of data, such as in the case of the backward-facing step flow. The results of the LM neural network demonstrate that the proposed method provides a practical guideline for the estimation of the number of hidden units essentially needed for Artificial Neural network (ANN) simulations.

Novel Approaches for the Prediction of Cancer Classification

In the past decade, DNA microarray technologies have had a great impact on cancer genome research; this technology has been viewed as a promising approach in predicting cancer classes and prognosis outcomes. In this paper, we proposed two systematic methods which can predict cancer classification. By applying the genetic algorithm gene selection (GAGS) method in order to find an optimal information gene subset, we avoid the over-fitting problem caused by attempting to apply a large number of genes to a small number of samples. By extracting significant samples (which we refer to as support vector samples because they are located only on support vectors), we allow the back propagation neural network (BPNN) to learn more tasks. We call this approach the multi-task support vector sample learning (MTSVSL) technique. We demonstrate experimentally that the GAGS and MTSVSL methods yield superior classification performance with application to leukemia and prostate cancer gene expression datasets. Our proposed GAGS and MTSVSL methods are novel approaches which are expedient and perform exceptionally well in cancer diagnosis and clinical outcome prediction.