Multi-task Learning Formulation Research Articles

Adaptive graph Laplacian MTL L1, L2 and LS-SVMs

Abstract Multi-Task Learning tries to improve the learning process of different tasks by solving them simultaneously. A popular Multi-Task Learning formulation for SVM is to combine common and task-specific parts. Other approaches rely on using a Graph Laplacian regularizer. Here we propose a combination of these two approaches that can be applied to L1, L2 and LS-SVMs. We also propose an algorithm to iteratively learn the graph adjacency matrix used in the Laplacian regularization. We test our proposal with synthetic and real problems, both in regression and classification settings. When the task structure is present, we show that our model is able to detect it, which leads to better results, and we also show it to be competitive even when this structure is not present.

Efficient multi-task learning with adaptive temporal structure for progression prediction

In this paper, we propose a novel efficient multi-task learning formulation for the class of progression problems in which its state will continuously change over time. To use the shared knowledge information between multiple tasks to improve performance, existing multi-task learning methods mainly focus on feature selection or optimizing the task relation structure. The feature selection methods usually fail to explore the complex relationship between tasks and thus have limited performance. The methods centring on optimizing the relation structure of tasks are not capable of selecting meaningful features and have a bi-convex objective function which results in high computation complexity of the associated optimization algorithm. Unlike these multi-task learning methods, motivated by a simple and direct idea that the state of a system at the current time point should be related to all previous time points, we first propose a novel relation structure, termed adaptive global temporal relation structure (AGTS). Then we integrate the widely used sparse group Lasso, fused Lasso with AGTS to propose a novel convex multi-task learning formulation that not only performs feature selection but also adaptively captures the global temporal task relatedness. Since the existence of three non-smooth penalties, the objective function is challenging to solve. We first design an optimization algorithm based on the alternating direction method of multipliers (ADMM). Considering that the worst-case convergence rate of ADMM is only sub-linear, we then devise an efficient algorithm based on the accelerated gradient method which has the optimal convergence rate among first-order methods. We show the proximal operator of several non-smooth penalties can be solved efficiently due to the special structure of our formulation. Experimental results on four real-world datasets demonstrate that our approach not only outperforms multiple baseline MTL methods in terms of effectiveness but also has high efficiency.

Towards Robust Visual Object Tracking with Independent Target-Agnostic Detection and Effective Siamese Cross-Task Interaction.

Advanced Siamese visual object tracking architectures are jointly trained using pair-wise input images to perform target classification and bounding box regression. They have achieved promising results in recent benchmarks and competitions. However, the existing methods suffer from two limitations: First, though the Siamese structure can estimate the target state in an instance frame, provided the target appearance does not deviate too much from the template, the detection of the target in an image cannot be guaranteed in the presence of severe appearance variations. Second, despite the classification and regression tasks sharing the same output from the backbone network, their specific modules and loss functions are invariably designed independently, without promoting any interaction. Yet, in a general tracking task, the centre classification and bounding box regression tasks are collaboratively working to estimate the final target location. To address the above issues, it is essential to perform target-agnostic detection so as to promote cross-task interactions in a Siamese-based tracking framework. In this work, we endow a novel network with a target-agnostic object detection module to complement the direct target inference, and to avoid or minimise the misalignment of the key cues of potential template-instance matches. To unify the multi-task learning formulation, we develop a cross-task interaction module to ensure consistent supervision of the classification and regression branches, improving the synergy of different branches. To eliminate potential inconsistencies that may arise within a multi-task architecture, we assign adaptive labels, rather than fixed hard labels, to supervise the network training more effectively. The experimental results obtained on several benchmarks, i.e., OTB100, UAV123, VOT2018, VOT2019, and LaSOT, demonstrate the effectiveness of the advanced target detection module, as well as the cross-task interaction, exhibiting superior tracking performance as compared with the state-of-the-art tracking methods.

SimVODIS++: Neural Semantic Visual Odometry in Dynamic Environments

Accurate estimation of 3D geometry and camera motion enables a wide range of tasks in robotics and autonomous vehicles. However, the lack of semantics and the performance degradation due to dynamic objects hinder its application to real-world scenarios. To overcome these limitations, we design a novel neural semantic visual odometry (VO) architecture on top of the simultaneous VO, object detection and instance segmentation (SimVODIS) network. Next, we propose an attentive pose estimation architecture with a multi-task learning formulation for handling dynamic objects and VO performance enhancement. The extensive experiments conducted in our work attest that the proposed SimVODIS++ improves the VO performance in dynamic environments. Further, SimVODIS++ focuses on salient regions while excluding feature-less regions. Performing the experiments, we have discovered and fixed the data leakage problem in the conventional experiment setting followed by numerous previous works—which we claim as one of our contributions. We make the source code public.

Multitask Deep Learning Reconstruction and Localization of Lesions in Limited Angle Diffuse Optical Tomography.

Diffuse optical tomography (DOT) leverages near-infrared light propagation through tissue to assess its optical properties and identify abnormalities. DOT image reconstruction is an ill-posed problem due to the highly scattered photons in the medium and the smaller number of measurements compared to the number of unknowns. Limited-angle DOT reduces probe complexity at the cost of increased reconstruction complexity. Reconstructions are thus commonly marred by artifacts and, as a result, it is difficult to obtain an accurate reconstruction of target objects, e.g., malignant lesions. Reconstruction does not always ensure good localization of small lesions. Furthermore, conventional optimization-based reconstruction methods are computationally expensive, rendering them too slow for real-time imaging applications. Our goal is to develop a fast and accurate image reconstruction method using deep learning, where multitask learning ensures accurate lesion localization in addition to improved reconstruction. We apply spatial-wise attention and a distance transform based loss function in a novel multitask learning formulation to improve localization and reconstruction compared to single-task optimized methods. Given the scarcity of real-world sensor-image pairs required for training supervised deep learning models, we leverage physics-based simulation to generate synthetic datasets and use a transfer learning module to align the sensor domain distribution between in silico and real-world data, while taking advantage of cross-domain learning. Applying our method, we find that we can reconstruct and localize lesions faithfully while allowing real-time reconstruction. We also demonstrate that the present algorithm can reconstruct multiple cancer lesions. The results demonstrate that multitask learning provides sharper and more accurate reconstruction.

Maximum Roaming Multi-Task Learning

Multi-task learning has gained popularity due to the advantages it provides with respect to resource usage and performance. Nonetheless, the joint optimization of parameters with respect to multiple tasks remains an active research topic. Sub-partitioning the parameters between different tasks has proven to be an efficient way to relax the optimization constraints over the shared weights, may the partitions be disjoint or overlapping. However, one drawback of this approach is that it can weaken the inductive bias generally set up by the joint task optimization. In this work, we present a novel way to partition the parameter space without weakening the inductive bias. Specifically, we propose Maximum Roaming, a method inspired by dropout that randomly varies the parameter partitioning, while forcing them to visit as many tasks as possible at a regulated frequency, so that the network fully adapts to each update. We study the properties of our method through experiments on a variety of visual multi-task data sets. Experimental results suggest that the regularization brought by roaming has more impact on performance than usual partitioning optimization strategies. The overall method is flexible, easily applicable, provides superior regularization and consistently achieves improved performances compared to recent multi-task learning formulations.

Joint Household Characteristic Prediction via Smart Meter Data

Predicting specific household characteristics (e.g., age of person, household income, cooking style, etc.) from their everyday electricity consumption (i.e., smart meter data) enables energy provider to develop many intelligent business applications or help consumers to reduce their energy consumption. However, most existing works intend to predict single household characteristic via smart meter data independently, and ignore the joint analysis of different characteristics. In this paper, we consider each characteristic as an independent task and intend to predict multiple household characteristics simultaneously by designing a new multi-task learning formulation: discriminative multi-task relationship learning (DisMTRL). Specifically, two main challenges need to be handled: 1) task relationship, that is the embedded structure of relationships among different characteristics and 2) feature learning, there exist redundant features in original training data. To achieve these, our DisMTRL model aims to obtain a simple but robust weight matrix through capturing the intrinsic relatedness among different characteristics by task covariance matrix (MTRL) and incorporating the discriminative features via feature covariance matrix (Dis). For model optimization, we employ an alternating minimization strategy to learn the optimal weight matrix as well as the relationship between tasks by converting feature learning regularization as trace minimization problem. For evaluation, we adopt a smart meter dataset collected from 4232 households in Ireland at a 30 min granularity over an interval of 1.5 years. The experimental results justify the effectiveness of our proposed model.

Correlation-Aware Sparse and Low-Rank Constrained Multi-Task Learning for Longitudinal Analysis of Alzheimer's Disease.

Alzheimer's disease (AD), as a severe neurodegenerative disease, is now attracting more and more researchers' attention in the healthcare. With the development of magnetic resonance imaging (MRI), the neuroimaging-based longitudinal analysis is gradually becoming an important research direction to understand and trace the process of the AD. In addition, regression analysis has been commonly adopted in the AD pattern analysis and progression prediction. However, most existing methods assume that all input features are equally related to the output variables, which ignore the difference in terms of the correlation. In this paper, we proposed a novel multi-task learning formulation, which considers a correlation-aware sparse and low-rank constrained regularization, for accurately predicting the cognitive scores of the patients at different time points and identifying the most predictive biomarkers. In addition, an efficient iterative algorithm is developed to optimize the proposed non-smooth convex objective formulation. We also have performed experiments using data from the AD neuroimaging initiative dataset to evaluate the proposed optimization formulation. Especially, we will predict cognitive scores of multiple time points through the baseline MRI features. The results not only indicate the rationality and correctness of the proposed method for predicting disease progression but also identify some stable and important MRI features that are consistent with the previous research.

Formula omitted] regularized nonlinear multi-task representation learning based cognitive performance prediction of Alzheimer’s disease

Alzheimer’s disease (AD) has been not only a substantial financial burden to the health care system but also the emotional hardship to patients and their families. Predicting cognitive performance of subjects from their magnetic resonance imaging (MRI) measures and identifying relevant imaging biomarkers are important research topics in the study of Alzheimer’s disease. Many previous works formulate the prediction task as a linear regression problem. The most critical limitation is that they assume a linear relationship between the MRI features and the cognitive outcomes. The linear models in original MRI feature spaces can be limited by their inability to exploit the nonlinear relation between the MRI features and cognitive measure prediction tasks. To better capture the complicated but more flexible relationship between the cognitive scores and the neuroimaging measures, we propose a ℓ2,1−ℓ1 norm regularized multi-kernel multi-task feature learning formulation with a joint sparsity inducing regularization. The formulation facilitates the shared kernel functions, as well as the high dimensional features in the kernel induced feature spaces simultaneously, to look for the common representation that are useful for all tasks by promoting use of few kernels and few learned features in each kernel. For optimization, we develop an alternating optimization method to effectively solve the proposed mixed norm regularized formulation. We evaluate the performance of the proposed method using the Alzheimer’s Disease Neuroimaging Initiative (ADNI) datasets and demonstrate that our proposed methods achieve not only clearly improved prediction performance for cognitive measurements with single MRI modality or multi-modalities data, but also a compact set of highly suggestive biomarkers relevant to AD.

Sparse shared structure based multi-task learning for MRI based cognitive performance prediction of Alzheimer’s disease

Alzheimer’s disease (AD), the most common form of dementia, not only causes progressive impairment of memory and other cognitive functions of patients, but also becomes the substantial financial burden to the health care system. There is thus an urgent need to (1) accurately predict the cognitive performance of the disease, and (2) identify potential MRI-related biomarkers most predictive of the estimation of cognitive outcomes. In this paper, we develop a novel multi-task learning formulation to explore the correlation existing in Magnetic Resonance Imaging (MRI) and cognitive measures by a mixed norm incorporating a hierarchical group sparsity and shared subspace uncovering regularization, to learn a shared structure from multiple related tasks with considering implicit shared subspace structure and explicit subset of features as well as Region-of-Interests (ROIs) simultaneously. An efficient alternating optimization algorithm is derived to solve the proposed non-convex and non-smooth objective formulation. We comprehensively evaluate the proposed algorithm for the cognitive outcome prediction including all subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset. The experimental results not only demonstrate the proposed method has superior performance over multiple state-of-the-art comparable approaches, but also identifies cognition-relevant MRI biomarkers that are consistent with prior knowledge.

Multitask Diffusion Adaptation Over Networks With Common Latent Representations

Online learning with streaming data in a distributed and collaborative manner can be useful in a wide range of applications. This topic has been receiving considerable attention in recent years with emphasis on both single-task and multitask scenarios. In single-task adaptation, agents cooperate to track an objective of common interest, while in multitask adaptation agents track multiple objectives simultaneously. Regularization is one useful technique to promote and exploit similarity among tasks in the latter scenario. This work examines an alternative way to model relations among tasks by assuming that they all share a common latent feature representation. As a result, a new multitask learning formulation is presented and algorithms are developed for its solution in a distributed online manner. We present a unified framework to analyze the mean-square-error performance of the adaptive strategies, and conduct simulations to illustrate the theoretical findings and potential applications.

Spatial Projection of Multiple Climate Variables Using Hierarchical Multitask Learning

Future projection of climate is typically obtained by combining outputs from multiple Earth System Models (ESMs) for several climate variables such as temperature and precipitation. While IPCC has traditionally used a simple model output average, recent work has illustrated potential advantages of using a multitask learning (MTL) framework for projections of individual climate variables. In this paper we introduce a framework for hierarchical multitask learning (HMTL) with two levels of tasks such that each super-task, i.e., task at the top level, is itself a multitask learning problem over sub-tasks. For climate projections, each super-task focuses on projections of specific climate variables spatially using an MTL formulation. For the proposed HMTL approach, a group lasso regularization is added to couple parameters across the super-tasks, which in the climate context helps exploit relationships among the behavior of different climate variables at a given spatial location. We show that some recent works on MTL based on learning task dependency structures can be viewed as special cases of HMTL. Experiments on synthetic and real climate data show that HMTL produces better results than decoupled MTL methods applied separately on the super-tasks and HMTL significantly outperforms baselines for climate projection.

Multi-Task Joint Sparse and Low-Rank Representation for the Scene Classification of High-Resolution Remote Sensing Image

Scene classification plays an important role in the intelligent processing of High-Resolution Satellite (HRS) remotely sensed images. In HRS image classification, multiple features, e.g., shape, color, and texture features, are employed to represent scenes from different perspectives. Accordingly, effective integration of multiple features always results in better performance compared to methods based on a single feature in the interpretation of HRS images. In this paper, we introduce a multi-task joint sparse and low-rank representation model to combine the strength of multiple features for HRS image interpretation. Specifically, a multi-task learning formulation is applied to simultaneously consider sparse and low-rank structures across multiple tasks. The proposed model is optimized as a non-smooth convex optimization problem using an accelerated proximal gradient method. Experiments on two public scene classification datasets demonstrate that the proposed method achieves remarkable performance and improves upon the state-of-art methods in respective applications.

Integrating gene set analysis and nonlinear predictive modeling of disease phenotypes using a Bayesian multitask formulation

BackgroundIdentifying molecular signatures of disease phenotypes is studied using two mainstream approaches: (i) Predictive modeling methods such as linear classification and regression algorithms are used to find signatures predictive of phenotypes from genomic data, which may not be robust due to limited sample size or highly correlated nature of genomic data. (ii) Gene set analysis methods are used to find gene sets on which phenotypes are linearly dependent by bringing prior biological knowledge into the analysis, which may not capture more complex nonlinear dependencies. Thus, formulating an integrated model of gene set analysis and nonlinear predictive modeling is of great practical importance.ResultsIn this study, we propose a Bayesian binary classification framework to integrate gene set analysis and nonlinear predictive modeling. We then generalize this formulation to multitask learning setting to model multiple related datasets conjointly. Our main novelty is the probabilistic nonlinear formulation that enables us to robustly capture nonlinear dependencies between genomic data and phenotype even with small sample sizes. We demonstrate the performance of our algorithms using repeated random subsampling validation experiments on two cancer and two tuberculosis datasets by predicting important disease phenotypes from genome-wide gene expression data.ConclusionsWe are able to obtain comparable or even better predictive performance than a baseline Bayesian nonlinear algorithm and to identify sparse sets of relevant genes and gene sets on all datasets. We also show that our multitask learning formulation enables us to further improve the generalization performance and to better understand biological processes behind disease phenotypes.

Multi-task learning for straggler avoiding predictive job scheduling

Parallel processing frameworks (Dean and Ghemawat, 2004) accelerate jobs by breaking them into tasks that execute in parallel. However, slow running or straggler tasks can run up to 8 times slower than the median task on a production cluster (Ananthanarayanan et al., 2013), leading to delayed job completion and inefficient use of resources. Existing straggler mitigation techniques wait to detect stragglers and then relaunch them, delaying straggler detection and wasting resources. We built Wrangler (Yadwadkar et al., 2014), a system that predicts when stragglers are going to occur and makes scheduling decisions to avoid such situations. To capture node and workload variability, Wrangler built separate models for every node and workload, requiring the time-consuming collection of substantial training data. In this paper, we propose multi-task learning formulations that share information between the various models, allowing us to use less training data and bring training time down from 4 hours to 40 minutes. Unlike naive multi-task learning formulations, our formulations capture the shared structure in our data, improving generalization performance on limited data. Finally, we extend these formulations using group sparsity inducing norms to automatically discover the similarities between tasks and improve interpretability.

Human action recognition via multi-task learning base on spatial–temporal feature

This study proposes a novel human action recognition method using regularized multi-task learning. First, we propose the part Bag-of-Words (PBoW) representation that completely represents the local visual characteristics of the human body structure. Each part can be viewed as a single task in a multi-task learning formulation. Further, we formulate the task of multi-view human action recognition as a learning problem penalized by a graph structure that is built according to the human body structure. Our experiments show that this method has significantly better performance in human action recognition than the standard Bag-of-Words+Support Vector Machine (BoW+SVM) method and other state-of-the-art methods. Further, the performance of the proposed method with simple global representation is as good as that of state-of-the-art methods for human action recognition on the TJU dataset (a new multi-view action dataset with RGB, depth, and skeleton data, which has been created by our group).

Multi-task learning with one-class SVM

Multi-task learning technologies have been developed to be an effective way to improve the generalization performance by training multiple related tasks simultaneously. The determination of the relatedness between tasks is usually the key to the formulation of a multi-task learning method. In this paper, we make the assumption that when tasks are related to each other, usually their models are close enough, that is, their models or their model parameters are close to a certain mean function. Following this task relatedness assumption, two multi-task learning formulations based on one-class support vector machines (one-class SVM) are presented. With the help of new kernel design, both multi-task learning methods can be solved by the optimization program of a single one-class SVM. Experiments conducted on both low-dimensional nonlinear toy dataset and high-dimensional textured images show that our approaches lead to very encouraging results.

Modeling disease progression via multi-task learning

Alzheimer's disease (AD), the most common type of dementia, is a severe neurodegenerative disorder. Identifying biomarkers that can track the progress of the disease has recently received increasing attentions in AD research. An accurate prediction of disease progression would facilitate optimal decision-making for clinicians and patients. A definitive diagnosis of AD requires autopsy confirmation, thus many clinical/cognitive measures including Mini Mental State Examination (MMSE) and Alzheimer's Disease Assessment Scale cognitive subscale (ADAS-Cog) have been designed to evaluate the cognitive status of the patients and used as important criteria for clinical diagnosis of probable AD. In this paper, we consider the problem of predicting disease progression measured by the cognitive scores and selecting biomarkers predictive of the progression. Specifically, we formulate the prediction problem as a multi-task regression problem by considering the prediction at each time point as a task and propose two novel multi-task learning formulations. We have performed extensive experiments using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). Specifically, we use the baseline MRI features to predict MMSE/ADAS-Cog scores in the next 4 years. Results demonstrate the effectiveness of the proposed multi-task learning formulations for disease progression in comparison with single-task learning algorithms including ridge regression and Lasso. We also perform longitudinal stability selection to identify and analyze the temporal patterns of biomarkers in disease progression. We observe that cortical thickness average of left middle temporal, cortical thickness average of left and right Entorhinal, and white matter volume of left Hippocampus play significant roles in predicting ADAS-Cog at all time points. We also observe that several MRI biomarkers provide significant information for predicting MMSE scores for the first 2 years, however very few are shown to be significant in predicting MMSE score at later stages. The lack of predictable MRI biomarkers in later stages may contribute to the lower prediction performance of MMSE than that of ADAS-Cog in our study and other related studies.

Learning Incoherent Sparse and Low-Rank Patterns from Multiple Tasks

We consider the problem of learning incoherent sparse and low-rank patterns from multiple tasks. Our approach is based on a linear multi-task learning formulation, in which the sparse and low-rank patterns are induced by a cardinality regularization term and a low-rank constraint, respectively. This formulation is non-convex; we convert it into its convex surrogate, which can be routinely solved via semidefinite programming for small-size problems. We propose to employ the general projected gradient scheme to efficiently solve such a convex surrogate; however, in the optimization formulation, the objective function is non-differentiable and the feasible domain is non-trivial. We present the procedures for computing the projected gradient and ensuring the global convergence of the projected gradient scheme. The computation of projected gradient involves a constrained optimization problem; we show that the optimal solution to such a problem can be obtained via solving an unconstrained optimization subproblem and an Euclidean projection subproblem. We also present two projected gradient algorithms and analyze their rates of convergence in details. In addition, we illustrate the use of the presented projected gradient algorithms for the proposed multi-task learning formulation using the least squares loss. Experimental results on a collection of real-world data sets demonstrate the effectiveness of the proposed multi-task learning formulation and the efficiency of the proposed projected gradient algorithms.