Regularized Multi-task Learning Research Articles

Bend-Net: Bending Loss Regularized Multitask Learning Network for Nuclei Segmentation in Histopathology Images

Bend-Net: Bending Loss Regularized Multitask Learning Network for Nuclei Segmentation in Histopathology Images

MTKSVCR: A novel multi-task multi-class support vector machine with safe acceleration rule

Regularized multi-task learning (RMTL) has shown good performance in tackling multi-task binary problems. Although RMTL can be used to handle multi-class problems based on “one-versus-one” and “one-versus-rest” techniques, the information of the samples is not fully utilized and the class imbalance problem occurs. Motivated by the regularization technique in RMTL, we propose an original multi-task multi-class model termed MTKSVCR based on “one-versus-one-versus-rest” strategy to achieve better testing accuracy. Due to the utilization of the idea of RMTL, the related information included in multiple tasks is mined by setting different penalty parameters before task-common and task-specific regularization terms. However, the proposed MTKSVCR is time-consuming since it employs all samples in each optimization problem. Therefore, a multi-parameter safe acceleration rule termed SA is further presented to reduce the time consumption. It identifies and deletes most of the superfluous samples corresponding to 0 elements in the dual optimal solution before solving. Then, only a reduced dual problem is to be solved and the computational efficiency is improved accordingly. The biggest advantage of the proposed SA lies in safety. Namely, it derives an identical optimal solution to the primal problem without SA. In addition, our method remains effective when multiple parameters change simultaneously. Experiments on different artificial datasets and benchmark datasets verify the validity of the proposed methods.

Efficacy of Regularized Multitask Learning Based on SVM Models.

This article investigates the efficacy of a regularized multitask learning (MTL) framework based on SVM (M-SVM) to answer whether MTL always provides reliable results and how MTL outperforms independent learning. We first find that the M-SVM is Bayes risk consistent in the limit of a large sample size. This implies that despite the task dissimilarities, the M-SVM always produces a reliable decision rule for each task in terms of the misclassification error when the data size is large enough. Furthermore, we find that the task-interaction vanishes as the data size goes to infinity, and the convergence rates of the M-SVM and its single-task counterpart have the same upper bound. The former suggests that the M-SVM cannot improve the limit classifier's performance; based on the latter, we conjecture that the optimal convergence rate is not improved when the task number is fixed. As a novel insight into MTL, our theoretical and experimental results achieved an excellent agreement that the benefit of the MTL methods lies in the improvement of the preconvergence-rate (PCR) factor (to be denoted in Section III) rather than the convergence rate. Moreover, this improvement of PCR factors is more significant when the data size is small. In addition, our experimental results of five other MTL methods demonstrate the generality of this new insight.

A unified analysis of convex and non‑convex ‑ball projection problems.

The task of projecting onto norm balls is ubiquitous in statistics and machine learning, yet the availability of actionable algorithms for doing so is largely limited to the special cases of . In this paper, we introduce novel, scalable methods for projecting onto the -ball for general . For , we solve the univariate Lagrangian dual via a dual Newton method. We then carefully design a bisection approach For , presenting theoretical and empirical evidence of zero or a small duality gap in the non-convex case. The success of our contributions is thoroughly assessed empirically, and applied to large-scale regularized multi-task learning and compressed sensing. The code implementing our methods is publicly available on Github.

Using Regularized Multi-Task Learning for Schizophrenia MRI Data Classification.

Machine learning techniques and magnetic resonance imaging methods have been widely used in computer-aided diagnosis and prognosis of severe brain diseases such as schizophrenia, Alzheimer, etc. Methods: In this paper, a regularized multi-task learning method for schizophrenia classification is proposed, and three MRI datasets of schizophrenia, collected from different data centers, are investigated. Firstly, slice extraction is used in image preprocessing. Then texture features of gray-level co-occurrence matrices are extracted from the above processed images. Finally, a p-norm regularized multi-task learning method is proposed to simultaneously learn the site-specific and site-shared features of the multi-site data, which can effectively discriminate schizophrenia patients from normal controls. The classification error rate on 10 datasets can be reduced from 10% to 30%. The proposed method obtains excellent results and provides objective evidence for clinical diagnosis and treatment of schizophrenia.

A Regularized Multi-Task Learning Approach for Cell Type Detection in Single-Cell RNA Sequencing Data.

Cell type prediction is one of the most challenging goals in single-cell RNA sequencing (scRNA-seq) data. Existing methods use unsupervised learning to identify signature genes in each cluster, followed by a literature survey to look up those genes for assigning cell types. However, finding potential marker genes in each cluster is cumbersome, which impedes the systematic analysis of single-cell RNA sequencing data. To address this challenge, we proposed a framework based on regularized multi-task learning (RMTL) that enables us to simultaneously learn the subpopulation associated with a particular cell type. Learning the structure of subpopulations is treated as a separate task in the multi-task learner. Regularization is used to modulate the multi-task model (e.g., W 1, W 2, … W t ) jointly, according to the specific prior. For validating our model, we trained it with reference data constructed from a single-cell RNA sequencing experiment and applied it to a query dataset. We also predicted completely independent data (the query dataset) from the reference data which are used for training. We have checked the efficacy of the proposed method by comparing it with other state-of-the-art techniques well known for cell type detection. Results revealed that the proposed method performed accurately in detecting the cell type in scRNA-seq data and thus can be utilized as a useful tool in the scRNA-seq pipeline.

A novel multi-task twin-hypersphere support vector machine for classification

Both regularized multi-task learning (RMTL) and direct multi-task twin support vector machine (DMTSVM) have shown good performances in dealing with multi-task problems. They all use hyperplanes to realize classification. However, the hyperplane cannot reflect the distribution of data well. Therefore, in this paper, we propose a novel multi-task twin hypersphere support vector machine (MTTHSVM) to solve multi-task classification problems. It will generate two hyperspheres rather than hyperplanes for each task. So, the proposed method could better describe the distribution information of all training samples compared with the existing RMTL and DMTSVM. Based on Hierarchical Bayes theory, MTTHSVM divides the center of each hypersphere into task-specific and task-common parts to better measure the commonality and individuality of tasks. Then the shared information contained in multiple related tasks could be adaptively mined well. Therefore, the prediction accuracy will be improved to some extent. Besides, MTTHSVM is superior to RMTL and DMTSVM in terms of computational efficiency. This is because our MTTHSVM just solves two smaller-sized quadratic programming problems without any matrix inverse operations. Experimental results on one artificial data set, thirty-five benchmark data sets and a real image data set Cifar100 have verified the effectiveness of our method.

Task Variance Regularized Multi-Task Learning

Multi-task Learning (MTL), which involves the simultaneous learning of multiple tasks, can achieve better performance than learning each task independently. It has achieved great success in various applications, ranging from Computer Vision (CV) to Natural Language Processing (NLP). In MTL, the losses of the including tasks are jointly optimized. However, it is common for these tasks to be competing. When the tasks are competing, minimizing the losses of some tasks increases the losses of others, which accordingly increases the task variance (variance between the task-specific loss); furthermore, it induces under-fitting in some tasks and over-fitting in others, which degenerates the generalization performance of an MTL model. To address this issue, it is necessary to control the task variance; thus, task variance regularization is a natural choice. While intuitive, task variance regularization remains unexplored in MTL. Accordingly, to fill this gap, we study the generalization error bound of MTL through the lens of task variance and propose the task variance matters the generalization performance of MTL. Furthermore, this paper investigates how the task variance might be effectively regularized, and consequently proposes a multi-task learning method based on adversarial multi-armed bandit. The proposed method, dubbed BanditMTL, regularizes the task variance by means of a mirror gradient ascent-descent algorithm. Adopting BanditMTL both in CV and NLP applications is found to achieve state-of-the-art performance. The results of extensive experiments back up our theoretical analysis and validate the superiority of our proposals.

Pan-cancer classification by regularized multi-task learning

Classifying pan-cancer samples using gene expression patterns is a crucial challenge for the accurate diagnosis and treatment of cancer patients. Machine learning algorithms have been considered proven tools to perform downstream analysis and capture the deviations in gene expression patterns across diversified diseases. In our present work, we have developed PC-RMTL, a pan-cancer classification model using regularized multi-task learning (RMTL) for classifying 21 cancer types and adjacent normal samples using RNASeq data obtained from TCGA. PC-RMTL is observed to outperform when compared with five state-of-the-art classification algorithms, viz. SVM with the linear kernel (SVM-Lin), SVM with radial basis function kernel (SVM-RBF), random forest (RF), k-nearest neighbours (kNN), and decision trees (DT). The PC-RMTL achieves 96.07% accuracy and 95.80% MCC score for a completely unknown independent test set. The only method that appears as the real competitor is SVM-Lin, which nearly equalizes the accuracy in prediction of PC-RMTL but only when complete feature sets are provided for training; otherwise, PC-RMTL outperformed all other classification models. To the best of our knowledge, this is a significant improvement over all the existing works in pan-cancer classification as they have failed to classify many cancer types from one another reliably. We have also compared gene expression patterns of the top discriminating genes across the cancers and performed their functional enrichment analysis that uncovers several interesting facts in distinguishing pan-cancer samples.

Trace-Norm Regularized Multi-Task Learning for Sea State Bias Estimation

Sea state bias (SSB) is an important component of errors for the radar altimeter measurements of sea surface height (SSH). However, existing SSB estimation methods are almost all based on single-task learning (STL), where one model is built on the data from only one radar altimeter. In this paper, taking account of the data from multiple radar altimeters available, we introduced a multi-task learning method, called trace-norm regularized multi-task learning (TNR-MTL), for SSB estimation. Corresponding to each individual task, TNR-MLT involves only three parameters. Hence, it is easy to implement. More importantly, the convergence of TNR-MLT is theoretically guaranteed. Compared with the commonly used STL models, TNR-MTL can effectively utilize the shared information between data from multiple altimeters. During the training of TNR-MTL, we used the JASON-2 and JASON-3 cycle data to solve two correlated SSB estimation tasks. Then the optimal model was selected to estimate SSB on the JASON-2 and the HY-2 70–71 cycle intersection data. For the JSAON-2 cycle intersection data, the corrected variance (M) has been reduced by 0.60 cm2 compared to the geophysical data records (GDR); while for the HY-2 cycle intersection data, M has been reduced by 1.30 cm2 compared to GDR. Therefore, TNR-MTL is proved to be effective for the SSB estimation tasks.

Safe sample screening for regularized multi-task learning

As a machine learning paradigm, multi-task learning (MTL) attracts increasing attention recently. It can improve the overall performance by exploiting the correlation among different tasks. It is especially helpful in dealing with small sample learning problems. As a classic multi-task learner, regularized multi-task learning (RMTL) inspired lots of multi-task learning researches in the past. Massive researches have shown the performance of RMTL when compared to single-task learners, i.e., support vector machine. However, the training complexity will be considerably large when training large datasets. To tackle such a problem, we propose safe screening rules for an improved regularized multi-task support vector machine (IRMTL). By statically detecting and removing inactive samples from multiple tasks simultaneously before solving the reduced optimization problem, both rules reduce the training time significantly without incurring performance degradation of the proposed method. The experimental results on 13 benchmark datasets and an image dataset also clearly demonstrate the effectiveness of safe screening rules for IRMTL.

Mobility-Aware Content Preference Learning in Decentralized Caching Networks

Due to the drastic increase of mobile traffic, wireless caching is proposed to serve repeated requests for content download. To determine the caching scheme for decentralized caching networks, the content preference learning problem based on mobility prediction is studied. We first formulate preference prediction as a decentralized regularized multi-task learning (DRMTL) problem without considering the mobility of mobile terminals (MTs). The problem is solved by a hybrid Jacobian and Gauss-Seidel proximal multi-block alternating direction method (ADMM) based algorithm, which is proven to conditionally converge to the optimal solution with a rate ${O}$ (1/ ${k}$ ). Then we use the tool of Markov renewal process to predict the moving path and sojourn time for MTs, and integrate the mobility pattern with the DRMTL model by reweighting the training samples and introducing a transfer penalty in the objective. We solve the problem and prove that the developed algorithm has the same convergence property but with different conditions. Through simulation we show the convergence analysis on proposed algorithms. Our real trace driven experiments illustrate that the mobility-aware DRMTL model can provide a more accurate prediction on geography preference than DRMTL model. Besides, the hit ratio achieved by most popular proactive caching (MPC) policy with preference predicted by mobility-aware DRMTL outperforms the MPC with preference from DRMTL and random caching (RC) schemes.

A Manifold Regularized Multi-Task Learning Model for IQ Prediction From Two fMRI Paradigms.

Multi-modal brain functional connectivity (FC) data have shown great potential for providing insights into individual variations in behavioral and cognitive traits. The joint learning of multi-modal data can utilize intrinsic association, and thus can boost learning performance. Although several multi-task based learning models have already been proposed by viewing feature learning on each modality as one task, most of them ignore the structural information inherent across the modalities, which may play an important role in extracting discriminative features. In this paper, we propose a new manifold regularized multi-task learning model by simultaneously considering between-subject and between-modality relationships. Specifically, the l2,1-norm (i.e., group-sparsity) regularizer is enforced to jointly select a few common features across different modalities. A novelly designed manifold regularizer is further imposed as a crucial underpinning to preserve the structural information both within and between modalities. Such designed regularizers will make our model more adaptive to realistic neuroimaging data, which are usually of small sample size but high dimensional features. Our model is validated on the Philadelphia Neurodevelopmental Cohort dataset, where our modalities are regarded as two types of functional MRI (fMRI) data collected under two paradigms. We conduct experimental studies on fMRI-based FC network data in two task conditions for intelligence quotient (IQ) prediction. The results show that our proposed model can not only achieve improved prediction performance, but also yield a set of IQ-relevant biomarkers. This paper develops a new multi-task learning model, enabling the discovery of significant biomarkers that may account for a proportion of the variance in human intelligence.

Evaluation of regularized multi-task leaning algorithms for single/multi-view human action recognition

Regularized multi-task learning (MTL) algorithms have been exploited in the field of pattern recognition and computer vision gradually, which can fully excavate the relationships of different related tasks. Therefore, many dramatically favorable approaches based on regularized MTL have been proposed. In the past decades, although the promising results about human action recognition have been achieved, most of existing action recognition algorithms focus on action descriptors, single/multi-view and multi-modality action recognition, and few works are related with MTL, especial of lacking the systematic evaluation of existing MTL algorithms for human action recognition. Thus, in the paper, seven popular regularized MTL algorithms in which different actions are considered as different tasks, are systematically exploited on two public multi-view action datasets. In detail, dense trajectory features are firstly extracted for each view, and then the shared codebook are constructed for all views by k-means, and then each video is coded by the shared codebook. Moreover, according to different regularized MTL algorithms, all actions or part of actions are considered as related, and then these actions are set to different tasks in MTL. Further, the effectiveness of different number of training samples from different action views is also evaluated for MTL. Large scale experimental results show that: 1) Regularized MTL is very useful for action recognition which can dig the latent relationship among different actions; 2) Not of all human actions are related, if irrelative actions are put together in MTL, its performance will fall; 3) With the increase of the training samples from different views, the relationships about different actions can be fully exploited, and it promotes the accuracy improvement of action recognition.

Human action recognition via multi-task learning base on spatial–temporal feature

This study proposes a novel human action recognition method using regularized multi-task learning. First, we propose the part Bag-of-Words (PBoW) representation that completely represents the local visual characteristics of the human body structure. Each part can be viewed as a single task in a multi-task learning formulation. Further, we formulate the task of multi-view human action recognition as a learning problem penalized by a graph structure that is built according to the human body structure. Our experiments show that this method has significantly better performance in human action recognition than the standard Bag-of-Words+Support Vector Machine (BoW+SVM) method and other state-of-the-art methods. Further, the performance of the proposed method with simple global representation is as good as that of state-of-the-art methods for human action recognition on the TJU dataset (a new multi-view action dataset with RGB, depth, and skeleton data, which has been created by our group).

Classifying Protein Sequences Using Regularized Multi-Task Learning.

Classification problems in which several learning tasks are organized hierarchically pose a special challenge because the hierarchical structure of the problems needs to be considered. Multi-task learning (MTL) provides a framework for dealing with such interrelated learning tasks. When two different hierarchical sources organize similar information, in principle, this combined knowledge can be exploited to further improve classification performance. We have studied this problem in the context of protein structure classification by integrating the learning process for two hierarchical protein structure classification database, SCOP and CATH. Our goal is to accurately predict whether a given protein belongs to a particular class in these hierarchies using only the amino acid sequences. We have utilized the recent developments in multi-task learning to solve the interrelated classification problems. We have also evaluated how the various relationships between tasks affect the classification performance. Our evaluations show that learning schemes in which both the classification databases are used outperform the schemes which utilize only one of them.

Multi-task least-squares support vector machines

There are often the underlying cross relatedness amongst multiple tasks, which is discarded directly by traditional single-task learning methods. Since multi-task learning can exploit these relatedness to further improve the performance, it has attracted extensive attention in many domains including multimedia. It has been shown through a meticulous empirical study that the generalization performance of Least-Squares Support Vector Machine (LS-SVM) is comparable to that of SVM. In order to generalize LS-SVM from single-task to multi-task learning, inspired by the regularized multi-task learning (RMTL), this study proposes a novel multi-task learning approach, multi-task LS-SVM (MTLS-SVM). Similar to LS-SVM, one only solves a convex linear system in the training phrase, too. What's more, we unify the classification and regression problems in an efficient training algorithm, which effectively employs the Krylow methods. Finally, experimental results on school and dermatology validate the effectiveness of the proposed approach.

Manifold Regularized Multitask Learning for Semi-Supervised Multilabel Image Classification

It is a significant challenge to classify images with multiple labels by using only a small number of labeled samples. One option is to learn a binary classifier for each label and use manifold regularization to improve the classification performance by exploring the underlying geometric structure of the data distribution. However, such an approach does not perform well in practice when images from multiple concepts are represented by high-dimensional visual features. Thus, manifold regularization is insufficient to control the model complexity. In this paper, we propose a manifold regularized multitask learning (MRMTL) algorithm. MRMTL learns a discriminative subspace shared by multiple classification tasks by exploiting the common structure of these tasks. It effectively controls the model complexity because different tasks limit one another's search volume, and the manifold regularization ensures that the functions in the shared hypothesis space are smooth along the data manifold. We conduct extensive experiments, on the PASCAL VOC'07 dataset with 20 classes and the MIR dataset with 38 classes, by comparing MRMTL with popular image classification algorithms. The results suggest that MRMTL is effective for image classification.