Multiple Related Tasks Research Articles

Multitask Learning for Classification Problem via New Tight Relaxation of Rank Minimization.

Multitask learning (MTL) is a joint learning paradigm, which fuses multiple related tasks together to achieve the better performance than single-task learning methods. It has been observed by many researchers that different tasks with certain similarities share a low-dimensional common yet latent subspace. In order to get the low-rank structure shared across tasks, trace norm has been used as a convex relaxation of the rank minimization problem. However, trace norm is not a tight approximation for the rank function. To address this important issue, we propose two novel regularization-based models to approximate the rank minimization problem by minimizing the k minimal singular values. For our new models, if the minimal singular values are suppressed to zeros, the rank would also be reduced. Compared with the standard trace norm, our new regularization-based models are the tighter approximations, which can help our models capture the low-dimensional subspace among multiple tasks better. Besides, it is an NP-hard problem to directly solve the exact rank minimization problem for our models. In this article, we proposed two simple but effective strategies to optimize our models, which tactically solves the exact rank minimization problem by setting a large penalizing parameter. Experimental results performed on synthetic and real-world benchmark datasets demonstrate that the proposed models have the ability of learning the low-rank structure shared across tasks and the better performance than other classical MTL methods.

Evolutionary Multitasking for Optimization Based on Generative Strategies

Evolutionary multitasking (EMT) is one of the emerging topics in evolutionary computation. EMT can solve multiple related optimization tasks simultaneously and enhance the optimization of each task via knowledge sharing among tasks. Many EMT algorithms have been proposed and achieved success in various problems, yet EMT for multi-objective optimization remains a big challenge. The existing multi-objective EMT algorithms tend to suffer from slow convergence and difficulty in generating high-quality knowledge. To alleviate these issues, this paper proposes a new EMT algorithm namely EMT-GS for multi-objective optimization based on two generative strategies. Particularly, generative adversarial networks and inertial differential evolution are introduced to generate transferable knowledge and offspring, respectively. A generative adversarial network is trained periodically for each source-target task pair, based on which helpful knowledge is generated from the source task and transferred to boost the solving of the target task. To accelerate the population convergence, the inertial differential evolution strategy is put forward to generate offspring in a promising direction according to the individuals from the previous generation and the transferred knowledge. The performance of EMT-GS is validated on three multitasking multi-objective benchmark problems. The experimental results highlight the excellent competitiveness of EMT-GS compared to other state-of-the-art multi-objective EMT algorithms.

Production quality prediction of multistage manufacturing systems using multi-task joint deep learning

A multistage manufacturing system with multiple manufacturing stages is the key and main production mode for enterprises to achieve lean production. Due to the variation propagation between stages and multiple related quality prediction tasks, it is difficult to accurately predict the quality of multistage manufacturing systems with multiple tasks. Traditional single stage and single task quality prediction methods permit the multi stages and multiple tasks separately, which ignore multi-stage effects or the quality-related relationship between multiple quality output indicators and reduce the efficiency of quality prediction. In this paper, a production quality prediction framework based on multi-task joint deep learning is proposed to simultaneously evaluate the multi-task quality of all stages in a multistage manufacturing system. To be specific, variation propagation cumulative impact between multiple manufacturing stages is innovatively expressed by designing a multi-scale convolutional neural network with control gates (MCNN-CG) to extract and propagate data features. Production quality with multi-tasks at all stages is jointly predicted by designing a multi-layer multi-gate mixture-of-experts multi-task (ML-MMoE) model with reducing multi-task predictive loss simultaneously. The soft parameter-sharing strategy and multi-gate attention strategy are separately designed to ensure information sharing while learning personalized features of tasks to improve quality prediction accuracy of each task. In addition, a loss function based on homoscedastic uncertainty and regularization is designed to automatically learn the weight between multi-stage and multi-task losses. Experiments on multistage assembly test data of an inertial navigation manufacturing system show that the proposed method performs better than traditional models. Compared to the single-stage model, the proposed multistage model has an average improvement of 8.5%, 20.0% and 23.3% in R2, MAE and RMSE respectively in the second stage. Compared with the traditional multi-stage model, the proposed model has an average improvement of 1.7%, 6.2% and 9.8% in R2, MAE and RMSE respectively in the second stage.

FedMPT: Federated Learning for Multiple Personalized Tasks Over Mobile Computing

Federated learning (FL) is a privacy-preserving collaborative learning framework that can be used in mobile computing where multiple user devices jointly train a deep learning model without uploading their data to a centralized server. An essential issue of FL is to reduce the significant communication overhead during training. Existing FL schemes mostly address this issue regarding single task learning. However, each user generally has multiple related tasks on the mobile device such as multi-content recommendation, and traditional FL schemes need to train an individual model per task which consumes a substantial number of resources. In this work, we formulate an FL problem with multiple personalized tasks, which aims to minimize the communication cost in learning different personalized tasks on each device. To solve the formulated problem, we incorporate multi-task learning into FL which trains a model for multiple tasks concurrently and propose an FL framework named FedMPT. FedMPT modifies the efficient acceleration algorithm and quantization compression strategy delicately to achieve superior performance regarding the communication efficiency. We implement and evaluate FedMPT on two datasets, Multi-MNIST and CelebA, in the FL environment. Experimental results show that FedMPT significantly outperforms the traditional FL scheme considering communication cost and average accuracy.

A Framework for Multi-Task Learning in Dynamic Adaptive Streaming Over HTTP

This paper presents a framework with a taxonomy for multi-task learning in the context of Dynamic Adaptive Streaming over HTTP (DASH). DASH is a widely used technology for video streaming, and multi-task learning has emerged as a promising approach to enhance the performance and user experience of DASH systems by jointly optimizing multiple related tasks. The framework provides a structured approach to design, train, and evaluate multi-task learning models in DASH, while the taxonomy categorizes the key components and approaches within the framework. The taxonomy includes task types, multitask learning approaches, input features, and training strategies. Task types encompass video quality adaptation, buffer management, bandwidth estimation, content pre-fetching, and resource allocation, representing the specific tasks involved in DASH. Multi-task learning approaches encompass methodologies such as shared representation learning, task-specific layers, multi-head architectures, knowledge distillation, and reinforcement learning, offering flexibility in model design and optimization. Input features cover video characteristics, network conditions, device capabilities, and user preferences, providing the necessary information for informed decision-making across tasks. Training strategies include joint training, alternate training, hierarchical training, task weighting, and task balancing, determining how the multi-task learning model is trained and optimized. By following the presented framework and taxonomy, researchers and practitioners can systematically approach the design, training, and evaluation of multi-task learning models in DASH. The framework enables the development of efficient and adaptive video streaming systems by leveraging the interdependencies among tasks. The taxonomy helps organize the components and approaches within the framework, aiding in a better understanding of the various aspects of multi-task learning in the DASH context. Overall, this framework and taxonomy provide a valuable resource for advancing the field of multitask learning in the dynamic and complex domain of video streaming over HTTP.

Fast forward approximation and multitask inversion of gravity anomaly based on UNet3+

SUMMARYGravity inversion is a process that obtains the spatial structure and physical properties of underground anomalies using surface collected gravity anomaly data. In recent years, the rapid development of deep learning (DL) has enabled the achievement of good results for gravity inversion methods based on DL. These methods aim to learn the mapping between geological models and gravity anomaly data by training a neural network with geological models as labels. However, using DL inversion requires generating a large amount of training data for each geological target and involves the forward calculation of the generated models, which inevitably consumes a large amount of time and storage space. To address this issue, we propose using a neural network to approximate the expensive forward computation with a fast evaluation alternative. After training, the network can reproduce gravity anomalies at any observation point. To evaluate the effectiveness of the forward model, we use the gravity anomalies predicted by the forward network for inversion network training. Additionally, to mitigate the problem of poor generalization of existing DL inversions, we propose using multitask learning. By learning multiple related tasks simultaneously, the generalization ability of the model improves, thus enhancing the performance of the main task. In this paper, a multitask UNet3+ network is proposed to realize anomaly bodies localization and density contrasts reconstruction simultaneously. Test results on the synthetic data set demonstrate that the gravity anomalies predicted by the forward network can be successfully inverted, and the multitask approach can predict subsurface geology more accurately than the single-task. To further illustrate the effectiveness of the algorithm, we apply this method to the inversion of the San Nicolas deposit in central Mexico.

Elaborate multi-task subspace learning with discrete group constraint

In multi-task learning (MTL), multiple related tasks can be learned simultaneously under the shared information to improve the generalization performance. However, most of MTL methods assume that all the learning tasks are related indeed and appropriate for joint learning. In some real situations, this assumption may not hold and further lead to the problem of negative transfer. Therefore, in this paper, we not only focus on researching the problem of robustly learning the common feature structure shared by tasks, but also discriminate with which tasks one task should share. By combining with the idea of subspace learning, we propose an elaborate multi-task subspace learning model (EMTSL) with discrete group structure constraint, which can cluster the learned tasks into a set of groups. By introducing the Schatten p-norm instead of trace norm, our model EMTSL can better approximate the low-rank constraint and also avoid the trivial solution. Furthermore, we design an efficient algorithm based on the re-weighted method to solve the proposed model. In addition, the convergence analysis of our algorithm is given in this paper. Experimental results on both synthetic and real-world datasets demonstrate the superiority of our method.

Batching of Tasks by Users of Pseudonymous Forums: Anonymity Compromise and Protection

There are a number of forums where people participate under pseudonyms. One example is peer review, where the identity of reviewers for any paper is confidential. When participating in these forums, people frequently engage in "batching": executing multiple related tasks (e.g., commenting on multiple papers) at nearly the same time. Our empirical analysis shows that batching is common in two applications we consider -- peer review and Wikipedia edits. In this paper, we identify and address the risk of deanonymization arising from linking batched tasks. To protect against linkage attacks, we take the approach of adding delay to the posting time of batched tasks. We first show that under some natural assumptions, no delay mechanism can provide a meaningful differential privacy guarantee. We therefore propose a "one-sided" formulation of differential privacy for protecting against linkage attacks. We design a mechanism that adds zero-inflated uniform delay to events and show it can preserve privacy. We prove that this noise distribution is in fact optimal in minimizing expected delay among mechanisms adding independent noise to each event, thereby establishing the Pareto frontier of the trade-off between the expected delay for batched and unbatched events. Finally, we conduct a series of experiments on Wikipedia and Bitcoin data that corroborate the practical utility of our algorithm in obfuscating batching without introducing onerous delay to a system.

Multi‐task deep neural networks for just‐in‐time software defect prediction on mobile apps

SummaryWith the development of smartphones, mobile applications play an irreplaceable role in our daily life, which characteristics often commit code changes to meet new requirements. This characteristic can introduce defects into the software. To provide immediate feedback to developers, previous researchers began to focus on just‐in‐time (JIT) software defect prediction techniques. JIT defect prediction aims to determine whether code commits will introduce defects into the software. It contains two scenarios, within‐project JIT defect prediction and cross‐project JIT defect prediction. Regardless of whether within‐project JIT defect prediction or cross‐project JIT defect prediction all need to have enough labeled data (within‐project JIT defect prediction assumes that have plenty of labeled data from the same project, while cross‐project JIT defect prediction assumes that have sufficient labeled data from source projects). However, in practice, both the source and target projects may only have limited labeled data. We propose the MTL‐DNN method based on multi‐task learning to solve this question. This method contains the data preprocessing layer, input layer, shared layers, task‐specific layers, and output layer. Where the common features of multiple related tasks are learned by sharing layers, and the unique features of each task are learned by the task‐specific layers. For verifying the effectiveness of the MTL‐DNN approach, we evaluate our method on 15 Android mobile apps. The experimental results show that our method significantly outperforms the state‐of‐the‐art single‐task deep learning and classical machine learning methods. This result shows that the MTL‐DNN method can effectively solve the problem of insufficient labeled training data for source and target projects.

A General Linguistic Steganalysis Framework Using Multi-Task Learning

Prevailing linguistic steganalysis approaches focus on learning sensitive features to distinguish a particular category of steganographic texts from non-steganographic texts, by performing binary classification. While it remains an unsolved problem and poses a significant threat to the security of cyberspace when various categories of non-steganographic or steganographic texts coexist. In this paper, we propose a general linguistic steganalysis framework named LS-MTL, which introduces the idea of multi-task learning to deal with the classification of various categories of steganographic and non-steganographic texts. LS-MTL captures sensitive linguistic features from multiple related linguistic steganalysis tasks and can concurrently handle diverse tasks with a constructed model. In the proposed framework, convolutional neural networks (CNNs) are utilized as private base models to extract sensitive features for each steganalysis task. Besides, a shared CNN is built to capture potential interaction information and share linguistic features among all tasks. Finally, LS-MTL incorporates the private and shared sensitive features to identify the detected text as steganographic or non-steganographic. Experimental results demonstrate that the proposed framework LS-MTL outperforms the baseline in the multi-category linguistic steganalysis task, while average Acc, Pre, and Rec are increased by 0.5%, 1.4%, and 0.4%, respectively. More ablation experimental results show that LS-MTL with the shared module has robust generalization capability and achieves good detection performance even in the case of spare data.

ALScA: A Framework for Using Auxiliary Learning Side-Channel Attacks to Model PUFs

Physical unclonable functions (PUFs) have emerged as potent hardware primitives owing to their intrinsic properties of being secret key-free, clone-proof, and lightweight. However, PUFs cannot avoid the threats of machine learning modeling and side-channel attacks (SCAs). Nevertheless, almost all attacks neglect the correlations between the mathematical model and side-channel models introduced by PUF internal parameters; thus, such attacks fail to exploit related data and struggle in modeling complex PUFs. To address this problem, we propose a framework for using auxiliary learning SCAs to model strong PUFs by learning multiple related tasks together. Side-channel information predictions are introduced as auxiliary tasks to facilitate the primary task of predicting response. The parameters hard for the primary task to learn can be shared by the auxiliary tasks that learn the same parameters more straightforwardly. Based on the proposed framework, we design a specific auxiliary learning power SCA that employs power level prediction as the auxiliary task. The proposed attack is implemented with the hard-parameter sharing and hierarchy sharing deep neural networks. Experimental results demonstrate that the proposed attack succeeds in modeling XOR APUF, MPUF, and iPUF and outperforms the state-of-the-art methods in modeling MPUF and iPUF. We evaluate the influences of task relatedness, architecture, and loss weight ratio. Furthermore, we propose a fine-grained classification-based method to generate the auxiliary task with an enhanced relationship to the primary task. According to the response, the class corresponding to a specific side-channel state is further divided into two subclasses. Experimental results demonstrate that the generated auxiliary task promotes performance and alleviates the adverse effects of improper architecture and parameters.

A Survey on Multi-Task Learning

Multi-Task Learning (MTL) is a learning paradigm in machine learning and its aim is to leverage useful information contained in multiple related tasks to help improve the generalization performance of all the tasks. In this paper, we give a survey for MTL from the perspective of algorithmic modeling, applications and theoretical analyses. For algorithmic modeling, we give a definition of MTL and then classify different MTL algorithms into five categories, including feature learning approach, low-rank approach, task clustering approach, task relation learning approach and decomposition approach as well as discussing the characteristics of each approach. In order to improve the performance of learning tasks further, MTL can be combined with other learning paradigms including semi-supervised learning, active learning, unsupervised learning, reinforcement learning, multi-view learning and graphical models. When the number of tasks is large or the data dimensionality is high, we review online, parallel and distributed MTL models as well as dimensionality reduction and feature hashing to reveal their computational and storage advantages. Many real-world applications use MTL to boost their performance and we review representative works in this paper. Finally, we present theoretical analyses and discuss several future directions for MTL.

A multi-objective multi-tasking evolutionary algorithm based inverse mapping and adaptive transformation strategy: IM-MFEA

Multi-tasking optimization algorithm attracts much attention because the knowledge transfer between tasks enables the algorithm to process multiple related tasks simultaneously. However, negative knowledge transfer occasionally occurs, which may weaken the performance of the algorithm. To reduce the impact of negative knowledge transfer, a multi-objective multi-tasking optimization algorithm (IM-MFEA) based on inverse model mapping and an objective transformation strategy is proposed. First, correlation analysis is applied in an inverse mapping strategy to improve the accuracy of the inverse mapping model. Then, following the pattern of using the source domain solutions to assist the optimization of the target domain, the adaptive transformation strategy is used to improve the quality of the source domain solution in the objective space. These transformed solutions are reconstructed through the inverse mapping strategy. Finally, these reconstructed source domain solutions are mated with the target domain solutions to generate competitive offspring individuals for the target domain. To verify the effectiveness of the IM-MFEA, comprehensive experiments were conducted on nine multi-objective multi-factorial optimization (MFO) benchmark problems. Empirical results demonstrate that IM-MFEA is superior to other algorithms in 90% of test instances by inverted generational distance (IGD) and hypervolume (HV) value indicators.

Local Learning-based Multi-task Clustering

Clustering plays an essential role in machine learning and data mining. Many real-world datasets for clustering are often different but related in the big data era. Recent research suggests that the individual performance of each learning task could be significantly improved by appropriately transferring knowledge among the related tasks. However, traditional clustering methods, which is limited to a single task, often ignore the correlation between multiple clustering tasks. Multi-task clustering (MTC) has attracted widespread interest recently by attempting to mine sufficient knowledge within multiple related tasks. Existing MTC methods still have the following limitations: 1) the intrinsic geometry of data is seldom considered; 2) the discriminative low-dimensional representation of data is not well explored; 3) the cluster structure of data is neglected in the process of clustering. In order to tackle the above issues, we propose a novel end-to-end Local Learning-based Multi-task Clustering (LLMC) method, which can simultaneously explore discriminative information in a low-dimensional subspace and expose the clustering results for multiple tasks. In particular, the proposed LLMC method can effectively integrate transfer learning, subspace learning, local manifold learning, and clustering. Specifically, a joint projection of heterogeneous features is introduced to control the number of features shared by all the tasks to transfer knowledge among tasks. An efficient iterative algorithm is designed to optimize the objective and is theoretically guaranteed to be convergent. Experimental results demonstrate the superiority of the proposed method over the state-of-the-art single-task and multi-task clustering methods.

Feature-Level Attention-Guided Multitask CNN for Fault Diagnosis and Working Conditions Identification of Rolling Bearing.

Accurate and real-time fault diagnosis (FD) and working conditions identification (WCI) are the key to ensuring the safe operation of mechanical systems. We observe that there is a close correlation between the fault condition and the working condition in the vibration signal. Most of the intelligent FD methods only learn some features from the vibration signals and then use them to identify fault categories. They ignore the impact of working conditions on the bearing system, and such a single-task learning method cannot learn the complementary information contained in multiple related tasks. Therefore, this article is devoted to mining richer and complementary globally shared features from vibration signals to complete the FD and WCI of rolling bearings at the same time. To this end, we propose a novel multitask attention convolutional neural network (MTA-CNN) that can automatically give feature-level attention to specific tasks. The MTA-CNN consists of a global feature shared network (GFS-network) for learning globally shared features and K task-specific networks with feature-level attention module (FLA-module). This architecture allows the FLA-module to automatically learn the features of specific tasks from globally shared features, thereby sharing information among different tasks. We evaluated our method on the wheelset bearing data set and motor bearing data set. The results show that our method has a better performance than the state-of-the-art deep learning methods and strongly prove that our multitask learning mechanism can improve the results of each task.

Active multitask learning with uncertainty-weighted loss for coronary calcium scoring.

The coronary artery calcification (CAC) score is an independent marker for the risk of cardiovascular events. Automatic methods for quantifying CAC could reduce workload and assist radiologists in clinical decision-making. However, large annotated datasets are needed for training to achieve very good model performance, which is an expensive process and requires expert knowledge. The number of training data required can be reduced in an active learning scenario, which requires only the most informative samples to be labeled. Multitask learning techniques can improve model performance by joint learning of multiple related tasks and extraction of shared informativefeatures. We propose an uncertainty-weighted multitask learning model for coronary calcium scoring in electrocardiogram-gated (ECG-gated), noncontrast-enhanced cardiac calcium scoring CT. The model was trained to solve the two tasks of coronary artery region segmentation (weak labels) and coronary artery calcification segmentation (strong labels) simultaneously in an active learning scenario to improve model performance and reduce the number of samples needed for training. We compared our model with a single-task U-Net and a sequential-task model as well as other state-of-the-art methods. The model was evaluated on 1275 individual patients in three different datasets (DISCHARGE, CADMAN, orCaScore), and the relationship between model performance and various influencing factors (image noise, metal artifacts, motion artifacts, image quality) wasanalyzed. Joint learning of multiclass coronary artery region segmentation and binary coronary calcium segmentation improved calcium scoring performance. Since shared information can be learned from both tasks for complementary purposes, the model reached optimal performance with only 12% of the training data and one-third of the labeling time in an active learning scenario. We identified image noise as one of the most important factors influencing model performance along with anatomical abnormalities and metalartifacts. Our multitask learning approach with uncertainty-weighted loss improves calcium scoring performance by joint learning of shared features and reduces labeling costs when trained in an active learningscenario.

Multi-Task Learning Based on Stochastic Configuration Neural Networks.

When the human brain learns multiple related or continuous tasks, it will produce knowledge sharing and transfer. Thus, fast and effective task learning can be realized. This idea leads to multi-task learning. The key of multi-task learning is to find the correlation between tasks and establish a fast and effective model based on these relationship information. This paper proposes a multi-task learning framework based on stochastic configuration neural networks. It organically combines the idea of the classical parameter sharing multi-task learning with that of constraint sharing configuration in stochastic configuration neural networks. Moreover, it provides an efficient multi-kernel function selection mechanism. The convergence of the proposed algorithm is proved theoretically. The experiment results on one simulation data set and four real life data sets verify the effectiveness of the proposed algorithm.

Multitask Learning Based on Least Squares Support Vector Regression for Stock Forecast

Various factors make stock market forecasting difficult and arduous. Single-task learning models fail to achieve good results because they ignore the correlation between multiple related tasks. Multitask learning methods can capture the cross-correlation among subtasks and achieve a satisfactory learning effect by training all tasks simultaneously. With this motivation, we assume that the related tasks are close enough to share a common model whereas having their own independent models. Based on this hypothesis, we propose a multitask learning least squares support vector regression (MTL-LS-SVR) algorithm, and an extension, EMTL-LS-SVR. Theoretical analysis shows that these models can be converted to linear systems. A Krylov-Cholesky algorithm is introduced to determine the optimal solutions of the models. We tested the proposed models by applying them to forecasts of the Chinese stock market index trend and the stock prices of five stated-owned banks. The experimental results demonstrate their validity.

A novel bundling learning paradigm for named entity recognition

Multi-task learning (MTL) takes advantage of the information gained from multiple related NLP tasks in order to improve performance across these tasks. MTL-based models for named entity recognition (NER) have traditionally included relation extraction and (or) coreference resolution, which requires additional data annotations in NER corpora, whereas these annotations are often unavailable. Indeed, we generally model the NER task using either a sequence labeling-based or span-based approach. Motivated by MTL, we propose a novel Bundling Learning (BL) paradigm for the NER task, which is achieved by bundling sequence labeling-based and span-based NER models together, thus allowing us to model the task from both token- and span-level perspectives. In addition, BL does not require additional data annotations compared to MTL. In experiments on NER and RE tasks, it is shown that BL consistently improves the performance of the two tasks across several benchmark datasets. Detailed analyses further confirm the effectiveness of BL.

A novel multi-task twin-hypersphere support vector machine for classification

Both regularized multi-task learning (RMTL) and direct multi-task twin support vector machine (DMTSVM) have shown good performances in dealing with multi-task problems. They all use hyperplanes to realize classification. However, the hyperplane cannot reflect the distribution of data well. Therefore, in this paper, we propose a novel multi-task twin hypersphere support vector machine (MTTHSVM) to solve multi-task classification problems. It will generate two hyperspheres rather than hyperplanes for each task. So, the proposed method could better describe the distribution information of all training samples compared with the existing RMTL and DMTSVM. Based on Hierarchical Bayes theory, MTTHSVM divides the center of each hypersphere into task-specific and task-common parts to better measure the commonality and individuality of tasks. Then the shared information contained in multiple related tasks could be adaptively mined well. Therefore, the prediction accuracy will be improved to some extent. Besides, MTTHSVM is superior to RMTL and DMTSVM in terms of computational efficiency. This is because our MTTHSVM just solves two smaller-sized quadratic programming problems without any matrix inverse operations. Experimental results on one artificial data set, thirty-five benchmark data sets and a real image data set Cifar100 have verified the effectiveness of our method.