Multi-task Learning Problem Research Articles

SAIPy: A Python package for single-station earthquake monitoring using deep learning

Seismology has witnessed significant advancements in recent years with the application of deep learning methods to address a broad range of problems. These techniques have demonstrated their remarkable ability to effectively extract statistical properties from extensive datasets, surpassing the capabilities of traditional approaches to an extent. In this study, we present SAIPy, an open-source Python package specifically developed for fast seismic data processing by implementing deep learning. SAIPy offers solutions for multiple seismological tasks, including earthquake signal detection, seismic phase picking, first motion polarity identification and magnitude estimation. We introduce upgraded versions of previously published models such as CREIME_RT capable of identifying earthquakes with an accuracy above 99.8% and a root mean squared error of 0.38 unit in magnitude estimation. These upgraded models outperform state-of-the-art approaches like the Vision Transformer network. SAIPy provides an API that simplifies the integration of these advanced models, including CREIME_RT, DynaPicker_v2, and PolarCAP, along with benchmark datasets. It also, to the best of our knowledge, introduces the first fully automated deep learning based pipeline to process continuous waveforms. The package has the potential to be used for real-time earthquake monitoring to enable timely actions to mitigate the impact of seismic events. Ongoing development efforts aim to further enhance SAIPy’s performance and incorporate additional features that enhance exploration efforts, and it also would be interesting to approach the retraining of the whole package as a multi-task learning problem. A detailed description of all functions is available in a supplementary document.

Subtask-Aware Representation Learning for Predicting Antibiotic Resistance Gene Properties via Gating-Controlled Mechanism.

The crisis of antibiotic resistance has become a significant global threat to human health. Understanding properties of antibiotic resistance genes (ARGs) is the first step to mitigate this issue. Although many methods have been proposed for predicting properties of ARGs, most of these methods focus only on predicting antibiotic classes, while ignoring other properties of ARGs, such as resistance mechanisms and transferability. However, acquiring all of these properties of ARGs can help researchers gain a more comprehensive understanding of the essence of antibiotic resistance, which will facilitate the development of antibiotics. In this paper, the task of predicting properties of ARGs is modeled as a multi-task learning problem, and an effective subtask-aware representation learning-based framework is proposed accordingly. More specifically, property-specific expert networks and shared expert networks are utilized respectively to learn subtask-specific features for each subtask and shared features among different subtasks. In addition, a gating-controlled mechanism is employed to dynamically allocate weights to subtask-specific semantics and shared semantics obtained respectively from property-specific expert networks and shared expert networks, thus adjusting distinctive contributions of subtask-specific features and shared features to achieve optimal performance for each subtask simultaneously. Extensive experiments are conducted on publicly available data, and experimental results demonstrate the effectiveness of the proposed framework on the task of ARGs properties prediction.

Predicting Multi-dimensional Surgical Outcomes with Multi-modal Mobile Sensing

Pre-operative prediction of post-surgical recovery for patients is vital for clinical decision-making and personalized treatments, especially with lumbar spine surgery, where patients exhibit highly heterogeneous outcomes. Existing predictive tools mainly rely on traditional Patient-Reported Outcome Measures (PROMs), which fail to capture the long-term dynamics of patient conditions before the surgery. Moreover, existing studies focus on predicting a single surgical outcome. However, recovery from spine surgery is multi-dimensional, including multiple distinctive but interrelated outcomes, such as pain interference, physical function, and quality of recovery. In recent years, the emergence of smartphones and wearable devices has presented new opportunities to capture longitudinal and dynamic information regarding patients' conditions outside the hospital. This paper proposes a novel machine learning approach, Multi-Modal Multi-Task Learning (M3TL), using smartphones and wristbands to predict multiple surgical outcomes after lumbar spine surgeries. We formulate the prediction of pain interference, physical function, and quality of recovery as a multi-task learning (MTL) problem. We leverage multi-modal data to capture the static and dynamic characteristics of patients, including (1) traditional features from PROMs and Electronic Health Records (EHR), (2) Ecological Momentary Assessment (EMA) collected from smartphones, and (3) sensing data from wristbands. Moreover, we introduce new features derived from the correlation of EMA and wearable features measured within the same time frame, effectively enhancing predictive performance by capturing the interdependencies between the two data modalities. Our model interpretation uncovers the complementary nature of the different data modalities and their distinctive contributions toward multiple surgical outcomes. Furthermore, through individualized decision analysis, our model identifies personal high risk factors to aid clinical decision making and approach personalized treatments. In a clinical study involving 122 patients undergoing lumbar spine surgery, our M3TL model outperforms a diverse set of baseline methods in predictive performance, demonstrating the value of integrating multi-modal data and learning from multiple surgical outcomes. This work contributes to advancing personalized peri-operative care with accurate pre-operative predictions of multi-dimensional outcomes.

End-to-end learning of adaptive coded modulation schemes for resilient wireless communications

Adaptive modulation and coding schemes play a crucial role in ensuring robust data transfer in wireless communications, especially when faced with changes or interference in the transmission channel. These schemes involve the use of variable coding rates, which can be achieved normally through code puncturing or shortening, and have been adopted in 4G and 5G communication standards. In recent works, auto-encoders for wireless communications have demonstrated the ability to learn short code representations that achieve gains over conventional codes. Such a methodology is attractive as it can learn optimal representations under a variety of channel conditions. However, due to its structure the auto-encoder does not currently support multiple code rates with a single model. This article draws upon the discipline of multi-task learning, as it applies to deep learning and therefore devises a branching architecture for the auto-encoder and custom training algorithm in training transmitter and receiver for adaptive modulation and coding. In this article we aim to demonstrate improvements in Block Error Rate over conventional methods in the Additive White Gaussian Noise channel, and to analyse the performance of the model under Rayleigh fading channels without retraining the auto-encoder on the new channel. This article demonstrates a novel approach towards training auto-encoder models to jointly learn adaptive modulation and coding schemes framed as a multi-task learning problem. The research outcomes extend end-to-end learning approaches to the design of adaptive wireless communications systems.

Marginal Bayesian Posterior Inference using Recurrent Neural Networks with Application to Sequential Models.

In Bayesian data analysis, it is often important to evaluate quantiles of the posterior distribution of a parameter of interest (e.g., to form posterior intervals). In multi-dimensional problems, when non-conjugate priors are used, this is often difficult generally requiring either an analytic or sampling-based approximation, such as Markov chain Monte-Carlo (MCMC), Approximate Bayesian computation (ABC) or variational inference. We discuss a general approach that reframes this as a multi-task learning problem and uses recurrent deep neural networks (RNNs) to approximately evaluate posterior quantiles. As RNNs carry information along a sequence, this application is particularly useful in time-series. An advantage of this risk-minimization approach is that we do not need to sample from the posterior or calculate the likelihood. We illustrate the proposed approach in several examples.

A transformer-based deep learning approach for fairly predicting post-liver transplant risk factors

Liver transplantation is a life-saving procedure for patients with end-stage liver disease. There are two main challenges in liver transplant: finding the best matching patient for a donor and ensuring transplant equity among different subpopulations. The current MELD scoring system evaluates a patient's mortality risk if not receiving an organ within 90 days. However, the donor-patient matching should also consider post-transplant risk factors, such as cardiovascular disease, chronic rejection, etc., which are all common complications after transplant. Accurate prediction of these risk scores remains a significant challenge. In this study, we used predictive models to solve the above challenges. Specifically, we proposed a deep learning model to predict multiple risk factors after a liver transplant. By formulating it as a multi-task learning problem, the proposed deep neural network was trained to simultaneously predict the five post-transplant risks and achieve equal good performance by exploiting task-balancing techniques. We also proposed a novel fairness-achieving algorithm to ensure prediction fairness across different subpopulations. We used electronic health records of 160,360 liver transplant patients, including demographic information, clinical variables, and laboratory values, collected from the liver transplant records of the United States from 1987 to 2018. The model's performance was evaluated using various performance metrics such as AUROC and AUPRC. Our experiment results highlighted the success of our multi-task model in achieving task balance while maintaining accuracy. The model significantly reduced the task discrepancy by 39 %. Further application of the fairness-achieving algorithm substantially reduced fairness disparity among all sensitive attributes (gender, age group, and race/ethnicity) in each risk factor. It underlined the potency of integrating fairness considerations into the task-balancing framework, ensuring robust and fair predictions across multiple tasks and diverse demographic groups.

A Deep Multi-Task Learning Approach for Bioelectrical Signal Analysis

Deep learning is a promising technique for bioelectrical signal analysis, as it can automatically discover hidden features from raw data without substantial domain knowledge. However, training a deep neural network requires a vast amount of labeled samples. Additionally, a well-trained model may be sensitive to the study object, and its performance may deteriorate sharply when transferred to other study objects. We propose a deep multi-task learning approach for bioelectrical signal analysis to address these issues. Explicitly, we define two distinct scenarios, the consistent source-target scenario and the inconsistent source-target scenario based on the motivation and purpose of the tasks. For each scenario, we present methods to decompose the original task and dataset into multiple subtasks and sub-datasets. Correspondingly, we design the generic deep parameter-sharing neural networks to solve the multi-task learning problem and illustrate the details of implementation with one-dimension convolutional neural networks (1D CNN), vanilla recurrent neural networks (RNN), recurrent neural networks with long short-term memory units (LSTM), and recurrent neural networks with gated recurrent units (GRU). In these two scenarios, we conducted extensive experiments on four electrocardiogram (ECG) databases. The results demonstrate the benefits of our approach, showing that our proposed method can improve the accuracy of ECG data analysis (up to 5.2%) in the MIT-BIH arrhythmia database.

Kill Two Birds With One Stone: A Multi-View Multi-Adversarial Learning Approach for Joint Air Quality and Weather Prediction

Accurate and timely air quality and weather predictions are of great importance to urban governance and human livelihood. Though many efforts have been made for air quality or weather prediction, most of them simply employ one another as feature input, which ignores the inner-connection between two predictive tasks. On one hand, the accurate prediction of one task can help improve another task's performance. On the other hand, geospatially distributed air quality and weather monitoring stations provide additional hints for city-wide spatiotemporal dependency modeling. Inspired by the above two insights, in this paper, we propose a multi-view multi-adversarial approach, entitled MasterGNN <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{+}$</tex-math></inline-formula> , to jointly predict air quality and weather conditions. First, we devise a multi-view graph learning block to model spatial autocorrelation based on geographical distance and environmental context. Then, a dedicated evolved recurrent network is proposed to dynamically capture the long-range and independent temporal autocorrelation for each monitoring station and time slot. After that, we develop a multi-adversarial graph learning framework to against observation noise propagation introduced by spatiotemporal modeling. Moreover, we present an adaptive training strategy by formulating multi-adversarial learning as a multi-task learning problem. Finally, extensive experiments on two real-world datasets show that MasterGNN <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{+}$</tex-math></inline-formula> achieves the best performance compared with seven baselines on both air quality and weather prediction tasks.

Multilinear multitask learning by transformed tensor singular value decomposition

In this paper, we study the problem of multilinear multitask learning (MLMTL), in which all tasks are stacked into a third-order tensor for consideration. In contrast to conventional multitask learning, MLMTL can explore inherent correlations among multiple tasks in a better manner by utilizing multilinear low rank structure. Existing approaches about MLMTL are mainly based on the sum of singular values for approximating low rank matrices obtained by matricizing the third-order tensor. However, these methods are suboptimal in the Tucker rank approximation. In order to elucidate intrinsic correlations among multiple tasks, we present a new approach by the use of transformed tensor nuclear norm (TTNN) constraint in the objective function. The main advantage of the proposed approach is that it can acquire a low transformed multi-rank structure in a transformed tensor by applying suitable unitary transformations which is helpful to determine principal components in grouping multiple tasks for describing their intrinsic correlations more precisely. Furthermore, we establish an excess risk bound of the minimizer of the proposed TTNN approach. Experimental results including synthetic problems and real-world images, show that the mean-square errors of the proposed method is lower than those of the existing methods for different number of tasks and training samples in MLMTL.

Multi-Task Deep Learning Games: Investigating Nash Equilibria and Convergence Properties

This paper conducts a rigorous game-theoretic analysis on multi-task deep learning, providing mathematical insights into the dynamics and interactions of tasks within these models. Multi-task deep learning has attracted significant attention in recent years due to its ability to leverage shared representations across multiple correlated tasks, leading to improved generalization and reduced training time. However, understanding and examining the interactions between tasks within a multi-task deep learning system poses a considerable challenge. In this paper, we present a game-theoretic investigation of multi-task deep learning, focusing on the existence and convergence of Nash equilibria. Game theory provides a suitable framework for modeling the interactions among various tasks in a multi-task deep learning system, as it captures the strategic behavior of learning agents sharing a common set of parameters. Our primary contributions include: casting the multi-task deep learning problem as a game where each task acts as a player aiming to minimize its task-specific loss function; introducing the notion of a Nash equilibrium for the multi-task deep learning game; demonstrating the existence of at least one Nash equilibrium under specific convexity and Lipschitz continuity assumptions for the loss functions; examining the convergence characteristics of the Nash equilibrium; and providing a comprehensive analysis of the implications and limitations of our theoretical findings. We also discuss potential extensions and directions for future research in the multi-task deep learning landscape.

Multitask Feature Learning as Multiobjective Optimization: A New Genetic Programming Approach to Image Classification.

Feature learning is a promising approach to image classification. However, it is difficult due to high image variations. When the training data are small, it becomes even more challenging, due to the risk of overfitting. Multitask feature learning has shown the potential for improving generalization. However, existing methods are not effective for handling the case that multiple tasks are partially conflicting. Therefore, for the first time, this article proposes to solve a multitask feature learning problem as a multiobjective optimization problem by developing a genetic programming approach with a new representation to image classification. In the new approach, all the tasks share the same solution space and each solution is evaluated on multiple tasks so that the objectives of all the tasks can be optimized simultaneously using a single population. To learn effective features, a new and compact program representation is developed to allow the new approach to evolving solutions shared across tasks. The new approach can automatically find a diverse set of nondominated solutions that achieve good tradeoffs between different tasks. To further reduce the risk of overfitting, an ensemble is created by selecting nondominated solutions to solve each image classification task. The results show that the new approach significantly outperforms a large number of benchmark methods on six problems consisting of 15 image classification datasets of varying difficulty. Further analysis shows that these new designs are effective for improving the performance. The detailed analysis clearly reveals the benefits of solving multitask feature learning as multiobjective optimization in improving the generalization.

Episodic task agnostic contrastive training for multi-task learning

Learning knowledge from different tasks to improve the general learning performance is crucial for designing an efficient algorithm. In this work, we tackle the Multi-task Learning (MTL) problem, where the learner extracts the knowledge from different tasks simultaneously with limited data. Previous works have been designing the MTL models by taking advantage of the transfer learning techniques, requiring the knowledge of the task index, which is not realistic in many practical scenarios. In contrast, we consider the scenario that the task index is not explicitly known, under which the features extracted by the neural networks are task agnostic. To learn the task agnostic invariant features, we implement model agnostic meta-learning by leveraging the episodic training scheme to capture the common features across tasks. Apart from the episodic training scheme, we further implemented a contrastive learning objective to improve the feature compactness for a better prediction boundary in the embedding space. We conduct extensive experiments on several benchmarks compared with several recent strong baselines to demonstrate the effectiveness of the proposed method. The results showed that our method provides a practical solution for real-world scenarios, where the task index is agnostic to the learner and can outperform several strong baselines, achieving state-of-the-art performances.

Communication-Efficient Federated Multitask Learning Over Wireless Networks

This article investigates the scheduling framework of the federated multitask learning (FMTL) problem with a hard-cooperation structure over wireless networks, in which the scheduling becomes more challenging due to the different convergence behaviors of different tasks. Based on the special model structure, we propose a dynamic user and task scheduling scheme with a block-wise incremental gradient aggregation algorithm, in which the neural network model is decomposed into a common feature-extraction module and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> task-specific modules. Different block gradients with respect to different modules can be scheduled separately. We further propose a Lyapunov-drift-based scheduling scheme that minimizes the overall communication latency by utilizing both the instantaneous data importance and the channel state information. We prove that the proposed scheme can converge almost surely to a KKT solution of the training problem such that the data-distortion issue is resolved. Simulation results illustrate that the proposed scheme significantly reduces the communication latency compared to the state-of-the-art baseline schemes.

Improving therapeutic synergy score predictions with adverse effects using multi-task heterogeneous network learning.

Drug combinations could trigger pharmacological therapeutic effects (TEs) and adverse effects (AEs). Many computational methods have been developed to predict TEs, e.g. the therapeutic synergy scores of anti-cancer drug combinations, or AEs from drug-drug interactions. However, most of the methods treated the AEs and TEs predictions as two separate tasks, ignoring the potential mechanistic commonalities shared between them. Based on previous clinical observations, we hypothesized that by learning the shared mechanistic commonalities between AEs and TEs, we could learn the underlying MoAs (mechanisms of actions) and ultimately improve the accuracy of TE predictions. To test our hypothesis, we formulated the TE prediction problem as a multi-task heterogeneous network learning problem that performed TE and AE learning tasks simultaneously. To solve this problem, we proposed Muthene (multi-task heterogeneous network embedding) and evaluated it on our collected drug-drug interaction dataset with both TEs and AEs indications. Our experimental results showed that, by including the AE prediction as an auxiliary task, Muthene generated more accurate TE predictions than standard single-task learning methods, which supports our hypothesis. Using a drug pair Vincristine-Dasatinib as a case study, we demonstrated that our method not only provides a novel way of TE predictions but also helps us gain a deeper understanding of the MoAs of drug combinations.

Differentially private transferrable deep learning with membership-mappings

Despite a recent surge of research interest in privacy and transferrable deep learning, optimizing the tradeoff between privacy requirements and performance of machine learning models remains a challenge. This motivates the development of an approach that optimizes both privacy-preservation mechanism and learning of the deep models for achieving a robust performance. This paper considers the problem of semi-supervised transfer and multi-task learning under differential privacy framework. An alternative conception of deep autoencoder, referred to as Conditionally Deep Membership-Mapping Autoencoder (CDMMA), is considered for transferrable deep learning. Under practice-oriented settings, an analytical solution for the learning of CDMMA can be derived by means of variational optimization. The paper proposes a transfer and multi-task learning approach that combines CDMMA with a tailored noise adding mechanism to transfer knowledge from source to target domain in a privacy-preserving manner.

Person Tracking by Detection Using Dual Visible-Infrared Cameras

We study the problem of cross-modality person reidentification (ReID) and tracking with dual visible-infrared (VI) cameras, while most existing efforts on tracking-by-detection have been paid on single-modality visible ReID which is inapplicable for poor-light environments. The major difficulties for cross-modality (e.g., VI) ReID stem from the large modality gap between three-channel visible images and one-channel infrared images and such unknown environmental factors as background clutter, occlusions, etc. To tackle these issues, we propose to enrich the diversities of visible and infrared images for intra- and cross-modality matching by using both the channel-aware data augmentation (DA) techniques (e.g., channel exchanged augmentation and random occlusions) and standard DA techniques. On top of these DA techniques, we incorporate ResNet50 and vision transformer (ViT) into the feature extraction backbone network and apply the dynamic weight average (DWA) strategy for learning loss weights by regarding the minimization of identity loss and triplet loss as a multitask learning problem. We then apply the proposed ReID approach for person tracking in the field of interests. The experiments on two public data sets, i.e., RegDB and SYSU-MM01, show that our approach can improve the performance of state-of-the-art rank-1, mAP, and mINP for cross-modality matching. In addition, the experiments on our data set show that tracking by VI-ReID using dual VI cameras can achieve an accuracy of around 0.24 m.

Watching the BiG artifacts: Exposing DeepFake videos via Bi-granularity artifacts

Recent years have witnessed significant advances in AI-based face manipulation techniques, known as DeepFakes, which has brought severe threats to society. Hence, an emerging and increasingly important research topic is how to detect DeepFake videos. In this paper, we propose a new DeepFake detection method based on Bi-granularity artifacts (BiG-Arts). We observe that the most of DeepFake video generation can commonly introduce bi-granularity artifacts: the intrinsic-granularity artifacts and extrinsic-granularity artifacts. Specifically, the intrinsic-granularity artifacts are caused by a common series of operations in model generation such as up-convolution or up-sampling, while the extrinsic-granularity artifacts are introduced by a common step in post-processing that blends the synthesized face to original video. To this end, we formulate DeepFake detection as multi-task learning problem, to simultaneously predict the intrinsic and extrinsic artifacts. Benefiting from the guidance of detecting Bi-granularity artifacts, our method is notably boosted in both within-datasets and cross-datasets scenarios. Extensive experiments are conducted on several DeepFake datasets, which corroborates the superiority of our method. Our method has been contributed as a part of the solution to achieve the Top-1 rank in DFGC competition (https://competitions.codalab.org/competitions/29583).

Heuristic optimisation of multi-task dynamic architecture neural network (DAN2)

This article proposes a novel method to optimise the Dynamic Architecture Neural Network (DAN2) adapted for a multi-task learning problem. The multi-task learning neural network adopts a multi-head and serial architecture with DAN2 layers acting as the basic subroutine. Adopting a dynamic architecture, the layers are added consecutively starting from a minimal initial structure. The optimisation method adopts an iterative heuristic scheme that sequentially optimises the shared layers and the task-specific layers until the solver converges to a small tolerance. Application of the method has demonstrated the applicability of the algorithm to simulated datasets. Comparable results to Artificial Neural Networks (ANNs) have been obtained in terms of accuracy and speed.

Accelerated Fire Detection and Localization at Edge

Fire-related incidents continue to be reported as a leading cause of life and property destruction. Automated fire detection and localization (AFDL) systems have grown in importance with the evolution of applied robotics, especially because use of robots in disaster situations can lead to avoidance of human fatality. The importance of AFDL on resource-constrained devices has further grown, as most unmanned vehicles (drones or ground vehicles) are battery operated with limited computational capacity, the disaster situations cannot guarantee uninterrupted communication with high-end resources in the cloud, and yet faster response time is a prime necessity. Traditional computer vision–based techniques require hand-engineered features on a case-by-case basis. Deep Learning –based classifiers perform well for fire/no-fire classification due to the availability of large datasets for training; however, a dearth of good fire localization datasets renders the localization performance below par. We have tried to address both problems with a multi-task learned cascaded model that triggers localization workflow only if the presence of fire is detected, through a strong classifier trained on available large fire datasets. This presents only fire images to a relatively weaker localization model, reducing false positives, false negatives, and thereby improving overall AFDL accuracy. The multi-task learning (MTL) approach for end-to-end training of a stitched classifier and object localizer model on diverse datasets enabled us to build a strong fire classifier and feature extractor. It also resulted in a single unified model, capable of running on “on-board” compute infrastructure without compromising on accuracy. To achieve the target inference rate for the AFDL deployment, we have investigated the effect of quantization and compression due to hardware acceleration on an MTL model. This article presents an approach to automate the hardware-software co-design to find the optimum parameter partitioning for a given MTL problem, especially when some parts of the model are hardware accelerated. We present combined evaluation results showing that our methodology and the corresponding AFDL model strikes a balance between the frames inferred per second and several accuracy metrics. We report fire localization accuracy in terms of mean average precision (object detection), which has not been done earlier for embedded AFDL systems.

Multiagent Meta-Reinforcement Learning for Adaptive Multipath Routing Optimization.

In this article, we investigate the routing problem of packet networks through multiagent reinforcement learning (RL), which is a very challenging topic in distributed and autonomous networked systems. In specific, the routing problem is modeled as a networked multiagent partially observable Markov decision process (MDP). Since the MDP of a network node is not only affected by its neighboring nodes' policies but also the network traffic demand, it becomes a multitask learning problem. Inspired by recent success of RL and metalearning, we propose two novel model-free multiagent RL algorithms, named multiagent proximal policy optimization (MAPPO) and multiagent metaproximal policy optimization (meta-MAPPO), to optimize the network performances under fixed and time-varying traffic demand, respectively. A practicable distributed implementation framework is designed based on the separability of exploration and exploitation in training MAPPO. Compared with the existing routing optimization policies, our simulation results demonstrate the excellent performances of the proposed algorithms.