Multi-task Framework Research Articles

Long-short diffeomorphism memory network for weakly-supervised ultrasound landmark tracking

Ultrasound is a promising medical imaging modality benefiting from low-cost and real-time acquisition. Accurate tracking of an anatomical landmark has been of high interest for various clinical workflows such as minimally invasive surgery and ultrasound-guided radiation therapy. However, tracking an anatomical landmark accurately in ultrasound video is very challenging, due to landmark deformation, visual ambiguity and partial observation. In this paper, we propose a long-short diffeomorphism memory network (LSDM), which is a multi-task framework with an auxiliary learnable deformation prior to supporting accurate landmark tracking. Specifically, we design a novel diffeomorphic representation, which contains both long and short temporal information stored in separate memory banks for delineating motion margins and reducing cumulative errors. We further propose an expectation maximization memory alignment (EMMA) algorithm to iteratively optimize both the long and short deformation memory, updating the memory queue for mitigating local anatomical ambiguity. The proposed multi-task system can be trained in a weakly-supervised manner, which only requires few landmark annotations for tracking and zero annotation for deformation learning. We conduct extensive experiments on both public and private ultrasound landmark tracking datasets. Experimental results show that LSDM can achieve better or competitive landmark tracking performance with a strong generalization capability across different scanner types and different ultrasound modalities, compared with other state-of-the-art methods.

A multi-task learning model for recommendation based on fusion of dynamic and static neighbors

To improve recommendation performance, this study introduces self-supervised learning into recommendation systems and proposes a multi-task learning recommendation framework that combines static neighbor and dynamic neighbor contrastive learning. Specifically, this study considers node relationships at both the graph and embedding levels, which can be defined in two aspects: (1) Static neighbors, which are positive nodes obtained by integrating information from user-item interaction graphs and social graphs. (2) Dynamic neighbors refer to nodes at the encoding level that are similar, but not necessarily the same, at each iteration; hence, they are termed dynamic neighbors. By employing static neighbor contrastive learning and dynamic neighbor contrastive learning as auxiliary tasks for the main recommendation task, we optimized the user and item embeddings. Through extensive experiments using multiple real datasets, we validate the effectiveness of the proposed approach and its components.

CAM based fine-grained spatial feature supervision for hierarchical yoga pose classification using multi-stage transfer learning

In this paper, we propose a technique for hierarchical yoga pose classification (YPC) in a multi-stage multi-tasking framework. We propose a three-stage transfer learning based end-to-end training methodology. Novelty lies in (a) proposed supervised contrastive combined loss function for stage-1 training, (b) proposed Encoder–Decoder network architecture with attention mechanism for stage-3 training, (c) proposed spatial context aware multi-tasking combined loss function for stage-3 training. Firstly, for stage-1 training, we propose the usage of linear combination of three loss functions: cross-entropy, self-supervised contrastive loss and supervised contrastive loss in a multi-tasking manner. We introduce radial and cosine margin into the formulation of self-supervised and supervised contrastive loss to pull feature embeddings of same class closer together compared to feature embeddings of different classes. Weights learned over stage-1 training are subsequently fine-tuned over cross-entropy multi-tasking loss in stage-2. These stage-2 weights are transfer learned and are further fine-tuned in stage-3 training. For stage-3 training, we propose the usage of spatial context aware multi-tasking combined loss function. This loss function leverages on the fine-grained spatial features obtained from HiResCAM. These are processed in parallel with features obtained from XGrad-CAM (sensitivity and conservation axioms satisfying features) to further supervise the learning of hierarchical yoga pose classifier. We exemplify our methodology on the publicly available Yoga-82 large-scale dataset. We report peak Top-1 YPC accuracy of 95.89% over 6 pose classes (Yoga-6), 93.85% over 20 pose classes (Yoga-20) and 90.0% over 82 pose classes (Yoga-82). Our proposed method achieves 6.1% improvement over Top-1 classification accuracy in Yoga-6 hierarchy, 9.3% improvement in Yoga-20 hierarchy and 10.9% improvement in Yoga-82 hierarchy in comparison with state-of-the-art (SOTA) methodology. We achieve the current best Top-1 classification accuracies in all the three YPC hierarchies.

Multi-Task Learning for Design Under Uncertainty With Multi-Fidelity Partially Observed Information

Abstract The assessment of system performance and identification of failure mechanisms in complex engineering systems often requires the use of computation-intensive finite element software or physical experiments, which are both costly and time-consuming. Moreover, when accounting for uncertainties in the manufacturing process, material properties, and loading conditions, the process of reliability-based design optimization (RBDO) for complex engineering systems necessitates the repeated execution of expensive tasks throughout the optimization process. To address this problem, this paper proposes a novel methodology for RBDO. First, a multi-fidelity surrogate modeling strategy is presented, leveraging partially observed information (POI) from diverse sources with varying fidelity and dimensionality to reduce computational cost associated with evaluating expensive high-dimensional complex systems. Second, a multi-task surrogate modeling framework is proposed to address the concurrent evaluation of multiple constraints for each design point. The multi-task framework aids in the development of surrogate models and enhances the effectiveness of reliability analysis and design optimization. The proposed multi-fidelity multi-task machine learning model utilizes a Bayesian framework, which significantly improves the performance of the predictive model and provides uncertainty quantification of the prediction. Additionally, the model provides a highly accurate and efficient framework for reliability-based design optimization through knowledge sharing. The proposed method was applied to two design case studies. By incorporating POI from various sources, the proposed approach improves the accuracy and efficiency of system performance prediction, while simultaneously addressing the cost and complexity associated with the design of complex systems.

Multi-task global optimization-based method for vascular landmark detection

Vascular landmark detection plays an important role in medical analysis and clinical treatment. However, due to the complex topology and similar local appearance around landmarks, the popular heatmap regression based methods always suffer from the landmark confusion problem. Vascular landmarks are connected by vascular segments and have special spatial correlations, which can be utilized for performance improvement. In this paper, we propose a multi-task global optimization-based framework for accurate and automatic vascular landmark detection. A multi-task deep learning network is exploited to accomplish landmark heatmap regression, vascular semantic segmentation, and orientation field regression simultaneously. The two auxiliary objectives are highly correlated with the heatmap regression task and help the network incorporate the structural prior knowledge. During inference, instead of performing a max-voting strategy, we propose a global optimization-based post-processing method for final landmark decision. The spatial relationships between neighboring landmarks are utilized explicitly to tackle the landmark confusion problem. We evaluated our method on a cerebral MRA dataset with 564 volumes, a cerebral CTA dataset with 510 volumes, and an aorta CTA dataset with 50 volumes. The experiments demonstrate that the proposed method is effective for vascular landmark localization and achieves state-of-the-art performance.

Federated Multitask Learning for Complaint Identification Using Graph Attention Network

Prior study on automatically identifying complaints on social media has relied on extensive feature engineering in centralized settings, with no consideration for the decentralized, non-identically independently distributed (Non-IID), and privacy-conscious aspect of complaints, which can hinder data collection, distribution, and learning. In this work, we propose a Graph Attention Network (GAT) based multi-task framework that intends to learn two closely related tasks, complaint detection (primary task) and sentiment classification (auxiliary task), simultaneously in federated-learning scenarios. We propose the Federated Combination (FedComb) algorithm, a two-sided adaptive optimization technique that simultaneously optimizes global and local models. The proposed methodology outperforms several baselines for the intended task of recognizing complaints in decentralized settings, according to quantitative and qualitative studies on two benchmark datasets. The resources are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/appy1608/IEEE-TAI_FedCI_GAT</uri> .

Short-term traffic flow prediction at isolated intersections based on parallel multi-task learning

This paper proposes a novel phase-based short-term traffic flow prediction method based on parallel multi-task learning for isolated intersections. Different from traditional short-term traffic flow prediction methods, we take the traffic flow of each phase as the minimum prediction unit, instead of directly utilising the traffic flow of a single lane with large random fluctuations. Meanwhile, we design a novel encoding and decoding structure whereby external influencing factors have been incorporated both into encoding and decoding operations. Furthermore, a fusion strategy is proposed to predict the traffic flow of each phase by integrating the traffic flows of lanes whose right of way are provided by the phase. In the fusion strategy, we develop a parallel multi-task prediction framework whereby a new loss function is defined to improve the prediction accuracy. Finally, the proposed method is tested with the traffic flow data collected from an intersection of South Changsheng Road located in the city of Jiaxing. The findings illustrate that the proposed method can achieve better prediction results at different sampling time scales, compared to the existing short-term traffic flow prediction methods.

An efficient multi-task learning CNN for driver attention monitoring

Driver Monitoring System (DMS), usually equipped with a camera, is an emerging vehicle safety system that can monitor driver attentiveness and trigger timely alarms when signs of inattention are detected. Since a single indicator (e.g., eye blink rate) is insufficient and unreliable to analyze driver attentiveness, almost all existing solutions train several independent models to identify driver facial states, such as face landmark, head pose, yawning, eye state, etc. However, apart from neglecting the inherent correlations between these related tasks, multiple models also raise challenges for vehicle safety-critical systems (e.g., hardware resources, software compatibility, and real-time response). In this paper, we propose a multi-task learning CNN framework (DANet) to unify the relevant tasks into one model and simultaneously output various driver facial states. By sharing the common features and parameters of highly related tasks, DANet avoids repetitive computations and mitigates single task overfitting. More importantly, the model provides a comprehensive overview of facial states while maintaining low complexity. We also propose two novel designs: (1) Dual-loss Block, which decomposes the pose estimation task into pose classification and coarse-to-fine regression; (2) Head Pose Penalization, which constrains the network to predict gaze direction based on predicted head pose. Our method achieves compelling results in both speed and accuracy on a vehicle computing platform, marking a momentous step in this field.

Constrained Multiobjective Optimization via Multitasking and Knowledge Transfer

Solving constrained multi-objective optimization problems (CMOPs) with various features and challenges via evolutionary algorithms is very popular. Existing methods usually adopt an additional helper problem to simplify and solve them by divide-and-conquer. This paper proposes a new multitasking framework for CMOPs, borrowing the idea of evolutionary multitasking optimization. The main contributions are i) a multitasking framework is proposed, where a CMOP is modeled as a multitasking optimization problem with three tasks. Then, it is solved by constraint-first, constraint-ignored, and constraint-relaxed multi-objective evolutionary algorithms; ii) a knowledge expression and a transfer strategy are devised to transfer the knowledge among the tasks; and iii) based on the proposed framework, a new two-stage algorithm is presented to solve CMOPs. The effectiveness of our approach is validated through experiments on four CMOP benchmark suites and 19 real-world CMOPs.

An Emotion-Aware Multitask Approach to Fake News and Rumor Detection Using Transfer Learning

Social networking sites, blogs, and online articles are instant sources of news for internet users globally. However, in the absence of strict regulations mandating the genuineness of every text on social media, it is probable that some of these texts are fake news or rumors. Their deceptive nature and ability to propagate instantly can have an adverse effect on society. This necessitates the need for more effective detection of fake news and rumors on the web. In this work, we annotate four fake news detection (EFN) and rumor detection datasets with their emotion class labels using transfer learning. We show the correlation between the legitimacy of a text with its intrinsic emotion for fake news and rumor detection, and prove that even within the same emotion class, fake and real news are often represented differently, which can be used for improved feature extraction. Based on this, we propose a multitask framework for fake news and rumor detection, predicting both the emotion and legitimacy of the text. We train a variety of deep learning models in single-task (STL) and multitask settings for a more comprehensive comparison. We further analyze the performance of our multitask approach for EFN in cross-domain settings to verify its efficacy for better generalization across datasets, and to verify that emotions act as a domain-independent feature. Experimental results verify that our multitask models consistently outperform their STL counterparts in terms of accuracy, precision, recall, and F1 score, both for in-domain and cross-domain settings. We also qualitatively analyze the difference in performance in STL and multitask learning (MTL) models.

Scaling Multiobjective Evolution to Large Data With Minions: A Bayes-Informed Multitask Approach.

In an era of pervasive digitalization, the growing volume and variety of data streams poses a new challenge to the efficient running of data-driven optimization algorithms. Targeting scalable multiobjective evolution under large-instance data, this article proposes the general idea of using subsampled small-data tasks as helpful minions (i.e., auxiliary source tasks) to quickly optimize for large datasets-via an evolutionary multitasking framework. Within this framework, a novel computational resource allocation strategy is designed to enable the effective utilization of the minions while guarding against harmful negative transfers. To this end, an intertask empirical correlation measure is defined and approximated via Bayes' rule, which is then used to allocate resources online in proportion to the inferred degree of source-target correlation. In the experiments, the performance of the proposed algorithm is verified on: 1) sample average approximations of benchmark multiobjective optimization problems under uncertainty and 2) practical multiobjective hyperparameter tuning of deep neural network models. The results show that the proposed algorithm can obtain up to about 73% speedup relative to existing approaches, demonstrating its ability to efficiently tackle real-world multiobjective optimization involving evaluations on large datasets.

Complaint and Severity Identification From Online Financial Content

The automatic detection of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">financial complaints</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FINCORP</i> ) can benefit businesses and online merchants. Compared with manually tagged complaints, they can use this information to monitor and address issues and effectively route them to appropriate teams. This can also promote greater transparency and accountability when dealing with consumer financial products and services, strengthening the firm’s brand value. In linguistic studies, complaints have been classified into severity categories based on the level of risk the complainant is prepared to accept. Furthermore, since emotions influence every speech act, an individual’s emotional state considerably impacts the complaint expression. In this article, we introduce a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FINCORP</i> resource, a collection of annotated complaints arising between financial institutions and consumers expressed in English on Twitter. The dataset has been enriched with the associated emotion, sentiment, and complaint severity classes. The dataset comprises 3149 complaint and 3133 noncomplaint instances spanning over ten domains (e.g., credit cards, mortgages). For a comprehensive evaluation of our dataset, we develop a multitask framework for complaint detection and severity classification with emotion recognition (ER) and sentiment classification as the additional tasks and compare it with several existing baselines. The corpus and code are available here: https://github.com/RohanBh23/FINCORP.

MTBullyGNN: A Graph Neural Network-Based Multitask Framework for Cyberbullying Detection

MTBullyGNN: A Graph Neural Network-Based Multitask Framework for Cyberbullying Detection

Enhanced encoder–decoder architecture for visual perception multitasking of autonomous driving

Visual perception plays a vital role in autonomous driving systems, demanding high accuracy and real-time inference speed to ensure safety. In this paper, we propose a multi-task framework that simultaneously performs object detection, drivable area segmentation, and lane line identification, addressing the requirements of accurate and efficient visual perception. Our approach utilizes a shared-encoder architecture with three separate decoders, targeting each specific task. We investigate three configurations for the shared encoder: a Convolutional Neural Network (CNN), a Polyp Vision Transformer (PVT), and a hybrid CNN+PVT model. Through extensive experimentation and comparative analysis on the challenging BD100K dataset, we evaluate the performance of these shared-encoder models and provide valuable insights into their strengths and weaknesses. Our research contributes to the advancement of multi-task visual perception for autonomous driving systems by achieving competitive results in terms of accuracy and efficiency. The source code is publicly available on GitHub to facilitate further research in this domain.

Multitask Weakly Supervised Generative Network for MR-US Registration.

Registering pre-operative modalities, such as magnetic resonance imaging or computed tomography, to ultrasound images is crucial for guiding clinicians during surgeries and biopsies. Recently, deep-learning approaches have been proposed to increase the speed and accuracy of this registration problem. However, all of these approaches need expensive supervision from the ultrasound domain. In this work, we propose a multitask generative framework that needs weak supervision only from the pre-operative imaging domain during training. To perform a deformable registration, the proposed framework translates a magnetic resonance image to the ultrasound domain while preserving the structural content. To demonstrate the efficacy of the proposed method, we tackle the registration problem of pre-operative 3D MR to transrectal ultrasonography images as necessary for targeted prostate biopsies. We use an in-house dataset of 600 patients, divided into 540 for training, 30 for validation, and the remaining for testing. An expert manually segmented the prostate in both modalities for validation and test sets to assess the performance of our framework. The proposed framework achieves a 3.58 mm target registration error on the expert-selected landmarks, 89.2% in the Dice score, and 1.81 mm 95th percentile Hausdorff distance on the prostate masks in the test set. Our experiments demonstrate that the proposed generative model successfully translates magnetic resonance images into the ultrasound domain. The translated image contains the structural content and fine details due to an ultrasound-specific two-path design of the generative model. The proposed framework enables training learning-based registration methods while only weak supervision from the pre-operative domain is available.

Bacterial image analysis using multi-task deep learning approaches for clinical microscopy.

Bacterial image analysis plays a vital role in various fields, providing valuable information and insights for studying bacterial structural biology, diagnosing and treating infectious diseases caused by pathogenic bacteria, discovering and developing drugs that can combat bacterial infections, etc. As a result, it has prompted efforts to automate bacterial image analysis tasks. By automating analysis tasks and leveraging more advanced computational techniques, such as deep learning (DL) algorithms, bacterial image analysis can contribute to rapid, more accurate, efficient, reliable, and standardised analysis, leading to enhanced understanding, diagnosis, and control of bacterial-related phenomena. Three object detection networks of DL algorithms, namely SSD-MobileNetV2, EfficientDet, and YOLOv4, were developed to automatically detect Escherichia coli (E. coli) bacteria from microscopic images. The multi-task DL framework is developed to classify the bacteria according to their respective growth stages, which include rod-shaped cells, dividing cells, and microcolonies. Data preprocessing steps were carried out before training the object detection models, including image augmentation, image annotation, and data splitting. The performance of the DL techniques is evaluated using the quantitative assessment method based on mean average precision (mAP), precision, recall, and F1-score. The performance metrics of the models were compared and analysed. The best DL model was then selected to perform multi-task object detections in identifying rod-shaped cells, dividing cells, and microcolonies. The output of the test images generated from the three proposed DL models displayed high detection accuracy, with YOLOv4 achieving the highest confidence score range of detection and being able to create different coloured bounding boxes for different growth stages of E. coli bacteria. In terms of statistical analysis, among the three proposed models, YOLOv4 demonstrates superior performance, achieving the highest mAP of 98% with the highest precision, recall, and F1-score of 86%, 97%, and 91%, respectively. This study has demonstrated the effectiveness, potential, and applicability of DL approaches in multi-task bacterial image analysis, focusing on automating the detection and classification of bacteria from microscopic images. The proposed models can output images with bounding boxes surrounding each detected E. coli bacteria, labelled with their growth stage and confidence level of detection. All proposed object detection models have achieved promising results, with YOLOv4 outperforming the other models.

Attention Guided Multi-Task Network for Joint CFO and Channel Estimation in OFDM Systems

The existence of high carrier frequency offset (CFO) and fading channels degrades the performance of communication systems in high mobility environments significantly. To address these challenges, in this paper, we propose an attention guided multi-task network for joint CFO and channel estimation in orthogonal frequency division multiplexing (OFDM) systems. Specifically, considering the correlation between CFO estimation and channel estimation, we construct a multi-task neural network framework which consists of a shared branch (SB), a CFO estimation branch (CFOEB) and a channel estimation branch (CEB). Among them, the SB extracts common features of two tasks, and the CFOEB and CEB make more detailed CFO and channel estimation, respectively. To further suppress the influence of CFO estimation error on channel estimation, we introduce an attention guided module (AGM) which completes the re-calibration of the channel dimension on the basis of different CFO characteristics. We train the whole network by an end-to-end paradigm and construct a suitable multi-task loss function to balance the losses between different tasks. Simulation results demonstrate that the proposed attention guided multi-task learning based joint CFO and channel estimation scheme outperforms the conventional methods and has greater robustness under various conditions.

Explicit Evolutionary Framework With Multitasking Feature Fusion for Optimizing Operational Parameters in Aluminum Electrolysis Process.

Collaboratively optimizing operational parameters through leveraging accumulated production experience is an innovative approach to reducing energy consumption in aluminum electrolysis cells (AECs). Due to the dynamic heterogeneity of various AECs, an explicit evolutionary multitasking (EMT) framework capable of incorporating different optimizers, has the potential to tackle this challenge effectively. However, there is a notable gap in theoretical research on multitasking collaborative evolutionary algorithms specifically applied to AECs. Meanwhile, existing explicit EMT algorithms often overlook the intertask correlation of feature information extracted in isolation from individual tasks. These issues significantly limit the development of synergistic effects in multitasking optimization for addressing parameter design in AECs. To address these limitations, this work proposes an explicit evolutionary framework with multitasking feature fusion (EMFF). This framework thoroughly considers the potential connections among feature information from different tasks. It achieves effective knowledge transfer by the design of a unique multitasking feature fusion mechanism, which enhances the information value of source tasks for target tasks. Furthermore, a transfer individual derivation (TID) strategy is introduced to ensure the rapid evolution of critical knowledge. Finally, comprehensive components and designed process are presented. Experimental results demonstrate EMFF's exceptional performance in various benchmark tests and real-world AEC parameter optimization cases.

Enforcing Water Balance in Multitask Deep Learning Models for Hydrological Forecasting

Abstract Accurate prediction of hydrological variables (HVs) is critical for understanding hydrological processes. Deep learning (DL) models have shown excellent forecasting abilities for different HVs. However, most DL models typically predicted HVs independently, without satisfying the principle of water balance. This missed the interactions between different HVs in the hydrological system and the underlying physical rules. In this study, we developed a DL model based on multitask learning and hybrid physically constrained schemes to simultaneously forecast soil moisture, evapotranspiration, and runoff. The models were trained using ERA5-Land data, which have water budget closure. We thoroughly assessed the advantages of the multitask framework and the proposed constrained schemes. Results showed that multitask models with different loss-weighted strategies produced comparable or better performance compared to the single-task model. The multitask model with a scaling factor of 5 achieved the best among all multitask models and performed better than the single-task model over 70.5% of grids. In addition, the hybrid constrained scheme took advantage of both soft and hard constrained models, providing physically consistent predictions with better model performance. The hybrid constrained models performed the best among different constrained models in terms of both general and extreme performance. Moreover, the hybrid model was affected the least as the training data were artificially reduced, and provided better spatiotemporal extrapolation ability under different artificial prediction challenges. These findings suggest that the hybrid model provides better performance compared to previously reported constrained models when facing limited training data and extrapolation challenges.

A Multi-Task Framework for Car Detection From High-Resolution UAV Imagery Focusing on Road Regions

A Multi-Task Framework for Car Detection From High-Resolution UAV Imagery Focusing on Road Regions