Multi-task Learning Architecture Research Articles

LungXpertAI: A deep multi-task learning model for chest CT scan analysis and COVID-19 detection

Addressing the urgent need for accurate COVID-19 identification and lung infection segmentation in CT scans, our study introduces LungXpertAI, a novel Multi-Task Learning architecture. Previous studies on multi-task structures have been limited in providing detailed insights into their architectures. Addressing these details is crucial for enhancing the performance of multi-task structures. Our proposed multi-task structure, based on the U-Net architecture, incorporates a shared encoder, a specialized segmentation decoder, and a multi-layer perceptron for classification. In the proposed approach, image processing algorithms, including histogram equalization, median filtering, and morphological operations have been employed in the pre-processing stage of input images. Also, by combining these algorithms in pairs, an attempt has been made to enhance the performance of tasks. Initial preprocessing and the integration of the Convolutional Block Attention Module (CBAM) contribute to effective feature extraction. Post-processing with Conditional Random Field (CRF) further refines segmentation outputs. CRF considers spatial dependencies within the segmentation results, contributing to more precise delineation of COVID-19-affected areas in CT scans. Evaluation on using dataset demonstrates the effectiveness of pairwise combinations of image processing algorithms, achieving superior results with a segmentation accuracy of 95.66 % and a classification accuracy of 95.62 %. The incorporation of the CBAM module improves classification accuracy to 96.22 %. Moreover, Integrating CBAM and CRF significantly improves segmentation accuracy to 96.51 %. The application of our proposed method steps to U-Net++ and ResUnet showcases their potential for enhancing multi-task structures. This study establishes new COVID-19 detection standards, promising progress in medical image analysis.

Improving AI-assisted video editing: Optimized footage analysis through multi-task learning

In recent years, AI-assisted video editing has shown promising applications. Understanding and analyzing camera language accurately is fundamental in video editing, guiding subsequent editing and production processes. However, many existing methods for camera language analysis overlook computational efficiency and deployment requirements in favor of improving classification accuracy. Consequently, they often fail to meet the demands of scenarios with limited computing power, such as mobile devices. To address this challenge, this paper proposes an efficient multi-task camera language analysis pipeline based on shared representations. This approach employs a multi-task learning architecture with hard parameter sharing, enabling different camera language classification tasks to utilize the same low-level feature extraction network, thereby implicitly learning feature representations of the footage. Subsequently, each classification sub-task independently learns the high-level semantic information corresponding to the camera language type. This method significantly reduces computational complexity and memory usage while facilitating efficient deployment on devices with limited computing power. Furthermore, to enhance performance, we introduce a dynamic task priority strategy and a conditional dataset downsampling strategy. The experimental results demonstrate that achieved a comprehensive accuracy surpassing all previous methods. Moreover, training time was reduced by 66.33%, inference cost decreased by 59.85%, and memory usage decreased by 31.95% on the 2-task dataset MovieShots; on the 4-task dataset AVE, training time was reduced by 95.34%, inference cost decreased by 97.23%, and memory usage decreased by 61.21%.

Gated ensemble of spatio-temporal mixture of experts for multi-task learning in ride-hailing system

Ride-hailing system requires efficient management of dynamic demand and supply to ensure optimal service delivery, pricing strategies, and operational efficiency. Designing spatio-temporal forecasting models separately in a task-wise and city-wise manner to forecast demand and supply-demand gap in a ride-hailing system poses a burden for the expanding transportation network companies. Therefore, a multi-task learning architecture is proposed in this study by developing gated ensemble of spatio-temporal mixture of experts network (GESME-Net) with convolutional recurrent neural network (CRNN), convolutional neural network (CNN), and recurrent neural network (RNN) for simultaneously forecasting these spatio-temporal tasks in a city as well as across different cities. Furthermore, a task adaptation layer is integrated with the architecture for learning joint representation in multi-task learning and revealing the contribution of the input features utilized in prediction. The proposed architecture is tested with data from Didi Chuxing for: (i) simultaneously forecasting demand and supply-demand gap in Beijing, and (ii) simultaneously forecasting demand across Chengdu and Xian. In both scenarios, models from our proposed architecture outperformed the single-task and multi-task deep learning benchmarks and ensemble-based machine learning algorithms.

MTLSC-Diff: Multitask learning with diffusion models for hyperspectral image super-resolution and classification

Hyperspectral image (HSIs) super-resolution (SR) can improve the spatial resolution of images for subsequent application tasks. In recent years, SR methods based on deep learning have gained widespread attention. However, most of the existing SR methods do not take into account the needs of specific application tasks when designing the network structure. These methods may not be able to efficiently generate high-quality images that satisfy the specific application tasks, leading to degradation of the performance of subsequent application tasks. To solve this problem, we propose a multi-task learning architecture based on the diffusion model, namely MTLSC-Diff. MTLSC-Diff combines the SR network and the classification network in a multi-task learning manner on the basis of the diffusion model. MTLSC-Diff achieves mutual guidance of the two tasks by iterating the image super-resolution and classification tasks, thus gradually reconstructing high-quality images and improving classification accuracy. The guided operations for each time step are performed by the specially designed Mutual-Guidance SR-Classification Synergy Module (M-GSCS). M-GSCS refines the multi-scale image obtained at the previous time step and uses the predicted high spatial resolution image for classification. Meanwhile, a class-guided SR dynamic refinement strategy (C-GSR) is proposed in M-GSCS, which uses multi-scale classification results to guide target scale images to learn new knowledge to further reconstruct high-quality images. Experimental results on relevant datasets show that our method significantly improves the super-resolution performance as well as the classification performance.

Multitask learning of a biophysically-detailed neuron model.

The human brain operates at multiple levels, from molecules to circuits, and understanding these complex processes requires integrated research efforts. Simulating biophysically-detailed neuron models is a computationally expensive but effective method for studying local neural circuits. Recent innovations have shown that artificial neural networks (ANNs) can accurately predict the behavior of these detailed models in terms of spikes, electrical potentials, and optical readouts. While these methods have the potential to accelerate large network simulations by several orders of magnitude compared to conventional differential equation based modelling, they currently only predict voltage outputs for the soma or a select few neuron compartments. Our novel approach, based on enhanced state-of-the-art architectures for multitask learning (MTL), allows for the simultaneous prediction of membrane potentials in each compartment of a neuron model, at a speed of up to two orders of magnitude faster than classical simulation methods. By predicting all membrane potentials together, our approach not only allows for comparison of model output with a wider range of experimental recordings (patch-electrode, voltage-sensitive dye imaging), it also provides the first stepping stone towards predicting local field potentials (LFPs), electroencephalogram (EEG) signals, and magnetoencephalography (MEG) signals from ANN-based simulations. While LFP and EEG are an important downstream application, the main focus of this paper lies in predicting dendritic voltages within each compartment to capture the entire electrophysiology of a biophysically-detailed neuron model. It further presents a challenging benchmark for MTL architectures due to the large amount of data involved, the presence of correlations between neighbouring compartments, and the non-Gaussian distribution of membrane potentials.

Multi-Task Learning in Natural Language Processing: An Overview

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this article, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Multi-task neural networks by learned contextual inputs

This paper explores learned-context neural networks. It is a multi-task learning architecture based on a fully shared neural network and an augmented input vector containing trainable task parameters. The architecture is interesting due to its powerful task adaption mechanism, which facilitates a low-dimensional task parameter space. Theoretically, we show that a scalar task parameter is sufficient for universal approximation of all tasks, which is not necessarily the case for more common architectures. Empirically it is shown that, for homogeneous tasks, the dimension of the task parameter may vary with the complexity of the tasks, but a small task parameter space is generally viable. The task parameter space is found to be well-behaved, which simplifies workflows related to updating models as new data arrives, and learning new tasks with the shared parameters are frozen. Additionally, the architecture displays robustness towards datasets where tasks have few data points. The architecture’s performance is compared to similar neural network architectures on ten datasets, with competitive results.

Multitask Learning for Concurrent Grading Diagnosis and Semi-Supervised Segmentation of Honeycomb Lung in CT Images

Honeycomb lung is a radiological manifestation of various lung diseases, seriously threatening patients’ lives worldwide. In clinical practice, the precise localization of lesions and assessment of their severity are crucial. However, accurate segmentation and grading are challenging for physicians due to the heavy annotation burden and diversity of honeycomb lungs. In this paper, we propose a multitask learning architecture for semi-supervised segmentation and grading diagnosis to achieve automatic localization and assessment of lesions. To the best of our knowledge, this is the first method that integrates a grading diagnosis task into honeycomb lung semi-supervised segmentation. Firstly, we adapt cross-learning to capture local features and long-range dependencies from the CNN and transformer. Secondly, considering the diversity of honeycomb lung lesions, the shape-edge aware constraint is designed to assist the model in locating lesions. Then, in order to better understand the different levels of information in the images, we develop global contrast and local contrast learning to enhance the model’s learning of semantic-level and pixel-level features. Lastly, aiming to improve the diagnostic accuracy, we propose a gradient thresholding algorithm to integrate the segmentation predictions into the grading diagnosis network. The experiment’s results based on the in-house honeycomb lung dataset demonstrate the superiority of our method. Compared to other methods, our approach achieves a state-of-the-art performance. In particular, in external data testing, our predictions are consistent with physicians in the majority of cases. In addition, the segmentation results based on the public Kvasir-SEG dataset also indicate that our method has good generalization ability.

Reconstruction of arterial blood pressure waveforms based on squeeze-and-excitation network models using electrocardiography and photoplethysmography signals

Arterial blood pressure (ABP) waveforms indicate the efficiency with which a patient’s blood responds to changes in the arterial flow. ABP waveforms can be used to analyze the cardiovascular health of patients, including heart rate analysis, stress analysis, and cardiovascular disease diagnosis. However, obtaining the ABP waveform is more difficult than recording the electrocardiography (ECG) and photoplethysmography (PPG) signals. Several studies have focused on the reconstruction of the ABP waveform by recombining bio-signals. Reconstructing ABP waveforms is considered a major challenge because of signals being distorted by waveform destruction, owing to the normalization and standardization of the input ABP waveform. This paper presents a model that can simultaneously output blood pressure estimation and ABP waveforms using the raw ABP signal as the input. Herein, the squeeze-and-excitation network (SE-Net) is coupled with the multi-task learning architecture. Three signals, namely, the ECG, PPG, and ECG–PPG, were trained as feature vectors using different variants of the U-Net model, and the results are compared. Additionally, we introduce a novel method for incorporating white Gaussian noise at different signal-to-noise ratios (SNRs) to augment the training data. The objective of the proposed approach is to evaluate the ABP signal reconstruction capability of the SE-Net at various SNR levels and analyze the overall performance and robustness of the model. In the case of systolic blood pressure, the root mean square error (RMSE) of the PPG-based values is 2.58 mmHg, and the Pearson correlation coefficient is r = 0.92 (p≤ 0.001). In the case of diastolic blood pressure(DBP), the RMSE was 3.35 mmHg, and the Pearson correlation coefficient was r = 0.89 (p ≤ 0.001). The results indicate the potential for replacing ECG signals with PPG signals to overcome the constraints of optimizing the non-invasive reconstruction of ABP waveforms using SE-Net. Furthermore, we demonstrate that the SE-Net-combined multi-task learning architecture model can simultaneously perform blood pressure estimation and output ABP waveforms without damaging the raw signal.

End-to-End Multi-task Learning Architecture for Brain Tumor Analysis with Uncertainty Estimation in MRI Images.

Brain tumors are a threat to life for every other human being, be it adults or children. Gliomas are one of the deadliest brain tumors with an extremely difficult diagnosis. The reason is their complex and heterogenous structure which gives rise to subjective as well as objective errors. Their manual segmentation is a laborious task due to their complex structure and irregular appearance. To cater to all these issues, a lot of research has been done and is going on to develop AI-based solutions that can help doctors and radiologists in the effective diagnosis of gliomas with the least subjective and objective errors, but an end-to-end system is still missing. An all-in-one framework has been proposed in this research. The developed end-to-end multi-task learning (MTL) architecture with a feature attention module can classify, segment, and predict the overall survival of gliomas by leveraging task relationships between similar tasks. Uncertainty estimation has also been incorporated into the framework to enhance the confidence level of healthcare practitioners. Extensive experimentation was performed by using combinations of MRI sequences. Brain tumor segmentation (BraTS) challenge datasets of 2019 and 2020 were used for experimental purposes. Results of the best modelwith four sequences show 95.1% accuracy for classification, 86.3% dicescore for segmentation, and a mean absolute error (MAE) of 456.59 for survival predictionon the test data. It is evident from the results that deep learning-based MTL models have the potential to automate the whole brain tumor analysis process and give efficient results with least inference time without human intervention. Uncertainty quantification confirms the idea that more data can improve the generalization ability and in turn can produce more accurate results with less uncertainty. The proposed model has the potential to be utilized in a clinical setup for the initial screening of glioma patients.

Multi-modal fusion for business process prediction in call center scenarios

Call centers are critical for gathering customer feedback, making them essential for business communication. Predicting the ongoing business process status accurately has become a focus in both academia and industry. However, current methods mainly analyze process sequence data from enterprise information systems, missing out on valuable data from other sources like call centers. Moreover, these methods often focus on a single task, ignoring the shared information across multiple tasks. This paper presents a novel method for business process prediction that fuses multi-modal data from both information systems and call centers. Specifically, the method combines sequence data from the enterprise information system and dialogue text data from the call center for a more enriched business process prediction. Additionally, to navigate the multi-task learning conundrum, we improve the existing MMoE algorithm and introduce a new multi-task learning architecture called Heterogeneous Multi-gate Mixture-of-experts. The experimental results over some current approaches like Transformer, CNN and LSTM show superior prediction performance compared to baseline models, demonstrating that our method can help call centers optimize their processes, improve customer service, and drive business success.

STSN-Net: Simultaneous Tooth Segmentation and Numbering Method in Crowded Environments with Deep Learning.

Accurate tooth segmentation and numbering are the cornerstones of efficient automatic dental diagnosis and treatment. In this paper, a multitask learning architecture has been proposed for accurate tooth segmentation and numbering in panoramic X-ray images. A graph convolution network was applied for the automatic annotation of the target region, a modified convolutional neural network-based detection subnetwork (DSN) was used for tooth recognition and boundary regression, and an effective region segmentation subnetwork (RSSN) was used for region segmentation. The features extracted using RSSN and DSN were fused to optimize the quality of boundary regression, which provided impressive results for multiple evaluation metrics. Specifically, the proposed framework achieved a top F1 score of 0.9849, a top Dice metric score of 0.9629, and an mAP (IOU = 0.5) score of 0.9810. This framework holds great promise for enhancing the clinical efficiency of dentists in tooth segmentation and numbering tasks.

Sequential sentence classification in research papers using cross-domain multi-task learning

The automatic semantic structuring of scientific text allows for more efficient reading of research articles and is an important indexing step for academic search engines. Sequential sentence classification is an essential structuring task and targets the categorisation of sentences based on their content and context. However, the potential of transfer learning for sentence classification across different scientific domains and text types, such as full papers and abstracts, has not yet been explored in prior work. In this paper, we present a systematic analysis of transfer learning for scientific sequential sentence classification. For this purpose, we derive seven research questions and present several contributions to address them: (1) We suggest a novel uniform deep learning architecture and multi-task learning for cross-domain sequential sentence classification in scientific text. (2) We tailor two transfer learning methods to deal with the given task, namely sequential transfer learning and multi-task learning. (3) We compare the results of the two best models using qualitative examples in a case study. (4) We provide an approach for the semi-automatic identification of semantically related classes across annotation schemes and analyse the results for four annotation schemes. The clusters and underlying semantic vectors are validated using k-means clustering. (5) Our comprehensive experimental results indicate that when using the proposed multi-task learning architecture, models trained on datasets from different scientific domains benefit from one another. Our approach significantly outperforms state of the art on full paper datasets while being on par for datasets consisting of abstracts.

Multi-task learning for predicting quality-of-life and independence in activities of daily living after stroke: a proof-of-concept study.

A health-related (HR) profile is a set of multiple health-related items recording the status of the patient at different follow-up times post-stroke. In order to support clinicians in designing rehabilitation treatment programs, we propose a novel multi-task learning (MTL) strategy for predicting post-stroke patient HR profiles. The HR profile in this study is measured by the Barthel index (BI) assessment or by the EQ-5D-3L questionnaire. Three datasets are used in this work and for each dataset six neural network architectures are developed and tested. Results indicate that an MTL architecture combining a pre-trained network for all tasks with a concatenation strategy conditioned by a task grouping method is a promising approach for predicting the HR profile of a patient with stroke at different phases of the patient journey. These models obtained a mean F1-score of 0.434 (standard deviation 0.022, confidence interval at 95% [0.428, 0.44]) calculated across all the items when predicting BI at 3 months after stroke (MaS), 0.388 (standard deviation 0.029, confidence interval at 95% [0.38, 0.397]) when predicting EQ-5D-3L at 6MaS, and 0.462 (standard deviation 0.029, confidence interval at 95% [0.454, 0.47]) when predicting the EQ-5D-3L at 18MaS. Furthermore, our MTL architecture outperforms the reference single-task learning models and the classic MTL of all tasks in 8 out of 10 tasks when predicting BI at 3MaS and has better prediction performance than the reference models on all tasks when predicting EQ-5D-3L at 6 and 18MaS. The models we present in this paper are the first models to predict the components of the BI or the EQ-5D-3L, and our results demonstrate the potential benefits of using MTL in a health context to predict patient profiles.

Bi-preference Learning Heterogeneous Hypergraph Networks for Session-based Recommendation

Session-based recommendation intends to predict next purchased items based on anonymous behavior sequences. Numerous economic studies have revealed that item price is a key factor influencing user purchase decisions. Unfortunately, existing methods for session-based recommendation only aim at capturing user interest preference, while ignoring user price preference. Actually, there are primarily two challenges preventing us from accessing price preference. First, the price preference is highly associated to various item features (i.e., category and brand), which asks us to mine price preference from heterogeneous information. Second, price preference and interest preference are interdependent and collectively determine user choice, necessitating that we jointly consider both price and interest preference for intent modeling. To handle above challenges, we propose a novel approach Bi-Preference Learning Heterogeneous Hypergraph Networks (BiPNet) for session-based recommendation. Specifically, the customized heterogeneous hypergraph networks with a triple-level convolution are devised to capture user price and interest preference from heterogeneous features of items. Besides, we develop a Bi-Preference Learning schema to explore mutual relations between price and interest preference and collectively learn these two preferences under the multi-task learning architecture. Extensive experiments on multiple public datasets confirm the superiority of BiPNet over competitive baselines. Additional research also supports the notion that the price is crucial for the task.

Visible Light Integrated Positioning and Communication: A Multi-Task Federated Learning Framework

Recently, visible light positioning and visible light communication are becoming a promising technology for integrated sensing and communication. However, the isolated design of positioning and communication has limited the system efficiency and performance. In this paper, a visible light integrated positioning and communication (VIPAC) framework is formulated, in which the positioning task for the sensing service and the channel estimation task for the communication service are integrated into a unified architecture. Firstly, a multi-task learning architecture, which is composed of a sparsity-aware shared network and two task-oriented sub-networks, is proposed to fully exploit the inherent sparse features of visible light channels, and achieve mutual benefits between the two tasks. The depth of the shared network can be adaptively adjusted to extract the optimal shared features, and the two sub-networks are further optimized for the two tasks, respectively. Moreover, the emerging federated learning technique is introduced to devise a multi-user cooperative VIPAC scheme, which further improves the generalization ability in spatiotemporally nonstationary environments while preserving data privacy. It is shown by theoretical analysis and simulation results that, the proposed scheme can significantly improve the performance of positioning and channel estimation in spatiotemporally nonstationary environments compared with existing benchmark schemes.

A Study on the Architecture of Artificial Neural Network Considering Injection-Molding Process Steps.

In this study, an artificial neural network (ANN) was established to predict product properties (mass, diameter, height) using six process conditions of the injection-molding process (melt temperature, mold temperature, injection speed, packing pressure, packing time, and cooling time) as input parameters. The injection-molding process consists of continuous sequential stages, including the injection stage, packing stage, and cooling stage. However, the related research tends to have an insufficient incorporation of structural characteristics based on these basic process stages. Therefore, in order to incorporate these process stages and characteristics into the ANN, a process-based multi-task learning technique was applied to the connection between the input parameters and the front-end of the hidden layer. This resulted in the construction of two network structures, and their performance was evaluated by comparing them with the typical network structure. The results showed that a multi-task learning architecture that incorporated process-level specific structures in the connections between the input parameters and the front end of the hidden layer yielded relatively better root mean square errors (RMSEs) values than a conventional neural network architecture, by as much as two orders of magnitude. Based on these results, this study has provided guidance for the construction of artificial neural networks for injection-molding processes that incorporates process-stage specific features and structures in the architecture.

The Effect of Grouping Output Parameters by Quality Characteristics on the Predictive Performance of Artificial Neural Networks in Injection Molding Process

In this study, a multi-input, multi-output-based artificial neural network (ANN) was constructed by classifying output parameters into different groups, considering the physical meanings and characteristics of product quality factors in the injection molding process. Injection molding experiments were conducted for bowl products, and a dataset was established. Based on this dataset, an ANN model was developed to predict the quality of molded products. The input parameters included melt temperature, mold temperature, packing pressure, packing time, and cooling time. The output parameters included mass, diameter, and height of the molded product. The output parameters were divided into two cases. In one case, diameter, and height, representing length, were grouped together, while mass was organized into a separate group. In the other case, mass, diameter and height were separated individually and applied to the ANN. A multi-task learning method was used to group the output parameters. The performance of the two constructed multi-task learning-based ANNs was compared with that of the conventional ANN where the output parameters were not separated and applied to a single layer. The comparative results showed that the multi-task learning architecture, which grouped the output parameters considering the physical meaning and characteristics of the quality of molded products, exhibited an improved prediction performance of about 32.8% based on the RMSE values.

Assisting Multimodal Named Entity Recognition by cross-modal auxiliary tasks

Although the existing Multimodal Named Entity Recognition (MNER) methods have achieved promising performance, they suffer from the following drawbacks in social media scenarios. Firstly, most existing methods are based on a strong assumption that the textual content and the associated images are matched, which is not always valid in real scenarios; Secondly, current methods fail to filter out modality-specific random noise, which impedes models from exploiting modality-shared features. In this paper, a novel multi-task multimodal learning architecture is put forward, which aims to improve Multimodal Named Entity Recognition (MNER) performance by cross-modal auxiliary tasks (CMAT). Specifically, we first separate the shared and task-specific features for the main task and auxiliary tasks respectively, which is accomplished by cross-modal gate-control mechanism. Subsequently, without extra pre-processing or annotations, we utilize the cross-modal matching to address the issue of mismatched image-text pairs, and the cross-modal mutual information maximization to optimize the most relevant cross-modal features. Moreover, experimental results on the two widely used datasets confirm the superiority of our proposed approach.

Keypoint Detection and Description through Deep Learning in Unstructured Environments

Feature extraction plays a crucial role in computer vision and autonomous navigation, offering valuable information for real-time localization and scene understanding. However, although multiple studies investigate keypoint detection and description algorithms in urban and indoor environments, far fewer studies concentrate in unstructured environments. In this study, a multi-task deep learning architecture is developed for keypoint detection and description, focused on poor-featured unstructured and planetary scenes with low or changing illumination. The proposed architecture was trained and evaluated using a training and benchmark dataset with earthy and planetary scenes. Moreover, the trained model was integrated in a visual SLAM (Simultaneous Localization and Maping) system as a feature extraction module, and tested in two feature-poor unstructured areas. Regarding the results, the proposed architecture provides a mAP (mean Average Precision) in a level of 0.95 in terms of keypoint description, outperforming well-known handcrafted algorithms while the proposed SLAM achieved two times lower RMSE error in a poor-featured area with low illumination, compared with ORB-SLAM2. To the best of the authors’ knowledge, this is the first study that investigates the potential of keypoint detection and description through deep learning in unstructured and planetary environments.