Multi-task Learning Network Research Articles

SPVINet: A Lightweight Multitask Learning Network for Assisting Visually Impaired People in Multiscene Perception

SPVINet: A Lightweight Multitask Learning Network for Assisting Visually Impaired People in Multiscene Perception

Multi-Task Visual Perception for Object Detection and Semantic Segmentation in Intelligent Driving

With the rapid development of intelligent driving vehicles, multi-task visual perception based on deep learning emerges as a key technological pathway toward safe vehicle navigation in real traffic scenarios. However, due to the high-precision and high-efficiency requirements of intelligent driving vehicles in practical driving environments, multi-task visual perception remains a challenging task. Existing methods typically adopt effective multi-task learning networks to concurrently handle multiple tasks. Despite the fact that they obtain remarkable achievements, better performance can be achieved through tackling existing problems like underutilized high-resolution features and underexploited non-local contextual dependencies. In this work, we propose YOLOPv3, an efficient anchor-based multi-task visual perception network capable of handling traffic object detection, drivable area segmentation, and lane detection simultaneously. Compared to prior works, we make essential improvements. On the one hand, we propose architecture enhancements that can utilize multi-scale high-resolution features and non-local contextual dependencies for improving network performance. On the other hand, we propose optimization improvements aiming at enhancing network training, enabling our YOLOPv3 to achieve optimal performance via straightforward end-to-end training. The experimental results on the BDD100K dataset demonstrate that YOLOPv3 sets a new state of the art (SOTA): 96.9% recall and 84.3% mAP50 in traffic object detection, 93.2% mIoU in drivable area segmentation, and 88.3% accuracy and 28.0% IoU in lane detection. In addition, YOLOPv3 maintains competitive inference speed against the lightweight YOLOP. Thus, YOLOPv3 stands as a robust solution for handling multi-task visual perception problems. The code and trained models have been released on GitHub.

Advancing cuffless blood pressure estimation: A PPG-based multi-task learning model for enhanced feature extraction and fusion

Cuffless continuous blood pressure (BP) monitoring is essential for personalized health management. Although existing cuffless BP estimation applies advanced machine learning techniques and integrates PPG signals, it is deficient in feature extraction and fusion. In addition, it is inefficient to train the model separately for different tasks. In this study, an advanced multi-domain and local–global feature parallel multi-task learning network (MDLG-MTLNet) is introduced. The MDLG-MTLNet was designed with three key aspects: first, temporal and multi-scale spatial features were extracted from PPG signals and their derivatives via a multi-scale spatial and temporal feature block; interaction of features from different domains was facilitated by the introduction of a local–global attention module that captured and efficiently fused local–global information; and lastly, intrinsic correlation between systolic (SBP) and diastolic blood pressure (DBP) was taken into account via a multi-task learning network to improve the overall performance of the model. On the MIMIC-II dataset, the MAEs of MDLG-MTLNet for DBP, SBP, and MBP were 2.64 mmHg, 1.57 mmHg, and 2.02 mmHg, respectively. These errors were superior to those of the existing methods, meeting the AAMI criteria, and earning an A grade according to the BHS protocol. The experimental results confirm the effectiveness of our proposed model in feature extraction and fusion.

A novel multi-task learning network for skin lesion classification based on multi-modal clues and label-level fusion

In this paper, we propose a multi-task learning (MTL) network based on the label-level fusion of metadata and hand-crafted features by unsupervised clustering to generate new clustering labels as an optimization goal. We propose a MTL module (MTLM) that incorporates an attention mechanism to enable the model to learn more integrated, variable information. We propose a dynamic strategy to adjust the loss weights of different tasks, and trade off the contributions of multiple branches. Instead of feature-level fusion, we propose label-level fusion and combine the results of our proposed MTLM with the results of the image classification network to achieve better lesion prediction on multiple dermatological datasets. We verify the effectiveness of the proposed model by quantitative and qualitative measures. The MTL network using multi-modal clues and label-level fusion can yield the significant performance improvement for skin lesion classification.

A multi-task learning network based on the Transformer network for airborne electromagnetic detection imaging and denoising

Abstract As an emerging geophysical exploration technology in recent years, airborne electromagnetic exploration has the advantages of adapting to diverse terrains, wide coverage, and providing a large amount of electromagnetic data, and can be applied to the rapid collection of large amounts of data. Scenarios are often used in fields such as deep geological structures, mineral resource exploration, and environmental engineering research. However, traditional airborne electromagnetic data inversion technology usually takes a long time to process a large amount of airborne electromagnetic data, and it is difficult to remove the noise in the later signals. Therefore, this paper proposes a multi-task learning network structure based on Transformer. By constraining the two network branches of imaging and denoising, a sub-network with simultaneous denoising and imaging is established to process aeronautical electromagnetic data. The noise test set is introduced for testing. This model achieved a 582.61% signal-to-noise ratio improvement in smooth Gaussian noise denoising, and a 129.69% and 112.74% signal-to-noise ratio improvement in non-smooth Gaussian noise and random impulse noise denoising, respectively. The method proposed in this article overcomes the shortcomings of traditional inversion imaging such as slow speed and low resolution, and at the same time eliminates the influence of noise in airborne electromagnetic data. This is of great significance for the application of deep learning in the field of geophysical exploration.

SRTRP-Net: A multi-task learning network for segmentation and prediction of stereotactic radiosurgery treatment response in brain metastases

Before the Stereotactic Radiosurgery (SRS) treatment, it is of great clinical significance to avoid secondary genetic damage and guide the personalized treatment plans for patients with brain metastases (BM) by predicting the response to SRS treatment of brain metastatic lesions. Thus, we developed a multi-task learning model termed SRTRP-Net to provide prior knowledge of BM ROI and predict the SRS treatment response of the lesion. In dual-encoder tumor segmentation Network (DTS-Net), two parallel encoders encode the original and mirrored multi-modal MRI images. The differences in the dual-encoder features between foreground and background are enhanced by the symmetrical visual difference block (SVDB). In the bottom layer of the encoder, a transformer is used to extract local contextual features in the spatial and depth dimensions of low-resolution images. Then, the decoder of DTS-Net provides the prior knowledge for predicting the response to SRS treatment by performing BM segmentation. SRS response prediction network (SRP-Net) directly utilizes shared multi-modal MRI features weighted by the signed distance map (SDM) of the masks. The bidirectional multi-dimensional feature fusion module (BMDF) fuses the shared features and the clinical text information features to obtain comprehensive tumor information for characterizing tumors and predicting SRS treatment response. Experiments based on internal and external clinical datasets have shown that SRTRP-Net achieves comparable or better results. We believe that SRTRP-Net can help clinicians accurately develop personalized first-time treatment regimens for BM patients and improve their survival.

Domain-alignment multitask learning network for partial discharge condition assessment with digital twin in gas-insulated switchgear

Deep-learning-driven methods have made great progress in the condition assessment of partial discharge (PD) which including diagnosis and location in gas-insulated switchgear (GIS). However, these methods perform diagnosis and location as two separate tasks and ignore the coupling relationship. In addition, these methods all require obtaining sufficient samples to develop models, and the model becomes ineffective when there is a significant difference in sample distribution. Therefore, we propose a novel domain-alignment multitask learning network (DAMTLN) for condition assessment including diagnosis and location assisted by digital twin. Firstly, a digital virtual model is established to assist the actual condition assessment of GIS PD. Then, a novel multitask network is constructed to mine the coupling relationship between the two tasks. Finally PD condition assessment guided by a digital twin model are achieved via a combination of local-maximum-mean-discrepancy-based and adversarial -based domain adaptation, in which fine-grained information on each category is captured. Experimental results show that the proposed DAMTLN achieved a diagnostic accuracy of 98.73%, and the mean absolute error of location was 9.06 cm, which were significantly better than the results of other methods. The DAMTLN thus provides a new avenue for PD diagnosis and location driven by ‘data–physics’ coupling.

A lightweight multi-task learning network based on key area guidance for counterfeit detection

Counterfeit detection traditionally relies on manual efforts, but manual detection efficiency is notably low. The accuracy of deep learning methods is challenging because of the insufficient samples, so it is crucial to allow the model to learn effective representation at a lower training cost. Given the above problems, we proposed a lightweight multi-task learning method that employs an uncomplicated auxiliary task to enhance the main task’s attention and reduce the training sample requirements. A key area guidance algorithm is designed to construct the auxiliary task, disturbing key image areas to generate new samples and training the auxiliary task to recognize the disturbance. This guides the main task in discerning authenticity from these key areas. Additionally, a tailored data preprocessing strategy was designed to improve the method’s performance further. Achieving an impressive 98.8% accuracy in identifying various counterfeiting points, our method outperforms existing advanced methods. Importantly, the method significantly reduces training costs. Even with an 80% reduction in the sample size, the method maintains a 92.1% accuracy, demonstrating minimal performance degradation compared to alternative methods.

Multi-task learning for segmentation and classification of breast tumors from ultrasound images

Segmentation and classification of breast tumors are critical components of breast ultrasound (BUS) computer-aided diagnosis (CAD), which significantly improves the diagnostic accuracy of breast cancer. However, the characteristics of tumor regions in BUS images, such as non-uniform intensity distributions, ambiguous or missing boundaries, and varying tumor shapes and sizes, pose significant challenges to automated segmentation and classification solutions. Many previous studies have proposed multi-task learning methods to jointly tackle tumor segmentation and classification by sharing the features extracted by the encoder. Unfortunately, this often introduces redundant or misleading information, which hinders effective feature exploitation and adversely affects performance. To address this issue, we present ACSNet, a novel multi-task learning network designed to optimize tumor segmentation and classification in BUS images. The segmentation network incorporates a novel gate unit to allow optimal transfer of valuable contextual information from the encoder to the decoder. In addition, we develop the Deformable Spatial Attention Module (DSAModule) to improve segmentation accuracy by overcoming the limitations of conventional convolution in dealing with morphological variations of tumors. In the classification branch, multi-scale feature extraction and channel attention mechanisms are integrated to discriminate between benign and malignant breast tumors. Experiments on two publicly available BUS datasets demonstrate that ACSNet not only outperforms mainstream multi-task learning methods for both breast tumor segmentation and classification tasks, but also achieves state-of-the-art results for BUS tumor segmentation. Code and models are available at https://github.com/qqhe-frank/BUS-segmentation-and-classification.git.

DBL-Net: A dual-branch learning network with information from spatial and frequency domains for tumor segmentation and classification in breast ultrasound image

Breast cancer is a common malignancy, ranking as one of the most prevalent cancers among women globally. Improving treatment success rates for breast cancer are crucially dependent on early detection and accurate diagnosis. Ultrasound imaging, as a non-invasive and low-radiation imaging technique, plays a significant role in early breast cancer detection. However, due to the characteristics of fuzzy tumor boundaries and irregular shapes of the tumor, accurate breast tumor classification and segmentation remain challenging tasks. This paper introduces a novel approach for breast ultrasound image segmentation and classification tasks. Firstly, we study the potential value of frequency domain information in breast image analysis and propose the DBL-Net, a multi-task learning network for simultaneous tumor segmentation and tumor classification. Secondly, we customize a Spatial-Frequency Domain Feature Encoding module (S-FEM) to better encode and fuse spatial and frequency domain features of breast ultrasound images. Furthermore, we design a Representational Perception Enhancer (RPE) to improve the feature representation capabilities and prediction accuracy of the model. Finally, We have carried out a comprehensive series of tests on the BUS dataset, and the findings clearly indicate that our method outperforms the current state-of-the-art methods.

Automatic and robust estimation of sex and chronological age from panoramic radiographs using a multi-task deep learning network: a study on a South Korean population.

Sex and chronological age estimation are crucial in forensic investigations and research on individual identification. Although manual methods for sex and age estimation have been proposed, these processes are labor-intensive, time-consuming, and error-prone. The purpose of this study was to estimate sex and chronological age from panoramic radiographs automatically and robustly using a multi-task deep learning network (ForensicNet). ForensicNet consists of a backbone and both sex and age attention branches to learn anatomical context features of sex and chronological age from panoramic radiographs and enables the multi-task estimation of sex and chronological age in an end-to-end manner. To mitigate bias in the data distribution, our dataset was built using 13,200 images with 100 images for each sex and age range of 15-80 years. The ForensicNet with EfficientNet-B3 exhibited superior estimation performance with mean absolute errors of 2.93 ± 2.61 years and a coefficient of determination of 0.957 for chronological age, and achieved accuracy, specificity, and sensitivity values of 0.992, 0.993, and 0.990, respectively, for sex prediction. The network demonstrated that the proposed sex and age attention branches with a convolutional block attention module significantly improved the estimation performance for both sex and chronological age from panoramic radiographs of elderly patients. Consequently, we expect that ForensicNet will contribute to the automatic and accurate estimation of both sex and chronological age from panoramic radiographs.

Multi-task global optimization-based method for vascular landmark detection

Vascular landmark detection plays an important role in medical analysis and clinical treatment. However, due to the complex topology and similar local appearance around landmarks, the popular heatmap regression based methods always suffer from the landmark confusion problem. Vascular landmarks are connected by vascular segments and have special spatial correlations, which can be utilized for performance improvement. In this paper, we propose a multi-task global optimization-based framework for accurate and automatic vascular landmark detection. A multi-task deep learning network is exploited to accomplish landmark heatmap regression, vascular semantic segmentation, and orientation field regression simultaneously. The two auxiliary objectives are highly correlated with the heatmap regression task and help the network incorporate the structural prior knowledge. During inference, instead of performing a max-voting strategy, we propose a global optimization-based post-processing method for final landmark decision. The spatial relationships between neighboring landmarks are utilized explicitly to tackle the landmark confusion problem. We evaluated our method on a cerebral MRA dataset with 564 volumes, a cerebral CTA dataset with 510 volumes, and an aorta CTA dataset with 50 volumes. The experiments demonstrate that the proposed method is effective for vascular landmark localization and achieves state-of-the-art performance.

Multi-Task Learning for Motion Analysis and Segmentation in 3D Echocardiography.

Characterizing left ventricular deformation and strain using 3D+time echocardiography provides useful insights into cardiac function and can be used to detect and localize myocardial injury. To achieve this, it is imperative to obtain accurate motion estimates of the left ventricle. In many strain analysis pipelines, this step is often accompanied by a separate segmentation step; however, recent works have shown both tasks to be highly related and can be complementary when optimized jointly. In this work, we present a multi-task learning network that can simultaneously segment the left ventricle and track its motion between multiple time frames. Two task-specific networks are trained using a composite loss function. Cross-stitch units combine the activations of these networks by learning shared representations between the tasks at different levels. We also propose a novel shape-consistency unit that encourages motion propagated segmentations to match directly predicted segmentations. Using a combined synthetic and in-vivo 3D echocardiography dataset, we demonstrate that our proposed model can achieve excellent estimates of left ventricular motion displacement and myocardial segmentation. Additionally, we observe strong correlation of our image-based strain measurements with crystal-based strain measurements as well as good correspondence with SPECT perfusion mappings. Finally, we demonstrate the clinical utility of the segmentation masks in estimating ejection fraction and sphericity indices that correspond well with benchmark measurements.

Simultaneous segmentation and classification of colon cancer polyp images using a dual branch multi-task learning network.

Accurate classification and segmentation of polyps are two important tasks in the diagnosis and treatment of colorectal cancers. Existing models perform segmentation and classification separately and do not fully make use of the correlation between the two tasks. Furthermore, polyps exhibit random regions and varying shapes and sizes, and they often share similar boundaries and backgrounds. However, existing models fail to consider these factors and thus are not robust because of their inherent limitations. To address these issues, we developed a multi-task network that performs both segmentation and classification simultaneously and can cope with the aforementioned factors effectively. Our proposed network possesses a dual-branch structure, comprising a transformer branch and a convolutional neural network (CNN) branch. This approach enhances local details within the global representation, improving both local feature awareness and global contextual understanding, thus contributing to the improved preservation of polyp-related information. Additionally, we have designed a feature interaction module (FIM) aimed at bridging the semantic gap between the two branches and facilitating the integration of diverse semantic information from both branches. This integration enables the full capture of global context information and local details related to polyps. To prevent the loss of edge detail information crucial for polyp identification, we have introduced a reverse attention boundary enhancement (RABE) module to gradually enhance edge structures and detailed information within polyp regions. Finally, we conducted extensive experiments on five publicly available datasets to evaluate the performance of our method in both polyp segmentation and classification tasks. The experimental results confirm that our proposed method outperforms other state-of-the-art methods.

Decoupling and Interacting Multi-Task Learning Network for Joint Speech and Accent Recognition

Decoupling and Interacting Multi-Task Learning Network for Joint Speech and Accent Recognition

Improving Tumor Classification by Reusing Self-predicted Segmentation of Medical Images as Guiding Knowledge.

Differential diagnosis of tumors is important for computer-aided diagnosis. In computer-aided diagnosis systems, expert knowledge of lesion segmentation masks is limited as it is only used during preprocessing or as supervision to guide feature extraction. To improve the utilization of lesion segmentation masks, this study proposes a simple and effective multitask learning network that improves medical image classification using self-predicted segmentation as guiding knowledge; we call this network RS 2-net. In RS 2-net, the predicted segmentation probability map obtained from the initial segmentation inference is added to the original image to form a new input, which is then reinput to the network for the final classification inference. We validated the proposed RS 2-net using three datasets: the pNENs-Grade dataset, which tested the prediction of pancreatic neuroendocrine neoplasm grading, and the HCC-MVI dataset, which tested the prediction of microvascular invasion of hepatocellular carcinoma, and ISIC 2017 public skin lesion dataset. The experimental results indicate that the proposed strategy of reusing self-predicted segmentation is effective, and RS 2-net outperforms other popular networks and existing state-of-the-art studies. Interpretive analytics based on feature visualization demonstrates that the improved classification performance of our reuse strategy is due to the semantic information that can be acquired in advance in a shallow network.

A Hybrid Multitask Learning Network for Hyperspectral Image Classification With Few Labels

A Hybrid Multitask Learning Network for Hyperspectral Image Classification With Few Labels

A Multi-Task Learning and Multi-Branch Network for DR and DME Joint Grading

Diabetic Retinopathy (DR) is one of the most common microvascular complications of diabetes. Diabetic Macular Edema (DME) is a concomitant symptom of DR. As the grade of lesion of DR and DME increase, the possibility of blindness can also increase significantly. In order to take the early interventions as soon as possible to reduce the likelihood of blindness, it is necessary to perform both DR and DME grading. We design a joint grading model based on multi-task learning and multi-branch networks (MaMNet) for DR and DME grading. The model mainly includes a multi-branch network (MbN), a feature fusion module, and a disease classification module. The MbN is formed by four branch structures, which can extract the low-level feature information of DME and DR in a targeted way; the feature fusion module is composed of a self-feature extraction module (SFEN), cross-feature extraction module (CFEN) and atrous spatial pyramid pooling module (ASPP). By combining various features collected from the aforementioned modules, the feature fusion module can provide more thorough discriminative features, which benefits the joint grading accuracy. The ISBI-2018-IDRiD challenge dataset is used to evaluate the performance of the proposed model. The experimental results show that based on the multi-task strategy the two grading tasks of DR and DME can provide each other with additional useful information. The joint accuracy of the model, the accuracy of DR and the accuracy of DME are 61.2%, 64.1% and 79.4% respectively.

An efficient instance segmentation approach for studying fission gas bubbles in irradiated metallic nuclear fuel

Gaseous fission products from nuclear fission reactions tend to form fission gas bubbles of various shapes and sizes inside nuclear fuel. The behavior of fission gas bubbles dictates nuclear fuel performances, such as fission gas release, grain growth, swelling, and fuel cladding mechanical interaction. Although mechanical understanding of the overall evolution behavior of fission gas bubbles is well known, lacking the quantitative data and high-level correlation between burnup/temperature and microstructure evolution blocks the development of predictive models and reduces the possibility of accelerating the qualification for new fuel forms. Historical characterization of fission gas bubbles in irradiated nuclear fuel relied on a simple threshold method working on low-resolution optical microscopy images. Advanced characterization of fission gas bubbles using scanning electron microscopic images reveals unprecedented details and extensive morphological data, which strains the effectiveness of conventional methods. This paper proposes a hybrid framework, based on digital image processing and deep learning models, to efficiently detect and classify fission gas bubbles from scanning electron microscopic images. The developed bubble annotation tool used a multitask deep learning network that integrates U-Net and ResNet to accomplish instance-level bubble segmentation. With limited annotated data, the model achieves a recall ratio of more than 90%, a leap forward compared to the threshold method. The model has the capability to identify fission gas bubbles with and without lanthanides to better understand the movement of lanthanide fission products and fuel cladding chemical interaction. Lastly, the deep learning model is versatile and applicable to the micro-structure segmentation of similar materials.

Multi-task Contexture Learning Network for automated vertebrae segmentation and tumor diagnosis from MRI

Automated vertebrae segmentation and tumor diagnosis from magnetic resonance imaging (MRI) scans play a crucial role in early detection of spinal tumors. However, the task is challenging due to the variability in fields of view across MRI images and the diverse appearances of tumors. To address these challenges, we propose a novel approach, the Multi-task Contexture Learning Network (MCLN), which concurrently performs accurate vertebrae segmentation and precise identification of tumors. Specifically, MCLN first constructs a spatial pyramid graph convolutional network and a label attention network to effectively capture long-range contextual information and generate a precise segmentation mask. Then, MCLN incorporates the input MRI image and the segmentation mask to recognize vertebral tumors utilizing a mixed kernel multi-loss network. We evaluate our proposed MCLN on a challenging dataset consisting of 600 MRI images, achieving impressive results with a mean Dice coefficient of 90.03% for vertebrae segmentation and a mean accuracy of 95.11% for tumor diagnosis. These results demonstrate the remarkable performance of MCLN in addressing automated vertebrae segmentation and tumor diagnosis from MRI.