Multi-task Deep Learning Framework Research Articles

A multimodal multitask deep learning framework for vibrotactile feedback and sound rendering

Data-driven approaches are often utilized to model and generate vibrotactile feedback and sounds for rigid stylus-based interaction. Nevertheless, in prior research, these two modalities were typically addressed separately due to challenges related to synchronization and design complexity. To this end, we introduce a novel multimodal multitask deep learning framework. In this paper, we developed a comprehensive end-to-end data-driven system that encompasses the capture of contact acceleration signals and sound data from various texture surfaces. This framework introduces novel encoder-decoder networks for modeling and rendering vibrotactile feedback through an actuator while routing sound to headphones. The proposed encoder-decoder networks incorporate stacked transformers with convolutional layers to capture both local variability and overall trends within the data. To the best of our knowledge, this is the first attempt to apply transformer-based data-driven approach for modeling and rendering of vibrotactile signals as well as sounds during tool-surface interactions. In numerical evaluations, the proposed framework demonstrates a lower RMS error compared to state-of-the-art models for both vibrotactile signals and sound data. Additionally, subjective similarity evaluation also confirm the superiority of proposed method over state-of-the-art.

Component Identification and Depth Estimation for Structural Images Based on Multi-Scale Task Interaction Network

Component identification and depth estimation are important for detecting the integrity of post-disaster structures. However, traditional manual methods might be time-consuming, labor-intensive, and influenced by subjective judgments of inspectors. Deep-learning-based image visual inspection is a new approach to overcome these problems, but repeated modeling is required for different inspection tasks, which limits inspection accuracy and practical deployment efficiency. In this study, it is observed that the matched ratios of pixel pairs between component identification and depth estimation reach a high value, which indicates the dual tasks are highly related. Therefore, the Multi-Scale Task Interaction Network (MTI-Net) is proposed for structural images to simultaneously accomplish both tasks for accurate and efficient structural inspection. It propagates distilled task information from lower to higher scales. Then, it aggregates the refined task features from all scales to produce the final per-task predictions. Experiments show that MTI-Net delivers the full potential of multi-task learning, with a smaller memory footprint and higher efficiency compared to single-task learning. For the evaluation metrics of model performance, the mean Intersection over Union (mIoU) of component identification improves by 2.30, and root mean square error (RMSE) drops by 0.36 m with the aid of the multi-task strategy. The multi-task deep learning framework has great potential value in engineering applications.

A multi-task deep learning framework for forecasting sparse demand of demand responsive transit

Demand Responsive Transport (DRT) is a flexible transportation service that adjusts its routes and schedules in real time based on user requests. Real-time demand forecasting has become a timely issue since the resolution of transportation planning has increased to provide flexible services to users. Although many previous studies have been conducted to design the demand forecasting model using the mobility service historical data, there are more considerations when predicting real-time demand for on-demand mobility services. Therefore, this study aims to develop a demand forecasting model for DRT services addressing two issues. First, repeated calls for service and abandoned requests are converted into the unit of demand to capture served and potential demand. Second, given the sparsity of DRT historical data, a multi-task deep learning model is proposed to predict both the probability and volume of demand simultaneously to mitigate the data sparsity problem. A zero-inflated loss function was introduced to consider the skewed demand distribution. To attain our goals, the historical dataset of the I-MOD service trial in Yeongjongdo, Inchon, Korea was used. The performance of the proposed model is compared against several baseline models, demonstrating its superiority in forecasting demand for mobility services. The results show that a multi-task structure can alleviate the data sparsity problem by excluding where demand is not generated from the forecasting problem. The proposed models can provide insights into real-time demand forecasting with sparse demand.

Dual-Task Network for Terrace and Ridge Extraction: Automatic Terrace Extraction via Multi-Task Learning

Terrace detection and ridge extraction from high-resolution remote sensing imagery are crucial for soil conservation and grain production on sloping land. Traditional methods use low-to-medium resolution images, missing detailed features and lacking automation. Terrace detection and ridge extraction are closely linked, with each influencing the other’s outcomes. However, most studies address these tasks separately, overlooking their interdependence. This research introduces a cutting-edge, multi-scale, and multi-task deep learning framework, termed DTRE-Net, designed for comprehensive terrace information extraction. This framework bridges the gap between terrace detection and ridge extraction, executing them concurrently. The network incorporates residual networks, multi-scale fusion modules, and multi-scale residual correction modules to enhance the model’s robustness in feature extraction. Comprehensive evaluations against other deep learning-based semantic segmentation methods using GF-2 terraced imagery from two distinct areas were undertaken. The results revealed intersection over union (IoU) values of 85.18% and 86.09% for different terrace morphologies and 59.79% and 73.65% for ridges. Simultaneously, we have confirmed that the connectivity of results is improved when employing multi-task learning for ridge extraction compared to directly extracting ridges. These outcomes underscore DTRE-Net’s superior capability in the automation of terrace and ridge extraction relative to alternative techniques.

A YOLOX-based Deep Instance Segmentation Neural Network for Cardiac Anatomical Structures in Fetal Ultrasound Images.

Echocardiography is an essential procedure for the prenatal examination of the fetus for congenital heart disease (CHD). Accurate segmentation of key anatomical structures in a four-chamber view is an essential step in measuring fetal growth parameters and diagnosing CHD. Currently, most obstetricians perform segmentation tasks manually, but the pixel-level operation is labor-intensive and requires extensive anatomical knowledge and clinical experience. As such, efficiently and accurately detecting structures from real-world fetal ultrasound images is a key challenge. In this paper, we propose a YOLOX-based deep instance segmentation neural network (i.e., IS-YOLOX) for cardiac anatomical structure location and segmentation in fetal ultrasound images. Specifically, we reconstruct a new instance segmentation branch based on a multi-task deep learning framework. We then design a new multi-level non-maximum suppression (NMS) mechanism to further improve the segmentation performance that consists of three levels of selection. Moreover, unlike two-stage instance segmentation approaches, our method does not rely on object detection results. To the best of our knowledge, this is the first study regarding instance segmentation on 13 types of anatomical structures in the fetal four-chamber view. Extensive experiments were carried out on clinical datasets, and the experimental results show that our method outperforms nine competitive baselines.

A Multi-Task Deep Learning Framework for Simultaneous Detection of Thoracic Pathology through Image Classification

A Multi-Task Deep Learning Framework for Simultaneous Detection of Thoracic Pathology through Image Classification

A Multi-Task Deep Learning Framework Integrating Segmentation and Reconstruction for Lensless Imaging

A Multi-Task Deep Learning Framework Integrating Segmentation and Reconstruction for Lensless Imaging

Automatic detection, segmentation, and classification of primary bone tumors and bone infections using an ensemble multi-task deep learning framework on multi-parametric MRIs: a multi-center study.

To develop an ensemble multi-task deep learning (DL) framework for automatic and simultaneous detection, segmentation, and classification of primary bone tumors (PBTs) and bone infections based on multi-parametric MRI from multi-center. This retrospective study divided 749 patients with PBTs or bone infections from two hospitals into a training set (N = 557), an internal validation set (N = 139), and an external validation set (N = 53). The ensemble framework was constructed using T1-weighted image (T1WI), T2-weighted image (T2WI), and clinical characteristics for binary (PBTs/bone infections) and three-category (benign/intermediate/malignant PBTs) classification. The detection and segmentation performances were evaluated using Intersection over Union (IoU) and Dice score. The classification performance was evaluated using the receiver operating characteristic (ROC) curve and compared with radiologist interpretations. On the external validation set, the single T1WI-based and T2WI-based multi-task models obtained IoUs of 0.71 ± 0.25/0.65 ± 0.30 for detection and Dice scores of 0.75 ± 0.26/0.70 ± 0.33 for segmentation. The framework achieved AUCs of 0.959 (95%CI, 0.955-1.000)/0.900 (95%CI, 0.773-0.100) and accuracies of 90.6% (95%CI, 79.7-95.9%)/78.3% (95%CI, 58.1-90.3%) for the binary/three-category classification. Meanwhile, for the three-category classification, the performance of the framework was superior to that of three junior radiologists (accuracy: 65.2%, 69.6%, and 69.6%, respectively) and comparable to that of two senior radiologists (accuracy: 78.3% and 78.3%). The MRI-based ensemble multi-task framework shows promising performance in automatically and simultaneously detecting, segmenting, and classifying PBTs and bone infections, which was preferable to junior radiologists. Compared with junior radiologists, the ensemble multi-task deep learning framework effectively improves differential diagnosis for patients with primary bone tumors or bone infections. This finding may help physicians make treatment decisions and enable timely treatment of patients. • The ensemble framework fusing multi-parametric MRI and clinical characteristics effectively improves the classification ability of single-modality models. • The ensemble multi-task deep learning framework performed well in detecting, segmenting, and classifying primary bone tumors and bone infections. • The ensemble framework achieves an optimal classification performance superior to junior radiologists' interpretations, assisting the clinical differential diagnosis of primary bone tumors and bone infections.

Uncertainty-aware trustworthy weather-driven failure risk predictor for overhead contact lines

Overhead contact lines (OCLs) are electric transmission lines that power railways, which are constantly threatened by external weather and environmental factors due to their outdoor location. Hence, for the long-term functioning of railway lines, a weather-driven risk predictor is an essential tool. Current prediction methods mainly adopt a single-point estimation system with fixed weights of neural networks and therefore cannot propagate the uncertainties within the data and model, resulting in unreliable predictions. To enhance safety-risk prevention capabilities, this paper proposes an uncertainty-aware trustworthy weather-driven failure-risk approach for OCLs, in a probabilistic deep multitask learning framework. Firstly, a deep Gaussian process is employed as the backbone model to deal with imbalanced weather-related failure datasets with limited fault samples. Moreover, a multi-task learning framework is embedded to simultaneously predict the multiple weather-driven failure risks under lightning, wind and haze. Finally, the predictive uncertainty is decomposed into epistemic and aleatory uncertainties, where epistemic and aleatory uncertainties account for the uncertainty within the model and data, respectively. Extensive experiments on actual OCLs are carried out to demonstrate the effectiveness of the proposed approach, which can effectively capture the predictive uncertainty and provide trustworthy predictive decisions of mitigating operational risk for railway operators.

Effectiveness of multi-task deep learning framework for EEG-based emotion and context recognition

Effectiveness of multi-task deep learning framework for EEG-based emotion and context recognition

ROSes-FINDER: a multi-task deep learning framework for accurate prediction of microorganism reactive oxygen species scavenging enzymes.

Reactive oxygen species (ROS) are highly reactive molecules that play important roles in microbial biological processes. However, excessive accumulation of ROS can lead to oxidative stress and cellular damage. Microorganism have evolved a diverse suite of enzymes to mitigate the harmful effects of ROS. Accurate prediction of ROS scavenging enzymes classes (ROSes) is crucial for understanding the mechanisms of oxidative stress and developing strategies to combat related diseases. Nevertheless, the existing approaches for categorizing ROS-related proteins exhibit certain drawbacks with regards to their precision and inclusiveness. To address this, we propose a new multi-task deep learning framework called ROSes-FINDER. This framework integrates three component methods using a voting-based approach to predict multiple ROSes properties simultaneously. It can identify whether a given protein sequence is a ROSes and determine its type. The three component methods used in the framework are ROSes-CNN, which extracts raw sequence encoding features, ROSes-NN, which predicts protein functions based on sequence information, and ROSes-XGBoost, which performs functional classification using ensemble machine learning. Comprehensive experiments demonstrate the superior performance and robustness of our method. ROSes-FINDER is freely available at https://github.com/alienn233/ROSes-Finder for predicting ROSes classes.

Identifying the serious clinical outcomes of adverse reactions to drugs by a multi-task deep learning framework

Adverse Drug Reactions (ADRs) have a direct impact on human health. As continuous pharmacovigilance and drug monitoring prove to be costly and time-consuming, computational methods have emerged as promising alternatives. However, most existing computational methods primarily focus on predicting whether or not the drug is associated with an adverse reaction and do not consider the core issue of drug benefit-risk assessment—whether the treatment outcome is serious when adverse drug reactions occur. To this end, we categorize serious clinical outcomes caused by adverse reactions to drugs into seven distinct classes and present a deep learning framework, so-called GCAP, for predicting the seriousness of clinical outcomes of adverse reactions to drugs. GCAP has two tasks: one is to predict whether adverse reactions to drugs cause serious clinical outcomes, and the other is to infer the corresponding classes of serious clinical outcomes. Experimental results demonstrate that our method is a powerful and robust framework with high extendibility. GCAP can serve as a useful tool to successfully address the challenge of predicting the seriousness of clinical outcomes stemming from adverse reactions to drugs.

Two-stage multi-task deep learning framework for simultaneous pelvic bone segmentation and landmark detection from CT images.

Pelvic bone segmentation and landmark definition from computed tomography (CT) images are prerequisite steps for the preoperative planning of total hip arthroplasty. In clinical applications, the diseased pelvic anatomy usually degrades the accuracies of bone segmentation and landmark detection, leading to improper surgery planning and potential operative complications. This work proposes a two-stage multi-task algorithm to improve the accuracy of pelvic bone segmentation and landmark detection, especially for the diseased cases. The two-stage framework uses a coarse-to-fine strategy which first conducts global-scale bone segmentation and landmark detection and then focuses on the important local region to further refine the accuracy. For the global stage, a dual-task network is designed to share the common features between the segmentation and detection tasks, so that the two tasks mutually reinforce each other's performance. For the local-scale segmentation, an edge-enhanced dual-task network is designed for simultaneous bone segmentation and edge detection, leading to the more accurate delineation of the acetabulum boundary. This method was evaluated via threefold cross-validation based on 81 CT images (including 31 diseased and 50 healthy cases). The first stage achieved DSC scores of 0.94, 0.97, and 0.97 for the sacrum, left and right hips, respectively, and an average distance error of 3.24mm for the bone landmarks. The second stage further improved the DSC of the acetabulum by 5.42%, and this accuracy outperforms the state-of-the-arts (SOTA) methods by 0.63%. Our method also accurately segmented the diseased acetabulum boundaries. The entire workflow took ~ 10s, which was only half of the U-Net run time. Using the multi-task networks and the coarse-to-fine strategy, this method achieved more accurate bone segmentation and landmark detection than the SOTA method, especially for diseased hip images. Our work contributes to accurate and rapid design of acetabular cup prostheses.

Lane-changing trajectory prediction based on multi-task learning

Abstract As a complex driving behaviour, lane-changing (LC) behaviour has a great influence on traffic flow. Improper lane-changing behaviour often leads to traffic accidents. Numerous studies are currently being conducted to predict lane-change trajectories to minimize dangers. However, most of their models focus on how to optimize input variables without considering the interaction between output variables. This study proposes an LC trajectory prediction model based on a multi-task deep learning framework to improve driving safety. Concretely, in this work, the coupling effect of lateral and longitudinal movement is considered in the LC process. Trajectory changes in two directions will be modelled separately, and the information interaction is completed under the multi-task learning framework. In addition, the trajectory fragments are clustered by the driving features, and trajectory type recognition is added to the trajectory prediction framework as an auxiliary task. Finally, the prediction process of lateral and longitudinal trajectory and LC style is completed by long short-term memory (LSTM). The model training and testing are conducted with the data collected by the driving simulator, and the proposed method expresses better performance in LC trajectory prediction compared with several traditional models. The results of this study can enhance the trajectory prediction accuracy of advanced driving assistance systems (ADASs) and reduce the traffic accidents caused by lane changes.

An Interpretable Multitask Framework BiLAT Enables Accurate Prediction of Cyclin-Dependent Protein Kinase Inhibitors.

The cyclin-dependent protein kinases (CDKs) are protein-serine/threonine kinases with crucial effects on the regulation of cell cycle and transcription. CDKs can be a hallmark of cancer since their excessive expression could lead to impaired cell proliferation. However, the selectivity profile of most developed CDK inhibitors is not enough, which have hindered the therapeutic use of CDK inhibitors. In this study, we propose a multitask deep learning framework called BiLAT based on SMILES representation for the prediction of the inhibitory activity of molecules on eight CDK subtypes (CDK1, 2, 4-9). The framework is mainly composed of an improved bidirectional long short-term memory module BiLSTM and the encode layer of the Transformer framework. Additionally, the data enhancement method of SMILES enumeration is applied to improve the performance of the model. Compared with baseline predictive models based on three conventional machine learning methods and two multitask deep learning algorithms, BiLAT achieves the best performance with the highest average AUC, ACC, F1-score, and MCC values of 0.938, 0.894, 0.911, and 0.715 for the test set. Moreover, we constructed a targeted external data set CDK-Dec for the CDK family, which mainly contains bait values screened by 3D similarity with active compounds. This dataset was utilized in the subsequent evaluation of our model. It is worth mentioning that the BiLAT model is interpretable and can be used by chemists to design and synthesize compounds with improved activity. To further verify the generalization ability of the multitask BiLAT model, we also conducted another evaluation on three public datasets (Tox21, ClinTox, and SIDER). Compared with several currently popular models, BiLAT shows the best performance on two datasets. These results indicate that BiLAT is an effective tool for accelerating drug discovery.

Identifying plant disease and severity from leaves: A deep multitask learning framework using triple-branch Swin Transformer and deep supervision

Accurate identification of plant disease is of great significance for intelligent agriculture. Currently, plant species, disease, and severity are considered as joint categories in most disease classification methods, which will increase the number of categories and decrease the generalization ability of these models. Compared to disease and severity, information on plant species is less important because different species may suffer from the same disease, and such information is usually known to users. Given this, this paper proposed a novel triple-branch Swin Transformer classification (TSTC) network for classification of disease and severity simultaneously and separately. The TSTC network consists of a multitask feature extraction module, a feature fusion module and a deep supervision module. Firstly, preliminary features are extracted using a triple-branch network, which is built based on Swin Transformer backbone under the multitask classification strategy (i.e., one for disease classification, one for severity classification and the other for deep supervision). After that, these features are fused using compact bilinear pooling technique to enhance the feature extractor’s learning ability and thus more discriminative features can be extracted. Finally, the deep supervision module combines losses from both hidden layers and the last layers of the TSTC so that it can be trained in the direction where all layers can work efficiently for disease and severity classifications. Compared to five widely used classification networks, experiments with the AI Challenger 2018 dataset shows that our proposed TSTC network achieves the highest accuracy with an overall accuracy of 99.00% for disease classification and 88.73% for severity classification.

MARSY: a multitask deep-learning framework for prediction of drug combination synergy scores.

Combination therapies have emerged as a treatment strategy for cancers to reduce the probability of drug resistance and to improve outcomes. Large databases curating the results of many drug screening studies on preclinical cancer cell lines have been developed, capturing the synergistic and antagonistic effects of combination of drugs in different cell lines. However, due to the high cost of drug screening experiments and the sheer size of possible drug combinations, these databases are quite sparse. This necessitates the development of transductive computational models to accurately impute these missing values. Here, we developed MARSY, a deep-learning multitask model that incorporates information on the gene expression profile of cancer cell lines, as well as the differential expression signature induced by each drug to predict drug-pair synergy scores. By utilizing two encoders to capture the interplay between the drug pairs, as well as the drug pairs and cell lines, and by adding auxiliary tasks in the predictor, MARSY learns latent embeddings that improve the prediction performance compared to state-of-the-art and traditional machine-learning models. Using MARSY, we then predicted the synergy scores of 133722 new drug-pair cell line combinations, which we have made available to the community as part of this study. Moreover, we validated various insights obtained from these novel predictions using independent studies, confirming the ability of MARSY in making accurate novel predictions. An implementation of the algorithms in Python and cleaned input datasets are provided in https://github.com/Emad-COMBINE-lab/MARSY.

GSN-HVNET: A Lightweight, Multi-Task Deep Learning Framework for Nuclei Segmentation and Classification.

Nuclei segmentation and classification are two basic and essential tasks in computer-aided diagnosis of digital pathology images, and those deep-learning-based methods have achieved significant success. Unfortunately, most of the existing studies accomplish the two tasks by splicing two related neural networks directly, resulting in repetitive computation efforts and a redundant-and-large neural network. Thus, this paper proposes a lightweight deep learning framework (GSN-HVNET) with an encoder-decoder structure for simultaneous segmentation and classification of nuclei. The decoder consists of three branches outputting the semantic segmentation of nuclei, the horizontal and vertical (HV) distances of nuclei pixels to their mass centers, and the class of each nucleus, respectively. The instance segmentation results are obtained by combing the outputs of the first and second branches. To reduce the computational cost and improve the network stability under small batch sizes, we propose two newly designed blocks, Residual-Ghost-SN (RGS) and Dense-Ghost-SN (DGS). Furthermore, considering the practical usage in pathological diagnosis, we redefine the classification principle of the CoNSeP dataset. Experimental results demonstrate that the proposed model outperforms other state-of-the-art models in terms of segmentation and classification accuracy by a significant margin while maintaining high computational efficiency.

Development of a deep learning-based tool to assist wound classification

This paper presents a deep learning-based wound classification tool that can assist medical personnel in non-wound care specialization to classify five key wound conditions, namely deep wound, infected wound, arterial wound, venous wound, and pressure wound, given color images captured using readily available cameras. The accuracy of the classification is vital for appropriate wound management. The proposed wound classification method adopts a multi-task deep learning framework that leverages the relationships among the five key wound conditions for a unified wound classification architecture. With differences in Cohen's kappa coefficients as the metrics to compare our proposed model with humans, the performance of our model was better or non-inferior to those of all human medical personnel. Our convolutional neural network-based model is the first to classify five tasks of deep, infected, arterial, venous, and pressure wounds simultaneously with good accuracy. The proposed model is compact and matches or exceeds the performance of human doctors and nurses. Medical personnel who do not specialize in wound care can potentially benefit from an app equipped with the proposed deep learning model.

Bridging structural MRI with cognitive function for individual level classification of early psychosis via deep learning.

Recent efforts have been made to apply machine learning and deep learning approaches to the automated classification of schizophrenia using structural magnetic resonance imaging (sMRI) at the individual level. However, these approaches are less accurate on early psychosis (EP) since there are mild structural brain changes at early stage. As cognitive impairments is one main feature in psychosis, in this study we apply a multi-task deep learning framework using sMRI with inclusion of cognitive assessment to facilitate the classification of patients with EP from healthy individuals. Unlike previous studies, we used sMRI as the direct input to perform EP classifications and cognitive estimations. The proposed deep learning model does not require time-consuming volumetric or surface based analysis and can provide additionally cognition predictions. Experiments were conducted on an in-house data set with 77 subjects and a public ABCD HCP-EP data set with 164 subjects. We achieved 74.9 ± 4.3% five-fold cross-validated accuracy and an area under the curve of 71.1 ± 4.1% on EP classification with the inclusion of cognitive estimations. We reveal the feasibility of automated cognitive estimation using sMRI by deep learning models, and also demonstrate the implicit adoption of cognitive measures as additional information to facilitate EP classifications from healthy controls.