Multi-task Deep Learning Method Research Articles

Magnetostrictive bi-perceptive flexible sensor for tracking bend and position of human and robot hand

The sensor that simultaneously perceives bending strain and magnetic field has the potential to detect the finger bending state and hand position of the human and robot. Based on unique magneto-mechanical coupling effect of magnetostrictive materials, the proposed a bi-perceptive flexible sensor, consisting of the Co–Fe film and magnetic sensing plane coils, can realize dual information perception of strain/magnetic field through the change of magnetization state. The sensor structure and interface circuit of the sensing system are designed to provide high sensitivity and fast response, based on the input–output characteristics of the simulation model. An asynchronous multi-task deep learning method is proposed, which takes the output of the position task as the partial input of the bending state task to analyze the output information of the sensor quickly and accurately. The sensing system, integrating with the proposed model, can better predict the bending state and approach distance of human or robot hand.

Detailed delineation of the fetal brain in diffusion MRI via multi-task learning.

Diffusion-weighted MRI is increasingly used to study the normal and abnormal development of fetal brain inutero. Recent studies have shown that dMRI can offer invaluable insights into the neurodevelopmental processes in the fetal stage. However, because of the low data quality and rapid brain development, reliable analysis of fetal dMRI data requires dedicated computational methods that are currently unavailable. The lack of automated methods for fast, accurate, and reproducible data analysis has seriously limited our ability to tap the potential of fetal brain dMRI for medical and scientific applications. In this work, we developed and validated a unified computational framework to (1) segment the brain tissue into white matter, cortical/subcortical gray matter, and cerebrospinal fluid, (2) segment 31 distinct white matter tracts, and (3) parcellate the brain's cortex and delineate the deep gray nuclei and white matter structures into 96 anatomically meaningful regions. We utilized a set of manual, semi-automatic, and automatic approaches to annotate 97 fetal brains. Using these labels, we developed and validated a multi-task deep learning method to perform the three computations. Our evaluations show that the new method can accurately carry out all three tasks, achieving a mean Dice similarity coefficient of 0.865 on tissue segmentation, 0.825 on white matter tract segmentation, and 0.819 on parcellation. The proposed method can greatly advance the field of fetal neuroimaging as it can lead to substantial improvements in fetal brain tractography, tract-specific analysis, and structural connectivity assessment.

Fault Diagnosis of the Rolling Bearing by a Multi-Task Deep Learning Method Based on a Classifier Generative Adversarial Network.

Accurate fault diagnosis is essential for the safe operation of rotating machinery. Recently, traditional deep learning-based fault diagnosis have achieved promising results. However, most of these methods focus only on supervised learning and tend to use small convolution kernels non-effectively to extract features that are not controllable and have poor interpretability. To this end, this study proposes an innovative semi-supervised learning method for bearing fault diagnosis. Firstly, multi-scale dilated convolution squeeze-and-excitation residual blocks are designed to exact local and global features. Secondly, a classifier generative adversarial network is employed to achieve multi-task learning. Both unsupervised and supervised learning are performed simultaneously to improve the generalization ability. Finally, supervised learning is applied to fine-tune the final model, which can extract multi-scale features and be further improved by implicit data augmentation. Experiments on two datasets were carried out, and the results verified the superiority of the proposed method.

Comprehensive assessment of the anterior segment in refraction corrected OCT based on multitask learning.

Anterior segment diseases are among the leading causes of irreversible blindness. However, a method capable of recognizing all important anterior segment structures for clinical diagnosis is lacking. By sharing the knowledge learned from each task, we proposed a fully automated multitask deep learning method that allows for simultaneous segmentation and quantification of all major anterior segment structures, including the iris, lens, cornea, as well as implantable collamer lens (ICL) and intraocular lens (IOL), and meanwhile for landmark detection of scleral spur and iris root in anterior segment OCT (AS-OCT) images. In addition, we proposed a refraction correction method to correct for the true geometry of the anterior segment distorted by light refraction during OCT imaging. 1251 AS-OCT images from 180 patients were collected and were used to train and test the model. Experiments demonstrated that our proposed network was superior to state-of-the-art segmentation and landmark detection methods, and close agreement was achieved between manually and automatically computed clinical parameters associated with anterior chamber, pupil, iris, ICL, and IOL. Finally, as an example, we demonstrated how our proposed method can be applied to facilitate the clinical evaluation of cataract surgery.

Predicting muscle invasion in bladder cancer based on MRI: A comparison of radiomics, and single-task and multi-task deep learning

Background and objectivesRadiomics and deep learning are two popular technologies used to develop computer-aided detection and diagnosis schemes for analysing medical images. This study aimed to compare the effectiveness of radiomics, single-task deep learning (DL) and multi-task DL methods in predicting muscle-invasive bladder cancer (MIBC) status based on T2-weighted imaging (T2WI). MethodsA total of 121 tumours (93 for training, from Centre 1; 28 for testing, from Centre 2) were included. MIBC was confirmed with pathological examination. A radiomics model, a single-task model, and a multi-task model based on T2WI were constructed in the training cohort with five-fold cross-validation, and validation was conducted in the external test cohort. Receiver operating characteristic (ROC) curve analysis was performed to evaluate the diagnostic performance of each model. DeLong's test and a permutation test were used to compare the performance of the models. ResultsThe area under the ROC curve (AUC) values of the radiomics, single-task and multi-task models in the training cohort were: 0.920, 0.933 and 0.932, respectively; and were 0.844, 0.884 and 0.932, respectively, in the test cohort. The multi-task model achieved better performance in the test cohort than did the other models. No statistically significant differences in AUC values and Kappa coefficients were observed between pairwise models, in either the training or test cohorts. According to the Grad-CAM feature visualization results, the multi-task model focused more on the diseased tissue area in some samples of the test cohort compared with the single-task model. ConclusionsThe T2WI-based radiomics, single-task, and multi-task models all exhibited good diagnostic performance in preoperatively predicting MIBC, in which the multi-task model had the best diagnostic performance. Compared with the radiomics method, our multi-task DL method had the advantage of saving time and effort. Compared with the single-task DL method, our multi-task DL method had the advantage of being more lesion-focused and more reliable for clinical reference.

Hair Follicle Classification and Hair Loss Severity Estimation Using Mask R-CNN.

Early and accurate detection of scalp hair loss is imperative to provide timely and effective treatment plans to halt further progression and save medical costs. Many techniques have been developed leveraging deep learning to automate the hair loss detection process. However, the accuracy and robustness of assessing hair loss severity still remain a challenge and barrier for transitioning such a technique into practice. The presented work proposes an efficient and accurate algorithm to classify hair follicles and estimate hair loss severity, which was implemented and validated using a multitask deep learning method via a Mask R-CNN framework. A microscopic image of the scalp was resized, augmented, then processed through pre-trained ResNet models for feature extraction. The key features considered in this study concerning hair loss severity include the number of hair follicles, the thickness of the hair, and the number of hairs in each hair follicle. Based on these key features, labeling of hair follicles (healthy, normal, and severe) were performed on the images collected from 10 men in varying stages of hair loss. More specifically, Mask R-CNN was applied for instance segmentation of the hair follicle region and to classify the hair follicle state into three categories, following the labeling convention (healthy, normal and severe). Based on the state of each hair follicle captured from a single image, an estimation of hair loss severity was determined for that particular region of the scalp, namely local hair loss severity index (P), and by combining P of multiple images taken and processed from different parts of the scalp, we constructed the hair loss severity estimation () and visualized in a heatmap to illustrate the overall hair loss type and condition. The proposed hair follicle classification and hair loss severity estimation using Mask R-CNN demonstrated a more efficient and accurate algorithm compared to other methods previously used, enhancing the classification accuracy by 4 to 15%. This performance supports its potential for use in clinical settings to enhance the accuracy and efficiency of current hair loss diagnosis and prognosis techniques.

Research on fault diagnosis of bearings in walking part of wall‐building robot based on roadside acoustic signal

AbstractThe walking part of the wall‐building robot is an important component whose health influences the robot's overall performance, and the bearing of the walking part is a critical but often disregarded component. As a result, based on acoustic signals from the roadside, this study proposes a method of bearing fault monitoring of walking parts to ensure the more stable operation of the wall‐building robot. To begin, due to the Doppler effect caused by the relative displacement between the sensor and the walking part, the collected acoustic signal has frequency shift and amplitude attenuation, so the multiple signal classification spectrum estimation method with added windows is first used to determine the position and speed of the walking part bearing in real time, and then obtain a resampled time series of the distorted signal. Second, the distorted signal is resampled by Morse acoustic theory to correct the distorted sound signal collected by the array sensor. Third, a multitask deep learning method with an attention mechanism is designed to jointly diagnose the fault type and damage degree of bearing by analyzing the corrected acoustic signal. The experimental results show that the method can effectively restore the distorted signal while still accurately and stably detecting the fault type and damage degree of the bearing under a variety of working conditions and external disturbances, making it more suitable for the work of wall‐building robots.

MTDeepM6A-2S: A two-stage multi-task deep learning method for predicting RNA N6-methyladenosine sites of Saccharomyces cerevisiae.

N6-methyladenosine (m6A) is one of the most important RNA modifications, which is involved in many biological activities. Computational methods have been developed to detect m6A sites due to their high efficiency and low costs. As one of the most widely utilized model organisms, many methods have been developed for predicting m6A sites of Saccharomyces cerevisiae. However, the generalization of these methods was hampered by the limited size of the benchmark datasets. On the other hand, over 60,000 low resolution m6A sites and more than 10,000 base resolution m6A sites of Saccharomyces cerevisiae are recorded in RMBase and m6A-Atlas, respectively. The base resolution m6A sites are often obtained from low resolution results by post calibration. In view of these, we proposed a two-stage deep learning method, named MTDeepM6A-2S, to predict RNA m6A sites of Saccharomyces cerevisiae based on RNA sequence information. In the first stage, a multi-task model with convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) deep framework was built to not only detect the low resolution m6A sites but also assign a reasonable probability for the predicted site. In the second stage, a transfer-learning strategy was used to build the model to predict the base resolution m6A sites from those low resolution m6A sites. The effectiveness of our model was validated on both training and independent test sets. The results show that our model outperforms other state-of-the-art models on the independent test set, which indicates that our model holds high potential to become a useful tool for epitranscriptomics analysis.

Multi-node load forecasting based on multi-task learning with modal feature extraction

Accurate multi-node load forecasting is the key to the safe, reliable, and economical operation of the power system. However, the dynamic nature of load and the coupling nature of networks are difficult to extract, making consistent and accurate forecasting of node load rather difficult. In this regard, this paper proposes a soft sharing multi-task deep learning method for multi-node load forecasting in the power system. It has the following aspects: (1) Considering the coupling characteristics of the node network, a multi-modal feature module, based on the inception strategy and gated temporal convolutional network (GTCN), is firstly designed to explore the coupling features implied in the node load data. (2) A novel multi-objective neural network model is proposed to achieve simultaneous prediction of multi-node load by integrating the multi-modal feature module and gated recurrent unit (GRU). For sharing the learning information of sub-networks, this paper uses the soft sharing mechanism to capture load features, which can better optimize the prediction task for each node load simultaneously. Load data from the New Zealand distribution network and AEMO are used to compare the proposed model’s performance in various scenarios using regression metrics such as mean absolute percentage error (MAPE), Weighted Mean Accuracy (WMA), root mean squared logarithmic error (RMSLE), and Diebold–Mariano (DM). The simulation results show that the proposed method can explore the spatial–temporal coupling characteristics in multi-node load data. Compared with existing state-of-the-art multi-node load prediction methods, our proposed method’s MAPE decrease 17.04% and 3.92% in Non-aggregation and Aggregation situations.

A neonatal dataset and benchmark for non-contact neonatal heart rate monitoring based on spatio-temporal neural networks

The digital revolution of noncontact physiological signal monitoring in clinical and home health care is underway, and deep learning techniques are incredibly popular. Camera-based physiological signal monitoring for adults has made considerable progress in recent years. However, most of existing methods and datasets are developed for adult subjects, and until now, there has been no neonatal public database that is collected for developing deep learning method. Thus, in this paper, we introduce a large-scale newborn baby database, named NBHR (newborn baby heart rate estimation database), to fill the abovementioned knowledge gap. A total of 9.6 h of clinical videos (1130 videos totaling 921 GB) and reference vital signs are recorded from 257 infants at 0–6 days old. The facial videos and corresponding synchronized physiological signals, including photoplethysmograph information, heart rate, and oxygen saturation level, are recorded in our database. This large-scale database could be used to develop deep learning methods to estimate heart rate or oxygen saturation levels. Furthermore, a multitask deep learning method, called NBHRnet, is proposed to estimate heart rate based on the NBHR database, and the model is succinct that it can be deployed on a computer without GPUs. The experimental results indicate that NBHRnet yields competitive performance in predicting infant heart rate, with a mean absolute error of 3.97 bpm and a mean absolute percentage error of 3.28%; additionally, it can estimate heart rate almost instantaneously (2 s/60 frames). Our datasets are freely publicly available by request.

A Multitask Deep-Learning Method for Predicting Membrane Associations and Secondary Structures of Proteins.

Prediction of residue-level structural attributes and protein-level structural classes helps model protein tertiary structures and understand protein functions. Existing methods are either specialized on only one class of proteins or developed to predict only a specific type of residue-level attribute. In this work, we develop a new deep-learning method, named Membrane Association and Secondary Structure Predictor (MASSP), for accurately predicting both residue-level structural attributes (secondary structure, location, orientation, and topology) and protein-level structural classes (bitopic, α-helical, β-barrel, and soluble). MASSP integrates a multilayer two-dimensional convolutional neural network (2D-CNN) with a long short-term memory (LSTM) neural network into a multitasking framework. Our comparison shows that MASSP performs equally well or better than the state-of-the-art methods in predicting residue-level secondary structures, boundaries of transmembrane segments, and topology. Furthermore, it achieves outstanding accuracy in predicting protein-level structural classes. MASSP automatically distinguishes the structural classes of input sequences and identifies transmembrane segments and topologies if present, making it broadly applicable to different classes of proteins. In summary, MASSP's good performance and broad applicability make it well suited for annotating residue-level attributes and protein-level structural classes at the proteome scale.

A New Multitask Learning Method for Tool Wear Condition and Part Surface Quality Prediction

Deep learning has been gradually used in the field of machining condition monitoring. However, at present only single-task prediction can be performed, which results in increased experimental costs, wasted datasets, and repetitive work. In this article, a new multitask learning method based on a deep belief network (DBN) is proposed, which can be used to predict the tool wear condition and part surface quality. The single-task data transmission of the last few hidden layers of the DBN network is improved to multitask parallel data transmission so that the improved DBN can realize multitask learning. The loss function of the multitask learning model is defined as the weighted sum of all single-task loss functions. According to the loss of different tasks in the iteration process, the weight of corresponding tasks can be adjusted automatically. Furthermore, the multitask deep learning method can realize information sharing, suppress overfitting, improve prediction accuracy, and require less computing time. Combined with the abovementioned improvements, a multitask model for tool wear and part surface quality was developed. Experimental verification was performed on a KVC850M three-axis vertical machining center. The results show that the accuracy of the proposed multitask prediction model is 99% for the tool wear prediction and 92.86% for part surface quality prediction.

Few-Shot Hyperspectral Image Classification With Unknown Classes Using Multitask Deep Learning

Current hyperspectral image classification assumes that a predefined classification system is closed and complete, and there are no unknown or novel classes in the unseen data. However, this assumption may be too strict for the real world. Often, novel classes are overlooked when the classification system is constructed. The closed nature forces a model to assign a label given a new sample and may lead to overestimation of known land covers (e.g., crop area). To tackle this issue, we propose a multitask deep learning method that simultaneously conducts classification and reconstruction in the open world (named MDL4OW) where unknown classes may exist. The reconstructed data are compared with the original data; those failing to be reconstructed are considered unknown, based on the assumption that they are not well represented in the latent features due to the lack of labels. A threshold needs to be defined to separate the unknown and known classes; we propose two strategies based on the extreme value theory for few-shot and many-shot scenarios. The proposed method was tested on real-world hyperspectral images; state-of-the-art results were achieved, e.g., improving the overall accuracy by 4.94% for the Salinas data. By considering the existence of unknown classes in the open world, our method achieved more accurate hyperspectral image classification, especially under the few-shot context.

Multi-task deep learning for cardiac rhythm detection in wearable devices

Wearable devices enable theoretically continuous, longitudinal monitoring of physiological measurements such as step count, energy expenditure, and heart rate. Although the classification of abnormal cardiac rhythms such as atrial fibrillation from wearable devices has great potential, commercial algorithms remain proprietary and tend to focus on heart rate variability derived from green spectrum LED sensors placed on the wrist, where noise remains an unsolved problem. Here we develop DeepBeat, a multitask deep learning method to jointly assess signal quality and arrhythmia event detection in wearable photoplethysmography devices for real-time detection of atrial fibrillation. The model is trained on approximately one million simulated unlabeled physiological signals and fine-tuned on a curated dataset of over 500 K labeled signals from over 100 individuals from 3 different wearable devices. We demonstrate that, in comparison with a single-task model, our architecture using unsupervised transfer learning through convolutional denoising autoencoders dramatically improves the performance of atrial fibrillation detection from a F1 score of 0.54 to 0.96. We also include in our evaluation a prospectively derived replication cohort of ambulatory participants where the algorithm performed with high sensitivity (0.98), specificity (0.99), and F1 score (0.93). We show that two-stage training can help address the unbalanced data problem common to biomedical applications, where large-scale well-annotated datasets are hard to generate due to the expense of manual annotation, data acquisition, and participant privacy.

A deep learning method for real-time intraoperative US image segmentation in prostate brachytherapy.

This paper addresses the detection of the clinical target volume (CTV) in transrectal ultrasound (TRUS) image-guided intraoperative for permanent prostate brachytherapy. Developing a robust and automatic method to detect the CTV on intraoperative TRUS images is clinically important to have faster and reproducible interventions that can benefit both the clinical workflow and patient health. We present a multi-task deep learning method for an automatic prostate CTV boundary detection in intraoperative TRUS images by leveraging both the low-level and high-level (prior shape) information. Our method includes a channel-wise feature calibration strategy for low-level feature extraction and learning-based prior knowledge modeling for prostate CTV shape reconstruction. It employs CTV shape reconstruction from automatically sampled boundary surface coordinates (pseudo-landmarks) to detect the low-contrast and noisy regions across the prostate boundary, while being less biased from shadowing, inherent speckles, and artifact signals from the needle and implanted radioactive seeds. The proposed method was evaluated on a clinical database of 145 patients who underwent permanent prostate brachytherapy under TRUS guidance. Our method achieved a mean accuracy of [Formula: see text] and a mean surface distance error of [Formula: see text]. Extensive ablation and comparison studies show that our method outperformed previous deep learning-based methods by more than 7% for the Dice similarity coefficient and 6.9 mm reduced 3D Hausdorff distance error. Our study demonstrates the potential of shape model-based deep learning methods for an efficient and accurate CTV segmentation in an ultrasound-guided intervention. Moreover, learning both low-level features and prior shape knowledge with channel-wise feature calibration can significantly improve the performance of deep learning methods in medical image segmentation.

Image Inpainting Detection Based on Multi-task Deep Learning Network

ABSTRACT Image inpainting can effectively repair damaged areas, but it can also be a way of image tampering when it is used to remove meaningful content from an image. Therefore, this paper focuses on the research of inpainting forensics, and proposes a multi-task deep learning method. In order to enhance the learning of texture features, the corresponding local binary pattern channels are added to the input of the network. Furthermore, considering that the multi-task object detection network Mask R-CNN cannot fully utilize the features of all scale feature information during the FPN feature extraction phase, the network in this paper combines Feature Pyramid Networks and back connections to extract more features. This network model can detect not only the images tampered by traditional inpainting methods, but also the images inpainted by deep learning methods. Experimental results on two large public data sets demonstrate the superior performance of the proposed method.

Answer Identification from Product Reviews for User Questions by Multi-Task Attentive Networks

Online Shopping has become a part of our daily routine, but it still cannot offer intuitive experience as store shopping. Nowadays, most e-commerce Websites offer a Question Answering (QA) system that allows users to consult other users who have purchased the product. However, users still need to wait patiently for others’ replies. In this paper, we investigate how to provide a quick response to the asker by plausible answer identification from product reviews. By analyzing the similarity and discrepancy between explicit answers and reviews that can be answers, a novel multi-task deep learning method with carefully designed attention mechanisms is developed. The method can well exploit large amounts of user generated QA data and a few manually labeled review data to address the problem. Experiments on data collected from Amazon demonstrate its effectiveness and superiority over competitive baselines.

Multitask Deep Learning With Spectral Knowledge for Hyperspectral Image Classification

In this letter, we propose a multitask deep learning method for classification of multiple hyperspectral data in a single training. Deep learning models have achieved promising results on hyperspectral image classification, but their performance highly rely on sufficient labeled samples, which are scarce on hyperspectral images. However, samples from multiple data sets might be sufficient to train one deep learning model, thereby improving its performance. To do so, we trained an identical feature extractor for all data, and the extracted features were fed into corresponding Softmax classifiers. Spectral knowledge was introduced to ensure that the shared features were similar across domains. Four hyperspectral data sets were used in the experiments. We achieved higher classification accuracies on three data sets (Pavia University, Pavia Center, and Indian Pines) and competitive results on the Salinas Valley data compared with the baseline. Spectral knowledge was useful to prevent the deep network from overfitting when the data shared similar spectral response. The proposed method tested on two deep CNNs successfully shows its ability to utilize samples from multiple data sets and enhance networks' performance.

Multi-task neural networks for joint hippocampus segmentation and clinical score regression

Feature representations extracted from hippocampus in magnetic resonance (MR) images are widely used in computer-aided Alzheimer’s disease (AD) diagnosis, and thus accurate segmentation for the hippocampus has been remaining an active research topic. Previous studies for hippocampus segmentation require either human annotation which is tedious and error-prone or pre-processing MR images via time-consuming non-linear registration. Although many automatic segmentation approaches have been proposed, their performance is often limited by the small size of hippocampus and complex confounding information around the hippocampus. In particular, human-engineered features extracted from segmented hippocampus regions (e.g., the volume of the hippocampus) are essential for brain disease diagnosis, while these features are independent of diagnosis models, leading to sub-optimal performance. To address these issues, we propose a multi-task deep learning (MDL) method for joint hippocampus segmentation and clinical score regression using MR images. The prominent advantages of our MDL method lie on that we don’t need any time-consuming non-linear registration for pre-processing MR images, and features generated by MDL are consistent with subsequent diagnosis models. Specifically, we first align all MR images onto a standard template, followed by a patch extraction process to approximately locate hippocampus regions in the template space. Using image patches as input data, we develop a multi-task convolutional neural network (CNN) for joint hippocampus segmentation and clinical score regression. The proposed CNN network contains two subnetworks, including 1) a U-Net with a Dice-like loss function for hippocampus segmentation, and 2) a convolutional neural network with a mean squared loss function for clinical regression. Note that these two subnetworks share a part of network parameters, to exploit the inherent association between these two tasks. We evaluate the proposed method on 407 subjects with MRI data from baseline Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. The experimental results suggest that our MDL method achieves promising results in both tasks of hippocampus segmentation and clinical score regression, compared with several state-of-the-art methods.