Multi-task Deep Learning Research Articles

Simultaneous Prediction of Wheat Yield and Grain Protein Content Using Multitask Deep Learning from Time-Series Proximal Sensing.

Wheat yield and grain protein content (GPC) are two main optimization targets for breeding and cultivation. Remote sensing provides nondestructive and early predictions of yield and GPC, respectively. However, whether it is possible to simultaneously predict yield and GPC in one model and the accuracy and influencing factors are still unclear. In this study, we made a systematic comparison of different deep learning models in terms of data fusion, time-series feature extraction, and multitask learning. The results showed that time-series data fusion significantly improved yield and GPC prediction accuracy with R2 values of 0.817 and 0.809. Multitask learning achieved simultaneous prediction of yield and GPC with comparable accuracy to the single-task model. We further proposed a two-to-two model that combines data fusion (two kinds of data sources for input) and multitask learning (two outputs) and compared different feature extraction layers, including RNN (recurrent neural network), LSTM (long short-term memory), CNN (convolutional neural network), and attention module. The two-to-two model with the attention module achieved the best prediction accuracy for yield (R2 = 0.833) and GPC (R2 = 0.846). The temporal distribution of feature importance was visualized based on the attention feature values. Although the temporal patterns of structural traits and spectral traits were inconsistent, the overall importance of both structural traits and spectral traits at the postanthesis stage was more important than that at the preanthesis stage. This study provides new insights into the simultaneous prediction of yield and GPC using deep learning from time-series proximal sensing, which may contribute to the accurate and efficient predictions of agricultural production.

Multitask deep learning with dynamic task balancing for quantum mechanical properties prediction.

Predicting quantum mechanical properties (QMPs) is very important for the innovation of material and chemistry science. Multitask deep learning models have been widely used in QMPs prediction. However, existing multitask learning models often train multiple QMPs prediction tasks simultaneously without considering the internal relationships and differences between tasks, which may cause the model to overfit easy tasks. In this study, we first proposed a multiscale dynamic attention graph neural network (MDGNN) for molecular representation learning. The MDGNN was designed in a multitask learning fashion that can solve multiple learning tasks at the same time. We then introduced a dynamic task balancing (DTB) strategy combining task differences and difficulties to reduce overfitting across multiple tasks. Finally, we adopted gradient-weighted class activation mapping (Grad-CAM) to analyze a deep learning model for frontier molecular orbital, highest occupied molecular orbital (HOMO) and lowest unoccupied molecular orbital (LUMO) energy level predictions. We evaluated our approach using two large QMPs datasets and compared the proposed method to the state-of-the-art multitask learning models. The MDGNN outperforms other multitask learning approaches on two datasets. The DTB strategy can further improve the performance of MDGNN significantly. Moreover, we show that Grad-CAM creates explanations that are consistent with the molecular orbitals theory. These advantages demonstrate that the proposed method improves the generalization and interpretation capability of QMPs prediction modeling.

Outcome and biomarker supervised deep learning for survival prediction in two multicenter breast cancer series

BackgroundPrediction of clinical outcomes for individual cancer patients is an important step in the disease diagnosis and subsequently guides the treatment and patient counseling. In this work, we develop and evaluate a joint outcome and biomarker supervised (estrogen receptor expression and ERBB2 expression and gene amplification) multitask deep learning model for prediction of outcome in breast cancer patients in two nation-wide multicenter studies in Finland (the FinProg and FinHer studies). Our approach combines deep learning with expert knowledge to provide more accurate, robust, and integrated prediction of breast cancer outcomes. Materials and methodsUsing deep learning, we trained convolutional neural networks (CNNs) with digitized tissue microarray (TMA) samples of primary hematoxylin-eosin-stained breast cancer specimens from 693 patients in the FinProg series as input and breast cancer-specific survival as the endpoint. The trained algorithms were tested on 354 TMA patient samples in the same series. An independent set of whole-slide (WS) tumor samples from 674 patients in another multicenter study (FinHer) was used to validate and verify the generalization of the outcome prediction based on CNN models by Cox survival regression and concordance index (c-index). Visual cancer tissue characterization, i.e., number of mitoses, tubules, nuclear pleomorphism, tumor-infiltrating lymphocytes, and necrosis was performed on TMA samples in the FinProg test set by a pathologist and combined with deep learning-based outcome prediction in a multitask algorithm. ResultsThe multitask algorithm achieved a hazard ratio (HR) of 2.0 (95% confidence interval [CI] 1.30–3.00), P < 0.001, c-index of 0.59 on the 354 test set of FinProg patients, and an HR of 1.7 (95% CI 1.2–2.6), P = 0.003, c-index 0.57 on the WS tumor samples from 674 patients in the independent FinHer series. The multitask CNN remained a statistically independent predictor of survival in both test sets when adjusted for histological grade, tumor size, and axillary lymph node status in a multivariate Cox analyses. An improved accuracy (c-index 0.66) was achieved when deep learning was combined with the tissue characteristics assessed visually by a pathologist. ConclusionsA multitask deep learning algorithm supervised by both patient outcome and biomarker status learned features in basic tissue morphology predictive of survival in a nationwide, multicenter series of patients with breast cancer. The algorithms generalized to another independent multicenter patient series and whole-slide breast cancer samples and provide prognostic information complementary to that of a comprehensive series of established prognostic factors.

Constructing Time-Series Momentum Portfolios with Deep Multi-Task Learning

Constructing Time-Series Momentum Portfolios with Deep Multi-Task Learning

Plasticity-Stability Preserving Multi-Task Learning for Remote Sensing Image Retrieval

Deep learning-based multi-task learning (MTL) methods have recently attracted attention for content-based image retrieval (CBIR) applications in remote sensing (RS). For a given set of tasks (e.g., scene classification, semantic segmentation, and image reconstruction), existing MTL methods employ a joint optimization algorithm on the direct aggregation of task-specific loss functions. Such an approach may provide limited CBIR performance when: 1) tasks compete or even distract each other; 2) one of the tasks dominates the whole learning procedure; or 3) characterization of each task is underperformed compared to single-task learning. This is mainly due to the lack of: 1) plasticity condition (which is associated with sensitivity to new information) or 2) stability condition (which is associated with protection from radical disruptions by new information) of the whole learning procedure. To avoid this issue, as a first time, we propose a novel plasticity-stability preserving MTL (PLASTA-MTL) approach to ensure the plasticity and the stability conditions of the whole learning procedure independently of the number and type of tasks. This is achieved by defining two novel loss functions. The first loss function is the plasticity preserving loss (PPL) function that aims to enforce the global image representation space to be sensitive to new information learned with each task. This is achieved by minimizing the difference of gradient magnitudes for the global representation and task-specific embedding spaces. The second loss function is the stability preserving loss (SPL) function that aims to protect the global representation space radically disrupted by a new task. This is achieved by minimizing the angular distances between the task gradients over global representation space. To effectively employ the proposed loss functions, we also introduce a novel sequential optimization algorithm. Experimental results show the effectiveness of the proposed approach compared to the state-of-the-art MTL methods in the context of CBIR.

Multitask Deep Learning for Segmentation and Lumbosacral Spine Inspection

Multi-task learning has achieved notable progress in many medical applications. In this paper, we propose a multi-task neural network, MRNet, for segmentation and spinal parameter inspection. It is developed based on the multi-path convolutional neural network for the robust detection of obscure regions on X-ray images. The proposed MRNet has two branches. One is for the segmentation of lumbar vertebrae, sacrum, and femoral heads. It shares the main features with the second branch for detection and classification by supervised learning. The output of the second branch is used to estimate the parameters for lumbosacral spine inspection. We conduct this research on our dataset collected and annotated by doctors for model training and performance evaluation. The datasets are used to train our MRNet as well as other networks for performance evaluation. Compared to the state-of-the-art techniques, the proposed MRNet is capable of X-ray image segmentation and parameter estimation with very limited training data. The results have demonstrated the feasibility of our MRNet for the segmentation of lumbar vertebrae, as well as the automated parameter prediction for lumbosacral spine inspection. Code is available at https://github.com/LuanTran07/BiLUnet-Lumbar-Spine.

Precipitation Retrieval From Fengyun-3D MWHTS and MWRI Data Using Deep Learning

In this paper, two multitask deep learning models, Multi-layer Perceptron (MLP) and Convolutional Neural Networks (CNN) are constructed to detect precipitation flags and retrieve precipitation rates simultaneously over the Northwest Pacific area. The retrieval results are verified by Integrated Multi-satellite Retrievals for GPM (IMERG). Compared with using a single payload, the results show the retrieval advantages of incorporating two microwave spaceborne payloads (MWHTS and MWRI). In addition, each separated task in multitasking deep learning models can outperform the single-task models. The MAE values of MLP and CNN are decreased by 0.05 mm/hr and 0.04 mm/hr, respectively. Nevertheless, it is still a challenge to extend the application of these models, which demands extracting the features learned from a certain area to other areas characterized by different precipitation features. A flexible network framework is developed with two Transfer Learning (TL) methods (freeze and fine-tuning) to tackle this problem. The pre-trained models (MLP and CNN) are trained from the Northwest Pacific area, whereas the transferred models are applied to the Northeast Pacific area. The encouraging results show that the performance of the TL model using fine-tune method has exceeded that of the comparison model without TL, which is trained by 4 times more data. The main advantage of using TL is the time reduction and efficiency improvements.

Tropical Cyclone Forecast Using Multitask Deep Learning Framework

A tropical cyclone is a robust weather system that affects human daily life. Accurate and rapid tropical cyclone forecast can guide human disaster prevention and mitigation work against tropical cyclones. The mainstream tropical cyclone forecasting method is numerical forecasting, which requires abundant prior knowledge and luxurious calculation. Nowadays, machine learning methods have received increasing attention for which they can overcome these disadvantages. However, existing machine learning methods usually ignored some potential factors since they mainly concentrated on one aspect of the tropical cyclone forecast. This letter proposes a multitask machine learning framework to forecast tropical cyclone path and intensity, which possesses two modules: one is the prediction module and the other is the estimate module. We use an improved generative adversarial network as the prediction module to predict the tropical cyclone spatial data at a certain moment in the future. Then, we use two different deep neural networks as the estimation module to extract the position and intensity from the generated prediction data. The method we propose is a general and relatively accurate tropical cyclone forecast method. We reach a 24-h path forecast error of 116 km and a 24-h intensity forecast error of 13.06 kt.

Accelerating Seismic Dip Estimation With Deep Learning

The seismic volumetric dip is a crucial seismic geometric attribute, which can provide useful information for assisting subsequent processing and interpretation. Waveform similarity scanning-based dip estimation (WSSB) delivers reliable dip estimation but encounters problems of expensive computation. To improve computing efficiency, we use multitask deep learning to simultaneously estimate the inline dip and crossline dip directly from a 3-D field seismic dataset. Our method considers dip estimation as a regression problem and trains a multilayer convolutional neural network with dual-channel output. It aims to output continuous values of seismic apparent dip from two directions simultaneously. To train the network, we propose an effective and efficient workflow to create a training sample dataset, which consists of field seismic cubes and the corresponding dip labels estimated by WSSB. After training, the network automatically learns how to extract rich and proper features that are important for dip estimation. By sliding the extraction window within the full 3-D seismic data, the network can output many overlapping dip cubes that are stacked to get two complete 3-D volumes of seismic dip. The final results of dip estimation by our method are similar to those by WSSB. We further demonstrate the accuracy of our approach by comparing the structural curvature. However, the computation time of our method is much less than that of WSSB. The proposed method can accurately estimate seismic volumetric dips with high computational efficiency.

Multi-Task Learning for Low-Frequency Extrapolation and Elastic Model Building From Seismic Data

Low-frequency signal content in seismic data as well as a realistic initial model are key ingredients for robust and efficient full-waveform inversions. However, acquiring low-frequency data is challenging in practice for active seismic surveys. Data-driven solutions show promise to extrapolate low-frequency data given a high-frequency counterpart. While being established for synthetic acoustic examples, the application of bandwidth extrapolation to field datasets remains non-trivial. Rather than aiming to reach superior accuracy in bandwidth extrapolation, we propose to jointly reconstruct low-frequency data and a smooth background subsurface model within a multi-task deep learning framework. We automatically balance data, model and trace-wise correlation loss terms in the objective functional and show that this approach improves the extrapolation capability of the network. We also design a pipeline for generating synthetic data suitable for field data applications. Finally, we apply the same trained network to synthetic and real marine streamer datasets and run an elastic full-waveform inversion from the extrapolated dataset.

Multi-Task Deep Learning Methods for Determining Railway Major Technical Standards

Multi-Task Deep Learning Methods for Determining Railway Major Technical Standards

A Multitask Deep Learning for Simultaneous Denoising and Inversion of 3-D Gravity Data

Noise present in real gravity data can lead to inaccurate inversion results. Multi-task strategy in deep learning provides a promising method to solve this problem. In this study, a multi-task framework is proposed for simultaneous inversion and denoising of noisy gravity data, in which the denoising task can constrain the inversion task. To extract multi-scale field information for high precision inversion, a novel backbone network, known as the cross-dimensional UNet (CDUNet), is proposed. CDUNet employs cross-dimensional skip-connections to transfer different scale field features, wherein transformation modules are used to convert two dimensional (2D) features extracted from gravity data to 3D features for density model reconstruction. A noisy dataset was synthesized to train the network, which comprised diverse density models having highly random geometric and physical characteristics. The test set evaluations showed that CDUNet and the multi-task framework could integrate well and the inversion accuracy of the network could reach 70.8% over the density perturbation area and 97% over the entire area. The synthetic examples showed that the inverted models characterized by distinct boundaries and relatively accurate values. Finally, the method was validated using real data from the Vinton salt dome in Texas and Louisiana, USA.

Multi-label Multi-task Deep Learning for Behavioral Coding.

We propose a methodology for estimating human behaviors in psychotherapy sessions using mutli-label and multi-task learning paradigms. We discuss the problem of behavioral coding in which data of human interactions is the annotated with labels to describe relevant human behaviors of interest. We describe two related, yet distinct, corpora consisting of therapist client interactions in psychotherapy sessions. We experimentally compare the proposed learning approaches for estimating behaviors of interest in these datasets. Specifically, we compare single and multiple label learning approaches, single and multiple task learning approaches, and evaluate the performance of these approaches when incorporating turn context. We demonstrate the prediction performance gains which can be achieved by using the proposed paradigms and discuss the insights these models provide into these complex interactions.

Decision Fusion Approach for Detecting Unknown Wafer Bin Map Patterns Based on a Deep Multitask Learning Model

Decision Fusion Approach for Detecting Unknown Wafer Bin Map Patterns Based on a Deep Multitask Learning Model

A MULTITASK DEEP-LEARNING SYSTEM FOR ASSESSMENT OF DIABETIC MACULAR ISCHEMIA ON OPTICAL COHERENCE TOMOGRAPHY ANGIOGRAPHY IMAGES.

We aimed to develop and test a deep-learning system to perform image quality and diabetic macular ischemia (DMI) assessment on optical coherence tomography angiography (OCTA) images. This study included 7,194 OCTA images with diabetes mellitus for training and primary validation and 960 images from three independent data sets for external testing. A trinary classification for image quality assessment and the presence or absence of DMI for DMI assessment were labeled on all OCTA images. Two DenseNet-161 models were built for both tasks for OCTA images of superficial and deep capillary plexuses, respectively. External testing was performed on three unseen data sets in which one data set using the same model of OCTA device as of the primary data set and two data sets using another brand of OCTA device. We assessed the performance by using the area under the receiver operating characteristic curves with sensitivities, specificities, and accuracies and the area under the precision-recall curves with precision. For the image quality assessment, analyses for gradability and measurability assessment were performed. Our deep-learning system achieved the area under the receiver operating characteristic curves >0.948 and area under the precision-recall curves >0.866 for the gradability assessment, area under the receiver operating characteristic curves >0.960 and area under the precision-recall curves >0.822 for the measurability assessment, and area under the receiver operating characteristic curves >0.939 and area under the precision-recall curves >0.899 for the DMI assessment across three external validation data sets. Grad-CAM demonstrated the capability of our deep-learning system paying attention to regions related to DMI identification. Our proposed multitask deep-learning system might facilitate the development of a simplified assessment of DMI on OCTA images among individuals with diabetes mellitus at high risk for visual loss.

A Multi-task Deep Learning System for Face Detection and Age Group Classification for Masked Faces

COVID-19 is an ongoing pandemic and according to the experts, using a face mask can reduce the spread of the disease. On the other hand, masks cause occlusion in faces and can create safety problems such as the recognition of the face and the estimation of its age. To prevent the spread of COVID-19, some countries have restrictions according to age groups. Also in different countries, people in some age groups have safety restrictions such as driving and consuming alcohol, etc. But these rules are difficult to follow due to occlusion in faces. Automated systems can assist to monitor these rules. In this study, a deep learning-based automated multi-task face detection and age group classification system is proposed for masked faces. The system first detects masked/no-masked-faces. Then, it classifies them according to age-groups. It works for multi-person regardless of indoor/outdoor environment. The system achieved 79.0% precision score for masked face detection using Faster R-CNN with resnet50 network. Also, 83.87% accuracy for classifying age groups with masked faces and 84.48% accuracy for no-masked faces using densenet201 network have been observed. It produced better results compared to the literature. The results are significant because they show that a reliable age classification for masked faces is possible.

Multi-Task Model for Esophageal Lesion Analysis Using Endoscopic Images: Classification with Image Retrieval and Segmentation with Attention.

The automatic analysis of endoscopic images to assist endoscopists in accurately identifying the types and locations of esophageal lesions remains a challenge. In this paper, we propose a novel multi-task deep learning model for automatic diagnosis, which does not simply replace the role of endoscopists in decision making, because endoscopists are expected to correct the false results predicted by the diagnosis system if more supporting information is provided. In order to help endoscopists improve the diagnosis accuracy in identifying the types of lesions, an image retrieval module is added in the classification task to provide an additional confidence level of the predicted types of esophageal lesions. In addition, a mutual attention module is added in the segmentation task to improve its performance in determining the locations of esophageal lesions. The proposed model is evaluated and compared with other deep learning models using a dataset of 1003 endoscopic images, including 290 esophageal cancer, 473 esophagitis, and 240 normal. The experimental results show the promising performance of our model with a high accuracy of 96.76% for the classification and a Dice coefficient of 82.47% for the segmentation. Consequently, the proposed multi-task deep learning model can be an effective tool to help endoscopists in judging esophageal lesions.

Joint bidding and pricing for electricity retailers based on multi-task deep reinforcement learning

The single-task deep reinforcement learning (STDRL)-based methods solve the joint bidding and pricing problem for the electricity retailer in a hierarchical electricity market by defining a bidding policy and a pricing policy separately, which may suffer from low learning efficiency, time-consuming training and local optimization. To deal with these issues, this paper proposes a novel Multi-task Deep reinforcement learning approach for Joint Bidding and Pricing (MDJBP) optimization model. MDJBP can deal with the bidding and pricing tasks concurrently through a shared long short-term memory (LSTM) representation network to distill meaningful temporal characteristics from high-dimensional environment states. Furthermore, we develop a deep neural network (DNN) structure consisting a regression branch for bidding task and a soft actor-critic (SAC) branch for pricing task with automating entropy adjustment and adaptive loss weighting to implement MDJBP. The proposed multi-task deep reinforcement learning (MTDRL)-based method is tested with the IEEE 30-bus system. Numerical results show that the proposed methodology succeeds in giving an optimal joint bidding and pricing policy by fully exploiting commonalities and differences between bidding task and pricing task, and thereby boosts the profit, improves learning efficiency, reduces training time, and enhances stability.

Multi-Task Deep Learning Model with an Attention Mechanism for Ship Accident Sentence Prediction

The number of ship accidents occurring in the Korean ocean has been steadily increasing year by year. The Korea Maritime Safety Tribunal (KMST) has published verdicts to ensure that the relevant personnel can share judgment on these accidents. As of 2020, there have been 3156 ship accidents; thus, it is difficult for the relevant personnel to study these various accidents by only reading the verdicts. Therefore, in this study, we propose a multi-task deep learning model with an attention mechanism for predicting the sentencing of ship accidents. The tasks are accident types, applied articles, and the sentencing of ship accidents. The proposed model was tested under verdicts published by the KMST between 2010 and 2019. Through experiments, we show that the proposed model can improve the performance of sentence prediction and can assist the relevant personnel to study these accidents.

3D deformation measurement in digital holographic interferometry using a multitask deep learning architecture.

The extraction of absolute phase from an interference pattern is a key step for 3D deformation measurement in digital holographic interferometry (DHI) and is an ill-posed problem. Estimating the absolute unwrapped phase becomes even more challenging when the obtained wrapped phase from the interference pattern is noisy. In this paper, we propose a novel multitask deep learning approach for phase reconstruction and 3D deformation measurement in DHI, referred to as TriNet, that has the capability to learn and perform two parallel tasks from the input image. The proposed TriNet has a pyramidal encoder-two-decoder framework for multi-scale information fusion. To our knowledge, TriNet is the first multitask approach to accomplish simultaneous denoising and phase unwrapping of the wrapped phase from the interference fringes in a single step for absolute phase reconstruction. The proposed architecture is more elegant than recent multitask learning methods such as Y-Net and state-of-the-art segmentation approaches such as UNet++. Further, performing denoising and phase unwrapping simultaneously enables deformation measurement from the highly noisy wrapped phase of DHI data. The simulations and experimental comparisons demonstrate the efficacy of the proposed approach in absolute phase reconstruction and 3D deformation measurement with respect to the existing conventional methods and state-of-the-art deep learning methods.