Multi-task Learning Architecture Research Articles

Multitask Siamese Network for Remote Photoplethysmography and Respiration Estimation.

Heart and respiration rates represent important vital signs for the assessment of a person’s health condition. To estimate these vital signs accurately, we propose a multitask Siamese network model (MTS) that combines the advantages of the Siamese network and the multitask learning architecture. The MTS model was trained by the images of the cheek including nose and mouth and forehead areas while sharing the same parameters between the Siamese networks, in order to extract the features about the heart and respiratory information. The proposed model was constructed with a small number of parameters and was able to yield a high vital-sign-prediction accuracy, comparable to that obtained from the single-task learning model; furthermore, the proposed model outperformed the conventional multitask learning model. As a result, we can simultaneously predict the heart and respiratory signals with the MTS model, while the number of parameters was reduced by 16 times with the mean average errors of heart and respiration rates being 2.84 and 4.21. Owing to its light weight, it would be advantageous to implement the vital-sign-monitoring model in an edge device such as a mobile phone or small-sized portable devices.

Multi-task learning approach for modulation and wireless signal classification for 5G and beyond: Edge deployment via model compression

Future communication networks must address the scarce spectrum to accommodate extensive growth of heterogeneous wireless devices. Efforts are underway to address spectrum coexistence, enhance spectrum awareness, and bolster authentication schemes. Wireless signal recognition is becoming increasingly more significant for spectrum monitoring, spectrum management, secure communications, among others. Consequently, comprehensive spectrum awareness on the edge has the potential to serve as a key enabler for the emerging beyond 5G (fifth generation) networks. State-of-the-art studies in this domain have (i) only focused on a single task – modulation or signal (protocol) classification – which in many cases is insufficient information for a system to act on, (ii) consider either radar or communication waveforms (homogeneous waveform category), and (iii) does not address edge deployment during neural network design phase. In this work, for the first time in the wireless communication domain, we exploit the potential of deep neural networks based multi-task learning (MTL) framework to simultaneously learn modulation and signal classification tasks while considering heterogeneous wireless signals such as radar and communication waveforms in the electromagnetic spectrum. The proposed MTL architecture benefits from the mutual relation between the two tasks in improving the classification accuracy as well as the learning efficiency with a lightweight neural network model. We additionally include experimental evaluations of the model with over-the-air collected samples and demonstrate first-hand insight on model compression along with deep learning pipeline for deployment on resource-constrained edge devices. We demonstrate significant computational, memory, and accuracy improvement of the proposed model over two reference architectures. In addition to modeling a lightweight MTL model suitable for resource-constrained embedded radio platforms, we provide a comprehensive heterogeneous wireless signals dataset for public use.

EMT-NET: EFFICIENT MULTITASK NETWORK FOR COMPUTER-AIDED DIAGNOSIS OF BREAST CANCER.

Deep learning-based computer-aided diagnosis has achieved unprecedented performance in breast cancer detection. However, most approaches are computationally intensive, which impedes their broader dissemination in real-world applications. In this work, we propose an efficient and light-weighted multitask learning architecture to classify and segment breast tumors simultaneously. We incorporate a segmentation task into a tumor classification network, which makes the backbone network learn representations focused on tumor regions. Moreover, we propose a new numerically stable loss function that easily controls the balance between the sensitivity and specificity of cancer detection. The proposed approach is evaluated using a breast ultrasound dataset with 1511 images. The accuracy, sensitivity, and specificity of tumor classification is 88.6%, 94.1%, and 85.3%, respectively. We validate the model using a virtual mobile device, and the average inference time is 0.35 seconds per image.

Multi-Task Based Deep Learning Approach for Open-Set Wireless Signal Identification in ISM Band

Wireless signal identification plays an important role in effectively implementing spectrum monitoring and management. However, in ISM (Industrial, Science and Medical) band, it becomes a challenging task due to the heterogeneity of variously emerging wireless techniques, and part of the potential unknown spectral occupants may even hinder the feasibility of wireless signal identification. To overcome such difficulties, we focus on open-set recognition (OSR) in this paper and present a multi-task learning architecture based on deep neural network for identifying known and unknown spectral occupants. A novel structured extension of the counterfactual GAN (CountGAN) architecture is proposed and we introduce a multi-tasking architecture to take advantage of the modulation-domain information from the captured signal, thus improving the representation of individual wireless signals and further enhancing model robustness and adaptability to open-set scenarios. In particular, Circle-loss in metric learning and extreme value theory are also applied to make tighter and clearer decision boundaries for signal identification, and further enhance the optimization of the decision boundaries between known and unknown classes. Numerical results indicate that the proposed framework consistently outperforms state-of-the-art OSR algorithms and several baselines for wireless signal identification task, both in terms of convergence performance and classification accuracy.

Multi-task Learning-Based Spoofing-Robust Automatic Speaker Verification System

Spoofing attacks posed by generating artificial speech can severely degrade the performance of a speaker verification system. Recently, many anti-spoofing countermeasures have been proposed for detecting varying types of attacks from synthetic speech to replay presentations. While there are numerous effective defenses reported on standalone anti-spoofing solutions, the integration for speaker verification and spoofing detection systems has obvious benefits. In this paper, we propose a spoofing-robust automatic speaker verification system for diverse attacks based on a multi-task learning architecture. This deep learning-based model is jointly trained with time-frequency representations from utterances to provide recognition decisions for both tasks simultaneously. Compared with other state-of-the-art systems on the ASVspoof 2017 and 2019 corpora, a substantial improvement of the combined system under different spoofing conditions can be obtained.

A multi-task learning-based framework for global maritime trajectory and destination prediction with AIS data

Automatic Identification System (AIS) was initially developed for tracking ships and avoiding collisions. As numerous small satellites were launched in recent years, millions of AIS messages are captured by these satellite AIS providers every day. The massive amount of data allow shipping firms and port operators to better predict vessels’ movement. In this study, a computational framework is developed to predict future trajectories and destinations of vessels by applying deep learning to the satellite AIS messages containing self-reporting positioning data. We employed deep learning approaches for large-scale maritime prediction and identifies the most suitable deep neural networks for this specific application. Three prediction models, Convolutional Neural Networks (CNN), Dense Neural Networks (DNN), and Long Short-Term Memory (LSTM) have been tested in the framework. A multi-task learning architecture is implemented based on test results. Through sharing parameters while training, this architecture enables long-term destination predictions by balancing short-term trajectory predictions. The framework is validated globally with the experiments covering five large-scale sea areas. The experimental results demonstrate that the LSTM model with an average accuracy percentage of 85.1% outperforms the CNN and DNN with accuracy percentages of 68.3% and 78.2% respectively. The LSTM model with multi-task architecture can achieve an accuracy of 87.0%. With the proposed computational framework, AIS data can be utilized for maritime predictions, supporting more complex marine applications and covering more expansive geographic areas.

TA-Net: Topology-Aware Network for Gland Segmentation.

Gland segmentation is a critical step to quantitatively assess the morphology of glands in histopathology image analysis. However, it is challenging to separate densely clustered glands accurately. Existing deep learning-based approaches attempted to use contour-based techniques to alleviate this issue but only achieved limited success. To address this challenge, we propose a novel topology-aware network (TA-Net) to accurately separate densely clustered and severely deformed glands. The proposed TA-Net has a multitask learning architecture and enhances the generalization of gland segmentation by learning shared representation from two tasks: instance segmentation and gland topology estimation. The proposed topology loss computes gland topology using gland skeletons and markers. It drives the network to generate segmentation results that comply with the true gland topology. We validate the proposed approach on the GlaS and CRAG datasets using three quantitative metrics, F1-score, object-level Dice coefficient, and object-level Hausdorff distance. Extensive experiments demonstrate that TA-Net achieves state-of-the-art performance on the two datasets. TA-Net outperforms other approaches in the presence of densely clustered glands.

3D deformation measurement in digital holographic interferometry using a multitask deep learning architecture.

The extraction of absolute phase from an interference pattern is a key step for 3D deformation measurement in digital holographic interferometry (DHI) and is an ill-posed problem. Estimating the absolute unwrapped phase becomes even more challenging when the obtained wrapped phase from the interference pattern is noisy. In this paper, we propose a novel multitask deep learning approach for phase reconstruction and 3D deformation measurement in DHI, referred to as TriNet, that has the capability to learn and perform two parallel tasks from the input image. The proposed TriNet has a pyramidal encoder-two-decoder framework for multi-scale information fusion. To our knowledge, TriNet is the first multitask approach to accomplish simultaneous denoising and phase unwrapping of the wrapped phase from the interference fringes in a single step for absolute phase reconstruction. The proposed architecture is more elegant than recent multitask learning methods such as Y-Net and state-of-the-art segmentation approaches such as UNet++. Further, performing denoising and phase unwrapping simultaneously enables deformation measurement from the highly noisy wrapped phase of DHI data. The simulations and experimental comparisons demonstrate the efficacy of the proposed approach in absolute phase reconstruction and 3D deformation measurement with respect to the existing conventional methods and state-of-the-art deep learning methods.

Pallet Recognition with Multi-Task Learning for Automated Guided Vehicles

As the need for efficient warehouse logistics has increased in manufacturing systems, the use of automated guided vehicles (AGVs) has also increased to reduce travel time. The AGVs are controlled by a system using laser sensors or floor-embedded wires to transport pallets and their loads. Because such control systems have only predefined palletizing strategies, AGVs may fail to engage incorrectly positioned pallets. In this study, we consider a vision sensor-based method to address this shortcoming by recognizing a pallet’s position. We propose a multi-task deep learning architecture that simultaneously predicts distances and rotation based on images obtained from a visionary sensor. These predictions complement each other in learning, allowing a multi-task model to learn and execute tasks impossible with single-task models. The proposed model can accurately predict the rotation and displacement of the pallets to derive information necessary for the control system. This information can be used to optimize a palletizing strategy. The superiority of the proposed model was verified by an experiment on images of stored pallets that were collected from a visionary sensor attached to an AGV.

Short-Term Multiple Load Forecasting Model of Regional Integrated Energy System Based on QWGRU-MTL

In order to improve the accuracy of the multiple load forecasting of a regional integrated energy system, a short-term multiple load forecasting model based on the quantum weighted GRU and multi-task learning framework is proposed in this paper. Firstly, correlation analysis is carried out using a maximum information coefficient to select the input of the model. Then, a multi-task learning architecture is constructed based on the quantum weighted GRU neural network, and the coupling information among multiple loads is learned through the sharing layer in order to improve the prediction accuracy of multiple loads. Finally, the PSO algorithm is used to optimize the parameters of the quantum weighted GRU. The simulation data of a regional integrated energy system in northern China are used to predict the power and cooling loads on summer weekdays and rest days, and the results show that, compared with the LSTM, GRU and single task learning QWGRU models, the proposed model is more effective in the multiple load forecasting of a regional integrated energy system.

Multi-task learning for virtual flow metering

Virtual flow metering (VFM) is a cost-effective and non-intrusive technology for inferring multiphase flow rates in petroleum assets. Inferences about flow rates are fundamental to decision support systems that operators extensively rely on. Data-driven VFM, where mechanistic models are replaced with machine learning models, has recently gained attention due to its promise of lower maintenance costs. While excellent performances in small sample studies have been reported in the literature, there is still considerable doubt about the robustness of data-driven VFM. In this paper, we propose a new multi-task learning (MTL) architecture for data-driven VFM. Our method differs from previous methods in that it enables learning across oil and gas wells. We study the method by modeling 55 wells from four petroleum assets and compare the results with two single-task baseline models. Our findings show that MTL improves robustness over single-task methods, without sacrificing performance. MTL yields a 25%–50% error reduction on average for the assets where single-task architectures are struggling.

Progressive Multi-task Learning with Controlled Information Flow for Joint Entity and Relation Extraction

Multitask learning has shown promising performance in learning multiple related tasks simultaneously, and variants of model architectures have been proposed, especially for supervised classification problems. One goal of multitask learning is to extract a good representation that sufficiently captures the relevant part of the input about the output for each learning task. To achieve this objective, in this paper we design a multitask learning architecture based on the observation that correlations exist between outputs of some related tasks (e.g. entity recognition and relation extraction tasks), and they reflect the relevant features that need to be extracted from the input. As outputs are unobserved, our proposed model exploits task predictions in lower layers of the neural model, also referred to as early predictions in this work. But we control the injection of early predictions to ensure that we extract good task-specific representations for classification. We refer to this model as a Progressive Multitask learning model with Explicit Interactions (PMEI). Extensive experiments on multiple benchmark datasets produce state-of-the-art results on the joint entity and relation extraction task.

A deep convolutional neural network to simultaneously localize and recognize waste types in images

Accurate waste classification is key to successful waste management. However, most current studies have focused exclusively on single-label waste classification from images, which goes against common sense. In this paper, we move beyond single-label waste classification and propose a benchmark for evaluating the multi-label waste classification and localization tasks to advance waste management via deep learning-based methods. We propose a multi-task learning architecture (MTLA) based on a convolutional neural network, which can be used to simultaneously identify and locate wastes in images. The MTLA comprises a backbone network with proposed attention modules, a novel multi-level feature pyramid network, and a group of joint learning multi-task subnets. To achieve joint optimization of waste identification and location, we designed the loss functions according to the concepts of focusing and joint. The proposed MTLA achieved performance similar to that of experts and had high scores for multiple tasks related to waste management. Its F1 score exceeded 95.50% (95.12% to 95.88%, with a 95% confidence interval) on the multi-label waste classification task, and the average precision score was over 81.50% (@IoU = 0.5) on the waste localization task. To improve interpretation, heatmaps were used to visualize the salient features extracted by the MTLA. The proposed MTLA is a promising auxiliary tool that can improve the automation of waste management systems.

Objective Measures of Cognitive Load Using Deep Multi-Modal Learning

The capability of measuring human performance objectively is hard to overstate, especially in the context of the instructor and student relationship within the process of learning. In this work, we investigate the automated classification of cognitive load leveraging the aviation domain as a surrogate for complex task workload induction. We use a mixed virtual and physical flight environment, given a suite of biometric sensors utilizing the HTC Vive Pro Eye and the E4 Empatica. We create and evaluate multiple models. And we have taken advantage of advancements in deep learning such as generative learning, multi-modal learning, multi-task learning, and x-vector architectures to classify multiple tasks across 40 subjects inclusive of three subject types --- pilots, operators, and novices. Our cognitive load model can automate the evaluation of cognitive load agnostic to subject, subject type, and flight maneuver (task) with an accuracy of over 80%. Further, this approach is validated with real-flight data from five test pilots collected over two test and evaluation flights on a C-17 aircraft.

Generalized and transferable patient language representation for phenotyping with limited data

The paradigm of representation learning through transfer learning has the potential to greatly enhance clinical natural language processing. In this work, we propose a multi-task pre-training and fine-tuning approach for learning generalized and transferable patient representations from medical language. The model is first pre-trained with different but related high-prevalence phenotypes and further fine-tuned on downstream target tasks. Our main contribution focuses on the impact this technique can have on low-prevalence phenotypes, a challenging task due to the dearth of data. We validate the representation from pre-training, and fine-tune the multi-task pre-trained models on low-prevalence phenotypes including 38 circulatory diseases, 23 respiratory diseases, and 17 genitourinary diseases. We find multi-task pre-training increases learning efficiency and achieves consistently high performance across the majority of phenotypes. Most important, the multi-task pre-training is almost always either the best-performing model or performs tolerably close to the best-performing model, a property we refer to as robust. All these results lead us to conclude that this multi-task transfer learning architecture is a robust approach for developing generalized and transferable patient language representations for numerous phenotypes.

Multiscale space-time-frequency feature-guided multitask learning CNN for motor imagery EEG classification

Objective. Motor imagery (MI) electroencephalography (EEG) classification is regarded as a promising technology for brain–computer interface (BCI) systems, which help people to communicate with the outside world using neural activities. However, decoding human intent accurately is a challenging task because of its small signal-to-noise ratio and non-stationary characteristics. Methods that directly extract features from raw EEG signals ignores key frequency domain information. One of the challenges in MI classification tasks is finding a way to supplement the frequency domain information ignored by the raw EEG signal. Approach. In this study, we fuse different models using their complementary characteristics to develop a multiscale space-time-frequency feature-guided multitask learning convolutional neural network (CNN) architecture. The proposed method consists of four modules: the space-time feature-based representation module, time-frequency feature-based representation module, multimodal fused feature-guided generation module, and classification module. The proposed framework is based on multitask learning. The four modules are trained using three tasks simultaneously and jointly optimized. Results. The proposed method is evaluated using three public challenge datasets. Through quantitative analysis, we demonstrate that our proposed method outperforms most state-of-the-art machine learning and deep learning techniques for EEG classification, thereby demonstrating the robustness and effectiveness of our method. Moreover, the proposed method is employed to realize control of robot based on EEG signal, verifying its feasibility in real-time applications. Significance. To the best of our knowledge, a deep CNN architecture that fuses different input cases, which have complementary characteristics, has not been applied to BCI tasks. Because of the interaction of the three tasks in the multitask learning architecture, our method can improve the generalization and accuracy of subject-dependent and subject-independent methods with limited annotated data.

Multi-task learning for gait-based identity recognition and emotion recognition using attention enhanced temporal graph convolutional network

Human gait conveys significant information that can be used for identity recognition and emotion recognition. Recent studies have focused more on gait identity recognition than emotion recognition and regarded these two recognition tasks as independent and unrelated. How to train a unified model to effectively recognize the identity and emotion from gait at the same time is a novel and challenging problem. In this paper, we propose a novel Attention Enhanced Temporal Graph Convolutional Network (AT-GCN) for gait-based recognition and motion prediction. Enhanced by spatial and temporal attention, the proposed model can capture discriminative features in spatial dependency and temporal dynamics. We also present a multi-task learning architecture, which can jointly learn representations for multiple tasks. It helps the emotion recognition task with limited data considerably benefit from the identity recognition task and helps the recognition tasks benefit from the auxiliary prediction task. Furthermore, we present a new dataset (EMOGAIT) that consists of 1, 440 real gaits, annotated with identity and emotion labels. Experimental results on two datasets demonstrate the effectiveness of our approach and show that our approach achieves substantial improvements over mainstream methods for identity recognition and emotion recognition.

Slag Removal Path Estimation by Slag Distribution Image and Multi-Task Deep Learning Network

This paper proposes an automatic slag removal path estimation in a ladle image using a deep neural network. Slag removal is a hazardous task in steel production, and it is conducted manually by experienced human operators. To automate the slag removal task using a robotic machine, a slag removal path in a ladle image must be determined to control the motion of the machine. We propose a novel image structure called Slag Distribution Image(SDI) and a Slag Removal Path Estimation Network(SRPENet) to estimate the slag removal path. SDI contains the slag distribution and the ladle boundary information. SRPENet is designed as a multi-task learning architecture to perform two tasks: the first task is to estimate the control point of the slag removal path and the second task is to estimate the goodness-score of the control points. The structure of SDI is designed to yield efficient performance in SRPENet. The ground truth and learning data of slag removal path are collected along the working trajectories of human experts. The final removal path is determined by post-processing of the outputs of SRPENet. In experiments using real ladle images, the amount of removed slag and the mechanical energy efficiency are evaluated and compared with the ground truth to present qualitative and quantitative analysis. In addition, two ablation studies are presented. The first study is using SDI or RGB image as the input of SRPENet. The second study is the presence or absence of the smooth term in the SRPENet’s loss function. The proposed method shows 119.83% and 96.2% performance compared to ground truth in the amount of removed slag and mechanical energy efficiency, respectively. Our method runs in real-time with about 53.168 msec/frame.

Bidirectional Guided Attention Network for 3-D Semantic Detection of Remote Sensing Images

Semantic segmentation and disparity estimation are in the research frontier of the computer vision and remote sensing (RS) fields. However, existing methods mostly deal with these two problems separately or use a combination of multiple models to solve these two tasks. Due to a lack of sufficient information sharing and fusion, they still have difficulties in coping with seasonal appearance differences in 3-D RS problems. In this article, we propose a novel multitask learning architecture that considers the bottom–up and up–bottom visual attention mechanism for 3-D semantic detection, named bidirectional guided attention network (BGA-Net). BGA-Net consists of five modules: unified backbone module (UBM), bidirectional guided attention module (BGAM), semantic segmentation module (SSM), feature matching module (FMM), and bidirectional fusion module (BFM). First, in UBM, we use a shared backbone to extract unified features and share them with three branches/modules (BGAM, SSM, and FMM). Then, SSM and FMM branches are applied to estimate segmentation and disparity maps, whereas the third branch/module (BGAM) shares the global features to guide the task-specific learning via attention mechanism. Finally, we fuse the results of the two tasks by BFM to improve the final performance. Extensive experiments demonstrate that: 1) our BGA-Net can handle the two tasks simultaneously and can be trained in an end-to-end way; 2) these modules fully take advantage of the two tasks’ information to share features and enhance the scene understanding ability, effectively against seasons change of RS images; and 3) BGA-Net has notable superiority and greater flexibility and also sets a new state of the art on the urban semantic 3-D (US3D) benchmark. Moreover, BGA-Net also provides insights into the intelligent interpretation of RS data images.

A Unified Multi-Task Learning Architecture for Fast and Accurate Pedestrian Detection

We present a unified multi-task learning architecture for fast and accurate pedestrian detection. Different from existing methods which often focus on either a new loss function or architecture, we propose an improved multi-task convolutional neural network learning architecture to effectively and efficiently interfuse the task of pedestrian detection and semantic segmentation. To achieve this, we integrate a lightweight semantic segmentation branch to Faster R-CNN detection framework that enables end-to-end hard parameter sharing in order to boost the detection performance and maintain computational efficiency as follows. Firstly, a Semantic Segmentation to Feature Module (SS2FM) refines the convolutional features in RPN stage by integrating the features generated from the semantic segmentation branch. Secondly, a Semantic Segmentation to Confidence Module (SS2CM) refines the classification confidence in RPN stage by fusing it with the semantic segmentation confidence. We also introduce an effective anchor matching point transform to alleviate the problem of feature misalignment for heavily occluded pedestrians. The proposed unified multi-task learning architecture lends itself well to more robust pedestrian detection in diverse scenarios with negligible computation overhead. In addition, the proposed architecture can achieve high detection performance with low resolution input images, which significantly reduces the computational complexity. Experiment results on CityPersons and Caltech datasets show that our method is the fastest among all state-of-the-art pedestrian detection methods while exhibiting competitive detection performance.