Multi-task Learning Approach Research Articles

Coordinating Multi-Energy Microgrids for Integrated Energy System Resilience: A Multi-Task Learning Approach

Coordinating Multi-Energy Microgrids for Integrated Energy System Resilience: A Multi-Task Learning Approach

A deep multi-task learning approach to identifying mummy berry infection sites, the disease stage, and severity.

Mummy berry is a serious disease that may result in up to 70 percent of yield loss for lowbush blueberries. Practical mummy berry disease detection, stage classification and severity estimation remain great challenges for computer vision-based approaches because images taken in lowbush blueberry fields are usually a mixture of different plant parts (leaves, bud, flowers and fruits) with a very complex background. Specifically, typical problems hindering this effort included data scarcity due to high manual labelling cost, tiny and low contrast disease features interfered and occluded by healthy plant parts, and over-complicated deep neural networks which made deployment of a predictive system difficult. Using real and raw blueberry field images, this research proposed a deep multi-task learning (MTL) approach to simultaneously accomplish three disease detection tasks: identification of infection sites, classification of disease stage, and severity estimation. By further incorporating novel superimposed attention mechanism modules and grouped convolutions to the deep neural network, enabled disease feature extraction from both channel and spatial perspectives, achieving better detection performance in open and complex environments, while having lower computational cost and faster convergence rate. Experimental results demonstrated that our approach achieved higher detection efficiency compared with the state-of-the-art deep learning models in terms of detection accuracy, while having three main advantages: 1) field images mixed with various types of lowbush blueberry plant organs under a complex background can be used for disease detection; 2) parameter sharing among different tasks greatly reduced the size of training samples and saved 60% training time than when the three tasks (data preparation, model development and exploration) were trained separately; and 3) only one-sixth of the network parameter size (23.98M vs. 138.36M) and one-fifteenth of the computational cost (1.13G vs. 15.48G FLOPs) were used when compared with the most popular Convolutional Neural Network VGG16. These features make our solution very promising for future mobile deployment such as a drone carried task unit for real-time field surveillance. As an automatic approach to fast disease diagnosis, it can be a useful technical tool to provide growers real time disease information that can prevent further disease transmission and more severe effects on yield due to fruit mummification.

Multimodal automatic assessment of acute pain through facial videos and heart rate signals utilizing transformer-based architectures.

Accurate and objective pain evaluation is crucial in developing effective pain management protocols, aiming to alleviate distress and prevent patients from experiencing decreased functionality. A multimodal automatic assessment framework for acute pain utilizing video and heart rate signals is introduced in this study. The proposed framework comprises four pivotal modules: the Spatial Module, responsible for extracting embeddings from videos; the Heart Rate Encoder, tasked with mapping heart rate signals into a higher dimensional space; the AugmNet, designed to create learning-based augmentations in the latent space; and the Temporal Module, which utilizes the extracted video and heart rate embeddings for the final assessment. The Spatial-Module undergoes pre-training on a two-stage strategy: first, with a face recognition objective learning universal facial features, and second, with an emotion recognition objective in a multitask learning approach, enabling the extraction of high-quality embeddings for the automatic pain assessment. Experiments with the facial videos and heart rate extracted from electrocardiograms of the BioVid database, along with a direct comparison to 29 studies, demonstrate state-of-the-art performances in unimodal and multimodal settings, maintaining high efficiency. Within the multimodal context, 82.74% and 39.77% accuracy were achieved for the binary and multi-level pain classification task, respectively, utilizing million parameters for the entire framework.

A vulnerability severity prediction method based on bimodal data and multi-task learning

Facing the increasing number of software vulnerabilities, the automatic analysis of vulnerabilities has become an important task in the field of software security. However, the existing severity prediction methods are mainly based on vulnerability descriptions and ignore the relevant features of vulnerability code, which only includes unimodal information and result in low prediction accuracy. This paper proposes a vulnerability severity prediction method based on bimodal data and multi-task learning. First the bimodal data, which consists of the description and source code of each vulnerability, is preprocessed. Next the GraphCodeBert is used for the word embedding module to extract different vulnerability features from the bimodal data. Then the Bi-GRU with attention mechanism is adopted for further feature extraction of vulnerability severity. Considering the strong correlation between the two tasks of vulnerability severity prediction and exploitability prediction, this paper proposes a multi-task learning approach, which allows the model to learn the connection and shared information between different tasks through a hard parameter sharing strategy, so as to achieve more accurate and reliable prediction of vulnerability severity. Experimental results show that the severity prediction method proposed in this paper outperforms state-of-the-art methods, and can achieve an average F1 score of 93.83 % on the public vulnerability dataset.

MCSSME: Multi-Task Contrastive Learning for Semi-supervised Singing Melody Extraction from Polyphonic Music

Singing melody extraction is an important task in the field of music information retrieval (MIR). The development of data-driven models for this task have achieved great successes. However, the existing models have two major limitations: firstly, most of the existing singing melody extraction models have formulated this task as a pixel-level prediction task. The lack of labeling data has limited the model for further improvements. Secondly, the generalization of the existing models are prone to be disturbed by the music genres. To address the issues mentioned above, in this paper, we propose a multi-Task contrastive learning framework for semi-supervised singing melody extraction, termed as MCSSME. Specifically, to deal with data scarcity limitation, we propose a self-consistency regularization (SCR) method to train the model on the unlabeled data. Transformations are applied to the raw signal of polyphonic music, which makes the network to improve its representation capability via recognizing the transformations. We further propose a novel multi-task learning (MTL) approach to jointly learn singing melody extraction and classification of transformed data. To deal with generalization limitation, we also propose a contrastive embedding learning, which strengthens the intra-class compactness and inter-class separability. To improve the generalization on different music genres, we also propose a domain classification method to learn task-dependent features by mapping data from different music genres to shared subspace. MCSSME evaluates on a set of well-known public melody extraction datasets with promising performances. The experimental results demonstrate the effectiveness of the MCSSME framework for singing melody extraction from polyphonic music using very limited labeled data scenarios.

Hot rolled prognostic approach based on hybrid Bayesian progressive layered extraction multi-task learning

Hot-rolled strip products have diverse applications, and enhancing the detection, diagnostics, and prognostics of product quality during hot rolling is essential. Nevertheless, the multivariable, strong coupling, nonlinear, and time-varying nature of the production process poses a rigorous challenge for accurate hot-rolled prognostics. This paper implements a progressive layered extraction (PLE) multi-task learning (MTL) framework to simultaneously estimate multiple quality indicators, such as strip crown, center line deviation, exit temperature, wedge, width, and symmetry flatness. Additionally, the paper proposes the implements of Hybrid Bayesian Neural Network (HBNN) experts and a gating network with attention mechanism to integrate private and shared task features. It also puts forth an auxiliary task involving a Variational Autoencoder with Generative Adversarial Networks (VAE-GAN) to extract latent states from the original sequence. Moreover, an adaptive joint loss optimization is employed to update the weight of individual task losses for MTL training problems, and three sets of field hot-rolled datasets are used for model evaluation. In the experimental validation, considering the noisy field data and limited conditions in the real hot rolled production, comparative experiments are conducted to demonstrate the improved generalization and robustness of the proposed MTL approach. These experiments involve different percentages of the total data, ranging from 5% to 20%, and various prediction horizons ranging from 1 to 50 steps for model establishment. In addition, the paper discusses the interpretation of the model and strategies for further enhancing model performance.

Efficient healthcare supply chain: A prioritized multi-task learning approach with task-specific regularization

Healthcare supply chain management is vital for the adequate and timely delivery of essential medical resources to healthcare facilities. In this context, a machine learning model is proposed in this paper to enhance the service of healthcare supply chains by predicting both the medical supply quantities and its timely delivery. This proposed work utilizes the multi-task learning as both of the interrelated tasks, such as the “shipped quantity” of the medical supplies and its “actual days to delivery”, simultaneously optimize their performance to improve supply chain predictions significantly. The prioritized multi-task learning with task-specific regularization provides better learning for the task “actual days to delivery” over “shipped quantity” by considering its significance in healthcare. Additionally, this task-specific regularization also prevents overfitting for the training of the model. The results and its analysis show a significant advancement of 0.3522 and 0.3531 mean absolute error and mean square error for predicting both the tasks. The proposed work outperforms the existing work in terms of mean absolute error, mean square error, root mean square error and R-squared (R2). In addition, the machine learning interpretation technique is used to assess the contribution of each feature in prediction by the proposed model.

Unveiling the dynamics of crisis events: Sentiment and emotion analysis via multi-task learning with attention mechanism and subject-based intent prediction

In the age of rapid internet expansion, social media platforms like Twitter have become crucial for sharing information, expressing emotions, and revealing intentions during crisis situations. They offer crisis responders a means to assess public sentiment, attitudes, intentions, and emotional shifts by monitoring crisis-related tweets. To enhance sentiment and emotion classification, we adopt a transformer-based multi-task learning (MTL) approach with attention mechanism, enabling simultaneous handling of both tasks, and capitalizing on task interdependencies. Incorporating attention mechanism allows the model to concentrate on important words that strongly convey sentiment and emotion. We compare three baseline models, and our findings show that BERTweet outperforms the standard BERT model and exhibits similar performance to RoBERTa in crisis tweets. Furthermore, we employ natural language processing techniques to extract key subject entities (e.g., police, victims) and leverage the publicly available commonsense knowledge model, COMET-ATOMIC 2020, to identify their intentions in given crisis scenarios. Evaluation of COMET-ATOMIC 2020 on subject-based intent prediction in crisis tweets reveals that BART was superior to GPT2-XL model, providing crisis responders with vital information for better decision making. Notably, the integration of sentiment and emotion classification, identification of attention words and subject-based intent prediction represents a novel methodology, not previously applied in the context of crisis scenarios.

Enhancing Demand Prediction: A Multi-Task Learning Approach for Taxis and TNCs

Taxis and Transportation Network Companies (TNCs) are important components of the urban transportation system. An accurate short-term forecast of passenger demand can help operators better allocate taxi or TNC services to achieve supply–demand balance in real time. As a result, drivers can improve the efficiency of passenger pick-ups, thereby reducing traffic congestion and contributing to the overall sustainability of the program. Previous research has proposed sophisticated machine learning and neural-network-based models to predict the short-term demand for taxi or TNC services. However, few of them jointly consider both modes, even though the short-term demand for taxis and TNCs is closely related. By enabling information sharing between the two modes, it is possible to reduce the prediction errors for both. To improve the prediction accuracy for both modes, this study proposes a multi-task learning (MTL) model that jointly predicts the short-term demand for taxis and TNCs. The model adopts a gating mechanism that selectively shares information between the two modes to avoid negative transfer. Additionally, the model captures the second-order spatial dependency of demand by applying a graph convolutional network. To test the effectiveness of the technique, this study uses taxi and TNC demand data from Manhattan, New York, as a case study. The prediction accuracy of single-task learning and multi-task learning models are compared, and the results show that the multi-task learning approach outperforms single-task learning and benchmark models.

Glioma subtype classification from histopathological images using in-domain and out-of-domain transfer learning: An experimental study

We provide in this paper a comprehensive comparison of various transfer learning strategies and deep learning architectures for computer-aided classification of adult-type diffuse gliomas. We evaluate the generalizability of out-of-domain ImageNet representations for a target domain of histopathological images, and study the impact of in-domain adaptation using self-supervised and multi-task learning approaches for pretraining the models using the medium-to-large scale datasets of histopathological images. A semi-supervised learning approach is furthermore proposed, where the fine-tuned models are utilized to predict the labels of unannotated regions of the whole slide images (WSI). The models are subsequently retrained using the ground-truth labels and weak labels determined in the previous step, providing superior performance in comparison to standard in-domain transfer learning with balanced accuracy of 96.91% and F1-score 97.07%, and minimizing the pathologist's efforts for annotation. Finally, we provide a visualization tool working at WSI level which generates heatmaps that highlight tumor areas; thus, providing insights to pathologists concerning the most informative parts of the WSI.

Model and Empirical Study on Multi-tasking Learning for Human Fall Detection

Many fall detection systems are being used to provide real-time responses to fall occurrences. Automated fall detection is challenging because it requires very high accuracy to be clinically acceptable. Recent research has tried to improve sensitivity while reducing the high rate of false positives. Nevertheless, there are still limitations in terms of having efficient learning approaches and proper datasets to train. To reduce false alarms, one approach is to add more nonfall data as negative samples to train the deep learning model. However, this approach increases class imbalance in the training set. To tackle this problem, we propose a multi-task deep learning approach that divides datasets into multiple training sets for multiple tasks. We prove this approach gives better results than a single-task model trained on all datasets. Many experiments are conducted to find the best combination of tasks for multi-task model training for fall detection.

Smart Motorcycle Helmet: Utilizing CNN for Multi-Task Learning

Enhancing road safety through education and enforcement relies on automated motorcycle helmet detection via video surveillance. However, existing methods encounter challenges like tracking individual motorcycles and distinguishing riders from passengers. To overcome these limitations, we propose a CNN-based multi-task learning (MTL) method. Our approach focuses on identifying and tracking individual motorcycles, particularly emphasizing rider-specific helmet use. We introduce the HELMET dataset, containing 91,000 annotated frames from 10,006 motorcycles across 12 observation sites in Myanmar, serving as a benchmark for future detection techniques. Leveraging concurrent visual similarity learning and helmet use classification, our MTL approach achieves enhanced efficiency and accuracy, operating at over 8 FPS on consumer hardware. With a weighted average F-measure of 67.3% for detecting riders and helmet use, our method underscores deep learning's accuracy and resource efficiency in gathering critical road safety data. Furthermore, we present an intelligent motorcycle helmet equipped with infrared transceivers, an image sensor, an embedded computation module, a charging module, a microphone, and earphones. Designed for large vehicle approach notification, the helmet utilizes image recognition modes for both day and night conditions. Experimental results demonstrate successful vehicle registration plate recognition for large trucks/buses, achieving up to 75% accuracy during the day and 70% at night. In real-time, the proposed intelligent motorcycle helmet detects approaching large vehicles within a 5-meter distance.

Career Mobility Analysis With Uncertainty-Aware Graph Autoencoders: A Job Title Transition Perspective

Career mobility analysis aims at discovering the movement patterns of employees across different job positions or grades, which can benefit various human resource-related applications. Indeed, recent studies in this direction mainly focus on modeling individual career trajectories, while the macroguidance for labor market assessment has been largely ignored. To this end, in this article, we propose to study career mobility from a market-driven perspective based on large-scale online professional networks (OPNs). Specifically, we propose an uncertainty-aware graph autoencoders (UnGAEs) framework, which can simultaneously discover potential job title transition patterns and predict job durations. In this phase, we first construct a job title transition graph based on massive career trajectory data from OPNs. Then, considering the inherent uncertainty in career mobility, we introduce a novel uncertainty-aware graph encoder (UnGE) to represent job titles as Gaussian embeddings. Furthermore, we design two task-specific decoders that can preserve the asymmetric relationships between job titles, namely the gravity-inspired decoder (GID) and the energy-inspired decoder (EID), for predicting potential transition patterns and corresponding duration, respectively. In particular, both tasks are modeled through a specially designed multitask learning approach. Finally, extensive experiments on a real-world dataset clearly demonstrate the effectiveness of UnGAE compared with state-of-the-art baselines, as well as some potential applications such as job title benchmarking and career path planning.

Probabilistic model updating of steel frame structures using strain and acceleration measurements: A multitask learning framework

This paper proposes a multitask learning framework for probabilistic model updating by jointly using strain and acceleration measurements. This framework can enhance the structural damage assessment and response prediction of existing steel frame structures with quantified uncertainty. Multitask learning may be used to address multiple similar inference tasks simultaneously to achieve a more robust prediction performance by transferring useful knowledge from one task to another, even in situations of data scarcity. In the proposed model-updating procedure, a spatial frame is decomposed into multiple planar frames that are viewed as multiple tasks and jointly analyzed based on the hierarchical Bayesian model, leading to robust estimation results. The procedure uses a displacement–stress relationship in the modal space because it directly reflects the elemental stiffness and requires no prior knowledge concerning the mass, unlike most existing model-updating techniques. Validation of the proposed framework by using a full-scale vibration test on a one-story, one-bay by one-bay moment resisting steel frame, wherein structural damage to the column bases is simulated by loosening the anchor bolts, is presented. The experimental results suggest that the displacement–stress relationship has sufficient sensitivity toward localized damage, and the Bayesian multitask learning approach may result in the efficient use of measurements such that the uncertainty involved in model parameter estimation is reduced. The proposed framework facilitates more robust and informative model updating.

Smart Motorcycle Helmet: A CNN-Based Multi-Task Learning Approach

Automated motorcycle helmet detection through video surveillance is crucial for enhancing road safety through education and enforcement. Existing approaches face limitations, such as difficulty in tracking individual motorcycles and distinguishing drivers from passengers. To address these challenges, we propose a CNN-based multi-task learning (MTL) method for identifying and tracking individual motorcycles, with a focus on rider-specific helmet use. We introduce the HELMET dataset, comprising 91,000 annotated frames from 10,006 motorcycles across 12 observation sites in Myanmar, providing a benchmark for future detection approaches. Our MTL approach, leveraging concurrent visual similarity learning and helmet use classification, achieves improved efficiency and accuracy. The proposed method operates at over 8 FPS on consumer hardware, yielding a weighted average F-measure of 67.3% for detecting riders and helmet use. Our work showcases deep learning as an accurate and resource-efficient means of collecting critical road safety data. we present an intelligent motorcycle helmet featuring infrared transceivers, an image sensor, an embedded computation module, a charging module, a microphone, and earphones. Designed for large vehicle approach intimation, the helmet uses image recognition modes for day and night conditions. Experimental results demonstrate successful vehicle registration plate recognition for large trucks/buses, achieving up to 75% accuracy during the day and 70% at night. The proposed intelligent motorcycle helmet detects approaching large vehicles in real time within a 5-meter distance.

Multitask learning for predicting pulmonary absorption of chemicals

Pulmonary absorption is an important route for drug delivery and chemical exposure. To streamline the chemical assessment process for the reduction of animal experiments, several animal-free models were developed for pulmonary absorption research. While Calu-3 and Caco-2 cells and their derived computational models were used in estimating pulmonary permeability, the ex vivo isolated perfused lung (IPL) models are considered more clinically relevant measurements. However, the IPL experiments are resource-consuming making it infeasible for the large-scale screening of potential inhaled toxicants and drugs. In silico models are desirable for estimating pulmonary absorption. This study presented a novel machine learning method that employed an extratrees-based multitask learning approach to predict the IPL absorption rate constant (kaIPL) of various chemicals. The shared permeability knowledge was extracted by simultaneously learning three relevant tasks of Caco-2 and Calu-3 cell permeability and IPL absorption rate. Seven informative physicochemical descriptors were identified. A rigorous evaluation of the developed prediction model showed good performance with a high correlation between predictions and observations (r = 0.84) in the independent test dataset. Two case studies of inhalation drugs and respiratory sensitizers revealed the potential application of this model, which may serve as a valuable tool for predicting pulmonary absorption of chemicals.

An ultra-short-term forecasting method for multivariate loads of user-level integrated energy systems in a microscopic perspective: based on multi-energy spatio-temporal coupling and dual-attention mechanism

An ultra-short-term multivariate load forecasting method under a microscopic perspective is proposed to address the characteristics of user-level integrated energy systems (UIES), which are small in scale and have large load fluctuations. Firstly, the spatio-temporal correlation of users’ energy use behavior within the UIES is analyzed, and a multivariate load input feature set in the form of a class image is constructed based on the various types of load units. Secondly, in order to maintain the feature independence and temporal integrity of each load during the feature extraction process, a deep neural network architecture with spatio-temporal coupling characteristics is designed. Among them, the multi-channel parallel convolutional neural network (MCNN) performs independent spatial feature extraction of the 2D load component pixel images at each moment in time, and feature fusion of various types of load features in high dimensional space. A bidirectional long short-term memory network (BiLSTM) is used as a feature sharing layer to perform temporal feature extraction on the fused load sequences. In addition, a spatial attention layer and a temporal attention layer are designed in this paper for the original input load pixel images and the fused load sequences, respectively, so that the model can better capture the important information. Finally, a multi-task learning approach based on the hard sharing mechanism achieves joint prediction of each load. The measured load data of a UIES is analyzed as an example to verify the superiority of the method proposed in this paper.

Multi-Task Deep Learning Approach for Sound Event Recognition and Tracking

Multi-Task Deep Learning Approach for Sound Event Recognition and Tracking

Multitask and Transfer Learning Approach for Joint Classification and Severity Estimation of Dysphonia.

Despite speech being the primary communication medium, it carries valuable information about a speaker's health, emotions, and identity. Various conditions can affect the vocal organs, leading to speech difficulties. Extensive research has been conducted by voice clinicians and academia in speech analysis. Previous approaches primarily focused on one particular task, such as differentiating between normal and dysphonic speech, classifying different voice disorders, or estimating the severity of voice disorders. This study proposes an approach that combines transfer learning and multitask learning (MTL) to simultaneously perform dysphonia classification and severity estimation. Both tasks use a shared representation; network is learned from these shared features. We employed five computer vision models and changed their architecture to support multitask learning. Additionally, we conducted binary 'healthy vs. dysphonia' and multiclass 'healthy vs. organic and functional dysphonia' classification using multitask learning, with the speaker's sex as an auxiliary task. The proposed method achieved improved performance across all classification metrics compared to single-task learning (STL), which only performs classification or severity estimation. Specifically, the model achieved F1 scores of 93% and 90% in MTL and STL, respectively. Moreover, we observed considerable improvements in both classification tasks by evaluating beta values associated with the weight assigned to the sex-predicting auxiliary task. MTL achieved an accuracy of 77% compared to the STL score of 73.2%. However, the performance of severity estimation in MTL was comparable to STL. Our goal is to improve how voice pathologists and clinicians understand patients' conditions, make it easier to track their progress, and enhance the monitoring of vocal quality and treatment procedures. Clinical and Translational Impact Statement: By integrating both classification and severity estimation of dysphonia using multitask learning, we aim to enable clinicians to gain a better understanding of the patient's situation, effectively monitor their progress and voice quality.

Bacterial image analysis using multi-task deep learning approaches for clinical microscopy.

Bacterial image analysis plays a vital role in various fields, providing valuable information and insights for studying bacterial structural biology, diagnosing and treating infectious diseases caused by pathogenic bacteria, discovering and developing drugs that can combat bacterial infections, etc. As a result, it has prompted efforts to automate bacterial image analysis tasks. By automating analysis tasks and leveraging more advanced computational techniques, such as deep learning (DL) algorithms, bacterial image analysis can contribute to rapid, more accurate, efficient, reliable, and standardised analysis, leading to enhanced understanding, diagnosis, and control of bacterial-related phenomena. Three object detection networks of DL algorithms, namely SSD-MobileNetV2, EfficientDet, and YOLOv4, were developed to automatically detect Escherichia coli (E. coli) bacteria from microscopic images. The multi-task DL framework is developed to classify the bacteria according to their respective growth stages, which include rod-shaped cells, dividing cells, and microcolonies. Data preprocessing steps were carried out before training the object detection models, including image augmentation, image annotation, and data splitting. The performance of the DL techniques is evaluated using the quantitative assessment method based on mean average precision (mAP), precision, recall, and F1-score. The performance metrics of the models were compared and analysed. The best DL model was then selected to perform multi-task object detections in identifying rod-shaped cells, dividing cells, and microcolonies. The output of the test images generated from the three proposed DL models displayed high detection accuracy, with YOLOv4 achieving the highest confidence score range of detection and being able to create different coloured bounding boxes for different growth stages of E. coli bacteria. In terms of statistical analysis, among the three proposed models, YOLOv4 demonstrates superior performance, achieving the highest mAP of 98% with the highest precision, recall, and F1-score of 86%, 97%, and 91%, respectively. This study has demonstrated the effectiveness, potential, and applicability of DL approaches in multi-task bacterial image analysis, focusing on automating the detection and classification of bacteria from microscopic images. The proposed models can output images with bounding boxes surrounding each detected E. coli bacteria, labelled with their growth stage and confidence level of detection. All proposed object detection models have achieved promising results, with YOLOv4 outperforming the other models.