Hierarchical Multi-task Learning Research Articles

HirMTL: Hierarchical Multi-Task Learning for dense scene understanding

In the realm of artificial intelligence, simultaneous multi-task learning is crucial, particularly for dense scene understanding. To address this, we introduce HirMTL, a novel hierarchical multi-task learning framework designed to enhance dense scene analysis. HirMTL is adept at facilitating interaction at the scale level, ensuring task-adaptive multi-scale feature fusion, and fostering task-level feature interchange. It leverages the inherent correlations between tasks to create a synergistic learning environment. Initially, HirMTL enables concurrent sharing and fine-tuning of features at the single-scale level. This is further extended by the Task-Adaptive Fusion module (TAF), which intelligently blends features across scales, specifically attuned to each task’s unique requirements. Complementing this, the Asymmetric Information Comparison Module (AICM) skillfully differentiates and processes both shared and unique features, significantly refining task-specific performance with enhanced accuracy. Our extensive experiments on various dense prediction tasks validate HirMTL’s exceptional capabilities, showcasing its superiority over existing multi-task learning models and underscoring the benefits of its hierarchical approach.

Hierarchical multi-task learning with self-supervised auxiliary task

Hierarchical multi-task learning with self-supervised auxiliary task

Hierarchical multi-task deep learning-assisted construction of human gut microbiota reactive oxygen species-scavenging enzymes database.

In the process of oxygen reduction, reactive oxygen species (ROS) are generated as intermediates, including superoxide anion (O2-), hydrogen peroxide (H2O2), and hydroxyl radicals (OH-). ROS can be destructive, and an imbalance between oxidants and antioxidants in the body can lead to pathological inflammation. Inappropriate ROS production can cause oxidative damage, disrupting the balance in the body and potentially leading to DNA damage in intestinal epithelial cells and beneficial bacteria. Microorganisms have evolved various enzymes to mitigate the harmful effects of ROS. Accurately predicting the types of ROS-scavenging enzymes (ROSes) is crucial for understanding the oxidative stress mechanisms and formulating strategies to combat diseases related to the "gut-organ axis." Currently, there are no available ROSes databases (DBs). In this study, we propose a systematic workflow comprising three modules and employ a hierarchical multi-task deep learning approach to collect, expand, and explore ROSes-related entries. Based on this, we have developed the human gut microbiota ROSes DB (http://39.101.72.186/), which includes 7,689 entries. This DB provides user-friendly browsing and search features to support various applications. With the assistance of ROSes DB, various communication-based microbial interactions can be explored, further enabling the construction and analysis of the evolutionary and complex networks of ROSes DB in human gut microbiota species.IMPORTANCEReactive oxygen species (ROS) is generated during the process of oxygen reduction, including superoxide anion, hydrogen peroxide, and hydroxyl radicals. ROS can potentially cause damage to cells and DNA, leading to pathological inflammation within the body. Microorganisms have evolved various enzymes to mitigate the harmful effects of ROS, thereby maintaining a balance of microorganisms within the host. The study highlights the current absence of a ROSes DB, emphasizing the crucial importance of accurately predicting the types of ROSes for understanding oxidative stress mechanisms and developing strategies for diseases related to the "gut-organ axis." This research proposes a systematic workflow and employs a multi-task deep learning approach to establish the human gut microbiota ROSes DB. This DB comprises 7,689 entries and serves as a valuable tool for researchers to delve into the role of ROSes in the human gut microbiota.

HiMAL: Multimodal Hierarchical Multi-task Auxiliary Learning framework for predicting Alzheimer's disease progression.

We aimed to develop and validate a novel multimodal framework Hierarchical Multi-task Auxiliary Learning (HiMAL) framework, for predicting cognitive composite functions as auxiliary tasks that estimate the longitudinal risk of transition from Mild Cognitive Impairment (MCI) to Alzheimer's Disease (AD). HiMAL utilized multimodal longitudinal visit data including imaging features, cognitive assessment scores, and clinical variables from MCI patients in the Alzheimer's Disease Neuroimaging Initiative dataset, to predict at each visit if an MCI patient will progress to AD within the next 6 months. Performance of HiMAL was compared with state-of-the-art single-task and multitask baselines using area under the receiver operator curve (AUROC) and precision recall curve (AUPRC) metrics. An ablation study was performed to assess the impact of each input modality on model performance. Additionally, longitudinal explanations regarding risk of disease progression were provided to interpret the predicted cognitive decline. Out of 634 MCI patients (mean [IQR] age: 72.8 [67-78], 60% male), 209 (32%) progressed to AD. HiMAL showed better prediction performance compared to all state-of-the-art longitudinal single-modality singe-task baselines (AUROC = 0.923 [0.915-0.937]; AUPRC = 0.623 [0.605-0.644]; all P < .05). Ablation analysis highlighted that imaging and cognition scores with maximum contribution towards prediction of disease progression. Clinically informative model explanations anticipate cognitive decline 6 months in advance, aiding clinicians in future disease progression assessment. HiMAL relies on routinely collected electronic health records (EHR) variables for proximal (6 months) prediction of AD onset, indicating its translational potential for point-of-care monitoring and managing of high-risk patients.

Negation handling for sentiment analysis task: approaches and performance analysis

Negation plays an essential role in sentiment analysis within natural language processing (NLP). Its integration involves two key aspects: identifying the scope of negation and incorporating this information into the sentiment model. Before delving into scope detection, the specific negation cue must be identified, with explicit and implicit negation cues being the two main types. Various methodologies, such as rule-based, machine learning, and hybrid approaches, address the negation scope detection challenge. Strategies for leveraging negation information in sentiment models encompass heuristic polarity modification, feature space augmentation, end-to-end approach, and hierarchical multi-task learning. Notably, there is a need for more studies addressing implicit negation cue detection, even within the state-of-the-art bidirectional encoder representation for transformers (BERT) approach. Some studies have employed reinforcement learning and hybrid techniques to address the implicit negation problem. Further exploration, particularly through a hybrid and multi-task learning approach, is warranted to make potential contributions to the nuanced challenges of handling negation in sentiment analysis, especially in complex sentence structures.

A Survey on Comprehensive Exploration of Modern Perception System

This review paper offers an in-depth analysis of recent advancements in image-based intelligent transportation systems. It provides insights into various approaches employed for detecting false images, license plate recognition, distributing emergency messages, as well as detecting and classifying vehicle accidents and road accidents. The research critically evaluates the pros and cons of deep learning techniques, including position-based routing, enhanced YOLOv8, optimal K-means with a convolutional neural network, and gated hierarchical multi-task learning. The investigation of an Internet of Things-based vehicle accident detection and classification system highlights the benefits of sensor fusion for precise accident detection; nonetheless, the evaluation raises questions regarding the scalability and reliability of smartphone sensors. The utility of a deep learning system for road accident identification utilizing artificially created multi-perspective movies is emphasized, along with warnings about possible limits in capturing real-world circumstances

The performance evaluation of teaching reform based on hierarchical multi-task deep learning

The performance evaluation of teaching reform based on hierarchical multi-task deep learning

A Hierarchical Multitask Learning Approach for the Recognition of Activities of Daily Living Using Data from Wearable Sensors.

Machine learning with deep neural networks (DNNs) is widely used for human activity recognition (HAR) to automatically learn features, identify and analyze activities, and to produce a consequential outcome in numerous applications. However, learning robust features requires an enormous number of labeled data. Therefore, implementing a DNN either requires creating a large dataset or needs to use the pre-trained models on different datasets. Multitask learning (MTL) is a machine learning paradigm where a model is trained to perform multiple tasks simultaneously, with the idea that sharing information between tasks can lead to improved performance on each individual task. This paper presents a novel MTL approach that employs combined training for human activities with different temporal scales of atomic and composite activities. Atomic activities are basic, indivisible actions that are readily identifiable and classifiable. Composite activities are complex actions that comprise a sequence or combination of atomic activities. The proposed MTL approach can help in addressing challenges related to recognizing and predicting both atomic and composite activities. It can also help in providing a solution to the data scarcity problem by simultaneously learning multiple related tasks so that knowledge from each task can be reused by the others. The proposed approach offers advantages like improved data efficiency, reduced overfitting due to shared representations, and fast learning through the use of auxiliary information. The proposed approach exploits the similarities and differences between multiple tasks so that these tasks can share the parameter structure, which improves model performance. The paper also figures out which tasks should be learned together and which tasks should be learned separately. If the tasks are properly selected, the shared structure of each task can help it learn more from other tasks.

Hierarchical Generative Network: A Hierarchical Multitask Learning Approach for Accelerated Composite Material Design and Discovery

Deep learning (DL) methods combined with computational simulations have shown promise for novel material discovery and establishing structure–response relationships within extensive design spaces. However, obtaining sufficient simulation data for precise predictions on nonlinear responses remains challenging, limiting DL models' predictive capabilities for unexplored composite configurations. To address this, we introduce the hierarchical generative network (HGNet), comprising three customized convolutional neural networks (NNs). HGNet predicts stress fields and crack patterns during mechanical failure events, encompassing linear response regimes, ultimate strength, and complete fracture. By jointly training these networks within a unified DL model, predictive capacity significantly improves. HGNet is applied to a two‐phase composite optimization problem with vast design configurations, demonstrating accurate predictions of complex stress distributions and fracture patterns in unseen configurations. Moreover, it explores uncharted designs with superior stiffness and strength across vast design spaces using genetic algorithms. The methodology provides insight into the model's reasoning through visualizing the inference process. Overall, this approach represents a potent and versatile tool for accelerating material discovery, facilitating a more efficient and cost‐effective approach to finding novel materials, even with limited training data.

Deep multi-task learning for nephropathy diagnosis on immunofluorescence images

Background and ObjectiveAs an advanced technique, immunofluorescence (IF) is one of the most widely-used medical image for nephropathy diagnosis, due to its ease of acquisition with low cost. In practice, the clinically collected IF images are commonly corrupted by blurs at different degrees, mainly because of the inaccurate focus at the acquisition stage. Although deep neural network (DNN) methods achieve the great success in nephropathy diagnosis, their performance dramatically drops over the blurred IF images. This significantly limits the potential of leveraging the advanced DNN techniques in real-world nephropathy diagnosis scenarios. MethodsThis paper first establishes two IF databases with synthetic blurs (IFVB) and real-world blurs (Real-IF) for nephropathy diagnosis, respectively, including 1,659 patients and 6,521 IF images with various degrees of blurs. According to the analysis on these two databases, we propose a deep hierarchical multi-task learning based nephropathy diagnosis (DeepMT-ND) method to bridge the gap between the low-level vision and high-level medical tasks. Specifically, DeepMT-ND simultaneously handles the main task of automatic nephropathy diagnosis, as well as the auxiliary tasks of image quality assessment (IQA) and de-blurring. ResultsExtensive experiments show the superiority of our DeepMT-ND in terms of diagnosis accuracy and generalization ability. For instance, our method performs better than nephrologists with at least 15.4% and 6.5% accuracy improvements in IFVB and Real-IF, respectively. Meanwhile, our method also achieves comparable performance in two auxiliary tasks of IQA and de-blurring on blurred IF images. ConclusionsIn this paper, we propose a new DeepMT-ND method for nephropathy diagnosis on blurred IF images. The proposed hierarchical multi-task learning framework provides the new scope to narrow the gap between the low-level vision and high-level medical tasks, and will contribute to nephropathy diagnosis in clinical scenarios. The diagnosis accuracy and generalization ability of DeepMT-ND are experimentally verified to be effective over both synthetic and real-world databases.

Generalized Fake Image Detection Method Based on Gated Hierarchical Multi-Task Learning

Generalized Fake Image Detection Method Based on Gated Hierarchical Multi-Task Learning

Refined analysis and a hierarchical multi-task learning approach for loan fraud detection

Fraud problems in loan application assessment cause significant losses for finance companies worldwide, and much research has focused on machine learning methods to improve the efficacy of fraud detection in some financial domains. However, diverse information falsification in individual fraud remains one of the most challenging problems in loan applications. To this end, we conducted an empirical study to explore the relationships between various fraud types and analyzed the factors influencing information fabrication. Weak relationships exist among different falsification types, and some essential factors play the same roles in different fraud types. In contrast, others have various or opposing effects on these types of frauds. Based on this finding, we propose a novel hierarchical multi-task learning approach to refine fraud-detection systems. Specifically, we first developed a hierarchical fraud category method to break down this problem into several subtasks according to the information types falsified by customers, reducing fraud identification's difficulty. Second, a heterogeneous network with a meta-path-based random walk and heterogeneous skip-gram model can solve the representation learning problem owing to the sophisticated relationships among the applicants' information. Furthermore, the final subtasks can be predicted using a multi-task learning approach with two prediction layers. The first layer provides the probabilities of general fraud categories as auxiliary information for the second layer, which is for specific subtask prediction. Finally, we conducted extensive experiments based on a real-world dataset to demonstrate the effectiveness of the proposed approach.

An approach framework of transfer learning, adversarial training and hierarchical multi-task learning - a case study of disinformation detection with offensive text

With society going online and disinformation getting accepted as a phenomena that we have to live with, there is a growing need to automatically detect offensive text on modern social media platforms. But the lack of enough balanced labeled data, constantly evolving socio-linguistic patterns and ever-changing definition of offensive text make it a challenging task. This is a common pattern witnessed in all disinformation detection tasks such as detection of propaganda, rumour, fake news, hate etc. The work described in this paper improves upon the existing body of techniques by bringing in an approach framework that can surpass the existing benchmarks. Firstly, it addresses the imbalanced and insufficient nature of available labeled dataset. Secondly, learning using relates tasks through multi-task learning has been proved to be an effective approach in this domain but it has the unrealistic requirement of labeled data for all related tasks. The framework presented here suitably uses transfer learning in lieu of multi-task learning to address this issue. Thirdly, it builds a model explicitly addressing the hierarchical nature in the taxonomy of disinformation being detected as that delivers a stronger error feedback to the learning tasks. Finally, the model is made more robust by adversarial training. The work presented in this paper uses offensive text detection as a case study and shows convincing results for the chosen approach. The framework adopted can be easily replicated in other similar learning tasks facing a similar set of challenges.

End-to-end aspect-based sentiment analysis with hierarchical multi-task learning

End-to-end aspect-based sentiment analysis (E2E-ABSA) is a sequence labeling task which detects aspect terms and the corresponding sentiment simultaneously. Previous works ignore the useful task-specific knowledge and embed the vital aspect and sentiment attributes implicitly in the intermediate layers. In this paper, we propose a hierarchical multi-task learning framework, which explicitly leverages task-related knowledge via the supervision of intermediate layers. Specifically, aspect term extraction, sentiment lexicon detection, and aspect sentiment detection are designed to encode the aspect boundary and sentiment information. The tasks are in charge of different perspectives and levels of knowledge, which provide multi-fold regulation effects to optimize the main task. Unlike vanilla multi-task learning, all the tasks are integrated into a hierarchical structure to help the higher-level tasks make full use of the lower-level tasks’ information. Experimental results on three datasets demonstrate that the proposed method achieves state-of-the-art results. Further analysis shows that the proposed method achieves better performance than single-task and vanilla multi-task learning methods and yields a more discriminative feature representation.

HMD-ARG: hierarchical multi-task deep learning for annotating antibiotic resistance genes

BackgroundThe spread of antibiotic resistance has become one of the most urgent threats to global health, which is estimated to cause 700,000 deaths each year globally. Its surrogates, antibiotic resistance genes (ARGs), are highly transmittable between food, water, animal, and human to mitigate the efficacy of antibiotics. Accurately identifying ARGs is thus an indispensable step to understanding the ecology, and transmission of ARGs between environmental and human-associated reservoirs. Unfortunately, the previous computational methods for identifying ARGs are mostly based on sequence alignment, which cannot identify novel ARGs, and their applications are limited by currently incomplete knowledge about ARGs.ResultsHere, we propose an end-to-end Hierarchical Multi-task Deep learning framework for ARG annotation (HMD-ARG). Taking raw sequence encoding as input, HMD-ARG can identify, without querying against existing sequence databases, multiple ARG properties simultaneously, including if the input protein sequence is an ARG, and if so, what antibiotic family it is resistant to, what resistant mechanism the ARG takes, and if the ARG is an intrinsic one or acquired one. In addition, if the predicted antibiotic family is beta-lactamase, HMD-ARG further predicts the subclass of beta-lactamase that the ARG is resistant to. Comprehensive experiments, including cross-fold validation, third-party dataset validation in human gut microbiota, wet-experimental functional validation, and structural investigation of predicted conserved sites, demonstrate not only the superior performance of our method over the state-of-art methods, but also the effectiveness and robustness of the proposed method.ConclusionsWe propose a hierarchical multi-task method, HMD-ARG, which is based on deep learning and can provide detailed annotations of ARGs from three important aspects: resistant antibiotic class, resistant mechanism, and gene mobility. We believe that HMD-ARG can serve as a powerful tool to identify antibiotic resistance genes and, therefore mitigate their global threat. Our method and the constructed database are available at http://www.cbrc.kaust.edu.sa/HMDARG/.2a9o_5z-RdRo9vpmWq5oHNVideo abstract (MP4 50984 kb)

Hierarchical Microbial Functions Prediction by Graph Aggregated Embedding.

Matching 16S rRNA gene sequencing data to a metabolic reference database is a meaningful way to predict the metabolic function of bacteria and archaea, bringing greater insight to the working of the microbial community. However, some operational taxonomy units (OTUs) cannot be functionally profiled, especially for microbial communities from non-human samples cultured in defective media. Therefore, we herein report the development of Hierarchical micrObial functions Prediction by graph aggregated Embedding (HOPE), which utilizes co-occurring patterns and nucleotide sequences to predict microbial functions. HOPE integrates topological structures of microbial co-occurrence networks with k-mer compositions of OTU sequences and embeds them into a lower-dimensional continuous latent space, while maximally preserving topological relationships among OTUs. The high imbalance among KEGG Orthology (KO) functions of microbes is recognized in our framework that usually yields poor performance. A hierarchical multitask learning module is used in HOPE to alleviate the challenge brought by the long-tailed distribution among classes. To test the performance of HOPE, we compare it with HOPE-one, HOPE-seq, and GraphSAGE, respectively, in three microbial metagenomic 16s rRNA sequencing datasets, including abalone gut, human gut, and gut of Penaeus monodon. Experiments demonstrate that HOPE outperforms baselines on almost all indexes in all experiments. Furthermore, HOPE reveals significant generalization ability. HOPE's basic idea is suitable for other related scenarios, such as the prediction of gene function based on gene co-expression networks. The source code of HOPE is freely available at https://github.com/adrift00/HOPE.

Simultaneous Diagnosis of Severity and Features of Diabetic Retinopathy in Fundus Photography Using Deep Learning.

Deep learning methods for diabetic retinopathy (DR) diagnosis are usually criticized as being lack of interpretability in the diagnostic result, thus limiting their application in clinic. Simultaneous prediction of DR related features during the DR severity diagnosis is able to resolve this issue by providing supporting evidence (i.e. DR related features) for the diagnostic result (i.e. DR severity). In this study, we propose a hierarchical multi-task deep learning framework for simultaneous diagnosis of DR severity and DR related features in fundus images. A hierarchical structure is introduced to incorporate the casual relationship between DR related features and DR severity levels. In the experiments, the proposed approach was evaluated on two independent testing sets using quadratic weighted Cohen's kappa coefficient, receiver operating characteristic analysis, and precision-recall analysis. A grader study was also conducted to compare the performance of the proposed approach with those of general ophthalmologists with different levels of experience. The results demonstrate that the proposed approach could improve the performance for both DR severity diagnosis and DR related feature detection when comparing with the traditional deep learning-based methods. It achieves performance close to general ophthalmologists with five years of experience when diagnosing DR severity levels, and general ophthalmologists with ten years of experience for referable DR detection.

Multi-modal transportation recommendation with unified route representation learning

Multi-modal transportation recommendation aims to provide the most appropriate travel route with various transportation modes according to certain criteria. After analyzing large-scale navigation data, we find that route representations exhibit two patterns: spatio-temporal autocorrelations within transportation networks and the semantic coherence of route sequences. However, there are few studies that consider both patterns when developing multi-modal transportation systems. To this end, in this paper, we study multi-modal transportation recommendation with unified route representation learning by exploiting both spatio-temporal dependencies in transportation networks and the semantic coherence of historical routes. Specifically, we propose to unify both dynamic graph representation learning and hierarchical multi-task learning for multi-modal transportation recommendations. Along this line, we first transform the multi-modal transportation network into time-dependent multi-view transportation graphs and propose a spatiotemporal graph neural network module to capture the spatial and temporal autocorrelation. Then, we introduce a coherent-aware attentive route representation learning module to project arbitrary-length routes into fixed-length representation vectors, with explicit modeling of route coherence from historical routes. Moreover, we develop a hierarchical multi-task learning module to differentiate route representations for different transport modes, and this is guided by the final recommendation feedback as well as multiple auxiliary tasks equipped in different network layers. Extensive experimental results on two large-scale real-world datasets demonstrate the performance of the proposed system outperforms eight baselines.

RAHM: Relation augmented hierarchical multi-task learning framework for reasonable medication stocking.

As an important task in digital preventive healthcare management, especially in the secondary prevention stage, active medication stocking refers to the process of preparing necessary medications in advance according to the predicted disease progression of patients. However, predicting preventive or even life-saving medicine for each patient is a non-trivial task. Existing models usually overlook the implicit hierarchical relation between patient's predicted diseases and medications, and mainly focus on single tasks (medication recommendation or disease prediction). To tackle this limitation, we propose a relation augmented hierarchical multi-task learning framework, named RAHM. which is capable of learning multi-level relation-aware patient representation for reasonable medication stocking. Specifically, the framework first leverages the underlying structural relations of Electronic Health Record (EHR) data to learn the low-level patient visit representation. Then, it uses a regular LSTM to encode the historical temporal disease information for disease-level patient representation learning. Further, a relation-aware LSTM (R-LSTM) is proposed to handle the relations between diseases and medication in longitudinal patient records, which can better integrate the historical information into the medication-level patient representation. In the learning process, two pseudo residual structures are introduced to mitigate the error propagation and preserve the valuable relation information of EHRs. To validate our method, extensive experiments have been conducted based on the real-world clinical dataset. The results demonstrate a consistent superiority of our framework over several baselines in suggesting reasonable stock medication.

Coarse-to-Fine Speech Emotion Recognition Based on Multi-Task Learning

Speech emotion recognition is very challenging because the definition of emotion is uncertain and the feature representation is complex. Accurate feature representation is one of the key factors for successful speech emotion recognition. Studies have shown that 3D data composed of static, deltas and delta-deltas of log-Mel spectrum is very effective in filtering irrelevant features. The challenge of speech emotion recognition is also reflected in the necessity of fine-grained classification. Typical applications of affective computing, such as psychological counseling and emotion regulation, require fine-grained emotion recognition. Based on the two inspirations, this paper proposes an end-to-end hierarchical multi-task learning framework, from coarse to fine to achieve fine-grained emotion recognition. Using 3D data as input, in the first stage, we train the coarse emotion type, and then use the result to assist the second stage training for the fine emotion type. By conducting the comparative experiments on the IEMOCAP corpus, we find that the classification idea of coarse-to-fine has a significant performance improvement over the baseline models.