Task-specific Representations Research Articles

Cascaded matching based on detection box area for multi-object tracking

In the existing tracking-by-detection paradigm, advanced approaches rely on appearance features to establish associations between current detections and trajectories. However, these methods are often plagued by issues such as sluggish tracking performance and suboptimal results, particularly when confronted with the unreliability of the appearance features. Considering these challenges, we propose a novel cascaded matching algorithm called the detection box area-based tracking algorithm (DBAT), which groups the detection boxes by area size and associates detections within each group in a cascaded manner. To enhance the accuracy of grouping, we introduce two crucial components to enhance the quality of detections: the compressed self-decoding module (CSDM) and the task collaboration module (TCM). To acquire more precise location information and augment feature richness, CSDM decomposes the input features into two one-dimensional feature encodings and one two-dimensional feature encoding. Subsequently, these feature encodings perform feature aggregation along both spatial directions to capture long-range dependencies and refine the accuracy of location information. Ultimately, these aggregated features engage with the original features, facilitating information fusion and elevating the overall feature representation. To alleviate potential conflicts between various tasks and bolster task-specific representations, TCM combines disparate receptive fields and decouples features through self-relationship and cross-relationship mappings, thereby concurrently enhancing learning across different tasks. Extensive experiments demonstrate that our proposed method achieves performance comparable to state-of-the-art methods on the MOT17, MOT20 and DanceTrack benchmark tests.

Disentanglement by Cyclic Reconstruction.

Deep neural networks have demonstrated their ability to automatically extract meaningful features from data. However, in supervised learning, information specific to the dataset used for training, but irrelevant to the task at hand, may remain encoded in the extracted representations. This remaining information introduces a domain-specific bias, weakening the generalization performance. In this work, we propose splitting the information into a task-related representation and its complementary context representation. We propose an original method, combining adversarial feature predictors and cyclic reconstruction, to disentangle these two representations in the single-domain supervised case. We then adapt this method to the unsupervised domain adaptation (UDA) problem, consisting of training a model capable of performing on both a source and a target domain. In particular, our method promotes disentanglement in the target domain, despite the absence of training labels. This enables the isolation of task-specific information from both domains and a projection into a common representation. The task-specific representation allows the efficient transfer of knowledge acquired from the source domain to the target domain. In the single-domain case, we demonstrate the quality of our representations on information retrieval tasks and the generalization benefits induced by sharpened task-specific representations. We then validate the proposed method on several classical domain adaptation (DA) benchmarks and illustrate the benefits of disentanglement for DA.

Generative Environment-Representation Instance-Based Learning: A Cognitive Model

Instance-Based Learning Theory (IBLT) suggests that humans learn to engage in dynamic decision making tasks through the accumulation of experiences, represented by the decision task features, the actions performed, and the utility of decision outcomes. This theory has been applied to the design of Instance-Based Learning (IBL) models of human behavior in a variety of contexts. One key feature of all IBL model applications is the method of accumulating instance-based memory and performing recognition-based retrieval. In simple tasks with few features, this knowledge representation and retrieval could hypothetically be done using all relevant information. However, these methods do not scale well to complex tasks when exhaustive enumeration of features is unfeasible. This requires cognitive modelers to design task-specific representations of state features, as well as similarity metrics, which can be time consuming and fail to generalize to related tasks. To address this issue, we leverage recent advancements in Artificial Neural Networks, specifically generative models (GMs), to learn representations of complex dynamic decision making tasks without relying on domain knowledge. We evaluate a range of GMs in their usefulness in forming representations that can be used by IBL models to predict human behavior in a complex decision making task. This work connects generative and cognitive models by using GMs to form representations and determine similarity.

GPL-GNN: Graph prompt learning for graph neural network

Despite the impressive results achieved in many areas of graph machine learning, through graph representation learning using supervised learning techniques, the limited availability of labeled training data has led to a bottleneck in terms of performance. To address this challenge, transfer learning has been proposed as an effective solution. It involves designing pre-training methods in an unsupervised manner to learn representations, which are then adapted to downstream tasks with limited labeled data. However, transfer learning can suffer from negative transfer when there is a major gap between the objectives of pre-training and the downstream tasks. To overcome these challenges, we introduce a novel framework, graph prompt learning-graph neural network (GPL-GNN), to narrow the gap between different tasks. GPL-GNN employs unsupervised methods, which require no labeled data, and incorporates unsupervised pre-trained structural representations into downstream tasks as prompt information. This information is combined with downstream data to train GNNs adapting them to the downstream tasks, and resulting in more adaptive, task-specific representations. Furthermore, the ability of GPL-GNN to learn graph representations without the constraints of pre-training and fine-tuning for model consistency increases the flexibility in choosing task-specific GNNs. In addition, the introduction of prototype networks as classification heads enables quick adaptation of GPL-GNNs to downstream tasks. Finally, we conduct extensive experiments on a benchmark dataset to demonstrate the effectiveness of GPL-GNN. The code is available in: https://github.com/chenzihaoww/GPL-GNN.

Comprehensive Analysis of Mammography Images Using Multi-Branch Attention Convolutional Neural Network

Breast cancer profoundly affects women’s lives; its early diagnosis and treatment increase patient survival chances. Mammography is a common screening method for breast cancer, and many methods have been proposed for automatic diagnosis. However, most of them focus on single-label classification and do not provide a comprehensive analysis concerning density, abnormality, and severity levels. We propose a method based on the multi-label classification of two-view mammography images to comprehensively diagnose a patient’s condition. It leverages the correlation between density type, lesion type, and states of lesions, which radiologists usually perform. It simultaneously classifies mammograms into the corresponding density, abnormality type, and severity level. It takes two-view mammograms (with craniocaudal and mediolateral oblique views) as input, analyzes them using ConvNeXt and the channel attention mechanism, and integrates the information from the two views. Finally, the fused information is passed to task-specific multi-branches, which learn task-specific representations and predict the relevant state. The system was trained, validated, and tested using two public domain benchmark datasets, INBreast and the Curated Breast Imaging Subset of DDSM (CBIS-DDSM), and achieved state-of-the-art results. The proposed computer-aided diagnosis (CAD) system provides a holistic observation of a patient’s condition. It gives the radiologists a comprehensive analysis of the mammograms to prepare a full report of the patient’s condition, thereby increasing the diagnostic precision.

Multimodal contrastive learning for face anti-spoofing

Multimodal face anti-spoofing systems adopt multiple sensor modalities, such as infrared, color, depth, and thermal, to distinguish between living and spoofing faces via complementary spoofing clues from each modality. One challenge is that when the multimodal face anti-spoofing system is placed in different environments, the sensor setup may not be unified, causing a certain sensor to be unavailable. To alleviate this issue, a two-stream face anti-spoofing method is proposed. The first stream focuses on extracting primary features from an available sensor by a baseline network. The second stream employs a multimodal contrastive learning strategy to acquire modality-agnostic and task-specific representations from another deployed sensor. Furthermore, a master–slave modulation fusion block is designed to effectively fuse features from the two streams. Experiments conducted on three public multimodal databases show the superior performance of the proposed method.

Classification of Tastants: A Deep Learning Based Approach.

Predicting the taste of molecules is of critical importance in the food and beverages, flavor, and pharmaceutical industries for the design and screening of new tastants. In this work, we have built deep learning models to classify sweet, bitter, and umami molecules- the three basic tastes whose sensation is mediated by G protein-coupled receptors. An extensive dataset containing 1466 bitter, 1764 sweet, and 238 umami tastants was curated from existing literature. We analyzed the chemical characteristics of the molecules, with special focus on the presence of different functional groups. A deep neural network model based on molecular descriptors and a graph neural network model were trained for taste prediction. The class imbalance due to fewer umami molecules was tackled using special sampling techniques. Both models show comparable performance during evaluation, but the graph-based model can learn task-specific representations from the molecular structure without requiring handcrafted features. We further explain the deep neural network predictions using Shapley additive explanations. Finally, we demonstrated the applicability of the models by screening bitter, sweet, and umami molecules from a large food database. This study develops an in-silico approach to classify molecules based on their taste by leveraging the recent progress in deep learning, which can serve as a powerful tool for tastant design.

Transformability, generalizability, but limited diffusibility: Comparing global vs. task-specific language representations in deep neural networks

This study investigates the integration of two prominent neural network representations into a hybrid cognitive model for solving a natural language task, where pre-trained large-language models serve as global learners and recurrent neural networks offer more “local” task-specific representations in the neural network. To explore the fusion of these two types of representations, we employ an autoencoder to transform them between each other or fuse them into a single model. Our exploration identifies a computational constraint, which we term limited diffusibility, highlighting the limitations of hybrid systems that operate with distinct types of representation. The findings from our hybrid system confirm the crucial role of global knowledge in adapting to a new learning task, as having only local knowledge greatly reduces the system’s transferability.

Simple But Powerful, a Language-Supervised Method for Image Emotion Classification

Image emotion classification is an important computer vision task to extract emotions from images. The methods for image emotion classification (IEC) are primarily based on label or distribution as a supervision signal, which neither has enough accessibility nor diversity, limiting the development of IEC research. Inspired by psychology research and the recent booming of large-scale pretrained language models. We figure out a language-supervised paradigm, which can cleverly combine the features of language and visual emotion to drive the visual model to gain stronger emotional discernment with language prompts. To practice the paradigm, we present a conceptually simple while empirically powerful framework for image emotion classification, SimEmotion. That we propose a prompt-based fine-tuning strategy to learn task-specific representations by composing a template with the emotion-level concept and entity-level information. Evaluations on four widely-used affective datasets, namely, Flickr and Instagram (FI), EmotionROI, Twitter I, and Twitter II, demonstrate that the proposed algorithm outperforms the state-of-the-art methods with a large margin ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , 8.42% absolute accuracy gain on EmotionROI) on image emotion classification tasks. Our codes will be publicly available for research purposes.

Boosting Low-Resource Speech Recognition in Air Traffic Communication via Pretrained Feature Aggregation and Multi-Task Learning

Developing a robust Automatic Speech Recognition (ASR) system usually requires a large amount of well-annotated samples which is extremely hard to build in the Air Traffic Control (ATC) due to domain-specific knowledge. In this work, we present a novel approach to improve ASR performance in the ATC domain by integrating Self-Supervised Learning and Multi-Task learning into a unified framework. Specifically, the proposed framework follows a two-stage training paradigm, i.e., (a) learning universal acoustic representations by employing the wav2vec 2.0 model and (b) jointly finetuning the model by the ASR, Speaker Role Identification, and Language Identification tasks. To capture the task-specific representations, an attention-guided feature aggregation module is dedicatedly designed to disentangle the discriminative representations from the pretrained features. In addition, the uncertainty-based loss combination strategy is employed to balance the loss weights for each task in a learnable manner. Finally, we conduct experiments to validate the technical improvements in a real-world ATC dataset. Experimental results demonstrated that the proposed framework outperforms competitive baselines among all tasks.

Directed Message Passing Neural Network for Predicting Power Conversion Efficiency in Organic Solar Cells.

Organic solar cells (OSCs) have emerged as a promising technology for renewable energy generation, and researchers are constantly exploring ways to improve their efficiency. For prediction of photovoltaic properties in OSCs, many machine learning models have been used in the past. All the models are used with fixed molecular descriptors and molecular fingerprints as input for power conversion efficiency (PCE) prediction. Recently, the graph neural network (GNN), which can model graph structures of the molecule, has received increasing attention as a method that could potentially overcome the limitations of fixed descriptors by learning the task-specific representations using graph convolutions. In this study, we have used the directed message passing neural network (D-MPNN), an emerging type of GNN for predicting PCE of organic solar cells, and the results are compared for the same train and test set with fixed descriptors and fingerprints. The excellent performance demonstrated by the D-MPNN model in this investigation highlights its potential for predicting PCE, surpassing the limitations of conventional fixed descriptors.

Assessment of Driver's Stress using Multimodal Biosignals and Regularized Deep Kernel Learning.

In this work, we classify the stress state of car drivers using multimodal physiological signals and regularized deep kernel learning. Using a driving simulator in a controlled environment, we acquire electrocardiography (ECG), electrodermal activity (EDA), photoplethysmography (PPG), and respiration rate (RESP) from N = 10 healthy drivers in experiments of 25min duration with different stress states (5min resting, 10min driving, 10min driving + answering cognitive questions). We manually remove unusable segments and approximately 4h of data remain. Multimodal time and frequency features are extracted and employed to regularized deep kernel machine learning based on a fusion framework. Task-specific representations of different physiological signals are combined using intermediate fusion. Subsequently, the fused multimodal features are fed a support vector machine (SVM) and a random forest (RF) for stress classification. The experimental results show that the proposed approach can discriminate between stress states. The combination of PPG and ECG using RF as classifier yields the highest F1-score of 0.97 in the test set. PPG only and RF yield a maximum F1-score of 0.90. Furthermore, subject-specific cross-validation improves performance. ECG and PPG signals are reliable in classifying the stress state of a car driver. In summary, the proposed framework could be extended to real-time stress state assessment in driving conditions.

Provable Pathways: Learning Multiple Tasks over Multiple Paths

Constructing useful representations across a large number of tasks is a key requirement for sample-efficient intelligent systems. A traditional idea in multitask learning (MTL) is building a shared representation across tasks which can then be adapted to new tasks by tuning last layers. A desirable refinement of using a shared one-fits-all representation is to construct task-specific representations. To this end, recent PathNet/muNet architectures represent individual tasks as pathways within a larger supernet. The subnetworks induced by pathways can be viewed as task-specific representations that are composition of modules within supernet's computation graph. This work explores the pathways proposal from the lens of statistical learning: We first develop novel generalization bounds for empirical risk minimization problems learning multiple tasks over multiple paths (Multipath MTL). In conjunction, we formalize the benefits of resulting multipath representation when adapting to new downstream tasks. Our bounds are expressed in terms of Gaussian complexity, lead to tangible guarantees for the class of linear representations, and provide novel insights into the quality and benefits of a multipath representation. When computation graph is a tree, Multipath MTL hierarchically clusters the tasks and builds cluster-specific representations. We provide further discussion and experiments for hierarchical MTL and rigorously identify the conditions under which Multipath MTL is provably superior to traditional MTL approaches with shallow supernets.

Generalized Zero-Shot Image Classification via Partially-Shared Multi-Task Representation Learning

Generalized Zero-Shot Learning (GZSL) holds significant research importance as it enables the classification of samples from both seen and unseen classes. A prevailing approach for GZSL is learning transferable representations that can generalize well to both seen and unseen classes during testing. This approach encompasses two key concepts: discriminative representations and semantic-relevant representations. “Semantic-relevant” facilitates the transfer of semantic knowledge using pre-defined semantic descriptors, while “discriminative” is crucial for accurate category discrimination. However, these two concepts are arguably inherently conflicting, as semantic descriptors are not specifically designed for image classification. Existing methods often struggle with balancing these two aspects and neglect the conflict between them, leading to suboptimal representation generalization and transferability to unseen classes. To address this issue, we propose a novel partially-shared multi-task representation learning method, termed PS-GZSL, which jointly preserves complementary and sharable knowledge between these two concepts. Specifically, we first propose a novel perspective that treats the learning of discriminative and semantic-relevant representations as optimizing a discrimination task and a visual-semantic alignment task, respectively. Then, to learn more complete and generalizable representations, PS-GZSL explicitly factorizes visual features into task-shared and task-specific representations and introduces two advanced tasks: an instance-level contrastive discrimination task and a relation-based visual-semantic alignment task. Furthermore, PS-GZSL employs Mixture-of-Experts (MoE) with a dropout mechanism to prevent representation degeneration and integrates a conditional GAN (cGAN) to synthesize unseen features for estimating unseen visual features. Extensive experiments and more competitive results on five widely-used GZSL benchmark datasets validate the effectiveness of our PS-GZSL.

Multi-task learning for few-shot biomedical relation extraction

Artificial intelligence (AI) has advanced rapidly, but it has limited impact on biomedical text understanding due to a lack of annotated datasets (a.k.a. few-shot learning). Multi-task learning, which uses data from multiple datasets and tasks with related syntax and semantics, has potential to address this issue. However, the effectiveness of this approach heavily relies on the quality of the available data and its transferability between tasks. In this paper, we propose a framework, built upon a state-of-the-art multi-task method (i.e. MT-DNN), that leverages different publicly available biomedical datasets to enhance relation extraction performance. Our model employs a transformer-based architecture with shared encoding layers across multiple tasks, and task-specific classification layers to generate task-specific representations. To further improve performance, we utilize a knowledge distillation technique. In our experiments, we assess the impact of incorporating biomedical datasets in a multi-task learning setting and demonstrate that it consistently outperforms state-of-the-art few-shot learning methods in cases of limited data. This results in significant improvement across most datasets and few-shot scenarios, particularly in terms of recall scores.

Knowledge distillation-enhanced multitask framework for recommendation

With the ever-growing amount of online information, recommender systems (RSs) act as information filtering tools and are widely used in various e-commerce platforms. Recommender methods generally adopt only one type of behavior data for single-task learning. Multitask learning is commonly used to simultaneously address multiple tasks utilizing as much information as possible. We propose a knowledge distillation-enhanced shared-bottom model for recommender multitask learning. The postview click-through rate, postview conversion rate and average transaction value are learned together to determine the final gross merchandise volume. First, a mixture of experts with gate networks is used as a shared bottom to learn task-specific representations for all tasks, while a tree distillation framework is designed as an expert for better feature selection. Then, we design a neural factorization machine as a task-specific prediction network to estimate individual goals. Finally, an objective function is designed based on task outputs to optimize the gross merchandise volume. Our proposed framework can enhance recommender preferences while avoiding generating a large number of parameters.This model serves the e-commerce scenario, with the following specific use scenario: building personalized GMV predictions for small scenarios on e-commerce platforms at low cost, which is a 0 to 1 process that is often highly rewarding. Ultimately, we propose to improve the reliability of e-commerce scenarios.

Cerebro-cerebellar networks facilitate learning through feedback decoupling

Behavioural feedback is critical for learning in the cerebral cortex. However, such feedback is often not readily available. How the cerebral cortex learns efficiently despite the sparse nature of feedback remains unclear. Inspired by recent deep learning algorithms, we introduce a systems-level computational model of cerebro-cerebellar interactions. In this model a cerebral recurrent network receives feedback predictions from a cerebellar network, thereby decoupling learning in cerebral networks from future feedback. When trained in a simple sensorimotor task the model shows faster learning and reduced dysmetria-like behaviours, in line with the widely observed functional impact of the cerebellum. Next, we demonstrate that these results generalise to more complex motor and cognitive tasks. Finally, the model makes several experimentally testable predictions regarding cerebro-cerebellar task-specific representations over learning, task-specific benefits of cerebellar predictions and the differential impact of cerebellar and inferior olive lesions. Overall, our work offers a theoretical framework of cerebro-cerebellar networks as feedback decoupling machines.

Heterogeneous Multi-Task Learning With Expert Diversity.

Predicting multiple heterogeneous biological and medical targets is a challenge for traditional deep learning models. In contrast to single-task learning, in which a separate model is trained for each target, multi-task learning (MTL) optimizes a single model to predict multiple related targets simultaneously. To address this challenge, we propose the Multi-gate Mixture-of-Experts with Exclusivity (MMoEEx). Our work aims to tackle the heterogeneous MTL setting, in which the same model optimizes multiple tasks with different characteristics. Such a scenario can overwhelm current MTL approaches due to the challenges in balancing shared and task-specific representations and the need to optimize tasks with competing optimization paths. Our method makes two key contributions: first, we introduce an approach to induce more diversity among experts, thus creating representations more suitable for highly imbalanced and heterogenous MTL learning; second, we adopt a two-step optimization (Finn et al., 2017 and Lee et al., 2020) approach to balancing the tasks at the gradient level. We validate our method on three MTL benchmark datasets, including UCI-Census-income dataset, Medical Information Mart for Intensive Care (MIMIC-III) and PubChem BioAssay (PCBA).

A multi-task learning model for building electrical load prediction

Buildings are one of the largest energy-consuming sectors in the world. Accurate forecasting of building electricity loads can bring significant environmental and economic benefits by reducing electricity use and the corresponding greenhouse gas emissions. Deep learning has witnessed great success in the past decade, so this work proposes a deep learning model to predict the electrical loads of a commercial building. We perform a data exploration on the training set, and the results show that the electricity load is relevant to the temperature. Thus, this work proposes a deep learning model based on multi-task learning (MTL) architecture to predict the hourly electricity load. In our proposed architecture, the main task is to predict the electricity load, while the auxiliary task is to predict the outdoor temperature. The auxiliary task can provide additional regularization to the model to prevent overfitting. Furthermore, the proposed model comprises task-shared and task-specific layers, giving a base to learn cross-task and task-specific representations, respectively. We conduct experiments to assess our proposed model and compare it with other alternatives. The experimental results show that our proposed model can significantly outperform the comparison methods. Furthermore, the analysis shows that our proposed model can benefit from a simple ensemble technique to improve the prediction performance. To validate the generalization of the proposed model, we perform a robustness analysis on three additional datasets. We demonstrate that the proposed MTL approach can provide superior predictive accuracy and robustness.

Task Encoding With Distribution Calibration for Few-Shot Learning

Few-shot learning is an extremely challenging task in computer vision that has attracted increased research attention in recent years. However, most recent methods do not fully use the task’s information, and few of the seen samples result in large intraclass differences among the same classes. In this paper, we propose a novel task encoding with distribution calibration (TEDC) model for few-shot learning, which uses the relationships among the feature distributions to reduce intraclass differences. In the TEDC model, an integrated feature extraction module (IFEM) is proposed, which extracts the multiangle visual features of an image and fuses them to obtain more representative features. To effectively utilize the task information, a novel task encoding module (TEM) is proposed, which obtains the task features by fusing all the seen samples’ information and uses them to adjust all the samples’ features for more generalizable task-specific representations. We also propose a distribution calibration module (DCM) to reduce the bias between the distribution of the support features and the query features in the same class. Extensive experiments show that our proposed TEDC model achieves an excellent performance and outperforms the state-of-the-art methods on three widely used few-shot classification benchmarks, specifically miniImageNet, tieredImageNet and CUB-200-2011.