Task Representation Research Articles

Two-stage fine-tuning with ChatGPT data augmentation for learning class-imbalanced data

Classification of long-tailed distributed data is a challenging problem, which suffers from serious class imbalance and hence poor performance on tail classes, which have only a few samples. Owing to this paucity of samples, learning on the tail classes is especially challenging for fine-tuning when transferring a pretrained model to a downstream task. In this work, we present a simple modification of standard fine-tuning to cope with these challenges. Specifically, we propose a two-stage fine-tuning. In Stage 1, we fine-tune the final layer of the pretrained model with class-balanced augmented data, generated using ChatGPT. As a large generative language model, ChatGPT is capable of generating novel and contextually similar responses to a given prompt, which makes it an excellent candidate for data augmentation. In Stage 2, we perform the standard fine-tuning. Our modification has several benefits: (1) it leverages pretrained representations by only fine-tuning a small portion of the model parameters while keeping the rest untouched; (2) it allows the model to learn an initial representation of the specific task; and importantly (3) it protects the learning of tail classes from being at a disadvantage during the model updating. We conduct extensive experiments on synthetic datasets of both two-class and multi-class tasks of text classification as well as a real-world application to ADME (i.e., absorption, distribution, metabolism, and excretion) semantic drug labeling. The experimental results show that the proposed two-stage fine-tuning outperforms vanilla fine-tuning and state-of-the-art methods on the above datasets.

Research on Bidirectional Multi-Span Feature Pyramid and Key Feature Capture Object Detection Network

UAV remote sensing (RS) image object detection is a very valuable and challenging technology. This article discusses the importance of key features and proposes an object detection network (URSNet) based on a bidirectional multi-span feature pyramid and key feature capture mechanism. Firstly, a bidirectional multi-span feature pyramid (BMSFPN) is constructed. In the process of bidirectional sampling, bicubic interpolation and cross layer fusion are used to filter out image noise and enhance the details of object features. Secondly, the designed feature polarization module (FPM) uses the internal polarization attention mechanism to build a powerful feature representation for classification and regression tasks, making it easier for the network to capture the key object features with more semantic discrimination. In addition, the anchor rotation alignment module (ARAM) further refines the preset anchor frame based on the key regression features extracted by FPM to obtain high-quality rotation anchors with a high matching degree and rich positioning visual information. Finally, the dynamic anchor optimization module (DAOM) is used to improve the ability of feature alignment and positive and negative sample discrimination of the model so that the model can dynamically select the candidate anchor to capture the key regression features so as to further eliminate the deviation between the classification and regression. URSNet has conducted comprehensive ablation and SOTA comparative experiments on challenging RS datasets such as DOTA-V2.0, DIOR and RSOD. The optimal experimental results (87.19% mAP, 108.2 FPS) show that URSNet has efficient and reliable detection performance.

Stimulating prefrontal cortex facilitates training transfer by increasing representational overlap.

A recent hypothesis characterizes difficulties in multitasking as being the price humans pay for our ability to generalize learning across tasks. The mitigation of these costs through training has been associated with reduced overlap of constituent task representations within frontal, parietal, and subcortical regions. Transcranial direct current stimulation, which can modulate functional brain activity, has shown promise in generalizing performance gains when combined with multitasking training. However, the relationship between combined transcranial direct current stimulation and training protocols with task-associated representational overlap in the brain remains unexplored. Here, we paired prefrontal cortex transcranial direct current stimulation with multitasking training in 178 individuals and collected functional magnetic resonance imaging data pre- and post-training. We found that 1mA transcranial direct current stimulation applied to the prefrontal cortex paired with multitasking training enhanced training transfer to spatial attention, as assessed via a visual search task. Using machine learning to assess the overlap of neural activity related to the training task in task-relevant brain regions, we found that visual search gains were predicted by changes in classification accuracy in frontal, parietal, and cerebellar regions for participants that received left prefrontal cortex stimulation. These findings demonstrate that prefrontal cortex transcranial direct current stimulation may interact with training-related changes to task representations, facilitating the generalization of learning.

Latent Representation Learning for Geospatial Entities

Representation learning has been instrumental in the success of machine learning, offering compact and performant data representations for diverse downstream tasks. In the spatial domain, it has been pivotal in extracting latent patterns from various data types, including points, polylines, polygons, and networked structures. However, existing approaches often fall short of explicitly capturing both semantic and spatial information, relying on proxies and synthetic features. This paper presents GeoNN, a novel graph neural network-based model designed to learn spatially-aware embeddings for geospatial entities. GeoNN leverages edge features generated from geodesic functions, dynamically selecting relevant features based on relative locations. It introduces both transductive (GeoNN-T) and inductive (GeoNN-I) models, ensuring effective encoding of geospatial features and scalability with entity changes. Extensive experiments demonstrate GeoNN’s effectiveness in location-sensitive superpixel-based graphs and real-world points of interest, outperforming baselines across various evaluation measures.

OPTIMISING THE LADDER RAISE TASK IN THE PHYSICAL APTITUDE TEST FOR FIRE-FIGHTERS

INTRODUCTION & AIM Identifying physically capable candidates is a crucial component of recruiting firefighters given the physical demands of the occupation. At the request of a local fire agency, a review of one test item within the physical aptitude testing protocol was undertaken with the aim of this pilot study to compare physical and perceptual performance between the ladder raise task and a suite of surrogate tests to determine suitability for use in recruitment screening. METHODS 23 participants (unskilled in firefighting) performed the criterion ladder task (‘Ladder’) and three surrogate tasks: stationary constant-tension overhead press (existing test item; ‘Reel’); barbell overhead walk/return (‘Barbell’); overhead modular ladder lift (Modular Under-Running Apparatus (‘MURA’)). The tests were segmented into six components relating to lifting and under-running the device. Task duration, task completion, perceptual and kinetic variables were measured. Pearson’s correlation, OLP regressions and paired t-tests assessed relationships between the Ladder and surrogate tasks. RESULTS The Reel most closely reflected the criterion task in component 1 (lifting) for both peak and average forces (dz =-0.11, p=0.920; dz=-0.11, p=0.532 respectively). The MURA and Barbell were most closely aligned to the Ladder in all other objective values with trivial-small differences evident. The MURA was most closely aligned with the Ladder in subjective measures for components-2/5 and components-3/4 including difficulty (dz= -0.41, p=0.133; dz=-0.32, p=0.250, respectively), fatigue (dz= -0.25, p=0.354; dz=-0.10, p=0.796, respectively) and task representation (7.2[1.6]; 7.4[1.6], respectively). CONCLUSION The currently used Reel task revealed low construct and content validity. The MURA was shown to be the most appropriate surrogate option for the local fire agency to further investigate for use within recruitment physical testing. Further investigation with a sample of career firefighters should be undertaken to verify face validity at the industry level.

Binary Graph Convolutional Network With Capacity Exploration.

The current success of Graph Neural Networks (GNNs) usually relies on loading the entire attributed graph for processing, which may not be satisfied with limited memory resources, especially when the attributed graph is large. This paper pioneers to propose a Binary Graph Convolutional Network (Bi-GCN), which binarizes both the network parameters and input node attributes and exploits binary operations instead of floating-point matrix multiplications for network compression and acceleration. Meanwhile, we also propose a new gradient approximation based back-propagation method to properly train our Bi-GCN. According to the theoretical analysis, our Bi-GCN can reduce the memory consumption by an average of ∼31x for both the network parameters and input data, and accelerate the inference speed by an average of ∼51x, on three citation networks, i.e., Cora, PubMed, and CiteSeer. Besides, we introduce a general approach to generalize our binarization method to other variants of GNNs, and achieve similar efficiencies. Although the proposed Bi-GCN and Bi-GNNs are simple yet efficient, these compressed networks may also possess a potential capacity problem, i.e., they may not have enough storage capacity to learn adequate representations for specific tasks. To tackle this capacity problem, an Entropy Cover Hypothesis is proposed to predict the lower bound of the width of Bi-GNN hidden layers. Extensive experiments have demonstrated that our Bi-GCN and Bi-GNNs can give comparable performances to the corresponding full-precision baselines on seven node classification datasets and verified the effectiveness of our Entropy Cover Hypothesis for solving the capacity problem.

Extended residual learning with one-shot imitation learning for robotic assembly in semi-structured environment.

Robotic assembly tasks require precise manipulation and coordination, often necessitating advanced learning techniques to achieve efficient and effective performance. While residual reinforcement learning with a base policy has shown promise in this domain, existing base policy approaches often rely on hand-designed full-state features and policies or extensive demonstrations, limiting their applicability in semi-structured environments. In this study, we propose an innovative Object-Embodiment-Centric Imitation and Residual Reinforcement Learning (OEC-IRRL) approach that leverages an object-embodiment-centric (OEC) task representation to integrate vision models with imitation and residual learning. By utilizing a single demonstration and minimizing interactions with the environment, our method aims to enhance learning efficiency and effectiveness. The proposed method involves three key steps: creating an object-embodiment-centric task representation, employing imitation learning for a base policy using via-point movement primitives for generalization to different settings, and utilizing residual RL for uncertainty-aware policy refinement during the assembly phase. Through a series of comprehensive experiments, we investigate the impact of the OEC task representation on base and residual policy learning and demonstrate the effectiveness of the method in semi-structured environments. Our results indicate that the approach, requiring only a single demonstration and less than 1.2 h of interaction, improves success rates by 46% and reduces assembly time by 25%. This research presents a promising avenue for robotic assembly tasks, providing a viable solution without the need for specialized expertise or custom fixtures.

Disentangled Relational Graph Neural Network with Contrastive Learning for knowledge graph completion

Learning disentangled entity representations has garnered significant attention in the field of knowledge graph completion (KGC). However, the existing methods inherently overlook the indicative role of relations and the correlation between latent factors and relations, leading to suboptimal entity representations for KGC tasks. In the current study, we introduce the Disentangled Relational Graph Neural Network with Contrastive Learning (DRGCL) method, designed to acquire disentangled entity representations guided by relations. In particular, we first devise the factor-aware relational message aggregation approach to learn entity representations under each semantic subspace and obtain latent factor representations by attention mechanisms. Subsequently, we propose a discrimination objective for factor-subspace pairs using a contrastive learning approach, which compels the factor representations to distinctly capture the information associated with different latent factors and promote the consistency between factor representations and semantic subspaces. Through disentanglement, our model can generate relation-aware scores tailored to the provided scenario. Extensive experiments have been conducted on three benchmark datasets and the results demonstrate the superiority of our method compared with strong baseline models.

A contrastive learning approach to integrate spatial transcriptomics and histological images

The rapid growth of spatially resolved transcriptomics technology provides new perspectives on spatial tissue architecture. Deep learning has been widely applied to derive useful representations for spatial transcriptome analysis. However, effectively integrating spatial multi-modal data remains challenging. Here, we present ConGcR, a contrastive learning-based model for integrating gene expression, spatial location, and tissue morphology for data representation and spatial tissue architecture identification. Graph convolution and ResNet were used as encoders for gene expression with spatial location and histological image inputs, respectively. We further enhanced ConGcR with a graph auto-encoder as ConGaR to better model spatially embedded representations. We validated our models using 16 human brains, four chicken hearts, eight breast tumors, and 30 human lung spatial transcriptomics samples. The results showed that our models generated more effective embeddings for obtaining tissue architectures closer to the ground truth than other methods. Overall, our models not only can improve tissue architecture identification’s accuracy but also may provide valuable insights and effective data representation for other tasks in spatial transcriptome analyses.

Perceived authenticity across three forms of educational simulations—the role of interactant representation, task alignment, and continuity of simulation

Authenticity in simulation-based learning is linked to cognitive processes implicated in learning. However, evidence on authenticity across formats is insufficient. We compared three case-based settings and investigated the effect of discontinuity in simulation on perceived authenticity. In a quasi-experiment, we compared formats of simulation in the context of medical education. All formats simulated anamnestic interviews with varying interactant and task representations using highly comparable designs. Interactants (patients) were simulated by (a) live actors (standardized patients), (b) live fellow students (roleplays), or (c) question menus and videoclips (virtual patients). The continuity of simulations varied. We measured perceived authenticity with three subscales: Realness, Involvement, and Spatial Presence. We employed confirmatory factor analysis (CFA) to assess measurement invariance across settings and analysis of variance on authenticity ratings to compare the effects of setting and discontinuous simulation. CFA supported the assumption of invariance. Settings differed in Realness and Spatial Presence but not Involvement. Discontinuous simulations yielded significantly lower ratings of authenticity than continuous simulations. The compared simulation modalities offer different advantages with respect to their perceived authenticity profiles. Lower levels of interactivity and reduced subtask representation do not necessarily lead to lower ratings of perceived authenticity. Spatial Presence can be as high for media-based simulation as for roleplays. Discontinuation of simulations by offering scaffolding impairs perceived authenticity. Scaffolds may be designed to avoid discontinuation of simulation to uphold perceived authenticity.

Human-in-the-loop error detection in an object organization task with a social robot.

In human-robot collaboration, failures are bound to occur. A thorough understanding of potential errors is necessary so that robotic system designers can develop systems that remedy failure cases. In this work, we study failures that occur when participants interact with a working system and focus especially on errors in a robotic system's knowledge base of which the system is not aware. A human interaction partner can be part of the error detection process if they are given insight into the robot's knowledge and decision-making process. We investigate different communication modalities and the design of shared task representations in a joint human-robot object organization task. We conducted a user study (N = 31) in which the participants showed a Pepper robot how to organize objects, and the robot communicated the learned object configuration to the participants by means of speech, visualization, or a combination of speech and visualization. The multimodal, combined condition was preferred by 23 participants, followed by seven participants preferring the visualization. Based on the interviews, the errors that occurred, and the object configurations generated by the participants, we conclude that participants tend to test the system's limitations by making the task more complex, which provokes errors. This trial-and-error behavior has a productive purpose and demonstrates that failures occur that arise from the combination of robot capabilities, the user's understanding and actions, and interaction in the environment. Moreover, it demonstrates that failure can have a productive purpose in establishing better user mental models of the technology.

Self supervised learning based emotion recognition using physiological signals.

The significant role of emotional recognition in the field of human-machine interaction has garnered the attention of many researchers. Emotion recognition based on physiological signals can objectively reflect the most authentic emotional states of humans. However, existing labeled Electroencephalogram (EEG) datasets are often of small scale. In practical scenarios, a large number of unlabeled EEG signals are easier to obtain. Therefore, this paper adopts self-supervised learning methods to study emotion recognition based on EEG. Specifically, experiments employ three pre-defined tasks to define pseudo-labels and extract features from the inherent structure of the data. Experimental results indicate that self-supervised learning methods have the capability to learn effective feature representations for downstream tasks without any manual labels.

Sequence-based malware detection using a single-bidirectional graph embedding and multi-task learning framework

As an important part of malware detection and classification, sequence-based analysis can be integrated into dynamic detection system for real-time detection. This work presents a novel learning method for malware detection models that leverages advances in graph embedding for fusing the n-gram data into a one-hot feature space with different transmission directions. By capturing the information flow, our method finds a better feature representation for detection tasks with rely solely on sequence information. To enhance the stability of feature representation, this work adopts a multi-task learning strategy which achieves better performance in independent testing. We evaluate our method on two different realworld datasets and compare it against four superior malware detection models. During malware detection using our method, we conducted in-depth discussions on feature length, graph embedding direction, model depth, and different multi-task learning strategies. Experimental and discussion results show that our method significantly outperforms alternative approaches across evaluation settings.

Characteristics of students’ task representation and its association with argumentative integrated writing performance

Task representation denotes students’ interpretation in which what a learning or assessment task required them to do. An argumentative integrated writing task which involves the use of reading materials as claims or evidences for composing an essay, makes the role of task representation more critical than others, as writers may be confused with whether their task is to focus on synthesizing the reading materials that they comprehend, or expressing their own views. With the aim of exploring the characteristics of task representation and its association with integrated writing, this study invited 474 secondary four students from Hong Kong to participate in think aloud writing protocol followed by stimulated recall interview (36 participants), and complete an integrated writing task and a questionnaire (438 participants). Three factors of the task representation were identified as source use, rhetorical purpose and text format, and significant positive correlations were found between the three factors and integrated writing performance. Theoretical and pedagogical implications are discussed.

Similarity in evoked responses does not imply similarity in macroscopic network states.

It is commonplace in neuroscience to assume that if two tasks activate the same brain areas in the same way, then they are recruiting the same underlying networks. Yet computational theory has shown that the same pattern of activity can emerge from many different underlying network representations. Here we evaluated whether similarity in activation necessarily implies similarity in network architecture by comparing region-wise activation patterns and functional correlation profiles from a large sample of healthy subjects (N = 242). Participants performed two executive control tasks known to recruit nearly identical brain areas, the color-word Stroop task and the Multi-Source Interference Task (MSIT). Using a measure of instantaneous functional correlations, based on edge time series, we estimated the task-related networks that differed between incongruent and congruent conditions. We found that the two tasks were much more different in their network profiles than in their evoked activity patterns at different analytical levels, as well as for a wide range of methodological pipelines. Our results reject the notion that having the same activation patterns means two tasks engage the same underlying representations, suggesting that task representations should be independently evaluated at both node and edge (connectivity) levels.

Multimodal semi-supervised learning for online recognition of multi-granularity surgical workflows

Purpose Surgical workflow recognition is a challenging task that requires understanding multiple aspects of surgery, such as gestures, phases, and steps. However, most existing methods focus on single-task or single-modal models and rely on costly annotations for training. To address these limitations, we propose a novel semi-supervised learning approach that leverages multimodal data and self-supervision to create meaningful representations for various surgical tasks. Methods Our representation learning approach conducts two processes. In the first stage, time contrastive learning is used to learn spatiotemporal visual features from video data, without any labels. In the second stage, multimodal VAE fuses the visual features with kinematic data to obtain a shared representation, which is fed into recurrent neural networks for online recognition. Results Our method is evaluated on two datasets: JIGSAWS and MISAW. We confirmed that it achieved comparable or better performance in multi-granularity workflow recognition compared to fully supervised models specialized for each task. On the JIGSAWS Suturing dataset, we achieve a gesture recognition accuracy of 83.3%. In addition, our model is more efficient in annotation usage, as it can maintain high performance with only half of the labels. On the MISAW dataset, we achieve 84.0% AD-Accuracy in phase recognition and 56.8% AD-Accuracy in step recognition. Conclusion Our multimodal representation exhibits versatility across various surgical tasks and enhances annotation efficiency. This work has significant implications for real-time decision-making systems within the operating room.

Does Knowledge of Results Affect Motor Skill Learning and Adaptation in Interception-Like Tasks?

Knowledge of results (KR), particularly its informational role, has often been regarded as redundant for learning interception-like tasks, such as coincidence–anticipation timing tasks. However, it is possible that the KR’s guiding effect might be detrimental to motor adaptation, instead of only redundant, leading to a dependency on KR and steering the sensorimotor system away from relevant information of the task. In this study, we aimed to investigate KR’s effect on learning a coincidence–anticipation timing tasks and on the adaptation to unpredictable perturbations. Two groups of participants practiced a coincidence–anticipation timing tasks with or without KR on 1 day and underwent testing the next day for learning (Retention test) and for adaptation to unpredictable perturbations (Exposure phase). Both groups exhibited similar learning results but failed to adapt to the perturbations, contradicting the assumption of negative guidance effects of KR and the positive effects of relying solely on intrinsic information (no KR). These findings suggest that motor adaptation may require specific information during the acquisition process, highlighting for more systematic analyses to understand this phenomenon better. Such insights could have practical implications in contexts like sports and rehabilitation, by providing learners with appropriate information for acquiring adaptive internal representations of tasks.

Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations

Generalization and sample efficiency have been long-standing issues concerning reinforcement learning, and thus the field of Offline Meta-Reinforcement Learning (OMRL) has gained increasing attention due to its potential of solving a wide range of problems with static and limited offline data. Existing OMRL methods often assume sufficient training tasks and data coverage to apply contrastive learning to extract task representations. However, such assumptions are not applicable in several real-world applications and thus undermine the generalization ability of the representations. In this paper, we consider OMRL with two types of data limitations: limited training tasks and limited behavior diversity and propose a novel algorithm called GENTLE for learning generalizable task representations in the face of data limitations. GENTLE employs Task Auto-Encoder (TAE), which is an encoder-decoder architecture to extract the characteristics of the tasks. Unlike existing methods, TAE is optimized solely by reconstruction of the state transition and reward, which captures the generative structure of the task models and produces generalizable representations when training tasks are limited. To alleviate the effect of limited behavior diversity, we consistently construct pseudo-transitions to align the data distribution used to train TAE with the data distribution encountered during testing. Empirically, GENTLE significantly outperforms existing OMRL methods on both in-distribution tasks and out-of-distribution tasks across both the given-context protocol and the one-shot protocol.

MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components

Meta-Reinforcement Learning (Meta-RL) aims to reveal shared characteristics in dynamics and reward functions across diverse training tasks. This objective is achieved by meta-learning a policy that is conditioned on task representations with encoded trajectory data or context, thus allowing rapid adaptation to new tasks from a known task distribution. However, since the trajectory data generated by the policy may be biased, the task inference module tends to form spurious correlations between trajectory data and specific tasks, thereby leading to poor adaptation to new tasks. To address this issue, we propose the Meta-RL with task unCertAinty feedback through decoupled context-aware Reward and Dynamics components (MetaCARD). MetaCARD distinctly decouples the dynamics and rewards when inferring tasks and integrates task uncertainty feedback from policy evaluation into the task inference module. This design effectively reduces uncertainty in tasks with changes in dynamics or/and reward functions, thereby enabling accurate task identification and adaptation. The experiment results on both Meta-World and classical MuJoCo benchmarks show that MetaCARD significantly outperforms prevailing Meta-RL baselines, demonstrating its remarkable adaptation ability in sophisticated environments that involve changes in both reward functions and dynamics.

Pano-NeRF: Synthesizing High Dynamic Range Novel Views with Geometry from Sparse Low Dynamic Range Panoramic Images

Panoramic imaging research on geometry recovery and High Dynamic Range (HDR) reconstruction becomes a trend with the development of Extended Reality (XR). Neural Radiance Fields (NeRF) provide a promising scene representation for both tasks without requiring extensive prior data. How- ever, in the case of inputting sparse Low Dynamic Range (LDR) panoramic images, NeRF often degrades with under-constrained geometry and is unable to reconstruct HDR radiance from LDR inputs. We observe that the radiance from each pixel in panoramic images can be modeled as both a signal to convey scene lighting information and a light source to illuminate other pixels. Hence, we propose the irradiance fields from sparse LDR panoramic images, which increases the observation counts for faithful geometry recovery and leverages the irradiance-radiance attenuation for HDR reconstruction. Extensive experiments demonstrate that the irradiance fields outperform state-of-the-art methods on both geometry recovery and HDR reconstruction and validate their effectiveness. Furthermore, we show a promising byproduct of spatially-varying lighting estimation. The code is available at https://github.com/Lu-Zhan/Pano-NeRF.