Shared Representation Research Articles

Motivated with joy or anxiety: Does approach-avoidance goal framing elicit differential reward-network activation in the brain?

Psychological research on human motivation repeatedly observed that approach goals (i.e., goals to attain success) increase task enjoyment and intrinsic motivation more strongly than avoidance goals (i.e., goals to avoid failure). The present study sought to address how the reward network in the brain—including the striatum and ventromedial prefrontal cortex—is involved when individuals engage in the same task with a focus on approach or avoidance goals. Participants reported stronger positive emotions when they focused on approach goals, but stronger anxiety and disappointment when they focused on avoidance goals. The fMRI analyses revealed that the reward network in the brain showed similar levels of activity to cues predictive of approach and avoidance goals. In contrast, the two goal states were associated with different patterns of activity in the visual cortex, hippocampus, and cerebellum during success and failure outcomes. Representation similarity analysis further revealed shared and different representations within the striatum and vmPFC between the approach and avoidance goal states, suggesting both the similarity and uniqueness of the mechanisms behind the two goal states. In addition, the distinct patterns of activation in the striatum were associated with distinct subjective experiences participants reported between the approach and the avoidance conditions. These results suggest the importance of examining the pattern of striatal activity in understanding the mechanisms behind different motivational states in humans.

Emotional Intelligence Attention Unsupervised Learning Using Lexicon Analysis for Irony-based Advertising

Social media platforms have made increasing use of irony in recent years. Users can express their ironic thoughts with audio, video, and images attached to text content. When you use irony, you are making fun of a situation or trying to make a point. It can also express frustration or highlight the absurdity of a situation. The use of irony in social media is likely to continue to increase, no matter the reason. By using syntactic information in conjunction with semantic exploration, we show that attention networks can be enhanced. Using learned embedding, unsupervised learning encodes word order into a joint space. By evaluating the entropy of an example class and adding instances, the active learning method uses the shared representation as a query to retrieve semantically similar sentences from a knowledge base. In this way, the algorithm can identify the instance with the maximum uncertainty and extract the most informative example from the training set. An ironic network trained for each labelled record is used to train a classifier (model). The partial training model and the original labelled data generate pseudo-labels for the unlabeled data. To correctly predict the label of a dataset, a classifier (attention network) updates the pseudo-labels for the remaining datasets. After the experimental evaluation of the 1,021 annotated texts, the proposed model performed better than the baseline models, achieving an F1 score of 0.63 on ironic tasks and 0.59 on non-ironic tasks. We also found that the proposed model generalized well to new instances of datasets.

Multi-label feature selection with global and local label correlation

In various application domains, high-dimensional multi-label data has become more prevalent, presenting two significant challenges: instances with high-dimensional features and a large number of labels. In the context of multi-label feature selection, the objective is to choose a subset of features from a given set that is highly pertinent for predicting multiple labels or categories associated with each instance. However, certain characteristics of multi-label classification, such as label dependencies and imbalanced label distribution, have often been overlooked although they hold valuable insights for designing effective multi-label feature selection algorithms. In this paper, we propose a feature selection model which exploits explicit global and local label correlations to select discriminative features across multiple labels. In addition, by representing the feature matrix and label matrix in a shared latent space, the model aims to capture the underlying correlations between features and labels. The shared representation can reveal common patterns or relationships that exist across multiple labels and features. An objective function involving L2,1-norm regularization is formulated, and an alternating optimization-based iterative algorithm is designed to obtain the sparse coefficients for multi-label feature selection. The proposed method was evaluated on 14 real-world multi-label datasets using six evaluation metrics, through comprehensive experiments. The results indicate its effectiveness, surpassing that of several representative methods.

Memory in the digital age

This article explores the configuration of collective memory under the impact of the digital turn. In recent debates, there has been a marked tendency to interpret ‘digital memory’ as a new type of memory, which is radically different from the traditional conceptualization. Even leading authors in the field claim that the digital revolution implies the end of collective memory. However, I argue that despite the transformations that memory undergoes in the digital age, these changes do not imply a new ontology of memory but rather a materialization of the theoretical claims made by Memory Studies since the field's inception. To support this hypothesis, I analyze digital memory in relation to three topics: first, I focus on the problematic definition of collective memory to demonstrate how the digital realm allows us to rethink the social nature of memory through a different concept of the social. By contrasting Halbwachs' notion of the social, which forms the basis of memory studies, with the alternative proposal of Gabriel Tarde, I argue that the latter enables us to refine the concept of the ‘collective’ that we have inherited from the founding figure of memory studies. Second, I delve into the new ontology of the digital archive showing how it materializes one of the defining features of collective memory: its mobile, dynamic, and procedural nature. Lastly, I address the inversion of the dialectic between memory and forgetting to highlight the specificity of these practices in the digital environment. I demonstrate how these changes effectively implement, surpassing older technologies, the concept of collective memory as a distributed and dynamic technological process that shapes our shared representations of the past.

Multitask and Transfer Learning Approach for Joint Classification and Severity Estimation of Dysphonia.

Despite speech being the primary communication medium, it carries valuable information about a speaker's health, emotions, and identity. Various conditions can affect the vocal organs, leading to speech difficulties. Extensive research has been conducted by voice clinicians and academia in speech analysis. Previous approaches primarily focused on one particular task, such as differentiating between normal and dysphonic speech, classifying different voice disorders, or estimating the severity of voice disorders. This study proposes an approach that combines transfer learning and multitask learning (MTL) to simultaneously perform dysphonia classification and severity estimation. Both tasks use a shared representation; network is learned from these shared features. We employed five computer vision models and changed their architecture to support multitask learning. Additionally, we conducted binary 'healthy vs. dysphonia' and multiclass 'healthy vs. organic and functional dysphonia' classification using multitask learning, with the speaker's sex as an auxiliary task. The proposed method achieved improved performance across all classification metrics compared to single-task learning (STL), which only performs classification or severity estimation. Specifically, the model achieved F1 scores of 93% and 90% in MTL and STL, respectively. Moreover, we observed considerable improvements in both classification tasks by evaluating beta values associated with the weight assigned to the sex-predicting auxiliary task. MTL achieved an accuracy of 77% compared to the STL score of 73.2%. However, the performance of severity estimation in MTL was comparable to STL. Our goal is to improve how voice pathologists and clinicians understand patients' conditions, make it easier to track their progress, and enhance the monitoring of vocal quality and treatment procedures. Clinical and Translational Impact Statement: By integrating both classification and severity estimation of dysphonia using multitask learning, we aim to enable clinicians to gain a better understanding of the patient's situation, effectively monitor their progress and voice quality.

L-VSM: Label-Driven View-Specific Fusion for Multiview Multilabel Classification.

In the task of multiview multilabel (MVML) classification, each instance is represented by several heterogeneous features and associated with multiple semantic labels. Existing MVML methods mainly focus on leveraging the shared subspace to comprehensively explore multiview consensus information across different views, while it is still an open problem whether such shared subspace representation is effective to characterize all relevant labels when formulating a desired MVML model. In this article, we propose a novel label-driven view-specific fusion MVML method named L-VSM, which bypasses seeking for a shared subspace representation and instead directly encodes the feature representation of each individual view to contribute to the final multilabel classifier induction. Specifically, we first design a label-driven feature graph construction strategy and construct all instances under various feature representations into the corresponding feature graphs. Then, these view-specific feature graphs are integrated into a unified graph by linking the different feature representations within each instance. Afterward, we adopt a graph attention mechanism to aggregate and update all feature nodes on the unified graph to generate structural representations for each instance, where both intraview correlations and interview alignments are jointly encoded to discover the underlying consensuses and complementarities across different views. Moreover, to explore the widespread label correlations in multilabel learning (MLL), the transformer architecture is introduced to construct a dynamic semantic-aware label graph and accordingly generate structural semantic representations for each specific class. Finally, we derive an instance-label affinity score for each instance by averaging the affinity scores of its different feature representations with the multilabel soft margin loss. Extensive experiments on various MVML applications have verified that our proposed L-VSM has achieved superior performance against state-of-the-art methods. The codes are available at https://gengyulyu.github.io/homepage/assets/codes/LVSM.zip.

Multi-Level Linguistic Alignment in a Dynamic Collaborative Problem-Solving Task.

Cocreating meaning in collaboration is challenging. Success is often determined by people's abilities to coordinate their language to converge upon shared mental representations. Here we explore one set of low-level linguistic behaviors, linguistic alignment, that both emerges from, and facilitates, outcomes of high-level convergence. Linguistic alignment captures the ways people reuse, that is, "align to," the lexical, syntactic, and semantic forms of others' utterances. Our focus is on the temporal change of multi-level linguistic alignment, as well as how alignment is related to communicative outcomes within a unique collaborative problem-solving paradigm. The primary task, situated within a virtual educational video game, requires creative thinking between three people where the paths for possible solutions are highly variable. We find that over time interactions are marked by decreasing lexical and syntactic alignment, with a trade-off of increasing semantic alignment. However, greater semantic alignment does not translate into better team performance. Overall, these findings provide greater clarity on the role of linguistic coordination within complex and dynamic collaborative problem-solvingtasks.

Cross-linguistic syntactic priming as rational expectation for syntactic repetition in the bilingual environment.

Recent research suggests that syntactic priming in language comprehension-the facilitated processing of repeated syntactic structures-arises from the expectation for syntactic repetition due to rational adaptation to the linguistic environment. To further evaluate the generalizability of this expectation adaptation account in cross-linguistic syntactic priming and explore the influence of second language (L2) proficiency, we conducted a self-paced reading study with Chinese L2 learners of English by utilizing the sentential complement-direct object (SC-DO) ambiguity. The results showed that participants exposed to clusters of SC structures subsequently processed repetitions of this structure more rapidly (i.e., larger priming effects) than those exposed to the same number of SC structures but spaced in time, despite the prime and target being in two different languages (Chinese and English). Furthermore, this difference in priming strength was more pronounced for participants with higher L2 (English) proficiency. These findings demonstrate that cross-linguistic syntactic priming is consistent with the expectation for syntactic repetition that rationally adapts to syntactic clustering properties in surrounding bilingual environments, and such adaptation is enhanced as L2 proficiency increases. Taken together, our study extends the expectation adaptation account to cross-linguistic syntactic priming and integrates the role of L2 proficiency, which can shed new light on the mechanisms underlying syntactic priming, bilingual shared syntactic representations and expectation-based sentence processing.

Shared Latent Membership Enables Joint Shape Abstraction and Segmentation with Deformable Superquadrics.

Part-level 3D shape representations are crucial to shape reasoning and understanding. Two key sub-tasks are (i) shape abstraction, creating primitive-based object parts, and (ii) shape segmentation, finding partition-based object parts. However, for 3D object point clouds, most advanced methods produce parts relying on task-specific priors, such as similarity metrics and primitive geometries, resulting in misleading parts that deviate from semantics. To address prior limitations, we establish a foundation for joint shape abstraction and shape segmentation as formal linear transformations within a shared latent space, encapsulating essential dual-purpose membership information linking points and object parts for mutual reinforcement. We demonstrate that the transformations are underpinned by a derivation based on k-means, Non-negative Matrix Factorization (NMF), and the attention mechanism. As a result, we introduce Latent Membership Pursuit (LMP) for joint optimization of shape abstraction and segmentation. LMP utilizes a shared latent representation of object part membership to autonomously identify common object parts in both tasks without any supervision and priors. Furthermore, we adapt deformable superquadrics (DSQs) for primitives to capture variable part-level geometric and semantic information. Experiments on benchmark datasets validate that our approach enables mutual learning of shape abstraction and segmentation, and promotes consistent interpretations of 3D object shapes across instances and even categories in a fully unsupervised manner.

The importance of observing the master's hand: Action Observation Training promotes the acquisition of new musical skills.

Via mirror mechanism, motor training approaches based on the alternation of action observation and execution (i.e., Action Observation Training-AOT) promote the acquisition of motor abilities. Previous studies showed that both visual and auditory stimuli may elicit a common motor representation of music-related gestures; however, the potentialities of AOT for the acquisition of musical skills are still underexplored. Twenty-one music-naïve participants underwent two blocks of training: AOT and Key-light Observation Training (KOT). AOT consisted of the observation of a melodic sequence played on a keyboard with the right hand by an expert model, followed by participant's imitation. Observation and execution were repeated six consecutive times (T1-T6). KOT followed the same procedure, except for the visual content of the stimulus, depicting the sequential highlighting of the piano keys corresponding to the melody. The rate of correct notes (C), the trainee-model similarity of key-pressure strength (S), and the trainee-model consistency of note duration (R) were collected across T1-T6. Both AOT and KOT improved musical performance. Noteworthy, AOT showed a higher learning magnitude relative to KOT in terms of C and S. Action Observation Training promotes the acquisition of key elements of melodic sequences, encompassing not only the accurate sequencing of notes but also their expressive characteristics, such as key-pressure dynamics. The convergence of listening and observation of actions onto a shared motor representation not only explains several pedagogical approaches applied in all musical cultures worldwide, but also enhances the potential efficacy of current procedures for music training.

Neural correlates of motor imagery and execution in real-world dynamic behavior: evidence for similarities and differences.

A large body of evidence shows that motor imagery and action execution behaviors result from overlapping neural substrates, even in the absence of overt movement during motor imagery. To date it is unclear how neural activations in motor imagery and execution compare for naturalistic whole-body movements, such as walking. Neuroimaging studies have not directly compared imagery and execution during dynamic walking movements. Here we recorded brain activation with mobile EEG during walking compared to during imagery of walking, with mental counting as a control condition. We asked 24 healthy participants to either walk six steps on a path, imagine taking six steps, or mentally count from one to six. We found beta and alpha power modulation during motor imagery resembling action execution patterns; a correspondence not found performing the control task of mental counting. Neural overlap occurred early in the execution and imagery walking actions, suggesting activation of shared action representations. Remarkably, a distinctive walking-related beta rebound occurred both during action execution and imagery at the end of the action suggesting that, like actual walking, motor imagery involves resetting or inhibition of motor processes. However, we also found that motor imagery elicits a distinct pattern of more distributed beta activity, especially at the beginning of the task. These results indicate that motor imagery and execution of naturalistic walking involve shared motor-cognitive activations, but that motor imagery requires additional cortical resources.

Multi-Task Learning for Motion Analysis and Segmentation in 3D Echocardiography.

Characterizing left ventricular deformation and strain using 3D+time echocardiography provides useful insights into cardiac function and can be used to detect and localize myocardial injury. To achieve this, it is imperative to obtain accurate motion estimates of the left ventricle. In many strain analysis pipelines, this step is often accompanied by a separate segmentation step; however, recent works have shown both tasks to be highly related and can be complementary when optimized jointly. In this work, we present a multi-task learning network that can simultaneously segment the left ventricle and track its motion between multiple time frames. Two task-specific networks are trained using a composite loss function. Cross-stitch units combine the activations of these networks by learning shared representations between the tasks at different levels. We also propose a novel shape-consistency unit that encourages motion propagated segmentations to match directly predicted segmentations. Using a combined synthetic and in-vivo 3D echocardiography dataset, we demonstrate that our proposed model can achieve excellent estimates of left ventricular motion displacement and myocardial segmentation. Additionally, we observe strong correlation of our image-based strain measurements with crystal-based strain measurements as well as good correspondence with SPECT perfusion mappings. Finally, we demonstrate the clinical utility of the segmentation masks in estimating ejection fraction and sphericity indices that correspond well with benchmark measurements.

Semi-Supervised Multimodal Representation Learning Through a Global Workspace.

Recent deep learning models can efficiently combine inputs from different modalities (e.g., images and text) and learn to align their latent representations or to translate signals from one domain to another (as in image captioning or text-to-image generation). However, current approaches mainly rely on brute-force supervised training over large multimodal datasets. In contrast, humans (and other animals) can learn useful multimodal representations from only sparse experience with matched cross-modal data. Here, we evaluate the capabilities of a neural network architecture inspired by the cognitive notion of a "global workspace" (GW): a shared representation for two (or more) input modalities. Each modality is processed by a specialized system (pretrained on unimodal data and subsequently frozen). The corresponding latent representations are then encoded to and decoded from a single shared workspace. Importantly, this architecture is amenable to self-supervised training via cycle-consistency: encoding-decoding sequences should approximate the identity function. For various pairings of vision-language modalities and across two datasets of varying complexity, we show that such an architecture can be trained to align and translate between two modalities with very little need for matched data (from four to seven times less than a fully supervised approach). The GW representation can be used advantageously for downstream classification and cross-modal retrieval tasks and for robust transfer learning. Ablation studies reveal that both the shared workspace and the self-supervised cycle-consistency training are critical to the system's performance.

Multi-Task Multi-Agent Reinforcement Learning With Interaction and Task Representations.

Multi-task multi-agent reinforcement learning (MT-MARL) is capable of leveraging useful knowledge across multiple related tasks to improve performance on any single task. While recent studies have tentatively achieved this by learning independent policies on a shared representation space, we pinpoint that further advancements can be realized by explicitly characterizing agent interactions within these multi-agent tasks and identifying task relations for selective reuse. To this end, this article proposes Representing Interactions and Tasks (RIT), a novel MT-MARL algorithm that characterizes both intra-task agent interactions and inter-task task relations. Specifically, for characterizing agent interactions, RIT presents the interactive value decomposition to explicitly take the dependency among agents into policy learning. Theoretical analysis demonstrates that the learned utility value of each agent approximates its Shapley value, thus representing agent interactions. Moreover, we learn task representations based on per-agent local trajectories, which assess task similarities and accordingly identify task relations. As a result, RIT facilitates the effective transfer of interaction knowledge across similar multi-agent tasks. Structurally, RIT develops universal policy structure for scalable multi-task policy learning. We evaluate RIT against multiple state-of-the-art baselines in various cooperative tasks, and its significant performance under both multi-task and zero-shot settings demonstrates its effectiveness.

MTL-Leak: Privacy Risk Assessment in Multi-Task Learning

Multi-task learning (MTL) supports simultaneous training over multiple related tasks and learns the shared representation. While improving the generalization ability of training on a single task, MTL has higher privacy risk than traditional single-task learning because more sensitive information is extracted and learned in a correlated manner. Unfortunately, very few works have attempted to address the privacy risks posed by MTL. In this paper, we first investigate such risk by designing model extraction attack (MEA) and membership inference attack (MIA) in MTL. Then we evaluate the privacy risks on six MTL model architectures and two popular MTL datasets, whose results show that both the number of tasks and the complexity of training data play an important role in the attack performance. Our investigation shows that MTL is more vulnerable than traditional single-task learning under both attacks.

Network Security Threat Intelligence Prediction in Network Traffic Analysis Based on Machine Learning

Abstract With the increasingly complex network architecture under the development of information technology, it is more and more difficult to recognize various network traffic. Based on the direction of machine learning for network traffic anomaly detection, this paper proposes a three-stage CNN-LSTM attention mechanism model for feature extraction of original traffic and a PCA-based improvement for spatial feature learning of CNN to construct its anomaly traffic detection model. Meanwhile, based on the abnormal traffic detection, with discriminative network and generative network as the main parts, we propose the cyber security threat intelligence prediction model based on the use of a domain adaptive model to realize the shared representation of source and target domains. In the experiment of the abnormal traffic detection model, the accuracy rate of this paper’s detection model is 93.56%, the check-all rate is 99.2%, the F1 value is 84.9%, and the Kappa value is 80.66%. The detection level in the face of DOS attacks is the highest, with 98.28% and 99.63% checking full rate and checking accuracy, respectively. In the experiments of the network security threat intelligence prediction model, the prediction model of this paper reduces the time consumed by about 72% on average compared to rule file matching in dealing with large-scale network traffic. In high-risk cybersecurity threat attack types, it has an average prediction accuracy of 88.83% and a recall rate of over 90%.

Rethinking Resource Competition in Multi-Task Learning: From Shared Parameters to Shared Representation

Rethinking Resource Competition in Multi-Task Learning: From Shared Parameters to Shared Representation

Deep Learning to Hash with Application to Cross-View Nearest Neighbor Search

Learning hash functions for approximate nearest neighbor search of high-dimensional data has received a surge of interests in recent years. Most existing methods are often concerned with learning hash functions for nearest neighbor search on high-dimensional data from a single source. In many real-world applications, data can be collected from diverse sources or represented using different feature descriptors. This raises an open challenge, i.e., the Cross-View Nearest Neighbor Search (CVNNS), where the representation of a query instance can be different from that of target instances to be retrieved in database. The key challenge of cross-view search is to learn an effective shared representation which can effectively connect the query instance and the target instances to be retrieved. In this paper, we present a new cross-view nearest neighbor search scheme by applying the emerging deep learning to hash techniques. In particular, we investigate two different architectures of deep Restricted Boltzmann Machines (RBMs) for learning to hash toward cross-view nearest neighbor search, and conduct extensive experiments to examine their empirical performance on diverse settings of cross-view image retrieval tasks. The encouraging results show that our technique outperforms the state-of-the-art approaches.

MMMViT: Multiscale multimodal vision transformer for brain tumor segmentation with missing modalities

Accurate segmentation of brain tumors from multimodal MRI sequences is a critical prerequisite for brain tumor diagnosis, prognosis, and surgical treatment. While one or more modalities are often missing in clinical practice, which can collapse most previous methods that rely on all modality data. To deal with this problem, the current state-of-the-art Transformer-related approach directly fuses available modality-specific features to learn a shared latent representation, with the aim of extracting common features that are robust to any combinatorial subset of all modalities. However, it is not trivial to directly learn a shared latent representation due to the diversity of combinatorial subsets of all modalities. Furthermore, correlations across modalities as well as global multiscale features are not exploited in this Transformer-related approach. In this work, we propose a Multiscale Multimodal Vision Transformer (MMMViT), which not only leverages correlations across modalities to decouple the direct fusing procedure into two simple steps but also innovatively fuses local multiscale features as the input of the intra-modal Transformer block to implicitly obtain the global multiscale features to adapt to brain tumors of various sizes. We experiment on the BraTs 2018 dataset for all modalities and various missing-modalities as input, and the results demonstrate that the proposed method achieves the state-of-the-art performance. Code is available at: https://github.com/qiuchengjian/MMMViT.

Visible-infrared person re-identification via specific and shared representations learning

The primary goal of visible-infrared person re-identification (VI-ReID) is to match pedestrian photos obtained during the day and night. The majority of existing methods simply generate auxiliary modalities to reduce the modality discrepancy for cross-modality matching. They capture modality-invariant representations but ignore the extraction of modality-specific representations that can aid in distinguishing among various identities of the same modality. To alleviate these issues, this work provides a novel specific and shared representations learning (SSRL) model for VI-ReID to learn modality-specific and modality-shared representations. We design a shared branch in SSRL to bridge the image-level gap and learn modality-shared representations, while a specific branch retains the discriminative information of visible images to learn modality-specific representations. In addition, we propose intra-class aggregation and inter-class separation learning strategies to optimize the distribution of feature embeddings at a fine-grained level. Extensive experimental results on two challenging benchmark datasets, SYSU-MM01 and RegDB, demonstrate the superior performance of SSRL over state-of-the-art methods.