Shared Representation Research Articles

DAKRS: Domain Adaptive Knowledge-Based Retrieval System for Natural Language-Based Vehicle Retrieval

Given Natural Language (NL) text descriptions, NL-based vehicle retrieval aims to extract target vehicles from a multi-view multi-camera traffic video pool. Due to inherent distinctions between textual and visual data, this is a challenging multi-modal retrieval task that requires robust feature extractors (e.g. neural network) to well-align the abstract representations of texts and images in the same domain. However, solutions to the problem have been challenged by the high data complexities of not only the multi-view, multi-camera attributes of visual data and the diverse range of textual descriptions but also a lack of high-volume datasets in this relatively new field, alongside a prominently large domain gap between training and test sets. Many existing approaches have developed computationally expensive models to separately extract the subspaces of language and vision before blending into the same shared representation space while only focusing on single-modal information and ignoring much of the multi-modal information to deal with the aforementioned issues. Hence, we propose a Domain Adaptive Knowledge-based Retrieval System (DAKRS) to effectively and efficiently align multi-modal knowledge in a setting of limited labels. Our contributions are threefold: (i) An efficient extension of Contrastive Language-Image Pre-training (CLIP)’s transfer learning into a baseline text-to-image multi-modular vehicle retrieval framework; (ii) A data enhancement module to create pseudo-vehicle tracks from the traffic video pool by leveraging the robustness of baseline retrieval model combine with background subtraction; and (iii) A SSDA (SSDA) scheme to engineer pseudo-labels for adapting model parameters to the target domain distribution. Experimental results are benchmarked on the Cityflow-NL dataset, illustrating our competitiveness against state-of-the-art performances in terms of effectiveness and efficiency without needing further post-processing or ensembling.

Multiscale Shared Learning for Fault Diagnosis of Rotating Machinery in Transportation Infrastructures

Rotating machinery is ubiquitous, and its failures constitute a major cause of the failures of transportation infrastructures. Most fault-diagnosis methods for rotating machinery are based on vibration-signal analysis because vibrations directly reflect the transient regime of machinery elements. This article proposes a novel multiscale shared-learning network (MSSLN) architecture to extract and classify the fault features inherent to multiscale factors of vibration signals. The architecture fuses layer-wise activations with multiscale flows, to enable the network to fully learn the shared representation with consistency across multiscale factors. This characteristic helps MSSLN provide more faithful diagnoses than existing single- and multiscale methods. Experiments on bearing and gearbox datasets are used to evaluate the fault-diagnosis performance of transportation infrastructures. Extensive experimental results and comprehensive analyses demonstrate the superiority of the proposed MSSLN in fault diagnosis for bearings and gearboxes, the two foundational elements in transportation infrastructures.

NIM-Nets: Noise-aware Incomplete Multi-view Learning Networks.

Data in real world are usually characterized in multiple views, including different types of features or different modalities. Multi-view learning has been popular in the past decades and achieved significant improvements. In this paper, we investigate three challenging problems in the field of incomplete multi-view representation learning, namely, i) how to reduce the influences produced by missing views in multi-view dataset, ii) how to learn a consistent and informative representation among different views and iii) how to alleviate the impacts of the inherent noise in multi-view data caused by high-dimensional features or varied quality for different data points. To address these challenges, we integrate these three tasks into a problem and propose a novel framework termed Noise-aware Incomplete Multi-view Learning Networks (NIM-Nets). NIM-Nets fully utilize incomplete data from different views to produce a multi-view shared representation which is consistent, informative and robust to noise. We model the inherent noise in data by defining the distribution Γ and assuming that each observation in the incomplete dataset is sampled from the distribution Γ. To the best of our knowledge, this is the first work to unify learning the consistent and informative representation, alleviating the impacts of noise in data and handling the view-missing patterns in multi-view learning into a framework. We also first give a definition of robustness and completeness for incomplete multi-view representation learning. Based on NIM-Nets, we present joint optimization models for classification and clustering, respectively. Extensive experiments on different datasets demonstrate the effectiveness of our method over the existing work based on classification and clustering tasks in terms of different metrics.

Shared or different: How linked are word production and comprehension?

Shared or different: How linked are word production and comprehension?

EFND: A Semantic, Visual, and Socially Augmented Deep Framework for Extreme Fake News Detection

Due to the exponential increase in internet and social media users, fake news travels rapidly, and no one is immune to its adverse effects. Various machine learning approaches have evaluated text and images to categorize false news over time, but they lack a comprehensive representation of relevant features. This paper presents an automated method for detecting fake news to counteract the spread of disinformation. The proposed multimodal EFND integrates contextual, social context, and visual data from news articles and social media to build a multimodal feature vector with a high level of information density. Using a multimodal factorized bilinear pooling, the gathered features are fused to improve their correlation and offer a more accurate shared representation. Finally, a Multilayer Perceptron is implemented over the shared representation for the classification of fake news. EFND is evaluated using a group of standard fake news datasets known as “FakeNewsNet”. EFND has outperformed the baseline and state-of-the-art machine learning and deep learning models. Furthermore, the results of ablation studies have demonstrated the efficacy of the proposed framework. For the PolitiFact and GossipCop datasets, the EFND has achieved an accuracy of 0.988% and 0.990%, respectively.

Federated data analytics: A study on linear models

As edge devices become increasingly powerful, data analytics are gradually moving from a centralized to a decentralized regime where edge computing resources are exploited to process more of the data locally. This regime of analytics is coined as Federated Data Analytics (FDA). Despite the recent success stories of FDA, most literature focuses exclusively on deep neural networks. In this work, we take a step back to develop an FDA treatment for one of the most fundamental statistical models: linear regression. Our treatment is built upon hierarchical modeling that allows borrowing strength across multiple groups. To this end, we propose two federated hierarchical model structures that provide a shared representation across devices to facilitate information sharing. Notably, our proposed frameworks are capable of providing uncertainty quantification, variable selection, hypothesis testing, and fast adaptation to new unseen data. We validate our methods on a range of real-life applications, including condition monitoring for aircraft engines. The results show that our FDA treatment for linear models can serve as a competing benchmark model for the future development of federated algorithms.

Pour une représentation partagée entre recherche et terrain autour du concept d’hybridation des enseignements

Le but de ce retour d’expérience est de proposer la coconstruction d’une représentation partagée du concept d’hybridation des enseignements à partir d’ateliers collaboratifs (workshop) associant chercheurs et enseignants du second degré. Il s’agit d’une étape préalable à la conduite d’une recherche participative réalisée dans le cadre d’un groupe thématique numérique qui questionne les pratiques enseignantes de l’hybridation. Au cours de ces ateliers, les représentations des enseignants à propos du concept d’hybridation ont pu être recueillies. Leur analyse a permis de mettre en évidence quatre dimensions comme autant d’axes de réflexion constitutifs du concept d’hybridation. Ce travail a donné lieu à l’élaboration d’un « modèle issu du terrain », que nous confrontons à un modèle issu de la recherche.

Novel transfer learning schemes based on Siamese networks and synthetic data

Transfer learning schemes based on deep networks which have been trained on huge image corpora offer state-of-the-art technologies in computer vision. Here, supervised and semi-supervised approaches constitute efficient technologies which work well with comparably small data sets. Yet, such applications are currently restricted to application domains where suitable deep network models are readily available. In this contribution, we address an important application area in the domain of biotechnology, the automatic analysis of CHO-K1 suspension growth in microfluidic single-cell cultivation, where data characteristics are very dissimilar to existing domains and trained deep networks cannot easily be adapted by classical transfer learning. We propose a novel transfer learning scheme which expands a recently introduced Twin-VAE architecture, which is trained on realistic and synthetic data, and we modify its specialized training procedure to the transfer learning domain. In the specific domain, often only few to no labels exist and annotations are costly. We investigate a novel transfer learning strategy, which incorporates a simultaneous retraining on natural and synthetic data using an invariant shared representation as well as suitable target variables, while it learns to handle unseen data from a different microscopy technology. We show the superiority of the variation of our Twin-VAE architecture over the state-of-the-art transfer learning methodology in image processing as well as classical image processing technologies, which persists, even with strongly shortened training times and leads to satisfactory results in this domain. The source code is available at https://github.com/dstallmann/transfer_learning_twinvae, works cross-platform, is open-source and free (MIT licensed) software. We make the data sets available at https://pub.uni-bielefeld.de/record/2960030.

Shared and distinct representations of visual regularities across levels of abstraction

Shared and distinct representations of visual regularities across levels of abstraction

Semi-supervised cross-modal hashing via modality-specific and cross-modal graph convolutional networks

Cross-modal hashing maps heterogeneous multimedia data into Hamming space for retrieving relevant samples across modalities, which has received great research interests due to its rapid retrieval and low storage cost. In real-world applications, due to high manual annotation cost of multi-media data, we can only make use of limited number of labeled data with rich unlabeled data. In recent years, several semi-supervised cross-modal hashing (SCH) methods have been presented. However, how to fully explore and jointly utilize the modality-specific (complementarity) and modality-shared (correlation) information for retrieval has not been well studied for existing SCH works. In this paper, we propose a novel SCH approach named Modality-specific and Cross-modal Graph Convolutional Networks (MCGCN). The network architecture contains two modality-specific channels and a cross-modal channel to learn modality-specific and shared representations for each modality, respectively. Graph convolutional network (GCN) is leveraged in these three channels to explore intra-modal and inter-modal similarity, and perform semantic information propagation from labeled data to unlabeled data. Modality-specific and shared representations for each modality are fused with attention scheme. To further reduce the modality gap, a discriminative model is designed, learning to classify the modality of representations, and network training is guided by adversarial scheme. Experiments on two widely used multi-modal datasets demonstrate MCGCN outperforms state-of-the-art semi-supervised/supervised cross-modal hashing methods.

Multi-Attribute Discriminative Representation Learning for Prediction of Adverse Drug-Drug Interaction.

Adverse drug-drug interaction (ADDI) is a significant life-threatening issue, posing a leading cause of hospitalizations and deaths in healthcare systems. This paper proposes a unified Multi-Attribute Discriminative Representation Learning (MADRL) model for ADDI prediction. Unlike the existing works that equally treat features of each attribute without discrimination and do not consider the underlying relationship among drugs, we first develop a regularized optimization problem based on CUR matrix decomposition for joint representative drug and discriminative feature selection such that the selected drugs and features can well approximate the original feature spaces and the critical factors discriminative to ADDIs can be properly explored. Different from the existing models that ignore the consistent and unique properties among attributes, a Generative Adversarial Network (GAN) framework is then designed to capture the inter-attribute shared and intra-attribute specific representations of adverse drug pairs for exploiting their consensus and complementary information in ADDI prediction. Meanwhile, MADRL is compatible with any kind of attributes and capable of exploring their respective effects on ADDI prediction. An iterative algorithm based on the alternating direction method of multipliers is developed for optimization. Experiments on publicly available dataset demonstrate the effectiveness of MADRL when compared with eleven baselines and its six variants.

Multi-dimensional shared representation learning with graph fusion network for Session-based Recommendation

The Session-based Recommendation (SBR) system aims to forecast anonymous users’ short-term decisions. Many prior research have demonstrated that using Graph Neural Networks (GNN) and Recurrent Neural Networks (RNN) to solve SBR tasks can lead to excellent results. However, the existing SBR models only use single feature extraction method to represent an item in the recommendation process. These models either use the GNN methods which ignore the sequence location features in the training process or use RNN methods which cannot obtain the weight features of every item in the sequence. All of them fail to extract features of different dimensions at the same time, which results in low-quality item representations. To tackle the aforementioned issue, this research introduces a novel method called Multi-dimensional Shared Representation Learning (MSR). (i) To get multi-dimensional features, we use the multi-dimensional feature extraction in a double-flow way based on transformer layers and graph attention network (GAT) layers. (ii) Through the MSR module, we train the joint representation of items in the session by using multiple GNNs, then merge the multi-dimensional features by using an attention mechanism. A variety of experiments have been carried out and the simulation results demonstrate the designed strategy largely surpasses the state-of-the-art proposal in recommendation accuracy.

Exploiting Japanese–Chinese Cognates with Shared Private Representations for NMT

Neural machine translation has achieved remarkable progress over the past several years; however, little attention has been paid to machine translation (MT) between Japanese and Chinese, which share a large proportion of cognate words that can be utilized as additional linguistic knowledge to enhance translation performance. In this article, we seek to strengthen the semantic correlation between Japanese and Chinese by leveraging cognate words that share common Chinese characters. Specifically, we experiment with three strategies: (1) a shared vocabulary with cognate lexicon induction, which models the commonality between source and target cognates; (2) a shared private representation with a dynamic gating mechanism, which models the language-specific features on the source side; and (3) an embedding shortcut, which enables the decoder to access the shared private representation with shortest distance and aids the training process. The experiments and analysis presented in this article demonstrate that our proposed approaches can significantly improve the performance of both Japanese-to-Chinese and Chinese-to-Japanese translations and verify the effectiveness of exploiting Japanese–Chinese cognates for MT.

Traffic flow and speed forecasting through a Bayesian deep multi-linear relationship network

Both traffic flow and speed forecasting are of great importance to intelligent transportation systems, which have been studied intensely in the past decades. In recent years, forecasting approaches based on deep learning have attracted the most extensive attention. However, most studies only predict traffic flow and speed separately, without taking advantage of the correlation between the two tasks. Unlike single-task learning, multi-task learning (MTL) approaches can learn related tasks simultaneously. MTL enhances the learning of each task by exploiting useful information from other related tasks. Nevertheless, traditional MTL methods only share transferable features at the feature layers and simply make an assumption that given the shared representation, the multiple tasks can be well correlated with each other, which leads to under-transfer and negative-transfer. In this paper, we propose a new MTL approach to forecast both traffic flow and speed, which shares transferable features at feature layers but learns the correlation between tasks at task-specific layers. By learning the multi-linear relationships between tasks and features as well as the transferable features, the proposed approach can mitigate the above-mentioned issues. Comprehensive experiments from a case study based on real data show that the proposed method steadily achieves the best forecasting results and good robustness.

SwitchNet: A modular neural network for adaptive relation extraction

This paper presents a portable toolkit, SwitchNet, for extracting relations from textual input. We summarize four data protocols for relation extraction tasks, including relation classification, relation extraction, triple extraction, and distant supervision relation extraction. This neural architecture is modular, so it can take as input data at different stages of the information extraction process (simple text, text and entities or entity pairs as relation candidates) and compute the rest of the process (named entity recognition and relation classification). We systematically design four information flows to integrate the above protocols by sharing network building blocks and switching different information flows. This framework can extract multiple triples (subject, predicate, object) in one pass. This framework enhances the use of relation classification models in end-to-end triple extraction by inferring pairs of entities of interest and using the shared representation mechanism.

A comprehensive update on CIDO: the community-based coronavirus infectious disease ontology

BackgroundThe current COVID-19 pandemic and the previous SARS/MERS outbreaks of 2003 and 2012 have resulted in a series of major global public health crises. We argue that in the interest of developing effective and safe vaccines and drugs and to better understand coronaviruses and associated disease mechenisms it is necessary to integrate the large and exponentially growing body of heterogeneous coronavirus data. Ontologies play an important role in standard-based knowledge and data representation, integration, sharing, and analysis. Accordingly, we initiated the development of the community-based Coronavirus Infectious Disease Ontology (CIDO) in early 2020.ResultsAs an Open Biomedical Ontology (OBO) library ontology, CIDO is open source and interoperable with other existing OBO ontologies. CIDO is aligned with the Basic Formal Ontology and Viral Infectious Disease Ontology. CIDO has imported terms from over 30 OBO ontologies. For example, CIDO imports all SARS-CoV-2 protein terms from the Protein Ontology, COVID-19-related phenotype terms from the Human Phenotype Ontology, and over 100 COVID-19 terms for vaccines (both authorized and in clinical trial) from the Vaccine Ontology. CIDO systematically represents variants of SARS-CoV-2 viruses and over 300 amino acid substitutions therein, along with over 300 diagnostic kits and methods. CIDO also describes hundreds of host-coronavirus protein-protein interactions (PPIs) and the drugs that target proteins in these PPIs. CIDO has been used to model COVID-19 related phenomena in areas such as epidemiology. The scope of CIDO was evaluated by visual analysis supported by a summarization network method. CIDO has been used in various applications such as term standardization, inference, natural language processing (NLP) and clinical data integration. We have applied the amino acid variant knowledge present in CIDO to analyze differences between SARS-CoV-2 Delta and Omicron variants. CIDO's integrative host-coronavirus PPIs and drug-target knowledge has also been used to support drug repurposing for COVID-19 treatment.ConclusionCIDO represents entities and relations in the domain of coronavirus diseases with a special focus on COVID-19. It supports shared knowledge representation, data and metadata standardization and integration, and has been used in a range of applications.

No semantic information is necessary to evoke general neural signatures of face familiarity: evidence from cross-experiment classification

Recent theories on the neural correlates of face identification stressed the importance of the available identity-specific semantic and affective information. However, whether such information is essential for the emergence of neural signal of familiarity has not yet been studied in detail. Here, we explored the shared representation of face familiarity between perceptually and personally familiarized identities. We applied a cross-experiment multivariate pattern classification analysis (MVPA), to test if EEG patterns for passive viewing of personally familiar and unfamiliar faces are useful in decoding familiarity in a matching task where familiarity was attained thorough a short perceptual task. Importantly, no additional semantic, contextual, or affective information was provided for the familiarized identities during perceptual familiarization. Although the two datasets originate from different sets of participants who were engaged in two different tasks, familiarity was still decodable in the sorted, same-identity matching trials. This finding indicates that the visual processing of the faces of personally familiar and purely perceptually familiarized identities involve similar mechanisms, leading to cross-classifiable neural patterns.

Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors.

Clinical diagnosis of the pediatric musculoskeletal system relies on the analysis of medical imaging examinations. In the medical image processing pipeline, semantic segmentation using deep learning algorithms enables an automatic generation of patient-specific three-dimensional anatomical models which are crucial for morphological evaluation. However, the scarcity of pediatric imaging resources may result in reduced accuracy and generalization performance of individual deep segmentation models. In this study, we propose to design a novel multi-task, multi-domain learning framework in which a single segmentation network is optimized over the union of multiple datasets arising from distinct parts of the anatomy. Unlike previous approaches, we simultaneously consider multiple intensity domains and segmentation tasks to overcome the inherent scarcity of pediatric data while leveraging shared features between imaging datasets. To further improve generalization capabilities, we employ a transfer learning scheme from natural image classification, along with a multi-scale contrastive regularization aimed at promoting domain-specific clusters in the shared representations, and multi-joint anatomical priors to enforce anatomically consistent predictions. We evaluate our contributions for performing bone segmentation using three scarce and pediatric imaging datasets of the ankle, knee, and shoulder joints. Our results demonstrate that the proposed approach outperforms individual, transfer, and shared segmentation schemes in Dice metric with statistically sufficient margins. The proposed model brings new perspectives towards intelligent use of imaging resources and better management of pediatric musculoskeletal disorders.

D2-Net: Dual Disentanglement Network for Brain Tumor Segmentation With Missing Modalities

Multi-modal Magnetic Resonance Imaging (MRI) can provide complementary information for automatic brain tumor segmentation, which is crucial for diagnosis and prognosis. While missing modality data is common in clinical practice and it can result in the collapse of most previous methods relying on complete modality data. Current state-of-the-art approaches cope with the situations of missing modalities by fusing multi-modal images and features to learn shared representations of tumor regions, which often ignore explicitly capturing the correlations among modalities and tumor regions. Inspired by the fact that modality information plays distinct roles to segment different tumor regions, we aim to explicitly exploit the correlations among various modality-specific information and tumor-specific knowledge for segmentation. To this end, we propose a Dual Disentanglement Network (D2-Net) for brain tumor segmentation with missing modalities, which consists of a modality disentanglement stage (MD-Stage) and a tumor-region disentanglement stage (TD-Stage). In the MD-Stage, a spatial-frequency joint modality contrastive learning scheme is designed to directly decouple the modality-specific information from MRI data. To decompose tumor-specific representations and extract discriminative holistic features, we propose an affinity-guided dense tumor-region knowledge distillation mechanism in the TD-Stage through aligning the features of a disentangled binary teacher network with a holistic student network. By explicitly discovering relations among modalities and tumor regions, our model can learn sufficient information for segmentation even if some modalities are missing. Extensive experiments on the public BraTS-2018 database demonstrate the superiority of our framework over state-of-the-art methods in missing modalities situations. Codes are available at https://github.com/CityU-AIM-Group/D2Net.

Long-term intergenerational transmission of memories of the Vajont disaster.

Previous literature documented the traumatic consequences of exposure to disasters on psychological functioning, but little attention has been paid to the intergenerational transmission of the memory of disasters. We explored long-term effects on the memory of the Vajont dam disaster in Northeast Italy that claimed 1,910 lives in 1963. We collected data from 52 two-generation families in which the first generations were born before the disaster and the second generations after. The families were divided into an experimental group whose first generation survived the disaster and a control group whose first generation had moved there afterward. The interviews included an open-ended narrative on the memory of the disaster. We coded free narratives focusing on the richness of the memories (i.e., length, causes, core, aftermath), analyzing negative emotions and salience of the natural and psychological domains. We applied generalized linear mixed models. The richness of the memories, including references to negative emotions, diminished with lower exposure and with intergenerational transmission. Moreover, the participants built a shared representation of the disaster that did not markedly differ across exposure or generation. The reported causes were attributed more to the natural rather than the human domain; the consequences more to the psychological compared to the material domain. Our findings highlight the processes through which collective memories of historical traumatic events are built over the long term and the way a collective identity develops to bear the burden of highly dramatic events and to transmit intergenerational lessons from the past. (PsycInfo Database Record (c) 2022 APA, all rights reserved).