Common Representation Research Articles

Attention-based stackable graph convolutional network for multi-view learning

In multi-view learning, graph-based methods like Graph Convolutional Network (GCN) are extensively researched due to effective graph processing capabilities. However, most GCN-based methods often require complex preliminary operations such as sparsification, which may bring additional computation costs and training difficulties. Additionally, as the number of stacking layers increases in most GCN, over-smoothing problem arises, resulting in ineffective utilization of GCN capabilities. In this paper, we propose an attention-based stackable graph convolutional network that captures consistency across views and combines attention mechanism to exploit the powerful aggregation capability of GCN to effectively mitigate over-smoothing. Specifically, we introduce node self-attention to establish dynamic connections between nodes and generate view-specific representations. To maintain cross-view consistency, a data-driven approach is devised to assign attention weights to views, forming a common representation. Finally, based on residual connectivity, we apply an attention mechanism to the original projection features to generate layer-specific complementarity, which compensates for the information loss during graph convolution. Comprehensive experimental results demonstrate that the proposed method outperforms other state-of-the-art methods in multi-view semi-supervised tasks.

StarPlat: A versatile DSL for graph analytics

Graphs model several real-world phenomena. With the growth of unstructured and semi-structured data, parallelization of graph algorithms is inevitable. Unfortunately, due to inherent irregularity of computation, memory access, and communication, graph algorithms are traditionally challenging to parallelize. To tame this challenge, several libraries, frameworks, and domain-specific languages (DSLs) have been proposed to reduce the parallel programming burden of the users, who are often domain experts. However, existing frameworks to model graph algorithms typically target a single architecture. In this paper, we present a graph DSL, named StarPlat, that allows programmers to specify graph algorithms in a high-level format, but generates code for three different backends from the same algorithmic specification. In particular, the DSL compiler generates OpenMP for multi-core systems, MPI for distributed systems, and CUDA for many-core GPUs. Since these three are completely different parallel programming paradigms, binding them together under the same language is challenging. We share our experience with the language design. Central to our compiler is an intermediate representation which allows a common representation of the high-level program, from which individual backend code generations begin. We demonstrate the expressiveness of StarPlat by specifying four graph algorithms: betweenness centrality computation, page rank computation, single-source shortest paths, and triangle counting. Using a suite of ten large graphs, we illustrate the effectiveness of our approach by comparing the performance of the generated codes with that obtained with hand-crafted library codes. We find that the generated code is competitive to library-based codes in many cases. More importantly, we show the feasibility to generate efficient codes for different target architectures from the same algorithmic specification of graph algorithms.

Temporal context modulates cross-modality time discrimination: Electrophysiological evidence for supramodal temporal representation

Although the peripheral nervous system lacks a dedicated receptor, the brain processes temporal information through different sensory channels. A critical question is whether temporal information from different sensory modalities at different times forms modality-specific representations or is integrated into a common representation in a supramodal manner. Behavioral studies on temporal memory mixing and the central tendency effect have provided evidence for supramodal temporal representations. We aimed to provide electrophysiological evidence for this proposal by employing a cross-modality time discrimination task combined with electroencephalogram (EEG) recordings. The task maintained a fixed auditory standard duration, whereas the visual comparison duration was randomly selected from the short and long ranges, creating two different audio–visual temporal contexts. The behavioral results showed that the point of subjective equality (PSE) in the short context was significantly lower than that in the long context. The EEG results revealed that the amplitude of the contingent negative variation (CNV) in the short context was significantly higher (more negative) than in the long context in the early stage, while it was lower (more positive) in the later stage. These results suggest that the audiovisual temporal context is integrated with the auditory standard duration to generate a subjective time criterion. Compared with the long context, the subjective time criterion in the short context was shorter, resulting in earlier decision-making and a preceding decrease in CNV. Our study provides electrophysiological evidence that temporal information from different modalities inputted into the brain at different times can form a supramodal temporal representation.

Partially shared federated multiview learning

With the diversification of data representations, the field of multiview learning has attracted the attention of many researchers. However, multiview data is stored in different devices in practical applications, and existing multiview learning methods fail to apply directly. Moreover, the data transmission process is often accompanied by the risk of privacy leakage. Therefore, this paper proposes a multiview learning method based on a vertical federated learning framework called partially shared federated multiview learning method (PSFedML) to handle this issue. Our proposed method uses a partial sharing approach that can simultaneously utilize consistent and complementary information between views to learn latent representations. In addition, the transmission of common latent representations between local clients and the central server avoids the risk of privacy leakage. We also give a corresponding iterative optimization algorithm. Experiments on real-world datasets demonstrate the efficiency and superiority of PSFedML.

Adaptive multi-label structure preserving network for cross-modal retrieval

To integrate the label structures, which describe semantic correlations among labels, into the learned common representations, many existing methods leverage the label embeddings learned according to label structures, to map the data of different modalities into a common representation space. However, these methods cannot fully discover the semantic correlation between labels. In this paper, we propose an Adaptive Multi-label Structure Preserving Network (AMLSPN) to dynamically learn the multi-label correlations and multi-label embeddings for learning common representations, which can preserve the label structures. Our method introduces a series of multi-label correlation matrices to capture the structures of multi-label nodes in the multi-label graph. Moreover, we present a novel hierarchical correlation loss to supervise the learning process of these multi-label correlation matrices. Additionally, we introduce a group GCN to enhance the training speed of our model. Extensive evaluations on three benchmark datasets demonstrate that our proposed AMLSPN outperforms the state-of-the-art methods.

Collaborative filtering with representation learning in the frequency domain

In the context of recommender systems, collaborative filtering is the method of predicting the ratings of a set of items given by a set of users based on partial knowledge of the ratings. Commonly, items and users are represented via vectors, and to predict ratings, approaches such as vector inner-product (aka matrix factorization) or more advanced nonlinear functions are applied. In this paper, while we adopt the common vectorial representation, we consider a general model in which the ratings are smooth functions of the item representations. Smoothness ensures similar items with nearby vectors will also get similar ratings as we expect from a human rater. We represent user smooth scoring functions in a so-called frequency domain and learn their representations alongside item representations using 1) an iterative optimization approach that maps items and users alternatively, and 2) a feedforward neural network consisting of interpretable layers. We also address the challenge of the distribution shift from observed to unobserved ratings (aka missing-not-at-random) with insights from the frequency domain. We evaluate the predictive power of our method and its robustness in missed-not-at-random settings on four popular benchmarks. Despite its simplicity and interpretability, our method yields a remarkable performance compared to the state-of-the-art.1

A unified representation framework for the evaluation of Optical Music Recognition systems

Modern-day Optical Music Recognition (OMR) is a fairly fragmented field. Most OMR approaches use datasets that are independent and incompatible between each other, making it difficult to both combine them and compare recognition systems built upon them. In this paper we identify the need of a common music representation language and propose the Music Tree Notation format, with the idea to construct a common endpoint for OMR research that allows coordination, reuse of technology and fair evaluation of community efforts. This format represents music as a set of primitives that group together into higher-abstraction nodes, a compromise between the expression of fully graph-based and sequential notation formats. We have also developed a specific set of OMR metrics and a typeset score dataset as a proof of concept of this idea.

Multimodal semantic enhanced representation network for micro-video event detection

Currently, micro-videos have gained widespread acceptance as a prominent form of user-generated content across various social media platforms. Accurate event analysis of micro-videos can greatly enhance the many diverse social media platforms applications. Although some studies have shown promising results from a multimodal perspective, there is still a challenge in extracting informative cues from inaccurate modalities, particularly for text modality that is prone to inaccuracies and noise. In this paper, we propose a multimodal semantically enhanced representation network (MSERN) for micro-video event detection. To better address inaccurate and noisy text sentences, we first extract visual concepts in the form of adjective-noun pairs (ANPs), through a fine-grained common representation module, to complement the textual descriptions. To maximize the acquisition of modality-specific cues from both visual and textual modalities, we then implement a coarse-grained private representation module to ensure that private representations encompass unique facets of the modalities beyond the common perspective. Finally, because two modules will collaborate, the fine-grained common and coarse-grained private representations are integrated to ensure a reinforced micro-video representation. We evaluate our proposed method on a micro-video event detection dataset and the experimental results show a superior performance compared to the state-of-the-art methods.

CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval

The first-stage retrieval aims to retrieve a subset of candidate documents from a huge collection both effectively and efficiently. Since various matching patterns can exist between queries and relevant documents, previous work tries to combine multiple retrieval models to find as many relevant results as possible. The constructed ensembles, whether learned independently or jointly, do not care which component model is more suitable to an instance during training. Thus, they cannot fully exploit the capabilities of different types of retrieval models in identifying diverse relevance patterns. Motivated by this observation, in this paper, we propose a Mixture-of-Experts (MoE) model consisting of representative matching experts and a novel competitive learning mechanism to let the experts develop and enhance their expertise during training. Specifically, our MoE model shares the bottom layers to learn common semantic representations and uses differently structured upper layers to represent various types of retrieval experts. Our competitive learning mechanism has two stages: (1) a standardized learning stage to train the experts equally to develop their capabilities to conduct relevance matching; (2) a specialized learning stage where the experts compete with each other on every training instance and get rewards and updates according to their performance to enhance their expertise on certain types of samples. Experimental results on retrieval benchmark datasets show that our method significantly outperforms the state-of-the-art baselines in the in-domain and out-of-domain settings.

Inverse mapping of quantum properties to structures for chemical space of small organic molecules

Computer-driven molecular design combines the principles of chemistry, physics, and artificial intelligence to identify chemical compounds with tailored properties. While quantum-mechanical (QM) methods, coupled with machine learning, already offer a direct mapping from 3D molecular structures to their properties, effective methodologies for the inverse mapping in chemical space remain elusive. We address this challenge by demonstrating the possibility of parametrizing a chemical space with a finite set of QM properties. Our proof-of-concept implementation achieves an approximate property-to-structure mapping, the QIM model (which stands for “Quantum Inverse Mapping”), by forcing a variational auto-encoder with a property encoder to obtain a common internal representation for both structures and properties. After validating this mapping for small drug-like molecules, we illustrate its capabilities with an explainability study as well as by the generation of de novo molecular structures with targeted properties and transition pathways between conformational isomers. Our findings thus provide a proof-of-principle demonstration aiming to enable the inverse property-to-structure design in diverse chemical spaces.

TSFAN: tensorized spatial-frequency attention network with domain adaptation for cross-session EEG-based biometric recognition

Objective. Electroencephalogram (EEG) signals are promising biometrics owning to their invisibility, adapting to the application scenarios with high-security requirements. However, It is challenging to explore EEG identity features without the interference of device and state differences of the subject across sessions. Existing methods treat training sessions as a single domain, affected by the different data distribution among sessions. Although most multi-source unsupervised domain adaptation (MUDA) methods bridge the domain gap between multiple source and target domains individually, relationships among the domain-invariant features of each distribution alignment are neglected. Approach. In this paper, we propose a MUDA method, Tensorized Spatial-Frequency Attention Network (TSFAN), to assist the performance of the target domain for EEG-based biometric recognition. Specifically, significant relationships of domain-invariant features are modeled via a tensorized attention mechanism. It jointly incorporates appropriate common spatial-frequency representations of pairwise source and target but also cross-source domains, without the effect of distribution discrepancy among source domains. Additionally, considering the curse of dimensionality, our TSFAN is approximately represented in Tucker format. Benefiting the low-rank Tucker Network, the TSFAN can scale linearly in the number of domains, providing us the great flexibility to extend TSFAN to the case associated with an arbitrary number of sessions. Main results. Extensive experiments on the representative benchmarks demonstrate the effectiveness of TSFAN in EEG-based biometric recognition, outperforming state-of-the-art approaches, as verified by cross-session validation. Significance. The proposed TSFAN aims to investigate the presence of consistent EEG identity features across sessions. It is achieved by utilizing a novel tensorized attention mechanism that collaborates intra-source transferable information with inter-source interactions, while remaining unaffected by domain shifts in multiple source domains. Furthermore, the electrode selection shows that EEG-based identity features across sessions are distributed across brain regions, and 20 electrodes based on 10–20 standard system are able to extract stable identity information.

CRE-TSCAE: A Novel Classification Model Based on Stacked Convolutional Autoencoder for Dual-Target RSVP-BCI Tasks.

The Rapid Serial Visual Presentation (RSVP) paradigm facilitates target identification in a rapid picture stream, which is applied extensively in military target surveillance and police monitoring. Most researchers concentrate on the single target RSVP-BCI whereas the study of dual-target is scarcely conducted, limiting RSVP application considerably. This paper proposed a novel classification model named Common Representation Extraction-Targeted Stacked Convolutional Autoencoder (CRE-TSCAE) to detect two targets with one nontarget in RSVP tasks. CRE generated a common representation for each target class to reduce variability from different trials of the same class and distinguish the difference between two targets better. TSCAE aimed to control uncertainty in the training process while requiring less target training data. The model learned a compact and discriminative feature through the training from several learning tasks so as to distinguish each class effectively. It was validated on the World Robot Contest 2021 and 2022 ERP datasets. Experimental results showed that CRE-TSCAE outperformed the state-of-the-art RSVP decoding algorithms and the Average ACC was 71.25%, improving 6.5% at least over the rest. It demonstrated that CRE-TSCAE showed a strong ability to extract discriminative latent features in detecting the differences among two targets with nontarget, which guaranteed increased classification accuracy. CRE-TSCAE provided an innovative and effective classification model for dual-target RSVP-BCI tasks and some insights into the neurophysiological distinction between different targets.

Low-Rank Tensor Regularized Views Recovery for Incomplete Multiview Clustering.

In real applications, it is often that the collected multiview data contain missing views. Most existing incomplete multiview clustering (IMVC) methods cannot fully utilize the underlying information of missing data or sufficiently explore the consistent and complementary characteristics. In this article, we propose a novel Low-rAnk Tensor regularized viEws Recovery (LATER) method for IMVC, which jointly reconstructs and utilizes the missing views and learns multilevel graphs for comprehensive similarity discovery in a unified model. The missing views are recovered from a common latent representation, and the recovered views conversely improve the learning of shared patterns. Based on the shared subspace representations and recovered complete multiview data, the multilevel graphs are learned by self-representation to fully exploit the consistent and complementary information among views. Besides, a tensor nuclear norm regularizer is introduced to pursue the global low-rank property and explore the interview correlations. An alternating direction minimization algorithm is presented to optimize the proposed model. Moreover, a new initialization method is proposed to promote the effectiveness of our method for latent representation learning and missing data recovery. Extensive experiments demonstrate that our method outperforms the state-of-the-art approaches.

Gender, sports, and belonging. The role of sports participation in residential preferences among girls and boys living in Norway's rural peripheries

The gender gap in residential preferences among rural youth has been documented in several studies – and for the most attributed to educational aspirations, employment, and income opportunities. The hereby study adds to this block of research by addressing the role of sports. Sports has been connected to place belonging. As a traditional masculine arena, sports can also be assumed to contribute to the gender gap in belonging. On this background, the study explores the role of sports participation in rural girls’ and boys’ residential preferences. As theoretical framework, feminist theories of gendered space contrasts feminist theories of sports participation, supplemented by elements from poststructuralist theories. Data consisted of school surveys from pupils aged 13–16 years living in Norway’s most peripheral municipalities (N=11,971). The analyses confirmed a gender gap in residential preferences. Boys more often considered living in their native municipality compared to girls. The role of sports was on the contrary more surprising. Rural girls who participated in sports had higher predicted probabilities for preferring to live in the native municipality compared to non-participants, but this was not the case for boys. Findings indicate that for girls living in rural Norway, organized sports represent arenas for positive place relations. This diverges from prevailing assumptions of gendered rural space. Furthermore, the study questions common representations of sports arenas as local hearths for masculine versions of Northern rural identity. New developments in leisure’s spatial and social organization within the rural periphery are discussed as potential reasons for these results. By combining and contrasting different strands of feminist thinking, the study gains new insights into gender, sports, and place belonging in the digital age.

Adaptive weighted multi-view subspace clustering method for recognizing urban functions from multi-source social sensing data

ABSTRACT Multi-source social sensing data provide new opportunities to identify urban functions from the perspective of human activity. The information embedded in multi-source data typically needs to be fused to obtain a comprehensive view of urban functions. Although multi-view clustering has been successfully used to fuse multi-source social sensing data, the adaptive determination of fusion weights for high-dimensional and noisy multi-source social sensing data remains challenging. Therefore, this study proposes an adaptive weighted multi-view subspace clustering (AWMSC) method. First, we use two neural networks to map multi-source data into a common latent representation and multiple specific latent representations, which serve as the query vector and input vectors of the attention mechanism, respectively. Then, the weight of each type of data is calculated based on the attention mechanism. Finally, the specific latent representations of the multi-source data are weighted and fused into a shared subspace representation, which is used as the input of the spectral clustering algorithm to obtain clustering results. AWMSC is applied to identify urban functional zones in Beijing using bus transactions, taxi trajectories, and points of interest datasets. The results show that AWMSC outperforms the typical single-view, weighted-average, and representative multi-view methods. AWMSC can obtain a comprehensive understanding of urban functional zones which may help government departments make more accurate strategic decisions.

Online Cross-modal Hashing With Dynamic Prototype

Online cross-modal hashing has received increasing attention due to its efficiency and effectiveness in handling cross-modal streaming data retrieval. Despite the promising performance, these methods mainly focus on the supervised learning paradigm, demanding expensive and laborious work to obtain clean annotated data. Existing unsupervised online hashing methods mostly struggle to construct instructive semantic correlations among data chunks, resulting in the forgetting of accumulated data distribution. To this end, we propose a Dynamic Prototype-based Online Cross-modal Hashing method, called DPOCH. Based on the pre-learned reliable common representations, DPOCH generates prototypes incrementally as sketches of accumulated data and updates them dynamically for adapting streaming data. Thereafter, the prototype-based semantic embedding and similarity graphs are designed to promote stability and generalization of the hashing process, thereby obtaining globally adaptive hash codes and hash functions. Experimental results on benchmarked datasets demonstrate that the proposed DPOCH outperforms state-of-the-art unsupervised online cross-modal hashing methods.

Fast unsupervised multi-modal hashing based on piecewise learning

Unsupervised hashing has been extensively applied in large-scale multi-modal retrieval by mapping original data from heterogeneous modalities into unified binary codes. However, there still remain challenges especially how to balance the individual modality-specific representations and common representation preserving intrinsic linkages among heterogeneous modalities. In this paper, we propose a novel fast Unsupervised Multi-modal Hashing based on Piecewise Learning, denoted as UMHPL, to deal with the mentioned issue. Initially, we formulate the problem as matrix factorization to derive the individual modality-specific latent representations and common latent representation with consensus matrices in a brief time. To maintain the integrality of multi-modal data, we integrate them by adaptive weight factors and nuclear norm minimization. Subsequently, we establish a connection between the individual modality-specific latent representations and common latent representation based on the piecewise hash learning framework to reinforce the discriminative competency of model, which leads the hash codes more compact. Finally, an effective discrete optimization algorithm in mathematical logic and functional analysis is proposed. Comprehensive experiments on Wiki, MIRFlirck, NUS-WIDE, and MSCOCO datasets demonstrate the superior performance of UMHPL to state-of-the-art hashing methods.

A unified framework for perceived magnitude and discriminability of sensory stimuli

The perception of sensory attributes is often quantified through measurements of sensitivity (the ability to detect small stimulus changes), as well as through direct judgments of appearance or intensity. Despite their ubiquity, the relationship between these two measurements remains controversial and unresolved. Here, we propose a framework in which they arise from different aspects of a common representation. Specifically, we assume that judgments of stimulus intensity (e.g., as measured through rating scales) reflect the mean value of an internal representation, and sensitivity reflects a combination of mean value and noise properties, as quantified by the statistical measure of Fisher information. Unique identification of these internal representation properties can be achieved by combining measurements of sensitivity and judgments of intensity. As a central example, we show that Weber’s law of perceptual sensitivity can coexist with Stevens’ power-law scaling of intensity ratings (for all exponents), when the noise amplitude increases in proportion to the representational mean. We then extend this result beyond the Weber’s law range by incorporating a more general and physiology-inspired form of noise and show that the combination of noise properties and sensitivity measurements accurately predicts intensity ratings across a variety of sensory modalities and attributes. Our framework unifies two primary perceptual measurements—thresholds for sensitivity and rating scales for intensity—and provides a neural interpretation for the underlying representation.

Object detection on low-resolution images with two-stage enhancement

Although deep learning-based object detection methods have achieved superior performance on conventional benchmark datasets, it is still difficult to detect objects from low-resolution (LR) images under diverse degradation conditions. To this end, a two-stage enhancement method for the LR image object detection (TELOD) framework is proposed. In the first stage, an extremely lightweight task disentanglement enhancement network (TDEN) is developed as a super-resolution (SR) sub-network before the detector. In the TDEN, the SR images can be obtained by applying the recurrent connection manner between an image restoration branch (IRB) and a resolution enhancement branch (REB) to enhance the input LR images. Specifically, the TDEN reduces the difficulty of image reconstruction by dividing the total image enhancement task into two sub-tasks, which are accomplished by the IRB and REB, respectively. Furthermore, a shared feature extractor is applied across two sub-tasks to explore common and accurate feature representations. In the second stage, an auxiliary feature enhancement head (AFEH) driven by high-resolution (HR) image priors is designed to improve the task-specific features produced by the detection Neck without any extra inference costs. In particular, the feature interaction module is built into the AFEH to integrate the features from the enhancement and detection phases to learn comprehensive information for detection. Extensive experiments show that the proposed TELOD significantly outperforms other methods. Specifically, the TELOD achieves mAP improvements of 1.8% and 3.3% over the second best method AERIS on degraded VOC and COCO datasets, respectively.

Toward Human-Like Representation Learning for Cognitive Architectures

Human-like learning includes an ability to learn concepts from a stream of embodiment sensor data. Echoing previous thoughts such as those from Barsalou that cognition and perception share a common representation system, we suggest an addendum to the common model of cognition. This addendum poses a simultaneous semantic memory and perception learning that bypasses working memory, and that uses parallel processing to learn concepts apart from deliberate reasoning. The goal is to provide a general outline for how to extend a class of cognitive architectures to implement a more human-like interface between cognition and embodiment of an agent, where a critical aspect of that interface is that it is dynamic because of learning.