DAS-MAE: A Self-Supervised Framework for Universal and High-Performance Representation Learning of Distributed Acoustic Sensing

• SAURL-TS is a novel framework for self-supervised time series representation learning. • SAURL-TS adaptively fits diverse temporal and spectral patterns across various time series. • SAURL-TS introduces adaptive augmentation to generate high-quality positive pairs. • SAURL-TS achieves strong performance without relying on negative samples. • SAURL-TS is validated through extensive and robust evaluation on classification and forecasting. Unsupervised time series representation learning, driven by recent advances in contrastive learning-based methods, has become a critical component for downstream tasks like forecasting and classification. However, time series data exhibit complex temporal dependencies and spectral patterns, posing challenges for existing approaches to adapt robustly. Moreover, existing contrastive learning-based approaches overlook frequency-domain information and struggle with selecting effective negative samples, further hindering model performance. To address these issues, we propose S a URL-TS, a novel self-adaptive framework for unsupervised time series representation learning. First, it dynamically learns dataset-specific augmentations to generate high-quality positive samples. Second, an adaptive self-supervised learning module with a multi-domain encoder captures both temporal and spectral patterns without relying on negative samples. Third, a representation-wise attention mechanism assigns dynamic weights to representations across domains. To the best of our knowledge, S a URL-TS is the first self-supervised learning framework to jointly model temporal and spectral patterns across both augmentation and learning stages. Extensive experiments confirm the superior performance of S a URL-TS over state-of-the-art models. Notably, its adaptive data augmentation module is plug-and-play and can be integrated into other contrastive learning frameworks, and its learning stage is capable of adapting to a wide range of time series patterns. Our codebase is available at https://github.com/YusenL/SAURL-TS .

Recently, models for user representation learning have been widely applied in click-through-rate (CTR) and conversion-rate (CVR) prediction. Usually, the model learns a universal user representation as the input for subsequent scenario-specific models. However, in numerous industrial applications (e.g., recommendation and marketing), the business always operates such applications as various online activities among different user segmentation. These segmentation are always created by domain experts. Due to the difference in user distribution (i.e., user segmentation) and business objectives in subsequent tasks, learning solely on universal representation may lead to detrimental effects on both model performance and robustness. In this paper, we propose a novel learning framework that can first learn general universal user representation through information bottleneck. Then, merge and learn a segmentation-specific or a task-specific representation through neural interaction. We design the interactive learning process by leveraging a bipartite graph architecture to model the representation learning and merging between contextual clusters and each user segmentation. Our proposed method is evaluated in two open-source benchmarks, two offline business datasets, and deployed on two online marketing applications to predict users’ CVR. The results demonstrate that our method can achieve superior performance and surpass the baseline methods.

Unsupervised spatial representation learning aims to automatically identify effective features of geographic entities (i.e., regions) from unlabeled yet structural geographical data. Existing network embedding methods can partially address the problem by: (1) regarding a region as a node in order to reformulate the problem into node embedding; (2) regarding a region as a graph in order to reformulate the problem into graph embedding. However, these studies can be improved by preserving (1) intra-region geographic structures, which are represented by multiple spatial graphs, leading to a reformulation of collective learning from relational graphs; (2) inter-region spatial autocorrelations, which are represented by pairwise graph regularization, leading to a reformulation of adversarial learning. Moreover, field data in real systems are usually lack of labels, an unsupervised fashion helps practical deployments. Along these lines, we develop an unsupervised Collective Graph-regularized dual-Adversarial Learning (CGAL) framework for multi-view graph representation learning and also a Graph-regularized dual-Adversarial Learning (GAL) framework for single-view graph representation learning. Finally, our experimental results demonstrate the enhanced effectiveness of our method.

Given the scarcity of labeled data, hyperspectral image (HSI) analysis has increasingly relied on self-supervised representation learning. Masked reconstruction and contrastive learning are two dominant paradigms that emphasize spectral–spatial fidelity and feature discrimination, respectively, yet their objectives are not naturally aligned in HSI: reconstruction is vulnerable to redundancy-induced shortcut solutions, while contrastive learning may under-emphasize subtle but class-critical spectral cues. Motivated by this gap, we propose a joint Masked Spectral Group (MSG) framework for hyperspectral representation learning that couples the two objectives through a shared spectral-group representation and objective-consistent structured masking. At its core, MSG builds on Spectral Group Embedding and Masking (SGEM), which organizes contiguous bands into compact groups and performs group-wise masking to suppress redundancy while preserving coherent spectral context for stable view formation. This design encourages reconstruction and discrimination to complement rather than interfere with each other, yielding representations that are both discriminative and transferable. Extensive experiments under linear evaluation and few-shot fine-tuning across multiple downstream benchmarks demonstrate that MSG consistently outperforms strong masked-reconstruction and contrastive learning baselines, achieving higher accuracy, more stable class-wise performance, and stronger transferability.

Educational process data, i.e., logs of detailed student activities in computerized or online learning platforms, has the potential to offer deep insights into how students learn. One can use process data for many downstream tasks such as learning outcome prediction and automatically delivering personalized intervention. In this paper, we propose a framework for learning representations of educational process data that is applicable across different learning scenarios. Our framework consists of a pre-training step that uses BERT-type objectives to learn representations from sequential process data and a fine-tuning step that further adjusts these representations on downstream prediction tasks. We apply our framework to the 2019 nation's report card data mining competition dataset that consists of student problem-solving process data and detail the specific models we use in this scenario. We conduct both quantitative and qualitative experiments to show that our framework results in process data representations that are both predictive and informative.

FEAT: A general framework for feature-aware multivariate time-series representation learning

This paper presents MoCoUTRL: a Momentum Contrastive Framework for Unsupervised Text Representation Learning. This model improves two aspects of recently popular contrastive learning algorithms in natural language processing (NLP). Firstly, MoCoUTRL employs multi-granularity semantic contrastive learning objectives, enabling a more comprehensive understanding of the semantic features of samples. Secondly, MoCoUTRL uses a dynamic dictionary to act as the approximately ground-truth representation for each token, providing the pseudo labels for token-level contrastive learning. The MoCoUTRL can extend the use of pre-trained language models (PLM) and even large-scale language models (LLM) into a plug-and-play semantic feature extractor that can fuel multiple downstream tasks. Experimental results on several publicly available datasets and further theoretical analysis validate the effectiveness and interpretability of the proposed method in this paper.

In this article, we present an optimal edge-weighted graph semantic correlation (EWGSC) framework for multi-view feature representation learning. Different from most existing multi-view representation methods, local structural information and global correlation in multi-view feature spaces are exploited jointly in the EWGSC framework, leading to a new and high-quality multi-view feature representation. Specifically, a novel edge-weighted graph model is first conceptualized and developed to preserve local structural information in each of the multi-view feature spaces. Then, the explored structural information is integrated with a semantic correlation algorithm, labeled multiple canonical correlation analysis (LMCCA), to form a powerful platform for effectively exploiting local and global relations across multi-view feature spaces jointly. We then theoretically verified the relation between the upper limit on the number of projected dimensions and the optimal solution to the multi-view feature representation problem. To validate the effectiveness and generality of the proposed framework, we conducted experiments on five datasets of different scales, including visual-based (University of California Irvine (UCI) iris database, Olivetti Research Lab (ORL) face database, and Caltech 256 database), text-image-based (Wiki database), and video-based (Ryerson Multimedia Lab (RML) audio-visual emotion database) examples. The experimental results show the superiority of the proposed framework on multi-view feature representation over state-of-the-art algorithms.

CommPOOL: An interpretable graph pooling framework for hierarchical graph representation learning

In this paper, we propose an iterative framework for self-supervised speaker representation learning based on a deep neural network (DNN). The framework starts with training a self-supervision speaker embedding network by maximizing agreement between different segments within an utterance via a contrastive loss. Taking advantage of DNN’s ability to learn from data with label noise, we propose to cluster the speaker embedding obtained from the previous speaker network and use the subsequent class assignments as pseudo labels to train a new DNN. Moreover, we iteratively train the speaker network with pseudo labels generated from the previous step to bootstrap the discriminative power of a DNN. Speaker verification experiments are conducted on the VoxCeleb dataset. The results show that our proposed iterative self-supervised learning framework outperformed previous works using self-supervision. The speaker network after 5 iterations obtains a 61% performance gain over the speaker embedding model trained with contrastive loss.

BackgroundAs the core functional carrier of life activities, the quality of protein representation directly affects the accuracy of downstream functional prediction. In recent years, multimodal deep learning methods have significantly improved the effectiveness of protein representation learning by virtue of their advantages in fusing sequence, structure, and chemical characteristics. However, current research still faces two core challenges: first, the guiding mechanism for structural information during multi-modal feature interaction has not been fully explored; second, existing fusion strategies mostly use static weight allocation mechanisms, which is difficult to adapt to sequence-structural features. The dynamic correlation between features leads to limited accuracy in identifying key functional residues.ResultsWe proposed ProGraphTrans, a multimodal dynamic collaborative framework for protein representation learning. ProGraphTrans builds a dynamic attention multimodal fusion mechanism and captures local sequential patterns through a multi-scale convolutional neural network.ConclusionsExperimental results on four protein downstream tasks show that ProGraphTrans not only outperforms other methods in various indicators but also demonstrates excellent interpretability, demonstrating its advantages and effectiveness as a protein representation method.

We present a novel framework for multivariate time series representation learning based on the transformer encoder architecture. The framework includes an unsupervised pre-training scheme, which can offer substantial performance benefits over fully supervised learning on downstream tasks, both with but even without leveraging additional unlabeled data, i.e., by reusing the existing data samples. Evaluating our framework on several public multivariate time series datasets from various domains and with diverse characteristics, we demonstrate that it performs significantly better than the best currently available methods for regression and classification, even for datasets which consist of only a few hundred training samples. Given the pronounced interest in unsupervised learning for nearly all domains in the sciences and in industry, these findings represent an important landmark, presenting the first unsupervised method shown to push the limits of state-of-the-art performance for multivariate time series regression and classification.

Single-cell RNA sequencing (scRNA-seq) has become a groundbreaking tool in life science research. Moreover, advances in Large Language Models (LLMs) have greatly catalyzed the development of cellular foundation models in transcriptomics. However, existing scRNA-seq models built on LLMs predominantly employ transformer architectures, which are constrained by quadratic computational complexity during inference, limiting their ability to process long sequences. Furthermore, current cellular foundation models often focus on a single objective like Masked Language Modeling (MLM), which limits their ability to capture semantically rich and biologically meaningful representations. Critically, the field lacks an efficient and general framework for scalable cell representation learning. To address these limitations, we introduce MambaCell, a multi-task self-supervised learning framework based on the bidirectional Mamba architecture. This novel framework integrates two complementary self-supervised tasks including Masked Gene Modeling (MGM) and Contrastive Learning (CL), enabling it to learn robust cell representations from large-scale, un-labeled datasets with reduced inference costs. Experimental results demonstrate that MambaCell achieves superior or comparable performance to state-of-the-art (SOTA) models on a series of downstream tasks, including cell type annotation, disease-related cell classification, single-cell batch integration, etc. Notably, MambaCell achieves faster inference speed and higher memory efficiency compared to transformer-based models with comparable parameters. These superior performance and efficiency gains underscore the effectiveness of MambaCell's architectural innovation, establishing it as a scalable solution for large-scale single-cell transcriptomic analysis.

Enterprise data typically involves multiple heterogeneous data sources and external data that respectively record business activities, transactions, customer demographics, status, behaviors, interactions and communications with the enterprise, and the consumption and feedback of its products, services, production, marketing, operations, and management, etc. They involve enterprise DNA associated with domain-oriented transactions and master data, informational and operational metadata, and relevant external data. A critical challenge in enterprise data science is to enable an effective ‘whole-of-enterprise’ data understanding and data-driven discovery and decision-making on all-round enterprise DNA. Accordingly, here we introduce a neural encoder Table2Vec for automated universal representation learning of entities such as customers from all-round enterprise DNA with automated data characteristics analysis and data quality augmentation. The learned universal representations serve as representative and benchmarkable enterprise data genomes (similar to biological genomes and DNA in organisms) and can be used for enterprise-wide and domain-specific learning tasks. Table2Vec integrates automated universal representation learning on low-quality enterprise data and downstream learning tasks. Such automated universal enterprise representation and learning cannot be addressed by existing enterprise data warehouses (EDWs), business intelligence and corporate analytics systems, where ‘enterprise big tables’ are constructed with reporting and analytics conducted by specific analysts on respective domain subjects and goals. It addresses critical limitations and gaps of existing representation learning, enterprise analytics and cloud analytics, which are analytical subject, task and data-specific, creating analytical silos in an enterprise. We illustrate Table2Vec in characterizing all-round customer data DNA in an enterprise on complex heterogeneous multi-relational big tables to build universal customer vector representations. The learned universal representation of each customer is all-round, representative and benchmarkable to support both enterprise-wide and domain-specific learning goals and tasks in enterprise data science. Table2Vec significantly outperforms the existing shallow, boosting and deep learning methods typically used for enterprise analytics. We further discuss the research opportunities, directions and applications of automated universal enterprise representation and learning and the learned enterprise data DNA for automated, all-purpose, whole-of-enterprise and ethical machine learning and data science.

Graph Neural Networks (GNNs) have been the standard toolkit for processing non-euclidean spatial data since their powerful capability in graph representation learning. Unfortunately, their training strategy for network parameters is inefficient since it is directly inherited from classic Neural Networks (NNs), ignoring the characteristic of GNNs. To alleviate this issue, experimental analyses are performed to investigate the knowledge captured in classifier parameters during network training. We conclude that the parameter features, i.e., the column vectors of the classifier parameter matrix, are cluster representations with high discriminability. And after a theoretical analysis, we conclude that the discriminability of these features is obtained from the feature propagation from nodes to parameters. Furthermore, an experiment verifies that compared with cluster centroids, the parameter features are more potential for augmenting the feature propagation between nodes. Accordingly, a novel GNN-specific training framework is proposed by simultaneously updating node representations and classifier parameters via a unified feature propagation scheme. Moreover, two augmentation schemes are implemented for the framework, named Full Propagation Augmentation (FPA) and Simplified Full Propagation Augmentation (SFPA). Specifically, FPA augmentates the feature propagation of each node with the updated classifier parameters. SFPA only augments nodes with the classifier parameters corresponding to their clusters. Theoretically, FPA is equivalent to optimizing a novel graph learning objective, which demonstrates the universality of the proposed framework to existing GNNs. Extensive experiments demonstrate the superior performance and the universality of the proposed framework.