High Dimensional Representation Research Articles

Language proficiency is associated with neural representational dimensionality of semantic concepts

Previous studies suggest that semantic concepts are characterized by high-dimensional neural representations and that language proficiency affects semantic processing. However, it is not clear whether language proficiency modulates the dimensional representations of semantic concepts at the neural level. To address this question, the present study adopted principal component analysis (PCA) and representational similarity analysis (RSA) to examine the differences in representational dimensionalities (RDs) and in semantic representations between words in highly proficient (Chinese) and less proficient (English) language. PCA results revealed that language proficiency increased the dimensions of lexical representations in the left inferior frontal gyrus, temporal pole, inferior temporal gyrus, supramarginal gyrus, angular gyrus, and fusiform gyrus. RSA results further showed that these regions represented semantic information and that higher semantic representations were observed in highly proficient language relative to less proficient language. These results suggest that language proficiency is associated with the neural representational dimensionality of semantic concepts.

A phenotypic drug discovery approach by latent interaction in deep learning

Contemporary drug discovery paradigms rely heavily on binding assays about the bio-physicochemical processes. However, this dominant approach suffers from overlooked higher-order interactions arising from the intricacies of molecular mechanisms, such as those involving cis -regulatory elements. It introduces potential impairments and restrains the potential development of computational methods. To address this limitation, I developed a deep learning model that leverages an end-to-end approach, relying exclusively on therapeutic information about drugs. By transforming textual representations of drug and virus genetic information into high-dimensional latent representations, this method evades the challenges arising from insufficient information about binding specificities. Its strengths lie in its ability to implicitly consider complexities such as epistasis and chemical–genetic interactions, and to handle the pervasive challenge of data scarcity. Through various modeling skills and data augmentation techniques, the proposed model demonstrates outstanding performance in out-of-sample validations, even in scenarios with unknown complex interactions. Furthermore, the study highlights the importance of chemical diversity for model training. While the method showcases the feasibility of deep learning in data-scarce scenarios, it reveals a promising alternative for drug discovery in situations where knowledge of underlying mechanisms is limited.

The prediction of bolted joint loosening state under transverse load based on a data-driven model

The impact of transverse load results in the loosening of bolted joints. The loosening condition is determined based on the transverse load and relevant parameters of the bolted joint. To efficiently forecast the loosening state of bolts under various transverse loads and relevant parameters, this study proposes a predictive model for the loosening state of bolted joints. Initially, the relationship between the loosening criteria and relevant parameters is established according to the bolt’s force conditions. Through elementary effect analysis, the high-dimensional representation model is improved, breaking down the loosening criteria function with high-dimensional parameter inputs into a series of low-dimensional functions. Subsequently, the prediction of the loosening state is achieved by training artificial neural networks to approximate multiple low-dimensional functions. The improved predictive model is utilized to determine the critical load for bolted joint loosening. Validation indicates the accuracy of the proposed model. Notably, the influence of the bearing friction coefficient and preload on the critical transverse load is more pronounced compared to the thread friction coefficient.

High-dimensional Ageome Representations of Biological Aging across Functional Modules.

The aging process involves numerous molecular changes that lead to functional decline and increased disease and mortality risk. While epigenetic aging clocks have shown accuracy in predicting biological age, they typically provide single estimates for the samples and lack mechanistic insights. In this study, we challenge the paradigm that aging can be sufficiently described with a single biological age estimate. We describe Ageome, a computational framework for measuring the epigenetic age of thousands of molecular pathways simultaneously in mice and humans. Ageome is based on the premise that an organism's overall biological age can be approximated by the collective ages of its functional modules, which may age at different rates and have different biological ages. We show that, unlike conventional clocks, Ageome provides a high-dimensional representation of biological aging across cellular functions, enabling comprehensive assessment of aging dynamics within an individual, in a population, and across species. Application of Ageome to longevity intervention models revealed distinct patterns of pathway-specific age deceleration. Notably, cell reprogramming, while rejuvenating cells, also accelerated aging of some functional modules. When applied to human cohorts, Ageome demonstrated heterogeneity in predictive power for mortality risk, and some modules showed better performance in predicting the onset of age-related diseases, especially cancer, compared to existing clocks. Together, the Ageome framework offers a comprehensive and interpretable approach for assessing aging, providing insights into mechanisms and targets for intervention.

Decoupling visual and identity features for adversarial palm-vein image attack

Palm-vein has been widely used for biometric recognition due to its resistance to theft and forgery. However, with the emergence of adversarial attacks, most existing palm-vein recognition methods are vulnerable to adversarial image attacks, and to the best of our knowledge, there is still no study specifically focusing on palm-vein image attacks. In this paper, we propose an adversarial palm-vein image attack network that generates highly similar adversarial palm-vein images to the original samples, but with altered palm-identities. Unlike most existing generator-oriented methods that directly learn image features via concatenated convolutional layers, our proposed network first maps palm-vein images into multi-scale high-dimensional shallow representation, and then develops attention-based dual-path feature learning modules to extensively exploit diverse palm-vein-specific features. After that, we design visual-consistency and identity-aware loss functions to specially decouple the visual and identity features to reconstruct the adversarial palm-vein images. By doing this, the visual characteristics of palm-vein images can be largely preserved while the identity information is removed in the adversarial palm-vein images, such that high-aggressive adversarial palm-vein samples can be obtained. Extensive white-box and black-box attack experiments conducted on three widely used databases clearly show the effectiveness of the proposed network.

Multimodal fusion for large-scale traffic prediction with heterogeneous retentive networks

Traffic speed prediction is a critical challenge in transportation research due to the complex spatiotemporal dynamics of urban mobility. This study proposes a novel framework for fusing diverse data modalities to enhance short-term traffic speed forecasting accuracy. We introduce the Heterogeneous Retentive Network (H-RetNet), which integrates multisource urban data into high-dimensional representations encoded with geospatial relationships. By combining the H-RetNet with a Gated Recurrent Unit (GRU), our model captures intricate spatial and temporal correlations. We validate the approach using a real-world Beijing traffic dataset encompassing social media, real estate, and point of interest data. Experiments demonstrate superior performance over existing methods, with the fusion architecture improving robustness. Specifically, we observe a 21.91% reduction in MSE, underscoring the potential of our framework to inform and enhance traffic management strategies.

Nested barycentric coordinate system as an explicit feature map for polyhedra approximation and learning tasks

We introduce a new embedding technique based on a nested barycentric coordinate system. We show that our embedding can be used to transform the problems of polyhedron approximation, piecewise linear classification and convex regression into one of finding a linear classifier or regressor in a higher dimensional (but nevertheless quite sparse) representation. Our embedding maps a piecewise linear function into an everywhere-linear function, and allows us to invoke well-known algorithms for the latter problem to solve the former. We explain the applications of our embedding to the problems of approximating separating polyhedra—in fact, it can approximate any convex body and unions of convex bodies—as well as to classification by separating polyhedra, and to piecewise linear regression.

LearnMSA2: deep protein multiple alignments with large language and hidden Markov models.

For the alignment of large numbers of protein sequences, tools are predominant that decide to align two residues using only simple prior knowledge, e.g. amino acid substitution matrices, and using only part of the available data. The accuracy of state-of-the-art programs declines with decreasing sequence identity and when increasingly large numbers of sequences are aligned. Recently, transformer-based deep-learning models started to harness the vast amount of protein sequence data, resulting in powerful pretrained language models with the main purpose of generating high-dimensional numerical representations, embeddings, for individual sites that agglomerate evolutionary, structural, and biophysical information. We extend the traditional profile hidden Markov model so that it takes as inputs unaligned protein sequences and the corresponding embeddings. We fit the model with gradient descent using our existing differentiable hidden Markov layer. All sequences and their embeddings are jointly aligned to a model of the protein family. We report that our upgraded HMM-based aligner, learnMSA2, combined with the ProtT5-XL protein language model aligns on average almost 6% points more columns correctly than the best amino acid-based competitor and scales well with sequence number. The relative advantage of learnMSA2 over other programs tends to be greater when the sequence identity is lower and when the number of sequences is larger. Our results strengthen the evidence on the rich information contained in protein language models' embeddings and their potential downstream impact on the field of bioinformatics. Availability and implementation: https://github.com/Gaius-Augustus/learnMSA, PyPI and Bioconda, evaluation: https://github.com/felbecker/snakeMSA.

Ionic species representations for materials informatics

High-dimensional representations of the elements have become common within the field of materials informatics to build useful, structure-agnostic models for the chemistry of materials. However, the characteristics of elements change when they adopt a given oxidation state, with distinct structural preferences and physical properties. We explore several methods for developing embedding vectors of elements decorated with oxidation states. Graphs generated from 110 160 crystals are used to train representations of 84 elements that form 336 species. Clustering these learned representations of ionic species in low-dimensional space reproduces expected chemical heuristics, particularly the separation of cations from anions. We show that these representations have enhanced expressive power for property prediction tasks involving inorganic compounds. We expect that ionic representations, necessary for the description of mixed valence and complex magnetic systems, will support more powerful machine learning models for materials.

The application of deep learning technology in integrated circuit design

This study addresses the intricate challenge of circuit layout optimization central to integrated circuit (IC) design, where the primary goals involve attaining an optimal balance among power consumption, performance metrics, and chip area (collectively known as PPA optimization). The complexity of this task, evolving into a multidimensional problem under multiple constraints, necessitates the exploration of advanced methodologies. In response to these challenges, our research introduces deep learning technology as an innovative strategy to revolutionize circuit layout optimization. Specifically, we employ Convolutional Neural Networks (CNNs) in developing an optimized layout strategy, a performance prediction model, and a system for fault detection and real-time monitoring. These methodologies leverage the capacity of deep learning models to learn from high-dimensional data representations and handle multiple constraints effectively. Extensive case studies and rigorous experimental validations demonstrate the efficacy of our proposed deep learning-driven approaches. The results highlight significant enhancements in optimization efficiency, with an average power consumption reduction of 120% and latency decrease by 1.5%. Furthermore, the predictive capabilities are markedly improved, evidenced by a reduction in the average absolute error for power predictions to 3%. Comparative analyses conclusively illustrate the superiority of deep learning methodologies over conventional techniques across several dimensions. Our findings underscore the potential of deep learning in achieving higher accuracy in predictions, demonstrating stronger generalization abilities, facilitating superior design quality, and ultimately enhancing user satisfaction. These advancements not only validate the applicability of deep learning in IC design optimization but also pave the way for future advancements in addressing the multidimensional challenges inherent to circuit layout optimization.

Spin Space Groups: Full Classification and Applications

In this work, we exhaust all the spin space symmetries, which fully characterize collinear, noncollinear, and commensurate spiral as well as incommensurate spiral magnetism, etc., and investigate enriched features of electronic bands that respect these symmetries. We achieve this by systematically classifying the so-called spin space groups (SSGs)—joint symmetry groups of spatial and spin operations that leave the magnetic structure unchanged. Generally speaking, they are accurate (approximate) symmetries in systems where spin-orbit coupling (SOC) is negligible (finite but weaker than the energy scale of interest), but we also show that specific SSGs could remain valid even in the presence of strong SOC. In recent years, SSGs have played increasingly pivotal roles in various fields such as altermagnetism, topological electronic states, and topological magnon, etc. However, due to its complexity, a complete SSG classification has not been completed up to now. By representing the SSGs as O(N) representations, we—for the first time—obtain the complete classifications of 1421, 9542, and 56 512 distinct SSGs for collinear (N=1), coplanar (N=2), and noncoplanar (N=3) magnetism, respectively. SSG not only fully characterizes the symmetry of spin degrees of freedom, but also gives rise to exotic electronic states, which, in general, form projective representations of magnetic space groups (MSGs). Surprisingly, electronic bands in SSGs exhibit features never seen in MSGs, such as (i) nonsymmorphic SSG Brillouin zone, where SSG operations behave as a glide or screw when acting on momentum, (ii) effective π flux, where translation operators anticommute with each other and yield duplicate bands, (iii) higher-dimensional representations unexplained by MSGs, and (iv) unconventional spin texture on a Fermi surface, which is completely determined by SSGs, independent of Hamiltonian details. To apply our theory, we identify the SSG for each of the 1595 published magnetic structures in the MAGNDATA database on the Bilbao Crystallographic Server. Material examples exhibiting the novel features (i)–(iv) are discussed with emphasis. We also investigate new types of SSG-protected topological electronic states that are unprecedented in MSGs. In particular, we propose a 3D Z2 topological insulator state with a fourfold degenerate Dirac point on the surface and a new scenario of anomalous Z2 helical states that appear on magnetic domain walls. Published by the American Physical Society 2024

ORASIS-MAE Harnesses the Potential of Self-Learning from Partially Annotated Clinical Eye Movement Records

Self-supervised learning (SSL) has gained significant attention in the past decade for its capacity to utilize non-annotated datasets to learn meaningful data representations. In the medical domain, the challenge of constructing large annotated datasets presents a significant limitation, rendering SSL an ideal approach to address this constraint. In this study, we introduce a novel pretext task tailored to stimulus-driven eye movement data, along with a denoising task to improve the robustness against simulated eye tracking failures. Our proposed task aims to capture both the characteristics of the pilot (brain) and the motor (eye) by learning to reconstruct the eye movement position signal using up to 12.5% of the unmasked eye movement signal patches, along with the entire REMOBI target signal. Thus, the encoder learns a high-dimensional representation using a multivariate time series of length 8192 points, corresponding to approximately 40 s. We evaluate the learned representation on screening eight distinct groups of pathologies, including dyslexia, reading disorder, and attention deficit disorder, across four datasets of varying complexity and size. Furthermore, we explore various head architecture designs along with different transfer learning methods, demonstrating promising results with improvements of up to approximately 15%, leading to an overall macro F1 score of 61% and 61.5% on the Saccade and the Vergence datasets, respectively. Notably, our method achieves macro F1 scores of 64.7%, 66.1%, and 61.1% for screening dyslexia, reading disorder, and attention deficit disorder, respectively, on clinical data. These findings underscore the potential of self-learning algorithms in pathology screening, particularly in domains involving complex data such as stimulus-driven eye movement analysis.

High-dimensional cortical signals reveal rich bimodal and working memory-like representations among S1 neuron populations

Complexity is important for flexibility of natural behavior and for the remarkably efficient learning of the brain. Here we assessed the signal complexity among neuron populations in somatosensory cortex (S1). To maximize our chances of capturing population-level signal complexity, we used highly repeatable resolvable visual, tactile, and visuo-tactile inputs and neuronal unit activity recorded at high temporal resolution. We found the state space of the spontaneous activity to be extremely high-dimensional in S1 populations. Their processing of tactile inputs was profoundly modulated by visual inputs and even fine nuances of visual input patterns were separated. Moreover, the dynamic activity states of the S1 neuron population signaled the preceding specific input long after the stimulation had terminated, i.e., resident information that could be a substrate for a working memory. Hence, the recorded high-dimensional representations carried rich multimodal and internal working memory-like signals supporting high complexity in cortical circuitry operation.

A General and Efficient Fault Diagnosis Approach Using Hyperdimensional Computing Enhanced by Local Feature Extraction

Abstract Ensuring the safe and reliable operation of industrial equipment is of great importance, and accordingly, machine fault diagnosis has attracted considerable attention. Recently, various deep learning-based fault diagnosis models have been widely developed due to the powerful ability of feature learning and nonlinear modeling of deep learning models, and these methods heavily rely on large amounts of data and high memory and resources. However, it is difficult to obtain enough data in practical applications, thus leading to the unsatisfactory performance of deep learning-based models and high cost. In this paper, we propose a new and efficient fault diagnosis approach based on Hyperdimensional computing (HDC), which can learn high-dimensional, random, and holographic distributed representations of input features. It has the advantages of single training, low training cost, and does not require a large amount of data. Therefore, we apply HDC to the field of few-shot fault diagnosis for the first time, and propose a simple and efficient approach for few-shot fault diagnosis based on hyperdimensional computing with local feature extraction(L-HDC). Instead of using original data, the model firstly extracts the local features of original data by hand-designed features, and then carries out HDC-encoding, training, and test classification. Through the comparison of various models and the analysis of various data, the effectiveness of our proposed model is verified. In the case of few-shot learning, the recognition accuracy is as high as 90\% and the training time of ordinary HDC model is 5-10 times that of L-HDC model, which is superior to other models.

AI-IMAGE REPRESENTATION AND LINEAR REPRENDER RENDERING

Image representation and rendering have become critical in numerous applications such as virtual reality, medical imaging, and computer graphics. Traditional rendering techniques often face challenges in efficiently handling complex scenes and achieving photorealistic results while maintaining low computational costs. The problem lies in the high-dimensional nature of image data, leading to slow processing times and reduced scalability. This research presents an AI-enhanced technique called Linear RepRender, which leverages deep learning to transform high-dimensional image representations into simplified linear forms for faster rendering. The proposed method employs a combination of convolutional neural networks (CNNs) and linear regression models to reduce image complexity. Specifically, the CNN extracts low-level and high-level features from the image, while the linear regression step approximates the scene’s core visual elements. This hybrid approach significantly improves rendering speed without sacrificing image quality. Furthermore, the method incorporates a loss function optimized for minimizing discrepancies between the rendered and ground truth images. Experimental results demonstrate that Linear RepRender outperforms traditional rendering algorithms, such as ray tracing and rasterization, in terms of computational efficiency and visual accuracy. On a dataset of complex 3D scenes, the proposed method achieved a 35% reduction in rendering time and a 22% improvement in peak signal-to-noise ratio (PSNR) compared to state-of-the-art methods. Additionally, Linear RepRender was able to handle up to 1.5 million polygons per scene with minimal visual artifacts, making it suitable for real-time applications.

DMRC-Net: Dual mechanism residual connection network for point cloud analysis

The purpose of point cloud analysis is to extract effective high-dimensional feature representations from point coordinates. However, existing methods only rely solely on global or local features. In this paper, we introduce the Dual Mechanism Residual Connection Network (DMRC-Net), a new end-to-end network architecture for more comprehensive representations. First, local features are encoded using graph convolution and fine-grained attention mechanisms, whereas global features are encoded using a multi-layer perceptron. Second, adaptive residual learning is designed to integrate two encoding results effectively. Furthermore, the structural information in local space is strengthened by collecting low-level geometric correlations. With these designs, DMRC-Net is able to capture both global and local details comprehensively. Experimental results show that our method performs well in point cloud classification, part segmentation, and semantic segmentation tasks.

Interpretable wind power forecasting combining seasonal-trend representations learning with temporal fusion transformers architecture

Wind power forecasting plays a critical role in the efficient integration of wind energy into power systems. Accurate short-term WPF helps maintain grid stability, optimize power dispatch, and reduce operational costs. However, current deep learning-based WPF architectures rely on mapping spatial–temporal features into a single high-dimensional representation, compromising performance and interpretability. In this work, the interpretable seasonal-trend representation algorithm (ISTR) is proposed to learn a couple of disentangled latent representations that describe seasonal-trend of wind power time series. The ISTR is developed in a self-supervised manner with no need for prior knowledge, providing sufficient interpretable information while avoiding the feature entanglement problem. The ISTR is then integrated into the temporal fusion transformer (TFT) model through seasonal time-point embedding to perform accurate multi-horizon WPF with interpretable meanings. In the case study, ISTR-TFT outperforms the current state-of-the-art methods by achieving an 8.9% improvement in mean absolute percentage error (MAPE) compared to DeepAR and a 15.7% improvement in pinball score. The proposed ISTR-TFT also shows enhanced robustness to noise and missing data scenarios. The interpretability results generated by ISTR-TFT reveal the importance of various input features and the learned temporal patterns, offering valuable insights for decision-making in power system operations.

Rethinking general time series analysis from a frequency domain perspective

Recently, Transformers and MLPs based models have dominated and made significant progress in time series analysis. However, these methods struggle to capture the complete periodic features to model the global dependencies of time series. To this end, we provide a new perspective to explore the potential of time series in the frequency domain. Specifically, we propose the DFT-based Amplitude-Phase Decoupling Network (APDNet), which learns the significance changes and periodicity characteristics of time series by decoupling the real and imaginary parts in the complex frequency domain. First, we propose the Fourier Temporal/Variable Interaction Module to model time dependency and dependency multiple variables, respectively. Secondly, we innovatively propose the Dimensional Expanded Feature Embedding Module (DEFEM), which expands time series into a higher-dimensional representation space to preserve more independent features. Finally, through extensive experiments on multiple real-world datasets, our APDNet achieves state-of-the-art performance in various tasks such as long-term and short-term forecasting, anomaly detection, imputation, and classification. Code is available at: https://github.com/CodeJester196/APDNet.

Multi-Output Prediction Model for Basic Oxygen Furnace Steelmaking Based on the Fusion of Deep Convolution and Attention Mechanisms

The objective of basic oxygen furnace (BOF) steelmaking is to achieve molten steel with final carbon content, temperature, and phosphorus content meeting the requirements. Accurate prediction of the above properties is crucial for end-point control in BOF steelmaking. Traditional prediction models typically use multi-variable input and single-variable output approaches, neglecting the coupling relationships between different property indicators, making it difficult to predict multiple outputs simultaneously. Consequently, a multi-output prediction model based on the fusion of deep convolution and attention mechanism networks (FDCAN) is proposed. The model inputs include scalar data, such as the properties of raw materials and target molten steel, and time series data, such as lance height, oxygen supply intensity, and bottom air supply intensity during the blowing process. The FDCAN model utilizes a fully connected module to extract nonlinear features from scalar data and a deep convolution module to process time series data, capturing high-dimensional feature representations. The attention mechanism then assigns greater weight to significant features. Finally, multiple multi-layer perceptron modules predict the outputs—final carbon content, temperature, and phosphorus content. This structure allows FDCAN to learn complex relationships within the input data and between input and output variables. The effectiveness of the FDCAN model is validated using actual BOF steelmaking data, achieving hit rates of 95.14% for final carbon content within ±0.015 wt%, 84.72% for final temperature within ±15 °C, and 88.89% for final phosphorus content within ±0.005 wt%.

Classification of compressed full-waveform airborne lidar data

ABSTRACT Airborne laser scanners produce 3D data that can be used for a range of applications, such as urban planning, facility monitoring, flood mapping, and forest management. Additional information on the surveyed area can be obtained from the backscattered waveforms recorded by modern light detection and ranging (lidar) sensors. However, the high-dimensional representation of full-waveform data has hindered progress in its use due to difficulties in processing and storage. This paper develops a quantized convolutional autoencoder network to compress lidar waveform data into a condensed feature representation, resulting in a compression rate of up to 20:1. This, together with height information, is fed into a U-net convolutional neural network that achieves an accuracy of 93.7% on six classes.