Linear Embedding Research Articles

An in-depth comparison of linear and non-linear joint embedding methods for bulk and single-cell multi-omics.

Multi-omic analyses are necessary to understand the complex biological processes taking place at the tissue and cell level, but also to make reliable predictions about, for example, disease outcome. Several linear methods exist that create a joint embedding using paired information per sample, but recently there has been a rise in the popularity of neural architectures that embed paired -omics into the same non-linear manifold. This work describes a head-to-head comparison of linear and non-linear joint embedding methods using both bulk and single-cell multi-modal datasets. We found that non-linear methods have a clear advantage with respect to linear ones for missing modality imputation. Performance comparisons in the downstream tasks of survival analysis for bulk tumor data and cell type classification for single-cell data lead to the following insights: First, concatenating the principal components of each modality is a competitive baseline and hard to beat if all modalities are available at test time. However, if we only have one modality available at test time, training a predictive model on the joint space of that modality can lead to performance improvements with respect to just using the unimodal principal components. Second, -omic profiles imputed by neural joint embedding methods are realistic enough to be used by a classifier trained on real data with limited performance drops. Taken together, our comparisons give hints to which joint embedding to use for which downstream task. Overall, product-of-experts performed well in most tasks and was reasonably fast, while early integration (concatenation) of modalities did quite poorly.

Towards a generalized hybrid deep learning model with optimized hyperparameters for malicious traffic detection in the Industrial Internet of Things

Detecting malicious attacks in Industrial Internet of Things (IIoT) is crucial to minimize downtime and financial losses. However, existing deep learning (DL) research faces limitations in reliability and generalization due to reliance on a single dataset. This single dataset approach hinders the effectiveness of the models when applied to new and unseen datasets. This paper addresses the issue using multiple-domain learning, creating a generalized DL framework for IIoT traffic classification. It combines three datasets: Edge-IIoTSet, WUSTL-IIoT-2021, and X-IIoTID. The proposed framework has two stages. The first stage involves the fusion of different datasets into a common feature space. To achieve this, an autoencoder architecture is proposed to match the dimensionality of the datasets into a common feature space. Subsequently, a modified locally linear embedding is used as a manifold alignment to ensure statistical matching of multiple datasets such that the maximum mean discrepancy between multiple datasets is minimized to compare and combine datasets effectively. The second stage presents a hybrid DL model with a convolutional neural network (CNN) and gated recurrent unit (GRU) for IIoT traffic classification. Bayesian optimization fine-tunes hyperparameters. Experiments on combined, single, and cross-datasets show the superiority of the framework. The CNN-GRU model achieved 97.68% accuracy, 97.70% recall, 97.67% precision, and 97.68% F1-score for binary classification on the combined dataset. Transfer learning improved accuracy from 50.95% to 97.80% and F1-score from 48.52% to 97.79% when trained on X-IIoTID and tested on Edge-IIoTSet. Multi-classification accuracy and weighted averages of recall, precision, and F1-score were approximately 98.85%.

From liquidity risk to systemic risk: A use of knowledge graph

In this paper, we use knowledge graph (KG) to study systemic risk in the banking industry. KG provides a graphic representation of the connections of entities of interest (known as vertices or nodes) with the strengths of connections being reflected by the lines connecting them (known as edges) or distances between them. As a result, KG is a natural tool for visualizing the relationships among financial institutions. Furthermore, various data and graph choices can present how differently entities of interest can be connected. In this paper, we draw KGs on two datasets: liquidity index and volatility and three different embedding methods: locally linear embedding, spectral embedding and principal component analysis. Our empirical results show, not surprisingly, that volatility and liquidity index are not similar in explaining how banks are connected. Embedding methods also matter.

Comparative performance analysis of binary variants of FOX optimization algorithm with half-quadratic ensemble ranking method for thyroid cancer detection

Thyroid cancer is a life-threatening condition that arises from the cells of the thyroid gland located in the neck’s frontal region just below the adam’s apple. While it is not as prevalent as other types of cancer, it ranks prominently among the commonly observed cancers affecting the endocrine system. Machine learning has emerged as a valuable medical diagnostics tool specifically for detecting thyroid abnormalities. Feature selection is of vital importance in the field of machine learning as it serves to decrease the data dimensionality and concentrate on the most pertinent features. This process improves model performance, reduces training time, and enhances interpretability. This study examined binary variants of FOX-optimization algorithms for feature selection. The study employed eight transfer functions (S and V shape) to convert the FOX-optimization algorithms into their binary versions. The vision transformer-based pre-trained models (DeiT and Swin Transformer) are used for feature extraction. The extracted features are transformed using locally linear embedding, and binary FOX-optimization algorithms are applied for feature selection in conjunction with the Naïve Bayes classifier. The study utilized two datasets (ultrasound and histopathological) related to thyroid cancer images. The benchmarking is performed using the half-quadratic theory-based ensemble ranking technique. Two TOPSIS-based methods (H-TOPSIS and A-TOPSIS) are employed for initial model ranking, followed by an ensemble technique for final ranking. The problem is treated as multi-objective optimization task with accuracy, F2-score, AUC-ROC and feature space size as optimization goals. The binary FOX-optimization algorithm based on the V_1 transfer function achieved superior performance compared to other variants using both datasets as well as feature extraction techniques. The proposed framework comprised a Swin transformer to extract features, a Fox optimization algorithm with a V1 transfer function for feature selection, and a Naïve Bayes classifier and obtained the best performance for both datasets. The best model achieved an accuracy of 94.75%, an AUC-ROC value of 0.9848, an F2-Score of 0.9365, an inference time of 0.0353 seconds, and selected 5 features for the ultrasound dataset. For the histopathological dataset, the diagnosis model achieved an overall accuracy of 89.71%, an AUC-ROC score of 0.9329, an F2-Score of 0.8760, an inference time of 0.05141 seconds, and selected 12 features. The proposed model achieved results comparable to existing research with small features space.

Feature Extraction from Satellite-Derived Hydroclimate Data: Assessing Impacts on Various Neural Networks for Multi-Step Ahead Streamflow Prediction

Enhancing the generalization capability of time-series models for streamflow prediction using dimensionality reduction (DR) techniques remains a major challenge in water resources management (WRM). In this study, we investigated eight DR techniques and their effectiveness in mitigating the curse of dimensionality, which hinders the performance of machine learning (ML) algorithms in the field of WRM. Our study delves into the most non-linear unsupervised representative DR techniques, including principal component analysis (PCA), kernel PCA (KPCA), multi-dimensional scaling (MDS), isometric mapping (ISOMAP), locally linear embedding (LLE), t-distributed stochastic neighbor embedding (t-SNE), Laplacian eigenmaps (LE), and autoencoder (AE), examining their effectiveness in multi-step ahead (MSA) streamflow prediction. In this study, we conducted a conceptual comparison of these techniques. Subsequently, we focused on their performance in four different case studies in the USA. Moreover, we assessed the quality of the transformed feature spaces in terms of the MSA streamflow prediction improvement. Through our investigation, we gained valuable insights into the performance of different DR techniques within linear/dense/convolutional neural network (CNN)/long short-term memory neural network (LSTM) and autoregressive LSTM (AR-LSTM) architectures. This study contributes to a deeper understanding of suitable feature extraction techniques for enhancing the capabilities of the LSTM model in tackling high-dimensional datasets in the realm of WRM.

Small Object Detection Algorithm for Underwater Organisms based on Improved Transformer

In view of the problem of difficult feature extraction and target missed detection, propose an improved transformer based underwater target detection algorithm. The proposed algorithm improves the linear embedding module of the Transformer to supplement the linear embedding information of small objects and builds the local relationship model of each layer feeling field, in order to achieve end-to-end accurate detection. An experiment on URPU underwater target detection data set is conducted. Compared with Swim Transformer, the proposed algorithm detection accuracy is 77.4%, 1. 5%, detection speed is 7 frame/s, 35.6%, the number is 30.4 MB, and compression is 84.1%. Experimental results have shown that the improved algorithm improves the performance of underwater small target detection

An anti-noise algorithm based on locally linear embedding and weighted XGBoost for fault diagnosis of T/R module

Due to the noise accompanied with fault signals, it is challenging to identify the discriminant information and the local geometric feature from the complex fault data for enhancing fault diagnosis accuracy. To address this challenge, this work proposed an anti-noise algorithm based on locally linear embedding integrated with diffusion distance and maximum correntropy criterion (DMLLE). In DMLLE, diffusion distance was adopted instead of the Euclidean distance for neighborhood construction. Meanwhile, the optimal weights were updated to reveal local geometry information based on the loss function of the maximum correntropy criterion. Subsequently, DMLLE is eventually developed to further restrain noise embedding into raw signals and obtain low dimensional features. Furthermore, weighted extreme gradient boosting is used to map the low dimensional features to the types of faults, which easily implements fault pattern recognition. Finally, two synthetic manifold datasets and fault data acquired from the transmit/receive (T/R) module are used to validate the performance of the proposed diagnosis methodology. Compared with the existing methods, the proposed diagnosis methodology generates a smoother flow structure by preserving the local neighborhood of the dataset with noises and realizes a higher accuracy of 94.41% on the T/R module dataset, which outperforms 3%–9% better than other classification models. Therefore, it can be concluded that the proposed diagnosis methodology can effectively extract intrinsic fault features by weakening the influence aroused by noise and achieve superior accuracy in fault diagnosis by addressing the problem of small sample size.

Research on intelligent clearing of weeds in wheat fields using spectral imaging and machine learning

The intelligent management of weed clearing can improve wheat yield and reduce the use of pesticides. In this paper, our contribution is to propose a method for wheat and weed recognition based on multispectral imaging and Membrane Search Algorithm Support Vector Machine (MSA-SVM), which is the basis for intelligent weed clearing. A spectral data acquisition system is set up in the laboratory. Then, a total of 700 groups of wheat and weed spectral data are collected. The classical SVM based on spectral data is applied to identify weeds to study the classification performance of wheat and weeds in different bands. The Membrane Search Algorithm (MSA) algorithm proposed by us was used for SVM parameters optimization, and the dimension reduction algorithm was adopted to reduce the interference of redundant information. After local linear embedding (LLE) dimensionality reduction of spectral data, MSA-SVM has the highest average recognition accuracy for weeds and wheat, which is 91.00%. The experimental results show that our research can distinguish wheat from weeds, which is of great significance for the development of smart agriculture.

Driving style-aware energy management for battery/supercapacitor electric vehicles using deep reinforcement learning

Driving style can significantly affect the energy consumption, battery lifespan, and driving economy of electric vehicles. In this context, this paper proposes a novel driving style-aware energy management strategy for electric vehicles with battery/supercapacitor hybrid energy storage systems based on deep reinforcement learning. Firstly, a semi-supervised support vector machine-based driving style recognition method is presented to recognize the driving style, where twenty features are extracted from limited labeled velocity/acceleration data and then reduced to six dimensions by locally linear embedding. The six dimension features are used to obtain accurate recognition results. Then a proximal policy optimization-based energy management strategy is proposed with the driving style as an additional input state, to optimize the power allocation and minimize the battery capacity loss cost. Extensive results illustrate the effectiveness of the proposed methods, e.g., the proposed driving style recognition method can recognize the real-time driving style with an accuracy of over 95%. Taking the recognized style as input, the proposed driving style-aware energy management strategy can reduce the battery capacity loss cost by 3.30–4.19% and 1.77–8.15%, compared with no driving style and incorrect driving style input energy management methods, respectively.

A novel untrained SSVEP-EEG feature enhancement method using canonical correlation analysis and underdamped second-order stochastic resonance.

Compared with the light-flashing paradigm, the ring-shaped motion checkerboard patterns avoid uncomfortable flicker or brightness modulation, improving the practical interactivity of brain-computer interface (BCI) applications. However, due to fewer harmonic responses and more concentrated frequency energy elicited by the ring-shaped checkerboard patterns, the mainstream untrained algorithms such as canonical correlation analysis (CCA) and filter bank canonical correlation analysis (FBCCA) methods have poor recognition performance and low information transmission rate (ITR). To address this issue, a novel untrained SSVEP-EEG feature enhancement method using CCA and underdamped second-order stochastic resonance (USSR) is proposed to extract electroencephalogram (EEG) features. In contrast to typical unsupervised dimensionality reduction methods such as common average reference (CAR), principal component analysis (PCA), multidimensional scaling (MDS), and locally linear embedding (LLE), CCA exhibits higher adaptability for SSVEP rhythm components. This study recruits 42 subjects to evaluate the proposed method and experimental results show that the untrained method can achieve higher detection accuracy and robustness. This untrained method provides the possibility of applying a nonlinear model from one-dimensional signals to multi-dimensional signals.

Diabetic Retinopathy Image Classification Using Shift Window Transformer

Diabetic retinopathy is one of the most dangerous complications for diabetic patients, leading to blindness if not diagnosed early. However, early diagnosis can control and prevent the disease from progressing to blindness. Transformers are considered state-of-the-art models in natural language processing that do not use convolutional layers. In transformers, means of multi-head attention mechanisms capture long-range contextual relations between pixels. For grading diabetic retinopathy, CNNs currently dominate deep learning solutions. However, the benefits of transformers, have led us to propose an appropriate transformer-based method to recognize diabetic retinopathy grades. A major objective of this research is to demonstrate that the pure attention mechanism can be used to determine diabetic retinopathy and that transformers can replace standard CNNs in identifying the degrees of diabetic retinopathy. In this study, a Swin Transformer-based technique for diagnosing diabetic retinopathy is presented by dividing fundus images into nonoverlapping batches, flattening them, and maintaining positional information using a linear and positional embedding procedure. Several multi-headed attention layers are fed into the resulting sequence to construct the final representation. In the classification step, the initial token sequence is passed into the SoftMax classification layer, which produces the recognition output. This work introduced the Swin transformer performance on the APTOS 2019 Kaggle for training and testing using fundus images of different resolutions and patches. The test accuracy, test loss, and test top 2 accuracies were 69.44%, 1.13, and 78.33%, respectively for 160*160 image size, patch size=2, and embedding dimension C=64. While the test accuracy was 68.85%, test loss: 1.12, and test top 2 accuracy: 79.96% when the patch size=4, and embedding dimension C=96. And when the size image is 224*224, patch size=2, and embedding dimension C=64, the test accuracy: 72.5%, test loss: 1.07, and test top 2 accuracy: 83.7%. When the patch size =4, embedding dimension C=96, the test accuracy was 74.51%, test loss: 1.02, and the test top 2 accuracy was 85.3%. The results showed that the Swin Transformer can achieve flexible memory savings. The proposed method highlights that an attention mechanism based on the Swin Transformer model is promising for the diabetic retinopathy grade recognition task.

Rigidity of proper holomorphic mappings between certain nonequidimensional unbounded non-hyperbolic domains

The Fock-Bargmann-Hartogs domain Dn,m(μ)(μ>0) in Cn+m is defined by the inequality ‖w‖2<e−μ‖z‖2, where (z,w)∈Cn×Cm, which is an unbounded non-hyperbolic domain in Cn+m. In this article, we prove that any proper holomorphic mapping from Dn,m(μ) to Dn,m+1(μ)(m≥3) is necessarily the standard linear embedding up to their automorphisms and therefore is a totally geodesic isometric embedding with respect to their Bergman metrics.

Swin transformer with multiscale 3D atrous convolution for hyperspectral image classification

Hyperspectral image (HSI) classification has attracted significant interest among researchers owing to its diverse practical applications. Convolutional neural networks (CNNs) have been extensively utilized for HSI classification. However, the effectiveness of CNN-based approaches is constrained by the fixed size and structure of the convolutional kernels, as well as their incapacity to capture global features. Moreover, these networks are inadequate in modeling the sequential characteristics of data. Recently, a promising approach, window-based multi-head self-attention has emerged to address the limitations of CNNs and incorporate efficient sequence modeling capabilities. This paper introduces a novel method, multiscale 3D atrous convolution with a lightweight swin transformer (MACLST), that effectively combines the strengths of two networks to capture both local and global features at different scales in HSI classification. The MACLST is designed to process HSI cubes as input and employs a spectral–spatial features extraction module based on multiscale 3D atrous convolution. This module involves parallel branches of 3D layers with varying atrous rates, enabling the extraction of features at multiple scales and resolutions. The extracted spectral–spatial features are fused and passed to the lightweight Swin transformer module as linear embeddings. This module captures long-range dependencies and learns effective feature representations of HSI. To reduce computational complexity, the swin transformer module is simplified and consists of only two stages, offering a more efficient version of the original swin transformer. The proposed MACLST model is extensively evaluated on five widely used benchmark HSI datasets, and the experimental results validate its superiority over state-of-the-art approaches with an overall accuracy of 99.00%, 99.59%, 99.95%, 98.71%, and 94.98% on the Indian Pines, University of Pavia, Salinas Valley, Houston University 2013, and Houston University 2018 datasets, respectively.

Extraction and Joint Method of PV–Load Typical Scenes Considering Temporal and Spatial Distribution Characteristics

Regarding the generation and integration of typical scenes of PV and loads in urban photovoltaic distribution networks, as well as the insufficient consideration of the spatiotemporal correlation between PV and loads, this paper proposes a typical scene extraction method based on local linear embedding, kernel density estimation, and a joint PV–load typical scene extraction method based on the FP-growth algorithm. Firstly, the daily operation matrices of PV and load are constructed by using the historical operation data of PV and load. Then, the typical scenes are extracted by the dimensionality reduction of local linear embedding and the kernel density estimation method. Finally, the strong association rules of PV–meteorological conditions and load–meteorological conditions are mined based on the FP-growth algorithm, respectively. The association of PV–load typical daily operation scenarios is completed using meteorological conditions as a link. This experiment involved one year of operation data of a distribution network containing PV in Qingyuan, Guangdong Province. The typical scene extraction joint method, Latin hypercube sampling method, and k-means clustering-based scene generation method proposed in this paper are used for comparison, respectively. The results show that compared to the other two scenario generation methods, the error between the typical scenario obtained by this method and the actual operating scenario of the distribution network is smaller. The extracted typical PV and load scenarios can better fit the actual PV and load operation scenarios, which have more reference value for the operation planning of actual distribution networks containing PV.

Data-Driven Linear Koopman Embedding for Networked Systems: Model-Predictive Grid Control

This article presents a data-learned linear Koopman embedding of nonlinear networked dynamics and uses it to enable real-time model predictive emergency voltage control in a power grid. The approach involves a novel data-driven “basis-dictionary free” lifting of the system dynamics into a higher dimensional linear space over which a model predictive control is exercised, making it both scalable and rapid for practical real-time implementation. A <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Koopman-inspired deep neural network</i> (KDNN) encoder–decoder architecture for the linear embedding of the underlying networked dynamics under distributive control is presented, in which the end-to-end components of the KDNN, comprising of a triple of transforms is learned from the system trajectory data in one go: a neural network (NN)-based lifting to a higher dimension, a linear dynamics within that higher dimension, and an NN-based projection to the original space. This data-learned approach relieves the burden of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ad hoc</i> selection of the nonlinear basis functions (e.g., radial or polynomial) used in conventional approaches for lifting to a higher dimensional linear space. We validate the efficacy and robustness of the approach via application to the standard IEEE 39-bus system.

Gradient-Enhanced Kriging for High-Dimensional Bayesian Optimization with Linear Embedding

This paper explores the application of gradient-enhanced (GE) kriging for Bayesian optimization (BO) problems with a high-dimensional parameter space. We utilize the active subspace method to embed the original parameter space in a low-dimensional subspace. The active subspace is detected by analyzing the spectrum of the empirical second-moment matrix of the gradients of the response function. By mapping the training data onto their respective subspace, the objective function and the constraint functions are efficiently approximated with low-dimensional GE-kriging models. In each cycle of the BO procedure, a new point is found by maximizing the constrained expected improvement function within a low-dimensional polytope, and it is mapped back to the original space for model evaluation. In this way, the computational costs are significantly reduced when compared with standard GE-kriging. We illustrate and assess the proposed approach with numerical experiments ranging from academic benchmark problems to aerodynamic engineering applications. The experimental results show that the proposed method is promising for optimizing high-dimensional expensive objective functions, especially for problems that exhibit a clear low-dimensional active subspace.

State-of-health estimation for lithium-ion batteries based on Bi-LSTM-AM and LLE feature extraction

With the increasing demands for battery safety management, data-driven method becomes a promising solution for highly accurate battery state of health (SOH) estimation. However, the data-driven method faces problems of poor interpretability and high dependence on input features. This paper proposes a SOH estimation method that integrates data-driven model and signal analysis method. Specifically, the differential thermal voltammetry (DTV) analysis method is used to analyze aging characteristics to obtain features strongly related to battery aging and solve the problem of poor interpretability of data-driven methods. The use of local linear embedding method (LLE) for feature extraction has improved model efficiency. A data-driven model is constructed with the Bi-directional long short-term memory (Bi-LSTM) as the core, and the attention mechanism (AM) is added to focus on important parts of the sequence to further improve the accuracy of the model. The proposed method is validated based on the Oxford battery degradation dataset, and the results show that the proposed method achieves high accuracy and strong robustness in SOH estimation with a root mean square error (RMSE) maintained at about 0.4%. This method has the potential to be employed on cloud platforms or end-cloud collaboration systems for online implementation.

Deep multi-view spectral clustering via ensemble

Graph-based methods have achieved great success in multi-view clustering. However, existing graph-based models generally utilize shallow and linear embedding functions to obtain the common spectral embedding for clustering assignments. In addition, the fusion similarity graphs from multiple views are generally obtained by a simple weighted-sum rule. To this end, we propose a novel deep multi-view spectral clustering via ensemble model (DMCE), which applies ensemble clustering to fuse the similarity graphs from different views. On this basis, we employ the graph auto-encoder to learn the common spectral embedding, which can be regarded as the indicator matrix directly. Moreover, a unified optimization framework is designed to update the variables in the proposed DMCE, which consists of graph reconstruction loss, orthogonal loss, and graph contrastive learning loss. Extensive experiments on six real-world benchmark datasets have demonstrated the effectiveness of our model compared with the state-of-the-art multi-view clustering methods.

A normalized differential sequence feature encoding method based on amino acid sequences.

Protein interactions are the foundation of all metabolic activities of cells, such as apoptosis, the immune response, and metabolic pathways. In order to optimize the performance of protein interaction prediction, a coding method based on normalized difference sequence characteristics (NDSF) of amino acid sequences is proposed. By using the positional relationships between amino acids in the sequences and the correlation characteristics between sequence pairs, NDSF is jointly encoded. Using principal component analysis (PCA) and local linear embedding (LLE) dimensionality reduction methods, the coded 174-dimensional human protein sequence vector is extracted using sequence features. This study compares the classification performance of four ensemble learning methods (AdaBoost, Extra trees, LightGBM, XGBoost) applied to PCA and LLE features. Cross-validation and grid search methods are used to find the best combination of parameters. The results show that the accuracy of NDSF is generally higher than that of the sequence matrix-based coding method (MOS) coding method, and the loss and coding time can be greatly reduced. The bar chart of feature extraction shows that the classification accuracy is significantly higher when using the linear dimensionality reduction method, PCA, compared to the nonlinear dimensionality reduction method, LLE. After classification with XGBoost, the model accuracy reaches 99.2%, which provides the best performance among all models. This study suggests that NDSF combined with PCA and XGBoost may be an effective strategy for classifying different human protein interactions.

Robust locally linear embedding and its application in analogue circuit fault diagnosis

During long-term operation of analogue circuits, fault diagnosis is important for preventing the occurrence of hazards. However, noise often accompanies sampled signals and makes the task of fault diagnosis more difficult. Therefore, developing a robust feature extraction technique is an indispensable part of fault diagnosis. The locally linear embedding (LLE) algorithm has recently emerged as a promising technique for dimensional reduction and feature extraction because it preserves linear neighborhoods, and it is quite effective when there is a locally linear dependent structure embedded in fault data. However, LLE is sensitive to noise. Therefore, the maximum correntropy criterion is adopted to resist non-Gaussian noise by seeking the optimal weight coefficient, and a half-quadratic optimization procedure is introduced to address the objective function. Moreover, softmax regression is applied to locate faults. Finally, two typical analogue circuit systems are used to demonstrate the robustness of the modified algorithm to non-Gaussian noise. The experimental results show that the robust LLE algorithm can outperform LLE in the extraction of fault features when there is non-Gaussian noise in the fault signals, and the proposed fault diagnosis method has a better effect in locating faults compared with other feature extraction methods.