t-SNE Visualization Research Articles

How Resilient Are Kolmogorov–Arnold Networks in Classification Tasks? A Robustness Investigation

Kolmogorov–Arnold Networks (KANs) are a novel class of neural network architectures based on the Kolmogorov–Arnold representation theorem, which has demonstrated potential advantages in accuracy and interpretability over Multilayer Perceptron (MLP) models. This paper comprehensively evaluates the robustness of various KAN architectures—including KAN, KAN-Mixer, KANConv_KAN, and KANConv_MLP—against adversarial attacks, which constitute a critical aspect that has been underexplored in current research. We compare these models with MLP-based architectures such as MLP, MLP-Mixer, and ConvNet_MLP across three traffic sign classification datasets: GTSRB, BTSD, and CTSD. The models were subjected to various adversarial attacks (FGSM, PGD, CW, and BIM) with varying perturbation levels and were trained under different strategies, including standard training, adversarial training, and Randomized Smoothing. Our experimental results demonstrate that KAN-based models, particularly the KAN-Mixer, exhibit superior robustness to adversarial attacks compared to their MLP counterparts. Specifically, the KAN-Mixer consistently achieved lower Success Attack Rates (SARs) and Degrees of Change (DoCs) across most attack types and datasets while maintaining high accuracy on clean data. For instance, under FGSM attacks with ϵ=0.01, the KAN-Mixer outperformed the MLP-Mixer by maintaining higher accuracy and lower SARs. Adversarial training and Randomized Smoothing further enhanced the robustness of KAN-based models, with t-SNE visualizations revealing more stable latent space representations under adversarial perturbations. These findings underscore the potential of KAN architectures to improve neural network security and reliability in adversarial settings.

MSBERT: Embedding Tandem Mass Spectra into Chemically Rational Space by Mask Learning and Contrastive Learning.

Tandem mass spectrometry (MS/MS) is a powerful technique for chemical analysis in many areas of science. The vast MS/MS spectral data generated in liquid chromatography-mass spectrometry (LC-MS) experiments require efficient analysis and interpretation methods for the following compound identification. In this study, we propose MSBERT based on self-supervised learning strategies to embed MS/MS spectra into reasonable embeddings for efficient compound identification. It adopts the transformer encoder as the backbone for mask learning and uses the same spectra with different masks for contrastive learning. MSBERT is trained on the GNPS data set and tested on the GNPS data set, the MoNA data set, and the MTBLS1572 data set. It exhibits enhanced library matching and analogous compound searching capabilities compared to existing methods. The recalls at 1, 5, and 10 on a GNPS test subset with structures not in the training set are 0.7871, 0.8950, and 0.9080, respectively. The results are better than those of Spec2Vec with 0.6898, 0.8276, and 0.8620, and DreaMS with 0.7158, 0.8327, and 0.8635. The rationality of embeddings is demonstrated by t-SNE visualization, structural similarity, spectra clustering, compound identification, and analogous compound searching. A user-friendly web server is provided for efficient spectral analysis, and the source code for MSBERT is available at https://github.com/zhanghailiangcsu/MSBERT.

A Machine Learning-Based Investigation of Integrin Expression Patterns in Cancer and Metastasis.

Integrins, a family of transmembrane receptor proteins, play complex roles in cancer development and metastasis. These roles could be better delineated through machine learning of transcriptomic data to reveal relationships between integrin expression patterns and cancer. We collected publicly available RNA-Seq integrin expression from 8 healthy tissues and their corresponding tumors, along with data from metastatic breast cancer. We then used machine learning methods, including t-SNE visualization and Random Forest classification, to investigate changes in integrin expression patterns. Integrin expression varied across tissues and cancers, and between healthy and cancer samples from the same tissue, enabling the creation of models that classify samples by tissue or disease status. The integrins whose expression was important to these classifiers were identified. For example, ITGA7 was key to classification of breast samples by disease status. Analysis in breast tissue revealed that cancer rewires co-expression for most integrins, but the co-expression relationships of some integrins remain unchanged in healthy and cancer samples. Integrin expression in primary breast tumors differed from their metastases, with liver metastasis notably having reduced expression. Integrin expression patterns vary widely across tissues and are greatly impacted by cancer. Machine learning of these patterns can effectively distinguish samples by tissue or disease status.

Identification of Patterns in CO2 Emissions among 208 Countries: K-Means Clustering Combined with PCA and Non-Linear t-SNE Visualization

Identification of Patterns in CO2 Emissions among 208 Countries: K-Means Clustering Combined with PCA and Non-Linear t-SNE Visualization

Is it possible to find the single nearest neighbor of a query in high dimensions?

We investigate an open question in the study of the curse of dimensionality: Is it possible to find the single nearest neighbor of a query in high dimensions? Using the notion of (in)distinguishability to examine whether the feature map of a kernel is able to distinguish two distinct points in high dimensions, we analyze this ability of a metric-based Lipschitz continuous kernel as well as that of the recently introduced Isolation Kernel. Between the two kernels, we show that only Isolation Kernel has distinguishability and it performs consistently well in four tasks: indexed search for exact nearest neighbor search, anomaly detection using kernel density estimation, t-SNE visualization and SVM classification in both low and high dimensions, compared with distance, Gaussian and three other existing kernels.

Sequential action-induced invariant representation for reinforcement learning

How to accurately learn task-relevant state representations from high-dimensional observations with visual distractions is a realistic and challenging problem in visual reinforcement learning. Recently, unsupervised representation learning methods based on bisimulation metrics, contrast, prediction, and reconstruction have shown the ability for task-relevant information extraction. However, due to the lack of appropriate mechanisms for the extraction of task information in the prediction, contrast, and reconstruction-related approaches and the limitations of bisimulation-related methods in domains with sparse rewards, it is still difficult for these methods to be effectively extended to environments with distractions. To alleviate these problems, in the paper, the action sequences, which contain task-intensive signals, are incorporated into representation learning. Specifically, we propose a Sequential Action–induced invariant Representation (SAR) method, which decouples the controlled part (i.e., task-relevant information) and the uncontrolled part (i.e., task-irrelevant information) in noisy observations through sequential actions, thereby extracting effective representations related to decision tasks. To achieve it, the characteristic function of the action sequence’s probability distribution is modeled to specifically optimize the state encoder. We conduct extensive experiments on the distracting DeepMind Control suite while achieving the best performance over strong baselines. We also demonstrate the effectiveness of our method at disregarding task-irrelevant information by applying SAR to real-world CARLA-based autonomous driving with natural distractions. Finally, we provide the analysis results of generalization drawn from the generalization decay and t-SNE visualization. Code and demo videos are available at https://github.com/DMU-XMU/SAR.git.

Domain Adaptation from Drilling to Geophysical Data for Mineral Exploration

This study utilizes domain adaptation to enhance the integration of diverse geoscience datasets, aiming to improve the identification of ore bodies. Traditional mineral exploration methods often face challenges in merging different geoscience data types, which leads to models that do not perform well across varying domains. Domain adaptation is a deep learning strategy aimed at adapting a model developed in one domain (source) to perform well in a different domain (target). To adapt models trained on detailed, labeled drilling data (source) to interpret broader, unlabeled geophysical data (target), Domain-Adversarial Neural Networks (DANNs) were applied, chosen for their robust performance in scenarios where the target domain does not provide labels. This approach was indirectly validated through the minimal overlap between regions identified as candidate ore and borehole locations marked as host rocks, with qualitative validation provided by t-Distributed Stochastic Neighbor Embedding (t-SNE) visualizations showing improved data integration across domains.

An innovative approach to design readily synthesizable polymers for all-polymer solar cells

The communalization of all-polymer solar cells depends on the cost of active layer materials. We have introduced a framework to find the easily synthesizable polymers. Breaking Retrosynthetically Interesting Chemical Substructures (BRICS) method is used to generate a large database of polymers and synthetic accessibility of generated polymers is predicted. The generated database is visualized using various methods. Energy levels of polymers are also predicted using pretrained machine learning models. Polymers are screened on the basis of predicted properties. Library of polymers are displayed using the T-distributed Stochastic Neighbor Embedding (t-SNE) visualization. Structure Activity Landscape Index (SALI) visualization is also used. A significant change is observed in synthetic accessibility score on structural changes. The histagradient boosting regressor is used to predict the energy levels of polymers that energy levels play significant role in the selection of materials for organic photovoltaic cells. Synthetic accessibility of polymers is analyzed and a significant number of polymers are easy to synthesize. Thirty polymers are selected through screening process that are potential candidates for organic photovoltaic cells.

Utilizing excitation-emission matrix fluorescence spectroscopy and convolutional neural networks for dark tea brand and aging period identification

This study utilized excitation-emission matrix (EEM) fluorescence spectroscopy in conjunction with convolutional neural networks (CNNs) to identify the dark tea brands and their aging periods, which were both key factors of market value. The adaptation of EEM data to the CNN model was examined, encompassing the configuration of the data structure and the implementation of data augmentation. Three typical CNN models including AlexNet, GoogLeNet, and ResNet were evaluated. ResNet was selected because it demonstrated the highest classification accuracy on the test set and superior clustering performance in the t-SNE visualization of extracted features. Analysis of feature contribution suggested that the CNN model exhibited enhanced sensitivity towards the boundaries of the fluorescence region. Furthermore, transfer learning was proved to be effective in facilitating adaptation of dark tea recognition models to databases with different sample sizes, which was meaningful for the application of this method in dark tea market supervision. Finally, the multi-task model was developed and achieved a 96.9 % accuracy in brand identification and 87.5 % in aging period determination, suggesting the advantages of multi-task learning in enhancing model generalization and training efficiency.

Class-incremental learning with Balanced Embedding Discrimination Maximization

Class incremental learning is committed to solving representation learning and classification assignments while avoiding catastrophic forgetting in scenarios where categories are increasing. In this work, a unified method named Balanced Embedding Discrimination Maximization (BEDM) is developed to make the intermediate embedding more distinctive. Specifically, we utilize an orthogonality constraint based on doubly-blocked Toeplitz matrix to minimize the correlation of convolution kernels, and an algorithm for similarity visualization is introduced. Furthermore, uneven samples and distribution shift among old and new tasks eventuate strongly biased classifiers. To mitigate the imbalance, we propose an adaptive balance weighting in softmax to compensate insufficient categories dynamically. In addition, hybrid embedding learning is introduced to preserve knowledge from old models, which involves less hyper-parameters than conventional knowledge distillation. Our proposed method outperforms the existing approaches on three mainstream benchmark datasets. Moreover, we technically visualize that our method can produce a more uniform similarity histogram and more stable spectrum. Grad-CAM and t-SNE visualizations further confirm its effectiveness. Code is available at https://github.com/wqzh/BEDM.

Exploring the chemical space of ionic liquids for CO2 dissolution through generative machine learning models

For discovering uncharted chemical space of ionic liquids (ILs) for CO2 dissolution, a reliable generative framework combining re-balanced variational autoencoder (VAE), artificial neural network (ANN), and particle swarm optimization (PSO) is developed based on a comprehensive experimental solubility database from literature. The re-balanced VAE transforms the chemical space of ILs into continuous latent space, which is demonstrated by t-distributed stochastic neighbor embedding (t-SNE) visualization and sampled ions of the latent space. ANN is connected with the re-balanced VAE to predict the CO2 solubility and the resultant VAE-ANN model achieves a low mean absolute error (MAE) of 0.022 on the test set. Lastly, the PSO algorithm is employed to search the latent space for optimal IL structures with the highest predicted solubility. A total of 5120 ILs are generated and optimized through 10 parallel runs of PSO. Their CO2 solubilities are predicted and compared to those of the 3735 ILs combined with the already-known cations and anions in the CO2 solubility database under 298.15 K and 100 kPa. The results demonstrate a notably larger distribution of higher CO2 solubility in optimized ILs after PSO, which effectively points out the significance and directions for exploring the wide IL chemical space.

Research on intelligent classification of weld defects based on improved ResNeXt network

Welding technology is widely used in the connection of metal workpieces, and then due to improper operation and other reasons, small defects may occur in the welding, which will lead to welding failure and hazardous accidents. Therefore, it is very important to accurately identify welding defects. The automatic defect identification method based on deep learning learns features directly from the full-focus mapping of weld defects and automatically classifies them; the ResNeXt network is selected for improvement, and the SK convolution unit is introduced into the original ResNeXt-50 network model to enhance the adaptive adjustment of sensory field; in order to alleviate the problem of the point-by-point convolution in the ResNeXt-50 which consumes a large amount of In order to alleviate the problem that point-by-point convolution in ResNeXt-50 consumes a large amount of computation, 1×1 group convolution is used instead of the original point-by-point convolution, and channel rearrangement is introduced to alleviate the problem of weakened information exchange of groups caused by group convolution. After improvement, the ResNeXt-50 network model can achieve a classification accuracy of 98.2%, which is 5.5% more than that of the original ResNeXt-50 on this paper's dataset, and the classification accuracy after T-sne visualization also has obvious separation boundary and clustering effects.

Fast and efficient context-aware embedding generation using fuzzy hashing for in-vehicle network intrusion detection

In the rapidly advancing field of automotive cybersecurity, the protection of In-Vehicle Networks (IVNs) against cyber threats is crucial. Current deep learning solutions offer robustness but at the cost of high computational demand and potential privacy breaches due to the extensive IVN data required for model training. Our study proposes a novel intrusion detection system (IDS) specifically designed for IVNs that prioritizes computational efficiency and data privacy. Utilizing fuzzy hashing techniques, we generate context-aware embeddings that effectively preserve the privacy of IVN data. Among the machine learning algorithms evaluated, the Support Vector Machine (SVM) emerged as the most effective, particularly when paired with TLSH hash embeddings. This combination achieved notable detection performance, as substantiated by T-SNE visualizations that demonstrate a distinct segregation of normal and attack traffic within the vector space. To validate the effectiveness and practicality of our proposed IDS, we conducted exhaustive experiments on the well-known car-hacking dataset and the more complex ROAD dataset, which includes diverse and sophisticated attack scenarios. Our findings reveal that the proposed lightweight IDS not only demonstrates high detection accuracy but also maintains this performance within the computational constraints of current IVN systems. The system's capability to operate effectively in real-time environments makes it a viable solution for modern automotive cybersecurity needs.

Electroencephalogram-Based ConvMixer Architecture for Recognizing Attention Deficit Hyperactivity Disorder in Children.

Attention deficit hyperactivity disorder (ADHD) is a neuro-developmental disorder that affects approximately 5-10% of school-aged children worldwide. Early diagnosis and intervention are essential to improve the quality of life of patients and their families. In this study, we propose ConvMixer-ECA, a novel deep learning architecture that combines ConvMixer with efficient channel attention (ECA) blocks for the accurate diagnosis of ADHD using electroencephalogram (EEG) signals. The model was trained and evaluated using EEG recordings from 60 healthy children and 61 children with ADHD. A series of experiments were conducted to evaluate the performance of the ConvMixer-ECA. The results showed that the ConvMixer-ECA performed well in ADHD recognition with 94.52% accuracy. The incorporation of attentional mechanisms, in particular ECA, improved the performance of ConvMixer; it outperformed other attention-based variants. In addition, ConvMixer-ECA outperformed state-of-the-art deep learning models including EEGNet, CNN, RNN, LSTM, and GRU. t-SNE visualization of the output of this model layer validated the effectiveness of ConvMixer-ECA in capturing the underlying patterns and features that separate ADHD from typically developing individuals through hierarchical feature learning. These outcomes demonstrate the potential of ConvMixer-ECA as a valuable tool to assist clinicians in the early diagnosis and intervention of ADHD in children.

Cluster Analysis of MSMES In Suleja, Nigeria: Insights From Fuzzy C-Means Clustering And T-SNE Visualizations

This study employs cluster analysis to segment Micro, Small, and Medium-sized Enterprises (MSMEs) in Suleja, Nigeria into distinct groups based on their financial and operational characteristics. We utilize Fuzzy C-Means Clustering and t-Distributed Stochastic Neighbor Embedding (t-SNE) visualizations to uncover hidden patterns and structures in the data. Our results reveal three distinct clusters, explaining 40.3% of the variance in the data (R² = 0.403). The clusters exhibit reasonable separation (Silhouette coefficient = 0.360) and distinct patterns in their variable means. Cluster 1 comprises high-performing MSMEs with high levels of engagement, participation, and satisfaction. Cluster 2 consists of low-performing MSMEs with low levels of engagement, participation, and satisfaction. Cluster 3 represents moderate-performing MSMEs with balanced levels of engagement, participation, and satisfaction. t-SNE visualizations confirm the clustering structure, revealing three distinct groups with varying densities and separation. The visualizations also highlight the relationships between the clusters, with Cluster 1 and Cluster 3 showing higher similarity than Cluster 2. Model performance metrics indicate a reasonable accuracy (Pearson's γ = 0.440) and a good balance between within-cluster sum of squares and between-cluster sum of squares (Calinski-Harabasz index = 13.087). The findings provide valuable insights for policymakers and practitioners seeking to support MSMEs in Nigeria. The results suggest that targeted interventions and support programs should focus on enhancing engagement, participation, and satisfaction among MSMEs, particularly for those in the low-performing cluster. The study contributes to the literature on MSMEs and cluster analysis, demonstrating the effectiveness of Fuzzy C-Means Clustering and t-SNE visualizations in uncovering meaningful patterns in MSMEs data.

Semi-supervised learning model synergistically utilizing labeled and unlabeled data for failure detection in optical networks

In optical networks, reliable failure detection is essential for maintaining quality of service. The methodology has evolved from traditional performance threshold-driven approaches to contemporary data-driven AI algorithms, predominantly employing supervised and unsupervised learning. However, with the advent of second-level telemetry, optical transport networks have amassed a wealth of unlabeled performance data, while labeled data remains limited due to the intensive effort required for annotation. In this scenario, to address the challenges of scarce labeled data in supervised learning and the accuracy issues in unsupervised methods, we propose an OpenFE-VIME semi-supervised model. This model synergizes the robustness of supervised approaches with the flexibility of unsupervised approaches. It not only leverages the abundant reservoir of unlabeled data but also addresses the challenges posed by the limited availability of labeled data, enabling reliable and efficient failure detection. Upon evaluation using performance data from OTN node devices in the operator’s optical backbone network, the OpenFE-VIME model demonstrates remarkable performance, achieving an F1-score of 0.947 and accuracy of 0.946, while significantly reducing false negative and false positive rates to 0.073 and 0.035, respectively. Moreover, our research explores the model’s capabilities in utilizing both labeled and unlabeled data and investigates the threshold for training convergence across various data ratios. Additionally, the model’s internal mechanisms and decision-making processes are interpreted using t-SNE visualization, offering enhanced insights into its operational efficacy.

Assessment of TGx-DDI genes for genotoxicity in a comprehensive panel of chemicals

Background The TGx-DDI biomarker identifies transcripts specifically induced by primary DNA damage. Profiling similarity of TGx-DDI signatures can allow clustering compounds by genotoxic mechanism. This transcriptomics-based approach complements conventional toxicology testing by enhancing mechanistic resolution. Methods Unsupervised hierarchical clustering and t-distributed stochastic neighbor embedding (tSNE) were utilized to assess similarity of publicly-available per- and polyfluoroalkyl substances (PFAS) and ToxCast chemicals based on TGx-DDI modulation. TempO-seq transcriptomic data after highest chemical concentrations were analyzed. Results Clustering discriminated between genotoxic and non-genotoxic compounds while drawing similarity among chemicals with shared mechanisms. PFAS largely clustered distinctly from classical mutagens. However, dynamic range across PFAS types and durations indicated variable potential for DNA damage. tSNE visualization reinforced phenotypic groupings, with genotoxins clustering separately from non-DNA damaging agents. Discussion Unsupervised learning approaches applied to TGx-DDI profiles effectively categorizes chemical genotoxicity potential, aiding elucidation of biological response pathways. This transcriptomics-based strategy gives further insight into the role and effect of individual TGx-DDI biomarker genes and complements existing assays by enhancing mechanistic resolution. Overall, TGx-DDI biomarker profiling holds promise for predictive safety screening.

SK-MMFMNet: A multi-dimensional fusion network of remote sensing images and EEG signals for multi-scale marine target recognition

Intelligent recognition of multi-scale marine targets remains pivotal in studying marine resources and transportation. Multi-scale marine target recognition faces challenges such as blurred image, noise interference, varied target sizes, and random target positions. However, these hardly affect the judgment of human brain which could adeptly capture multi-scale targets and disregard noise interference. Therefore, this study proposes an innovative approach to recognize multi-scale marine targets through taking full advantages of the texture, color and structural information provided by remote sensing images and the quick classification ability of human brains, called Selective Kernel & Multi-dimensional Multimodal Data Fusion Module Network (SK-MMFMNet), which fuses remote sensing images and electroencephalography (EEG) signals to improve the accuracy of classifying multi-scale marine targets. In this study, we construct a multi-scale marine target dataset, which includes both remote sensing images of islands, wind turbines, and ships and their corresponding EEG signals from subjects while viewing remote sensing images. Then, the proposed approach extends the Multimodal Transfer Module (MMTM) based on attention mechanism to a dual fusion module across channel and spatial dimensions to fusing MobileNetV3 and EEGNet. Also, we embed the Selective Kernel Module into MobileNetV3 for addressing multi-scale features. The average experimental results across the three multi-scale marine target sub-dataset show that SK-MMFMNet exhibited accuracy improvements of 2.88 %, 21.60 %, and 1.08 %, moreover, F1-Score increments of 24.60 %, 162.22 %, and 14.32 % compared to MobileNetV3, EEGNet, and MMTMNet (MMTM-based fusion network). Visual analysis via Grad-CAM demonstrates that benefiting from EEG signals and Selective Kernel Module, our proposed SK-MMFMNet adjusts the network attention to exactly focus on the multi-scale target area, and thus achieves the best performance. Meanwhile, T-SNE visualization also proves the effectiveness of the three fusion modules and EEG signals for feature extraction. This study offers a valuable and promising insight for intelligent recognition of multi-scale marine targets.

MAE-EEG-Transformer: A transformer-based approach combining masked autoencoder and cross-individual data augmentation pre-training for EEG classification

Convolutional neural networks (CNN) may not be ideal for extracting global temporal features from non-stationary Electroencephalogram (EEG) signals. The application of the masking-based method in EEG classification is not well studied, and there is a shortage of commonly accepted models for verifying inter-individual results in motor imagery classification tasks. The MAE-EEG-Transformer, a transformer with masking mechanism, is proposed in this article. It pre-trains by randomly masking signals and forces the model to learn semantic features. The pre-trained encoder module is fine-tuned and moved to the classification task to obtain the category of EEG signals. The effectiveness of features with and without pre-training is compared using t-SNE visualization to demonstrate pre-training’s inter-subject efficacy. The MAE EEG Transformer was extensively evaluated across three prevalent datasets in EEG-based motor imagery, demonstrating performance comparable to state-of-the-art models while requiring only approximately 20% of the computational cost (results in Table 1, 2, 3 and 4).

Navigating the Chemical Space of ENR Inhibitors: A Comprehensive Analysis.

Antimicrobial resistance is a global health threat that requires innovative strategies against drug-resistant bacteria. Our study focuses on enoyl-acyl carrier protein reductases (ENRs), in particular FabI, FabK, FabV, and InhA, as potential antimicrobial agents. Despite their promising potential, the lack of clinical approvals for inhibitors such as triclosan and isoniazid underscores the challenges in achieving preclinical success. In our study, we curated and analyzed a dataset of 1412 small molecules recognized as ENR inhibitors, investigating different structural variants. Using advanced cheminformatic tools, we mapped the physicochemical landscape and identified specific structural features as key determinants of bioactivity. Furthermore, we investigated whether the compounds conform to Lipinski rules, PAINS, and Brenk filters, which are crucial for the advancement of compounds in development pipelines. Furthermore, we investigated structural diversity using four different representations: Chemotype diversity, molecular similarity, t-SNE visualization, molecular complexity, and cluster analysis. By using advanced bioinformatics tools such as matched molecular pairs (MMP) analysis, machine learning, and SHAP analysis, we were able to improve our understanding of the activity cliques and the precise effects of the functional groups. In summary, this chemoinformatic investigation has unraveled the FAB inhibitors and provided insights into rational antimicrobial design, seamlessly integrating computation into the discovery of new antimicrobial agents.