Data Dimensionality Reduction Methods Research Articles

Comparison of Data Visualization, Outlier Detection and Data Dimensionality Reduction Methods

With the deepening of the digital age of information, people's daily data is getting larger and larger, and it is more and more difficult to quantify and process. At this time, the data processing means becomes particularly important. This paper compares and analyzes some methods from data visualization to data dimensionality reduction to outlier detection. In this paper, two different types of datasets, ModelNet40, and red wine quality, are used to introduce the visualization method of the Farthest Point Sampling (FPS). This method can have a clear visual effect on the data dimension and scale, and allow users to observe the structure, type, and scale of the data. In data dimensionality reduction, the study uses Principal Component Analysis (PCA), T-Distributed Stochastic Neighbor Embedding (t-SNE), Triplets Manifold Approximation and Projection (TriMAP), Uniform Manifold Approximation and Projection (UMAP), Pairwise Controlled Manifold Approximation Projection (PaCMAP), and Autoencoder to compare their dimensionality reduction effects. Through these methods, this paper finds that different methods have different effects on different datasets. Therefore, in data dimensionality reduction, it can get twice the result with half the effort by choosing the appropriate method. Finally, this paper also detects outliers. Outliers in datasets will make it difficult for people to process data and make subsequent results inaccurate, so it is necessary to identify outliers. This paper involves methods such as isolation forest and Density-Based Spatial Clustering of Applications with Noise (DBSCAN). Through this paper, the methods of different datasets are analyzed and summarized.

A New and Effective Classification Method for Complex Time Series Based on Information Measure

The growing importance of time series information measure raises questions about how to effectively cluster a large number of nonlinear complex time series data and accurately extract more hidden information from them. In this paper, a clustering measurement and classification method for complex time series, the symmetrical exponential Tsallis relative information (SETRI) measure, is proposed, which aims to address these problems. The intrinsic characteristics of different types of time series information could be validly identified by this method. The modified multidimensional scaling (MDS) method, based on the SETRI measure, has the ability to display the data in the form of graphs for intuitive exhibition and accomplish the process of dimension reduction. The introduction of weighted permutation patterns allows a higher-accuracy classification not only for the time series dissimilarity quantification, but also for avoiding dispensable errors. Besides, the feasibility of the modified MDS classification method is visually and quantitatively verified by the simulated and real-world data. Compared with other MDS methods, the proposed method has better performance, which is reflected in the validity and rationality of the clustering results, thus further verifying the feasibility of the proposed method. Therefore, the new results will be helpful to develop complex data clustering and dimensionality reduction methods.

Contrastive Learning Dimensionality Reduction Method Based on Manifold Learning

With the development of the times, more and more high-dimensional datasets come into people's view. In order to reduce the time complexity and space complexity of downstream tasks, data dimensionality reduction becomes the primary choice. Classical dimensionality reduction algorithms are mainly divided into linear dimensionality reduction algorithms and nonlinear dimensionality reduction algorithms. Some of the traditional dimensionality reduction methods have the problems of not considering the nonlinear structure of the original dataset and the existence of weak generalisation, which makes the dimensionality reduction effect not good or model need to be recalculated because of the addition of new samples. In order to solve these problems, the research in this paper is a comparative learning dimensionality reduction method based on manifold learning. The idea of manifold learning using geodesic distance can fully consider the nonlinear structure of the original dataset. In this paper, comparative learning is the main framework. When the neural network completes the training, it only need to take the new data as input to calculate, the result can be obtained, no need to reconstruct the model which means the generality is high. Starting from the related work, this paper briefly introduces manifold learning, comparative learning and neural network algorithms. Subsequently, an innovative model is proposed, including three modules, Isomap to extract nonlinear structure, expanding neighbourhood to make pseudo-labels, and comparative learning training. Detailed analyses are carried out through experiments, comparing with PCA and LLE algorithms with the geodetic distance retention rate as an indicator, which proves that the data dimensionality reduction method of this model is more effective and ubiquitous.

Deep Semi-Supervised Learning Method for False Data Detection Against Forgery and Concealing of Faults in Cyber-Physical Power Systems

This paper proposes a deep semi-supervised numerical false data detection (DSS-NFDD) method, where hidden data in labeled and unlabeled datasets are leveraged simultaneously, to detect the false data in CPPS. Two types of false data are considered in this study: one is the forged fault data, which makes operators mistakenly believe that there is a fault in their operating system; the other is the false data used to conceal actual faults. First, a data dimension reduction method is proposed based on the PageRank algorithm to avoid the excessive noise caused by high-dimension data. Then, a semi-supervised deep learning framework is established to detect false data samples, which consists of two parts: one is the priori estimation module, and the other is a false data scoring network. A novel concentration loss function is presented for training the false data scoring network, which minimizes the impacts of noise pollution and sample bias. Next, a false feature location method is proposed to help human operators eliminate anomalies. Finally, the effectiveness and the superiorities of the proposed DSS-NFDD method are verified by analyzing the simulation results for the IEEE-39 bus and IEEE-118 bus systems.

Incremental semi-supervised graph learning NMF with block-diagonal

Non-negative matrix factorization (NMF), as a good data dimensionality reduction method, is widely used in the field of image recognition. Incremental non-negative matrix factorization (INMF) as an improvement solves the problem of inefficiency caused by repeated running of data samples during online learning. However, in the traditional incremental non-negative matrix factorization algorithm, the newly added training samples do not contain label information. Some samples may be marked in both the initial sample and the new training sample in the real application scenario. In order to make full use of the label information carried by the dataset, in this paper, we propose a semi-supervised non-negative matrix factorization model for batch incremental data, incremental semi-supervised graph learning NMF with block diagonal (ISGDNMF). The model is divided into three cases according to the degree of label-carrying for the new batch data: all label-carrying, no label-carrying, and partial label-carrying. The label information is also used to add a diagonal structure to the coefficient matrix, which makes it possible to have stronger discriminatory ability and to distinguish different classes of images more easily. And graph regularization information is added in order to maintain the spatial-geometric structure of the data. Experiments on six image datasets show that this algorithm has superior performance relative to the other seven NMF-based algorithms.

Spatial assessments in texture analysis: what the radiologist needs to know.

To date, studies investigating radiomics-based predictive models have tended to err on the side of data-driven or exploratory analysis of many thousands of extracted features. In particular, spatial assessments of texture have proven to be especially adept at assessing for features of intratumoral heterogeneity in oncologic imaging, which likewise may correspond with tumor biology and behavior. These spatial assessments can be generally classified as spatial filters, which detect areas of rapid change within the grayscale in order to enhance edges and/or textures within an image, or neighborhood-based methods, which quantify gray-level differences of neighboring pixels/voxels within a set distance. Given the high dimensionality of radiomics datasets, data dimensionality reduction methods have been proposed in an attempt to optimize model performance in machine learning studies; however, it should be noted that these approaches should only be applied to training data in order to avoid information leakage and model overfitting. While area under the curve of the receiver operating characteristic is perhaps the most commonly reported assessment of model performance, it is prone to overestimation when output classifications are unbalanced. In such cases, confusion matrices may be additionally reported, whereby diagnostic cut points for model predicted probability may hold more clinical significance to clinical colleagues with respect to related forms of diagnostic testing.

Nondestructive and rapid detection of foreign materials in wolfberry by hyperspectral imaging combing with chemometrics

The foreign materials in wolfberries, such as wolfberry flowers, leaves, branches and other plant fruits, can affect the overall grade and economic value of wolfberry. Identifying and removing these foreign materials is a crucial step after wolfberry picking. This study aims to provide a method of hyperspectral imaging combing with chemometrics (HSIcC) for identification of foreign materials in wolfberries. Wolfberry flower, leaf, branch and dogwood are taken as the foreign materials in this paper. Hyperspectral images of wolfberries and four foreign materials are collected in wavelength range of 370–1060 nm. And the analytical model is established. Different spectral preprocessing algorithms, including vector normalization (VN), the savitzky–golay (SG) method, the first-derivative, the second-derivative, and standard normal variable (SNV) transformation, are used and compared. Principal component analysis (PCA) and competitive adaptive reweighted sampling (CARS) are employed as data dimension reduction methods. K-Nearest Neighbor (KNN), Random Forest (RF) and Support Vector Machine (SVM) are utilized for modeling. The optimal results are obtained by “SG+CARS +KNN” model, with accuracy for training, validation, and testing set of 100%, 100% and 100%, respectively. The results show that HSIcC can provide a rapid and nondestructive on-line detection method for foreign materials identification of wolfberries.

Trends and Advances on The K-Hyperparameter Tuning Techniques in High-Dimensional Space Clustering

Clustering is one of the tasks performed during exploratory data analysis with an extensive and wealthy history in a variety of disciplines. Application of clustering in computational medicine is one such application of clustering that has proliferated in the recent past. K-means algorithms are the most popular because of their ability to adapt to new examples besides scaling up to large datasets. They are also easy to understand and implement. However, with k-means algorithms, k-hyperparameter tuning is a long standing challenge. The sparse and redundant nature of the high-dimensional datasets makes the k-hyperparameter tuning in high-dimensional space clustering a more challenging task. A proper k-hyperparameter tuning has a significant effect on the clustering results. A number of state-of-the art k-hyperparameter tuning techniques in high-dimensional space have been proposed. However, these techniques perform differently in a variety of high-dimensional datasets and data-dimensionality reduction methods. This article uses a five-step methodology to investigate the trends and advances on the state of the art k-hyperparameter tuning techniques in high-dimensional space clustering, data dimensionality reduction methods used with these techniques, their tuning strategies, nature of the datasets applied with them as well as the challenges associated with the cluster analysis in high-dimensional spaces. The metrics used in evaluating these techniques are also reviewed. The results of this review, elaborated in the discussion section, makes it efficient for data science researchers to undertake an empirical study among these techniques; a study that subsequently forms the basis for creating improved solutions to this k-hyperparameter tuning problem.

A Software Defect Prediction Approach Based on Hybrid Feature Dimensionality Reduction

Software defect prediction (SDP) is designed to assist software testing, which can reasonably allocate test resources to reduce costs and improve development efficiency. In order to improve the prediction performance, researchers have designed many defect-related features for SDP. However, feature redundancy (FR) and irrelevance caused by the increasing dimensions of data will greatly degrade the performance of defect prediction. In order to solve the problems, researchers have proposed various data dimensionality reduction methods. These methods can be simply divided into two categories of methods: feature selection and feature extraction. However, the two categories of methods have their own advantages and limitation. In this paper, we propose a Hybrid Feature Dimensionality Reduction Approach (HFDRA) for SDP, which combines the two different kinds of methods, to improve the performance of SDP. HFDRA approach can be divided into two stages: feature selection and feature extraction. First, HFDRA divides the original features into several feature subsets through a clustering algorithm in the feature selection stage. Then, in the feature extraction stage, kernel principal component analysis (KPCA) is used to reduce the dimensionality of each feature subset. Finally, the reduced-dimensional data is used to build the prediction model. In the empirical study, we use 22 projects from AEEEM, SOFTLAB, MORP, and ReLink as experiment object. In this paper, we first compare our approach with seven baseline methods and three state-of-the-art methods. Then, we analyze the relationship between FR and prediction performance. Experiment results show that our approach outperforms the state-of-the-art data dimensionality reduction methods for defect prediction.

Joint Feature Selection and Extraction With Sparse Unsupervised Projection.

Feature selection and feature extraction, in the field of data dimensionality reduction, are the two main strategies. Nevertheless, each of these two strategies has its own advantages and disadvantages. The features chosen by feature selection method have complete physical meaning. However, feature selection cannot reveal the implicit structural information of the samples. In this article, the methods proposed by us combine both feature selection and feature extraction, called joint feature selection and extraction with sparse unsupervised projection (SUP) and graph optimization SUP (GOSUP). A constraint on the number of nonzero rows of the projection matrix is added, which ensures the sparsity of the projection matrix, and only the features corresponding to the nonzero rows of the projection matrix are selected for the feature extraction procedure. We invoke a newly proposed algorithm to tackle this constrained optimization problem. A new concept of "purification matrix" is invented, the use of which could better eliminate meaningless information of samples in subspace. The performance on several datasets verifies the effectiveness of the proposed method for data dimensionality reduction.

Efficient data dimensionality reduction method for improving road crack classification algorithms

AbstractAutomatic crack classification plays an essential role in road maintenance. Using many features for the classification is inefficient for implementing embedded systems with low computational resources makes it difficult. Therefore, this work proposes a new data dimensionality reduction (DDR) for crack classification algorithms (DDR4CC). DDR4CC reduces the required information about the cracks to only four features. Using these features, the images can be classified into longitudinal, transverse, and alligator cracks or healthy pavement. DDR4CC is compared with eight DDR methods, and the reduced set of features is analyzed using five different classification algorithms. Besides, five different datasets, generated by a combination of several public datasets, are used. We are proposing a simple DDR method with high interpretability of the data, obtaining very fast computation and high accuracy. Experiments show that DDR4CC enhances the results of the classification algorithms, providing almost perfect classifiers with a minimum computation time.

A New Data Dimension Reduction Method Based On Convolution In The Application Of Authenticity Identification Of Traditional Chinese Medicine LongGu

As a commonly used Chinese herbal medicine in clinic, LongGu is often used as a powder, which is difficult to identify manually. In this study, unsupervised learning method was used to model and identify LongGu, calcining LongGu and LongGu counterfeits by infrared spectroscopy. After preprocessing the original data, the current commonly used principal component analysis method, which was used to reduce the data dimension, and then the KMeans algorithm was used to realize the classification and identification of the samples. The classification results were better in the theoretical samples, but not in the actual samples. Based on this problem, this paper proposes a convolution-based machine learning feature dimension reduction method for spectral data. Comparing with the data method of principal component analysis, the partial classification effect is obviously better on the theoretical data. The effect of identifying calcining LongGu and LongGu counterfeits in actual data has been significantly improved, and it also provides a reference for the application of machine learning technology in the field of spectral identification of traditional Chinese medicine.

Vis-NIR and SWIR hyperspectral imaging method to detect bruises in pomegranate fruit.

Fresh pomegranate fruit is susceptible to bruising, a common type of mechanical damage during harvest and at all stages of postharvest handling. Accurate and early detection of such damages in pomegranate fruit plays an important role in fruit grading. This study investigated the detection of bruises in fresh pomegranate fruit using hyperspectral imaging technique. A total of 90 sample of pomegranate fruit were divided into three groups of 30 samples, each representing purposefully induced pre-scanning bruise by dropping samples from 100 cm and 60 cm height on a metal surface. The control has no pre-scanning bruise (no drop). Two hyperspectral imaging setups were examined: visible and near infrared (400 to 1000 nm) and short wavelength infrared (1000 to 2500 nm). Region of interest (ROI) averaged reflectance spectra was implemented to reduce the image data. For all hypercubes a principal components analysis (PCA) based background removal were done prior to segmenting the region of interest (ROI) using the Evince® multi-variate analysis software 2.4.0. Then the average spectrum of the ROI of each sample was computed and transferred to the MATLAB 2022a (The MathWorks, Inc., Mass., USA) for classification. A two-layer feed-forward artificial neural network (ANN) is used for classification. The accuracy of bruise severity classification ranged from 80 to 96.7%. When samples from both bruise severity (Bruise damage induced from a 100cm and 60 cm drop heights respectively) cases were merged, class recognition accuracy were 88.9% and 74.4% for the SWIR and Vis-NIR, respectively. This study implemented the method of selecting out informative bands and disregarding the redundant ones to decreases the data size and dimension. The study developed a more compact classification model by the data dimensionality reduction method. This study demonstrated the potential of using hyperspectral imaging technology in sensing and classification of bruise severity in pomegranate fruit. This work provides the foundation to build a compact and fast multispectral imaging-based device for practical farm and packhouse applications.

Topology identification method for residential areas in low-voltage distribution networks based on unsupervised learning and graph theory

• Improve traditional data dimensionality reduction and clustering method, and propose tSNE-DBSCAN-LLE algorithm. • Four-level topology information can be identify by the method. • Noise-containing data and incomplete data collected by smart meters can also be used effectively. • A complete topology diagram can be simply generated to visualize the identified information. • Test results derive the different identification capabilities that the method can play under different noise environments. The information of a low-voltage (LV) distribution network is important for power supply departments to monitor grid information, analyze faults and optimize grid operation status. However, the current mainstream methods are not able to comprehensively update the topology information of LV distribution networks in real time. Therefore, this paper proposes an unsupervised learning and graph theory-based method to identify four-level topology information and generate a topology diagram for low-voltage distribution network. Firstly, four-level topology information are identified based on the tSNE-DBSCAN-LLE algorithm. Then, the identied information is used to simply generate a topology diagram. Finally, the simulation data and the actual data from three LV distribution networks are analyzed to show the effectiveness and advantageousness of the proposed method.

Construction and validation of a prognostic model for gastrointestinal stromal tumors based on copy number alterations and clinicopathological characteristics.

The increasing incidence of gastrointestinal stromal tumors (GISTs) has led to the discovery of more novel prognostic markers. We aim to establish an unsupervised prognostic model for the early prediction of the prognosis of future patients with GISTs and to guide clinical treatment. We downloaded the GISTs dataset through the cBioPortal website. We extracted clinical information and pathological information, including the microsatellite instability (MSI) score, fraction genome altered (FGA) score, tumor mutational burden (TMB), and copy number alteration burden (CNAB), of patients with GISTs. For survival analysis, we used univariate Cox regression to analyze the contribution of each factor to prognosis and calculated a hazard ratio (HR) and 95% confidence interval (95% CI). For clustering groupings, we used the t-distributed stochastic neighbor embedding (t-SNE) method for data dimensionality reduction. Subsequently, the k-means method was used for clustering analysis. A total of 395 individuals were included in the study. After dimensionality reduction with t-SNE, all patients were divided into two subgroups. Cluster 1 had worse OS than cluster 2 (HR=3.45, 95% CI, 2.22-5.56, P<0.001). The median MSI score of cluster 1 was 1.09, and the median MSI score of cluster 2 was 0.24, which were significantly different (P<0.001). The FGA score of cluster 1 was 0.28, which was higher than that of cluster 2 (P<0.001). In addition, both the TMB and CNAB of cluster 1 were higher than those of cluster 2, and the P values were less than 0.001. Based on the CNA of GISTs, patients can be divided into high-risk and low-risk groups. The high-risk group had a higher MSI score, FGA score, TMB and CNAB than the low-risk group. In addition, we established a prognostic nomogram based on the CNA and clinicopathological characteristics of patients with GISTs.

Classification of Plates and Trihedral Corner Reflectors Based on Linear Wavefront Phase-Modulated Beam

Wavefront-modulated beams such as vortex beams have attracted much attention in the field of target recognition due to the introduced degrees of freedom. However, traditional wavefront-modulated beams are doughnut shaped, and are not suitable for radar detection or tracking. To solve this problem, a linear wavefront phase-modulated beam with a maximum radiation intensity in the center was proposed in a previous study. In this paper, we continue to study target characteristics under the linear wavefront phase-modulated beam. Through analysis of the target scattering based on the physical optics (PO) method, we find that a part of the monostatic or bistatic radar cross-section (RCS) of the target could be obtained by changing the phase gradient of the modulated beam. Taking this part of RCS for feature extraction, we recognize the plates and trihedral corner reflectors through the support vector machine (SVM) method. For data visualization, we use the t-distributed stochastic neighbor embedding (t-SNE) method for data dimensionality reduction. The results show that the recognition probability of the plates and trihedral corner reflectors can reach 91% with an antenna array having an aperture of 20 wavelengths when the signal-to-noise ratio (SNR) is 20 dB, while the traditional plane beam cannot classify these two targets directly.

Accounting for regional water recyclability or scarcity using Machine Learning and Pinch Analysis

Water stress is becoming a major concern worldwide because of the lack of fresh resources to meet growing water demand in the face of climate change. Resources recycling is a viable option, but the main dilemma is to define a proper water quality grading system. This paper proposes a hybrid framework combining Machine Learning (ML) with Process Integration (PI) tools for assessing the regional water scarcity and recycling potential. The procedure involves defining the quality of water resources using supervised or unsupervised ML. Supervised ML (Classification) is employed when the data samples' origins or quality levels are known. The data can be sampled from an existing recycling system. The unsupervised ML (Clustering) method is used when quality levels are unknown. Data dimensionality reduction or expansion methods are used on the dataset to yield better classification or clustering outcomes. Once the hierarchical quality classes/clusters are revealed, the PI approach of Pinch Analysis is applied with the defined quality categories for planning water exchange systems (e.g., urban water networks or industrial parks). The method not only identifies the quality bottleneck of the system but also reveals the fresh resources deficit or excess of system supplies based on the defined quality clusters. This novel concept is demonstrated with case studies featuring different water sources and scenarios. Results show that the hybrid approach can categorise the water sources effectively, and depending on the number of defined clusters/categories, the water recycling potential can be different (e.g. with 5 clusters, the recyclability rate is 44%, while with 2 clusters, the recyclability rate can increase to 78% for the case study). The framework could serve as a guideline for regional authorities to manage the water resources according to their own water resources and properties database.

Artificial intelligence informed toxicity screening of amine chemistries used in the synthesis of hybrid organic–inorganic perovskites

AbstractThis article describes a machine learning guided framework for screening the potential toxicity impact of amine chemistries used in the synthesis of hybrid organic–inorganic perovskites. Using a combination of a probabilistic molecular fingerprint technique that encodes bond connectivity (MinHash) coupled to non‐linear data dimensionality reduction methods (Uniform Manifold Approximation and Projection), we develop an “Amine Atlas.” We show how the Amine Atlas can be used to rapidly screen the relative toxicity levels of amine molecules used in the synthesis of 2D and 3D perovskites and help identify safer alternatives. Our work also serves as a framework for rapidly identifying molecular similarity guided, structure–function relationships for safer materials chemistries that also incorporate sustainability/toxicity concerns.

Combining Spectral and Texture Features of UAS-Based Multispectral Images for Maize Leaf Area Index Estimation

The leaf area index (LAI) is of great significance for crop growth monitoring. Recently, unmanned aerial systems (UASs) have experienced rapid development and can provide critical data support for crop LAI monitoring. This study investigates the effects of combining spectral and texture features extracted from UAS multispectral imagery on maize LAI estimation. Multispectral images and in situ maize LAI were collected from test sites in Tongshan, Xuzhou, Jiangsu Province, China. The spectral and texture features of UAS multispectral remote sensing images are extracted using the vegetation indices (VIs) and the gray-level co-occurrence matrix (GLCM), respectively. Normalized texture indices (NDTIs), ratio texture indices (RTIs), and difference texture indices (DTIs) are calculated using two GLCM-based textures to express the influence of two different texture features on LAI monitoring at the same time. The remote sensing features are prescreened through correlation analysis. Different data dimensionality reduction or feature selection methods, including stepwise selection (ST), principal component analysis (PCA), and ST combined with PCA (ST_PCA), are coupled with support vector regression (SVR), random forest (RF), and multiple linear regression (MLR) to build the maize LAI estimation models. The results reveal that ST_PCA coupled with SVR has better performance, in terms of the VIs + DTIs (R2 = 0.876, RMSE = 0.239) and VIs + NDTIs (R2 = 0.877, RMSE = 0.236). This study introduces the potential of different texture indices for maize LAI monitoring and demonstrates the promising solution of using ST_PCA to realize the combining of spectral and texture features for improving the estimation accuracy of maize LAI.

Adaptive Data Dimensionality Reduction for Chemical Process Modeling Based on the Information Criterion Related to Data Association and Redundancy

Chemical process modeling is the basis for research and applications in related fields. With the development of industrial informatization, data-driven process modeling techniques are increasingly applied in chemical processes, helping to obtain more accurate results with less model development costs. However, due to the high-dimensional nonlinear characteristics of most chemical processes, problems such as the “curse of dimensionality” and information redundancy will render the models more prone to overfitting with reduced accuracies and weaker generalization abilities. Many data dimensionality reduction methods are adopted to mitigate the above problems, but most are limited by inaccurate association measurements and weak redundancy exclusion. In this paper, the extensive existence of data associations and information redundancies is first revealed through the analysis from an information-theoretic perspective. Then, a feature selection method based on conditional refined maximal information coefficient maximization (CRMICM) is proposed to improve the consistency of association measurement and the accuracy of redundancy estimation with limited samples. The final prediction modeling test for an actual fluidized catalytic cracking (FCC) process proves the extensive association between the variables and targets. Only a few variables are essential for the modeling, while the rest are redundant. Compared with other methods, CRMICM achieves the best dimensionality reduction effects on the FCC process data regarding the number of features and model prediction accuracy, showing its good applicability for chemical processes.