Data Dimensionality Reduction Methods Research Articles

Comparison of interfacial injected energy between simulated lightning return stroke experiment and common Joule heat & Arc heat model application

A previous study has shown that the thermal damage of the Joule thermal arc heat transfer model is lighter than that of a natural lightning strike. Therefore, this paper focuses on the return stroke current and proposes an improved experimental method of simulated return damage without using arc-inducing wire. Combining the data with the inversion model of injected energy, the energy transfer characteristics of the samples are characterized. Furthermore, a data dimensionality reduction method based on multiple correlation coefficients is used to discuss the impact of the current peak/rise rate/wave tail time on the injected energy discrepancy. The results indicate a positive correlation between the current peak and current rise rate with the injected energy discrepancy. When the tail time exceeds 15 microseconds, the injected energy discrepancy decreases as the tail time increases. The thermal source characteristics of energy transfer during the return stroke process are determined. During the initial phase of the return stroke current, interfacial energy transfer includes contributions from ion enthalpy flux, Joule heating, and electronic enthalpy flux. When the tail time exceeds 15 microseconds, the contribution of ion enthalpy flux to the injected energy diminishes with increasing tail time.

What charge-off rates are predictable by macroeconomic latent factors?

Charge-offs provide critical insights into the risk level of loan portfolios within the banking system, signaling potential systemic risks that could lead to deep recessions. Utilizing consolidated financial statements, we have compiled the net charge-off rate (COR) data from the 10 largest U.S. bank holding companies (BHCs) for disaggregated loans, including business loans, real estate loans, and consumer loans, as well as the average COR for each loan category among these top 10 banks. We propose factor-augmented forecasting models for CORs that incorporate latent common factor estimates, including targeted factors, via an array of data dimensionality reduction methods for a large panel of macroeconomic predictors. Our models have demonstrated superior performance compared with benchmark forecasting models, particularly well for business loan and real estate loan CORs, while predicting consumer loan CORs remains challenging especially at short horizons. Notably, real activity factors improve the out-of-sample predictability over the benchmarks for business loan CORs even when financial sector factors are excluded.

Evolution and Spatiotemporal Response of Ecological Environment Quality to Human Activities and Climate: Case Study of Hunan Province, China

Human beings are facing increasingly serious threats to the ecological environment with industrial development and urban expansion. The changes in ecological environmental quality (EEQ) and their driving factors are attracting increased attention. As such, simple and effective ecological environmental quality monitoring processes must be developed to help protect the ecological environment. Based on the RSEI, we improved the data dimensionality reduction method using the coefficient of variation method, constructing RSEI-v using Landsat and MODIS data. Based on RSEI-v, we quantitatively monitored the characteristics of the changes in EEQ in Hunan Province, China, and the characteristics of its spatiotemporal response to changes in human activities and climate factors. The results show the following: (1) RSEI-v and RSEI perform similarly in characterizing ecological environment quality. The calculated RSEI-v is a positive indicator of EEQ, but RSEI is not. (2) The high EEQ values in Hunan are concentrated in the eastern and western mountainous areas, whereas low values are concentrated in the central plains. (3) A total of 49.40% of the area was experiencing substantial changes in EEQ, and the areas with significant decreases (accounting for 2.42% of the total area) were concentrated in the vicinity of various cities, especially the Changsha–Zhuzhou–Xiangtan urban agglomeration. The areas experiencing substantial EEQ increases (accounting for 16.97% of the total area) were concentrated in the eastern and western forests. (4) The areas experiencing substantial EEQ decreases, accounting for more than 60% of the area, were mainly affected by human activities. The areas surrounding Changsha and Hengyang experienced noteworthy decreases in EEQ. The areas where the EEQ was affected by precipitation and temperature were mainly concentrated in the eastern and western mountainous areas. This study provides a valuable reference for ecological environment quality monitoring and environmental protection.

Comparison of Data Visualization, Outlier Detection and Data Dimensionality Reduction Methods

With the deepening of the digital age of information, people's daily data is getting larger and larger, and it is more and more difficult to quantify and process. At this time, the data processing means becomes particularly important. This paper compares and analyzes some methods from data visualization to data dimensionality reduction to outlier detection. In this paper, two different types of datasets, ModelNet40, and red wine quality, are used to introduce the visualization method of the Farthest Point Sampling (FPS). This method can have a clear visual effect on the data dimension and scale, and allow users to observe the structure, type, and scale of the data. In data dimensionality reduction, the study uses Principal Component Analysis (PCA), T-Distributed Stochastic Neighbor Embedding (t-SNE), Triplets Manifold Approximation and Projection (TriMAP), Uniform Manifold Approximation and Projection (UMAP), Pairwise Controlled Manifold Approximation Projection (PaCMAP), and Autoencoder to compare their dimensionality reduction effects. Through these methods, this paper finds that different methods have different effects on different datasets. Therefore, in data dimensionality reduction, it can get twice the result with half the effort by choosing the appropriate method. Finally, this paper also detects outliers. Outliers in datasets will make it difficult for people to process data and make subsequent results inaccurate, so it is necessary to identify outliers. This paper involves methods such as isolation forest and Density-Based Spatial Clustering of Applications with Noise (DBSCAN). Through this paper, the methods of different datasets are analyzed and summarized.

A New and Effective Classification Method for Complex Time Series Based on Information Measure

The growing importance of time series information measure raises questions about how to effectively cluster a large number of nonlinear complex time series data and accurately extract more hidden information from them. In this paper, a clustering measurement and classification method for complex time series, the symmetrical exponential Tsallis relative information (SETRI) measure, is proposed, which aims to address these problems. The intrinsic characteristics of different types of time series information could be validly identified by this method. The modified multidimensional scaling (MDS) method, based on the SETRI measure, has the ability to display the data in the form of graphs for intuitive exhibition and accomplish the process of dimension reduction. The introduction of weighted permutation patterns allows a higher-accuracy classification not only for the time series dissimilarity quantification, but also for avoiding dispensable errors. Besides, the feasibility of the modified MDS classification method is visually and quantitatively verified by the simulated and real-world data. Compared with other MDS methods, the proposed method has better performance, which is reflected in the validity and rationality of the clustering results, thus further verifying the feasibility of the proposed method. Therefore, the new results will be helpful to develop complex data clustering and dimensionality reduction methods.

Contrastive Learning Dimensionality Reduction Method Based on Manifold Learning

With the development of the times, more and more high-dimensional datasets come into people's view. In order to reduce the time complexity and space complexity of downstream tasks, data dimensionality reduction becomes the primary choice. Classical dimensionality reduction algorithms are mainly divided into linear dimensionality reduction algorithms and nonlinear dimensionality reduction algorithms. Some of the traditional dimensionality reduction methods have the problems of not considering the nonlinear structure of the original dataset and the existence of weak generalisation, which makes the dimensionality reduction effect not good or model need to be recalculated because of the addition of new samples. In order to solve these problems, the research in this paper is a comparative learning dimensionality reduction method based on manifold learning. The idea of manifold learning using geodesic distance can fully consider the nonlinear structure of the original dataset. In this paper, comparative learning is the main framework. When the neural network completes the training, it only need to take the new data as input to calculate, the result can be obtained, no need to reconstruct the model which means the generality is high. Starting from the related work, this paper briefly introduces manifold learning, comparative learning and neural network algorithms. Subsequently, an innovative model is proposed, including three modules, Isomap to extract nonlinear structure, expanding neighbourhood to make pseudo-labels, and comparative learning training. Detailed analyses are carried out through experiments, comparing with PCA and LLE algorithms with the geodetic distance retention rate as an indicator, which proves that the data dimensionality reduction method of this model is more effective and ubiquitous.

Deep Semi-Supervised Learning Method for False Data Detection Against Forgery and Concealing of Faults in Cyber-Physical Power Systems

This paper proposes a deep semi-supervised numerical false data detection (DSS-NFDD) method, where hidden data in labeled and unlabeled datasets are leveraged simultaneously, to detect the false data in CPPS. Two types of false data are considered in this study: one is the forged fault data, which makes operators mistakenly believe that there is a fault in their operating system; the other is the false data used to conceal actual faults. First, a data dimension reduction method is proposed based on the PageRank algorithm to avoid the excessive noise caused by high-dimension data. Then, a semi-supervised deep learning framework is established to detect false data samples, which consists of two parts: one is the priori estimation module, and the other is a false data scoring network. A novel concentration loss function is presented for training the false data scoring network, which minimizes the impacts of noise pollution and sample bias. Next, a false feature location method is proposed to help human operators eliminate anomalies. Finally, the effectiveness and the superiorities of the proposed DSS-NFDD method are verified by analyzing the simulation results for the IEEE-39 bus and IEEE-118 bus systems.

Incremental semi-supervised graph learning NMF with block-diagonal

Non-negative matrix factorization (NMF), as a good data dimensionality reduction method, is widely used in the field of image recognition. Incremental non-negative matrix factorization (INMF) as an improvement solves the problem of inefficiency caused by repeated running of data samples during online learning. However, in the traditional incremental non-negative matrix factorization algorithm, the newly added training samples do not contain label information. Some samples may be marked in both the initial sample and the new training sample in the real application scenario. In order to make full use of the label information carried by the dataset, in this paper, we propose a semi-supervised non-negative matrix factorization model for batch incremental data, incremental semi-supervised graph learning NMF with block diagonal (ISGDNMF). The model is divided into three cases according to the degree of label-carrying for the new batch data: all label-carrying, no label-carrying, and partial label-carrying. The label information is also used to add a diagonal structure to the coefficient matrix, which makes it possible to have stronger discriminatory ability and to distinguish different classes of images more easily. And graph regularization information is added in order to maintain the spatial-geometric structure of the data. Experiments on six image datasets show that this algorithm has superior performance relative to the other seven NMF-based algorithms.

Spatial assessments in texture analysis: what the radiologist needs to know.

To date, studies investigating radiomics-based predictive models have tended to err on the side of data-driven or exploratory analysis of many thousands of extracted features. In particular, spatial assessments of texture have proven to be especially adept at assessing for features of intratumoral heterogeneity in oncologic imaging, which likewise may correspond with tumor biology and behavior. These spatial assessments can be generally classified as spatial filters, which detect areas of rapid change within the grayscale in order to enhance edges and/or textures within an image, or neighborhood-based methods, which quantify gray-level differences of neighboring pixels/voxels within a set distance. Given the high dimensionality of radiomics datasets, data dimensionality reduction methods have been proposed in an attempt to optimize model performance in machine learning studies; however, it should be noted that these approaches should only be applied to training data in order to avoid information leakage and model overfitting. While area under the curve of the receiver operating characteristic is perhaps the most commonly reported assessment of model performance, it is prone to overestimation when output classifications are unbalanced. In such cases, confusion matrices may be additionally reported, whereby diagnostic cut points for model predicted probability may hold more clinical significance to clinical colleagues with respect to related forms of diagnostic testing.

Nondestructive and rapid detection of foreign materials in wolfberry by hyperspectral imaging combing with chemometrics

The foreign materials in wolfberries, such as wolfberry flowers, leaves, branches and other plant fruits, can affect the overall grade and economic value of wolfberry. Identifying and removing these foreign materials is a crucial step after wolfberry picking. This study aims to provide a method of hyperspectral imaging combing with chemometrics (HSIcC) for identification of foreign materials in wolfberries. Wolfberry flower, leaf, branch and dogwood are taken as the foreign materials in this paper. Hyperspectral images of wolfberries and four foreign materials are collected in wavelength range of 370–1060 nm. And the analytical model is established. Different spectral preprocessing algorithms, including vector normalization (VN), the savitzky–golay (SG) method, the first-derivative, the second-derivative, and standard normal variable (SNV) transformation, are used and compared. Principal component analysis (PCA) and competitive adaptive reweighted sampling (CARS) are employed as data dimension reduction methods. K-Nearest Neighbor (KNN), Random Forest (RF) and Support Vector Machine (SVM) are utilized for modeling. The optimal results are obtained by “SG+CARS +KNN” model, with accuracy for training, validation, and testing set of 100%, 100% and 100%, respectively. The results show that HSIcC can provide a rapid and nondestructive on-line detection method for foreign materials identification of wolfberries.

Trends and Advances on The K-Hyperparameter Tuning Techniques in High-Dimensional Space Clustering

Clustering is one of the tasks performed during exploratory data analysis with an extensive and wealthy history in a variety of disciplines. Application of clustering in computational medicine is one such application of clustering that has proliferated in the recent past. K-means algorithms are the most popular because of their ability to adapt to new examples besides scaling up to large datasets. They are also easy to understand and implement. However, with k-means algorithms, k-hyperparameter tuning is a long standing challenge. The sparse and redundant nature of the high-dimensional datasets makes the k-hyperparameter tuning in high-dimensional space clustering a more challenging task. A proper k-hyperparameter tuning has a significant effect on the clustering results. A number of state-of-the art k-hyperparameter tuning techniques in high-dimensional space have been proposed. However, these techniques perform differently in a variety of high-dimensional datasets and data-dimensionality reduction methods. This article uses a five-step methodology to investigate the trends and advances on the state of the art k-hyperparameter tuning techniques in high-dimensional space clustering, data dimensionality reduction methods used with these techniques, their tuning strategies, nature of the datasets applied with them as well as the challenges associated with the cluster analysis in high-dimensional spaces. The metrics used in evaluating these techniques are also reviewed. The results of this review, elaborated in the discussion section, makes it efficient for data science researchers to undertake an empirical study among these techniques; a study that subsequently forms the basis for creating improved solutions to this k-hyperparameter tuning problem.

A Software Defect Prediction Approach Based on Hybrid Feature Dimensionality Reduction

Software defect prediction (SDP) is designed to assist software testing, which can reasonably allocate test resources to reduce costs and improve development efficiency. In order to improve the prediction performance, researchers have designed many defect-related features for SDP. However, feature redundancy (FR) and irrelevance caused by the increasing dimensions of data will greatly degrade the performance of defect prediction. In order to solve the problems, researchers have proposed various data dimensionality reduction methods. These methods can be simply divided into two categories of methods: feature selection and feature extraction. However, the two categories of methods have their own advantages and limitation. In this paper, we propose a Hybrid Feature Dimensionality Reduction Approach (HFDRA) for SDP, which combines the two different kinds of methods, to improve the performance of SDP. HFDRA approach can be divided into two stages: feature selection and feature extraction. First, HFDRA divides the original features into several feature subsets through a clustering algorithm in the feature selection stage. Then, in the feature extraction stage, kernel principal component analysis (KPCA) is used to reduce the dimensionality of each feature subset. Finally, the reduced-dimensional data is used to build the prediction model. In the empirical study, we use 22 projects from AEEEM, SOFTLAB, MORP, and ReLink as experiment object. In this paper, we first compare our approach with seven baseline methods and three state-of-the-art methods. Then, we analyze the relationship between FR and prediction performance. Experiment results show that our approach outperforms the state-of-the-art data dimensionality reduction methods for defect prediction.

Joint Feature Selection and Extraction With Sparse Unsupervised Projection.

Feature selection and feature extraction, in the field of data dimensionality reduction, are the two main strategies. Nevertheless, each of these two strategies has its own advantages and disadvantages. The features chosen by feature selection method have complete physical meaning. However, feature selection cannot reveal the implicit structural information of the samples. In this article, the methods proposed by us combine both feature selection and feature extraction, called joint feature selection and extraction with sparse unsupervised projection (SUP) and graph optimization SUP (GOSUP). A constraint on the number of nonzero rows of the projection matrix is added, which ensures the sparsity of the projection matrix, and only the features corresponding to the nonzero rows of the projection matrix are selected for the feature extraction procedure. We invoke a newly proposed algorithm to tackle this constrained optimization problem. A new concept of "purification matrix" is invented, the use of which could better eliminate meaningless information of samples in subspace. The performance on several datasets verifies the effectiveness of the proposed method for data dimensionality reduction.

Efficient data dimensionality reduction method for improving road crack classification algorithms

AbstractAutomatic crack classification plays an essential role in road maintenance. Using many features for the classification is inefficient for implementing embedded systems with low computational resources makes it difficult. Therefore, this work proposes a new data dimensionality reduction (DDR) for crack classification algorithms (DDR4CC). DDR4CC reduces the required information about the cracks to only four features. Using these features, the images can be classified into longitudinal, transverse, and alligator cracks or healthy pavement. DDR4CC is compared with eight DDR methods, and the reduced set of features is analyzed using five different classification algorithms. Besides, five different datasets, generated by a combination of several public datasets, are used. We are proposing a simple DDR method with high interpretability of the data, obtaining very fast computation and high accuracy. Experiments show that DDR4CC enhances the results of the classification algorithms, providing almost perfect classifiers with a minimum computation time.

A New Data Dimension Reduction Method Based On Convolution In The Application Of Authenticity Identification Of Traditional Chinese Medicine LongGu

As a commonly used Chinese herbal medicine in clinic, LongGu is often used as a powder, which is difficult to identify manually. In this study, unsupervised learning method was used to model and identify LongGu, calcining LongGu and LongGu counterfeits by infrared spectroscopy. After preprocessing the original data, the current commonly used principal component analysis method, which was used to reduce the data dimension, and then the KMeans algorithm was used to realize the classification and identification of the samples. The classification results were better in the theoretical samples, but not in the actual samples. Based on this problem, this paper proposes a convolution-based machine learning feature dimension reduction method for spectral data. Comparing with the data method of principal component analysis, the partial classification effect is obviously better on the theoretical data. The effect of identifying calcining LongGu and LongGu counterfeits in actual data has been significantly improved, and it also provides a reference for the application of machine learning technology in the field of spectral identification of traditional Chinese medicine.

Vis-NIR and SWIR hyperspectral imaging method to detect bruises in pomegranate fruit.

Fresh pomegranate fruit is susceptible to bruising, a common type of mechanical damage during harvest and at all stages of postharvest handling. Accurate and early detection of such damages in pomegranate fruit plays an important role in fruit grading. This study investigated the detection of bruises in fresh pomegranate fruit using hyperspectral imaging technique. A total of 90 sample of pomegranate fruit were divided into three groups of 30 samples, each representing purposefully induced pre-scanning bruise by dropping samples from 100 cm and 60 cm height on a metal surface. The control has no pre-scanning bruise (no drop). Two hyperspectral imaging setups were examined: visible and near infrared (400 to 1000 nm) and short wavelength infrared (1000 to 2500 nm). Region of interest (ROI) averaged reflectance spectra was implemented to reduce the image data. For all hypercubes a principal components analysis (PCA) based background removal were done prior to segmenting the region of interest (ROI) using the Evince® multi-variate analysis software 2.4.0. Then the average spectrum of the ROI of each sample was computed and transferred to the MATLAB 2022a (The MathWorks, Inc., Mass., USA) for classification. A two-layer feed-forward artificial neural network (ANN) is used for classification. The accuracy of bruise severity classification ranged from 80 to 96.7%. When samples from both bruise severity (Bruise damage induced from a 100cm and 60 cm drop heights respectively) cases were merged, class recognition accuracy were 88.9% and 74.4% for the SWIR and Vis-NIR, respectively. This study implemented the method of selecting out informative bands and disregarding the redundant ones to decreases the data size and dimension. The study developed a more compact classification model by the data dimensionality reduction method. This study demonstrated the potential of using hyperspectral imaging technology in sensing and classification of bruise severity in pomegranate fruit. This work provides the foundation to build a compact and fast multispectral imaging-based device for practical farm and packhouse applications.

Topology identification method for residential areas in low-voltage distribution networks based on unsupervised learning and graph theory

• Improve traditional data dimensionality reduction and clustering method, and propose tSNE-DBSCAN-LLE algorithm. • Four-level topology information can be identify by the method. • Noise-containing data and incomplete data collected by smart meters can also be used effectively. • A complete topology diagram can be simply generated to visualize the identified information. • Test results derive the different identification capabilities that the method can play under different noise environments. The information of a low-voltage (LV) distribution network is important for power supply departments to monitor grid information, analyze faults and optimize grid operation status. However, the current mainstream methods are not able to comprehensively update the topology information of LV distribution networks in real time. Therefore, this paper proposes an unsupervised learning and graph theory-based method to identify four-level topology information and generate a topology diagram for low-voltage distribution network. Firstly, four-level topology information are identified based on the tSNE-DBSCAN-LLE algorithm. Then, the identied information is used to simply generate a topology diagram. Finally, the simulation data and the actual data from three LV distribution networks are analyzed to show the effectiveness and advantageousness of the proposed method.

Construction and validation of a prognostic model for gastrointestinal stromal tumors based on copy number alterations and clinicopathological characteristics.

The increasing incidence of gastrointestinal stromal tumors (GISTs) has led to the discovery of more novel prognostic markers. We aim to establish an unsupervised prognostic model for the early prediction of the prognosis of future patients with GISTs and to guide clinical treatment. We downloaded the GISTs dataset through the cBioPortal website. We extracted clinical information and pathological information, including the microsatellite instability (MSI) score, fraction genome altered (FGA) score, tumor mutational burden (TMB), and copy number alteration burden (CNAB), of patients with GISTs. For survival analysis, we used univariate Cox regression to analyze the contribution of each factor to prognosis and calculated a hazard ratio (HR) and 95% confidence interval (95% CI). For clustering groupings, we used the t-distributed stochastic neighbor embedding (t-SNE) method for data dimensionality reduction. Subsequently, the k-means method was used for clustering analysis. A total of 395 individuals were included in the study. After dimensionality reduction with t-SNE, all patients were divided into two subgroups. Cluster 1 had worse OS than cluster 2 (HR=3.45, 95% CI, 2.22-5.56, P<0.001). The median MSI score of cluster 1 was 1.09, and the median MSI score of cluster 2 was 0.24, which were significantly different (P<0.001). The FGA score of cluster 1 was 0.28, which was higher than that of cluster 2 (P<0.001). In addition, both the TMB and CNAB of cluster 1 were higher than those of cluster 2, and the P values were less than 0.001. Based on the CNA of GISTs, patients can be divided into high-risk and low-risk groups. The high-risk group had a higher MSI score, FGA score, TMB and CNAB than the low-risk group. In addition, we established a prognostic nomogram based on the CNA and clinicopathological characteristics of patients with GISTs.

Classification of Plates and Trihedral Corner Reflectors Based on Linear Wavefront Phase-Modulated Beam

Wavefront-modulated beams such as vortex beams have attracted much attention in the field of target recognition due to the introduced degrees of freedom. However, traditional wavefront-modulated beams are doughnut shaped, and are not suitable for radar detection or tracking. To solve this problem, a linear wavefront phase-modulated beam with a maximum radiation intensity in the center was proposed in a previous study. In this paper, we continue to study target characteristics under the linear wavefront phase-modulated beam. Through analysis of the target scattering based on the physical optics (PO) method, we find that a part of the monostatic or bistatic radar cross-section (RCS) of the target could be obtained by changing the phase gradient of the modulated beam. Taking this part of RCS for feature extraction, we recognize the plates and trihedral corner reflectors through the support vector machine (SVM) method. For data visualization, we use the t-distributed stochastic neighbor embedding (t-SNE) method for data dimensionality reduction. The results show that the recognition probability of the plates and trihedral corner reflectors can reach 91% with an antenna array having an aperture of 20 wavelengths when the signal-to-noise ratio (SNR) is 20 dB, while the traditional plane beam cannot classify these two targets directly.

Accounting for regional water recyclability or scarcity using Machine Learning and Pinch Analysis

Water stress is becoming a major concern worldwide because of the lack of fresh resources to meet growing water demand in the face of climate change. Resources recycling is a viable option, but the main dilemma is to define a proper water quality grading system. This paper proposes a hybrid framework combining Machine Learning (ML) with Process Integration (PI) tools for assessing the regional water scarcity and recycling potential. The procedure involves defining the quality of water resources using supervised or unsupervised ML. Supervised ML (Classification) is employed when the data samples' origins or quality levels are known. The data can be sampled from an existing recycling system. The unsupervised ML (Clustering) method is used when quality levels are unknown. Data dimensionality reduction or expansion methods are used on the dataset to yield better classification or clustering outcomes. Once the hierarchical quality classes/clusters are revealed, the PI approach of Pinch Analysis is applied with the defined quality categories for planning water exchange systems (e.g., urban water networks or industrial parks). The method not only identifies the quality bottleneck of the system but also reveals the fresh resources deficit or excess of system supplies based on the defined quality clusters. This novel concept is demonstrated with case studies featuring different water sources and scenarios. Results show that the hybrid approach can categorise the water sources effectively, and depending on the number of defined clusters/categories, the water recycling potential can be different (e.g. with 5 clusters, the recyclability rate is 44%, while with 2 clusters, the recyclability rate can increase to 78% for the case study). The framework could serve as a guideline for regional authorities to manage the water resources according to their own water resources and properties database.