Algorithm For High-dimensional Data Research Articles

Scalable Neural Network Algorithms for High Dimensional Data

The boundary for machine learning engineers lately has moved from the restricted data to the algorithms' failure to involve every one of the data in the time permitted. Due of this, scientists are presently worried about the adaptability of machine learning algorithms notwithstanding their exactness. The key to success for many computer vision and machine learning challenges is having big training sets. A few published systematic reviews were taken into account in this topic. Recent systematic reviews may include both more recent and older research on the subject under study. Thus, the publications we examined were all recent. The review utilized information that were gathered somewhere in the range of 2010 and 2021. System: In this paper, we make a modified brain organization to eliminate possible components from extremely high layered datasets. Both a totaled level and an exceptionally fine-grained level of translation are feasible for these highlights. It is basically as easy to grasp non-straight connections as it is a direct relapse. We utilize the method on a dataset for item returns in web based shopping that has 15,555 aspects and 5,659,676 all out exchanges. Result and conclusion: We compare 87 various models to show that our approach not only produces higher predicted accuracy than existing techniques, but is also interpretable. The outcomes show that feature selection is a useful strategy for enhancing scalability. The method is sufficiently abstract to be used with many different analytics datasets

Feature screening algorithm for high dimensional data

Currently, feature screening is becoming an important topic in the fields of machine learning and high-dimensional data analysis. Filtering out irrelevant features from a set of variables is considered to be an important preliminary step that should be performed before any data analysis. Many approaches have been proposed to the same topic after the work of Fan and Lv (J. Royal Stat. Soc., Ser. B. 70 (5), 849–911 (2008)), who introduced the sure screening property. However, the performance of these methods differs from one paper to another. In this work, we aim to add to this list a new algorithm performing feature screening inspired by the Kendall interaction filter (J. Appl. Stat. 50 (7), 1496–1514 (2020)) when the response variable is continuous. The good behavior of our algorithm is proved through a comparison with an existing method, proposed in this work under several simulation scenarios.

Unsupervised spectral feature selection algorithms for high dimensional data

Unsupervised spectral feature selection algorithms for high dimensional data

An Auto-Encoder with Genetic Algorithm for High Dimensional Data: Towards Accurate and Interpretable Outlier Detection

When dealing with high-dimensional data, such as in biometric, e-commerce, or industrial applications, it is extremely hard to capture the abnormalities in full space due to the curse of dimensionality. Furthermore, it is becoming increasingly complicated but essential to provide interpretations for outlier detection results in high-dimensional space as a consequence of the large number of features. To alleviate these issues, we propose a new model based on a Variational AutoEncoder and Genetic Algorithm (VAEGA) for detecting outliers in subspaces of high-dimensional data. The proposed model employs a neural network to create a probabilistic dimensionality reduction variational autoencoder (VAE) that applies its low-dimensional hidden space to characterize the high-dimensional inputs. Then, the hidden vector is sampled randomly from the hidden space to reconstruct the data so that it closely matches the input data. The reconstruction error is then computed to determine an outlier score, and samples exceeding the threshold are tentatively identified as outliers. In the second step, a genetic algorithm (GA) is used as a basis for examining and analyzing the abnormal subspace of the outlier set obtained by the VAE layer. After encoding the outlier dataset’s subspaces, the degree of anomaly for the detected subspaces is calculated using the redefined fitness function. Finally, the abnormal subspace is calculated for the detected point by selecting the subspace with the highest degree of anomaly. The clustering of abnormal subspaces helps filter outliers that are mislabeled (false positives), and the VAE layer adjusts the network weights based on the false positives. When compared to other methods using five public datasets, the VAEGA outlier detection model results are highly interpretable and outperform or have competitive performance compared to current contemporary methods.

A Distributed Attribute Reduction Algorithm for High-Dimensional Data under the Spark Framework

Attribute reduction is an important issue in rough set theory. However, the rough set theory-based attribute reduction algorithms need to be improved to deal with high-dimensional data. A distributed version of the attribute reduction algorithm is necessary to enable it to effectively handle big data. The partition of attribute space is an important research direction. In this paper, a distributed attribution reduction algorithm based on cosine similarity (DARCS) for high-dimensional data pre-processing under the Spark framework is proposed. First, to avoid the repeated calculation of similar attributes, the algorithm gathers similar attributes based on similarity measure to form multiple clusters. And then one attribute is selected randomly as a representative from each cluster to form a candidate attribute subset to participate in the subsequent reduction operation. At the same time, to improve computing efficiency, an improved method is introduced to calculate the attribute dependency in the divided sub-attribute space. Experiments on eight datasets show that, on the premise of avoiding critical information loss, the reduction ability and computing efficiency of DARCS have been improved by 0.32 to 39.61% and 31.32 to 93.79% respectively compared to the distributed version of attribute reduction algorithm based on a random partitioning of the attributes space.

A visual uncertainty analytics approach for weather forecast similarity measurement based on fuzzy clustering

Forecast calibration methods based on historical similar atmospheric state are effective means weather forecast accuracy. Conventional approaches search similar forecasts on the basis of predefined similarity formulas and provide calibration recommendations to forecasters. However, these approaches ignore the uncertainty of similarity measurement, which affects calibration efficacy significantly. This study proposes a similarity weight adaptive algorithm for high-dimensional data on the basis of fuzzy clustering to characterize the uncertainty of similarity measurements. Without prior knowledge, the algorithm computes the uncertainty of the similarity between data in the fuzzy set space iteratively on the basis of membership and then determine weight distribution by maximizing the differentiating ability of each dimension. This study further presents a visual analysis framework on the basis of the weight adaptive algorithm for the exploration of uncertainty in meteorological data and the optimization of similarity measurement method. This framework has coordinated views and intuitive interactions to enable the visualization of the similarity uncertainty distribution and support the iterative visual analysis of similarity weight distribution in each dimension that combines domain knowledge. We illustrate a case study using real-world meteorological data to verify the efficacy of the proposed approach.

The γ-OMP algorithm for feature selection with application to gene expression data

Feature selection for predictive analytics is the problem of identifying a minimal-size subset of features that is maximally predictive of an outcome of interest. To apply to molecular data, feature selection algorithms need to be scalable to tens of thousands of features. In this paper, we propose γ-OMP, a generalisation of the highly-scalable Orthogonal Matching Pursuit feature selection algorithm. γ-OMP can handle (a)various types of outcomes, such as continuous, binary, nominal, time-to-event, (b)discrete (categorical)features, (c)different statistical-based stopping criteria, (d)several predictive models (e.g., linear or logistic regression), (e)various types of residuals, and (f)different types of association. We compare γ-OMP against LASSO, a prototypical, widely used algorithm for high-dimensional data. On both simulated data and several real gene expression datasets, γ-OMP is on par, or outperforms LASSO in binary classification (case-control data), regression (quantified outcomes), and time-to-event data (censored survival times). γ-OMP is based on simple statistical ideas, it is easy to implement and to extend, and our extensive evaluation shows that it is also effective in bioinformatics analysis settings.

A Wrapper Feature Selection Based on Ensemble Learning Algorithm for High Dimensional Data

A Wrapper Feature Selection Based on Ensemble Learning Algorithm for High Dimensional Data

Cross Breed Clustering Algorithm for High Dimensional Data

Clustering plays a major role in machine learning and also in data mining. Deep learning is fast growing domain in present world. Improving the quality of the clustering results by adopting the deep learning algorithms. Many clustering algorithm process various datasets to get the better results. But for the high dimensional data clustering is still an issue to process and get the quality clustering results with the existing clustering algorithms. In this paper, the cross breed clustering algorithm for high dimensional data is utilized. Various datasets are used to get the results.

Optimizing Data Stream Representation: An Extensive Survey on Stream Clustering Algorithms

Analyzing data streams has received considerable attention over the past decades due to the widespread usage of sensors, social media and other streaming data sources. A core research area in this field is stream clustering which aims to recognize patterns in an unordered, infinite and evolving stream of observations. Clustering can be a crucial support in decision making, since it aims for an optimized aggregated representation of a continuous data stream over time and allows to identify patterns in large and high-dimensional data. A multitude of algorithms and approaches has been developed that are able to find and maintain clusters over time in the challenging streaming scenario. This survey explores, summarizes and categorizes a total of 51 stream clustering algorithms and identifies core research threads over the past decades. In particular, it identifies categories of algorithms based on distance thresholds, density grids and statistical models as well as algorithms for high dimensional data. Furthermore, it discusses applications scenarios, available software and how to configure stream clustering algorithms. This survey is considerably more extensive than comparable studies, more up-to-date and highlights how concepts are interrelated and have been developed over time.

Review of Traditional and Ensemble Clustering Algorithms for High Dimensional Data

High-dimensional data is explained by a huge quantity of features, introduces new issues to clustering. The so-named 'high dimensionality', creates initially to explain the common increase in time complexity of several computational issues, so the performances of the general clustering algorithms are unsuccessful. Accordingly, several works have been focused on introducing new techniques and clustering algorithms for handling higher dimensionality data. Regular to all clustering algorithms is the fact with the purpose of they need a various fundamental evaluation of similarity among data objects. However still, the existing clustering algorithms have some open research issues. In this review work, we provide a summary of the result of high-dimensional data space and their implications for various clustering algorithms. It also presents a detailed overview of many clustering algorithms with several types: subspace methods, modelbased clustering, density-based clustering methods; partition based clustering methods, etc., including a more detailed description of recent work of their own advantages and disadvantages for solving higher dimensionality data problem. The scope of the future work to extend the present clustering methods and algorithms are also discussed at end of the work.

Distributed Dimensionality Reconstruction Algorithm for High Dimensional Data in Internet of Brain Things

There are a large number of high-dimensional data in the Internet of Brain Things, and the data is reduced from the high-dimensional to low-dimensional to maintain the similarity between the data, thereby effectively ensuring the operating speed of the Internet of Brain Things. In the traditional method, the distributed dimensionality reduction reconstruction algorithm of the High dimensional data has poor dimensionality reduction effect and serious data distortion after reconstruction. A method for dimensionality reconstruction of high-dimensional data in the Internet of Brain Things is proposed. By using the algorithm of linear discriminant analysis, the projection matrix of high dimensional data is constructed to solve it. According to the solution results, using improved implicit variable model to establish a high dimensional large data dimensionality reduction model for the Internet of Brain Things. The fitness value of data after dimensionality reduction is calculated by quantum immune clonal algorithm, and the optimal individual and optimal solution are determined. The data reconfiguration is realized through the optimal solution marshalling. Experimental results show that the proposed algorithm can effectively improve the dimensionality reduction of High dimensional data in the Internet of Brain Things. After reconstruction, the reconstructed data retain accurate data information, the reliability of reconstructed data is high, and the computational complexity is not high, the need for small storage space, and the advantages of strong promotion ability.

A Novel Clustering and Mining Algorithm for High Dimensional Data based on Uncertainty Criteria and Fuzzy Mathematics

A Novel Clustering and Mining Algorithm for High Dimensional Data based on Uncertainty Criteria and Fuzzy Mathematics

Optimized Feature Selection Algorithm for High Dimensional Data

Objectives: This research paper, based on fuzzy entropy, adapts a new method along with firefly concept, seeks to select quality features. At the same time it removes redundant and irrelevant attributes in high dimensional data. Methods/Statistical Analysis: Feature selection can be understood as a data prepossessing method in order to reduce dimensionality, eliminate irrelevant data and sharpening of accuracy. In the pattern space, fuzzy entropy is used to estimate the knowledge of pattern distribution. The study of the lightning quality of the fireflies has led to the introduction of the Firefly Algorithm for computing models. This work proposes an algorithm for selecting features by integrating fuzzy entropy and firefly algorithm. Our proposed algorithm's performances are analyzed using four different high dimensional data sets WILT, ORL, LC and LTG. Findings: The algorithm which is introduced here is further experimented with four variant data sets and the results shows that this algorithm out performs the traditional feature selection method. Also our proposed algorithm achieves maximum relevance and minimum level of redundancy. The performance metrics such as sensitivity, specificity and accuracy gives significant improvement when compared with existing FCBF algorithm. Applications/Improvements: Our optimized proposed algorithm efficiently improves the performance by eliminating redundant, noisy and insignificant features and can be applied on all high dimensional data sets.

Efficient Feature Subset Selection Algorithm for High Dimensional Data

<p>Feature selection approach solves the dimensionality problem by removing irrelevant and redundant features. Existing Feature selection algorithms take more time to obtain feature subset for high dimensional data. This paper proposes a feature selection algorithm based on Information gain measures for high dimensional data termed as IFSA (Information gain based Feature Selection Algorithm) to produce optimal feature subset in efficient time and improve the computational performance of learning algorithms. IFSA algorithm works in two folds: First apply filter on dataset. Second produce the small feature subset by using information gain measure. Extensive experiments are carried out to compare proposed algorithm and other methods with respect to two different classifiers (Naive bayes and IBK) on microarray and text data sets. The results demonstrate that IFSA not only produces the most select feature subset in efficient time but also improves the classifier performance.</p>

Efficient Feature Subset Selection Algorithm for High Dimensional Data

<p>Feature selection approach solves the dimensionality problem by removing irrelevant and redundant features. Existing Feature selection algorithms take more time to obtain feature subset for high dimensional data. This paper proposes a feature selection algorithm based on Information gain measures for high dimensional data termed as IFSA (Information gain based Feature Selection Algorithm) to produce optimal feature subset in efficient time and improve the computational performance of learning algorithms. IFSA algorithm works in two folds: First apply filter on dataset. Second produce the small feature subset by using information gain measure. Extensive experiments are carried out to compare proposed algorithm and other methods with respect to two different classifiers (Naive bayes and IBK) on microarray and text data sets. The results demonstrate that IFSA not only produces the most select feature subset in efficient time but also improves the classifier performance.</p>

Efficiency and Effectiveness of Clustering Algorithms for High Dimensional Data

ering high dimensional data is challenging due to its dimensionality problem and it affects time complexity and accuracy of clustering methods. This paper presents the F- measure and Euclidean distance based performance efficiency and effectiveness of K-means and Agglomerative hierarchical clustering methods on Text and Microarray datasets by varying cluster values. Efficiency concerns about computational time required to build up dataset and effectiveness concerns about accuracy to cluster the data. Experimental results on different datasets demonstrate that K- means clustering algorithm is favourable in terms of effectiveness where as Agglomerative hierarchical clustering is efficient in time for text datasets used for empirical study. KeywordsK-means, Agglomerative hierarchical, F-measure, Precision, Recall.

Scalable Nearest Neighbor Algorithms for High Dimensional Data

For many computer vision and machine learning problems, large training sets are key for good performance. However, the most computationally expensive part of many computer vision and machine learning algorithms consists of finding nearest neighbor matches to high dimensional vectors that represent the training data. We propose new algorithms for approximate nearest neighbor matching and evaluate and compare them with previous algorithms. For matching high dimensional features, we find two algorithms to be the most efficient: the randomized k-d forest and a new algorithm proposed in this paper, the priority search k-means tree. We also propose a new algorithm for matching binary features by searching multiple hierarchical clustering trees and show it outperforms methods typically used in the literature. We show that the optimal nearest neighbor algorithm and its parameters depend on the data set characteristics and describe an automated configuration procedure for finding the best algorithm to search a particular data set. In order to scale to very large data sets that would otherwise not fit in the memory of a single machine, we propose a distributed nearest neighbor matching framework that can be used with any of the algorithms described in the paper. All this research has been released as an open source library called fast library for approximate nearest neighbors (FLANN), which has been incorporated into OpenCV and is now one of the most popular libraries for nearest neighbor matching.

Integrated constraint based clustering algorithm for high dimensional data

Dimension selection, dimension weighting and data assignment are three circular dependent essential tasks for high dimensional data clustering and each such task is challenging. To meet the challenge of high dimensional data clustering, constraints have been employed in several previous works. However, these constraint based algorithms use constraints to help accomplish only one of the three essential tasks. In this paper, we propose an integrated constraint based clustering (ICBC) algorithm for high dimensional data, which exploits constraints to accomplish all the three essential tasks. Firstly we generalize the dimension selection technique of CDCDD algorithm such that dimension selection and dimension weighting could be accomplished simultaneously. Then we propose a novel constraint based data assignment method which assigns all the data points to their corresponding clusters based on the selected dimensions and dimension weights. Finally we use an optimization technique to iteratively refine the initial dimension weights and centroids, and reassign data accordingly till convergence. Experimental results on both synthetic data sets and real data sets show that our proposed ICBC algorithm outperforms typical unsupervised algorithms and other constraint based algorithms in terms of accuracy. ICBC also outperforms the other algorithms that implement dimension selection in terms of efficiency and scalability.

Fuzzy C Means Clustering Algorithm for High Dimensional Data Using Feature Subset Selection Technique

Feature choice involves characteristic a set of the foremost helpful options that produces compatible results because the original entire set of options. A feature choice rule is also evaluated from each the potency and effectiveness points of read. Whereas the potency considerations the time needed to search out a set of options, the effectiveness is expounded to the standard of the set of options. Supported these criteria, an economical Fuzzy C Means (FCM) is projected and by experimentation evaluated in this paper. The quick rule works in 2 steps. Within the commencement, options area unit divided into clusters by exploitation graph-theoretic cluster ways. Within the second step, the foremost representative feature that's powerfully associated with target categories is chosen from every cluster to create a set of options. Options in numerous clusters area unit comparatively freelance; the clustering-based strategy of quick incorporates a high chance of manufacturing a set of helpful and independent options. To make sure the potency of quick, we have a tendency to adopt the economical Fuzzy C Means (FCM) cluster technique. The potency associated effectiveness of the quick rule area unit evaluated through an empirical study. in depth experiments area unit dole out to match quick and a number of other representative feature choice algorithms, namely, FCBF, ReliefF, CFS, Consist, and FOCUS-SF, with relevancy four kinds of well- known classifiers, namely, the chance primarily based Naive Thomas Bayes, the tree-based C4.5, the instance- based IB1,and also the rule-based liquidator before and once feature choice. The results, on thirty five in public on the market real-world high-dimensional image, microarray, and text knowledge, demonstrate that the quick not solely produces smaller subsets of options however additionally improves the performances of the four kinds of classifiers. Index Terms: feature subset selection, relevance, redundancy and high dimensionality.