Patterns In High-dimensional Datasets Research Articles

Detecting the ultra low dimensionality of real networks

Reducing dimension redundancy to find simplifying patterns in high-dimensional datasets and complex networks has become a major endeavor in many scientific fields. However, detecting the dimensionality of their latent space is challenging but necessary to generate efficient embeddings to be used in a multitude of downstream tasks. Here, we propose a method to infer the dimensionality of networks without the need for any a priori spatial embedding. Due to the ability of hyperbolic geometry to capture the complex connectivity of real networks, we detect ultra low dimensionality far below values reported using other approaches. We applied our method to real networks from different domains and found unexpected regularities, including: tissue-specific biomolecular networks being extremely low dimensional; brain connectomes being close to the three dimensions of their anatomical embedding; and social networks and the Internet requiring slightly higher dimensionality. Beyond paving the way towards an ultra efficient dimensional reduction, our findings help address fundamental issues that hinge on dimensionality, such as universality in critical behavior.

Automatic Lithofacies Classification with t-SNE and K-Nearest Neighbors Algorithm

One of the critical processes in the exploration of hydrocarbons is the identification and prediction of lithofacies that constitute the reservoir. One of the cheapest and most efficient ways to carry out that process is from the interpretation of well log data, which are often obtained continuously and in the majority of drilled wells. The main methodologies used to correlate log data to data obtained in well cores are based on statistical analyses, machine learning models and artificial neural networks. This study aims to test an algorithm of dimension reduction of data together with an unsupervised classification method of predicting lithofacies automatically. The performance of the methodology presented was compared to predictions made with artificial neural networks. We used the t-Distributed Stochastic Neighbor Embedding (t-SNE) as an algorithm for mapping the wells logging data in a smaller feature space. Then, the predictions of facies are performed using a KNN algorithm. The method is assessed in the public dataset of the Hugoton and Panoma fields. Prediction of facies through traditional artificial neural networks obtained an accuracy of 69%, where facies predicted through the t-SNE + K-NN algorithm obtained an accuracy of 79%. Considering the nature of the data, which have high dimensionality and are not linearly correlated, the efficiency of t SNE+KNN can be explained by the ability of the algorithm to identify hidden patterns in a fuzzy boundary in data set. It is important to stress that the application of machine learning algorithms offers relevant benefits to the hydrocarbon exploration sector, such as identifying hidden patterns in high-dimensional datasets, searching for complex and non-linear relationships, and avoiding the need for a preliminary definition of mathematic relations among the model’s input data.

Outlying Sequence Detection in Large Data Sets: A data-driven approach

Outliers refer to observations that do not conform to the expected patterns in high-dimensional data sets. When such outliers signify risks (e.g., in fraud detection) or opportunities (e.g., in spectrum sensing), harnessing the costs associated with the risks or missed opportunities necessitates mechanisms that can identify them effectively. Designing such mechanisms involves striking an appropriate balance between reliability and cost of sensing, as two opposing performance measures, where improving one tends to penalize the other. This article poses and analyzes outlying sequence detection in a hypothesis testing framework under different outlier recovery objectives and different degrees of knowledge about the underlying statistics of the outliers.

A new method for mining disjunctive emerging patterns in high-dimensional datasets using hypergraphs

We investigate in this paper the problem of mining disjunctive emerging patterns in high-dimensional biomedical datasets. Disjunctive emerging patterns are sets of features that are very frequent among samples of a target class, cases in a case–control study, for example, and are very rare among all other samples. We, for the very first time, demonstrate that this problem can be solved using minimal transversals in a hypergraph. We propose a new divide-and-conquer algorithm that enables us to efficiently compute disjunctive emerging patterns in parallel and distributed environments. We conducted experiments using real-world microarray gene expression datasets to assess the performance of our approach. Our results show that our approach is more efficient than the state-of-the-art solution available in the literature. In this sense, we contribute to the area of bioinformatics and data mining by providing another useful alternative to identify patterns distinguishing samples with different class labels, such as those in case–control studies, for example.

Nuclear Potential Clustering As a New Tool to Detect Patterns in High Dimensional Datasets

We present a new approach for the clustering of high dimensional data without prior assumptions about the structure of the underlying distribution. The proposed algorithm is based on a concept adapted from nuclear physics. To partition the data, we model the dynamic behaviour of nucleons interacting in an N-dimensional space. An adaptive nuclear potential, comprised of a short-range attractive (strong interaction) and a long-range repulsive term (Coulomb force) is assigned to each data point. By modelling the dynamics, nucleons that are densely distributed in space fuse to build nuclei (clusters) whereas single point clusters repel each other. The formation of clusters is completed when the system reaches the state of minimal potential energy. The data are then grouped according to the particles' final effective potential energy level. The performance of the algorithm is tested with several synthetic datasets showing that the proposed method can robustly identify clusters even when complex configurations are present. Furthermore, quantitative MRI data from 43 multiple sclerosis patients were analyzed, showing a reasonable splitting into subgroups according to the individual patients' disease grade. The good performance of the algorithm on such highly correlated non-spherical datasets, which are typical for MRI derived image features, shows that Nuclear Potential Clustering is a valuable tool for automated data analysis, not only in the MRI domain.

Efficient colossal pattern mining in high dimensional datasets

‘Frequent pattern mining’ is considered as an important data mining problem which has been extensively studied over the last decade. There are a large number of algorithms which have been developed for frequent pattern mining on a traditional commercial dataset which usually contains a huge number of transactions besides a small number of items in each transaction. The advent of bioinformatics contributed to the development of new form of datasets – called high dimensional – which are characterized by small number of transactions and large number of items in each transaction. The running time of traditional algorithms increases exponentially with increasing average transaction length, thus these algorithms cannot be suitable for the high dimensional datasets. On the other hand, the mining algorithms on high dimensional datasets create a very large output set as result which includes small and mid-size frequent patterns which do not bear any useful information for scientists. Colossal pattern mining is described as a solution to reduce the amount of output set of mining patterns. Due to ignoring the mining of the small and mid-sized patterns, mining process speed is increased in colossal patterns mining algorithms. Therefore, only very large (colossal) patterns are extracted and mined in this approach. In this paper we represent an efficient vertical bottom up method to conduct mining of frequent colossal patterns in high dimensional datasets. In our algorithm, we use a bit matrix to compress the dataset and make it easy to use in mining process. Our experimental result shows that our algorithm attains very good mining efficiencies on various input datasets. Furthermore, our performance study shows that this algorithm outperforms substantially the best former algorithms.

STUDENTJAMA. The challenges of whole-genome approaches to common diseases.

RECENT TECHNOLOGICAL ADVANCES MAY SOON ENABLE THE study of hundreds of thousands of human singlenucleotide polymorphisms (SNPs) at the population level. Because strategies for analyzing these data have not kept pace with the laboratory methods that generate the data, however, it is unlikely that these advances will immediately lead to an improved understanding of the genetic contribution to common human diseases. In addition, the underlying genetics of common diseases such as sporadic breast cancer or essential hypertension are far more complex than that of rare mendelian diseases such as cystic fibrosis and sickle cell anemia. As a result, several important technical challenges will need to be overcome to identify susceptibility genes that can be used to improve the prevention, diagnosis, and treatment of common diseases. These challenges include developing statistical methods to analyze genetic data, selecting appropriate genetic variables, and interpreting interactions between individual genes. Although specific DNA sequence variations have been linked to a variety of rare diseases, they have not been as informative for predicting the onset of more common conditions. This difference is illustrated when comparing familial (rare) and sporadic (common) forms of breast cancer. Women with a strong family history for breast cancer, for example, can be tested for specific mutations in the BRCA1 and BRCA2 genes, which result in 50% chance of developing the disease. The risk for sporadic breast cancer, however, cannot adequately be predicted by DNA sequence variations alone. Similar to other common diseases, the underlying genetic etiology of sporadic breast cancer probably involves many genes, each of which influences susceptibility primarily through nonadditive interactions with other genes (termed “epistasis”) and with environmental factors. It is possible that interactions between genes are ubiquitous in the underlying etiology of most common diseases, given the complex molecular interactions that occur during biological processes such as transcription, translation, and signal transduction. Knowledge about DNA sequence variations from many different genes, in the context of environmental exposure, may thus be necessary to apply genetic information to human health. This problem suggests several challenges in identifying susceptibility genes from the entire human genome. First, powerful statistical and computational methods will need to be developed to model the relationship between combinations of SNPs and disease susceptibility. Due to the large number of potential genotypes, analyzing SNP combinations is a far more difficult task than assessing each SNP individually. The difficulty increases exponentially with the number of SNPs under consideration. For example, while a single SNP with 3 genotypes has only 3 categories, 2 SNPs with 3 genotypes will have 9 possible 2-locus genotype combinations. With 3 SNPs, the number of combinations increases to 27. As the number of possible combinations increases, it may become impossible to recruit enough subjects into epidemiological studies to represent every possible genotypic combination. This problem has been referred to as the “curse of dimensionality.” This limitation may be partially addressed with statistical and modeling approaches. Traditional parametric statistical approaches (ie, methods that compute estimates of population parameters) such as logistic regression do not deal with the dimensionality problem very effectively and are thus not well suited to detecting and characterizing gene-gene interactions. This is due to the inaccuracy of parameter estimates when there are too many variables in relation to the amount of data. By contrast, nonparametric “data-mining” methods usually do not require a prespecified statistical hypothesis and thus are better suited to search for trends or patterns in high-dimensional data sets. Although data-mining techniques such as multifactor dimensionality reduction (MDR) and neural networks may be more powerful than parametric statistical approaches, they have their own limitations. Neural network models, for example, can be very difficult to interpret and their results may not be intuitive. Furthermore, data-mining approaches may be influenced by chance patterns in data, which can result in false-positive results. A second challenge is the selection of genetic variables that should be included for analysis. If complex interactions between genes explain most of the heritability of common diseases, then combinations of SNPs will need to be evaluated from a list of hundreds of thousands of candidates. When single, functional polymorphisms each have a statistically detectable independent effect, each polymorphism can be evaluated individually for an association with disease, followed by an analysis of gene-gene interactions that considers only those polymorphisms. This greatly reduces the number of potential combinations of variables that must be examined. When SNPs do not have independent effects, however, it is impossible for most current computer technologies to analyze the resulting astronomical number of possible combinations. For instance, if 300000 SNPs have been measured at a density of 1 SNP every 10 kilobases (kb), and if 10 statistical evaluations can be computed each second, then evaluation of each individual SNP would require 30000 seconds (ie, 8.3 hours) of computer time. Exhaustive evaluation of the approximately 4 10