Large Volumes Of High-dimensional Data Research Articles

Time and memory scalable algorithms for clustering tendency assessment of big data

Large-volume and high-dimensional big datasets are being generated quickly. They are expected to provide data-driven solutions for various pressing challenges such as cybersecurity, credit card fraud, network intrusion detection, etc. The visual assessment of clustering tendency (VAT) family of algorithms provides an excellent tool for assessing clustering tendency and subsequent clustering of these novel datasets to extract groups and anomalies to better understand the data generation phenomenon. However, VAT and improved VAT (iVAT) algorithms are plagued with O(n2) time and space complexity as they use Prim's algorithm for building Euclidean minimum spanning tree (EMST) of the data points, leaving it inapplicable to large datasets. This paper develops three novel time- and memory-scalable algorithms for fast computation of VAT EMST by reducing the search space of the following possible EMST edge and using an efficient data structure, the k-d tree. Next, we develop a novel approach to compute iVAT reordered dissimilarity image (RDI) in a novel memory-preserving manner using the EMST edge lengths calculated previously without having to compute memory intensive n×n VAT RDI. We experimented on several synthetic and real-life datasets to showcase that the proposed approaches can help quickly assess their clustering tendency in a memory-saving manner. We also demonstrate the applicability of the proposed methods for the anomaly detection task in large volumes of high-dimensional data.

Tracking Pandemics: A MEC-Enabled IoT Ecosystem with Learning Capability

The COVID-19 pandemic has resulted in unprecedented challenges to global society and the healthcare system in particular. The main objective of this article is to introduce an end-to-end Internet of Things (IoT) ecosystem for healthcare that uses an open source hardware and interoperable IoT standard for eHealth monitoring in general, and COVID-19 symptoms (e.g., fever, coughing, and fatigue) in particular. The system is designed to monitor the physical conditions of human subjects and send the data to a hierarchical multi-access edge computing (MEC) framework. Such a system is expected to be cognizant, taskable (i.e., tasks can be assigned to any computing process in the system), and adaptable. To this end, we demonstrate how a learning method can be introduced in the ecosystem to achieve taskability and efficiency. Specifically, the proposed system utilizes a shared representation learning process to extract actionable information from large volumes of high-dimensional data obtained from IoT edge devices. These edge devices are enabled with tri-sensors for real-time monitoring of COVID-19 symptoms. The feasibility of the proposed system is evaluated by testing real datasets.

A Rapid Hybrid Clustering Algorithm for Large Volumes of High Dimensional Data

Clustering large volumes of high-dimensional data is a challenging task. Many clustering algorithms have been developed to address either handling datasets with a very large sample size or with a very high number of dimensions, but they are often impractical when the data is large in both aspects. To simultaneously overcome both the ‘curse of dimensionality’ problem due to high dimensions and scalability problems due to large sample size, we propose a new fast clustering algorithm called FensiVAT. FensiVAT is a hybrid, ensemble-based clustering algorithm which uses fast data-space reduction and an intelligent sampling strategy. In addition to clustering, FensiVAT also provides visual evidence that is used to estimate the number of clusters (cluster tendency assessment) in the data. In our experiments, we compare FensiVAT with nine state-of-the-art approaches which are popular for large sample size or high-dimensional data clustering. Experimental results suggest that FensiVAT, which can cluster large volumes of high-dimensional datasets in a few seconds, is the fastest and most accurate method of the ones tested.

Sketched Subspace Clustering

The immense amount of daily generated and communicated data presents unique challenges in their processing. Clustering, the grouping of data without the presence of ground-truth labels, is an important tool for drawing inferences from data. Subspace clustering (SC) is a relatively recent method that is able to successfully classify nonlinearly separable data in a multitude of settings. In spite of their high clustering accuracy, SC methods incur prohibitively high computational complexity when processing large volumes of high-dimensional data. Inspired by random sketching approaches for dimensionality reduction, the present paper introduces a randomized scheme for SC, termed Sketch-SC, tailored for large volumes of high-dimensional data. Sketch-SC accelerates the computationally heavy parts of state-of-the-art SC approaches by compressing the data matrix across both dimensions using random projections, thus enabling fast and accurate large-scale SC. Performance analysis as well as extensive numerical tests on real data corroborate the potential of Sketch-SC and its competitive performance relative to state-of-the-art scalable SC approaches.

Estimating the intrinsic dimension of datasets by a minimal neighborhood information

Analyzing large volumes of high-dimensional data is an issue of fundamental importance in data science, molecular simulations and beyond. Several approaches work on the assumption that the important content of a dataset belongs to a manifold whose Intrinsic Dimension (ID) is much lower than the crude large number of coordinates. Such manifold is generally twisted and curved; in addition points on it will be non-uniformly distributed: two factors that make the identification of the ID and its exploitation really hard. Here we propose a new ID estimator using only the distance of the first and the second nearest neighbor of each point in the sample. This extreme minimality enables us to reduce the effects of curvature, of density variation, and the resulting computational cost. The ID estimator is theoretically exact in uniformly distributed datasets, and provides consistent measures in general. When used in combination with block analysis, it allows discriminating the relevant dimensions as a function of the block size. This allows estimating the ID even when the data lie on a manifold perturbed by a high-dimensional noise, a situation often encountered in real world data sets. We demonstrate the usefulness of the approach on molecular simulations and image analysis.

Parallel high-dimensional multi-objective feature selection for EEG classification with dynamic workload balancing on CPU–GPU architectures

Many bioinformatics applications that analyse large volumes of high-dimensional data comprise complex problems requiring metaheuristics approaches with different types of implicit parallelism. For example, although functional parallelism would be used to accelerate evolutionary algorithms, the fitness evaluation of the population could imply the computation of cost functions with data parallelism. This way, heterogeneous parallel architectures, including central processing unit (CPU) microprocessors with multiple superscalar cores and accelerators such as graphics processing units (GPUs) could be very useful. This paper aims to take advantage of such CPU–GPU heterogeneous architectures to accelerate electroencephalogram classification and feature selection problems by evolutionary multi-objective optimization, in the context of brain computing interface tasks. In this paper, we have used the OpenCL framework to develop parallel master-worker codes implementing an evolutionary multi-objective feature selection procedure in which the individuals of the population are dynamically distributed among the available CPU and GPU cores.

AutoWeka: toward an automated data mining software for QSAR and QSPR studies.

In biology and chemistry, a key goal is to discover novel compounds affording potent biological activity or chemical properties. This could be achieved through a chemical intuition-driven trial-and-error process or via data-driven predictive modeling. The latter is based on the concept of quantitative structure-activity/property relationship (QSAR/QSPR) when applied in modeling the biological activity and chemical properties, respectively, of compounds. Data mining is a powerful technology underlying QSAR/QSPR as it harnesses knowledge from large volumes of high-dimensional data via multivariate analysis. Although extremely useful, the technicalities of data mining may overwhelm potential users, especially those in the life sciences. Herein, we aim to lower the barriers to access and utilization of data mining software for QSAR/QSPR studies. AutoWeka is an automated data mining software tool that is powered by the widely used machine learning package Weka. The software provides a user-friendly graphical interface along with an automated parameter search capability. It employs two robust and popular machine learning methods: artificial neural networks and support vector machines. This chapter describes the practical usage of AutoWeka and relevant tools in the development of predictive QSAR/QSPR models. The software is freely available at http://www.mt.mahidol.ac.th/autoweka.

Locality-Sensitive Hashing for Data with Categorical and Numerical Attributes Using Dual Hashing

Locality-sensitive hashing techniques have been developed to efficiently handle nearest neighbor searches and similar pair identification problems for large volumes of high-dimensional data. This study proposes a locality-sensitive hashing method that can be applied to nearest neighbor search problems for data sets containing both numerical and categorical attributes. The proposed method makes use of dual hashing functions, where one function is dedicated to numerical attributes and the other to categorical attributes. The method consists of creating indexing structures for each of the dual hashing functions, gathering and combining the candidates sets, and thoroughly examining them to determine the nearest ones. The proposed method is examined for a few synthetic data sets, and results show that it improves performance in cases of large amounts of data with both numerical and categorical attributes.

A global geometric framework for nonlinear dimensionality reduction.

Scientists working with large volumes of high-dimensional data, such as global climate patterns, stellar spectra, or human gene distributions, regularly confront the problem of dimensionality reduction: finding meaningful low-dimensional structures hidden in their high-dimensional observations. The human brain confronts the same problem in everyday perception, extracting from its high-dimensional sensory inputs-30,000 auditory nerve fibers or 10(6) optic nerve fibers-a manageably small number of perceptually relevant features. Here we describe an approach to solving dimensionality reduction problems that uses easily measured local metric information to learn the underlying global geometry of a data set. Unlike classical techniques such as principal component analysis (PCA) and multidimensional scaling (MDS), our approach is capable of discovering the nonlinear degrees of freedom that underlie complex natural observations, such as human handwriting or images of a face under different viewing conditions. In contrast to previous algorithms for nonlinear dimensionality reduction, ours efficiently computes a globally optimal solution, and, for an important class of data manifolds, is guaranteed to converge asymptotically to the true structure.