Large-scale Multidimensional Data Research Articles

An Exploration of the Application of Principal Component Analysis in Big Data Processing

Abstract With the arrival of the significant data era, efficiently processing large-scale multidimensional data has become challenging. As a powerful data dimensionality reduction tool, Principal Component Analysis (PCA) plays a vital role in big data processing, especially in information extraction and data simplification, showing unique advantages. The research aims to simplify the data processing process and improve the data processing efficiency by PCA method. The research method adopts the basic theory of PCA, the improvement of the weighted principal component analysis algorithm, and standardized and homogenized data processing techniques to process large-scale multidimensional data sets. The results show that the data dimensionality is significantly reduced after using PCA, for example, in the Analysis of the earnings quality of listed companies in the e-commerce industry, the cumulative variance contribution rate of the first four principal components extracted by PCA reaches 81.623%, which effectively removes the primary information of the original data. PCA not only reduces the complexity of the data, but also retains a large amount of crucial information, which is a significant application value for the processing of big data, especially in the fields of data compression and pattern recognition.

Back temporal autoregressive matrix factorization for high-dimensional time series prediction

While prediction for large-scale multidimensional spatio-temporal data is widely used in applications such as business analytics and open source metrics, service data collected from these applications often have values that are missing from the data. The accuracy of prediction results requires that the prediction model used is highly robust and scalable, something that is typically lacking in traditional prediction models. To this end, we propose a back temporal autoregressive matrix factorization (BTAMF) framework, which can construct an efficient prediction model for time series data with missing values by mining the trend and periodic characteristics of the prediction results. It first uses the target data and its left neighbors to construct trend autoregressive and then uses the target data and its associated periodic data to construct periodic autoregressive to mine the trend and periodicity properties of time series data. We then integrate the tensor/matrix factorization and both autoregressive models into a single optimization model for missing value completion and prediction through optimization of the objective functions. This scheme, which exploits global and local information of the target data, can directly predict the data without pre-completing the missing values, which provides a strong guarantee of the temporal integrity of the time series data. To validate the efficient completion and prediction effects of BTAMF in real-world scenarios, we select ten real-world behavioral datasets from GitHub and seven comparison models for experiments. Our experimental results show that BTAMF has a stronger advantage over current state-of-the-art methods in terms of scalability and prediction accuracy.

Dataopsy: Scalable and Fluid Visual Exploration using Aggregate Query Sculpting.

We present aggregate query sculpting (AQS), a faceted visual query technique for large-scale multidimensional data. As a "born scalable" query technique, AQS starts visualization with a single visual mark representing an aggregation of the entire dataset. The user can then progressively explore the dataset through a sequence of operations abbreviated as P6: pivot (facet an aggregate based on an attribute), partition (lay out a facet in space), peek (see inside a subset using an aggregate visual representation), pile (merge two or more subsets), project (extracting a subset into a new substrate), and prune (discard an aggregate not currently of interest). We validate AQS with DATAOPSY, a prototype implementation of AQS that has been designed for fluid interaction on desktop and touch-based mobile devices. We demonstrate AQS and Dataopsy using two case studies and three application examples.

Ganos

Recently, the trend of developing digital twins for smart cities has driven a need for managing large-scale multidimensional, dynamic, and scene-oriented spatial data. Due to larger data scale and more complex data structure, queries over such data are more complicated and expensive than those on traditional spatial data, which poses challenges to the system efficiency and deployment costs. The existing spatial databases have limited support in both data types and operations. Therefore, a new-generation spatial database with excellent performance and effective deployment costs is needed. This paper presents Ganos, a cloud-native spatial database engine of PolarDB for PostgreSQL that is developed by Alibaba Cloud, to efficiently manage multidimensional, dynamic, and scene-oriented spatial data. Ganos models 3D space and spatio-temporal dynamics as first-class citizens. Also, it natively supports spatial/spatio-temporal data types such as 3DMesh, Trajectory, Raster, PointCloud, etc. Besides, it implements a novel extended-storage mechanism that utilizes cloud-native object storage to reduce storage costs and enable uniform operations on the data in different storages. To facilitate processing "big" queries, Ganos extends PolarDB and provides spatial-oriented multi-level parallelism under the architecture of decoupling compute from storage in cloud-native databases, which achieves elasticity and excellent query performance. We demonstrate Ganos in real-life case studies. The performance of Ganos is evaluated using real datasets, and promising results are obtained. Finally, based on the extensive deployment and application of Ganos, the lessons learned from our customers and the expectations of modern cloud applications for new spatial database features are discussed.

Big Data Analytics for Mobility Prediction and Its Classification

Kern forecasting is the detection of which customers may leave a service or cancel a subscription for a service. This is an important forecast for many businesses because gaining new customers will cost more than retaining existing customers. The Retail Banking Seer Forecast is an AI-based model that helps customers assess the likelihood of your bank being blocked. Odor is a good indicator of growth. By comparing and analyzing these two metrics that monitor lost customers, and growth rates, new customers, Kern rates accurately tell you how much your business is growing over time. If growth is greater than recession, you can say that your business is growing. With increasing skill requirements and requirements for quality of experience, mobility forecasting has become widely used for mobile communication and has become one of the key processors that use historical transport information to predict the future locations of traffic users. Predictive maintenance refers to the use of data-driven, efficient maintenance methods, via designed to analyze the Status and maintenance of equipment When Predict what needs to be done. Forecasting maintenance is a type of maintenance that directly monitors the health, condition and performance of an asset in real time. Predictive maintenance is aimed at minimizing costly and unexpected breakdowns and gives the manufacturer the opportunity to plan maintenance around their own production schedule. Some examples of the use of forecast maintenance and forecast maintenance sensors include vibration analysis, oil analysis, Includes thermal imaging and equipment monitoring. This approach guarantees cost of savings in routine or time-based preventive maintenance, due to the tasks are only done when guaranteed. Predictive maintenance techniques to determine the condition of equipment in service Designed to help and evaluate when maintenance should be done. Big Data Analytics is structured, semi-structured, Is the use of advanced analytical techniques against very large, diverse datasets. and an unstructured data of various sizes ranging from different sources to terabytes to zeta bytes. Choosing a career in the field of big data and Analytics can be an exciting career endeavor, and it can be the type of role you are trying to find. Machine learning (ML) is a prediction that considers large-scale multidimensional data from a variety of sources allows you to create models. Several studies have been conducted on the use of ML algorithms to predict road traffic. Traffic forecast is floating car data and traffic flow, average traffic speed and based on historical traffic data such as traffic events The task is to predict real-time traffic information.

Compressed Tensor Completion: A Robust Technique for Fast and Efficient Data Reconstruction in Wireless Sensor Networks

In recent times, Compressive sensing (CS) based data collection has become an essential technique for Wireless Sensor Networks (WSN) because of its benefits of low data communication costs. This work develops a Compressed Tensor Completion (CTC) approach to recover the original signals effectively and with better accuracy from the randomly projected sea surface temperature signals. Tensor learning approaches show their superiority towards identifying and extracting correlations in spatio-temporal compressed signals. Initially, this research frames the 3D tensor. Subsequently, we implement compressed matrix factorization using Randomised Singular Value Decomposition (RSVD). This research utilizes a general strategy to define conditions between the matrices and tensor environments to obtain accurate signal measurements. The proposed approach uses Pacific Sea surface temperature data and brain image data for simulation. The simulation results show that the proposed method yields higher accuracy at a lower sampling rate, minimizing data transmission costs and extending sensor lifespan. In addition, CTC has outperformed other approaches in classifying large-scale multi-dimensional sensor data.

Genetic, Pharmacogenomic, and Immune Landscapes of Enhancer RNAs Across Human Cancers.

This study provides insights into the impact of the enhancer RNA (eRNA) landscape on patient outcome, the immune microenvironment, and drug response across cancers and develops a multidimensional data portal to facilitate eRNA research.

A pan-cancer immunogenomic atlas for immune checkpoint blockade immunotherapy.

The ability to identify robust genomic signatures that predict response to immune checkpoint blockade is restricted by limited sample sizes and ungeneralizable performance across cohorts. To address these challenges, we established Cancer-Immu (http://bioinfo.vanderbilt.edu/database/Cancer-Immu/) a comprehensive platform that integrates large-scale multidimensional omics data, including genetic, bulk, and single-cell transcriptomic, proteomic, and dynamic genomic profiles, with clinical phenotypes to explore consistent and rare immunogenomic connections. Currently Cancer-Immu has incorporated data for 3,652 samples for 16 cancer types. It provides easy access to immunogenomic data and empowers researchers to translate omics datasets into biological insights and clinical applications.

MLkit: A machine-learning-powered automatic workflow for classification of cancer samples.

e13583 Background: In the era of data explosion, precision classification of cancer samples based on multi-dimensional medical data provides more insights into disease mechanism and useful hints on clinical treatment associated with tissue of origin, recurrence tendency and prognostic of chemotherapy or immunotherapy. We developed an automatic workflow MLkit to select features from large-scale multi-dimensional medical data and conduct classification through various machine learning techniques. Methods: MLkit is an automatic and one-stop workflow for classification of cancer samples with four modules: preprocessing (missing data remove or imputation and feature standardization), feature selection (unsupervised multi-statistics and supervised multiple machine estimators with recursive feature elimination and cross-validation), modeling (hyper-parameter, performance evaluation and probability calibration) and prediction. Most of current machine learning algorithms were implemented in this workflow, including linear model (logistic regression, ridge regression and stochastic gradient descent), ensemble model (gradient boosting, random forest, xgboost, catboost, lightgmb and stacking), support vector kernel (linear and non-linear), naive Bayes, k-nearest neighbors and multi-layer perceptron neural network. To evaluate the performance of this workflow, we utilized it to fit a model used for prediction of tissue of origin based on 450K DNA methylation data of 2,210 samples from lung, kidney and breast cancer patients collected in TCGA. Results: MLkit performed well in the prediction of tissue of origin for independent validation sets of cancer patients with stable feature selection, automatic hyper-parameters and efficient probability calibration, in which the model achieved AUCs ranged from 0.85 to 0.96. In addition, we also utilized this workflow on extensive real world data and most of results showed superior accuracy and stable performance. Conclusions: MLkit facilitates automated and one-stop classification of cancer samples using machine learning algorithms. It can be operated with simple command line, making it accessible to a broad range of users. The well performance of this workflow based on multi-dimensional medical data can help to improve the discovery of tumor biomarker and optimize clinical follow-up and therapeutic treatment for cancer patients.

Ensemble Pruning via Quadratic Margin Maximization

Ensemble models refer to methods that combine a typically large number of weak learners into a stronger composite model. The output of an ensemble method is the result of fitting a base-learning algorithm to a given data set, and obtaining diverse answers by re-weighting the observations or by re-sampling them using a given probabilistic selection. A key challenge of using ensembles in large-scale multidimensional data lies in the complexity and the computational burden associated with them. The models created by ensembles are often difficult, if not impossible, to interpret and their implementation requires more computational power than individual learning algorithms. Recent research effort in the field has concentrated on reducing ensemble size, while maintaining predictive accuracy. We propose a method to prune an ensemble solution by optimizing its margin distribution, while increasing its diversity. The proposed algorithm results in an ensemble that uses only a fraction of the original weak learners, with generally improved estimated generalization performance. We analyze and test our method on both synthetic and real data sets. The analysis shows that the proposed method compares favorably to the original ensemble solutions and to other existing ensemble pruning methodologies.

RETRACTED: Research on multidimensional information collection algorithm of marine buoy wireless communication network

The large-scale multidimensional database and data warehouse have become a common phenomenon in various commercial and scientific applications. These data warehouses, several tens of millions of groups, including the respective tens or hundreds of the classification size. One of these databases' main challenges is to extract the critical data's meaning: structure from them, looking for a pattern, will discover a derived cause-and-effect relationship. Promising technology for the analysis of these multidimensional database is visualized. (A project of the plane) in addition to visualizing multidimensional data. Visual analysis partially assigns separately simply and intuitively estimate the similarities and differences between objects. However, when projecting the multifaceted data on a plane, a disadvantage of this analysis is that some information is inevitably lost. The agreed methods, have a combination of the process of classification and visualization. A marine radio is the fastest solution to resolve the sea of systematically interconnected problems; it is still in the early stages of development. Communicate with the marine environment; marine Internet applications are currently on the market. Development of communication systems and challenges of the ocean, need a discussion on the development of maritime radio communication networks.

High-Performance Tucker Factorization on Heterogeneous Platforms

Given large-scale multi-dimensional data (e.g., (user, movie, time; rating) for movie recommendations), how can we extract latent concepts/relations of such data? Tensor factorization has been widely used to solve such problems with multi-dimensional data, which are modeled as tensors. However, most tensor factorization algorithms exhibit limited scalability and speed since they require huge memory and heavy computational costs while updating factor matrices. In this paper, we propose GTA, a general framework for Tucker factorization on heterogeneous platforms. GTA performs alternating least squares with a row-wise update rule in a fully parallel way, which significantly reduces memory requirements for updating factor matrices. Furthermore, GTA provides two algorithms: GTA-PART for partially observable tensors and GTA-FULL for fully observable tensors, both of which accelerate the update process using GPUs and CPUs. Experimental results show that GTA exhibits $5.6 \sim 44.6 \times$5.6∼44.6× speed-up for large-scale tensors compared to the state-of-the-art. In addition, GTA scales near linearly with the number of GPUs and computing nodes used for experiments.

VASEM: visual analytics system for electron microscopy data bank

Electron Microscopy Data Bank (EMDB) provides biological researchers with a platform for uploading and downloading data obtained mainly by electron microscopy. With the continuous expansion of the total amount of data stored in EMDB in recent years, its influence has also been continuously improved. In order to make it easier for domain scientists to make better use of these data, we analyzed the EMDB meta-information structure and developed a visual analytics system, called VASEM, to provide convenient data exploration, searching, and analysis tools. Domain scientists can use our system to interactively filter massive EMDB data into relatively small subset for analysis and obtain subset depiction from visualization results. A modified parallel coordinates plot suitable for large-scale multi-dimensional ranking data visualization is also proposed for EM data subset depiction tasks. We present two usage scenarios to describe how the proposed system achieves the design goals.

Principal component analysis with tensor train subspace

Tensor train is a hierarchical tensor network structure that helps alleviate the curse of dimensionality by parameterizing large-scale multidimensional data via a set of network of low-rank tensors. Associated with such a construction is a notion of Tensor Train subspace and in this paper we propose a TT-PCA algorithm for estimating this structured subspace from the given data. By maintaining low rank tensor structure, TT-PCA is empirically more robust to noise as compared to PCA or Tucker-PCA. This is borne out numerically by testing the proposed approach on the Extended YaleFace Dataset B, MINIST Dataset, CIFAR-10 dataset. This paper shows that the TT-PCA methods achieve less storage requirements, and have computationally faster online implementation with improved classification performance.

GDCRNATools: an R/Bioconductor package for integrative analysis of lncRNA, miRNA and mRNA data in GDC.

The large-scale multidimensional omics data in the Genomic Data Commons (GDC) provides opportunities to investigate the crosstalk among different RNA species and their regulatory mechanisms in cancers. Easy-to-use bioinformatics pipelines are needed to facilitate such studies. We have developed a user-friendly R/Bioconductor package, named GDCRNATools, for downloading, organizing and analyzing RNA data in GDC with an emphasis on deciphering the lncRNA-mRNA related competing endogenous RNAs regulatory network in cancers. Many widely used bioinformatics tools and databases are utilized in our package. Users can easily pack preferred downstream analysis pipelines or integrate their own pipelines into the workflow. Interactive shiny web apps built in GDCRNATools greatly improve visualization of results from the analysis. GDCRNATools is an R/Bioconductor package that is freely available at Bioconductor (http://bioconductor.org/packages/devel/bioc/html/GDCRNATools.html). Detailed instructions, manual and example code are also available in Github (https://github.com/Jialab-UCR/GDCRNATools).

Scalable distributed data cube computation for large-scale multidimensional data analysis on a Spark cluster

A data cube is a powerful analytical tool that stores all aggregate values over a set of dimensions. It provides users with a simple and efficient means of performing complex data analysis while assisting in decision making. Since the computation time for building a data cube is very large, however, efficient methods for reducing the data cube computation time are needed. Previous works have developed various algorithms for efficiently generating data cubes using MapReduce, which is a large-scale distributed parallel processing framework. However, MapReduce incurs the overhead of disk I/Os and network traffic. To overcome these MapReduce limitations, Spark was recently proposed as a memory-based parallel/distributed processing framework. It has attracted considerable research attention owing to its high performance. In this paper, we propose two algorithms for efficiently building data cubes. The algorithms fully leverage Spark’s mechanisms and properties: Resilient Distributed Top-Down Computation (RDTDC) and Resilient Distributed Bottom-Up Computation (RDBUC). The former is an algorithm for computing the components (i.e., cuboids) of a data cube in a top-down approach; the latter is a bottom-up approach. The RDTDC algorithm has three key functions. (1) It approximates the size of the cuboid using the cardinality without additional Spark action computation to determine the size of each cuboid during top-down computation. Thus, one cuboid can be computed from the upper cuboid of a smaller size. (2) It creates an execution plan that is optimized to input the smaller sized cuboid. (3) Lastly, it uses a method of reusing the result of the already computed cuboid by top-down computation and simultaneously computes the cuboid of several dimensions. In addition, we propose the RDBUC bottom-up algorithm in Spark, which is widely used in computing Iceberg cubes to maintain only cells satisfying a certain condition of minimum support. This algorithm incorporates two primary strategies: (1) reducing the input size to compute aggregate values for a dimension combination (e.g., A, B, and C) by removing the input, which does not satisfy the Iceberg cube condition at its lower dimension combination (e.g., A and B) computed earlier. (2) We use a lazy materialization strategy that computes every combination of dimensions using only transformation operations without any action operation. It then stores them in a single action operation. To prove the efficiency of the proposed algorithms using a lazy materialization strategy by employing only one action operation, we conducted extensive experiments. We compared them to the cube() function, a built-in cube computation library of Spark SQL. The results showed that the proposed RDTDC and RDBUC algorithms outperformed Spark SQL cube().

Automation strategies for large-scale 3D image analysis

Abstract New imaging techniques enable visualizing and analyzing a multitude of unknown phenomena in many areas of science at high spatio-temporal resolution. The rapidly growing amount of image data, however, can hardly be analyzed manually and, thus, future research has to focus on automated image analysis methods that allow one to reliably extract the desired information from large-scale multidimensional image data. Starting with infrastructural challenges, we present new software tools, validation benchmarks and processing strategies that help coping with large-scale image data. The presented methods are illustrated on typical problems observed in developmental biology that can be answered, e.g., by using time-resolved 3D microscopy images.

Golay Code Transformations for Ensemble Clustering in Application to Medical Diagnostics

Clinical Big Data streams have accumulated large-scale multidimensional data about patients’ medical conditions and drugs along with their known side effects. The volume and the complexity of this Big Data streams hinder the current computational procedures. Effective tools are required to cluster and systematically analyze this amorphous data to perform data mining methods including discovering knowledge, identifying underlying relationships and predicting patterns. This paper presents a novel computation model for clustering tremendous amount of Big Data streams. The presented approach is utilizing the error-correction Golay Code. This clustering methodology is unique. It outperforms all other conventional techniques because it has linear time complexity and does not impose predefined cluster labels that partition data. Extracting meaningful knowledge from these clusters is an essential task; therefore, a novel mechanism that facilitates the process of predicting patterns and likelihood diseases based on a semi-supervised technique is presented.

Madagascar: open-source software project for multidimensional data analysis and reproducible computational experiments

<p class="p1"><span class="s1">The Madagascar software package is designed for analysis of large-scale multidimensional data, such as those occurring in exploration geophysics. Madagascar provides a framework for reproducible research. By “reproducible research” we refer to the discipline of attaching software codes and data to computational results reported in publications. The package contains a collection of (a) computational modules, (b) data-processing scripts, and (c) research papers. Madagascar is distributed on SourceForge under a GPL v2 license <a href="https://sourceforge.net/projects/rsf/">https://sourceforge.net/projects/rsf/</a>. By October 2013, more than 70 people from different organizations around the world have contributed to the project, with increasing year-to-year activity. The Madagascar website is <a href="http://www.ahay.org/">http://www.ahay.org/</a>.</span>

WEB APPLICATION FOR LARGE-SCALE MULTIDIMENSIONAL DATA VISUALIZATION

In this paper, we present an approach of the web application (as a service) for data mining oriented to the multidimensional data visualization. This paper focuses on visualization methods as a tool for the visual presentation of large-scale multidimensional data sets. The proposed implementation of such a web application obtains a multidimensional data set and as a result produces a visualization of this data set. It also supports different configuration parameters of the data mining methods used. Parallel computation has been used in the proposed implementation to run the algorithms simultaneously on different computers.