Data Partitioning Scheme Research Articles

A hierarchical data partitioning strategy for irregular applications: a case study in digital pathology

A hierarchical data partitioning strategy for irregular applications: a case study in digital pathology

Unfixed Seasonal Partition Based on Symbolic Aggregate Approximation for Forecasting Solar Power Generation Using Deep Learning

Solar energy is an important alternative energy source, and it is essential to forecast solar power generation for efficient power management. Due to the seasonal characteristics of weather features, seasonal data partition strategies help develop prediction models that perform better in extreme weather-related situations. Most existing studies rely on fixed season partitions, such as meteorological and astronomical, where the start and end dates are specific. However, even if the countries are in the same Northern or Southern Hemisphere, seasonal changes can occur due to abnormal climates such as global warming. Therefore, we propose a novel unfixed seasonal data partition based on Symbolic Aggregate Approximation (SAX) to forecast solar power generation. Here, symbolic representations generated by SAX are used to select seasonal features and obtain seasonal criteria. We then employ two-layer stacked LSTM and combine predictions from various seasonal features and partitions through ensemble methods. The datasets used in the experiments are from real-world solar panel plants such as in Gyeongju, South Korea; and in California, USA. The results of the experiments show that the proposed methods perform better than non-partitioned or fixed-partitioned solar power generation forecasts. They outperform them by 2.2% to 3.5%; and 1.6% to 6.5% in the Gyeongju and California datasets, respectively.

Spatial-Aware and Load Balancing Distributed Data Partitioning Strategies for Content-Based Multimedia Retrieval.

Content-Based Multimedia Retrieval (CBMR) has become very popular in several applications, driven by the growing routine use of multimedia data. Since the datasets used in real-world applications are very large and descriptor's dimensionality is high, querying is an expensive, albeit important functionality. Further, exact search is prohibitive in most cases, motivating the use of Approximate Nearest Neighbour Search (ANNS) algorithms, trading accuracy for performance. These have been mainly developed targeting a sequential execution in a single node. However, the large and increasing datasets used and the high query loads submitted to those systems typically surpass the memory and computing resources available in a single node. This motivated the development of parallel distributed memory ANNS solutions to meet the computing capabilities required by those applications. A common problem that must be handled when using distributed memory systems is data partitioning and its impact on load imbalance. Several data partitioning approaches have already been proposed, including elaborated spatial-aware strategies. However, little effort has been put into carefully analyzing the performance of those strategies at scale. Here, we evaluated the commonly used data partitioning strategies in ANNS and identified their limitations to propose a novel class of partitioning algorithms that can minimize load imbalance while improving data locality to attain high performance on the distributed memory search. Experimentally, we found that our proposed algorithms (SABBS and SABBSR) improved search performance by up to 1.64× compared to the best previous solution. In a distributed memory weak scaling evaluation, with up to 12 billion 128-dimensional descriptors and 60 compute nodes, the gains were maintained as the system scaled with our novel approaches. These results demonstrate the efficiency of our new algorithms for billion-scale ANNS and the importance of considering not only data locality but also data and load imbalance in the data partitioning.

Optimization of multidimensional feature engineering and data partitioning strategies in heart disease prediction models

The relentless rise in heart disease incidence, a leading global cause of death, presents a significant public health challenge. Precise prediction of heart disease risk and early interventions are crucial. This study investigates the performance improvement of heart disease prediction models using machine learning and deep learning algorithms. Initially, we utilized the Heart Failure Prediction Dataset from Kaggle. After preprocessing to ensure data quality, three distinct feature engineering techniques were applied: PCA for dimensionality reduction, ET for feature selection, and Pearson's correlation coefficient for feature selection. We assessed their impact on model performance. The dataset was then partitioned into three different data split ratios—1:9, 2:8, and 3:7—to determine their specific effects on model performance. Twelve machine learning classifiers—LGBM, Adaboost, XGB, RF, DT, KNN, LR, GNB, ET, SVC, GB, and Bagging—were trained and evaluated based on five key metrics: accuracy, recall, precision, F1 score, and training time. The influence of different feature engineering methods and data partitioning ratios on model performance were systematically analyzed using paired-sample t-tests. Among the feature engineering methods compared, the Bagging classifier, when combined with feature selection via ET, exhibited superior performance. It achieved an accuracy of 97.48 % and an F1-Score of 97.48 % with a data split ratio of 1:9 between the test and training sets. With a 2:8 split, the accuracy was 94.96 % and the F1-Score was 94.95 %. For a 3:7 split, the accuracy was 94.12 % and the F1-Score was 94.11 %. Paired sample T-test results indicate that feature selection using Pearson correlation coefficient can shorten training duration, but this also leads to a decline in classifier performance. After applying PCA dimensionality reduction, compared with the control group, there was no significant difference in the training efficiency and efficacy of the classifier. However, feature selection through ET significantly reduced the training time for various classifiers while ensuring their performance.

Dynamic heatmap pyramid computation for massive high-parallel spatial streaming in urban environments

ABSTRACT It is necessary to integrate data and information effectively in order to build an integrated digital and refined urban system. A multi-granularity and multi-view urban heat map formed by integrating multi-source urban information flow can assist in immediate decision-making. Managing urban big data in the form of streams requires a stable operating environment that can efficiently handle millions of sensors and devices connected to the Internet of Things (IoT) in a highly concurrent environment. Urban management data contain spatiotemporal multi-dimensional information that presents the complexity of spatiotemporal dynamic associations, further increasing the difficulty of data streaming. To overcome these challenges, we have proposed a spatiotemporal-pyramid (ST-pyramid) model that organizes multidimensional and dynamic data streams logically using a data partition strategy based on geographical grid subdivision. Accordingly, we have proposed a load-balanced heatmap pyramid computation framework that can be used to build streaming processing procedures in a distributed environment. In addition, we implemented an urban traffic heatmap prototype system based on an open-source Flink framework. The experimental results show that the real-time heatmap pyramid construction algorithm proposed in this paper has high throughput, low latency, and flexible scalability and can provide large-scale public services in time in digital earth construction.

Two-dimensional data partitioning for non-negative matrix tri-factorization

As a two-sided clustering and dimensionality reduction paradigm, Non-negative Matrix Tri-Factorization (NMTF) has attracted much attention in machine learning and data mining researchers due to its excellent performance and reliable theoretical support. Unlike Non-negative Matrix Factorization (NMF) methods applicable to one-sided clustering only, NMTF introduces an additional factor matrix and uses the inherent duality of data to realize the mutual promotion of sample clustering and feature clustering, thus showing great advantages in many scenarios (e.g., text co-clustering). However, the existing methods for solving NMTF usually involve intensive matrix multiplication, which is characterized by high time and space complexities, that is, there are limitations of slow convergence of the multiplicative update rules and high memory overhead. In order to solve the above problems, this paper develops a distributed parallel algorithm with a 2-dimensional data partition scheme for NMTF (i.e., PNMTF-2D). Experiments on multiple text datasets show that the proposed PNMTF-2D can substantially improve the computational efficiency of NMTF (e.g., the average iteration time is reduced by up to 99.7% on Amazon) while ensuring the effectiveness of convergence and co-clustering.

Scrutinizing different predictive modeling validation methodologies and data-partitioning strategies: new insights using groundwater modeling case study

Groundwater salinity is a critical factor affecting water quality and ecosystem health, with implications for various sectors including agriculture, industry, and public health. Hence, the reliability and accuracy of groundwater salinity predictive models are paramount for effective decision-making in managing groundwater resources. This pioneering study presents the validation of a predictive model aimed at forecasting groundwater salinity levels using three different validation methods and various data partitioning strategies. This study tests three different data validation methodologies with different data-partitioning strategies while developing a group method of data handling (GMDH)-based model for predicting groundwater salinity concentrations in a coastal aquifer system. The three different methods are the hold-out strategy (last and random selection), k-fold cross-validation, and the leave-one-out method. In addition, various combinations of data-partitioning strategies are also used while using these three validation methodologies. The prediction model’s validation results are assessed using various statistical indices such as root mean square error (RMSE), means squared error (MSE), and coefficient of determination (R2). The results indicate that for monitoring wells 1, 2, and 3, the hold-out (random) with 40% data partitioning strategy gave the most accurate predictive model in terms of RMSE statistical indices. Also, the results suggested that the GMDH-based models behave differently with different validation methodologies and data-partitioning strategies giving better salinity predictive capabilities. In general, the results justify that various model validation methodologies and data-partitioning strategies yield different results due to their inherent differences in how they partition the data, assess model performance, and handle sources of bias and variance. Therefore, it is important to use them in conjunction to obtain a comprehensive understanding of the groundwater salinity prediction model's behavior and performance.

Simulating federated learning for steatosis detection using ultrasound images

We aimed to implement four data partitioning strategies evaluated with four federated learning (FL) algorithms and investigate the impact of data distribution on FL model performance in detecting steatosis using B-mode US images. A private dataset (153 patients; 1530 images) and a public dataset (55 patient; 550 images) were included in this retrospective study. The datasets contained patients with metabolic dysfunction-associated fatty liver disease (MAFLD) with biopsy-proven steatosis grades and control individuals without steatosis. We employed four data partitioning strategies to simulate FL scenarios and we assessed four FL algorithms. We investigated the impact of class imbalance and the mismatch between the global and local data distributions on the learning outcome. Classification performance was assessed with area under the receiver operating characteristic curve (AUC) on a separate test set. AUCs were 0.93 (95% CI 0.92, 0.94) for source-based partitioning scenario with FedAvg, 0.90 (95% CI 0.89, 0.91) for a centralized model, and 0.83 (95% CI 0.81, 0.85) for a model trained in a single-center scenario. When data was perfectly balanced on the global level and each site had an identical data distribution, the model yielded an AUC of 0.90 (95% CI 0.88, 0.92). When each site contained data exclusively from one single class, irrespective of the global data distribution, the AUC fell in the range of 0.34–0.70. FL applied to B-mode US images provide performance comparable to a centralized model and higher than single-center scenario. Global data imbalance and local data heterogeneity influenced the learning outcome.

Bayesian cross-validation by parallel Markov chain Monte Carlo

Brute force cross-validation (CV) is a method for predictive assessment and model selection that is general and applicable to a wide range of Bayesian models. Naive or ‘brute force’ CV approaches are often too computationally costly for interactive modeling workflows, especially when inference relies on Markov chain Monte Carlo (MCMC). We propose overcoming this limitation using massively parallel MCMC. Using accelerator hardware such as graphics processor units, our approach can be about as fast (in wall clock time) as a single full-data model fit. Parallel CV is flexible because it can easily exploit a wide range data partitioning schemes, such as those designed for non-exchangeable data. It can also accommodate a range of scoring rules. We propose MCMC diagnostics, including a summary of MCMC mixing based on the popular potential scale reduction factor (R^\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\widehat{\ extrm{R}}$$\\end{document}) and MCMC effective sample size (ESS^\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\widehat{\ extrm{ESS}}$$\\end{document}) measures. We also describe a method for determining whether an R^\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\widehat{\ extrm{R}}$$\\end{document} diagnostic indicates approximate stationarity of the chains, that may be of more general interest for applications beyond parallel CV. Finally, we show that parallel CV and its diagnostics can be implemented with online algorithms, allowing parallel CV to scale up to very large blocking designs on memory-constrained computing accelerators.

Privacy-Enhanced Cooperative Storage Scheme for Contact-Free Sensory Data in AIoT with Efficient Synchronization

The growing popularity of contact-free smart sensing has contributed to the development of the Artificial Intelligence of Things (AIoT). The contact-free sensory data has great potential to mine and analyze the hidden information for AIoT-enabled applications. However, due to the limited storage resource of contact-free smart sensing devices, data is naturally stored in the cloud, which is at risk of privacy leakage. Cloud storage is generally considered insecure. On one hand, the openness of the cloud environment makes the data easy to be attacked, and the complex AIoT environment also makes the data transmission process vulnerable to the third party. On the other hand, the Cloud Service Provider (CSP) is untrusted. In this article, to ensure the security of data from contact-free smart sensing devices, a Cloud-Edge-End cooperative storage scheme is proposed, which takes full advantage of the differences in the cloud, edge, and end. Firstly, the processed sensory data is stored separately in the three layers by utilizing well-designed data partitioning strategy. This scheme can increase the difficulty of privacy leakage in the transmission process and avoid internal and external attacks. Besides, the contact-free sensory data is highly time-dependent. Therefore, combined with the Cloud-Edge-End cooperation model, this article proposes a delta-based data update method and extends it into a hybrid update mode to improve the synchronization efficiency. Theoretical analysis and experimental results show that the proposed cooperative storage method can resist various security threats in bad situations and outperform other update methods in synchronization efficiency, significantly reducing the synchronization overhead in AIoT.

An Adapted Ant Colony Optimization for Feature Selection

ABSTRACT As information technologies evolve, they generate vast and ever-expanding datasets. This wealth of high-dimensional data presents challenges, including increased computational demands and difficulties in extracting valuable insights. The aim of feature selection is to address this complexity by reducing data dimensions with minimal information loss. Our proposed feature selection approach, the Feature Selection via Ant Colony Optimization algorithm, employs heuristic distance directly in its probability function, instead of using its inverse. The algorithm bypasses the need for sub-attribute sets, running multiple iterations to create a frequency order list from the collected routes, which informs feature importance. The efficacy of this technique has been validated through comparative experiments with other methods from scientific literature. To ensure fairness, these experiments used identical datasets, data partitioning strategies, classifiers, and performance metrics. Initially, the algorithm was compared with fifteen different algorithms, and subsequently benchmarked against three selected methods. The impact of feature selection on classification performance was statistically verified through comparisons before and after the feature selection process. Convergence performance of the proposed method has also been evaluated. Our findings robustly support the efficacy of the introduced approach in managing complex, multidimensional data effectively.

Evaluation of machine learning models for cytochrome P450 3A4, 2D6, and 2C9 inhibition.

Cytochrome P450 (CYP) enzymes are involved in the metabolism of approximately 75% of marketed drugs. Inhibition of the major drug-metabolizing P450s could alter drug metabolism and lead to undesirable drug-drug interactions. Therefore, it is of great significance to explore the inhibition of P450s in drug discovery. Currently, machine learning including deep learning algorithms has been widely used for constructing in silico models for the prediction of P450 inhibition. These models exhibited varying predictive performance depending on the use of machine learning algorithms and molecular representations. This leads to the difficulty in the selection of appropriate models for practical use. In this study, we systematically evaluated the conventional machine learning and deep learning models for three major P450 enzymes, CYP3A4, CYP2D6, and CYP2C9 from several perspectives, such as algorithms, molecular representation, and data partitioning strategies. Our results showed that the XGBoost and CatBoost algorithms coupled with the combined fingerprint/physicochemical descriptor features exhibited the best performance with Area Under Curve (AUC) of 0.92, while the deep learning models were generally inferior to the conventional machine learning models (average AUC reached 0.89) on the same test sets. We also found that data volume and sampling strategy had a minor effect on model performance. We anticipate that these results are helpful for the selection of molecular representations and machine learning/deep learning algorithms in the P450 model construction and the future model development of P450 inhibition.

Applying Hybrid Clustering in Pulsar Candidate Sifting with Multi-modality for FAST Survey

Pulsar search is always the basis of pulsar navigation, gravitational wave detection and other research topics. Currently, the volume of pulsar candidates collected by the Five-hundred-meter Aperture Spherical radio Telescope (FAST) shows an explosive growth rate that has brought challenges for its pulsar candidate filtering system. Particularly, the multi-view heterogeneous data and class imbalance between true pulsars and non-pulsar candidates have negative effects on traditional single-modal supervised classification methods. In this study, a multi-modal and semi-supervised learning based on a pulsar candidate sifting algorithm is presented, which adopts a hybrid ensemble clustering scheme of density-based and partition-based methods combined with a feature-level fusion strategy for input data and a data partition strategy for parallelization. Experiments on both High Time Resolution Universe Survey II (HTRU2) and actual FAST observation data demonstrate that the proposed algorithm could excellently identify pulsars: On HTRU2, the precision and recall rates of its parallel mode reach 0.981 and 0.988 respectively. On FAST data, those of its parallel mode reach 0.891 and 0.961, meanwhile, the running time also significantly decreases with the increment of parallel nodes within limits. Thus, we can conclude that our algorithm could be a feasible idea for large scale pulsar candidate sifting for FAST drift scan observation.

Block size estimation for data partitioning in HPC applications using machine learning techniques

The extensive use of HPC infrastructures and frameworks for running data-intensive applications has led to a growing interest in data partitioning techniques and strategies. In fact, application performance can be heavily affected by how data are partitioned, which in turn depends on the selected size for data blocks, i.e. the block size. Therefore, finding an effective partitioning, i.e. a suitable block size, is a key strategy to speed-up parallel data-intensive applications and increase scalability. This paper describes a methodology, namely BLEST-ML (BLock size ESTimation through Machine Learning), for block size estimation that relies on supervised machine learning techniques. The proposed methodology was evaluated by designing an implementation tailored to dislib, a distributed computing library highly focused on machine learning algorithms built on top of the PyCOMPSs framework. We assessed the effectiveness of the provided implementation through an extensive experimental evaluation considering different algorithms from dislib, datasets, and infrastructures, including the MareNostrum 4 supercomputer. The results we obtained show the ability of BLEST-ML to efficiently determine a suitable way to split a given dataset, thus providing a proof of its applicability to enable the efficient execution of data-parallel applications in high performance environments.

Estimation of Paddy leaves Chlorophyll Content Using Convolutional Neural Network (CNN)

In contemporary agriculture, the demand for cutting-edge machine-learning techniques to elevate crop assessment and management is paramount. This study introduces an innovative approach employing a Convolutional Neural Network (CNN) with Inception_v3 as its base model to estimate chlorophyll levels in paddy leaves. The primary aim is to craft a robust, precise model capable of non-destructively predicting chlorophyll content, promising substantial improvements in the efficiency of evaluating paddy crop health and nutritional status. The dataset comprises 566 images of paddy leaves, spanning 122 unique chlorophyll content levels. A meticulous data partitioning strategy allocates 244 images for model training, with 122 and 180 images for validation and testing, respectively. Model performance metrics include a test loss of 1.19 and a test accuracy of 0.81. Leveraging the Inception_v3 architecture empowers the CNN model to extract intricate, distinguishing features from paddy leaf images. This capability enables the model to discern subtle variations in chlorophyll content across different classes, underpinning its promising predictive prowess. Future research directions may explore potential model enhancements and dataset expansion, marking significant progress toward revolutionizing crop health assessment in modern agriculture.

A Batch Postprocessing Method Based on an Adaptive Data Partitioning Strategy for DEM Differencing From Global Data Products

A Batch Postprocessing Method Based on an Adaptive Data Partitioning Strategy for DEM Differencing From Global Data Products

Survey of image deduplication for cloud storage

Increased growth of real-life communication has motivated the creation, transmission, and digital storage of vast volumes of images and video data on the cloud. The explosive increase in virtual/visual image data on cloud servers requires efficient storage utilization that can be addressed using image deduplication technology. Even though the virtual and visual image properties are different, the existing literature uses a similar approach for deduplication checks, which motivated us to consider both image types for this review. This article aims to provide a detailed survey of state-of-the-art visuals as well as virtual image deduplication techniques in a cloud environment, summarizing and organizing them by developing a five-dimensional taxonomy for analysing the features and performance with several non-overlapping categories in each dimension. These include: 1) location of applying deduplication; 2) image feature extraction; 3) time of application; 4) image data partitioning strategy; 5) involvement of user dataset level. Existing image deduplication techniques are categorized into two main categories based on whether the technique involves security. A comparison of techniques is discussed across a set of functional and performance parameters. The current issues are highlighted with the possible future directions to motivate further research studies on the topic.

Complete Plastid Genomes of Nine Species of Ranunculeae (Ranunculaceae) and Their Phylogenetic Inferences.

The tribe Ranunculeae, Ranunculaceae, comprising 19 genera widely distributed all over the world. Although a large number of Sanger sequencing-based molecular phylogenetic studies have been published, very few studies have been performed on using genomic data to infer phylogenetic relationships within Ranunculeae. In this study, the complete plastid genomes of nine species (eleven samples) from Ceratocephala, Halerpestes, and Ranunculus were de novo assembled using a next-generation sequencing method. Previously published plastomes of Oxygraphis and other related genera of the family were downloaded from GenBank for comparative analysis. The complete plastome of each Ranunculeae species has 112 genes in total, including 78 protein-coding genes, 30 transfer RNA genes, and four ribosomal RNA genes. The plastome structure of Ranunculeae samples is conserved in gene order and arrangement. There are no inverted repeat (IR) region expansions and only one IR contraction was found in the tested samples. This study also compared plastome sequences across all the samples in gene collinearity, codon usage, RNA editing sites, nucleotide variability, simple sequence repeats, and positive selection sites. Phylogeny of the available Ranunculeae species was inferred by the plastome data using maximum-likelihood and Bayesian inference methods, and data partitioning strategies were tested. The phylogenetic relationships were better resolved compared to previous studies based on Sanger sequencing methods, showing the potential value of the plastome data in inferring the phylogeny of the tribe.

Prediction and model evaluation for space–time data

ABSTRACT Evaluation metrics for prediction error, model selection and model averaging on space–time data are understudied and poorly understood. The absence of independent replication makes prediction ambiguous as a concept and renders evaluation procedures developed for independent data inappropriate for most space–time prediction problems. Motivated by air pollution data collected during California wildfires in 2008, this manuscript attempts a formalization of the true prediction error associated with spatial interpolation. We investigate a variety of cross-validation (CV) procedures employing both simulations and case studies to provide insight into the nature of the estimand targeted by alternative data partition strategies. Consistent with recent best practice, we find that location-based cross-validation is appropriate for estimating spatial interpolation error as in our analysis of the California wildfire data. Interestingly, commonly held notions of bias-variance trade-off of CV fold size do not trivially apply to dependent data, and we recommend leave-one-location-out (LOLO) CV as the preferred prediction error metric for spatial interpolation.

An advanced ensemble clustering approach for data partitioning and mining to optimize performance in variable refrigerant flow systems

This study presents a data mining-based approach to develop clustering ensemble models for data partitioning of variable refrigerant flow (VRF) system, with the aim of improving the quality of pattern detection and laying a solid foundation for further data mining. This approach mainly includes three parts. First, data preprocessing combines handling outliers, feature filtering, stable data filtering and data normalization. Secondly, consensus clustering is used to obtain an appropriate number of clusters. To generate base clusterings, we applied three clustering algorithms - k-means, self-organizing maps, and fuzzy C-means - in a 1:1:1 ratio. These base clusterings were selected based on the adjusted rand index as the underlying selection criterion. For clustering ensemble, we utilized three integration methods, namely K-modes, latent class analysis, and linkage clustering ensemble. Finally, analyze the metric difference and time consumption of the integration clustering results of different methods, and obtain a suitable data partitioning scheme for VRF system. This case study examines the utilization of the proposed method to analyze the operation data of a VRF system in an office building. The results indicate that the quality of the clustering derived by the ensemble clustering model is notably superior to that of the individual base clustering. Moreover, the latent class analysis method demonstrates remarkable advantages in achieving top-quality clustering outcomes. These findings suggest that the proposed method can be effective in optimizing VRF system operation while leveling up the accuracy of data analysis. Ensemble models have considerable merit in devising fault identification and diagnosis techniques, optimizing operation, and exploring the links between VRF systems and smart grids.