Data Partitioning Research Articles

Adaptive Bayesian learning for making risk-aware decisions: A case of trauma survival prediction

Decision tree (DT) models provide a transparent approach to prediction of patient’s outcomes within a probabilistic framework. Averaging over DT models under certain conditions can deliver reliable estimates of predictive posterior probability distributions, which is of critical importance in the case of predicting an individual patient’s outcome. Reliable estimations of the distribution can be achieved within the Bayesian framework using Markov chain Monte Carlo (MCMC) and its Reversible Jump extension enabling DT models to grow to a reasonable size. Existing MCMC strategies however have limited ability to control DT structures and tend to sample overgrown DT models, making unreasonably small partitions, thus deteriorating the uncertainty calibration. This happens because the MCMC explores a DT model parameter space within a limited knowledge of the distribution of data partitions. We propose a new adaptive strategy which overcomes this limitation, and show that in the case of predicting trauma outcomes the number of data partitions can be significantly reduced, so that the unnecessary uncertainty of estimating the predictive posterior density is avoided. The proposed and existing strategies are compared in terms of entropy which, being calculated for predicted posterior distributions, represents the uncertainty in decisions. In this framework, the proposed method has outperformed the existing sampling strategies, so that the unnecessary uncertainty in decisions is efficiently avoided.

Two-layer partitioned and deletable deep bloom filter for large-scale membership query

The recently proposed Learned Bloom Filter (LBF) provides a new perspective on large-scale membership queries by using machine learning to replace the traditional bloom filter. However, reducing the false positive rate (FPR) of the learned model with small memory usage, and supporting deletion efficiently become the new issues. In this paper, we propose a novel Two-layer Partitioned and Deletable Deep Bloom Filter (PDDBF) for large-scale membership query, which can reduce the FPR with small memory usage and support deletion efficiently. The proposed PDDBF consists of three main parts: (1) Data partition. To improve the classification accuracy of the learned model, the K-means cluster with the elbow method is used for the data partition. (2) Deep Bloom Filter. To reduce the FPR, deep learning models are used to construct multiple independent learning mechanisms, which correspond to the clusters obtained by part1. (3) Partitioned backup filter. To support deletion under the premise of ensuring low FPR and reducing query time consumption, combine the perfect hash (PH) table and counting bloom filters (CBFs) on the basis of the partition bloom filter. Experiments show that the proposed PDDBF reduces the FPR 87.13% with the same memory usage compared with the state-of-the-art PLBF on real-world URLs data set. Moreover, the PDDBF reduces the FPR 99.68% with the same memory usage and reduces the query time consumption to 2.61x that of the PLBF after data deletion, respectively.

A novel double-hybrid learning method for modal frequency-based damage assessment of bridge structures under different environmental variation patterns

Monitoring of modal frequencies under an unsupervised learning framework is a practical strategy for damage assessment of civil structures, especially bridges. However, the key challenge is related to high sensitivity of modal frequencies to environmental and/or operational changes that may lead to economic and safety losses. The other challenge pertains to different environmental and/or operational variation patterns in modal frequencies due to differences in structural types, materials, and applications, measurement periods in terms of short and/or long monitoring programs, geographical locations of structures, weather conditions, and influences of single or multiple environmental and/or operational factors, which may cause barriers to employing state-of-the-art unsupervised learning approaches. To cope with these issues, this paper proposes a novel double-hybrid learning technique in an unsupervised manner. It contains two stages of data partitioning and anomaly detection, both of which comprise two hybrid algorithms. For the first stage, an improved hybrid clustering method based on a coupling of shared nearest neighbor searching and density peaks clustering is proposed to prepare local information for anomaly detection with the focus on mitigating environmental and/or operational effects. For the second stage, this paper proposes an innovative non-parametric hybrid anomaly detector based on local outlier factor. In both stages, the number of nearest neighbors is the key hyperparameter that is automatically determined by leveraging a self-adaptive neighbor searching algorithm. Modal frequencies of two full-scale bridges are utilized to validate the proposed technique with several comparisons. Results indicate that this technique is able to successfully eliminate different environmental and/or operational variations and correctly detect damage.

Real-Time Interactive Parallel Visualization of Large-Scale Flow-Field Data

With the increasing demand for high precision in numerical simulations using computational fluid dynamics (CFD), the use of large-scale grids for discretized solutions has become a trend, resulting in an explosive growth of flow-field data size. To address the challenges posed by large-scale flow-field data for real-time interactive visualization, this paper proposes novel strategies for data partitioning and communication management. Firstly, we propose a data-partitioning strategy based on grid segmentation. This approach constructs metadata to create file viewports for each process and performs grid partitioning. Subsequently, it reconstructs sub-grids within each process and utilizes a coordinate-mapping algorithm to map global coordinates to local process coordinates, facilitating access to attribute variables through a lookup table. Secondly, we introduce a real-time interactive method for large-scale flow fields. This method leverages the system architecture of high-speed interconnection among compute nodes in a cluster environment and low-speed interconnection between service nodes and rendering nodes. It enables coordinated management of parallel rendering and synchronized rendering methods. The experimental results on typical flow-field data demonstrate that the proposed data-partitioning strategy improves the loading speed of millions of grid-level data by a factor of 7, surpassing ParaView’s performance by 1.5 times. Furthermore, it achieves system load balancing. Real-time interaction experiments with datasets containing 500 million and 800 million grid cells exhibit millisecond-level latencies, demonstrating the effectiveness of the proposed communication management method in meeting the real-time interactive visualization demands of large-scale flow-field data.

Autonomous data partitioning for type-2 fuzzy set based time series

Autonomous data partitioning for type-2 fuzzy set based time series

Improving Structured Grid-Based Sparse Matrix-Vector Multiplication and Gauss–Seidel Iteration on GPDSP

Structured grid-based sparse matrix-vector multiplication and Gauss–Seidel iterations are very important kernel functions in scientific and engineering computations, both of which are memory intensive and bandwidth-limited. GPDSP is a general purpose digital signal processor, which is a very significant embedded processor that has been introduced into high-performance computing. In this paper, we designed various optimization methods, which included a blocking method to improve data locality and increase memory access efficiency, a multicolor reordering method to develop Gauss–Seidel fine-grained parallelism, a data partitioning method designed for GPDSP memory structures, and a double buffering method to overlap computation and access memory on structured grid-based SpMV and Gauss–Seidel iterations for GPDSP. At last, we combined the above optimization methods to design a multicore vectorization algorithm. We tested the matrices generated with structured grids of different sizes on the GPDSP platform and obtained speedups of up to 41× and 47× compared to the unoptimized SpMV and Gauss–Seidel iterations, with maximum bandwidth efficiencies of 72% and 81%, respectively. The experiment results show that our algorithms could fully utilize the external memory bandwidth. We also implemented the commonly used mixed precision algorithm on the GPDSP and obtained speedups of 1.60× and 1.45× for the SpMV and Gauss–Seidel iterations, respectively.

DTEC: Decision tree-based evidential clustering for interpretable partition of uncertain data

Recently, the evidential clustering has been developed as a promising clustering framework for uncertain data, which generalizes those hard, fuzzy, possibilistic and rough clustering. However, the resulting cluster assignments are less interpretable in terms of human cognition, which limits its applications in those security, privacy or ethic related fields. In this study, the unsupervised decision tree model is introduced into the evidential clustering framework to improve the interpretability of the evidential partition. A Decision Tree-based Evidential Clustering (DTEC) algorithm is developed to build an unsupervised evidential decision tree, which uses the paths from the root node to leaf nodes to achieve the interpretability of each cluster. The proposed algorithm is composed of three procedures, i.e., cutting-point selection, node evidential splitting, and cluster adjustment, in which the first two procedures are carried out iteratively to build a preliminary unsupervised decision tree and the last procedure is designed to adjust the preliminary decision tree if the number of clusters is available. Both synthetic and real datasets are used to evaluate the performance of the proposed algorithm, and the experimental results demonstrate the good performance of the proposal compared with some representative fuzzy, evidential or decision tree-based clustering algorithms.

Clustering Daerah Rawan Bencana Alam Di Indonesia Berdasarkan Provinsi Dengan Metode K-Means

Indonesia is a country located in a strategic geological position and is at the confluence of three world plates. This position causes Indonesia to become a country frequently hit by natural disasters such as earthquakes, landslides and other natural disasters. Based on data obtained from the Regional Disaster Management Agency (BPBD), from year to year the number of natural disasters always increases. The increase in the occurrence of natural disasters has led to the need for further research to find out how vulnerable areas are in Indonesia and can later reduce the risks posed when natural disasters occur. Researchers will use the K-Means Clustering method to classify natural disaster areas based on their level of vulnerability to natural disasters. Clustering is a process for grouping data into several clusters or groups. K-Means is a method for grouping non-hierarchical data (partitions) which can divide data into two or more groups. The source of this dataset comes from the Central Bureau of Statistics (BPS) and consists of 12 columns and the data used are landslides and no natural disasters. Then a search for the optimal K value was carried out using the elbow method and Silhouette Analysis which produced 2 clusters namely cluster 0 and cluster 1. The results of the clustering indicated that cluster 1 was classified as a disaster-prone area in Indonesia including Aceh, North Sumatra, West Java, Central Java, East Java.

Dominant Partitioning of Discontinuities of Rock Masses Based on DBSCAN Algorithm

In the analysis of rock slope stability and rock mass hydraulics, the dominant partitioning of discontinuities of rock masses is a very important concept, and it is still a key for establishing the three-dimensional (3-D) network model of random discontinuities. The traditional graphical analysis method is inadequate and greatly influenced by subjective experience. A new method using density-based spatial clustering of applications with noise (DBSCAN) algorithm is proposed for the dominant partitioning of discontinuities of rock mass. In the proposed method, we do not need to determine the centers of every cluster before clustering, and the acnodes or outliers can be eliminated effectively after clustering. Firstly, the spatial coordinate transformation of the discontinuity occurrence is carried out and the objective function is established by using the sine value of the angle of the unit normal vector as the similarity measure standard. The DBSCAN algorithm is used to establish the optimal clustering centers by searching the global optimal solution of the objective function, and the fuzzy C-means clustering algorithm is optimized and the mathematical model of the advantage grouping of rock discontinuities is established. The new method and the fuzzy C-means method are compared and verified by using the artificially randomly generated discontinuity occurrence data. The proposed method is a better method than the fuzzy C-means method in general cases, and it can provide more accurate results by eliminating the acnodes or outliers. Finally, the proposed method is applied to discontinuity orientation partition data at Maji dam site, Nujiang River, and there is good agreement with the in situ measurement.

Day-ahead seasonal solar radiation prediction, combining VMD and STACK algorithms

Abstract This article proposes a method for accurately predicting solar irradiance over a 24-hour horizon to forecast photovoltaic energy generation in a positive-energy building. In order to make this prediction, the input data are divided into seasons and preprocessed using the variational mode decomposition (seasonal-VMD) method. The VMD method is used for extracting high-bandwidth features from the input data, decomposing them into a finite number of smooth modes and focusing on specific frequency ranges. Hence, the accuracy of signal extraction using the VMD method can be improved by selecting particular parameters judiciously, which impacts the smoothing and frequency concentration of the extracted signal. In this regard, the salp swarm algorithm (SSA) is employed to identify the optimal VMD parameters that can be used to enhance extraction accuracy. In addition, the obtained residual between the observed solar irradiation data and their decomposed modes is treated to enhance the prediction process. A stacking algorithm (STACK) is used to predict the following 24-hour solar irradiance modes and the residual, which are finally summed to reconstruct the desired signal. The performances of the proposed prediction method are evaluated using two quantitative evaluation indices: the normalized root mean square percentage error (NRMSPE) and normalized mean absolute percentage error (NMAPE). The proposed model is trained on data collected for three years in Rabat (2019–22). The performance of the proposed model is evaluated by predicting the 24-hour solar irradiance for a different season. The proposed approach seasonal-VMD-STACK is compared with two other methods in the case of using VMD-based STACK without season partition and STACK method only. Moreover, the proposed method has exhibited stability and proven good results with an NRMSPE of 3.87% and an NMAPE of 1.58% for cloudy days during the test phase. The results demonstrate that residual preprocessing, seasonal input data partition and appropriate selection of VMD parameters improve the performance and accuracy of the prediction.

SD^2: Slicing and Dicing Scholarly Data for Interactive Evaluation of Academic Performance.

Comprehensively evaluating and comparing researchers' academic performance is complicated due to the intrinsic complexity of scholarly data. Different scholarly evaluation tasks often require the publication and citation data to be investigated in various manners. In this paper, we present an interactive visualization framework, SD^2, to enable flexible data partition and composition to support various analysis requirements within a single system. SD^2 features the hierarchical histogram, a novel visual representation for flexibly slicing and dicing the data, allowing different aspects of scholarly performance to be studied and compared. We also leverage the state-of-the-art set visualization technique to select individual researchers or combine multiple scholars for comprehensive visual comparison. We conduct multiple rounds of expert evaluation to study the effectiveness and usability of SD^2 and revise the design and system implementation accordingly. The effectiveness of SD^2 is demonstrated via multiple usage scenarios with each aiming to answer a specific, commonly raised question.

A fuzzy-based cascade ensemble model for improving extreme wind speeds prediction

A novel fuzzy-based cascade ensemble of regression models is proposed to address a problem of extreme wind speed events forecasting, using data from atmospheric reanalysis models. Although this problem has been mostly explored in the context of classification tasks, the innovation of this paper arises from tackling a continuous predictive domain, aiming at providing an accurate estimation of the extreme wind speed values. The proposed layered framework involves a successive partition of the training data into fuzzy-soft clusters according to the target variable value, and further training a specific regression model within each designated cluster, so that each model can analyze a particular part of the target domain. Finally, predictions made by individual models are integrated into a fuzzy-based ensemble, where a pertinence value is designated to each model based on the previous layer’s prediction, and on the defined membership functions for each cluster. A Differential Evolution (DE) optimization algorithm is adopted to find the optimal way to perform data partitioning. Fast training randomized neural networks methods are used as final regression schemes. The performance of the proposed methodology has been assessed by comparison against state-of-the-art methods in real data from three wind farms in Spain.

SLR: A million-scale comprehensive crossword dataset for simultaneous learning and reasoning

The progress of the natural language understanding (NLU) community has put forward higher demands for the knowledge reserve and reasoning ability of the model. However, existing schemes for improving model capabilities often split them into two separate tasks. The task of enabling models to learn and reason about knowledge simultaneously has not received sufficient attention. In this paper, we propose a novel crossword-based NLU task that imparts knowledge information to a model by solving crossword clues and simultaneously trains the model to infer new knowledge from existing knowledge. To this end, we construct a comprehensive crossword dataset SLR containing more than 4 million unique clue-answer pairs. Compared to existing crossword datasets, SLR is more comprehensive and contains linguistic knowledge, expertise in various fields, and commonsense knowledge. Meanwhile, to evaluate the reasoning ability of the model, we design clever details in the reasoning of the answers, most clues require the solver to reason through two or more pieces of knowledge to arrive at an answer. We analyze the composition of the dataset and the similarities and differences of the various types of clues via sampling and consider various data partitioning methods to enhance the generalization ability of the training set. Furthermore, we test the performance of several different advanced models and methods on this dataset and analyze the strengths and weaknesses of each. An interesting conclusion is that even powerful language models perform poorly on tasks that require reasoning.

CSI: Contrastive data Stratification for Interaction prediction and its application to compound-protein interaction prediction.

Accurately predicting the likelihood of interaction between two objects (compound-protein sequence, user-item, author-paper, etc.) is a fundamental problem in Computer Science. Current deep-learning models rely on learning accurate representations of the interacting objects. Importantly, relationships between the interacting objects, or features of the interaction, offer an opportunity to partition the data to create multi-views of the interacting objects. The resulting congruent and non-congruent views can then be exploited via contrastive learning techniques to learn enhanced representations of the objects. We present a novel method, Contrastive Stratification for Interaction Prediction (CSI), to stratify (partition) a dataset in a manner that can be exploited via Contrastive Multiview Coding to learn embeddings that maximize the mutual information across congruent data views. CSI assigns a key and multiple views to each data point, where data partitions under a particular key form congruent views of the data. We showcase the effectiveness of CSI by applying it to the compound-protein sequence interaction prediction problem, a pressing problem whose solution promises to expedite drug delivery (drug-protein interaction prediction), metabolic engineering, and synthetic biology (compound-enzyme interaction prediction) applications. Comparing CSI with a baseline model that does not utilize data stratification and contrastive learning, and show gains in average precision ranging from 13.7% to 39% using compounds and sequences as keys across multiple drug-target and enzymatic datasets, and gains ranging from 16.9% to 63% using reaction features as keys across enzymatic datasets. Code and dataset available at https://github.com/HassounLab/CSI.

Automated diagnosis of epileptic seizures using EEG image representations and deep learning

BackgroundThe identification of seizure and its complex waveforms in electroencephalography (EEG) through manual examination is time consuming, tedious, and susceptible to human mistakes. These issues have prompted the design of an automated seizure detection system that can assist the neurophysiologists by providing a fast and accurate analysis. MethodsExisting automated seizure detection systems are either machine learning based or deep learning based. Machine learning based algorithms employ handcrafted features with sophisticated feature selection approaches. As a result of which their performance varies with the choice of the feature extraction and selection techniques employed. On the other hand, deep learning-based methods automatically deduce the best subset of features required for the categorization task but they are computationally expensive and lacks generalization on clinical EEG datasets. To address the above stated limitations and motivated by the advantage of continuous wavelet transform's (CWT) in elucidating the non-stationary nature of the EEG signals in a better way, we propose an approach based on EEG image representations (constructed via applying WT at different scale and time intervals) and transfer learning for seizure detection. Firstly, the pre-trained model is fine-tuned on the EEG image representations and thereafter features are extracted from the trained model by performing activations on different layers of the network. Subsequently, the features are passed through a Support Vector Machine (SVM) for categorization using a 10-fold data partitioning scheme. Results and comparison with existing methodsThe proposed mechanism results in a ceiling level of classification performance (accuracy=99.50/98.67, sensitivity=100/100 & specificity=99/96) for both the standard and the clinical dataset that are better than the existing state-of-the art works. ConclusionThe rapid advancement in the field of deep learning has created a paradigm shift in automated diagnosis of epilepsy. The proposed tool has effectually marked the relevant EEG segments for the clinician to review thereby reducing the time burden in scanning the long duration EEG records.

Phylogenetic congruence, conflict and consilience between molecular and morphological data

Morphology and molecules are important data sources for estimating evolutionary relationships. Modern studies often utilise morphological and molecular partitions alongside each other in combined analyses. However, the effect of combining phenomic and genomic partitions is unclear. This is exacerbated by their size imbalance, and conflict over the efficacy of different inference methods when using morphological characters. To systematically address the effect of topological incongruence, size imbalance, and tree inference methods, we conduct a meta-analysis of 32 combined (molecular + morphology) datasets across metazoa. Our results reveal that morphological-molecular topological incongruence is pervasive: these data partitions yield very different trees, irrespective of which method is used for morphology inference. Analysis of the combined data often yields unique trees that are not sampled by either partition individually, even with the inclusion of relatively small quantities of morphological characters. Differences between morphology inference methods in terms of resolution and congruence largely relate to consensus methods. Furthermore, stepping stone Bayes factor analyses reveal that morphological and molecular partitions are not consistently combinable, i.e. data partitions are not always best explained under a single evolutionary process. In light of these results, we advise that the congruence between morphological and molecular data partitions needs to be considered in combined analyses. Nonetheless, our results reveal that, for most datasets, morphology and molecules can, and should, be combined in order to best estimate evolutionary history and reveal hidden support for novel relationships. Studies that analyse only phenomic or genomic data in isolation are unlikely to provide the full evolutionary picture.

A Clustering Algorithm for Tunnel Boring Machine Data Based on Ridge Regression and Fuzzy C-Means

A fuzzy clustering data partitioning method based on ridge regression is proposed to address the high correlation between the data attributes of the tunnel boring machine. The method utilizes the fuzzy partition matrix as the weight for ridge regression analysis and then employs the resulting regression equation as the clustering model to iteratively approach the target minimum value. This process enables data clustering based on the relevant characteristics of the data attributes. Experimental results using functional data sets demonstrate that this method achieves higher accuracy in clustering functional correlation data compared to traditional methods. Additionally, the proposed algorithm is evaluated using measured data from a bid section of a city subway. Results indicate that the algorithm effectively addresses the problem of data classification and key performance index prediction, with a misclassification rate and prediction error of 23.8% and 2.11%, respectively. These findings meet the engineering requirements and provide support for the subsequent data analysis of the tunnel boring machine.

Leveraging Spark and PySpark for Data-Driven Success: Insights and Best Practices Including Parallel Processing, Data Partitioning, and Fault Tolerance Mechanisms

In the burgeoning field of big data analytics, efficient data ingestion pipelines are crucial for handling vast volumes of data effectively. Apache Spark and its Python API, PySpark, have emerged as leading platforms for constructing robust data ingestion architectures due to their superior processing speeds, scalability, and fault tolerance capabilities. This paper delves into the optimization of data ingestion pipelines using Spark and PySpark, focusing on best practices and techniques that enhance performance and reliability. The discussion begins with an overview of Spark and PySpark, explaining their significance in the big data ecosystem and their roles in data ingestion. It highlights the core components relevant to data ingestion, including Spark Core and Spark SQL, which facilitate efficient data processing and integration. The paper further explores critical strategies such as data partitioning, parallel processing, and the judicious use of caching and persistence to improve data throughput and query performance. Fault tolerance, a pivotal aspect of data ingestion pipelines, is thoroughly examined, emphasizing Spark’s built-in mechanisms like RDD lineage and Data Frame operations that ensure data integrity and recovery without manual intervention. Additionally, the paper addresses performance tuning techniques, offering guidance on configuring Spark settings to optimize resource utilization and throughput during data ingestion tasks. Practical case studies are presented to illustrate how various industries implement these practices to overcome specific data ingestion challenges. These examples provide insights into the application of theoretical concepts in real-world scenarios, reinforcing the practical benefits of Spark and PySpark in diverse operational environments. This comprehensive analysis aims to equip data engineers and IT professionals with the knowledge to leverage Spark and PySpark effectively, enhancing their data ingestion pipelines' efficiency, scalability, and resilience, thereby supporting more informed decision-making and streamlined data operations in their organizations.

An automated malaria cells detection from thin blood smear images using deep learning.

Timely and rapid diagnosis is crucial for faster and proper malaria treatment planning. Microscopic examination is the gold standard for malaria diagnosis, where hundreds of millions of blood films are examined annually. However, this method's effectiveness depends on the trained microscopist's skills. With the increasing interest in applying deep learning in malaria diagnosis, this study aims to determine the most suitable deep-learning object detection architecture and their applicability to detect and distinguish red blood cells as either malaria-infected or non-infected cells. The object detectors Yolov4, Faster R-CNN, and SSD 300 are trained with images infected by all five malaria parasites and from four stages of infection with 80/20 train and test data partition. The performance of object detectors is evaluated, and hyperparameters are optimized to select the best-performing model. The best-performing model was also assessed with an independent dataset to verify the models' ability to generalize in different domains. The results show that upon training, the Yolov4 model achieves a precision of 83%, recall of 95%, F1-score of 89%, and mean average precision of 93.87% at a threshold of 0.5. Conclusively, Yolov4 can act as an alternative in detecting the infected cells from whole thin blood smear images. Object detectors can complement a deep learning classification model in detecting infected cells since they eliminate the need to train on single-cell images and have been demonstrated to be more feasible for a different target domain.

Comparison of MRI Sequences to Predict ATRX Status Using Radiomics-Based Machine Learning.

ATRX is an important molecular marker according to the 2021 WHO classification of adult-type diffuse glioma. We aim to predict the ATRX mutation status non-invasively using radiomics-based machine learning models on MRI and to determine which MRI sequence is best suited for this purpose. In this retrospective study, we used MRI images of patients with histologically confirmed glioma, including the sequences T1w without and with the administration of contrast agent, T2w, and the FLAIR. Radiomics features were extracted from the corresponding MRI images by hand-delineated regions of interest. Data partitioning into training data and independent test data was repeated 100 times to avoid random effects. Feature preselection and subsequent model development were performed using Lasso regression. The T2w sequence was found to be the most suitable and the FLAIR sequence the least suitable for predicting ATRX mutations using radiomics-based machine learning models. For the T2w sequence, our seven-feature model developed with Lasso regression achieved a mean AUC of 0.831, a mean accuracy of 0.746, a mean sensitivity of 0.772, and a mean specificity of 0.697. In conclusion, for the prediction of ATRX mutation using radiomics-based machine learning models, the T2w sequence is the most suitable among the commonly used MRI sequences.