Partitioning Technique Research Articles

Learning-based Similarity Join for Power Data

The increasing instrumentation of physical and computing processes has given us unprecedented capabilities to collect massive volumes of time series. Power data is a typical kind of time series. Considering that the original time series data has ineluctable limitations such as uneven distribution, non-uniform length, poor sampling rate and noisy, we propose a learning=based similarity join for power data consisting of RNN encoder and matrix model. In addition, we develop the partition techniques by grouping process nodes following the matrix join model, ensuring the accuracy and efficiency of similarity join for data series. We conduct experiments on real data-set to evaluate the performance of our approach, demonstrating the effectiveness and scalability of our method.

Revalidation Technique on Landslide Susceptibility Modelling: An Approach to Local Level Disaster Risk Management in Kuala Lumpur, Malaysia

Landslide susceptibility modelling in tropical climates is hindered by incomplete inventory due to rapid development and natural processes that obliterate field evidence, making validation a challenge. Susceptibility modelling was conducted in Kuala Lumpur, Malaysia using a new spatial partitioning technique for cross-validation. This involved a series of two alternating east-west linear zones, where the first zone served as the training dataset and the second zone was the test dataset, and vice versa. The results show that the susceptibility models have good compatibility with the selected landslide conditioning factors and high predictive accuracy. The model with the highest area under curve (AUC) values (SRC = 0.92, PRC = 0.90) was submitted to the City Council of Kuala Lumpur for land use planning and development control. Rainfall-induced landslides are prominent within the study area, especially during the monsoon period. An extreme rainfall event in December 2021 that triggered 122 landslides provided an opportunity to conduct retrospective validation of the model; the high predictive capability (AUC of PRC = 0.93) was reaffirmed. The findings proved that retrospective validation is vital for landslide susceptibility modelling, especially where the inventory is not of the best quality. This is to encourage wider usage and acceptance among end users, especially decision-makers in cities, to support disaster risk management in a changing climate.

Performance Analysis and Deployment of Partitioning Strategies in Apache Spark

Data is flourishing day by day to a large extent, the data that need to be analyzed are not just large, but it may be high dimensional, heterogeneous, complex, unstructured, incomplete and noisy as well. Apache Spark framework is used for high-performance computing of Big Data. The proper division of the dataset can impact the degree of parallelism achieved. In Apache Spark, partitioning techniques help to manage the dataset in a distributed fashion. The optimal number of partitions and associated data with these partitions ensures the appropriate storage and quick access to data. Inbuilt libraries and flexibility utilizing the available techniques help Apache Spark to divide the dataset as a convenience. In this study, partitioning techniques available in Spark have been discussed and implemented. The pros and cons of the partitioning techniques have been highlighted. Furthermore, the performance of partitioning techniques has been compared in terms of execution time which help to select the suitable partitioning strategy for the different jobs.

Graph Wedgelets: Adaptive Data Compression on Graphs Based on Binary Wedge Partitioning Trees and Geometric Wavelets

We introduce graph wedgelets - a tool for data compression on graphs based on the representation of signals by piecewise constant functions on adaptively generated binary graph partitionings. The adaptivity of the partitionings, a key ingredient to obtain sparse representations of a graph signal, is realized in terms of recursive wedge splits adapted to the signal. For this, we transfer adaptive partitioning and compression techniques known for 2D images to general graph structures and develop discrete variants of continuous wedgelets and binary space partitionings. We prove that continuous results on best <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$m$</tex-math></inline-formula> -term approximation with geometric wavelets can be transferred to the discrete graph setting and show that our wedgelet representation of graph signals can be encoded and implemented in a simple way. Finally, we illustrate that this graph-based method can be applied for the compression of images as well.

Learning in horizontal scaling MongoDB data sharding on windows

MongoDB is a multi-storage NoSQL database. With the fast growth of technology, the number of large databases has exponentially increased, and relational databases cannot fulfill the need for managing such large amounts of data. To address this issue, intensive research was made on the NoSQL (Not Only SQL) data management model, specifically focusing on MongoDB, and a novel approach to manage large-scale remote sensing data is proposed. Social networking sites generate massive amounts of data, and Structured Query Language (SQL) and NoSQL are predominantly used for data storage. NoSQL proves to be a superior choice for web applications due to its ability to handle large data volumes, unlike Relational Database Management Systems (RDBMS). While various data partitioning techniques such as hash and round-robin exist, they are not efficient for small transactions involving only a few tuples. The results demonstrate that the proposed method of different segmentation overcomes the limitations of conventional approaches, facilitating horizontal expansion of the database and making it more suitable for managing extensive stored data. This research provides an essential technical support for effectively managing large amounts of stored data.

A Branch-independence-based Reliability Assessment Approach for Transmission Systems

This paper proposes a branch-independence-based reliability assessment approach for transmission systems. The approach consists of branch decoupling and state-space partition techniques. By integrating an impact-increment-based reliability index calculation model and the proposed branch decoupling technique, a proportion of sampled contingency states no longer need to be analyzed using the time-consuming optimal power flow (OPF) algorithm. In this way, the technique speeds up the calculation of reliability indices. Since first-order contingency states have a high probability of being sampled, we propose a state-space partition technique to replace first-order contingency state simulation with first-order contingency state enumeration. Consequently, the calculation of reliability indices is further accelerated by avoiding a large amount of repetitive OPF analyses during simulation process without affecting reliability index accuracy. The validity and applicability of our approach are verified using the IEEE 118-bus and IEEE 145-bus systems. Numerical results indicate that the proposed approach can improve computational efficiency without decreasing accuracy.

Low-Temperature Partitioning: A Simple Screening Method for Determining Diethylene Glycol in Beer by Gas Chromatography

Diethylene glycol is an extremely toxic substance to humans. Recently, cases of beer contamination in Brazil have raised awareness of the need for developing simple screening methods to evaluate this type of compound. This research developed a liquid-liquid extraction with low-temperature partitioning technique to determine diethylene glycol in beer via gas chromatography. Employing a flame-ionization detector simplifies the method, lowers its cost and therefore, it can be used as screening step to assess the possibility of contamination. A gas chromatograph coupled to a mass spectrometer would be used only for a confirmatory analysis. The optimized method was validated for the main figures of merit, and it proved to be adequate, with good values of recovery rate (94-106%), limit of detection (3.0 mg L-1), and quantification (10.0 mg L-1). Accuracy, in terms of repeatability and intermediate accuracy, showed variation coefficients lower than or equal to 20%. This method was applied to 28 samples of beers marketed in Brazil, and diethylene glycol was found above the limit of detection in three of them (10.7%). These results were confirmed by a gas chromatograph coupled to a mass spectrometer, which showed the reliability of the screening method for determining diethylene glycol in beer samples.

Opt-CoInfer: Optimal collaborative inference across IoT and cloud for fast and accurate CNN inference

For fast and accurate Convolutional Neural Network (CNN) inference of massive Internet of Things (IoT) data, Collaborative Inference (CI) based on partition and compression techniques needs to carefully select the collaboration scheme considering both application scenario and inference requirement. However, with emerging partition and compression techniques, it is hard to select a proper collaboration scheme due to the exponential growth of the scheme space and the time-consuming evaluation of a single scheme. Therefore, we present Opt-CoInfer to search the optimal collaboration scheme for collaborative inference (i.e., the fastest/most accurate scheme satisfying the latency/accuracy requirement in any specific scenario). Generally, Opt-CoInfer achieves the optimum by updating the local optimum and reducing the feasible set to empty in an iterative method. Specifically, in each iteration, Opt-CoInfer selects the promising scheme and updates the local optimum according to a statistical model built with the evaluated schemes, then reduces the feasible set by the local optimum and some key observations of collaboration schemes. Experiments show that Opt-CoInfer outperforms single-end and state-of-the-art CI approaches with the same inference requirements in various application scenarios (e.g., up to 3.49× faster with the same accuracy requirement and as low as 37.41% accuracy loss with the same latency requirement).

Topological and Spectral Properties of Wavy Zigzag Nanoribbons

Low-dimensional graphene-based nanomaterials are interesting due to their cutting-edge electronic and magnetic properties. Their large surface area, strong mechanical resistance, and electronic properties have enabled potential pharmaceutical and opto-electronic applications. Graphene nanoribbons (GNRs) are graphene strips of nanometer size possessing zigzag and armchair edge geometries with tunable widths. Despite the recent developments in the characterization, design and synthesis of GNRs, the study of electronic, magnetic and topological properties, GNRs continue to pose a challenge owing to their multidimensionality. In this study, we obtain the topological and electronic properties of a series of wave-like nanoribbons comprising nanographene units with zigzag-shaped edges. The edge partition techniques based on the convex components are employed to compute the mathematical formulae of molecular descriptors for the wave-like zigzag GNRs. We have also obtained the spectral and energetic properties including HOMO-LUMO gaps, bond delocalization energies, resonance energies, 13C NMR and ESR patterns for the GNRs. All of these computations reveal zero to very low HOMO-LUMO gaps that make these nanoribbons potential candidates for topological spintronics.

Flexible tree-structured regression models for discrete event times

Discrete hazard models are widely applied for the analysis of time-to-event outcomes that are intrinsically discrete or grouped versions of continuous event times. Commonly, one assumes that the effect of explanatory variables on the hazard can be described by a linear predictor function. This, however, may be not appropriate when non-linear effects or interactions between the explanatory variables occur in the data. To address this issue, we propose a novel class of discrete hazard models that utilizes recursive partitioning techniques and allows to include the effects of explanatory variables in a flexible data-driven way. We introduce a tree-building algorithm that inherently performs variable selection and facilitates the inclusion of non-linear effects and interactions, while the favorable additive form of the predictor function is kept. In a simulation study, the proposed class of models is shown to be competitive with alternative approaches, including a penalized parametric model and Bayesian additive regression trees, in terms of predictive performance and the ability to detect informative variables. The modeling approach is illustrated by two real-world applications analyzing data of patients with odontogenic infection and lymphatic filariasis.

Stochastic learning control of adiabatic speedup in a non-Markovian open qutrit system

Precise and efficient control of quantum systems is essential to perform quantum information-processing tasks. In terms of adiabatic speedup via leakage elimination operator approach, for a closed system, the ideal pulse control conditions have been theoretically derived by P-Q partitioning technique. However, it is a challenge to design the corresponding control pulses for an open system, which requires noisy environments to be addressed. In this paper, we apply the stochastic search procedures to an open qutrit system and successfully obtain the optimal control pulses for significant adiabatic speedup. The calculation results show that these optimal pulses allow us to acquire higher fidelities than the ideal pulses. The improvement of fidelity is large for relatively strong system-bath coupling strength and high bath temperature. For certain coupling strength and bath temperature, the maximal improvement can be achieved for a critical characteristic frequency which represents the memory time of the environment. Our investigation indicates that the stochastic search procedures are powerful tools to design control pulses for combating the detrimental effects of the environment.

Block Effective Hamiltonian Theory and Its Application.

Block effective Hamiltonian theory (BEHT) is presented in this work. Configuration interaction functions are divided into P, Q, and R spaces. Effective Hamiltonian is constructed with the partitioning technique within the P space. The eigenvalue problem of the effective Hamiltonian is then solved iteratively. It is demonstrated that the ground-state energies of N2, HF, and F2 calculated with BEHT converge to the multireference configuration interaction energies from below and the iteration number becomes smaller as BEHT energy becomes closer to the exact energy. The accuracy of BEHT is better than that of the second-order multireference perturbation theory, although the matrix elements in both methods are the same. The ionization potentials of the singlet state of HF, the doublet state of F, and the triplet state of NH and the potential energy curves of CH4, C2, and N2 are calculated with BEHT and compared with experiments and results of CASSCF, CCSD, and CCSD(T) and the results of the full configuration interaction if available. The iteration numbers are all less than 10 in this study. These calculations show the good performances of BEHT in comparison with other theoretical approximation methods.

Indexing Metric Spaces for Exact Similarity Search

With the continued digitization of societal processes, we are seeing an explosion in available data. This is referred to as big data. In a research setting, three aspects of the data are often viewed as the main sources of challenges when attempting to enable value creation from big data: volume, velocity, and variety. Many studies address volume or velocity, while fewer studies concern the variety. Metric spaces are ideal for addressing variety because they can accommodate any data as long as it can be equipped with a distance notion that satisfies the triangle inequality. To accelerate search in metric spaces, a collection of indexing techniques for metric data have been proposed. However, existing surveys offer limited coverage, and a comprehensive empirical study exists has yet to be reported. We offer a comprehensive survey of existing metric indexes that support exact similarity search: we summarize existing partitioning, pruning, and validation techniques used by metric indexes to support exact similarity search; we provide the time and space complexity analyses of index construction; and we offer an empirical comparison of their query processing performance. Empirical studies are important when evaluating metric indexing performance, because performance can depend highly on the effectiveness of available pruning and validation as well as on the data distribution, which means that complexity analyses often offer limited insights. This article aims at revealing strengths and weaknesses of different indexing techniques to offer guidance on selecting an appropriate indexing technique for a given setting, and to provide directions for future research on metric indexing.

Parallelism-Optimizing Data Placement for Faster Data-Parallel Computations

Systems performing large data-parallel computations, including online analytical processing (OLAP) systems like Druid and search engines like Elasticsearch, are increasingly being used for business-critical real-time applications where providing low query latency is paramount. In this paper, we investigate an underexplored factor in the performance of data-parallel queries: their parallelism. We find that to minimize the tail latency of data-parallel queries, it is critical to place data such that the data items accessed by each individual query are spread across as many machines as possible so that each query can leverage the computational resources of as many machines as possible. To optimize parallelism and minimize tail latency in real systems, we develop a novel parallelism-optimizing data placement algorithm that defines a linearly-computable measure of query parallelism, uses it to frame data placement as an optimization problem, and leverages a new optimization problem partitioning technique to scale to large cluster sizes. We apply this algorithm to popular systems such as Solr and MongoDB and show that it reduces p99 latency by 7-64% on data-parallel workloads.

An optimal real-time pricing strategy for aggregating distributed generation and battery storage systems in energy communities: A stochastic bilevel optimization approach

The expansion of distributed electricity generation and the increasing capacity of installed battery storage systems at the community level have posed challenges to efficient technical and economic operation of the power systems. With advances in smart-grid infrastructure, many innovative demand response business models have sought to tackle these challenges, while creating financial benefits for the participating actors. In this context, we propose an optimal real-time pricing (ORTP) approach for the aggregation of distributed energy resources within energy communities. We formulate the interaction between a community-owned profit-maximizing aggregator and the users (consumers with electricity generation and storage potential, known as “prosumagers”, and electric vehicles) as a stochastic bilevel disjunctive program. To solve the problem efficiently, we offer a novel solution algorithm, which applies a linear quasi-relaxation approach and an innovative dynamic partitioning technique. We introduce benchmark tariffs and solution algorithms and assess the performance of the proposed pricing strategy and solution algorithm in four case studies. Our results show that the ORTP strategy increases community welfare while providing useful grid services. Furthermore, our findings reveal the superior computational efficiency of our proposed solution algorithm in comparison to benchmark algorithms.

Anspannung: Introduction to concept and quantification of mental strain exemplified on data taken in five countries

In Dükers action theory Anspannung , which we translated as “psychological tension” (PT), is described as a directly experienced valid indicator for the extent of mental strain. In German-speaking regions the Category Partitioning technique (CP) has proven to be a useful method for accurately quantifying the experienced PT. Outside Germany, however, the concept of PT and the CP technique for measuring it have found little resonance, as it seemed that the central terms could not be meaningfully translated into English. To challenge these language barriers, test the applicability and usefulness of the PT concept, and evaluate the CP scaling method, we used the CP technique to quantify the level of PT required by 32 imagined everyday situations. To do this we adapted descriptions of the everyday situations from the German into English, Japanese, Korean, and Mandarin Chinese, and enrolled N = 158 participants from five countries (Canada, Germany, Japan, South Korea, and Taiwan). The results show a remarkable agreement between the data collected in the five cohorts. The experimental data point to the universality of the experience of mental load in culturally and linguistically diverse societies. They also point to the need to design scaling techniques so that respondents can describe their immediate sensations as they would in everyday life.

Partitioning local seismogram wavefields using continuous wavelet transform methods for IRIS wavefield experiment arrays

SUMMARY We applied nonlinear thresholding and scale–time gating in the continuous wavelet transform (CWT) domain to denoise, identify and characterize seismic phases contained in gradiometer and phased array waveforms of four seismic events recorded during the 2016 Incorporated Research Institutions of Seismology Wavefields Experiment in northern Oklahoma. A dense, 80-element three component phased array was subset from the linear array deployments to examine background noise, waveform coherence and seismic wave composition for local explosion and earthquake waveforms. CWT techniques were also used to significantly improve gradiometery analyses for data recorded by the geodetic array subexperiment. We observed as much as two orders of magnitude gain in the data signal-to-noise ratio. We also saw improvement in array beam quality after denoising the seismic data. Using the signal partitioning technique, we were able to extract and identify many phases based on their positions on the scale–time plane. CWT denoising and wavefield decomposition techniques also improved gradiometry analysis results from the 112-element geodetic array (also called the gradiometer) since waves could be separated before the computation of wave attributes. The operations of removing noise and gating out signal phases improved signal coherence across array records and provided clear P wave onsets on horizontal records, which can mitigate phase picking error and resulting event location uncertainty.

Calculating PI Related Indices and Its Polynomial of Hyaluronic Acid and Conjugates

A large number of medical experiments have confirmed that the features of drugs have a close correlation with their molecular structure. Drug properties can be obtained by studying the molecular structure of corresponding drugs. The calculation of the topological index of a chemical graph enables scientists to have a better understanding of the physical chemistry and biological characteristics of drugs. In this paper, we focus on Hyaluronic Acid-Curcumin/Paclitaxel conjugate which are widely used in the manufacture of anticancer drugs. Therefore in this paper we have computed the Padmakar-Ivan related indices and its polynomial of the Hyaluronic Acid-curcumin/paclitaxel conjugate by using edge partitioning technique.

Identification of geochemical anomalies related to mineralization: A case study from porphyry copper deposits in the Qulong-Jiama mining district of Tibet, China

Geochemical exploration data plays a vital role in mineral prospectivity mapping (MPM) for discovering unknown mineral deposits. In this study, compositional balance analysis (CoBA), unsupervised and supervised learning are collaboratively used to improve the practice of identification of geochemical anomalies related to mineralization in the Qulong-Jiama mining district of Tibet. For CoBA, five balances of geochemical elements were constructed by the sequential binary partition (SBP) technique, facilitating the interpretation of geochemical/geological processes and contrasting with geochemical anomalies generated by unsupervised and supervised learning. The iterative self-organizing data analysis techniques algorithm (ISODATA) and isolation forest (iForest) are used for preprocessing the geochemical data and therefore optimizing the appropriate training sets (e.g., positive and negative training samples) for supervised learning. Different training datasets including locations of known mineralization, randomly selected positive samples from resulting clusters by ISODATA and outliers by iForest are fed to the support vector machine (SVM) and random forests (RF) algorithm. The results of high AUC values indicate that supervised learning can effectively delineate prospective areas both with SVM and RF, however, the excessive large area of high-prospectivity and a bias toward known mineralization makes the outcomes of MPM unpractical. Nevertheless, a hybrid of the CoBA, unsupervised learning and supervised learning could alleviate such situations and provide insights into MPM using the geochemical data.

A subgroup dominance-based benefit of the doubt method for addressing rank reversals: A case study of the human development index in Europe

The benefit of the doubt (BoD) is a non-parametric weighting method that aims to maximize the relative composite indicator value of each decision-making unit (DMU). A well-known issue of BoD-based ranking of DMUs is the rank reversal problem when DMUs are deleted or added. The fundamental reason for rank reversals is that deleting (or adding) a DMU influences the generated weight by changing the frontier surface. To address the rank reversal problem, this paper proposes a subgroup dominance-based BoD model (SD-BoD). Based on the individual weights obtained from a classic BoD model, a dominance-based pairwise comparison mechanism is put forward to ensure comparability of different DMUs. Moreover, motivated by the data envelopment analysis idea, we construct a sequential frontier surface partition technique and establish a DMU ranking framework based on subgroup dominance, which helps reduce weight variation due to DMU addition or deletion. Finally, the effectiveness of the SD-BoD method is illustrated by comparing and analyzing the Human Development Indices of the 28 European regions.