Scalable Algorithm Research Articles

A Review on Data Mining Issues, Solution & Techniques

Data mining, the process of discovering patterns and knowledge from large datasets, has become a cornerstone of decision-making across various domains including business, healthcare, and finance. This paper reviews the current landscape of data mining applications, exploring the diverse techniques employed and the challenges faced. Key issues include data quality, privacy concerns, and the scalability of mining algorithms in the face of increasingly large and complex datasets. We discuss solutions to these challenges, such as advancements in data cleaning methodologies, privacy-preserving techniques, and the development of scalable algorithms. Additionally, we provide a comprehensive overview of current data mining techniques, including clustering, classification, and association rule mining, and evaluate their effectiveness in addressing real-world problems. The paper aims to provide understanding of both the potential and limitations of data mining, offering insights into future research directions and technological advancements needed to overcome existing hurdles.

Certified algorithms for equilibrium states of local quantum Hamiltonians

Predicting observables in equilibrium states is a central yet notoriously hard question in quantum many-body systems. In the physically relevant thermodynamic limit, certain mathematical formulations of this task have even been shown to result in undecidable problems. Using a finite-size scaling of algorithms devised for finite systems often fails due to the lack of certified convergence bounds for this limit. In this work, we design certified algorithms for computing expectation values of observables in the equilibrium states of local quantum Hamiltonians, both at zero and positive temperature. Importantly, our algorithms output rigorous lower and upper bounds on these values. This allows us to show that expectation values of local observables can be approximated in finite time, contrasting related undecidability results. When the Hamiltonian is commuting on a 2-dimensional lattice, we prove fast convergence of the hierarchy at high temperature and as a result for a desired precision ε, local observables can be approximated by a convex optimization program of quasi-polynomial size in 1/ε.

Dynamic Byzantine Fault-Tolerant Consensus Algorithm with Supervised Feedback Mechanisms

Among the existing consensus algorithms, there are very few low-latency consensus algorithms that can simultaneously take into account node dynamics, fault tolerance, and scalability, and which are applicable to large-scale open scenarios where low latency is required. Therefore, this paper proposes a low-latency scalable dynamic consensus algorithm with high fault tolerance utilizing a combination of layered and threshold signature technologies, known briefly as LTSBFT. Firstly, LTSBFT achieves linear communication complexity through the utilization of threshold signature technology and a two-layer structure. Secondly, the mutual supervision feedback mechanism among nodes enables the attainment of linear complexity for reaching consensus on the faulty upper-layer nodes during the view-change. Lastly, a node dynamic protocol was proposed for the first time to support dynamic changes in the number of nodes during the consensus process. That is to say, consensus can still be reached when the number of nodes dynamically changes without interrupting the client’s request for consensus in the network. The experimental results indicate that LTSBFT exhibits lower communication latency and higher throughput compared to the classic HotStuff and PBFT algorithms. Furthermore, in comparison to double-layer PBFT, the LTSBFT has been demonstrated to have improved fault tolerance.

A Blockchain and PKI-Based Secure Vehicle-to-Vehicle Energy-Trading Protocol

With the increasing awareness for sustainable future and green energy, the demand for electric vehicles (EVs) is growing rapidly, thus placing immense pressure on the energy grid. To alleviate this, local trading between EVs should be encouraged. In this paper, we propose a blockchain and public key infrastructure (PKI)-based secure vehicle-to-vehicle (V2V) energy-trading protocol. A permissioned blockchain utilizing the proof of authority (PoA) consensus and smart contracts is used to securely store data. Encrypted communication is ensured through transport layer security (TLS), with PKI managing the necessary digital certificates and keys. A multi-leader, multi-follower Stackelberg game-based trade algorithm is formulated to determine the optimal energy demands, supplies, and prices. Finally, we propose a detailed communication protocol that ties all the components together, enabling smooth interaction between them. Key findings, such as system behavior and performance, scalability of the trade algorithm and the blockchain, smart contract execution costs, etc., are presented through numerical results by implementing and simulating the protocol in various scenarios. This work not only enhances local energy trading among EVs, encouraging efficient energy usage and reducing burden on the power grid, but also paves a way for future research in sustainable energy management.

Lightweight and accurate aphid detection model based on an improved deep-learning network

Rapid and accurate detection of bamboo aphids can help prevent large-scale aphid infestations from occurring, which is of great significance for increasing bamboo shoot production and economic benefits. Herein, a lightweight and accurate model, SCA-YOLOv5s, was established by integrating ShuffleNetv2 and Coordinate Attention with the YOLOv5s model to detect Takecallis taiwanus on the yellow sticky traps. Specifically, we first replaced the backbone network of YOLOv5s with ShuffleNetv2 to reduce the number of parameters and computational complexity of the model. Second, an anchor optimization method was proposed by combining linear scaling and k-means algorithm to generate appropriate anchor boxes for detecting small-sized alate aphids. Third, the coordinate attention mechanism was added to the neck network to improve the feature extraction ability. To verify the performance of the proposed SCA-YOLOv5s model, eight detection models were constructed with existing deep learning methods, including SSD300, YOLOv3, Faster R-CNN, YOLOv4, YOLOv4-Tiny, YOLOX-Tiny, YOLOv7-Tiny, and YOLOv5s. Results reveal that the SCA-YOLOv5s model achieved higher detection accuracy than the other eight models. Its mean average precision reached 92.2 %. The proposed model has a size of only 6.7 MB, its floating-point operations (FLOPs) is 7.4 × 109, its inference time is 6.6 ms, and compared with YOLOv5s, it is 53.47 % smaller in model size, 55.15 % lower in FLOPs, and 0.8 ms faster in inference time. The results indicate that the proposed model can maintain high detection accuracy while minimizing computation and inference time, which is crucial for deployment in remote areas with low information technology. This study provides valuable technical support for the control of aphids in bamboo forests.

GPU accelerated Staggered Update Procedure (SUP)

The advancement in programmable capability of graphics hardware has paved new opportunities in the domain of high performance computing (HPC). The computational fluid dynamics (CFD) community, being a significant user of HPC, has started exploiting the inherent data parallelism in the numerical solvers to be able to make efficient use of these many-core, high throughput accelerator based processors. In the present work, we examine the process of accelerating our CPU based Staggered Update Procedure (SUP) solver, i.e., a higher order accurate cell-centred finite volume solver by off-loading the computationally most expensive region of the code pertaining to the explicit residual computation. We have adopted OpenACC, a directive based programming model to expose parallelism in the code. The framework evolved for GPU porting in the context of SUP is also of value to those intending to port their CFD solvers based on classical finite volume methodology. The performance analysis is conducted using scalar convection–diffusion equations in both two- and three-dimensions. The findings demonstrate a speedup factor of 9 (in case of 2D) and 28 (in case of 3D) when considering the explicit residual alone, achieved with a single NVIDIA Tesla V100 GPU card. In addition, we could establish superior algorithmic scalability by the way of recovering near perfect serial performance, on the heterogeneous CPU+GPU architecture. Further, overall code acceleration can be achieved by porting other parts of the solver on GPU.

Benchmarking Quantum Generative Learning: A Study on Scalability and Noise Resilience using QUARK

AbstractQuantum computing promises a disruptive impact on machine learning algorithms, taking advantage of the exponentially large Hilbert space available. However, it is not clear how to scale quantum machine learning (QML) to industrial-level applications. This paper investigates the scalability and noise resilience of quantum generative learning applications. We consider the training performance in the presence of statistical noise due to finite-shot noise statistics and quantum noise due to decoherence to analyze the scalability of QML methods. We employ rigorous benchmarking techniques to track progress and identify challenges in scaling QML algorithms, and show how characterization of QML systems can be accelerated, simplified, and made reproducible when the QUARK framework is used. We show that QGANs are not as affected by the curse of dimensionality as QCBMs and to which extent QCBMs are resilient to noise.

Efficient First-Order Algorithms for Large-Scale, Non-Smooth Maximum Entropy Models with Application to Wildfire Science.

Maximum entropy (MaxEnt) models are a class of statistical models that use the maximum entropy principle to estimate probability distributions from data. Due to the size of modern data sets, MaxEnt models need efficient optimization algorithms to scale well for big data applications. State-of-the-art algorithms for MaxEnt models, however, were not originally designed to handle big data sets; these algorithms either rely on technical devices that may yield unreliable numerical results, scale poorly, or require smoothness assumptions that many practical MaxEnt models lack. In this paper, we present novel optimization algorithms that overcome the shortcomings of state-of-the-art algorithms for training large-scale, non-smooth MaxEnt models. Our proposed first-order algorithms leverage the Kullback-Leibler divergence to train large-scale and non-smooth MaxEnt models efficiently. For MaxEnt models with discrete probability distribution of n elements built from samples, each containing m features, the stepsize parameter estimation and iterations in our algorithms scale on the order of O(mn) operations and can be trivially parallelized. Moreover, the strong ℓ1 convexity of the Kullback-Leibler divergence allows for larger stepsize parameters, thereby speeding up the convergence rate of our algorithms. To illustrate the efficiency of our novel algorithms, we consider the problem of estimating probabilities of fire occurrences as a function of ecological features in the Western US MTBS-Interagency wildfire data set. Our numerical results show that our algorithms outperform the state of the art by one order of magnitude and yield results that agree with physical models of wildfire occurrence and previous statistical analyses of wildfire drivers.

Music Similarity Ranking: Case Study Fairphonic

Fairphonic Pte Ltd is a technology company operating in the music sector. Fairphonic provides services to detect content on social media that is suspected of copyright infringement. Fairphonic utilizes audio features for the detection process. The current algorithm used by Fairphonic requires pairwise comparison, and the content to be compared is collected through a scraping process immediately after the process is run. Fairphonic has hundreds of thousands of music data in their database. Fairphonic desires a more scalable algorithm to compare an input music piece with the entire Fairphonic music catalog. This research uses features such as Harmonic Pitch Class Profile (HPCP), Chroma, and Rhythm Pattern. The study compares previously researched algorithms, namely binary similarity matrix, Euclidean distance, and similarity matrix profile. The results show that the combination of HPCP with the binary similarity matrix yields the highest Mean Average Precision of 0.989. Speed testing by performing comparisons 10 times shows that the combination of Chroma and the similarity matrix profile is 72% faster compared to the combination of HPCP with the binary similarity matrix. The author recommends the Chroma and similarity matrix profile algorithm for music similarity ranking due to its faster process.

The Multiple Roles of Structured Multipurpose Databases in Fusion Research

Most databases in fusion research are devoted to a single topic, such as energy confinement, H-modes, profiles, or disruptions. To allow for a wider range of analysis, modeling, and validation tasks, the JETPEAK broad-based multipurpose database has been developed for JET. This database currently includes 27 065 stationary state (∂/∂t ≈ 0) samples and nearly 1000 scalar, one-dimensional (profiles), and two-dimensional (R and Z dependent) variables grouped into topical structures. A similar database has been created for the Tokamak à Configuration Variable (TCV), comprising 65 000 samples reaching back to early TCV operation in the 1990s. The breadth and flexibility of these databases allows them to be used for a wide variety of investigations such as modeling tasks, confinement scaling, testing, validation and benchmarking of algorithms and modeling codes, and long-term monitoring of device conditions, as well as for documentation.

PrismX: A Single-Machine System for Querying Big Graphs

We demonstrate PrismX (PRAM with SSDs as Memory eXtension), a single-machine system for graph analytics. PrismX allows users to make practical use of existing PRAM algorithms without any change. To cope with the limited DRAM capacity, it employs NVMe SSDs as memory extension. Leveraging graph preprocessing, PrismX implements a series of system optimization strategies, which automatically and transparently adapt to the runtime workload, no matter whether the computation is CPU-bound or I/O-bound. We demonstrate PrismX for its (1) ease of programming by reusing PRAM algorithms, (2) efficiency by comparing with the state-of-the-art graph systems, single-machine or multi-machine, in-memory or out-of-core; (3) parallel scalability of in-memory PRAM algorithms, reducing runtime when more CPU cores are available; and (4) applications in credit risk assessment.

Evolutionary Constrained Multiobjective Optimization: Scalable High-Dimensional Constraint Benchmarks and Algorithm

Evolutionary constrained multiobjective optimization has received extensive attention and research in the past two decades, and a lot of benchmarks have been proposed to test the effect of the constrained multiobjective evolutionary algorithms (CMOEAs). Specially, the constraint functions are highly correlated with the objective values, which makes the features of constraints too monotonic and differ from the properties of the real-world problems. Accordingly, previous CMOEAs cannot solve real-world problems well, which generally involve decision space constraints with multi-modal/non-linear features. Therefore, we propose a new benchmark framework and design a suite of new test functions with scalable high-dimensional decision space constraints. To be specific, different high-dimensional constraint functions and mixed linkages in variables are considered to be close to realistic features. In this framework, several parameter interfaces are provided, so that users can easily adjust the parameters to obtain the variant functions and test the generalization performance of the algorithms. Different types of existing CMOEAs are employed to test the use of the proposed test functions, and the results show that they are easy to fall into local feasible regions. Therefore, we improve one evolutionary multitasking-based CMOEA to better handle these problems, in which a new search algorithm is designed to enhance the search abilities of populations. Compared with the existing CMOEAs, the proposed CMOEA presents better performance.

Rock: Cleaning Data with both ML and Logic Rules

We demonstrate Rock, a system for cleaning relational data. Rock highlights the following unique features: (1) it extends logic rules by embedding machine learning models as predicates, to benefit from both ML and logic deduction; (2) it supports entity resolution, conflict resolution, timeliness deduction and missing data imputation in a unified process; and (3) it provides parallelly scalable algorithms for rule discovery, error detection and error correction, in batch and incremental modes. We will demonstrate Rock for its (a) easy-to-use interface, (b) scalability when cleaning large datasets, (c) accuracy for detecting and correcting errors across multiple tables, and (d) applications at banks and HR departments.

Rapid Initialization Method of Unmanned Aerial Vehicle Swarm Based on VIO-UWB in Satellite Denial Environment

In environments where satellite signals are blocked, initializing UAV swarms quickly is a technical challenge, especially indoors or in areas with weak satellite signals, making it difficult to establish the relative position of the swarm. Two common methods for initialization are using the camera for joint SLAM initialization, which increases communication burden due to image feature point analysis, and obtaining a rough positional relationship using prior information through a device such as a magnetic compass, which lacks accuracy. In recent years, visual–inertial odometry (VIO) technology has significantly progressed, providing new solutions. With improved computing power and enhanced VIO accuracy, it is now possible to establish the relative position relationship through the movement of drones. This paper proposes a two-stage robust initialization method for swarms of more than four UAVs, suitable for larger-scale satellite denial scenarios. Firstly, the paper analyzes the Cramér–Rao lower bound (CRLB) problem and the moving configuration problem of the cluster to determine the optimal anchor node for the algorithm. Subsequently, a strategy is used to screen anchor nodes that are close to the lower bound of CRLB, and an optimization problem is constructed to solve the position relationship between anchor nodes through the relative motion and ranging relationship between UAVs. This optimization problem includes quadratic constraints as well as linear constraints and is a quadratically constrained quadratic programming problem (QCQP) with high robustness and high precision. After addressing the anchor node problem, this paper simplifies and improves a fast swarm cooperative positioning algorithm, which is faster than the traditional multidimensional scaling (MDS) algorithm. The results of theoretical simulations and actual UAV tests demonstrate that the proposed algorithm is advanced, superior, and effectively solves the UAV swarm initialization problem under the condition of a satellite signal rejection.

Re2Pair: Increasing the Scalability of RePair by Decreasing Memory Usage.

The RePair compression algorithm produces a context-free grammar by iteratively substituting the most frequently occurring pair of consecutive symbols with a new symbol until all consecutive pairs of symbols appear only once in the compressed text. It is widely used in the settings of bioinformatics, machine learning, and information retrieval where random access to the original input text is needed. For example, in pangenomics, RePair is used for random access to a population of genomes. BigRePair improves the scalability of the original RePair algorithm by using Prefix-Free Parsing (PFP) to preprocess the text prior to building the RePair grammar. Despite the efficiency of PFP on repetitive text, there is a scalability issue with the size of the parse which causes a memory bottleneck in BigRePair. In this paper, we design and implement recursive RePair (denoted as ), which builds the RePair grammar using recursive PFP. Our novel algorithm faces the challenge of constructing the RePair grammar without direct access to the parse of text, relying solely on the dictionary of the text and the parse and dictionary of the parse of the text. We compare to BigRePair using SARS-CoV-2 haplotypes and haplotypes from the 1000 Genomes Project. We show that our method achieves over a 40% peak memory reduction and a speed up ranging between 12% to 79% compared to BigRePair when compressing the largest input texts in all experiments. is made publicly available under the GNU public license here: https://github.com/jkim210/Recursive-RePair.

A survey on scalable consensus algorithms for blockchain technology

The process of reaching an agreement on a value within a distributed network, known as a consensus problem, is a defining feature of blockchain. This consensus problem can be seen in various applications like load balancing, transaction validation in blockchain, and distributed computing. In recent years, many researchers have provided solutions to this problem. Hence we have presented a survey in which we delved into blockchain consensus algorithms and conducted a comparative analysis of all the consensus algorithms to provide information about each protocol’s advantages and drawbacks. This survey starts with the standard proof-of-work consensus protocol applied in bitcoin cryptocurrency and its limitations on the ground of the following parameters: throughput (transactions per second), latency, forks, fault tolerance, double spending attacks, and power consumption. The rest of the consensus algorithms in this paper have been systematically covered to address the limitations of proof-of-work. This paper also covered Raft and PBFT consensus algorithms suitable for permissioned networks. Although the PBFT consensus protocol has a high throughput and a low latency, it has limited node scalability. The PBFT has a low byzantine fault tolerant rate. This paper also covers PoEWAL for blockchain-based IoT applications and WBFT, which prevents corrupt nodes from taking part in consensus. A comparative analysis of the consensus algorithms provides an explicit knowledge of the present research, which also offers guidance for future study.

Prime Time Tactics—Sieve Tweaks and Boosters

In a landscape where interest in prime sieving has waned and practitioners are few, we are still hoping for a domain renaissance, fueled by a resurgence of interest and a fresh wave of innovation. Building upon years of extensive research and experimentation, this article aims to contribute by presenting a heterogeneous compilation of generic tweaks and boosters aimed at revitalizing prime sieving methodologies. Drawing from a wealth of resurfaced knowledge and refined sieving algorithms, techniques, and optimizations, we unveil a diverse array of strategies designed to elevate the efficiency, accuracy, and scalability of prime sieving algorithms; these tweaks and boosters represent a synthesis of old wisdom and new discoveries, offering practical guidance for researchers and practitioners alike.

Rule learning by modularity

In this paper, we present a modular methodology that combines state-of-the-art methods in (stochastic) machine learning with well-established methods in inductive logic programming (ILP) and rule induction to provide efficient and scalable algorithms for the classification of vast data sets. By construction, these classifications are based on the synthesis of simple rules, thus providing direct explanations of the obtained classifications. Apart from evaluating our approach on the common large scale data sets MNIST, Fashion-MNIST and IMDB, we present novel results on explainable classifications of dental bills. The latter case study stems from an industrial collaboration with Allianz Private Krankenversicherung which is an insurance company offering diverse services in Germany.

Visibility graph-based covariance functions for scalable spatial analysis in non-convex partially Euclidean domains.

We present a new method for constructing valid covariance functions of Gaussian processes for spatial analysis in irregular, non-convex domains such as bodies of water. Standard covariance functions based on geodesic distances are not guaranteed to be positive definite on such domains, while existing non-Euclidean approaches fail to respect the partially Euclidean nature of these domains where the geodesic distance agrees with the Euclidean distances for some pairs of points. Using a visibility graph on the domain, we propose a class of covariance functions that preserve Euclidean-based covariances between points that are connected in the domain while incorporating the non-convex geometry of the domain via conditional independence relationships. We show that the proposed method preserves the partially Euclidean nature of the intrinsic geometry on the domain while maintaining validity (positive definiteness) and marginal stationarity of the covariance function over the entire parameter space, properties which are not always fulfilled by existing approaches to construct covariance functions on non-convex domains. We provide useful approximations to improve computational efficiency, resulting in a scalable algorithm. We compare the performance of our method with those of competing state-of-the-art methods using simulation studies on synthetic non-convex domains. The method is applied to data regarding acidity levels in the Chesapeake Bay, showing its potential for ecological monitoring in real-world spatial applications on irregular domains.

Fast-forwarding molecular ground state preparation with optimal control on analog quantum simulators

We show that optimal control of the electron dynamics is able to prepare molecular ground states, within chemical accuracy, with evolution times approaching the bounds imposed by quantum mechanics. We propose a specific parameterization of the molecular evolution only in terms of interaction already present in the molecular Hamiltonian. Thus, the proposed method solely utilizes quantum simulation routines, retaining their favorable scalings. Due to the intimate relationships between variational quantum algorithms and optimal control, we compare, when possible, our results with state-of-the-art methods in the literature. We found that the number of parameters needed to reach chemical accuracy and algorithmic scaling is in line with compact adaptive strategies to build variational Ansätze. The algorithm, which is also suitable for quantum simulators, is implemented by emulating a digital quantum processor (up to 16 qubits) and tested on different molecules and geometries spanning different degrees of electron correlation.