Scalable Algorithm Research Articles

Recovery of biological signals lost in single-cell batch integration with CellANOVA.

Data integration to align cells across batches has become a cornerstone of single-cell data analysis, critically affecting downstream results. Currently, there are no guidelines for when the biological differences between samples are separable from batch effects. Here we show that current paradigms for single-cell data integration remove biologically meaningful variation and introduce distortion. We present a statistical model and computationally scalable algorithm, CellANOVA (cell state space analysis of variance), that harnesses experimental design to explicitly recover biological signals that are erased during single-cell data integration. CellANOVA uses a 'pool-of-controls' design concept, applicable across diverse settings, to separate unwanted variation from biological variation of interest and allow the recovery of subtle biological signals. We apply CellANOVA to diverse contexts and validate the recovered biological signals by orthogonal assays. In particular, we show that CellANOVA is effective in the challenging case of single-cell and single-nucleus data integration, where it recovers subtle biological signals that can be validated and replicated by external data.

The real-time application and effectiveness assessment of an intelligent physical fitness testing system in physical training

An intelligent physical fitness testing system leverages advanced technologies to monitor and evaluate individuals’ fitness levels accurately. It integrates real-time data acquisition, and analysis, to support personalized physical training and health management. This study aims to evaluate the practical application and effectiveness of an intelligent system for real-time physical fitness testing in the context of physical training. Our suggested model employs portable sensing devices and we proposed a novel Northern Goshawk optimization-driven Gate Customized Long Short-Term Memory (NG-GC-LSTM) for enhancing accuracy in evaluating the individuals’ physical fitness levels. Data acquisition involves gathering bio-sensing data from 25 individuals during diverse physical training activities. The Min-Max Scaling algorithm is utilized to pre-process the obtained sensor data. We employed a Short-Time Fourier Transform (STFT) for extracting crucial features from the processed data. In our proposed framework, the NG optimization algorithm iteratively fine-tunes the GC-LSTM architecture for the accurate evaluation of an established intelligent physical fitness testing system. The recommended model is executed in Python software. During the result analysis phase, we assess the efficacy of our model’s performance across a variety of parameters. Additionally, we conduct comparative analyses with existing methodologies. The obtained outcomes demonstrate the efficacy and superiority of the suggested framework.

Data Collection and Preprocessing in Web Usage Mining: Implementation and Analysis

Data collection and data preprocessing are crucial stages in web usage mining, mainly because of the unstructured, diverse, and noisy nature of log data. During data collection, log file datasets are loaded and merged. Effective and comprehensive data preprocessing plays a vital role in ensuring the efficiency and scalability of algorithms used in the pattern discovery phase of web usage mining. This work aims to address these phases by introducing two innovative approaches. The first approach focuses on determining the device used for accessing the web, distinguishing between computers and mobile devices. The second approach aims to determine user sessions and complete paths by utilizing the referrer URL. The entire preprocessing pipeline has been implemented using the C# programming language, and the source code is available on GitHub at the following link: https://github.com/Mohammed91/Web-Usage-Mining.

MUPPAAL: Efficient Elimination and Reduction of Useless Mutants in Real‐Time Model‐Based Systems

ABSTRACTTo assess test quality, mutation testing (MT) creates mutants by injecting artificial faults into the system and evaluates the ability of tests to distinguish these mutants. Tests distinguishing more mutants have also been proven empirically to detect more real faults. MT has been applied to many domains. We focus on MT for timed safety‐critical systems modelled as Timed Automata (TA). While powerful, MT usually yields equivalent and duplicate mutants, the former having the same behaviour as the original system and the latter other mutants. Such useless mutants bring no value, waste execution time and can be difficult to detect. We integrate useless mutant detection and removal strategies in our mutation framework MUPPAAL. MUPPAAL leverages existing equivalence‐avoiding mutation operators and focuses on detecting mutant duplicates using a scalable bisimulation algorithm and a fast approximate one based on biased simulation. We also demonstrate how to design an operator that reduces the occurrence of mutant duplicates. We evaluate MUPPAAL on six systems, demonstrating that (1) mutant duplicates account for up to 32% of all generated mutants, (2) our bisimulation approach scales effectively with these systems and (3) biased simulations further enhance performance. Our heuristic is 10 times faster than bisimulation and limits the exploration to two times the number of exact duplicates compared to up to 10 times for the baseline.

Robustly learning the Hamiltonian dynamics of a superconducting quantum processor

Precise means of characterizing analog quantum simulators are key to developing quantum simulators capable of beyond-classical computations. Here, we precisely estimate the free Hamiltonian parameters of a superconducting-qubit analog quantum simulator from measured time-series data on up to 14 qubits. To achieve this, we develop a scalable Hamiltonian learning algorithm that is robust against state-preparation and measurement (SPAM) errors and yields tomographic information about those SPAM errors. The key subroutines are a novel super-resolution technique for frequency extraction from matrix time-series, tensorESPRIT, and constrained manifold optimization. Our learning results verify the Hamiltonian dynamics on a Sycamore processor up to sub-MHz accuracy, and allow us to construct a spatial implementation error map for a grid of 27 qubits. Our results constitute an accurate implementation of a dynamical quantum simulation that is precisely characterized using a new diagnostic toolkit for understanding, calibrating, and improving analog quantum processors.

Parallelization of the Bison Algorithm Applied to Data Classification

In data science and machine learning, efficient and scalable algorithms are paramount for handling large datasets and complex tasks. Classification algorithms, in particular, play a crucial role in a wide range of applications, from image recognition and natural language processing to fraud detection and medical diagnosis. Traditional classification methods, while effective, often struggle with scalability and efficiency when applied to massive datasets. This challenge has driven the development of innovative approaches that leverage modern computational frameworks and parallel processing capabilities. This paper presents the Bison Algorithm, applied to classification problems. The algorithm, inspired by the social behavior of bison, aims to enhance the accuracy of classification tasks. The Bison Algorithm is implemented using PySpark, leveraging the distributed computing power to handle large datasets efficiently. This study compares the performance of the Bison Algorithm on several dataset sizes using speedup and scaleup as the performance measure.

MDTA: An efficient, scalable and fast Multiple Disjoint Tree Algorithm for dynamic environments

Emerging applications such as telemedicine, the tactile Internet or live streaming place high demands on low latency to ensure a satisfactory Quality of Experience (QoE). In these scenarios the use of trees can be particularly interesting to efficiently deliver traffic to groups of users because they further enhance network performance by providing redundancy and fault tolerance, ensuring service continuity when network failure or congestion scenarios occur. Furthermore, if trees are isolated from each other (they do not share common communication elements as links and/or nodes), their benefits are further enhanced since events such as failures or congestion in one tree do not affect others. However, the challenge of computing fully disjoint trees (both link- and node-disjoint) introduces significant mathematical complexity, resulting in longer computation times, which negatively impacts latency-sensitive applications.In this article, we propose a novel algorithm designed to rapidly compute multiple fully (either link- or node-) disjoint trees while maintaining efficiency and scalability, specifically focused on targeting the low-latency requirements of emerging services and applications. The proposed algorithm addresses the complexity of ensuring disjointedness between trees without sacrificing performance. Our solution has been tested in a variety of network environments, including both wired and wireless scenarios.The results showcase that our proposed method is approximately 100 times faster than existing techniques, while achieving a comparable success rate in terms of number of obtained disjoint trees. This significant improvement in computational speed makes our approach highly suitable for the low-latency requirements of next-generation networks.

A Review of Generalized Linear Latent Variable Models and Related Computational Approaches

ABSTRACTGeneralized linear latent variable models (GLLVMs) have become mainstream models in this analysis of correlated, m‐dimensional data. GLLVMs can be seen as a reduced‐rank version of generalized linear mixed models (GLMMs) as the latent variables which are of dimension induce a reduced‐rank covariance structure for the model. Models are flexible and can be used for various purposes, including exploratory analysis, that is, ordination analysis, estimating patterns of residual correlation, multivariate inference about measured predictors, and prediction. Recent advances in computational tools allow the development of efficient, scalable algorithms for fitting GLLMVs for any response distribution. In this article, we discuss the basics of GLLVMs and review some options for model fitting. We focus on methods that are based on likelihood inference. The implementations available in R are compared via simulation studies and an example illustrates how GLLVMs can be applied as an exploratory tool in the analysis of data from community ecology.

Patterns of referral to interprofessional services among frail older adults presenting to emergency departments in Canada.

Geriatric Emergency Department (ED) Guidelines recommend optimizing transitions of care for older patients with complex needs. In this study, we investigated referral patterns to interprofessional services, including occupational therapy, physiotherapy, dietician, social work, home care, and specialized geriatric services, among older adults presenting to the ED with high-risk characteristics. We recruited community-dwelling older adults presenting to 10 EDs across Ontario, Quebec, and Newfoundland, Canada, from April 2017 to July 2018. To observe processes of care in the ED, we deployed a two-stage high-risk case-finding and focused comprehensive assessment process based on the interRAI ED-Screener and ED Contact Assessment to identify and characterize older adults at high risk. We analyzed the secondary data using descriptive statistics and logistic regression. We screened 5265 individuals with the ED Screener, further assessed 1479 with the ED Contact Assessment, and analyzed data from a subset of 1055 community-dwelling older adults assessed with the ED Contact Assessment. Participants in our study sample had a mean age of 83 years, 58% were female, and many had a complex burden of cognitive and functional impairment and social needs. Over half of this high-needs sample were referred to general home care services (62.7%), occupational therapy (59.3%), and physiotherapy services (55.2%), while 16% were referred to specialized geriatric services. We also found a significant positive association between interprofessional referrals and the Assessment Urgency Algorithm and Institutional Risk Scale. The most important determinants of referral to interprofessional services were hospital province, functional, clinical, and social burden and support measures. The referral patterns identified suggest that patient needs and risk intensity did not always guide referral patterns in the Canadian EDs investigated. We suggest that EDs critically examine the appropriateness of their documentation and referral systems for supporting person-centered care provision.

Dynamic Programming Implementation for Delivery Route Optimization in E-Commerce Logistics

The rapid growth of e-commerce has created new challenges in logistics optimization, particularly in terms of delivery route efficiency. This research develops a dynamic programming model to optimize delivery routes in the context of e-commerce in Indonesia. Using a modified Vehicle Routing Problem with Time Windows (VRPTW) approach, we implemented an algorithm that considers various factors such as distance, time, and cost. Simulations using synthetic datasets showed efficiency improvements of 18.7% in travel distance and 22.3% in delivery time compared to conventional methods. Field trials with an e-commerce partner resulted in a 21.5% reduction in travel distance and an increase in on-time delivery rate from 87% to 94%. Sensitivity analysis revealed that the algorithm's performance is most affected by demand fluctuations and changes in traffic conditions. Implementation challenges include integration with existing systems and consideration of workforce impact. This research opens avenues for further development in algorithm scalability, integration of sustainability factors, and adaptation to various geographical contexts, demonstrating significant potential for improving e-commerce logistics efficiency in the future.

A robust approach to Gaussian process implementation

Abstract. Gaussian process (GP) regression is a flexible modeling technique used to predict outputs and to capture uncertainty in the predictions. However, the GP regression process becomes computationally intensive when the training spatial dataset has a large number of observations. To address this challenge, we introduce a scalable GP algorithm, termed MuyGPs, which incorporates nearest-neighbor and leave-one-out cross-validation during training. This approach enables the evaluation of large spatial datasets with state-of-the-art accuracy and speed in certain spatial problems. Despite these advantages, conventional quadratic loss functions used in the MuyGPs optimization, such as root mean squared error (RMSE), are highly influenced by outliers. We explore the behavior of MuyGPs in cases involving outlying observations and, subsequently, develop a robust approach to handle and mitigate their impact. Specifically, we introduce a novel leave-one-out loss function based on the pseudo-Huber function (LOOPH) that effectively accounts for outliers in large spatial datasets within the MuyGPs framework. Our simulation study shows that the LOOPH loss method maintains accuracy despite outlying observations, establishing MuyGPs as a powerful tool for mitigating unusual observation impacts in the large data regime. In the analysis of US ozone data, MuyGPs provides accurate predictions and uncertainty quantification, demonstrating its utility in managing data anomalies. Through these efforts, we advance the understanding of GP regression in spatial contexts.

Amoeba Monte Carlo algorithms for random trees with controlled branching activity: Efficient trial move generation and universal dynamics.

The reptation Monte Carlo algorithm is a simple, physically motivated and efficient method for equilibrating semidilute solutions of linear polymers. Here, we propose two simple generalizations for the analog Amoeba algorithm for randomly branching chains, which allowus to efficiently deal with random trees with controlled branching activity. We analyze the rich relaxation dynamics of Amoeba algorithms and demonstrate the existence of an unexpected scaling regime for the tree relaxation. Our results suggest that the equilibration time for Amoeba algorithms scales in general like N^{2}〈n_{lin}〉^{Δ}, where N denotes the number of tree nodes, 〈n_{lin}〉 the mean number of linear segments the trees are composed of, and Δ≃0.4.

Robust brain MRI image classification with SIBOW-SVM

Primary Central Nervous System tumors in the brain are among the most aggressive diseases affecting humans. Early detection and classification of brain tumor types, whether benign or malignant, glial or non-glial, is critical for cancer prevention and treatment, ultimately improving human life expectancy. Magnetic Resonance Imaging (MRI) is the most effective technique for brain tumor detection, generating comprehensive brain scans. However, human examination can be error-prone and inefficient due to the complexity, size, and location variability of brain tumors. Recently, automated classification techniques using machine learning methods, such as Convolutional Neural Networks (CNNs), have demonstrated significantly higher accuracy than manual screening. However, deep learning-based image classification methods, including CNNs, face challenges in estimating class probabilities without proper model calibration (Guo et al., 2017; Minderer et al., 2021). In this paper, we propose a novel brain tumor image classification method called SIBOW-SVM, which integrates the Bag-of-Features model with SIFT feature extraction and weighted Support Vector Machines. This new approach can effectively extract hidden image features, enabling differentiation of various tumor types, provide accurate label predictions, and estimate probabilities of images belonging to each class, offering high-confidence classification decisions. We have also developed scalable and parallelable algorithms to facilitate the practical implementation of SIBOW-SVM for massive image datasets. To benchmark our method, we apply SIBOW-SVM to a public dataset of brain tumor MRI images containing four classes: glioma, meningioma, pituitary, and normal. Our results demonstrate that the new method outperforms state-of-the-art techniques, including CNNs, in terms of uncertainty quantification, classification accuracy, computational efficiency, and data robustness.

All-to-all reconfigurability with sparse and higher-order Ising machines

Domain-specific hardware to solve computationally hard optimization problems has generated tremendous excitement. Here, we evaluate probabilistic bit (p-bit) based Ising Machines (IM) on the 3-Regular 3-Exclusive OR Satisfiability (3R3X), as a representative hard optimization problem. We first introduce a multiplexed architecture that emulates all-to-all network functionality while maintaining highly parallelized chromatic Gibbs sampling. We implement this architecture in a single Field-Programmable Gate Array (FPGA) and show that running the adaptive parallel tempering algorithm demonstrates competitive algorithmic and prefactor advantages over alternative IMs by D-Wave, Toshiba, and Fujitsu. We also implement higher-order interactions that lead to better prefactors without changing algorithmic scaling for the XORSAT problem. Even though FPGA implementations of p-bits are still not quite as fast as the best possible greedy algorithms accelerated on Graphics Processing Units (GPU), scaled magnetic versions of p-bit IMs could lead to orders of magnitude improvements over the state of the art for generic optimization.

Blockchain Consensus Mechanisms: A Bibliometric Analysis (2014–2024) Using VOSviewer and R Bibliometrix

Blockchain consensus mechanisms play a critical role in ensuring the security, decentralization, and integrity of distributed networks. As blockchain technology expands beyond cryptocurrencies into broader applications such as supply chain management and healthcare, the importance of efficient and scalable consensus algorithms has grown significantly. This study provides a comprehensive bibliometric analysis of blockchain and consensus mechanism research from 2014 to 2024, using tools such as VOSviewer and R’s Bibliometrix package. The analysis traces the evolution from foundational mechanisms like Proof of ork (PoW) to more advanced models such as Proof of Stake (PoS) and Byzantine Fault Tolerance (BFT), with particular emphasis on Ethereum’s “The Merge” in 2022, which marked the historic shift from PoW to PoS. Key findings highlight emerging themes, including scalability, security, and the integration of blockchain with state-of-the-art technologies like artificial intelligence (AI), the Internet of Things (IoT), and energy trading. The study also identifies influential authors, institutions, and countries, emphasizing the collaborative and interdisciplinary nature of blockchain research. Through thematic analysis, this review uncovers the challenges and opportunities in decentralized systems, underscoring the need for continued innovation in consensus mechanisms to address efficiency, sustainability, scalability, and privacy concerns. These insights offer a valuable foundation for future research aimed at advancing blockchain technology across various industries.

Observing Quantum Measurement Collapse as a Learnability Phase Transition

During a quantum measurement, superpositions of states with different observable properties probabilistically collapse into one with a sharp value of the measured observable. In macroscopic quantum systems, this collapse arises via a continuous measurement-induced phase transition (MIPT) at a critical value of the strength of interaction with the measurement apparatus. MIPTs lie outside established paradigms for equilibrium or nonequilibrium critical phenomena and delineate distinct, stable dynamical and computational phases of matter. Quantum computers enable programmable simulation of the interaction of a measurement apparatus with a dynamical quantum system, to explore MIPT phenomena over a range of system sizes while retaining quantum coherence. Yet, existing experimental protocols rely on fundamentally nonscalable postselection techniques or direct classical simulation of quantum circuits. Here, we report the scalable observation of finite-size scaling evidence for an observable-sharpening MIPT in monitored quantum circuits in a chain of Yb+171 ions in Quantinuum’s H1-1 trapped-ion quantum processor. By leveraging an equivalent description as a statistical physics problem, we implement scalable classical algorithms to infer the value of the measured observable from a single experimental shot. This technique enables a truly scalable protocol to observe observable-sharpening MIPTs in generic classes of circuits that cannot be directly classically simulated and also provides enhanced means to detect and suppress errors in the quantum simulation. Published by the American Physical Society 2024

An Algorithm for Clustering with Confidence-Based Must-Link and Cannot-Link Constraints

We study here the semisupervised k-clustering problem where information is available on whether pairs of objects are in the same or different clusters. This information is available either with certainty or with a limited level of confidence. We introduce the pair-wise confidence constraints clustering (PCCC) algorithm, which iteratively assigns objects to clusters while accounting for the information provided on the pairs of objects. Our algorithm uses integer programming for the assignment of objects, which allows us to include relationships as hard constraints that are guaranteed to be satisfied or as soft constraints that can be violated subject to a penalty. This flexibility distinguishes our algorithm from the state of the art, in which all pair-wise constraints are considered hard or all are considered soft. We developed an enhanced multistart approach and a model-size reduction technique for the integer program that contribute to the effectiveness and efficiency of the algorithm. Unlike existing algorithms, our algorithm scales to large-scale instances with up to 60,000 objects, 100 clusters, and millions of cannot-link constraints (which are the most challenging constraints to incorporate). We compare the PCCC algorithm with state-of-the-art approaches in an extensive computational study. Even though the PCCC algorithm is more general than the state-of-the-art approaches in its applicability, it outperforms the state-of-the-art approaches on instances with all hard or all soft constraints in terms of both run time and various metrics of solution quality. The code of the PCCC algorithm is publicly available on GitHub. History: Accepted by Ram Ramesh, Area Editor for Data Science and Machine Learning. Funding: The research of D. S. Hochbaum was supported by the AI Institute NSF Award [Grant 2112533]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.0419 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2023.0419 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Physics-Based Abnormal Trajectory Gap Detection

Given trajectories with gaps (i.e., missing data), we investigate algorithms to identify abnormal gaps in trajectories which occur when a given moving object did not report its location, but other moving objects in the same geographic region periodically did. The problem is important due to its societal applications, such as improving maritime safety and regulatory enforcement for global security concerns, such as illegal fishing, illegal oil transfers, and trans-shipments. The problem is challenging due to the difficulty of bounding the possible locations of the moving object during a trajectory gap, and the very high computational cost of detecting gaps in such a large volume of location data. The current literature on anomalous trajectory detection assumes linear interpolation within gaps, which may not be able to detect abnormal gaps since objects within a given region may have traveled away from their shortest path. In preliminary work, we introduced an abnormal gap measure that uses a classical space-time prism model to bound an object's possible movement during the trajectory gap and provided a scalable memoized gap detection algorithm (Memo-AGD). In this article, we propose a space time-aware gap detection (STAGD) approach to leverage space-time indexing and merging of trajectory gaps. We also incorporate a dynamic region merge-based (DRM) approach to efficiently compute gap abnormality scores. We provide theoretical proofs that both algorithms are correct and complete and also provide analysis of asymptotic time complexity. Experimental results on synthetic and real-world maritime trajectory data show that the proposed approach substantially improves computation time over the baseline technique.

Automatically designing counterfactual regret minimization algorithms for solving imperfect-information games

Strategic decision-making in imperfect-information games is an important problem in artificial intelligence. Counterfactual regret minimization (CFR), a family of iterative algorithms, has been the workhorse for solving these types of games since its inception. In recent years, a series of novel CFR variants have been proposed, significantly improving the convergence rate of vanilla CFR. However, most of these new variants are hand-designed by researchers through trial and error, often based on different motivations, which generally requires a tremendous amount of effort and insight. This work proposes AutoCFR, a systematic framework that meta-learns novel CFR algorithms through evolution, easing the burden of manual algorithm design. We first design a search language that is rich enough to represent various CFR variants. We then exploit a scalable regularized evolution algorithm with a set of acceleration techniques to efficiently search over the combinatorial space of algorithms defined by this language. The learned novel CFR algorithm can generalize to new imperfect-information games not seen during training and performs on par with or better than existing state-of-the-art CFR variants. In addition to superior empirical performance, we also theoretically show that the learned algorithm converges to an approximate Nash equilibrium. Extensive experiments across diverse imperfect-information games highlight the scalability, extensibility, and generalizability of AutoCFR, establishing it as a general-purpose framework for solving imperfect-information games.

QCLUE: a quantum clustering algorithm for multi-dimensional datasets

Clustering algorithms are at the basis of several technological applications, and are fueling the development of rapidly evolving fields such as machine learning. In the recent past, however, it has become apparent that they face challenges stemming from datasets that span more spatial dimensions. In fact, the best-performing clustering algorithms scale linearly in the number of points, but quadratically with respect to the local density of points. In this work, we introduce qCLUE, a quantum clustering algorithm that scales linearly in both the number of points and their density. qCLUE is inspired by CLUE, an algorithm developed to address the challenging time and memory budgets of Event Reconstruction (ER) in future High-Energy Physics experiments. As such, qCLUE marries decades of development with the quadratic speedup provided by quantum computers. We numerically test qCLUE in several scenarios, demonstrating its effectiveness and proving it to be a promising route to handle complex data analysis tasks – especially in high-dimensional datasets with high densities of points.