Scalable Algorithm Research Articles

Scalable implicit solvers with dynamic mesh adaptation for a relativistic drift-kinetic Fokker–Planck–Boltzmann model

In this work we consider a relativistic drift-kinetic model for runaway electrons along with a Fokker–Planck operator for small-angle Coulomb collisions, a radiation damping operator, and a secondary knock-on (Boltzmann) collision source. We develop a new scalable fully implicit solver utilizing finite volume and conservative finite difference schemes and dynamic mesh adaptivity. A new data management framework in the PETSc library based on the p4est library is developed to enable simulations with dynamic adaptive mesh refinement (AMR), distributed memory parallelization, and dynamic load balancing of computational work. This framework and the runaway electron solver building on the framework are able to dynamically capture both bulk Maxwellian at the low-energy region and a runaway tail at the high-energy region. To effectively capture features via the AMR algorithm, a new AMR indicator prediction strategy is proposed that is performed alongside the implicit time evolution of the solution. This strategy is complemented by the introduction of computationally cheap feature-based AMR indicators that are analyzed theoretically. Numerical results quantify the advantages of the prediction strategy in better capturing features compared with nonpredictive strategies; and we demonstrate trade-offs regarding computational costs. The robustness with respect to model parameters, algorithmic scalability, and parallel scalability are demonstrated through several benchmark problems including manufactured solutions and solutions of different physics models. We focus on demonstrating the advantages of using implicit time stepping and AMR for runaway electron simulations.

ResORR: A globally scalable and satellite data-driven algorithm for river flow regulation due to reservoir operations

We propose a globally scalable algorithm, ResORR (Reservoir Operations driven River Regulation), to predict regulated river flow and tested it over the heavily regulated basin of the Cumberland River in the US. ResORR was found able to model regulated river flow due to upstream reservoir operations of the Cumberland River. Over a mountainous basin dominated by high rainfall, ResORR was effective in capturing extreme flooding modified by upstream hydropower dam operations. On average, ResORR improved regulated river flow simulation by more than 50% across all performance metrics when compared to a hydrologic model without a regulation module. ResORR is a timely software algorithm for understanding human regulation of surface water as satellite-estimated reservoir state is expected to improve globally with the recently launched Surface Water and Ocean Topography (SWOT) mission.

Downlink Learning-aided Precoding for Cell-Free networks Specially D esigned for beyond 5G Communication System

Objectives: With increase in number of access points in Cell-Free Massive Multiple Input Multiple Output (CFMM) system, spatial complexity of calculating precoding vector increases with traditional precoding schemes. So, to reduce computational complexity, network cost, and run time, a new deep learning -aided precoder is specially designed for CFMM system. The performance of downlink (DL) Cell-Free Massive Multiple Input Multiple Output (CFMM) system operating under Rayleigh fading channel model is analyze using new deep learning-based precoding scheme. Methods: We introduce new deep learning-aided precoder and an improved version of basic scalable pilot assignment algorithm which enhances system performance. We derive closed- form expression for average DL spectral efficiency (SE) for the proposed scheme, which is then compared with Minimum Mean Square Error Successive Interference Cancellation (MMSESIC), Regularized Zero Forcing (RZF), Least Square (LS) and Maximum Ratio (MR) combining techniques. We analyze the proposed scheme with perfect channel state information (CSI), instantaneous CSI, coherent transmission, non-coherent transmission, different pilot configuration. Findings: The spectral efficiency obtained with CFMM-MMSESIC and the proposed scheme is 3 bits/s/Hz, and 2.5bits/s/Hz respectively, which clearly indicates the difference of 20 % only. So, the proposed scheme gives near optimal results, and we can use simple linear combining techniques instead of complex non-linear combining techniques. Also, the proposed scheme with coherent data transmission gives performance curve, which is very close to MMSESIC scheme as desired. The performance of the proposed system is improved by reducing run time and computational complexity as compared to conventional linear precoding schemes. Novelty: The proposed precoder improves performance by reducing run time and computational complexity. Advanced pilot assignment algorithm enhances performance by reducing pilot contamination. Keywords: Cell-Free Massive Multiple Input Multiple Output, Pilot Contamination, Precoding, Minimum Mean Square Error Successive Interference Cancellation, Maximum Ratio, Regularized Zero Forcing (RZF), Least Square (LS)

Limited Query Graph Connectivity Test

We propose a combinatorial optimisation model called Limited Query Graph Connectivity Test. We consider a graph whose edges have two possible states (On/Off). The edges' states are hidden initially. We could query an edge to reveal its state. Given a source s and a destination t, we aim to test s−t connectivity by identifying either a path (consisting of only On edges) or a cut (consisting of only Off edges). We are limited to B queries, after which we stop regardless of whether graph connectivity is established. We aim to design a query policy that minimizes the expected number of queries. Our model is mainly motivated by a cyber security use case where we need to establish whether attack paths exist in a given network, between a source (i.e., a compromised user node) and a destination (i.e., a high-privilege admin node). Edge query is resolved by manual effort from the IT admin, which is the motivation behind query minimization. Our model is highly related to Stochastic Boolean Function Evaluation (SBFE). There are two existing exact algorithms for SBFE that are prohibitively expensive. We propose a signifcantly more scalable exact algorithm. While previous exact algorithms only scale for trivial graphs (i.e., past works experimented on at most 20 edges), we empirically demonstrate that our algorithm is scalable for a wide range of much larger practical graphs (i.e., graphs representing Windows domain networks with tens of thousands of edges). We also propose three heuristics. Our best-performing heuristic is via limiting the planning horizon of the exact algorithm. The other two are via reinforcement learning (RL) and Monte Carlo tree search (MCTS). We also derive an algorithm for computing the performance lower bound. Experimentally, we show that all our heuristics are near optimal. The heuristic building on the exact algorithm outperforms all other heuristics, surpassing RL, MCTS and eight existing heuristics ported from SBFE and related literature.

Optimal Makespan in a Minute Timespan! A Scalable Multi-Robot Goal Assignment Algorithm for Minimizing Mission Time

We study a variant of the multi-robot goal assignment problem where a unique goal to each robot needs to be assigned while minimizing the largest cost of movement among the robots, called makespan. A significant step in solving this problem is to find the cost associated with the robot-goal pairs, which requires solving a complex path planning problem. We present OM, a scalable optimal algorithm that solves the multi-robot goal assignment problem by computing the paths for a significantly less number of robot-goal pairs compared to the state-of-the-art algorithms, leading to a computationally superior mechanism to solve the problem. We extensively evaluate our algorithm for hundreds of robots on randomly generated and standard workspaces. Our experimental results demonstrate that the proposed algorithm achieves a noticeable speedup over two state-of-the-art baseline algorithms.

An Improved NLCS Algorithm Based on Series Reversion and Elliptical Model Using Geosynchronous Spaceborne–Airborne UHF UWB Bistatic SAR for Oceanic Scene Imaging

Geosynchronous spaceborne–airborne (GEO-SA) ultra-high-frequency ultra-wideband bistatic synthetic aperture radar (UHF UWB BiSAR) provides high-precision images for marine and polar environments, which are pivotal in glacier monitoring and sea ice thickness measurement for polar ocean mapping and navigation. Contrasting with traditional high-frequency BiSAR, it faces unique challenges, such as the considerable spatial variability, significant range–azimuth coupling, and vast volumes of echo data, which impede high-resolution image reconstruction. This paper presents an improved bistatic nonlinear chirp scaling (NLCS) algorithm for imaging oceanic scenes with GEO-SA UHF UWB BiSAR. This methodology extends the two-dimensional (2-D) spectrum up to the sixth order via the method of series reversion (MSR) to meet accuracy demands and then employs an elliptical model to elucidate the alterations in the azimuth frequency modulation (FM) rate mismatch. Initially, the imaging geometry and signal model are introduced, and then a separation of bistatic slant ranges based on the configuration is proposed. In addition, during range processing, after eliminating linear range cell migration (RCM), the derivation process for the sixth-order 2-D spectrum is detailed and an improved filter is applied to correct the high-order RCM. Finally, during azimuth processing, the causes of the FM rate mismatch are analyzed, a cubic perturbation function derived from the elliptical model is used for FM rate equalization, and a unified sixth-order filter is applied to complete the azimuth compression. Experimental results with point targets and natural oceanic scenes validate the outstanding efficacy of the proposed NLCS algorithm, particularly in imaging quality enhancements for GEO-SA UHF UWB BiSAR.

Learning Sliding Policy of Flat Multi-target Objects in Clutter Scenes

In clutter scenes, one or several targets need to be obtained, which is hard for robot manipulation task. Especially, when the targets are flat objects like book, plates, due to limitation of common robot end-effectors, it will be more challenging. By employing pre-grasp operation like sliding, it becomes feasible to rearrange objects and shift the target towards table edge, enabling the robot to grasp it from a lateral perspective. In this paper, the proposed method transfers the task into a Parameterized Action Markov Decision Process to solve the problem, which is based on deep reinforcement learning. The mask images are taken as one of observations to the network for avoiding the impact of noise of original image. In order to improve data utilization, the policynetwork predicts the parameters for the sliding primitive of each object, which is weight-sharing, and then the Q-network selects the optimal execution target. Meanwhile, extra reward mechanism is adopted for improving the efficiency of task actions to cope with multiple targets. In addition, an adaptive policy scaling algorithm is proposed to improve the speed and adaptability of policy training. In both simulation and real system, our method achieves a higher task success rate and requires fewer actions to accomplish the flat multi-target sliding manipulation task within clutter scene, which verifies the effectiveness of ours.

Assessing Routing Algorithms for Payment Channel Networks

Payment Channel Networks (PCNs) are a promising approach to overcome scalability issues of blockchains. To achieve efficient payments in PCNs, it is necessary to route transactions between a payer and a payee. Especially in large-scale PCNs, multi-hop routing becomes necessary, since transactions need to be relayed by nodes. For this, a scalable routing algorithm is needed, which fits the individual objectives of PCN users. In this article, we study whether routing protocols from the field of Wireless Sensor Networks can be applied in PCNs. To this end, we first derive requirements for routing in PCNs, select suitable approaches, and analyze to which degrees they perform well and meet the requirements. We adapt selected protocols and evaluate them with regard to the lengths of payment paths, fees, and success ratio.

Parallel Algorithms for Hierarchical Nucleus Decomposition

Nucleus decompositions have been shown to be a useful tool for finding dense subgraphs. The coreness value of a clique represents its density based on the number of other cliques it is adjacent to. One useful output of nucleus decomposition is to generate a hierarchy among dense subgraphs at different resolutions. However, existing parallel algorithms for nucleus decomposition do not generate this hierarchy, and only compute the coreness values. This paper presents a scalable parallel algorithm for hierarchy construction, with practical optimizations, such as interleaving the coreness computation with hierarchy construction and using a concurrent union-find data structure in an innovative way to generate the hierarchy. We also introduce a parallel approximation algorithm for nucleus decomposition, which achieves much lower span in theory and better performance in practice. We prove strong theoretical bounds on the work and span (parallel time) of our algorithms. On a 30-core machine with two-way hyper-threading, our parallel hierarchy construction algorithm achieves up to a 58.84x speedup over the state-of-the-art sequential hierarchy construction algorithm by Sariyuce et al. and up to a 30.96x self-relative parallel speedup. On the same machine, our approximation algorithm achieves a 3.3x speedup over our exact algorithm, while generating coreness estimates with a multiplicative error of 1.33x on average.

Bayesian variable selection for matrix autoregressive models

A Bayesian method is proposed for variable selection in high-dimensional matrix autoregressive models which reflects and exploits the original matrix structure of data to (a) reduce dimensionality and (b) foster interpretability of multidimensional relationship structures. A compact form of the model is derived which facilitates the estimation procedure and two computational methods for the estimation are proposed: a Markov chain Monte Carlo algorithm and a scalable Bayesian EM algorithm. Being based on the spike-and-slab framework for fast posterior mode identification, the latter enables Bayesian data analysis of matrix-valued time series at large scales. The theoretical properties, comparative performance, and computational efficiency of the proposed model is investigated through simulated examples and an application to a panel of country economic indicators.

Correlation-informed ordered dictionary learning for imaging in complex media

We propose a method for imaging in scattering media when large and diverse datasets are available. It has two steps. Using a dictionary learning algorithm the first step estimates the true Green's function vectors as columns in an unordered sensing matrix. The array data comes from many sparse sets of sources whose location and strength are not known to us. In the second step, the columns of the estimated sensing matrix are ordered for imaging using the multidimensional scaling algorithm with connectivity information derived from cross-correlations of its columns, as in time reversal. For these two steps to work together, we need data from large arrays of receivers so the columns of the sensing matrix are incoherent for the first step, as well as from sub-arrays so that they are coherent enough to obtain connectivity needed in the second step. Through simulation experiments, we show that the proposed method is able to provide images in complex media whose resolution is that of a homogeneous medium.

Cluster modeling of the interaction of statyonary SH-waves with a system of curvilinear cracks in a half-space

A method is proposed for solving problems of mathematical physics for semi-infinite media containing systems of curvilinear cuts. Based on singular integral equations (SIE), a unified approach to solving the problem has been developed. Numerical implementation was carried out using parallelisation. A graph of the duration of the initial cluster hour is given for calculating an array of searched functions in the context of a parabolic form as a function of the number of processes for one variant of implementation. It is shown that the entire algorithm scales well and has an efficient number of processes. The expected result was obtained, because the structure of the calculation process in models based on SIE is mostly unchanged and built on well-defined procedures. It turned out that 150–200 processes are effective. An accuracy of 10–12 was achieved with the number of collocation points of the contour of each section N = 300, because the algorithm for numerical solution of the problem uses interpolation by Chebyshev polynomials in accordance with the fact that the unknown function has a key feature at the ends of the section, which causes a higher speed of convergence of the algorithm. The study of the question of the further increase in the accuracy of the result was not conducted. The corresponding dynamic boundary value problems for a restrained and force-free half-plane are studied. The influence of the curvature of defects, their interaction and proximity of the boundary on the magnitude and nature of the change in the dynamic stress intensity coefficients was studied. To check the reliability of the algorithm, two tests were carried out: removing three inhomogeneities at a distance of 106 of their length from each other and saturating the system by increasing the number of geometrically equal reflectors to 13–15. When removed, the characteristics of each reflector tend to the characteristics of a single one, and when saturated, they tend to the results of the corresponding periodic problem. The agreement between the results showed good reliability of the algorithm. The proposed method can be used to assess the influence of various mechanical or geometric factors on the strength of bodies with defects.

EVENT DRIVEN KUBERNETES AUTOSCALER FOR CLOUD NATIVE APPS

Kubernetes and Docker have emerged to be a key solution for managing and organizing containers in microservice applications. With autoscaling, one of Kubernetes key features that dynamically modifies resources like CPU and memory to fit changing demands, it gives us a strong and efficient way to manage our application. Yet, the conventional autoscaling methods, such as the Horizontal Pod Autoscaler (HPA), often have difficulties on customary metrics, like CPU and memory utilization, which may not be fully suitable with the constant changes in the modern applications. This project aims at solving this problem by suggesting a unique solution which is Event Driven Kubernetes Autoscaler for Cloud Native Apps that is specifically made to work with the scaling needs of the application based on the message queue traffic. The custom scaling algorithm is an important part of the project, which unlike the previous traditional autoscalers who needed predefined metrics and did not provide much flexibility, the custom Metric based Autoscaler allows developers to select their own metrics and scaling parameters, thus providing flexibility and great user experience as they can meet the performance needs of all the applications. Keywords: Kubernetes, Docker, Microservices, Autoscaling, Cloud-Native, Horizontal Pod Autoscaler (HPA)

Transfer Learning-Based B-Line Assessment of Lung Ultrasound for Acute Heart Failure

ObjectiveB-lines assessed by lung ultrasound (LUS) outperform physical exam, chest radiograph, and biomarkers for the associated diagnosis of acute heart failure (AHF) in the emergent setting. The use of LUS is however limited to trained professionals and suffers from interpretation variability. The objective was to utilize transfer learning to create an AI-enabled software that can aid novice users to automate LUS B-line interpretation. MethodsData from an observational AHF LUS study provided standardized cine clips for AI model development and evaluation. A total of 49,952 LUS frames from 30 patients were hand scored and trained on a convolutional neural network (CNN) to interpret B-lines at the frame level. A random independent evaluation set of 476 LUS clips from 60 unique patients assessed model performance. The AI models scored the clips on both a binary and ordinal 0–4 multiclass assessment. ResultsA multiclassification AI algorithm had the best performance at the binary level when applied to the independent evaluation set, AUC of 0.967 (95% CI 0.965–0.970) for detecting pathologic conditions. When compared to expert blinded reviewer, the 0–4 multiclassification AI algorithm scale had a reported linear weighted kappa of 0.839 (95% CI 0.804–0.871). ConclusionsThe multiclassification AI algorithm is a robust and well performing model at both binary and ordinal multiclass B-line evaluation. This algorithm has the potential to be integrated into clinical workflows to assist users with quantitative and objective B-line assessment for evaluation of AHF.

Tree-informed Bayesian multi-source domain adaptation: cross-population probabilistic cause-of-death assignment using verbal autopsy.

Determining causes of deaths (CODs) occurred outside of civil registration and vital statistics systems is challenging. A technique called verbal autopsy (VA) is widely adopted to gather information on deaths in practice. A VA consists of interviewing relatives of a deceased person about symptoms of the deceased in the period leading to the death, often resulting in multivariate binary responses. While statistical methods have been devised for estimating the cause-specific mortality fractions (CSMFs) for a study population, continued expansion of VA to new populations (or "domains") necessitates approaches that recognize between-domain differences while capitalizing on potential similarities. In this article, we propose such a domain-adaptive method that integrates external between-domain similarity information encoded by a prespecified rooted weighted tree. Given a cause, we use latent class models to characterize the conditional distributions of the responses that may vary by domain. We specify a logistic stick-breaking Gaussian diffusion process prior along the tree for class mixing weights with node-specific spike-and-slab priors to pool information between the domains in a data-driven way. The posterior inference is conducted via a scalable variational Bayes algorithm. Simulation studies show that the domain adaptation enabled by the proposed method improves CSMF estimation and individual COD assignment. We also illustrate and evaluate the method using a validation dataset. The article concludes with a discussion of limitations and future directions.

Time and memory scalable algorithms for clustering tendency assessment of big data

Large-volume and high-dimensional big datasets are being generated quickly. They are expected to provide data-driven solutions for various pressing challenges such as cybersecurity, credit card fraud, network intrusion detection, etc. The visual assessment of clustering tendency (VAT) family of algorithms provides an excellent tool for assessing clustering tendency and subsequent clustering of these novel datasets to extract groups and anomalies to better understand the data generation phenomenon. However, VAT and improved VAT (iVAT) algorithms are plagued with O(n2) time and space complexity as they use Prim's algorithm for building Euclidean minimum spanning tree (EMST) of the data points, leaving it inapplicable to large datasets. This paper develops three novel time- and memory-scalable algorithms for fast computation of VAT EMST by reducing the search space of the following possible EMST edge and using an efficient data structure, the k-d tree. Next, we develop a novel approach to compute iVAT reordered dissimilarity image (RDI) in a novel memory-preserving manner using the EMST edge lengths calculated previously without having to compute memory intensive n×n VAT RDI. We experimented on several synthetic and real-life datasets to showcase that the proposed approaches can help quickly assess their clustering tendency in a memory-saving manner. We also demonstrate the applicability of the proposed methods for the anomaly detection task in large volumes of high-dimensional data.

A Lightweight, Effective, and Efficient Model for Label Aggregation in Crowdsourcing

Due to the presence of noise in crowdsourced labels, label aggregation (LA) has become a standard procedure for post-processing these labels. LA methods estimate true labels from crowdsourced labels by modeling worker quality. However, most existing LA methods are iterative in nature. They require multiple passes through all crowdsourced labels, jointly and iteratively updating true labels and worker qualities until a termination condition is met. As a result, these methods are burdened with high space and time complexities, which restrict their applicability in scenarios where scalability and online aggregation are essential. Furthermore, defining a suitable termination condition for iterative algorithms can be challenging. In this article, we view LA as a dynamic system and represent it as a Dynamic Bayesian Network. From this dynamic model, we derive two lightweight and scalable algorithms: LA onepass and LA twopass . These algorithms can efficiently and effectively estimate worker qualities and true labels by traversing all labels at most twice, thereby eliminating the need for explicit termination conditions and multiple traversals over the crowdsourced labels. Due to their dynamic nature, the proposed algorithms are also capable of performing label aggregation online. We provide theoretical proof of the convergence property of the proposed algorithms and bound the error of the estimated worker qualities. Furthermore, we analyze the space and time complexities of our proposed algorithms, demonstrating their equivalence to those of majority voting. Through experiments conducted on 20 real-world datasets, we demonstrate that our proposed algorithms can effectively and efficiently aggregate labels in both offline and online settings, even though they traverse all labels at most twice. The code is on https://github.com/yyang318/LA_onepass .

Analyzing E-commerce Trends through Big Data

Big data is a new driver of the world economic and societal changes. The world’s data collection is reaching a tipping point for major technological changes that can bring new ways in decision making, managing our health, cities, finance and education. While the data complexities are increasing including data’s volume, variety, velocity and veracity, the real impact hinges on our ability to uncover the `value’ in the data through Big Data Analytics technologies. Big Data Analytics poses a grand challenge on the design of highly scalable algorithms and systems to integrate the data and uncover large hidden values from datasets that are diverse, complex, and of a massive scale.

A parallel Breadth-First Search using shared memory level-synchronization

Breadth-first search (BFS) stands as a cornerstone in graph exploration techniques, enabling systematic traversal of a provided graph. As the digital era continues to burgeon, there has been a marked upswing in the need to process vast graph-based data sets. To harness the power of such data effectively, it becomes imperative to use computational tools efficiently. Parallelizing BFS emerges as a pivotal strategy in this regard, leveraging the expansive capabilities of multiprocessor systems to maximize efficiency. This manuscript introduces a level-synchronous parallel BFS that is predicated on the shared-memory model. Recognizing the potential pitfalls of such an approach, especially regarding overhead induced by implicit barriers and critical sections, meticulous optimization techniques are infused into the model. These strategies are not mere afterthoughts; they are woven into the fabric of the design, ensuring smooth operations even when scaled. To validate the efficacy of this model, a rigorous assessment is carried out using the Graph500 benchmark. This offers insights into the performance scale of the parallel BFS algorithm, especially focusing on its speedup in relation to the number of operational threads. Concluding this exploration, the paper delineates prospective avenues for refining and further enhancing the proposed parallel implementation, aiming for even greater efficiencies in future endeavors.

Causality Assessment Between Drugs and Fatal Cerebral Haemorrhage Using Electronic Medical Records: Comparative Evaluation of Disease-Specific and Conventional Methods.

A new algorithm for causality assessment of drugs and fatal cerebral haemorrhage (ACAD-FCH) was published in 2021. However, its use in clinical practice has not been verified. This study aimed to explore the practical value of the ACAD-FCH when applying information available in clinical practice. The medical records of patients who died at the University of Tokyo Hospital in 2020 were reviewed, and cases with intracranial haemorrhage were selected. Two evaluators independently assessed these cases using three methods (the ACAD-FCH, Naranjo algorithm, and WHO-UMC scale). The number of 'Yes', 'No', and 'No information/Do not know' responses to each question by both evaluators were summed and compared. Inter-rater reliability was evaluated for each method using agreement rates and kappa coefficients with 95% confidence intervals (CI). Among 316 deaths, 24 cases with intracranial haemorrhage were evaluated. The proportion of ‛No information/Do not know' responses for each question was 35.6% (95% CI 31.4-40.6%) for the ACAD-FCH and 66.9% (95% CI 62.5-71.1%) for the Naranjo algorithm. The respective agreement rates and kappa coefficients were 0.917 (0.798-1.00) and 0.867 (0.675-1.00) for the ACAD-FCH, 0.708 (0.512-0.904) and 0.139 (-0.236 to 0.513) for the Naranjo algorithm, and 0.50 (0.284-0.716) and 0.326 (0.110-0.541) for the WHO-UMC scale, respectively. Our findings suggest the utility of the ACAD-FCH when assessing death cases with intracranial haemorrhage. However, larger studies including intra-rater assessments are warranted for further validation of this algorithm.