Distributed Computing System Research Articles

On Batch-Processing Based Coded Computing for Heterogeneous Distributed Computing Systems

In recent years, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">coded distributed computing</i> (CDC) has attracted significant attention, because it can efficiently facilitate many delay-sensitive computation tasks against unexpected latencies in distributed computing systems. Despite such a salient feature, many design challenges and opportunities remain. In this paper, we focus on practical computing systems with heterogeneous computing resources, and design a novel CDC approach, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">batch-processing based coded computing</i> (BPCC), which exploits the fact that every computing node can obtain some coded results before it completes the whole task. To this end, we first describe the main idea of the BPCC framework, and then formulate an optimization problem for BPCC to minimize the task completion time by configuring the computation load. Through formal theoretical analyses, extensive simulation studies, and comprehensive real experiments on the Amazon EC2 computing clusters, we demonstrate promising performance of the proposed BPCC scheme, in terms of high computational efficiency and robustness to uncertain disturbances.

Design of a digital communication platform to food donations

In this paper, a communication strategy between the individuals involved in the donation and collection of food is presented and thus, in this way, reduce the loss or waste of the same, which is one of the causes of food insecurity in Mexico and in the world. The platform is implemented as a distributed computing system. The system is accessed through a mobile application that operates on Android. The server implements the PHP language and stores the information in a relational model using the MySQL manager. In addition, the communication between the client and the server is done through the REST protocol encapsulating the data in JSON In this system, donors can see the location of nearby altruistic organizations, consult information and contact them through social networks. (WhatsApp, Facebook, and Twitter). It also publishes donation offers for organizations to contact them and view reports of donations made. Organizations visualize the offer of users and contact them through social networks. Likewise, they must record the follow-up of each donation received in order to ensure the delivery of the food.

Alchemy: Distributed financial quantitative analysis system with high‐level programming model

AbstractNowadays, the financial securities investment and transaction are processed digitally. The securities companies have accumulated a large amount of financial data. Given that, quantitative analysis has become a common method for securities investors. However, as the scale of data continues to grow, data storing and processing have increasingly become a big challenge to financial quantitative analysts. First, most existing stand‐alone quantitative analysis systems are hard to accelerate data processing in a distributed way. Second, existing distributed computing frameworks such as Apache Spark demands financial quantitative analysts with expertise knowledge on distributed computing. Third, transition from traditional stand‐alone financial quantitative analysis (FQA) system to such distributed computing system often introduces learning efforts and development overheads. To solve these problems, we propose Alchemy, a distributed processing platform tailored for FQA with high‐level programming interface. Alchemy offers distributed computing service and provides a pipeline‐based programming model which shares many similarities with the traditional stand‐alone systems, reducing learning efforts for financial quantitative analysts. The performance of Alchemy is evaluated in both experimental environment and a real‐world production environment in Huatai Securites which is a leading financial company in China. The results show that Alchemy is able to achieve up to 300 times speedup compared against existing financial quantitative analysis systems.

Features of complementary objects usage in distributed virtual environment

. In this paper, complementary objects design and usage is considered. The presented examples illustrate the programming paradigms necessary for correct usage of complementary objects that are essential in distributed computation systems.

DQN as an alternative to Market-based approaches for Multi-Robot processing Task Allocation (MRpTA)

Multi-robot task allocation (MRTA) problems require that robots make complex choices based on their understanding of a dynamic and uncertain environment. As a distributed computing system, the Multi-Robot System (MRS) must handle and distribute processing tasks (MRpTA). Each robot must contribute to the overall efficiency of the system based solely on a limited knowledge of its environment. Market-based methods are a natural candidate to deal processing tasks over a MRS but recent and numerous developments in reinforcement learning and especially Deep Q-Networks (DQN) provide new opportunities to solve the problem. In this paper we propose a new DQN-based method so that robots can learn directly from experience, and compare it with Market-based approaches as well with centralized and purely local solutions. Our study shows the relevancy of learning-based methods and also highlight research challenges to solve the processing load-balancing problem in MRS.

Stochastic identification of the “Object-attribute” table based on the modified Rabiner’s method

This article is about solving the problem of stochastic identification of the “Object-attribute” table based on subsets of stochastic ergodic matrices. The table has N rows and m columns. The identification is based on the implementation of the modified Rabiner’s method. We assume that the elements of m columns of the table are a discrete Markov chain of length N. The identification of each column is based on calculating the maximum probability that the Markov chain is generated based on the distribution law represented by one ergodic stochastic matrix from a given subset. An algorithm for solving this problem is proposed. Estimates of the time and hardware complexity of this algorithm, which are executed in parallel on a distributed computing system, are obtained. The dependence of the obtained estimates on the number of rows and columns of the identified table is determined. The value of N has a linear effect on the time complexity of an algorithm that implements MMR and is executed in parallel. A promising direction for future research is the distributed implementation of the proposed Algorithm.

Wireless Distributed Computing System Using High-Gain Array Yagi-Antenna

Wireless Distributed Computing System Using High-Gain Array Yagi-Antenna

Googling About Blockchain

Modern technology allows people to communicate directly using voice and video calls, emails, instant messages traveling from one device to another. Communication is done by maintaining trust in each other without a third party, no matter how far away they are. But when in a communication between two points, it is needed to make a transaction, then the procedures require the trust of a third party to establish trust such as a law firm or a bank, resulting in high costs, fraud, and inefficiency. Blockchain technology is coming to change this regime radically. Blockchain using mathematics and cryptography provides an open and decentralized base of data in each transaction that contains a value such as money, goods, real estate, work, or even notes. Creates a log for each transaction which the whole community can verify. The blockchain combining cryptography and distributed computer systems provide secure, direct peer-to-peer transactions without the need for third parties.

Particle Swarm Optimization for Load Balancing in Distributed Computing Systems – A Survey

Development of technology like Cloud Computing and its widespread usage has given rise to exponential increase in the volume of traffic. With this increase in huge traffic the resources in the network would either be insufficient to handle the traffic or the situation may cause some of the resources to be over utilized or underutilized. This condition leads to reduced performance of the system. To improve the performance of the system the traffic requires to be regulated such that all the resources are utilized conferring to their capacity which is known as load balancing. Load balancing has been one of the concerns in the distributed computing systems where the computing nodes do not have a global view of the network. There have been constant efforts to provide an efficient solution for load balancing through the approaches like game theory, fuzzy logic, heuristics and metaheuristics. Even though various solutions exist for balancing the load, the issue is challenging as there does not exist one best fit solution. The paper aims at the study of how Particle Swarm Optimization approach is used to achieve an optimal solution for load balancing in distributed computing system.

METHOD OF SPECTRAL CLUSTERING OF PAYMENTS AND RAW MATERIALS SUPPLY FOR THE COMPLIANCE AUDIT PLANNING

Context. The analytical procedures used in the audit are currently based on data mining techniques. The work solves the problem of increasing the efficiency and effectiveness of analytical audit procedures by clustering based on spectral decomposition. The object of the research is the process of auditing the compliance of payment and supply sequences for raw materials.&#x0D; Objective. The aim of the work is to increase the effectiveness and efficiency of the audit due to the method of spectral clustering of sequences of payment and supply of raw materials while automating procedures for checking their compliance.&#x0D; Method. The vectors of features are generated for the objects of the sequences of payment and supply of raw materials, which are then used in the proposed method. The created method improves the traditional spectral clustering method by automatically determining the number of clusters based on the explained and sample variance rule; automatic determination of the scale parameter based on local scaling (the rule of K-nearest neighbors is used); resistance to noise and random outliers by replacing the k-means method with a modified PAM method, i.e. replacing centroid clustering with medoid clustering. As in the traditional approach, the data can be sparse, and the clusters can have different shapes and sizes. The characteristics of evaluating the quality of spectral clustering are selected.&#x0D; Results. The proposed spectral clustering method was implemented in the MATLAB package. The results obtained made it possible to study the dependence of the parameter values on the quality of clustering.&#x0D; Conclusions. The experiments carried out have confirmed the efficiency of the proposed method and allow us to recommend it for practical use in solving audit problems. Prospects for further research may lie in the creation of intelligent parallel and distributed computer systems for general and special purposes, which use the proposed method for segmentation, machine learning and pattern recognition tasks.

Improving service quality in a congested network with random breakdowns

This paper employs some strategies for improving service quality in a congested network consisting of facilities and customers. The quality is measured considering both customers’ travel distances and congestion delays. The strategies include opening new facilities, increasing the service capacities, and incorporating multiple backup services for customers, empowering the network to distribute the facilities’ workload smoothly. The goal is to optimally configure the congested network while improving the service quality. Each facility is modeled as an M/M/1 queuing system and might be broken down frequently during its serving process. The recovery starts immediately, which is a multi-step process, and each step takes an exponentially-distributed random time. The problem is first modeled as a mixed-integer nonlinear program; then, reformulated as a mixed-integer second-order cone program and can be optimally solved. An efficient solution algorithm based on Lagrangian decomposition is also developed. The numerical experiments illustrate the high efficiency of the solution methodology. Several managerial implications, including determining the best backup service level, are provided. Last, an example in distributed computing system design is presented to demonstrate the applicability of the model.

Coded Computing for Secure Boolean Computations

The growing size of modern datasets necessitates splitting a large scale computation into smaller computations and operate in a distributed manner. Adversaries in a distributed system deliberately send erroneous data in order to affect the computation for their benefit. Boolean functions are the key components of many applications, e.g., verification functions in blockchain systems and design of cryptographic algorithms. We consider the problem of computing a Boolean function in a distributed computing system with particular focus on security against Byzantine workers. Any Boolean function can be modeled as a multivariate polynomial with high degree in general. However, the security threshold (i.e., the maximum number of adversarial workers can be tolerated such that the correct results can be obtained) provided by the recent proposed Lagrange Coded Computing (LCC) can be extremely low if the degree of the polynomial is high. We propose three different schemes called coded Algebraic normal form (ANF), coded Disjunctive normal form (DNF) and coded polynomial threshold function (PTF). The key idea of the proposed schemes is to model it as the concatenation of some low-degree polynomials and threshold functions. In terms of the security threshold, we show that the proposed coded ANF and coded DNF are optimal by providing a matching outer bound.

The cooperative sorting strategy for connected and automated vehicle platoons

This paper presents a “cooperative vehicle sorting” strategy that seeks to optimally sort connected and automated vehicles (CAVs) in a multi-lane platoon to reach an ideally organized platoon. In the proposed method, a CAV platoon is firstly discretized into a grid system, where a CAV moves from one cell to another in discrete time-space domain. Then, the cooperative sorting problem is modeled as a path-finding problem in the graphic domain. The problem is solved by the deterministic A* algorithm with a stepwise strategy, where only one vehicle can move within a movement step. The resultant shortest path is further optimized with an integer linear programming algorithm to minimize the sorting time by allowing multiple movements within a step. To improve the algorithm running time and address multiple shortest paths, a distributed stochastic A* algorithm (DSA*) is developed by introducing random disturbances to the edge costs to break uniform paths (with equal path cost). Numerical experiments are conducted to demonstrate the effectiveness of the proposed DSA* method. The results report shorter sorting time and significantly improved algorithm running time due to the use of DSA*. In addition, we find that the optimization performance can be further improved by increasing the number of processes in the distributed computing system.

Energy-minimized Scheduling of Real-time Parallel Workflows on Heterogeneous Distributed Computing Systems

Today's large-scale parallel workflows are often processed on heterogeneous distributed computing platforms. From an economic perspective, computing resource providers should minimize the cost while offering high service quality. It has become well-recognized that energy consumption accounts for a large part of a computing system's total cost, and timeliness and reliability are two important service indicators. This work studies the problem of scheduling a parallel workflow that minimizes the system energy consumption under the constraints of response time and reliability. We first mathematically formulate this problem as a Non-linear Mixed Integer Programming problem. Since this problem is hard to solve directly, we present some highly-efficient heuristic solutions. Specifically, we first develop an algorithm that minimizes the schedule length while meeting reliability requirement, on top of which we propose a processor-merging algorithm and a slack time reclamation algorithm using a dynamic voltage frequency scaling (DVFS) technique to reduce energy consumption. The processor-merging algorithm tries to turn off some energy-inefficient processors such that energy consumption can be minimized. The DVFS technique is applied to scale down the processor frequency at both processor and task levels to reduce energy consumption. Experimental results on two real-life workflows and extensive synthetic parallel workflows demonstrate their effectiveness.

Архітектурні особливості систем розподілених обчислень

Recently, there has been an increasing penetration of information technology in almost all areas of human life. The development of information technology is associated with the emergence of new tasks that require significant computing resources and can not be solved on a conventional computer. A large amount of computing requires the creation of so-called supercomputers, which is not always technically possible. But there is another way to solve this problem, when a complex task is divided into a number of subtasks that run in parallel. And here come in handy distributed computing system. In general, a distributed computing system is a virtual machine that consists of several nodes connected by a network. That is, a certain three-dimensional problem is divided into several simple subtasks and connections are established between them. But such a system will be operational only when the tasks between the nodes are distributed correctly, and the sequence of their execution will take place according to a given algorithm. The article analyzes the architectural features of distributed computing systems. The main task of distributed computing technologies is to provide access to globally distributed resources and solve problems that require significant computing power and can not be implemented on a conventional computer. The complexity of global tasks is due to the fact that the necessary data can be accessed on different computers. In addition, distributed computing systems, which are formed from autonomous resources, can change their architecture dynamically. Management of such distributed computer systems requires the search for new computational models and the search for architectural solutions to build new systems that would meet the current level of development of information technology.

Modeling and Analysis of Relation between Load Balancing and Scalability in Distributed Computing Systems

Modeling and Analysis of Relation between Load Balancing and Scalability in Distributed Computing Systems

Containerization in ATLAS Software Development and Data Production

The ATLAS experiment’s software production and distribution on the grid benefits from a semi-automated infrastructure that provides up-to-date information about software usability and availability through the CVMFS distribution service for all relevant systems. The software development process uses a Continuous Integration pipeline involving testing, validation, packaging and installation steps. For opportunistic sites that can not access CVMFS, containerized releases are needed. These standalone containers are currently created manually to support Monte-Carlo data production at such sites. In this paper we will describe an automated procedure for the containerization of ATLAS software releases in the existing software development infrastructure, its motivation, integration and testing in the distributed computing system.

Building a Distributed Computing System for LDMX

Particle physics experiments rely extensively on computing and data services, making e-infrastructure an integral part of the research collaboration. Constructing and operating distributed computing can however be challenging for a smaller-scale collaboration. The Light Dark Matter eXperiment (LDMX) is a planned small-scale accelerator-based experiment to search for dark matter in the sub-GeV mass region. Finalizing the design of the detector relies on Monte-Carlo simulation of expected physics processes. A distributed computing pilot project was proposed to better utilize available resources at the collaborating institutes, and to improve scalability and reproducibility. This paper outlines the chosen lightweight distributed solution, presenting requirements, the component integration steps, and the experiences using a pilot system for tests with large-scale simulations. The system leverages existing technologies wherever possible, minimizing the need for software development, and deploys only non-intrusive components at the participating sites. The pilot proved that integrating existing components can dramatically reduce the effort needed to build and operate a distributed e-infrastructure, making it attainable even for smaller research collaborations.

基于Ray的服务于自动驾驶的远程分布式计算系统

基于Ray的服务于自动驾驶的远程分布式计算系统

Learning-Driven Interference-Aware Workload Parallelization for Streaming Applications in Heterogeneous Cluster

In the past few years, with the rapid development of CPU-GPU heterogeneous computing, the issue of task scheduling in the heterogeneous cluster has attracted a great deal of attention. This problem becomes more challenging with the need for efficient co-execution of tasks on the GPUs. However, the uncertainty of heterogeneous cluster and the interference caused by resource contention among co-executing tasks can lead to the unbalanced use of computing resource and further cause the degradation in performance of computing platform. In this article, we propose a two-stage task scheduling approach for streaming applications based on deep reinforcement learning and neural collaborative filtering, which considers fine-grained task division and task interference on the GPU. Specifically, the Learning-Driven Workload Parallelization (LDWP) method selects an appropriate execution node for the mutually independent tasks. By using the deep Q-network, the cluster-level scheduling model is online learned to perform the current optimal scheduling actions according to the runtime status of cluster environments and characteristics of tasks. The Interference-Aware Workload Parallelization (IAWP) method assigns subtasks with dependencies to the appropriate computing units, taking into account the interference of subtasks on the GPU by using neural collaborative filtering. For making the learning of neural network more efficient, we use pre-training in the two-stage scheduler. Besides, we use transfer learning technology to efficiently rebuild task scheduling model referring to the existing model. We evaluate our learning-driven and interference-aware task scheduling approach on a prototype platform with other widely used methods. The experimental results show that the proposed strategy can averagely improve the throughout for distributed computing system by 26.9 percent and improve the GPU resource utilization by around 14.7 percent.