Large-scale Distributed Systems Research Articles

Augmented sensing-based on-orbit configuration estimation for large-scale distributed spacecraft systems

On-orbit configuration estimation is one of the most important tasks for large-scale distributed spacecraft systems, yet incomplete measurement topologies limit the acquisition of partial inter-satellite relative navigation information. In this article, an on-orbit configuration estimation algorithm based on augmented sensing is proposed, which enables each spacecraft to extend the estimation range beyond the observation range without any communication overhead. Referring to the Bayesian inference process, the cooperative behavior of the large-scale distributed spacecraft system is applied to infer relative motion states of unobservable spacecrafts. Under the framework of scaled unscented transformations (SUT), we achieve high-accuracy configuration estimation for the large-scale distributed spacecraft system. Finally, the proposed algorithm is verified in a high-orbiting gravitational wave detector constellation with inter-satellite laser interferometers.

A simple and efficient Distributed Trigger Counting algorithm based on local thresholds

Consider a large-scale distributed system in which each computing device is observing triggers from an external source. Distributed Trigger Counting (DTC) algorithm is used to detect the state of the system when the aggregated number of the observed triggers reaches a predefined value. In this paper, we propose a simple and efficient DTC algorithm: Cascading Thresholds (CT). We mathematically show that CT is an optimal DTC algorithm in terms of the total number of exchanged messages among the devices (message complexity). For the maximum number of received messages per device (MaxRcv), CT is sub-optimal. The average message complexity of CT is O(Nlog(W/N)), and MaxRcv of it is O(klog(W/N)+N), where W is the number of triggers to be detected, N is the number of devices, and k is the degree of a node in the tree-like structure. Compared to the previous optimal algorithm (TreeFill), CT is much simpler: in our implementation the code size is about 2.5 times smaller. Also, unlike TreeFill CT does not require complicated mechanisms including distributed locking. Experimental results show that CT has a lower message complexity and MaxRcv compared to the previous work (CoinRand and RingRand). Furthermore, CT and TreeFill show a similar performance. From its simplicity, CT is more practical than previous work including TreeFill, CoinRand and RingRand.

Near-Optimal Resilient Aggregation Rules for Distributed Learning Using 1-Center and 1-Mean Clustering with Outliers

Byzantine machine learning has garnered considerable attention in light of the unpredictable faults that can occur in large-scale distributed learning systems. The key to secure resilience against Byzantine machines in distributed learning is resilient aggregation mechanisms. Although abundant resilient aggregation rules have been proposed, they are designed in ad-hoc manners, imposing extra barriers on comparing, analyzing, and improving the rules across performance criteria. This paper studies near-optimal aggregation rules using clustering in the presence of outliers. Our outlier-robust clustering approach utilizes geometric properties of the update vectors provided by workers. Our analysis show that constant approximations to the 1-center and 1-mean clustering problems with outliers provide near-optimal resilient aggregators for metric-based criteria, which have been proven to be crucial in the homogeneous and heterogeneous cases respectively. In addition, we discuss two contradicting types of attacks under which no single aggregation rule is guaranteed to improve upon the naive average. Based on the discussion, we propose a two-phase resilient aggregation framework. We run experiments for image classification using a non-convex loss function. The proposed algorithms outperform previously known aggregation rules by a large margin with both homogeneous and heterogeneous data distributions among non-faulty workers. Code and appendix are available at https://github.com/jerry907/AAAI24-RASHB.

Self-Organizing Democratized Learning: Toward Large-Scale Distributed Learning Systems.

Emerging cross-device artificial intelligence (AI) applications require a transition from conventional centralized learning systems toward large-scale distributed AI systems that can collaboratively perform complex learning tasks. In this regard, democratized learning (Dem-AI) lays out a holistic philosophy with underlying principles for building large-scale distributed and democratized machine learning systems. The outlined principles are meant to study a generalization in distributed learning systems that go beyond existing mechanisms such as federated learning (FL). Moreover, such learning systems rely on hierarchical self-organization of well-connected distributed learning agents who have limited and highly personalized data and can evolve and regulate themselves based on the underlying duality of specialized and generalized processes. Inspired by Dem-AI philosophy, a novel distributed learning approach is proposed in this article. The approach consists of a self-organizing hierarchical structuring mechanism based on agglomerative clustering, hierarchical generalization, and corresponding learning mechanism. Subsequently, hierarchical generalized learning problems in recursive forms are formulated and shown to be approximately solved using the solutions of distributed personalized learning problems and hierarchical update mechanisms. To that end, a distributed learning algorithm, namely DemLearn, is proposed. Extensive experiments on benchmark MNIST, Fashion-MNIST, FE-MNIST, and CIFAR-10 datasets show that the proposed algorithm demonstrates better results in the generalization performance of learning models in agents compared to the conventional FL algorithms. The detailed analysis provides useful observations to further handle both the generalization and specialization performance of the learning models in Dem-AI systems.

Continuous agile cyber–physical systems architectures based on digital twins

Modern cyber-physical systems, for the most part, are large-scale multilevel heterogeneous distributed systems that integrate subsystems of different kinds and are built on the Internet of Things platforms, where system structure and behavior are not constant. Managing such systems and keeping them in working condition throughout their lifetime is a difficult task. The proposed article discusses one of the possible approaches to solving this problem, based on the use of well-known continuous and agile architecture paradigms. However, there are currently no effective mechanisms for implementing these paradigms. The proposed article suggests a new approach to implementing continuous agile architectures by utilizing digital twins and proposes a reference architecture for a run-time dynamic digital twin. This method is unique because it builds a series of dynamic digital twins that model the system in real time, utilizing data about system events. Build the first models using the models used in earlier stages of the system lifecycle. This gives the following opportunities: i) a way to use dynamic digital twins to implement the continuous agile architecture paradigm; ii) a generalized three-level model of the life cycle of the continuous agile architecture; iii) a reference architecture for dynamic digital twins; and iv) a set of models that are all about using dynamic digital twins. The suggested approach enables the management of heterogeneous multilevel cyber-physical systems with variable structure and behavior variability.

Методика формирования комплексной системы синхронизации и доставки шкалы времени для крупной распределенной системы технологического назначения

Purpose: To develop scientific and technical solutions for constructing a comprehensive synchronization system and delivery of the time scale, which includes subsystems of network clock synchronization (NCS) and unified time (UT), as a unified timekeeping system addressing the challenges of frequency, time, and phase synchronization in large-scale distributed technological purpose systems. Methods: Research methods are based on the fundamental provisions of the theory of systems and the theory of decision-making. Results: A methodology for the formation of a comprehensive synchronization system and the delivery of a time scale is proposed, which is characterized by an integrated approach to building unified time and frequency systems in the communication networks of the technological segment, the independence of the types of generator equipment used and the means of delivery of NCS and UT signals, as well as meeting the requirements of potential consumers of frequency and time services. The proposed methodology for the formation of a comprehensive synchronization system and delivery of a time scale will allow to reasonably approach building systems for delivering unified time and frequency signals in complex heterogeneous communication networks. Practical significance: Solutions based on the presented methodology will allow the provision of an infrastructure independent of external communication operators to ensure the unity of network clock synchronization and unified time. This is necessary for the correct operation of telecommunication equipment of the technological communication network. The methodology will also meet the needs for providing a full range of synchronization and unified time signal services to existing and prospective consumers. Additionally, it will enable a gradual transition of the organization of communication networks from channel-swithching to packet-oriented technologies due to a flexible approach to the routes and transmission systems used for the distribution of synchronization signals and unified time.

Efficient asynchronous federated neuromorphic learning of spiking neural networks

Spiking Neural Networks (SNNs) can be trained on resource-constrained devices at low computational costs. There has been little attention to training them on a large-scale distributed system like federated learning. Federated Learning (FL) can be exploited to perform collaborative training for higher accuracy, involving multiple resource-constrained devices. In this paper, we introduce SNNs into asynchronous federated learning (AFL), which adapts to the statistical heterogeneity of users and complex communication environments. A novel fusion weight based on information age and average spike rate is designed, which aims to reduce the impact of model staleness. Numerical experiments validate SNNs on federated learning with MNIST, FashionMNIST, CIFAR10 and SVHN benchmarks, achieving better accuracy and desirable convergence under Non-IID settings.

Boosting Erasure-Coded Multi-Stripe Repair in Rack Architecture and Heterogeneous Clusters: Design and Analysis

Large-scale distributed storage systems have introduced erasure codes to guarantee high data reliability, yet inevitably at the expense of high repair costs. In practice, storage nodes are usually divided into different racks, and data blocks in storage nodes are often organized into multiple stripes independently manipulated by erasure code. Due to the scarcity and heterogeneity of the cross-rack bandwidth, the cross-rack network transmission dominates the entire repair costs. We argue that when erasure code is deployed in a rack architecture, existing repair techniques are limited in different aspects: neglecting the heterogeneous cross-rack bandwidth, less consideration for multi-stripe failure, and no special treatment on repair link scheduling. In this paper, we present CMRepair, a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cross-rack Multi-stripe Repair</i> technique that aims to reduce the repair time for multi-stripes failure repair in heterogeneous erasure-coded clusters. CMRepair first carefully chooses the nodes for reading/repairing blocks and searches for the multi-stripe repair solution. It adopts different algorithms to adjust per stripe solution, including the Computation Time Priority (CTP) algorithm based on the greedy idea and the Repair Time Priority (RTP) algorithm based on the meta-heuristics idea. Furthermore, CMRepair selectively schedules the execution orders of cross-rack links, with the primary objective of saturating the unused upload/download bandwidth resources and avoiding network congestion. The experiments show that CMRepair with the CTP algorithm can reduce 27.59%-58.12% of the repair time while only introducing negligible computation overhead, and CMRepair with the RTP algorithm can reduce 33.52%-97.75% of the repair time in an acceptable computation time, over existing repair techniques.

Independence enhancement of distributed generation systems by integrating shared energy storage system and energy community with internal market

The connection of large-scale distributed generation systems, primarily distributed solar and wind power, to the public grid, will cause a considerable burden on its dispatching. How to improve the independence of distributed generation systems is crucial. This study thoroughly investigates the possibility of enhancing the independence of distributed generation systems by combining feasible solutions such as integrating energy community, installing shared energy storage, and adopting rational dispatching strategies while ensuring the benefits of multiple parties. A two-level framework for optimizing energy community scheduling and shared energy storage system sizing is proposed. The upper layer uses a multi-objective approach to optimize the size of the shared energy storage system, which ensures the economy of the shared energy storage system and the independence of the energy community. The lower layer dispatches power intending to maximize social welfare and allocates cooperative benefits through the Sharply value method, thus ensuring the interests of community participants. And a dynamic electricity price strategy in response to the imbalance between supply and demand to guide community dispatching is proposed, which reduces the peak-to-valley range while ensuring economic performance. The case results show that compared to the distributed generation systems operating alone, integrating the energy community increased the self-sufficiency and self-consumption rates of the community by 6.01 % and 6.01 %, respectively, and installing shared energy storage system increased the self-sufficiency and self-consumption rates of the community by 13.50 % and 19.35 %, respectively.

Over-the-Air Federated Multi-Task Learning Over MIMO Multiple Access Channels

With the explosive growth of data and wireless devices, federated learning (FL) over wireless medium has emerged as a promising technology for large-scale distributed intelligent systems. Yet, the urgent demand for ubiquitous intelligence will generate a large number of concurrent FL tasks, which may seriously aggravate the scarcity of communication resources. By exploiting the analog superposition of electromagnetic waves, over-the-air computation (AirComp) is an appealing solution to alleviate the burden of communication required by FL. However, sharing frequency-time resources in over-the-air computation inevitably brings about the problem of inter-task interference, which poses a new challenge that needs to be appropriately addressed. In this paper, we study over-the-air federated multi-task learning (OA-FMTL) over the multiple-input multiple-output (MIMO) multiple access (MAC) channel. We propose a novel model aggregation method for the alignment of local gradients of different devices, which alleviates the straggler problem in over-the-air computation due to the channel heterogeneity. We establish a communication-learning analysis framework for the proposed OA-FMTL scheme by considering the spatial correlation between devices, and formulate an optimization problem for the design of transceiver beamforming and device selection. To solve this problem, we develop an algorithm by using alternating optimization (AO) and fractional programming (FP), which effectively mitigates the impact of inter-task interference on the FL learning performance. We show that due to the use of the new model aggregation method, device selection is no longer essential, thereby avoiding the heavy computational burden involved in selecting active devices. Numerical results demonstrate the validity of the analysis and the superb performance of the proposed scheme.

Data Distribution for Heterogeneous Storage Systems

The exponential growth of data in many science and engineering domains poses significant challenges to storage systems. Data distribution is a critical component in large-scale distributed storage systems and plays a vital role in placing petabytes of data and beyond, among tens to hundreds of thousands of storage devices. Meantime, heterogeneous storage systems, such as those having devices with hard disk drives (HDDs) and storage class memories (SCMs), have become increasingly popular for massive data storage due to their distinct and complement characteristics. This paper presents a new data distribution algorithm called SUORA (Scalable and Uniform storage via Optimally-adaptive and Random number Addressing) specifically for heterogeneous devices to maximize the benefits of them. SUORA provides a fully symmetric, highly efficient methodology to distribute data across a hybrid and tiered storage cluster. It divides heterogeneous devices into different buckets and segments, and adopts pseudo-random functions to map data onto them with the balanced consideration of capacity, performance and life-time. By analyzing hotness and access patterns, SUORA gradually moves hot data from HDDs to SCMs to optimize the throughput, and moves cold data reversely for load balance. It combines data replication with migration to significantly reduce movement overhead while making data placement more adaptive to different workloads. Extensive evaluations on simulation and Sheepdog storage system show that, with considering distinct characteristics of various devices thoroughly, SUORA improves the overall performance efficiency of heterogeneous storage systems.

Smash: Flexible, Fast, and Resource-efficient Placement and Lookup of Distributed Storage

Large-scale distributed storage systems, such as object stores, usually apply hashing-based placement and lookup methods to achieve scalability and resource efficiency. However, when object locations are determined by hash values, placement becomes inflexible, failing to optimize or satisfy application requirements such as load balance, failure tolerance, parallelism, and network/system performance. This work presents a novel solution to achieve the best of two worlds: flexibility while maintaining cost-effectiveness and scalability. The proposed method Smash is an object placement and lookup method that achieves full placement flexibility, balanced load, low resource cost, and short latency. Smash utilizes a recent space-efficient data structure and applies it to object-location lookups. We implement Smash as a prototype system and evaluate it in a public cloud. The analysis and experimental results show that Smash achieves full placement flexibility, fast storage operations, fast recovery from node dynamics, and lower DRAM cost (&lt;60%) compared to existing hash-based solutions such as Ceph and MapX.

Power Big Data Analysis Platform Design Based on Hadoop

To better analyze and process power data to obtain effective information, a power big data analysis scheme based on Hadoop architecture is proposed. We analyze the cloud computing environment Hadoop distributed platform to obtain massive data of large-scale distributed power system. According to the characteristics of intelligent power consumption data, common data mining algorithm modules such as parallel classification algorithm and parallel real-time clustering algorithm are designed, and the implementation of clustering algorithm with different principles is further analyzed. Then, HBase is adopted to access data in a distributed way, and MapReduce is used to realize the visual management of power GIS data. The experimental results show that the parallel processing method of power big data based on Hadoop has high efficiency and good scalability, and the algorithm has good identification ability for massive data in the cluster mode.

Research on Distributed Measurement and Control System Based on FlexRay Bus

With the purpose of the design problem of electromagnetic launch distributed measurement and control system, this paper proposes a design method of scalable large-scale distributed interconnected measurement and control system based on FlexRay bus. The task execution time deviation of each node in the distributed system, including phase deviation and frequency deviation, is reduced by using the macro tick generation process and clock synchronization process to achieve task synchronization in the distributed system. The synchronization of timing generation is realized based on the universal clock technology. When the timing interval is zero, the pulse power units are guaranteed to be close to the ideal full synchronization, thus ensuring that the released energy during transmission is close to the design value. In the state of adjustable timing, when the pulse power supply module conducts pulse discharge, the trigger discharge times of the modules are set according to the load needs. The flat top, sawtooth, and other characteristic current waveforms are adjusted accurately. At the same time, a combined multi-level protection structure based on hardware and software is realized. The verification results show that the distributed electromagnetic launch measurement and control system implemented in this paper can meet the high efficiency, fast and continuous launch target.

A Literature Review of Big Data and Metadata Management Systems

Abstract: Metadata is essential for the successful operation of data files, as it acts as the underlying data that helps maintain and manage the namespace, permissions, and address of the file data blocks. The performance of metadata services has a direct impact on the overall performance of the file system, as these operations can account for a significant portion, up to 80%, of file system activity. Large-scale distributed file systems (DFS) have emerged to meet the rising demand for data storing. These systems provide users with access to numerous storage units spread across various locations. The widespread use of large-scale DFSs as the foundation for a range of computing systems and applications, particularly in the era of Big Data, highlights the importance of efficient metadata management. This paper seeks to offer an enhanced knowledge of the current status of metadata systems in large-scale DFSs, this paper presents a comprehensive overview of the latest research in the field. DFSs are defined by three aspects: scalability, performance, and availability. Moreover, the study evaluates various challenges and limitations in the current research, providing valuable insight for future studies. Overall, the study attempts to give a complete knowledge of the present scenario of metadata services across large-scale DFSs and to recommend areas for future research and development.

Optimal packing and planning for large-scale distributed rooftop photovoltaic systems under complex shading effects and rooftop availabilities

Rooftop photovoltaics (PVs) are considered a promising solution to alleviating current cities’ escalating energy usage and carbon emissions. In high-density cities, complex shading effects and rooftop availabilities (caused by diversified rooftop obstacles and irregular rooftop outlines) jointly make planning of large-scale distributed rooftop PV systems critically challenging. This study proposed an optimal packing and planning method for large-scale distributed rooftop PV systems under complex shading and rooftop availabilities, tackling the challenges by decoupling optimal packing and planning into two-step optimization. Utilizing horizontal-level genetic algorithm, the method first optimized PV-panels packing on irregular-shaped rooftops to maximize area utilization. Second, adopting sequential integer linear programming, the method optimized the planning of individual rooftop packing levels to minimize levelized cost of electricity (LCOE). Based on a 139-rooftop Hong Kong case study, the method was verified against 1 billion Monte-Carlo solutions, which reduced LCOE by 48.0% at most and achieved the lowest LCOE of 0.365 HKD/kWh. Further analysis showed that the proposed method outperformed a rule-based planning method because of its better utilization of high solar-energy-intensity areas, reducing the LCOE by 15.4%. In practice, the method can be used to facilitate deployment of large-scale distributed rooftop PV, enhancing overall system cost-effectiveness and city decarbonization.

New Optimal Linear Codes With Hierarchical Locality

Locally repairable codes with hierarchical locality (H-LRCs) are designed to correct different numbers of erasures, which play a crucial role in large-scale distributed storage systems. In this paper, we construct three classes of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula> -ary optimal H-LRCs by employing matrix product codes, concatenated codes and cyclic codes, respectively. The first two constructions are based on the idea of constructing new codes from old, which produces several new classes of optimal H-LRCs whose lengths can reach up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$q^{2}+q$ </tex-math></inline-formula> or unbounded. The final construction generates a class of new optimal cyclic H-LRCs whose lengths divide <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$q-1$ </tex-math></inline-formula> . Compared with the previously known ones, our constructions are new in the sense that their parameters are not covered by the codes available in the literature.

HV-SNSP: A Low-Overhead Data Recovery Method Based on Cross-Checking

The failure of a single unreliable commodity component is very common in large-scale distributed storage systems. In order to ensure the reliability of data in large-scale distributed storage systems, many studies have emerged one after another. Among them, Erasure Codes are widely used in actual storage systems, such as Hadoop Distributed File System (HDFS), which provides high fault-tolerance with lower storage overhead. However, usually the recovery of erasure-coded storage system when encountering node failure results in severe cross-node and cross-rack bandwidth consumption, which affects the efficiency of failure recovery and wastes additional resources. In this paper, we improve the erasure coding storage strategy in distributed storage systems, and propose a low-overhead data recovery method based on cross-checking, namely HV-SNSP. In HV-SNSP, horizontal and vertical cross parity checking is realized by adding RS parity inside the data node, that is, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{H}^{\mathrm {RS(n, k)}}$ </tex-math></inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{V}^{\mathrm {RS(n', \textrm {}k')}}$ </tex-math></inline-formula> storage architecture. Based on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{H}^{\mathrm {RS(n, \textrm {}k)}}$ </tex-math></inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{V}^{\mathrm {RS(n', k')}}$ </tex-math></inline-formula> , a low-cost supply node selection strategy, namely SNSP, is designed, and nodes with shorter network distance and lower load are selected to participate in recovery. This strategy can effectively reduce the amount of data transmission, shorten the recovery time, and improve the recovery efficiency. The experimental results show that compared with traditional RS, HV-SNSP can reduce the amount of cross-rack data transmission by 62.5% during data recovery, and can shorten the recovery time by up to 42.41%; Compared with D3, HV-SNSP can reduce the occupation of cross-rack bandwidth by 25% and shorten the recovery time by 36.58%.

Scalable Long-Term Safety Certificate for Large-Scale Systems

This letter focuses on safe control problems for high-dimensional systems with large uncertainties. A major challenge is the computation load to account for long outlook horizons in large-scale systems. This challenge is tackled using an integration of probabilistic forward invariance, the comparison theorem, and PDE techniques. Specifically, we propose a probabilistic certificate for long-term safety that only requires myopically ensuring linear control constraints and evaluating two-dimensional PDEs regardless of the system dimension. The certificate is constructed by obtaining a long-term safe probability bound as a solution of the PDE using the comparison theorem and applying a new notion of probabilistic forward invariance on the probability bound. The use of forward invariance directly on probabilistic reachability allows our method to carry the former’s computation efficiency and the latter’s control over long-term behaviors. Its capability to efficiently ensure long-term safety for high-dimensional systems can be useful in many large-scale distributed autonomous systems operating with limited onboard resources in latency-critical environments.

Byzantine-Robust Distributed Learning With Compression

Communication between workers and the master node to collect local stochastic gradients is a key bottleneck in a large-scale distributed learning system. Various recent works have proposed to compress the local stochastic gradients to mitigate the communication overhead. However, robustness to malicious attacks is rarely considered in such a setting. In this work, we investigate the problem of Byzantine-robust compressed distributed learning, where the attacks from Byzantine workers can be arbitrarily malicious. We theoretically point out that different to the attacks-free compressed stochastic gradient descent (SGD), its vanilla combination with geometric median-based robust aggregation seriously suffers from the compression noise in the presence of Byzantine attacks. In light of this observation, we propose to reduce the compression noise with gradient difference compression so as to improve the Byzantine-robustness. We also observe the impact of the intrinsic stochastic noise caused by selecting random samples, and adopt the stochastic average gradient algorithm (SAGA) to gradually eliminate the inner variations of regular workers. We theoretically prove that the proposed algorithm reaches a neighborhood of the optimal solution at a linear convergence rate, and the asymptotic learning error is in the same order as that of the state-of-the-art uncompressed method. Finally, numerical experiments demonstrate the effectiveness of the proposed method.