Distributed Computing Research Articles

Enhancing Rate-Splitting-Based Distributed Edge Computing via Multi-Group Information Recycling

To address the straggling effect in distributed edge computing, existing methods often introduce extra computation or communication costs. Recently, information recycling has emerged as an efficient solution that avoids such extra overhead. Nonetheless, the performance of the existing information recycling-assisted rate-splitting-based distributed edge computing scheme significantly hinges on the single common stream, whose rate is limited by the edge node (EN) with the worst channel quality. To this end, a multi-group information recycling scheme for rate-splitting-based distributed edge computing is proposed, where the ENs are clustered into multiple groups, each with its own common stream. In this way, the proposed scheme alleviates the communication limitation of a single common stream, thereby boosting the information recycling mechanism for promoted computation collaboration. A K-medoids-based grouping algorithm is designed for the general multi-antenna case, and a more efficient continuous partitioning-based grouping algorithm is proposed for the special case of single-antenna. Besides, a convex–concave procedure-based algorithm is developed to solve the corresponding latency optimization problem. Simulations demonstrate that the proposed scheme offers significant and robust advantages across various communication and computation conditions. In the considered scenarios, the proposed scheme can substantially reduce the processing latency by up to 51.0% compared to the conventional information recycling scheme.

Whale Optimized Distributed Computing Data Lake for Energy Storage

This paper presents the Whale Seahorse Optimization Distributed Computing (WSODS) algorithm, a novel approach that combines the Whale Optimization Algorithm (WOA) and Seahorse Optimization Algorithm (SOA) within a distributed computing framework. WSODS aims to address complex optimization challenges across various domains, including power storage systems and data lake architectures. The algorithm's performance was evaluated based on key metrics such as data processing time, system throughput, resource utilization, and scalability. The evaluation results indicate that WSODS significantly enhances system performance. In the context of power storage systems, WSODS improves energy efficiency, storage capacity, charging and discharging rates, and round-trip efficiency. For data lake architectures, WSODS achieves lower data processing times, higher system throughput, and competitive resource utilization while demonstrating good scalability with increasing data loads. The evaluation results indicate that WSODS significantly enhances system performance. In the context of power storage systems, WSODS improves energy efficiency from 92% to 98%, storage capacity from 480 MWh to 500 MWh, charging rate from 45 MW to 55 MW, discharging rate from 55 MW to 64 MW, and round-trip efficiency from 88% to 94% over 100 iterations. For data lake architectures, WSODS achieves lower data processing times (450 seconds compared to 600 seconds for GA), higher system throughput (75 MB/s compared to 55 MB/s for GA), and competitive resource utilization (80% CPU and 65% memory), while demonstrating good scalability with processing time for a 2x data load at 920 seconds compared to 1250 seconds for GA. These findings suggest that WSODS is a versatile and robust optimization tool capable of driving advancements in energy efficiency, data analytics, and distributed computing. Further research and real-world applications are recommended to fully explore its potential and capabilities.

A Blockchain and Zero Knowledge Proof Based Data Security Transaction Method in Distributed Computing

In distributed computing, data trading mechanisms are essential for ensuring the sharing of data across multiple computing nodes. Nevertheless, they currently encounter considerable obstacles, including low accuracy in matching trading parties, ensuring fairness in transactions, and safeguarding data privacy throughout the trading process. In order to address these issues, we put forward a data trading security scheme based on zero-knowledge proofs and smart contracts. In the phase of preparing the security parameters, the objective is to reduce the complexity of generating non-interactive zero-knowledge proofs and to enhance the efficiency of data trading. In the pre-trading phase, we devise attribute atomic matching smart contracts based on precise data property alignment, with the objective of achieving fine-grained matching of data attributes between trading parties. In the trading execution phase, lightweight cryptographic algorithms based on elliptic curve cryptography (ECC) and non-interactive zero-knowledge proofs are employed for the dual encryption of trading data and the generation of attribute proof contracts, thus ensuring the security and privacy of the data. The results of experiments conducted on the Ethereum platform in an industrial IoT scenario demonstrate that our scheme maintains stable and low-cost consumption while ensuring accuracy in matching and privacy protection.

Analytic computation for stationary distributions of D-BMAP/D -MSP(a,b)/1 queueing system

ABSTRACT This paper analyses single server correlated batch arrival and correlated batch service queue with innumerable waiting space in discrete-time. The arrival and service processes are governed by discrete batch Markovian arrival process and discrete Markovian service process with general bulk service rule, respectively. Observing two consecutive random epochs, we construct general structure transition probability matrix and it is then reblocked to the desired M/G/1 structure. The UL-type RG-factorization approach combined with spectral expansion method is adopted to unravel the queue length distribution at random epoch. The relationships are undertaken to obtain the queue length distributions at pre-arrival, outside observer's, intermediate and post-departure epochs. Our most challenging effort is to derive the probability mass functions of waiting time and service batch size for arbitrary customer in an arriving batch. We address the warehouse management process for finished products from dealer to retailer as a potential practical use of our batch queue with correlation in both arrival and service, and provide the effect of our queueing behaviour in this application based on our analytical and computational investigation. Based on the implementation of parametrized numeric experiments with numerous categories, we impart our computational results to validate the analytical findings reported in this investigation.

Open Geospatial Engine: A Cloud-based Spatiotemporal Computing Platform

Abstract. The quantity and diversity of Earth spatiotemporal big data have significantly increased in recent years, providing the potential to comprehensively analyse complex spatiotemporal problems from new perspectives. However, integrating, managing, and applying multi-source heterogeneous Earth spatiotemporal big data remains a challenge. To address this issue, this study proposes a cloud-based spatiotemporal computing platform, the Open Geospatial Engine (OGE), for the unified organization and joint analysis of Earth spatiotemporal big data in multiple dimensions and scales. The framework of this platform comprises three modules: data management, computing engine, and service interface. The data management module adopts the GeoCube model to effectively integrate and manage multi-source heterogeneous spatiotemporal data through multi-dimensional aligned tiles. The computing engine module seamlessly maps the GeoCube model to the cloud environment by extending Spark RDD, making the GeoCube model capable of high-performance distributed computing. Following OGC specifications, the service interface module integrates and shares data, operators, and spatiotemporal analysis models through extensible APIs in an interactive development environment. Combined with a series of high-performance optimization techniques, OGE simplifies data queries and the construction of complex analytical applications by integrating these three modules. The applicability of OGE is demonstrated by case studies involving multi-dimensional queries and joint analysis of long-time-series and heterogeneous spatiotemporal data.

Multiplexed quantum frequency conversion.

We study multiplexed quantum frequency conversion (m-QFC) on a single waveguide, where a single pump beam simultaneously converts multiple signal beams at distinct wavelengths. Using a three-peak periodically poled lithium niobate (PPLN) waveguide, we demonstrate the multiplexed upconversion in the telecom band with internal efficiencies up to 73.6%. Tested with a multichannel photon-pair source, the quantum correlation is preserved well after the conversion, with coincidence-to-accidental ratio reaching 767. These results highlight a viable device approach to multiplexed quantum key distribution, quantum sensing, and quantum computing.

Constructed encoded data based coded distributed DNN training for edge computing scenario

Deep learning in unmanned aerial vehicle (UAV) deployment encounters two problems: 1, One UAV may fail to store and execute large Deep Neural Network (DNN) model. 2, One UAV fails to accomplish real time services. One possible solution is the group of UAVs (nodes) collectively generate swarm intelligence in the form of edge computing scenario, namely in the distributed computing (DC) mode with clever arrangement, say Coded Distributed Computing (CDC). In CDC systems, the redundant computation introduced by linear coding can compensate stragglers. However, since linear property cannot pass the nonlinear activation function in Deep Neural Network (DNN) training, coding/decoding for CDC need to be applied layer by layer, which slows down the training. To avoid layer-by-layer coding/decoding, we propose a novel DNN training scheme based on constructing encoded data. This construction process lies before the training process (can be done before training without any impact on training efficiency). Based on both the original data and the newly constructed encoded data, the training phase can take advantage of the (n,k) property (Wait for the first k returned data) and hence improve the training speed. The training process does not require encoding/decoding operations, and hence significantly improves the training speed. Experimental results show that the training scheme based on constructed encoded data can achieve prediction accuracy approximating that of the centralized one and significantly reduce the latency compared to the layer-by-layer linear encoding and decoding scheme.

Partitioning of a 2-bit hash function across 66 communicating cells.

Powerful distributed computing can be achieved by communicating cells that individually perform simple operations. Here, we report design software to divide a large genetic circuit across cells as well as the genetic parts to implement the subcircuits in their genomes. These tools were demonstrated using a 2-bit version of the MD5 hashing algorithm, which is an early predecessor to the cryptographic functions underlying cryptocurrency. One iteration requires 110 logic gates, which were partitioned across 66 Escherichia coli strains, requiring the introduction of a total of 1.1 Mb of recombinant DNA into their genomes. The strains were individually experimentally verified to integrate their assigned input signals, process this information correctly and propagate the result to the cell in the next layer. This work demonstrates the potential to obtain programable control of multicellular biological processes.

Automatic Architecture Design for Distributed Quantum Computing

Abstract In distributed quantum computing (DQC), quantum hardware design mainly focuses on providing as many as possible high-quality inter-chip connections. Meanwhile, quantum software tries best to reduce the required number of remote quantum gates between chips. However, this "hardware first, software follows" methodology may not fully exploit the potential of DQC. Inspired by classical software-hardware co-design, this paper explores the design space of application-specific DQC architectures. More specifically, we propose AutoArch, an automated quantum chip network (QCN) structure design tool. With qubits grouping followed by a customized QCN design, AutoArch can generate a near-optimal DQC architecture suitable for target quantum algorithms. Experimental results show that the DQC architecture generated by AutoArch can outperform other general QCN architectures when executing target quantum algorithms.

Choosing of Telecommunications Equipment Suppliers Based on the Choquet Integral and Fuzzy Measure Variance Minimization

Telecommunications equipment suppliers play an important role in the functioning of corporations with large-scale distributed computer networks such as system integrators, Internet service providers and others. An important condition of sustainable development for these companies is effective and reasonable decision making concerning the choice of suppliers. At the same time, the amount of information to be considered in many industries for making decisions has grown several times. In accordance with the above, decision making support systems for the reliable choice of suppliers started to be developed. At the same time, the approaches to designing such systems possess some disadvantages. In order to overcome these disadvantages this article suggests an approach to decision making support for choosing telecommunication equipment suppliers based on the Choquet Integral and minimization of fuzzy measure variance. It deals with the task of criteria normalization, fuzzy measure identification, gives examples of expert estimations for criteria and also the results of supplier assessments based on which the decision is made.

Self-stabilizing multivalued consensus in asynchronous crash-prone systems

The multivalued consensus problem is a fundamental issue in fault-tolerant distributed computing. It encompasses a wide range of agreement problems where processes must unanimously decide on a specific value v∈V, with |V|≥2. Existing solutions that handle process crash failures simplify the multivalued consensus problem by reducing it to the binary consensus problem. Examples of such solutions include Mostéfaoui-Raynal-Tronel [IPL 2000] and Zhang-Chen [IPL 2009].In this work, we aim to design an even more reliable solution by leveraging the concept of self-stabilization, which provides a strong form of fault tolerance. Self-stabilizing algorithms can recover from transient faults, which represent any deviation from the system's intended behavior (as long as the algorithm code remains intact) in addition to process and communication failures.To the best of our knowledge, this work presents the first self-stabilizing algorithm for multivalued consensus in asynchronous message-passing systems susceptible to process failures and transient faults. Our solution uses, at most, n concurrent invocations of binary consensus. This is another way we advance state-of-the-art solutions compared to previous non-self-stabilizing ones. For example, Mostéfaoui-Raynal-Tronel's solution requires an unbounded number of sequential invocations of binary consensus.

Big Data-driven MLOps workflow for annual high-resolution land cover classification models

Developing an annual and global high-resolution land cover map is one of the most ambitious tasks in remote sensing, with increasing importance due to the continual rise in validated data and satellite imagery. The success of land cover classification models largely hinges on the data quality, coupled with the application of Big Data techniques and distributed computing. This is essential for efficiently processing the extensive volume of available satellite data. However, maintaining the lifecycle of several annual Machine Learning models presents a complex challenge. The rise of Machine Learning Operations offers an opportunity to automate the maintenance of these models, a feature particularly beneficial in systems that require generating new models each year alongside the continuous integration of validated data. This article details the development of an end-to-end MLOps workflow, meticulously integrating land cover classification models that employ Big Data strategies for processing large-scale, high-resolution spatial data. The workflow is designed within a Kubernetes environment, achieving on-demand auto-scaling, distributed computing, and load balancing. This integration demonstrates the practicality and efficiency of managing and deploying models that treat satellite imagery in an automated, scalable framework, thus marking a significant advancement in remote sensing and MLOps.

An expandable and cost-effective data center network

With the rapid growth of data volume, the escalating complexity of data businesses, and the increasing reliance on the Internet for daily life and production, the scale of data centers is constantly expanding. The data center network (DCN) is a bridge connecting large-scale servers in data centers for large-scale distributed computing. How to build a DCN structure that is flexible and cost-effective, while maintaining its topological properties unchanged during network expansion has become a challenging issue. In this paper, we propose an expandable and cost-effective DCN, namely HHCube, which is based on the half hypercube structure. Further, we analyze some characteristics of HHCube, including connectivity, diameter, and bisection bandwidth of the HHCube. We also design an efficient algorithm to find the shortest path between any two distinct nodes and present a fault-tolerant routing scheme to obtain a fault-tolerant path between any two distinct fault-free nodes in HHCube. Meanwhile, we present two local diagnosis algorithms to determine the status of nodes in HHCube under the PMC model and MM* model, respectively. Our results demonstrate that despite the presence of up to 25% faulty nodes in HHCube, both algorithms achieve a correct diagnosis rate exceeding 90%. Finally, we compare HHCube with state-of-the-art DCNs including Fat-Tree, DCell, BCube, Ficonn, and HSDC, and the experimental results indicate that the HHCube is an excellent candidate for constructing expandable and cost-effective DCNs.

Retraction Note: Healthcare framework for identification of strokes based on versatile distributed computing and machine learning

Retraction Note: Healthcare framework for identification of strokes based on versatile distributed computing and machine learning

Replicated multistage interconnection networks: QoS evaluation for parallel and distributed computing

Introduction to big data becomes very important with dealing in high-performance parallel distributed computing, especially in those systems where communication among a number of processors is required. The current paper takes into consideration different topologies for the Shuffle Exchange Network (SEN) in order to attain optimal data transfer among such scenarios. SEN topologies are a key feature in connecting several processors and implementing the data transfer among them where a single processor fails to handle the load. The study hereby reports in the performance, reliability, and cost analysis of these topologies. These topologies, some of which are advocated to have better performance in many studies, include replicated networks. Our research demystifies claims made in earlier papers that replicated networks have inflated reliability and higher costs due to their additional links. Researchers are therefore guided on accurate performance data that will lead them to make optimum choices of SEN topologies for targeted applications. These findings further highlight that the trade-offs between reliability and cost must be carefully considered during network design so as to arrive at better results for big data communication and computation in parallel and distributed systems. This work provides important insights into the correct evaluation of SEN topologies, helping to correct the misleading facts and to have better network selection for various real-time application realizations.

Scalability through Pulverisation: Declarative deployment reconfiguration at runtime

In recent years, the infrastructure supporting the execution of situated distributed computations evolved at a fast pace. Modern collective adaptive applications – as found in the Internet of Things, swarm robotics, and social computing – are designed to be executed on very diverse devices and to be deployed on infrastructures composed of devices ranging from cloud servers to wearable devices, constituting together a cloud–edge continuum. The availability of such an infrastructure opens to better resource utilisation and performance but, at the same time, introduces new challenges to software designers, as applications must be conceived to be able to adapt to changing deployment domains and conditions. In this paper, we introduce a practical framework for the development of systems based on the concept of pulverisation, meant to neatly separate business logic and deployment concerns, allowing applications to be defined independently of the infrastructure they will execute upon, thus supporting scalability. The framework is based on a domain-specific language capturing, in a declarative fashion: pulverised application components, device capabilities, resource allocation, and (runtime re-) configuration policies. The framework, implemented in Kotlin multiplatform and available as open source, is then evaluated in a small-scale real-world demo and in a city-scale simulated scenario, demonstrating the feasibility of the approach and its potential benefits in achieving better trade-offs between performance and resource utilisation.

Microservice-Based Vehicular Network for Seamless and Ultra-Reliable Communications of Connected Vehicles

The fifth-generation (5G) cellular infrastructure is expected to bring about the widespread use of connected vehicles. This technological progress marks the beginning of a new era in vehicular networks, which includes a range of different types and services of self-driving cars and the smooth sharing of information between vehicles. Connected vehicles have also been announced as a main use case of the sixth-generation (6G) cellular, with ultimate requirements beyond the 5G (B5G) and 6G eras. These networks require full coverage, extremely high reliability and availability, very low latency, and significant system adaptability. The significant specifications set for vehicular networks pose considerable design and development challenges. The goals of establishing a latency of 1 millisecond, effectively handling large amounts of data traffic, and facilitating high-speed mobility are of utmost importance. To address these difficulties and meet the demands of upcoming networks, e.g., 6G, it is necessary to improve the performance of vehicle networks by incorporating innovative technology into existing network structures. This work presents significant enhancements to vehicular networks to fulfill the demanding specifications by utilizing state-of-the-art technologies, including distributed edge computing, e.g., mobile edge computing (MEC) and fog computing, software-defined networking (SDN), and microservice. The work provides a novel vehicular network structure based on micro-services architecture that meets the requirements of 6G networks. The required offloading scheme is introduced, and a handover algorithm is presented to provide seamless communication over the network. Moreover, a migration scheme for migrating data between edge servers was developed. The work was evaluated in terms of latency, availability, and reliability. The results outperformed existing traditional approaches, demonstrating the potential of our approach to meet the demanding requirements of next-generation vehicular networks.

Preface

From 22nd to 24th December 2023, the 2023 3rd International Conference on Computational Modeling, Simulation and Data Analysis (CMSDA 2023) was held in Sanya, China via virtual form. The Conference attracted about 150 researchers, engineers, teachers and students worldwide working in or interested in the fields related to computational modeling, simulation and data analysis. The general objectives of the CMSDA 2023 were: To generate an adequate academic and scientific forum to present the work and experiences of researchers/teachers/students, contributing, in this way, to the production of new knowledge in the fields of computational modeling and data analysis. To create a path of establishing certain relationships for the speakers, authors and listeners, and providing opportunities for collaboration among the universities and institutions, in order to promote scientific research activities and develop relevant technologies. The subject areas in which the accepted papers were presented cover but are but not limited to: Perceptual Issues in Visualization and Modeling; Mathematical and Numerical Methods in Simulation and Modeling; Parallel and Distributed Computing Simulation; Big Data Visual Analysis; Data Analysis Algorithms and Systems, etc. We have invited renowned professors at home and abroad in these academic fields who shared with the attendees the latest innovations and research results in the domains of computational modeling, data mining and analysis. The Conference mainly featured keynote speeches by renowned experts and presentations of peer-reviewed papers by the authors. Among them, Prof. Daowen Qiu (Sun Yat-Sen University, China) shared his research on Distributed Quantum Algorithm for Simon’s Problem. Limited by today’s physical devices, quantum circuits are usually noisy and difficult to be designed deeply. The novel computing architecture of distributed quantum computing is expected to reduce the noise and depth of quantum circuits. And in his study, he presented the Simon’s problem in distributed scenarios and designed a distributed quantum algorithm to solve the problem. The algorithm proposed by him has the advantage of not only exponential acceleration compared with the classical distributed computing, but also square acceleration compared with the best distributed quantum algorithm proposed before. The Conference Proceedings was made so fruitful with all the excellent contributions collected from both domestic and international channels including the wonderful presentations in the Conference. It is a pleasure to thank the organizers, sponsors and all the participants and contributors for making the Conference possible and interesting. And our appreciation also to the members of Journal of Physics: Conference Series for providing us help in publishing this paper volume. The Committee of CMSDA 2023 List of Committee Member is available in this pdf.

Simulation Framework for Research and Education in Distributed Computing

Simulation Framework for Research and Education in Distributed Computing

Report on the Collab-a-Thon at ECIR 2024

We present a report on the Collab-a-thon, a series of sessions at the European Conference on Information Retrieval (ECIR) 2024 designed to help foster new collaborations during a conference. This report presents the motivation and design of the Collab-a-thon, a summary of the discussions covered at each session, and a set of recommendations for conducting similar events in the future. The event is set to run again at ECIR 2025 and planning is underway to pilot the event in a different community at the IEEE International Conference on Distributed Computing Systems (ICDCS) 2025. Date : 25--27 March 2024. Website : https://www.ecir2024.org/collab-a-thon/.