Large-scale Distributed Environments Research Articles

The Diagnosis-Effective Sampling of Application Traces

Distributed tracing is cutting-edge technology used for monitoring, managing, and troubleshooting native cloud applications. It offers a more comprehensive and continuous observability, surpassing traditional logging methods, and is indispensable for navigating modern complex software architectures. However, the sheer volume of generated traces is staggering in distributed applications, and the direct storage and utilization of every trace is impractical due to associated operational costs. This entails a sampling strategy to select which traces warrant storage and analysis. Historically, sampling methods have included a rate-based approach, often relying heavily on a manual configuration. There is a need for a more intelligent approach, and we propose a hierarchical sampling methodology to address multiple requirements concurrently. Initial rate-based sampling mitigates the overwhelming volume of traces, as no further analysis can be performed on this level. In the next stage, more nuanced analysis is facilitated based on the previous foundation, incorporating information regarding trace properties and ensuring the preservation of vital process details even under extreme conditions. This comprehensive approach not only aids in the visualization and conceptualization of applications but also enables more targeted analysis in later stages. As we delve deeper into the sampling hierarchy, the technique becomes tailored to specific purposes, such as the simplification of application troubleshooting. In this context, the sampling strategy prioritizes the retention of erroneous traces from dominant processes, thus facilitating the identification and resolution of underlying issues. The focus of this paper is to reveal the impact of sampling on troubleshooting efficiency. Leveraging intelligent and explainable artificial intelligence solutions enables the detection of malfunctioning microservices and provides transparent insights into root causes. We advocate for using rule-induction systems, which offer explainability and efficacy in decision-making processes. By integrating advanced sampling techniques with machine-learning-driven intelligence, we empower organizations to navigate the complexities of large-scale distributed cloud environments effectively.

Distributed Tracing for Troubleshooting of Native Cloud Applications via Rule-Induction Systems

Diagnosing IT issues is a challenging problem for large-scale distributed cloud environments due to complex and non-deterministic interrelations between the system components. Modern monitoring tools rely on AI-empowered data analytics for detection, root cause analysis, and rapid resolution of performance degradation. However, the successful adoption of AI solutions is anchored on trust. System administrators will not unthinkingly follow the recommendations without sufficient interpretability of solutions. Explainable AI is gaining popularity by enabling improved confidence and trust in intelligent solutions. For many industrial applications, explainable models with moderate accuracy are preferable to highly precise black-box ones. This paper shows the benefits of rule-induction classification methods, particularly RIPPER, for the root cause analysis of performance degradations. RIPPER reveals the causes of problems in a set of rules system administrators can use in remediation processes. Native cloud applications are based on the microservices architecture to consume the benefits of distributed computing. Monitoring such applications can be accomplished via distributed tracing, which inspects the passage of requests through different microservices. We discuss the application of rule-learning approaches to trace traffic passing through a malfunctioning microservice for the explanations of the problem. Experiments performed on datasets from cloud environments proved the applicability of such approaches and unveiled the benefits.

Experimental Performance Analysis of a Scalable Distributed Hyperledger Fabric for a Large-Scale IoT Testbed.

Blockchain technology, with its decentralization characteristics, immutability, and traceability, is well-suited for facilitating secure storage, sharing, and management of data in decentralized Internet of Things (IoT) applications. Despite the increasing development of blockchain platforms, there is still no comprehensive approach for adopting blockchain technology in IoT systems. This is due to the blockchain’s limited capability to process substantial transaction requests from a massive number of IoT devices. Hyperledger Fabric (HLF) is a popular open-source permissioned blockchain platform hosted by the Linux Foundation. This article reports a comprehensive empirical study that measures HLF’s performance and identifies potential performance bottlenecks to better meet the requirements of blockchain-based IoT applications. The study considers the implementation of HLF on distributed large-scale IoT systems. First, a model for monitoring the performance of the HLF platform is presented. It addresses the overhead challenges while delivering more details on system performance and better scalability. Then, the proposed framework is implemented to evaluate the impact of varying network workloads on the performance of the blockchain platform in a large-scale distributed environment. In particular, the performance of the HLF is evaluated in terms of throughput, latency, network size, scalability, and the number of peers serviceable by the platform. The obtained experimental results indicate that the proposed framework can provide detailed real-time performance evaluation of blockchain systems for large-scale IoT applications.

Scalable Access Control Scheme of Internet of Things Based on Blockchain

With the development of Internet of things (IoT) technology, a large-scale, heterogeneous and dynamic distributed IoT environment has gradually formed between different IoTs. In order to solve the scalability problem of restricted device access management in the Internet of things, a distributed access control system model of the Internet of things based on blockchain technology is proposed. The system model adopts a single smart contract, which simplifies the whole process in the blockchain network and reduces the communication overhead between nodes. According to the simulation results and evaluation, it is proved that the solution has good scalability.

BLOCKCHAIN TECHNOLOGY INTO STEEL INDUSTRY CURRENT STATE OF FOREIGN PAYMENT ACTIVITY

The purpose of the article is to analyze the current state of foreign trade in the industrial blockchain of the steel industry, the existing digital platform has gradually formed a large-scale heterogeneous distributed environment. The problems of further development of the metallurgical complex of Ukraine in the world electronic system of international cooperation are determined. Methodology. The survey is based on a blockchain technology is an interlinked systematic chain of blocks that contains transaction history and other user data. It works under the principle of decentralized distributed digital ledger. Results. Experimental tests prove that production companies, logistics, international payments and consumers can participate in the information certification of steel products via the modern information system. Consumers can understand the real product manufacturing process, effectively avoiding of the incomplete information and low transparency in the traditional information traceability process, and effectively trace the quality of steel products. The system provides an effective payments scheme for promoting the transformation and upgrading of the modern technology steel industry. Practical implications. In view of the low transparency of information traceability of current steel products and the defects of information islands, in this article the blockchain-based steel smart contract payment technology quality traceability system is developed and adopted the alliance chain mode and the new digital blockchain platform. The article describes the use of smart payment technology in a full-fledged financial transaction as a transfer funds in cryptocurrency to electronic bank accounts, as well as the transfer of commercial information, including contracts. To do this, the assets and terms of the contract are encoded and placed in the block chain, then the contract is distributed is saved on a set of network nodes and is executed after the condition is triggered. The fulfillment of the obligations of the parties is checked automatically. It is worth noting that the fact that smart contracts only react to transactions. If the asset or the currency is transferred to the program, it starts monitor compliance with the terms of the contract. How as soon as they are fulfilled, the seller receives money, and the buyer is a shipment goods. Value/originality. Blockchain technologies enable the use of business information systems that are highly resistant to technical failures and malicious attacks. Costs and time of carrying out business operations are significant.

A local external coupling matrix solution and dynamic processing in medical cyber-physical cloud systems

Abstract Medical cyber-physical cloud systems (MCPCS) are new generation medical cyber-physical systems (MCPS), which introduce cloud computing to integrate multiple MCPS to improve quality of service (QoS) and expand service scope in hospitals. When used in large-scale distributed environment, MCPCS need to perform a stability analysis to guarantee collaboration among multiple MCPS. In this paper, we propose a local topology search algorithm (LTS) to obtain the local external coupling matrices, which are important input parameters for the local synchronization stability analysis of MCPCS. LTS generates an optimal multi-fork tree from the source end MCPS to multiple destination end MCPS. Based on the relationship between nodes in the optimal multi-fork tree, the local external coupling matrices applied to the local synchronization stability analysis of MCPCS are then solved. Considering the variability of network topology, we further propose a dynamic processing method called external coupling matrix change processing (ECMCP) to ensure the real-time update of the local external coupling matrices and dimensional consistency. Experimental results show that the proposed solution of LTS & ECMCP has lower costs of nodes and their communications than existing method.

Blockchain Technology for Applications in Internet of Things—Mapping From System Design Perspective

Internet of Things (IoT) refers to networks with billions of physical devices for collecting, sharing, and utilizing data in the virtual world. Most of IoT applications centralize security assurance in creating, authenticating, transferring, or delating system components. However, the centralization exposes its limitations to meet security needs of a rapidly growing number of things world-widely. How to scale up the applications with assured security becomes a critical challenge. Blockchain technology (BCT) is a promising solution to provide security and protect privacy in a large scale; especially, smart contracts offer opportunities to improve the reliability of IoT applications. Smart contracts establish trusts for both of data and executed processes. Recently, many literature surveys and positioning articles have been published on the integration of BCT with IoT, but they are limited to superficial discussions of technical potentials, and very few of them have a thorough exploration of the challenges in developing BCT for IoT at technical levels. This paper uses the system design approach to scrutinize the state of the art of study on BCT-based applications and clarify critical research areas of enabling BCT for security assurance: 1) the relations of BCT and IoT are modeled and discussed; 2) the needs of eliminating threats in IoT-based applications are defined as functional requirements (FRs), existing works on enabling technologies of BCT are defined as the physical solutions (PSs); and 3) the mappings between FRs and PSs are established to identify the limitations and the critical areas for the applications of BCT in large-scale distributed environment.

Large-scale distributed semantic augmented reality services – A performance evaluation

This paper presents an architecture for large-scale distributed AR services and an experimental evaluation of the main component – Semantic Augmented Reality Middleware. The architecture is based on a client-server design, which supports semantic modeling and building contextual AR presentations for a large number of users. The architecture is based on the Service Oriented Architecture (SOA) paradigm that enables building distributed systems that provide application functionality as services to either end-user applications or other services. The proposed architecture consists of two client-side components and multiple server-side distributed service applications – semantic middleware and AR service providers. The presented findings are promising and demonstrate that application of semantic web techniques can be a useful solution to implement large-scale contextual distributed AR environments.

Secure and Efficient Multi-Authority Attribute-Based Encryption Scheme From Lattices

Attribute-based encryption (ABE) mechanism on lattices can against quantum attack and guarantees fine-grained access control of encrypted data, and it has been widely used in cloud computing and privacy protection. However, the high complexity of the trapdoor generation and delegation and preimage sampling are the main barriers for the development of ABE schemes in a post-quantum world. Thus, we proposed a security and efficient multi-authority ABE scheme from lattices in this paper. The scheme is based on the theory of MP12 and is proved to be secure under the standard model. In this scheme, we construct an optimized sampling algorithm to generate a $\mathbb {Z}_{q}$ -invertible matrix with a lower runtime. Based on it, combining the trapdoor generation and delegation algorithm in MP12, we designed multiple attribute authorities which can manage different attribute sets and generate private keys for the user independently. In addition, the Shamir’s secret sharing technique is introduced to support policies expressed in any monotone access structures. Compared with the existing related schemes, the proposed scheme can improve the functional agility and the flexibility of the access policy prominently, and it also can achieve a better performance with less lattice dimension, trapdoor storage occupation, and ciphertext expansion rate. The analysis shows that our scheme is feasible and superior in the large-scale distributed environment.

CDLB: a cross-domain load balancing mechanism for software defined networks in cloud data centre

Currently, cross-domain load balancing is one of the core issues for software defined networks (SDN) in cloud data centre, which can optimise resource allocation. In this paper, we propose a cross-domain load balancing mechanism, CDLB, based on Extensive Messaging and Presence Protocol (XMPP) for SDN in cloud data centre. Different from poll method, XMPP based push model is introduced in the proposed scheme, which can avoid wasting network and computing resources in large-scale distributed network environment. The proposed scheme enables all the controllers in the flat distributed control plane to share the same consistent global-view network information in real time through XMPP and XMPP publish/subscribe extension. Thus, the problem of non-real time information synchronisation can be resolved and cross-domain load balancing can be realised. The simulations show the efficiency of the proposed scheme.

CDLB: a cross-domain load balancing mechanism for software defined networks in cloud data centre

Currently, cross-domain load balancing is one of the core issues for software defined networks (SDN) in cloud data centre, which can optimise resource allocation. In this paper, we propose a cross-domain load balancing mechanism, CDLB, based on Extensive Messaging and Presence Protocol (XMPP) for SDN in cloud data centre. Different from poll method, XMPP based push model is introduced in the proposed scheme, which can avoid wasting network and computing resources in large-scale distributed network environment. The proposed scheme enables all the controllers in the flat distributed control plane to share the same consistent global-view network information in real time through XMPP and XMPP publish/subscribe extension. Thus, the problem of non-real time information synchronisation can be resolved and cross-domain load balancing can be realised. The simulations show the efficiency of the proposed scheme.

Iteration acceleration for distributed learning systems

Abstract During the implementation of iterative machine learning algorithms for objective function optimization in large-scale distributed environment, they can be blocked when some of the machines failed. In this paper, a hybrid approach is proposed to balance the performance and efficiency. In each iteration, the results from failure machines are abandoned. We will discuss the relationship between accuracy and abandon rate which can be formulated as inequations. It is argued that the speed of this process is highly effective and efficient. The algorithm is demonstrated using real distributed environment and shows significant improvement of speedup results.

Sensor data distribution and knowledge inference framework for a cognitive-based distributed storage sink environment

Large-scale sensor data distributions and knowledge inferences are major challenges for cognitive-based distributed storage environments. Cognitive storage sinks play an essential role in addressing these challenges. In a data-concentrated distributed cognitive sensor environment, cognitive storage sinks regulate the data distribution operations and infer knowledge from the large amounts of sensor data that are distributed across the conventional sensors. Embedding cognitive functions in conventional sensors is unreasonable, and the knowledge-processing limitations of conventional sensors create a serious problem. To overcome this problem, we propose a cognitive co-sensor platform across a large-scale distributed environment. Further, we propose a distributed data distribution framework (DDD-framework) for effective data distributions and a distributed knowledge inference framework (DKI-framework) that infers useful patterns for building knowledge intelligence. The analysis and discussion demonstrate that these frameworks can be adequately instigated for the purpose of optimal data distribution and knowledge inference within the horizon of a real-time distributed environment.

Secure Pub-Sub: Blockchain-Based Fair Payment With Reputation for Reliable Cyber Physical Systems

The cyber physical system (CPS) has gained considerable success in large-scale distributed integration environment. In such systems, the sensor devices collect data which would be disseminated via reliable manner to all interested co-operant entities from the physical world. However, highly unreliable environment of CPS, for example, a number of limitations of existing network middle wares, makes secure and reliable data distribution services a challenge issue. In this paper, we propose a new architecture called secure pub-sub (SPS) without middle ware, i.e., blockchain-based fair payment with reputation. In SPS, publishers publish a topic on the blockchain and subscribers specify an interest message by making a deposit to subscribing the topic. Then, if the interest message matches the topic, the publisher transmits the encrypted content of the topic to the blockchain such that the subscribers can decrypt the ciphertext to obtain the content, and mark the publisher as its reputation. Finally, the publisher receives the payment from the subscriber. The new proposal provides confidentiality and reliability of data, anonymity of subscribers and payment fairness between the publishers and subscribers. Different from the traditional pub-sub services, no trusted third party is involved in our system due to employing blockchain technique. The security of the proposed SPS is analyzed as well. The implementation of the protocol on Ethereum of smart contract demonstrates the validity of SPS.

Resource discovery for distributed computing systems: A comprehensive survey

Large-scale distributed computing environments provide a vast amount of heterogeneous computing resources from different sources for resource sharing and distributed computing. Discovering appropriate resources in such environments is a challenge which involves several different subjects. In this paper, we provide an investigation on the current state of resource discovery protocols, mechanisms, and platforms for large-scale distributed environments, focusing on the design aspects. We classify all related aspects, general steps, and requirements to construct a novel resource discovery solution in three categories consisting of structures, methods, and issues. Accordingly, we review the literature, analyzing various aspects for each category.

A checkpointed league championship algorithm-based cloud scheduling scheme with secure fault tolerance responsiveness

In its simplest structure, cloud computing technology is a massive collection of connected servers residing in a datacenter and continuously changing to provide services to users on-demand through a front-end interface. The failure of task during execution is no more an accident but a frequent attribute of scheduling systems in a large-scale distributed environment. Recently, some computational intelligence techniques have been mostly utilized to decipher the problems of scheduling in the cloud environment, but only a few emphasis on the issue of fault tolerance. This research paper puts forward a Checkpointed League Championship Algorithm (CPLCA) scheduling scheme to be used in the cloud computing system. It is a fault-tolerance aware task scheduling mechanisms using the checkpointing strategy in addition to tasks migration against unexpected independent task execution failure. The simulation results show that, the proposed CPLCA scheme produces an improvement of 41%, 33% and 23% as compared with the Ant Colony Optimization (ACO), Genetic Algorithm (GA) and the basic league championship algorithm (LCA) respectively as parametrically measured using the total average makespan of the schemes. Considering the total average response time of the schemes, the CPLCA scheme produces an improvement of 54%, 57% and 30% as compared with ACO, GA and LCA respectively. It also turns out significant failure decrease in jobs execution as measured in terms of failure metrics and performance improvement rate. From the results obtained, CPLCA provides an improvement in both tasks scheduling performance and failure awareness that is more appropriate for scheduling in the cloud computing model.

An enhanced active caching strategy for data-intensive computations in distributed GIS

Caching can prepare data for computational tasks in advance by tracking the requirements and behaviors of distributed geographical information systems to reduce network latency and improve computational performance. This paper presents an enhanced method to actively cache data for data-intensive computations that considers both data relationships and the timeliness of those relationships. First, the access correlations, the correlation steps and the times of the correlations are computed based on the behaviors of the computational tasks. Because the influence of historically accessed records will decrease gradually over time, only recently accessed records are used. To track changes in the relationships and prevent cache waste problems, each record is given a different age-based weight. A conditional caching probability can then be computed based on the timeliness relationships, which can be used to find the appropriate data to compute simultaneously. Finally, we present several experiments that compare the proposed method with techniques that use other data placement strategies, active caching strategies and passive caching algorithms. The results show that the proposed model has better performance than other algorithms in all respects. In addition, the proposed model results in a lower cache replacement ratio. The experiments with different data sets on different data scales indicate that the proposed algorithm can also be used in large-scale distributed environments.

Semantic Grid resource discovery based on SKOS ontology

In large-scale distributed environments like dynamic Grids, an efficient and scalable resource discovery mechanism is required. Hybrid techniques based on node clustering are considered among solutions that guarantee the efficiency and the scalability at the same time. However, lack of a good representation of nodes in these environments often results in an irrelevant grouping of nodes. Combining hybrid techniques with Semantic Web technologies would bring more benefits to resource discovery. This paper introduces a semantic clustering of nodes based on their domains of interest to form groups called federations. It presents a new process for constructing federations and a three-layered overlay network. The proposed process is based on a SKOS lightweight ontology that describes domains of applications in the Grid. We conduct extensive simulations to evaluate the performance of the proposed approach; the experimental results demonstrate its efficiency and its ability to scale up with the system size.

Optimized Resource Provisioning Method for Computational Grid

Grid computing is an accumulation of heterogeneous, dynamic resources from multiple administrative areas which are geographically distributed that can be utilized to reach a mutual end. Development of resource provisioning-based scheduling in large-scale distributed environments like grid computing brings in new requirement challenges that are not being believed in traditional distributed computing environments. Computational grid is applying the resources of many systems in a network to a single problem at the same time. Grid scheduling is the method by which work specified by some means is assigned to the resources that complete the work in the environment which cannot fulfill the user requirements considerably. The satisfaction of users while providing the resources might increase the beneficiary level of resource suppliers. Resource scheduling has to satisfy the multiple constraints specified by the user. The option of resource with the satisfaction of multiple constraints is the most tedious process. This trouble is solved by bringing out the particle swarm optimization based heuristic scheduling algorithm which attempts to select the most suitable resource from the set of available resources. The primary parameters that are taken in this work for selecting the most suitable resource are the makespan and cost. The experimental result shows that the proposed method yields optimal scheduling with the atonement of all user requirements.

A multi-agent protocol for service level agreement negotiation in cloud federations

The emergence of complex cloud services and the growing number of QoS metrics entail the need of automation to transform business requirements into QoS constraints. Furthermore, in the emergent context of Cloud Federations providers need to share knowledge to improve interoperability when negotiating SLA Service Level Agreements. Intelligent software agents are capable of supporting activities related to SLA negotiation, as well as sharing knowledge into large-scale distributed environments. Since they are usually provided with different ontologies, understanding the meaning of the messages requires the availability of common, global ontologies among customers and providers. In order to deal with this issue, we discuss a multi-agent SLA negotiation protocol which does not need a common global agent ontology. The protocol takes into account different concerns related to the understanding of semantic and technical terms, and the knowledge about the agent abilities on semantic and technical terms are shared among the Cloud Federation.