Heterogeneous Distributed System Research Articles

Scheduling-Efficient Framework for Neural Network on Heterogeneous Distributed Systems and Mobile Edge Computing Systems

As the volume of machine learning training data sets and the quantity of model parameters continue to grow, the pattern in which machine learning models are trained alone can no longer accommodate large-scale data environments. However, distributed systems and mobile edge computing systems are unpredictable and have heterogeneous nodes, resulting in interruptions in training or low convergence rate. In addition, existing distributed machine learning frameworks cannot guarantee a good convergence rate and speedup ratio in a variety of operating environments. Considering the above shortcomings, this paper proposes an adaptive scheduling framework for machine learning based on a heterogeneous distributed system and mobile edge computing system for machine learning model optimization. The framework detects and analyzes the dynamic changes of resources in the distributed system and mobile edge computing system through the resource detection system; then, the task scheduling system adaptively modifies the environmental parameters and schedules calculations. Relevant experiments conducted with the public data set show that the robustness and scalability of the framework are significantly better than the traditional distributed machine learning framework under the premise of ensuring high convergence rate.

Off-Line Time Aware Scheduling of Bag-of-Tasks on Heterogeneous Distributed System

The resource allocation for bag-of-tasks in the heterogeneous distributed system is to distribute the tasks to proper processors such that the makespan is minimized. It is a well-known NP-hard problem, and is even more complex and challenging when the processors have off-line time. To tackle this challenging problem, first, we set up a mathematical model for this problem which minimizes the makespan of the bag-of-tasks with the off-line time segment of the processors. Second, to solve the model efficiently, we propose two new algorithms: a new scheduling algorithm referred to as sorting-allocation-pulling scheduling algorithm which first allocate the tasks to available time segment on proper processors and then pulls them to the formerly available time segment for the sake of minimizing the makespan, and an effective genetic algorithm with a novel local search operator and a well-designed modify operator. Finally, the numerical simulation experiments are conducted, and the two proposed algorithms are compared. The experimental results indicate the effectiveness of the proposed model and algorithms.

Task allocation based modified bacterial foraging algorithm for maximising reliability of a heterogeneous distributed system

modified bacterial foraging algorithm for allocation and scheduling tasks on processors and links with high reliability is proposed (MBFA). MBFA ensure high reliability without redundancy of processors or links by using an efficient and simple structure. MBFA is improved by adding an efficient model to deal with the weak solution (i.e., allocation with low reliability). Also a simulated annealing technique is integrated into MBFA in order to strengthen the search of best solution. The efficiency and effectiveness of MBFA are tested using random generated problems. MBFA is tested over the published approaches HPSO and GAA using density parameter (DS), number of nodes (N) and the communication to computation ratio parameter (CCR). The results show the superiority of MBFA over the published approaches in all test cases.

Towards Ubiquitous Intelligent Computing: Heterogeneous Distributed Deep Neural Networks

For the pursuit of ubiquitous computing, distributed computing systems containing the cloud, edge devices, and Internet-of-Things devices are highly demanded. However, existing distributed frameworks do not tailor for the fast development of Deep Neural Network (DNN), which is the key technique behind many intelligent applications nowadays. Based on prior exploration on distributed deep neural networks (DDNN), we propose Heterogeneous Distributed Deep Neural Network (HDDNN) over the distributed hierarchy, targeting at ubiquitous intelligent computing. While being able to support basic functionalities of DNNs, our framework is optimized for various types of heterogeneity, including heterogeneous computing nodes, heterogeneous neural networks, and heterogeneous system tasks. Besides, our framework features parallel computing, privacy protection and robustness, with other consideration for the combination of heterogeneous distributed system and DNN. Extensive experiments demonstrate that our framework is capable of utilizing hierarchical distributed system better for DNN and tailoring DNN for real-world distributed system properly, which is with low response time, high performance, and better user experience.

Simulation modelling of the heterogeneous distributed information processing systems

This study reviews simulation modelling of the heterogeneous distributed information processing systems. The relevance of simulation modelling for the considered class of systems is substantiated. The purpose of the study is outlined and solutions of the tasks required for its achievement are described. Based on the structural modelling carried-out earlier, main types of the model elements and the relations between them are allocated. The network structure generation algorithm is presented. The behaviour and the structure of model elements are described. Results of the modelling are displayed. At the conclusion, the possibilities for further development of the model are considered, and the conclusion on its practical application for real systems is made.

The impact of workload variability on the energy efficiency of large-scale heterogeneous distributed systems

Previous studies have shown that the workload variability has a serious impact on the performance of large-scale distributed architectures, since it may cause significant fluctuations in service demands. Energy efficiency is one of the aspects of such platforms that are of paramount importance and therefore it is imperative to investigate how it may also be affected by this factor. Towards this direction, in this paper we investigate via simulation the impact of workload variability, in terms of computational volume and interarrival times, on the energy consumption of a large-scale heterogeneous distributed system. The workload consists of real-time bag-of-tasks jobs that arrive dynamically at the system. The execution rate and power consumption characteristics of the processors are modeled after real-world processors, according to the Standard Performance Evaluation Corporation (SPEC) Power benchmark. Four heuristics are employed for the scheduling of the workload, two commonly used baseline policies and two proposed energy-aware heuristics. The simulation results reveal that the workload variability has a significant impact on the energy consumption of the system and that the severity of the impact depends on the employed scheduling technique.

Multicriterion problem of allocation of resources in the heterogeneous distributed information processing systems

This study reviews the problem of allocation of resources in the heterogeneous distributed information processing systems, which may be formalized in the form of a multicriterion multi-index problem with the linear constraints of the transport type. The algorithms for solution of this problem suggest a search for the entire set of Pareto-optimal solutions. For some classes of hierarchical systems, it is possible to significantly speed up the procedure of verification of a system of linear algebraic inequalities for consistency due to the reducibility of them to the stream models or the application of other solution schemes (for strongly connected structures) that take into account the specifics of the hierarchies under consideration.

Service Request Scheduling based on Quantification Principle using Conjoint Analysis and Z-score in Cloud

Service request scheduling has a major impact on the performance of the service processing design in a large-scale distributed computing environment like cloud systems. It is desirable to have a service request scheduling principle that evenly distributes the workload among the servers, according to their capacities. The capacities of the servers are termed high or low relative to one another. Therefore, there is a need to quantify the server capacity to overcome this subjective assessment. Subsequently, a method to split and distribute the service requests based on this quantified server capacity is also needed. The novelty of this research paper is to address these requirements by devising a service request scheduling principle for a heterogeneous distributed system using appropriate statistical methods, namely Conjoint analysis and Z-score. Suitable experiments were conducted and the experimental results show considerable improvement in the performance of the designed service request scheduling principle compared to a few other existing principles. Areas of further improvement have also been identified and presented.

On the Dependability of Highly Heterogeneous and Open Distributed Systems

This paper introduces an architecture of distributed systems that facilitates the implementation of a substantial range of dependable system properties, i.e., properties that span an entire system, or a set of components dispersed throughout it. This architecture, called GDS, for governed distributed system, governs the system by controlling the flow of messages between its actors, independently of the internals of the interacting actors. This governance is done via an enforced collection of interaction laws organized into a modular and conflict free hierarchical ensemble. This ensemble of laws is sensitive to the history of interaction; and it is enforced in a decentralized manner, and is thus scalable. The dependable system properties that can be implemented under GDS can have the following beneficial consequences, among others: a) the ability to establish regularities over the system, rendering it more coherent, and easier to reason about; b) the ability to provide a degree of trust among the disparate actor of the system; and c) the ability to ensure compliance with interaction protocols that are essential for distributed computing. Consequently, the GDS architecture can have a significant impact on the following important system qualities: security, fault tolerance, auditability, and manageability.

Energy-Efficient Scheduling Algorithms for Real-Time Parallel Applications on Heterogeneous Distributed Embedded Systems

Energy consumption minimization is one of the primary design requirements for heterogeneous distributed systems. State-of-the-art algorithms are used to study the problem of minimizing the energy consumption of a real-time parallel application with precedence constrained tasks on a heterogeneous distributed system by introducing the concept of latest finish time (LFT) to reclaim the slack time based on the dynamic voltage and frequency scaling (DVFS) energy-efficient design optimization technique. However, the use of DVFS technique alone is insufficient, and the energy consumption reduction is limited because scaling down the frequency is restricted in practice. Furthermore, these studies merely minimize energy consumption through a local energy-efficient scheduling algorithm, such as reducing the energy consumption for each task on the fixed processor, rather than a global energy-efficient scheduling algorithm, such as reducing the energy consumption for each task on different processors. This study solves the problem of minimizing the energy consumption of a real-time parallel application on heterogeneous distributed systems by using the combined non-DVFS and global DVFS-enabled energy-efficient scheduling algorithms. The non-DVFS energy-efficient scheduling (NDES) algorithm is solved by introducing the concept of deadline slacks to reduce the energy consumption while satisfying the deadline constraint. The global DVFS-enabled energy-efficient scheduling (GDES) algorithm is presented by moving the tasks to the processor slacks that generate minimum dynamic energy consumptions. Results of the experiments show that the combined NDES&GDES algorithm can save up to 36.25-55.65 percent of energy compared with state-of-the-art counterparts under different scales, parallelism, and heterogeneity degrees of parallel applications.

Effective Load Balance Scheduling Schemes for Heterogeneous Distributed System

Importance of distributed systems for distributing the workload on the processors is globally accepted. It is an agreed fact that divides and conquers is the effective strategy for load balancing problems. In today’s time, load balancing is the major issue for scheduling algorithm such as in Parallel and Distributed Systems including Grid and Cloud Computing and many more. Load Balancing is the phenomena of spreading or distributing the workload among the processors so that all processors keep busy for all the time, in order to prevent ideal time of processors. In this work, presents a load balancing algorithm for heterogeneous distributed system (HeDS) with aim of minimizing the load imbalance factor (LIF). The proposed algorithm is using optimization techniques such as Max-Max and Min-Max strategy and applied on Folded Crossed Cube (FCC) network. Makespan, speedup and average resource utilization are also evaluated for performance matrices. The experimental results of the proposed algorithms have showed better in comparison to with previous work under various test conditions.

Software and experience with managing workflows for the computing operation of the CMS experiment

We present a system deployed in the summer of 2015 for the automatic assignment of production and reprocessing workflows for simulation and detector data in the frame of the Computing Operation of the CMS experiment at the CERN LHC. Processing requests involves a number of steps in the daily operation, including transferring input datasets where relevant and monitoring them, assigning work to computing resources available on the CMS grid, and delivering the output to the Physics groups. Automation is critical above a certain number of requests to be handled, especially in the view of using more efficiently computing resources and reducing latency. An effort to automatize the necessary steps for production and reprocessing recently started and a new system to handle workflows has been developed. The state-machine system described consists in a set of modules whose key feature is the automatic placement of input datasets, balancing the load across multiple sites. By reducing the operation overhead, these agents enable the utilization of more than double the amount of resources with robust storage system. Additional functionality were added after months of successful operation to further balance the load on the computing system using remote read and additional resources. This system contributed to reducing the delivery time of datasets, a crucial aspect to the analysis of CMS data. We report on lessons learned from operation towards increased efficiency in using a largely heterogeneous distributed system of computing, storage and network elements.

A hybrid Big Bang-Big Crunch algorithm for energy optimization on heterogeneous distributed system

A hybrid algorithm is presented for global optimization of the energy consumption rather than makespan of the system. The cost profile is optimized by extending the execution time of the tasks on dynamic voltage scalable processing elements in embedded environment with meeting the execution constraints. The modified Big Bang Big Crunch (BBBC) method improves the scheduling efficiency of the tasks by simulating one of the theories of the evolution of the universe. BBBC algorithm generates disorder data points inspired by energy dissipation procedure of the Big Bang phase, and moves those data points to a single representative data point inspired by a center of mass cost approach in the Big Crunch phase. The Logistic and Sinusoidal chaotic maps are investigated and utilized to improve the movement step of the BBBC algorithm. The proposed hybrid algorithm is tested on several benchmark data sets and its performance is compared with those of Ant Colony Optimization and HEFT strategies. The simulation experiment shows that the presented evolutionary optimization algorithm is robust and suitable for energy saving.

Different aspects of workflow scheduling in large-scale distributed systems

As large-scale distributed systems gain momentum, the scheduling of workflow applications with multiple requirements in such computing platforms has become a crucial area of research. In this paper, we investigate the workflow scheduling problem in large-scale distributed systems, from the Quality of Service (QoS) and data locality perspectives. We present a scheduling approach, considering two models of synchronization for the tasks in a workflow application: (a) communication through the network and (b) communication through temporary files. Specifically, we investigate via simulation the performance of a heterogeneous distributed system, where multiple soft real-time workflow applications arrive dynamically. The applications are scheduled under various tardiness bounds, taking into account the communication cost in the first case study and the I/O cost and data locality in the second. The simulation results provide useful insights into the impact of tardiness bound and data locality on the system performance.

Integrating Dynamic-TDMA Communication Channels into COTS Ethernet Networks

Real-time Ethernet (RTE) is widely recognized for its potential to provide a unified communication backbone for next-generation heterogeneous distributed systems. However, most of the existing research in RTE technologies has traditionally focused on formal models and theoretical analyzes of timing properties, usually omitting the associated implementation challenges for testing them in practice. This gap between theory and practice prevents experimental validation of the claimed properties, which in turn hinders the pace of innovation and adoption of the technology in industrial settings. This paper aims at narrowing the theory-practice gap by characterizing a comprehensive open-source RTE framework that explores emerging challenges in real-time networking, including the provision of ultra-low latency and jitter, dynamic bandwidth management, and segmentation within large networks. This work integrates research on formal abstractions for dynamic time-division multiple access arbitration and technological insights from modern hardware infrastructure, and uses a representative distributed video processing application to provide reproducible evidence of the achieved properties in multihop Ethernet settings. By leveraging readily available technology and an open-source design, the proposed framework facilitates further exploration and experimental validation of properties that are beyond the scope of current commercial technologies, encouraging evidence-based discussions to accelerate development and adoption of new standards for next-generation industrial networks.

Multi-objective mapping of full-mission simulators on heterogeneous distributed multi-processor systems

Full-mission simulators (FMSs) are considered the most critical simulation tool belonging to the flight simulator family. FMSs include a faithful reproduction of fighter aircraft. They are used by armed forces for design, training, and investigation purposes. Due to the criticality of their timing constraints and the high computation cost of the whole simulation, FMSs need to run in a high-performance computing system. Heterogeneous distributed systems are among the leading computing platforms and can guarantee a significant increase in performance by providing a large number of parallel powerful execution resources. One of the most persistent challenges raised by these platforms is the difficulty of finding an optimal mapping of n tasks on m processing elements. The mapping problem is considered a variant of the quadratic assignment problem, in which an exhaustive search cannot be performed. The mapping problem is an NP-hard problem and solving it requires the use of meta-heuristics, and it becomes more challenging when one has to optimize more than one objective with respect to the timing constraints. Multi-objective evolutionary algorithms have proven their efficiency when tackling this problem. Most of the existent works deal with the task mapping by considering either a single objective or homogeneous architectures. Therefore, the main contribution of this paper is a framework based on the model-driven design paradigm allowing us to map a set of intercommunicating real-time tasks making up the FMS model onto the heterogeneous distributed multi-processor system model. We propose a multi-objective approach based on the well-known optimization algorithm “Non-dominated Sorting Genetic Algorithm-II” satisfying the tight timing constraints of the simulation and minimizing makespan, communication cost, and memory consumption simultaneously.

Achieving high performance and privacy-preserving query over encrypted multidimensional big metering data

With the proliferation of smart grids, traditional utilities are struggling to handle the increasing amount of metering data. Outsourcing the metering data to heterogeneous distributed systems has the potential to provide efficient data access and processing. In an untrusted heterogeneous distributed system environment, employing data encryption prior to outsourcing can be an effective way to preserve user privacy. However, how to efficiently query encrypted multidimensional metering data stored in an untrusted heterogeneous distributed system environment remains a research challenge. In this paper, we propose a high performance and privacy-preserving query (P2Q) scheme over encrypted multidimensional big metering data to address this challenge. In the proposed scheme, encrypted metering data are stored in the server of an untrusted heterogeneous distributed system environment. A Locality Sensitive Hashing (LSH) based similarity search approach is then used to realize the similarity query. To demonstrate utility of the proposed LSH-based search approach, we implement a prototype using MapReduce for the Hadoop distributed environment. More specifically, for a given query, the proxy server will return K top similar data object identifiers. An enhanced Ciphertext-Policy Attribute-based Encryption (CP-ABE) policy is then used to control access to the search results. Therefore, only the requester with an authorized query attribute can obtain the correct secret keys to retrieve the metering data. We then prove that the P2Q scheme achieves data confidentiality and preserves the data owner’s privacy in a semi-trusted cloud. In addition, our evaluations demonstrate that the P2Q scheme can significantly reduce response time and provide high search efficiency without compromising on search quality (i.e. suitable for multidimensional big data search in heterogeneous distributed system, such as cloud storage system).

Efficient and reliable service selection for heterogeneous distributed software systems

The service-oriented paradigm is emerging as a new approach to heterogeneous distributed software systems composed of services accessed locally or remotely by middleware technology. How to select the optimal composited service from a set of functionally equivalent services with different quality of service (QoS) attributes has become an active focus of research in the service community. However, existing middleware solutions or approaches are inefficient as they search all solution spaces. More importantly, they inherently neglect QoS uncertainty owing to the dynamic network environment. In this paper, based on a service composition middleware framework, we propose an efficient and reliable service selection approach that attempts to select the best reliable composited service by filtering low-reliability services through the computation of QoS uncertainty. The approach first employs information theory and probability theory to abandon high-QoS-uncertainty services and downsize the solution space. A reliability fitness function is then designed to select the best reliable service for composited services. We experimented with real-world and synthetic datasets and compared our approach with other approaches. Our results show that our approach is not only fast, but also finds more reliable composited services.

A Survey of Static Scheduling Algorithm for Distributed Computing System

static scheduling algorithms are widely used to evaluate the performance of distributed computing system. In such systems, purpose of scheduling algorithm is to allocate tasks to available processor so as to efficiently utilize this processor and to reduce the makespan, total computational cost and various other such factors, with the motive of achieving optimal solution. Scheduling algorithms are classified into two broad categories i.e., static scheduling algorithms and dynamic scheduling algorithms. In this paper we are discussing various static scheduling algorithm and numerous problems in various levels of the homogeneous and heterogeneous distributed systems. Also we are comparing these algorithms on the basis of various factors such as speedup, time complexity, scheduling length ratio, normalized scheduling length and so on.

ParaDIME: Parallel Distributed Infrastructure for Minimization of Energy for data centers

Dramatic environmental and economic impact of the ever increasing power and energy consumption of modern computing devices in data centers is now a critical challenge. On the one hand, designers use technology scaling as one of the methods to face the phenomenon called dark silicon (only segments of a chip function concurrently due to power restrictions). On the other hand, designers use extreme-scale systems such as teradevices to meet the performance needs of their applications which in turn increases the power consumption of the platform. In order to overcome these challenges, we need novel computing paradigms that address energy efficiency. One of the promising solutions is to incorporate parallel distributed methodologies at different abstraction levels.The FP7 project ParaDIME focuses on this objective to provide different distributed methodologies (software–hardware techniques) at different abstraction levels to attack the power-wall problem. In particular, the ParaDIME framework will utilize: circuit and architecture operation below safe voltage limits for drastic energy savings, specialized energy-aware computing accelerators, heterogeneous computing, energy-aware runtime, approximate computing and power-aware message passing. The major outcome of the project will be a noval processor architecture for a heterogeneous distributed system that utilizes future device characteristics, runtime and programming model for drastic energy savings of data centers. Wherever possible, ParaDIME will adopt multidisciplinary techniques, such as hardware support for message passing, runtime energy optimization utilizing new hardware energy performance counters, use of accelerators for error recovery from sub-safe voltage operation, and approximate computing through annotated code. Furthermore, we will establish and investigate the theoretical limits of energy savings at the device, circuit, architecture, runtime and programming model levels of the computing stack, as well as quantify the actual energy savings achieved by the ParaDIME approach for the complete computing stack with the real environment.