Enterprise Desktop Grid Research Articles

Dynamical systems analysis using many-task interactive cloud computing

The article states an approach for developing a distributed application architecture for dynamic systems analysis in a hybrid cloud. The problem that was posed as an example is based on the calculation of the Lyapunov characteristic exponents for trajectories of dynamical systems in phase space. Parallelization of the computational process is achieved by supplying the list of running tasks for computing Lyapunov characteristic exponents, depending on the result of finished tasks during calculations. We have implemented a distributed application to make this experiment automated. The architecture of application elements is established on the principles of the enterprise desktop grid and cloud systems. A vital detail of the application architecture is the deployment process based on a cloud hybrid computing environment using the Jupyter server running on the DigitalOcean cloud platform and the Everest cloud platform simultaneously with the Maple package.

Task Scheduling in Desktop Grids: Open Problems

AbstractWe survey the areas of Desktop Grid task scheduling that seem to be insufficiently studied so far and are promising for efficiency, reliability, and quality of Desktop Grid computing. These topics include optimal task grouping, “needle in a haystack” paradigm, game-theoretical scheduling, domain-imposed approaches, special optimization of the final stage of the batch computation, and Enterprise Desktop Grids.

Video conference in the fog: an economical approach based on enterprise desktop grid

There exist two classical and well-understood approaches to video-processing tasks (such as mixing or trans-coding) for videoconferencing. The first one is using a centralized multipoint control unit (MCU), hardware- or software-based, deployed on-premises or in the cloud. In the second approach, the video-processing tasks are directly handled in endpoints (i.e., equipment such as PCs, laptops, and tablets that are involved in the video session). Performance is then restricted by device characteristics, especially in the case of mobile devices. In this paper, we propose a third alternative approach. It has been shown that there exist significant computational resources in user equipment deployed in enterprises, which are under-utilized most of the time. In this paper, we propose a system, which distributes real-time video-processing tasks on these available resources. A dedicated multi-attribute decision-making (MADM) method is designed in order to take into account the variety of attributes impacting Quality of Experience. We enumerate a comprehensive list of events, which may cause distribution or redistribution of video-processing activities and provide simple algorithms to tackle all these cases. We then test the MADM algorithm by means of simulations in order to study the impact of the main critical parameters.

Высокопроизводительный виртуальный скрининг в Enterprise Desktop Grid на базе BOINC

In the paper we present the experience of developing and using an Enterprise Desktop Grid infrastructure based on BOINC software platform and designed for virtual screening using open-source software and open databases of chemical compounds models. According to the idea of Enterprise Desktop Grid, a virtual single computational network integrates non-specialized computers within an organization or group of users that are directly interested in solving an applied problem. We describe and evaluate the performance of the infrastructure created withing a joint research project between the L¨ubeck Institute of Experimental Dermatology (former Clinic of dermatology, allergology and venerology) of the University of L¨ubeck and the Institute of Applied Mathematical Research of the Karelian Research Centre of the Russian Academy of Sciences. We summarize the project and briefly overview the results of the virtual screening.

State-based predictions with self-correction on Enterprise Desktop Grid environments

The abundant computing resources in current organizations provide new opportunities for executing parallel scientific applications and using resources. The Enterprise Desktop Grid Computing (EDGC) paradigm addresses the potential for harvesting the idle computing resources of an organization’s desktop PCs to support the execution of the company’s large-scale applications. In these environments, the accuracy of response-time predictions is essential for effective metascheduling that maximizes resource usage without harming the performance of the parallel and local applications. However, this accuracy is a major challenge due to the heterogeneity and non-dedicated nature of EDGC resources. In this paper, two new prediction techniques are presented based on the state of resources. A thorough analysis by linear regression demonstrated that the proposed techniques capture the real behavior of the parallel applications better than other common techniques in the literature. Moreover, it is possible to reduce deviations with a proper modeling of prediction errors, and thus, a Self-adjustable Correction method (SAC) for detecting and correcting the prediction deviations was proposed with the ability to adapt to the changes in load conditions. An extensive evaluation in a real environment was conducted to validate the SAC method. The results show that the use of SAC increases the accuracy of response-time predictions by 35%. The cost of predictions with self-correction and its accuracy in a real environment was analyzed using a combination of the proposed techniques. The results demonstrate that the cost of predictions is negligible and the combined use of the prediction techniques is preferable.

Scalable data management for map-reduce-based data-intensive applications: a view for cloud and hybrid infrastructures

As map-reduce emerges as a leading programming paradigm for data-intensive computing, today’s frameworks which support it still have substantial shortcomings that limit its potential scalability. In this paper, we discuss several directions where there is room for such progress: they concern storage efficiency under massive data access concurrency, scheduling, volatility and fault-tolerance. We place our discussion in the perspective of the current evolution towards an increasing integration of large-scale distributed platforms (clouds, cloud federations, enterprise desktop grids, etc.). We propose an approach which aims to overcome the current limitations of existing map-reduce frameworks, in order to achieve scalable, concurrency-optimised, fault-tolerant map-reduce data processing on hybrid infrastructures. This approach will be evaluated with real-life bio-informatics applications on existing Nimbus-powered cloud testbeds interconnected with desktop grids.

Scheduling Task Parallel Applications for Rapid Turnaround on Enterprise Desktop Grids

Desktop Grids are popular platforms for high throughput applications, but due to their inherent resource volatility it is difficult to exploit them for applications that require rapid turnaround. Efficient desktop Grid execution of short-lived applications is an attractive proposition and we claim that it is achievable via intelligent resource selection. We propose three general techniques for resource selection: resource prioritization, resource exclusion, and task duplication. We use these techniques to instantiate several scheduling heuristics. We evaluate these heuristics through trace-driven simulations of four representative desktop Grid configurations. We find that ranking desktop resources according to their clock rates, without taking into account their availability history, is surprisingly effective in practice. Our main result is that a heuristic that uses the appropriate combination of resource prioritization, resource exclusion, and task replication can achieve performance within a factor of 1.7 of optimal in practice.

Characterizing resource availability in enterprise desktop grids

Desktop grids, which use the idle cycles of many desktop PCs, are one of the largest distributed systems in the world. Despite the popularity and success of many desktop grid projects, the heterogeneity and volatility of hosts within desktop grids have been poorly understood. Yet, resource characterization is essential for accurate simulation and modelling of such platforms. In this paper, we present application-level traces of four real desktop grids that can be used for simulation and modelling purposes. In addition, we describe aggregate and per host statistics that reflect the heterogeneity and volatility of desktop grid resources. Finally, we apply our characterization to develop a performance model for desktop grid applications for various task granularities, and then use a cluster equivalence metric to quantify the utility of the desktop grid relative to that of a dedicated cluster for task-parallel applications.

Entropia: architecture and performance of an enterprise desktop grid system

The exploitation of idle cycles on pervasive desktop PC systems offers the opportunity to increase the available computing power by orders of magnitude (10×–1000×). However, for desktop PC distributed computing to be widely accepted within the enterprise, the systems must achieve high levels of efficiency, robustness, security, scalability, manageability, unobtrusiveness, and openness/ease of application integration. We describe the Entropia distributed computing system as a case study, detailing its internal architecture and philosophy in attacking these key problems. Key aspects of the Entropia system include the use of: (1) binary sandboxing technology for security and unobtrusiveness, (2) a layered architecture for efficiency, robustness, scalability and manageability, and (3) an open integration model to allow applications from many sources to be incorporated. Typical applications for the Entropia System includes molecular docking, sequence analysis, chemical structure modeling, and risk management. The applications come from a diverse set of domains including virtual screening for drug discovery, genomics for drug targeting, material property prediction, and portfolio management. In all cases, these applications scale to many thousands of nodes and have no dependences between tasks. We present representative performance results from several applications that illustrate the high performance, linear scaling, and overall capability presented by the Entropia system.