Opportunistic Resource Research Articles

Computation offloading to edge cloud and dynamically resource-sharing collaborators in Internet of Things

With the diversity of the communication technology and the heterogeneity of the computation resources at network edge, both the edge cloud and peer devices (collaborators) can be scavenged to provide computation resources for the resource-limited Internet-of-Things (IoT) devices. In this paper, a novel cooperative computing paradigm is proposed, in which the computation resources of IoT device, opportunistically idle collaborators and dedicated edge cloud are fully exploited. Computation/offloading assistance is provided by collaborators at idle/busy states, respectively. Considering the channel randomness and opportunistic computation resource share of collaborators, we study the stochastic offloading control for an IoT device, regarding how much computation load is processed locally, offloaded to the edge cloud and a collaborator. The problem is formulated into a finite horizon Markov decision problem with the objective of minimizing the expected total energy consumption of the IoT device and the collaborator, subject to satisfying the hard computation deadline constraint. Optimal offloading policy is derived based on the stochastic optimization theory, which demonstrates that the energy consumption can be reduced by a proportional factor through the cooperative computing. More energy saving is achieved with better wireless channel condition or higher computation energy efficiency of collaborators. Simulation results validate the optimality of the proposed policy and the efficiency of the cooperative computing between end devices and edge cloud, compared to several other offloading schemes.

Characterization of sparse WLAN data traffic in opportunistic indoor environments as a prior for coexistence scenarios of modern wireless technologies

CR enabled radio environment provides the seamless operating framework to cope with the requirements of next generation wireless systems like higher data rates, massive machine time communication and interference free coexistence scenarios of wireless technologies. Traffic characterization studies assist to optimize coexistence framework by providing necessary information about the usage patterns of wireless services in observed radio bands. Prior knowledge about the observed sparse wireless local area network (WLAN) data traffic is a key for opportunistic radio resource allocation and utilization in the given coexistence scenarios of WLAN either with cellular systems (including 5G-NR/4G Long Term Evolution (LTE)) or with low power wireless systems (including Bluetooth and ZigBee). Parametric methodology is adapted to model the sparse WLAN data traffic observed in 2.4 GHz Industrial, Scientific and Medical (ISM) band as spectral and temporal activity. The multivariate Gaussian mixture model (MGMM) is proposed in both scenarios where either dependency is considered between the observed WLAN data traffic or assuming that the observed traffic is independent and identically distributed (i.i.d). It is to be validated that in order to have an efficient characterization of sparse WLAN data traffic, the dependency (correlation) between neighbored frequency subbands must be considered. By considering the dependency either between neighbored frequency subbands or neighbored time domain signals actually help to characterize the sparse WLAN data traffic in more realistic way which could not be possible by assuming them i.i.d. Such statistics to characterize sparse data traffic really help as user activity for efficient allocation and utilization of radio resources in CR enabled coexistence scenarios of wireless technologies where WLAN acitivity is considered nominal either as secondary user or classify it grey spectrum as primary user.

A Blockchain-Based Computing Architecture for Mobile Ad Hoc Cloud

Mobile ad-hoc cloud can exploit the computing resources (e.g., smartphones, vehicles, and unmanned systems) scattered in the mobile environment to form a self-organized ad-hoc local resource pool for providing opportunistic computing services. However, the highly dynamic and distributed characteristics of the mobile ad-hoc network environment bring great challenges in privacy and security in such opportunistic resource sharing. In this article, we first discuss the attractive features of blockchain for providing such resource sharing services in the mobile ad-hoc network environment in a secure and trustful way and then discuss the problems caused when using existing consensus protocols in such an environment. We accordingly devise a blockchain based trustful mobile ad-hoc cloud architecture, AdChain cloud. We describe the functions at different layers in this architecture, including the network layer, blockchain layer, and smart contract layer. To adapt to the high dynamics of the mobile ad-hoc network environment, we design a stability-aware consensus protocol at the blockchain layer. Simulation results show that our solution can achieve improved performance as compared with existing work.

HEP Analyses on Dynamically Allocated Opportunistic Computing Resources

The current experiments in high energy physics (HEP) have a huge data rate. To convert the measured data, an enormous number of computing resources is needed and will further increase with upgraded and newer experiments. To fulfill the ever-growing demand the allocation of additional, potentially only temporary available non-HEP dedicated resources is important. These so-called opportunistic resources cannot only be used for analyses in general but are also well-suited to cover the typical unpredictable peak demands for computing resources. For both use cases, the temporary availability of the opportunistic resources requires a dynamic allocation, integration, and management, while their heterogeneity requires optimization to maintain high resource utilization by allocating best matching resources. To find the best matching resources which should be allocated is challenging due to the unpredictable submission behavior as well as an ever-changing mixture of workflows with different requirements.Instead of predicting the best matching resource, we base our decisions on the utilization of resources. For this reason, we are developing the resource manager TARDIS (Transparent Adaptive Resource Dynamic Integration System) which manages and dynamically requests or releases resources. The decision of how many resources TARDIS has to request is implemented in COBalD (COBald - The Opportunistic Balancing Daemon) to ensure further allocation of well-used resources while reducing the amount of insufficiently used ones. TARDIS allocates and manages resources from various resource providers such as HPC centers or commercial and public clouds while ensuring a dynamic allocation and efficient utilization of these heterogeneous opportunistic resources.Furthermore, TARDIS integrates the allocated opportunistic resources into one overlay batch system which provides a single point of entry for all users. In order to provide the dedicated HEP software environment, we use virtualization and container technologies.In this contribution, we give an overview of the dynamic integration of opportunistic resources via TARDIS/COBalD in our HEP institute as well as how user analyses benefit from these additional resources.

Federation of compute resources available to the German CMS community

The German CMS community (DCMS) as a whole can benefit from the various compute resources, available to its different institutes. While Grid-enabled and National Analysis Facility resources are usually shared within the community, local and recently enabled opportunistic resources like HPC centers and cloud resources are not. Furthermore, there is no shared submission infrastructure available.Via HTCondor’s [1] mechanisms to connect resource pools, several remote pools can be connected transparently to the users and therefore used more efficiently by a multitude of user groups. In addition to the statically provisioned resources, also dynamically allocated resources from external cloud providers as well as HPC centers can be integrated. However, the usage of such dynamically allocated resources gives rise to additional complexity. Constraints on access policies of the resources, as well as workflow necessities have to be taken care of. To maintain a well-defined and reliable runtime environment on each resource, virtualization and containerization technologies such as virtual machines, Docker, and Singularity, are used.

Boosting Performance of Data-intensive Analysis Workflows with Distributed Coordinated Caching

Data-intensive end-user analyses in high energy physics require high data throughput to reach short turnaround cycles. This leads to enormous challenges for storage and network infrastructure, especially when facing the tremendously increasing amount of data to be processed during High-Luminosity LHC runs. Including opportunistic resources with volatile storage systems into the traditional HEP computing facilities makes this situation more complex.Bringing data close to the computing units is a promising approach to solve throughput limitations and improve the overall performance. We focus on coordinated distributed caching by coordinating workows to the most suitable hosts in terms of cached files. This allows optimizing overall processing efficiency of data-intensive workows and efficiently use limited cache volume by reducing replication of data on distributed caches.We developed a NaviX coordination service at KIT that realizes coordinated distributed caching using XRootD cache proxy server infrastructure and HTCondor batch system. In this paper, we present the experience gained in operating coordinated distributed caches on cloud and HPC resources. Furthermore, we show benchmarks of a dedicated high throughput cluster, the Throughput-Optimized Analysis-System (TOpAS), which is based on the above-mentioned concept.

Opportunistic scheduling and resources consolidation system based on a new economic model

This paper presents a new opportunistic scheduling and resource consolidation system based on an economic model related to different service level agreements (SLAs) classes. The goal is to address the problem of companies that manage a private infrastructure of machines, i.e., a cloud platform and would like to optimize the scheduling of several requests submitted online by users. For the sake of simplicity of the presentation, the proposed economic model has two SLAs classes (qualitative and quantitative) with three Quality of Service for each SLA class (Premium, Advanced and Best effort). The consequence of this choice as well as the need to serve requests as they come have an impact on the algorithmic ways to consolidate an infrastructure. Indeed, our system proposes a new allocation heuristic that adapts the number of active machines in the cloud according to the global resources usage of all machines inside the infrastructure. This heuristic can be examined as a consolidation heuristic, based on the idea that the system can make reasonable choices, based on the SLAs, for the placement and the allocation of resources for each request. Experimentation with our system is conducted on Prezi (Web workload) and Google Cloud Data (HPC-oriented workload) traces, and they demonstrate the potential of our approach under different scenarios. From a methodological point of view, we propose a general framework which is limited in scope, for the sake of simplicity in reading the paper, with a small number of SLAs, but the idea can be extended to many more SLAs and performance metrics. In this way, the user or the provider operating the cloud have more latitude, thanks to our multi-criteria approach, to control the workload without a sacrifice on performance.

Moving Towards Universal Health Coverage in Haiti

ABSTRACT Haiti announced in 2018 its aim to achieve universal health coverage. In this paper, we discuss what this objective means for the country and what next steps should be taken. To contextualize the notion, we framed Haiti en route to the 2030 goal and analyzed qualitatively the status quo in terms of geographic, financial, and service access. For each dimension, we focused on the context, the government’s policies and political agendas, their implementation progress, and key influential factors. Our analysis found little progress and numerous challenges. Geographic access was limited due principally to the insufficient number of facilities, difficulties in reaching health facilities, and local customs. Financial coverage was low because of the government’s insufficient budgets, inefficient budget allocation, and ineffective management. Service access also had room for significant improvement for a lack of basic infrastructure and resources, gaps between the essential service package guidelines, health professionals’ skills, and the needs, as well as deficiencies in people-centered care. These factors affected not only health service coverage but also its quality. We found that the root causes of these issues were composed of unstable financing mechanisms, opportunistic resource allocation, and ineffective management control systems. We suggest that to overcome these issues and achieve universal health coverage with decent service quality, Haiti’s health system needs to be reformed by implementing strategic financing, decentralized management systems, and community engagement in primary health care.

High-Throughput Cloud Computing with the Cloudscheduler VM Provisioning Service

We describe a high-throughput computing system for running jobs on public and private computing clouds using the HTCondor job scheduler and the cloudscheduler VM provisioning service. The distributed cloud computing system is designed to simultaneously use dedicated and opportunistic cloud resources at local and remote locations. It has been used for large-scale production particle physics workloads for many years using thousands of cores on three continents. A decade after its initial design and implementation, cloudscheduler has been modernized to take advantage of new software designs, improved operating system capabilities and support packages. The updated cloudscheduler is more resilient and scalable, with expanded capabilities. We present an overview of the original design and then describe the new version of the distributed compute cloud system. We conclude with a review of the current status and future plans.

Effective Dynamic Integration and Utilization of Heterogenous Compute Resources

Increased operational effectiveness and the dynamic integration of only temporarily available compute resources (opportunistic resources) becomes more and more important in the next decade, due to the scarcity of resources for future high energy physics experiments as well as the desired integration of cloud and high performance computing resources. This results in a more heterogenous compute environment, which gives rise to huge challenges for the computing operation teams of the experiments. At the Karlsruhe Institute of Technology (KIT) we design solutions to tackle these challenges. In order to ensure an efficient utilization of opportunistic resources and unified access to the entire infrastructure, we developed the Transparent Adaptive Resource Dynamic Integration System (TARDIS). A scalable multi-agent resource manager providing interfaces to provision as well as dynamically and transparently integrate resources of various providers into one common overlay batch system. Operational effectiveness is guaranteed by relying on COBalD – the Opportunistic Balancing Daemon and its simple approach of taking into account the utilization and allocation of the different resource types, in order to run the individual workflows on the best-suited resource respectively. In this contribution we will present the current status of integrating various HPC centers and cloud providers into the compute infrastructure at the Karlsruhe Institute of Technology as well as our experiences gained in a production environment.

Lightweight dynamic integration of opportunistic resources

To satisfy future computing demands of the Worldwide LHC Computing Grid (WLCG), opportunistic usage of third-party resources is a promising approach. While the means to make such resources compatible with WLCG requirements are largely satisfied by virtual machines and containers technologies, strategies to acquire and disband many resources from many providers are still a focus of current research. Existing meta-schedulers that manage resources in the WLCG are hitting the limits of their design when tasked to manage heterogeneous resources from many diverse resource providers.To provide opportunistic resources to the WLCG as part of a regular WLCG site, we propose a new meta-scheduling approach suitable for opportunistic, heterogeneous resource provisioning. Instead of anticipating future resource requirements, our approach observes resource usage and promotes well-used resources. Following this approach, we have developed an inherently robust meta-scheduler, COBalD, for managing diverse, heterogeneous resources given unpredictable resource requirements. This paper explains the key concepts of our approach, and discusses the benefits and limitations of our new approach to dynamic resource provisioning compared to previous approaches.

Raythena: a vertically integrated scheduler for ATLAS applications on heterogeneous distributed resources

The ATLAS experiment has successfully integrated HighPerformance Computing resources (HPCs) in its production system. Unlike the current generation of HPC systems, and the LHC computing grid, the next generation of supercomputers is expected to be extremely heterogeneous in nature: different systems will have radically different architectures, and most of them will provide partitions optimized for different kinds of workloads. In this work we explore the applicability of concepts and tools realized in Ray (the high-performance distributed execution framework targeting large-scale machine learning applications) to ATLAS event throughput optimization on heterogeneous distributed resources, ranging from traditional grid clusters to Exascale computers. We present a prototype of Raythena, a Ray-based implementation of the ATLAS Event Service (AES), a fine-grained event processing workflow aimed at improving the efficiency of ATLAS workflows on opportunistic resources, specifically HPCs. The AES is implemented as an event processing task farm that distributes packets of events to several worker processes running on multiple nodes. Each worker in the task farm runs an event-processing application (Athena) as a daemon. The whole system is orchestrated by Ray, which assigns work in a distributed, possibly heterogeneous, environment. For all its flexibility, the AES implementation is currently comprised of multiple separate layers that communicate through ad-hoc command-line and filebased interfaces. The goal of Raythena is to integrate these layers through a feature-rich, efficient application framework. Besides increasing usability and robustness, a vertically integrated scheduler will enable us to explore advanced concepts such as dynamically shaping of workflows to exploit currently available resources, particularly on heterogeneous systems.

A fully unprivileged CernVM-FS

The CernVM File System provides the software and container distribution backbone for most High Energy and Nuclear Physics experiments. It is implemented as a file system in user-space (Fuse) module, which permits its execution without any elevated privileges. Yet, mounting the file system in the first place is handled by a privileged suid helper program that is installed by the Fuse package on most systems. The privileged nature of the mount system call is a serious hindrance to running CernVM-FS on opportunistic resource and supercomputers. Fortunately, recent developments in the Linux kernel and in the Fuse user-space libraries enabled fully unprivileged mounting for Fuse file systems (as of RHEL 8), or at least outsourcing the privileged mount system call to a custom, external process. This opens the door to several, very appealing new ways to use CernVM-FS, such as a generally usable “super pilot” consisting of the pilot code bundled with Singularity and CernVM-FS, or the on-demand instantiation of unprivileged, ephemeral containers to publish new CernVM-FS content from anywhere. In this contribution, we discuss the integration of these new Linux features with CernVM-FS and show some of its most promising, new applications.

Predicting resource usage for enhanced job scheduling for opportunistic resources in HEP

To overcome the computing challenge in High Energy Physics available resources must be utilized as efficiently as possible. This targets algorithmic challenges in the workflows itself but also the scheduling of jobs to compute resources. To enable the best possible scheduling, job schedulers require accurate information about resource consumption of a job before it is even executed. It is the responsibility of the user to provide an accurate resource estimate required for jobs. However, this is quite a challenge for users as they (i) want to ensure their jobs to run correctly, (ii) must manage to deal with heterogeneous compute resources and (iii) face intransparent library dependencies and frequent updates. Users therefore tend to specify resource requests with an ample buffer. This inaccuracy results in inefficient utilisation by either blocking unused resources or exceeding reserved resources. Especially in the context of opportunistic resource provisioning the inaccuracies have an even broader impact that does not even target utilisation of resources but also composition of the most suitable resources. The contribution of this paper is an analysis of production and end-user workflows in HEP with regards to optimizing the various resources types. We further propose a method to improve user estimates.

Fast distributed compilation and testing of large C++ projects

High energy physics experiments traditionally have large software codebases primarily written in C++ and the LHCb physics software stack is no exception. Compiling from scratch can easily take 5 hours or more for the full stack even on an 8-core VM. In a development workflow, incremental builds often do not significantly speed up compilation because even just a change of the modification time of a widely used header leads to many compiler and linker invokations. Using powerful shared servers is not practical as users have no control and maintenance is an issue. Even though support for building partial checkouts on top of published project versions exists, by far the most practical development workflow involves full project checkouts because of off-the-shelf tool support (git, intellisense, etc.) This paper details a deployment of distcc, a distributed compilation server, on opportunistic resources such as development machines. The best performance operation mode is achieved when preprocessing remotely and profiting from the shared CernVM File System. A 10 (30) fold speedup of elapsed (real) time is achieved when compiling Gaudi, the base of the LHCb stack, when comparing local compilation on a 4 core VM to remote compilation on 80 cores, where the bottleneck becomes non-distributed work such as linking. Compilation results are cached locally using ccache, allowing for even faster rebuilding. A recent distributed memcached-based shared cache is tested as well as a more modern distributed system by Mozilla, sccache, backed by S3 storage. These allow for global sharing of compilation work, which can speed up both central CI builds and local development builds. Finally, we explore remote caching and execution services based on Bazel, and how they apply to Gaudi-based software for distributing not only compilation but also linking and even testing.

Cloudscheduler: a VM provisioning system for a distributed compute cloud

We describe a high-throughput computing system for running jobs on public and private computing clouds using the HTCondor job scheduler and the cloudscheduler VM provisioning service. The distributed cloud computing system is designed to simultaneously use dedicated and opportunistic cloud resources at local and remote locations. It has been used for large scale production particle physics workloads for many years using thousands of cores on three continents. Cloudscheduler has been modernized to take advantage of new software designs, improved operating system capabilities and support packages. The result is a more resilient and scalable system, with expanded capabilities.

IEEE Access Special Section Editorial: Mobile Edge Computing

Mobile edge computing (MEC) is emerging as a promising computing architecture by pushing computation and storage closer to end-users with both strategically deployed edge servers and opportunistic available resources. Essentially different from the traditional definition of cloud computing, mobile edge computing can provide solutions to problems faced by mobile cloud computing and support a large number of different mobile and the Internet of Things (IoT) applications and services. However, to maximize the resource utilization and user service experiences, it is envisioned that mobile edge computing and cloud computing should be designed and operated under a seamless integrated hierarchical architecture, though there are still many unprecedented technical challenges for applying mobile edge computing to the emerging mobile and IoT applications.

Managing the CERN Batch System with Kubernetes

The CERN Batch Service faces many challenges in order to get ready for the computing demands of future LHC runs. These challenges require that we look at all potential resources, assessing how efficiently we use them and that we explore different alternatives to exploit opportunistic resources in our infrastructure as well as outside of the CERN computing centre. Several projects, like BEER, Helix Nebula Science Cloud and the new OCRE project, have proven our ability to run batch workloads on a wide range of non-traditional resources. However, the challenge is not only to obtain the raw compute resources needed but how to define an operational model that is cost and time efficient, scalable and flexible enough to adapt to a heterogeneous infrastructure. In order to tackle both the provisioning and operational challenges it was decided to use Kubernetes. By using Kubernetes we benefit from a de-facto standard in containerised environments, available in nearly all cloud providers and surrounded by a vibrant ecosystem of open-source projects. Leveraging Kubernetes’ built-in functionality, and other open-source tools such as Helm, Terraform and GitLab CI, we have deployed a first cluster prototype which we discuss in detail. The effort has simplified many of the existing operational procedures we currently have, but has also made us rethink established procedures and assumptions that were only valid in a VM-based cloud environment. This contribution presents how we have adopted Kubernetes into the CERN Batch Service, the impact its adoption has in daily operations, a comparison on resource usage efficiency and the experience so far evolving our infrastructure towards this model.

Standalone containers with ATLAS offline software

This paper describes the deployment of the offline software of the ATLAS experiment at LHC in containers for use in production workflows such as simulation and reconstruction. To achieve this goal we are using Docker and Singularity, which are both lightweight virtualization technologies that can encapsulate software packages inside complete file systems. The deployment of offline releases via containers removes the interdependence between the runtime environment needed for job execution and the configuration of the computing nodes at the sites. Docker or Singularity would provide a uniform runtime environment for the grid, HPCs and for a variety of opportunistic resources. Additionally, releases may be supplemented with a detector’s conditions data, thus removing the need for network connectivity at computing nodes, which is normally quite restricted for HPCs. In preparation to achieve this goal, we have built Docker and Singularity images containing single full releases of ATLAS software for running detector simulation and reconstruction jobs in runtime environments without a network connection. Unlike similar efforts to produce containers by packing all possible dependencies of every possible workflow into heavy images (≈ 200GB), our approach is to include only what is needed for specific workflows and to manage dependencies efficiently via software package managers. This approach leads to more stable packaged releases where the dependencies are clear and the resulting images have more portable sizes ( 16GB). In an effort to cover a wider variety of workflows, we are deploying images that can be used in raw data reconstruction. This is particularly challenging due to the high database resource consumption during the access to the experiment’s conditions payload when processing data. We describe here a prototype pipeline in which images are provisioned only with the conditions payload necessary to satisfy the jobs’ requirements. This database-on-demand approach would keep images slim, portable and capable of supporting various workflows in a standalone fashion in environments with no network connectivity.

Conflict-Free Scheduling in Cellular V2X Communications

Cellular V2X, the “Vehicle to Everything” standard, defines a framework for information exchange among vehicles and other network entities. In one of the main modes, LTE V2X relies on a central scheduler to minimize the consumed resources in a conflict-free manner. This NP-hard problem leads to a scheduling strategy that assigns separate resources to the conflicting links and can enjoy from opportunistic resource reuse. In this paper, a novel polynomial-time heuristic, MUCS, is introduced, which models this scheduling as a Vehicle Routing Problem. The existing conflict-free schedulers face major incompetence in satisfying the LTE and 5G V2X requirements mainly due to their reliance on simplifications that are unnatural to the V2X environment. On the contrary, MUCS can flexibly accommodate general Device-to-Device topologies as the basis of V2X networks without imposing any packet segmentation. This way, MUCS minimizes the control information overhead in the cellular V2X standard. Due to scalability and a high quality resource utilization (near-optimal in certain conditions), compared to the existing literature, MUCS is desirable for LTE V2X and the 5G networks.