Modern Workloads Research Articles

Breathing New Life into an Old Tree: Resolving Logging Dilemma of B + -tree on Modern Computational Storage Drives

Having dominated databases and various data management systems for decades, B + -tree is infamously subject to a logging dilemma: One could improve B + -tree speed performance by equipping it with a larger log, which nevertheless will degrade its crash recovery speed. Such a logging dilemma is particularly prominent in the presence of modern workloads that involve intensive small writes. In this paper, we propose a novel solution, called per-page logging based B + -tree, which leverages the emerging computational storage drive (CSD) with built-in transparent compression to fundamentally resolve the logging dilemma. Our key idea is to divide the large single log into many small (e.g., 4KB), highly compressible per-page logs , each being statically bounded with a B + -tree page. All per-page logs together form a very large over-provisioned log space for B + -tree to improve its operational speed performance. Meanwhile, during crash recovery, B + -tree does not need to scan any per-page logs, leading to a recovery latency independent from the total log size. We have developed and open-sourced a fully functional prototype. Our evaluation results show that, under small-write intensive workloads, our design solution can improve B + -tree operational throughput by up to 625.6% and maintain a crash recovery time of as low as 19.2 ms, while incurring a minimal storage overhead of only 0.5-1.6%.

NDP-RANK: Prediction and ranking of NDP systems performance using machine learning

The near-data processing (NDP) paradigm has recently gained popularity as a promising method for mitigating the memory wall challenges of future computing systems. Modern 3D-stacked DRAM memories can be leveraged to prevent unnecessary data movement between the main memory and the CPU. However, identifying which type of NDP system, in terms of processing elements and the type of high performance 3D memory used, is suitable for a given application is not a trivial task. Understanding the interactions between modern workloads and the memory subsystem requires a lot of effort. Each memory type presents its unique advantages and shortcomings. In addition, memory access patterns vary greatly across applications. Therefore, the performance of a given application on a given memory type is difficult to intuitively predict. There is no specific NDP system or memory type that can provide high performance for every application.We propose NDP-RANK, a machine learning framework that can efficiently decide which NDP system is suitable for an application. NDP-RANK’s performance prediction is based on an input set of application and microarchitecture characteristics. We build machine learning models that can be used to conduct design-space exploration of NDP systems and accurately predict performance of previously unseen applications on several NDP architectures. Our models are orders of magnitude faster than architectural simulation. They can accurately predict performance with coefficients of determination ranging between 0.89 and 0.96, and root mean square errors ranging between 0.04 and 0.16.

Velox

The ad-hoc development of new specialized computation engines targeted to very specific data workloads has created a siloed data landscape. Commonly, these engines share little to nothing with each other and are hard to maintain, evolve, and optimize, and ultimately provide an inconsistent experience to data users. In order to address these issues, Meta has created Velox, a novel open source C++ database acceleration library. Velox provides reusable, extensible, high-performance, and dialect-agnostic data processing components for building execution engines, and enhancing data management systems. The library heavily relies on vectorization and adaptivity, and is designed from the ground up to support efficient computation over complex data types due to their ubiquity in modern workloads. Velox is currently integrated or being integrated with more than a dozen data systems at Meta, including analytical query engines such as Presto and Spark, stream processing platforms, message buses and data warehouse ingestion infrastructure, machine learning systems for feature engineering and data preprocessing (PyTorch), and more. It provides benefits in terms of (a) efficiency wins by democratizing optimizations previously only found in individual engines, (b) increased consistency for data users, and (c) engineering efficiency by promoting reusability.

Diagnosing the coexistence of Poissonity and self-similarity in memory workloads

Accurately modeling the inter-arrival distribution is critical for synthesizing memory traces to evaluate performance. The intensity and complexity of memory accesses have increased greatly in modern applications. This motivates us to re-examine two fundamental issues: (1) Can the arrival process of memory accesses in modern workloads still be described as a Poisson process? (2) Do modern memory workloads present self-similarity? Via analyzing SPEC CPU2017, this paper demonstrates the coexistence of both Poissonity and self-similarity in modern memory workloads, which seems to reconcile the contradiction between Poisson and self-similar characteristics. Specifically, memory access intervals at a small time scale, such as milliseconds, approximately follow an exponential distribution and memory accesses are independent of each other. At such time scales, memory access series can be approximated by a Poisson process. For the aggregated process of memory access series at a large time scale, such as several minutes, the non-degenerate structures of the auto-correlation function show the presence of self-similarity in memory workloads. All Hurst parameters estimated are greater than 0.5, confirming the presence of self-similarity.

MoRS: An Approximate Fault Modeling Framework for Reduced-Voltage SRAMs

On-chip memory (usually based on Static RAMs-SRAMs) are crucial components for various computing devices including heterogeneous devices, e.g, GPUs, FPGAs, ASICs to achieve high performance. Modern workloads such as Deep Neural Networks (DNNs) running on these heterogeneous fabrics are highly dependent on the on-chip memory architecture for efficient acceleration. Hence, improving the energy-efficiency of such memories directly leads to an efficient system. One of the common methods to save energy is undervolting i.e., supply voltage underscaling below the nominal level. Such systems can be safely undervolted without incurring faults down to a certain voltage limit. This safe range is also called voltage guardband. However, reducing voltage below the guardband level without decreasing frequency causes timing-based faults. In this paper, we propose MoRS, a framework that generates the first approximate undervolting fault model using real faults extracted from experimental undervolting studies on SRAMs to build the model. We inject the faults generated by MoRS into the on-chip memory of the DNN accelerator to evaluate the resilience of the system under the test. MoRS has the advantage of simplicity without any need for high-time overhead experiments while being accurate enough in comparison to a fully randomly-generated fault injection approach. We evaluate our experiment in popular DNN workloads by mapping weights to SRAMs and measure the accuracy difference between the output of the MoRS and the real data. Our results show that the maximum difference between real fault data and the output fault model of MoRS is 6.21%, whereas the maximum difference between real data and random fault injection model is 23.2%. In terms of average proximity to the real data, the output of MoRS outperforms the random fault injection approach by 3.21x.

Memory-Aware Functional IR for Higher-Level Synthesis of Accelerators

Specialized accelerators deliver orders of a magnitude of higher performance than general-purpose processors. The ever-changing nature of modern workloads is pushing the adoption of Field Programmable Gate Arrays (FPGAs) as the substrate of choice. However, FPGAs are hard to program directly using Hardware Description Languages (HDLs). Even modern high-level HDLs, e.g., Spatial and Chisel, still require hardware expertise. This article adopts functional programming concepts to provide a hardware-agnostic higher-level programming abstraction. During synthesis, these abstractions are mechanically lowered into a functional Intermediate Representation (IR) that defines a specific hardware design point. This novel IR expresses different forms of parallelism and standard memory features such as asynchronous off-chip memories or synchronous on-chip buffers. Exposing such features at the IR level is essential for achieving high performance. The viability of this approach is demonstrated on two stencil computations and by exploring the optimization space of matrix-matrix multiplication. Starting from a high-level representation for these algorithms, our compiler produces low-level VHSIC Hardware Description Language (VHDL) code automatically. Several design points are evaluated on an Intel Arria 10 FPGA, demonstrating the ability of the IR to exploit different hardware features. This article also shows that the designs produced are competitive with highly tuned OpenCL implementations and outperform hardware-agnostic OpenCL code.

Benchmarking a New Paradigm: Experimental Analysis and Characterization of a Real Processing-in-Memory System

Many modern workloads, such as neural networks, databases, and graph processing, are fundamentally memory-bound. For such workloads, the data movement between main memory and CPU cores imposes a significant overhead in terms of both latency and energy. A major reason is that this communication happens through a narrow bus with high latency and limited bandwidth, and the low data reuse in memory-bound workloads is insufficient to amortize the cost of main memory access. Fundamentally addressing this data movement bottleneck requires a paradigm where the memory system assumes an active role in computing by integrating processing capabilities. This paradigm is known as processing-in-memory (PIM). Recent research explores different forms of PIM architectures, motivated by the emergence of new 3D-stacked memory technologies that integrate memory with a logic layer where processing elements can be easily placed. Past works evaluate these architectures in simulation or, at best, with simplified hardware prototypes. In contrast, the UPMEM company has designed and manufactured the first publicly-available real-world PIM architecture. The UPMEM PIM architecture combines traditional DRAM memory arrays with general-purpose in-order cores, called DRAM Processing Units (DPUs), integrated in the same chip. This paper provides the first comprehensive analysis of the first publicly-available real-world PIM architecture. We make two key contributions. First, we conduct an experimental characterization of the UPMEM-based PIM system using microbenchmarks to assess various architecture limits such as compute throughput and memory bandwidth, yielding new insights. Second, we present PrIM (Processing-In-Memory benchmarks), a benchmark suite of 16 workloads from different application domains (e.g., dense/sparse linear algebra, databases, data analytics, graph processing, neural networks, bioinformatics, image processing), which we identify as memory-bound. We evaluate the performance and scaling characteristics of PrIM benchmarks on the UPMEM PIM architecture, and compare their performance and energy consumption to their modern CPU and GPU counterparts. Our extensive evaluation conducted on two real UPMEM-based PIM systems with 640 and 2,556 DPUs provides new insights about suitability of different workloads to the PIM system, programming recommendations for software designers, and suggestions and hints for hardware and architecture designers of future PIM systems.

Rearchitecting in-memory object stores for low latency

Low latency is increasingly critical for modern workloads, to the extent that compute functions are explicitly scheduled to be co-located with their in-memory object stores for faster access. However, the traditional object store architecture mandates that clients interact with the server via inter-process communication (IPC). This poses a significant performance bottleneck for low-latency workloads. Meanwhile, in many important emerging AI workloads, such as parallel tree search and reinforcement learning, all the worker processes accessing the object store belong to a single user. We design Lightning, an in-memory object store rearchitected for modern, low-latency workloads in a single-user, multi-process setting. Lightning departs from the traditional design by adopting a shared memory model, enabling clients to directly access the object store without IPC boundary. Instead, client isolation is achieved by a novel integration of Intel Memory Protect Keys (MPK) hardware, transaction logging, and formal verification. Our evaluations show that Lightning outperforms state-of-the-art in-memory object stores by up to 9.0x on five standard NoSQL workloads and up to 4.5x in scaling up a Python tree search program. Lightning improves the throughput of a popular reinforcement learning framework that uses an in-memory object store for data sharing by up to 40%.

Resiliency Engineering in Cloud-Native Environments: Fail-Safe Mechanisms for Modern Workloads

Resiliency Engineering in Cloud-Native Environments: Fail-Safe Mechanisms for Modern Workloads

A Survey on Domain-Specific Memory Architectures

The never-ending demand for high performance and energy efficiency is pushing designers towards an increasing level of heterogeneity and specialization in modern computing systems. In such systems, creating efficient memory architectures is one of the major opportunities for optimizing modern workloads (e.g., computer vision, machine learning, graph analytics, etc.) that are extremely data-driven. However, designers demand proper design methods to tackle the increasing design complexity and address several new challenges, like the security and privacy of the data to be elaborated.This paper overviews the current trend for the design of domain-specific memory architectures. Domain-specific architectures are tailored for the given application domain, with the introduction of hardware accelerators and custom memory modules while maintaining a certain level of flexibility. We describe the major components, the common challenges, and the state-of-the-art design methodologies for building domain-specific memory architectures. We also discuss the most relevant research projects, providing a classification based on our main topics.

Extended performance accounting using Valgrind tool

Modern workloads, parallel or sequential, usually suffer from insufficient memory and computing performance. Common trends to improve workload performance include the utilizations of complex functional units or coprocessors, which are able not only to provide accelerated computations but also independently fetch data from memory generating complex address patterns, with or without support of control flow operations. Such coprocessors usually are not adopted by optimizing compilers and should be utilized by special application interfaces by hand. On the other hand, memory bottlenecks may be avoided with proper use of processor prefetch capabilities which load necessary data ahead of actual utilization time, and the prefetch is also adopted only for simple cases making programmers to do it usually by hand. As workloads are fast migrating to embedded applications a problem raises how to utilize all hardware capabilities for speeding up workload at moderate efforts. This requires precise analysis of memory access patterns at program run time and marking hot spots where the vast amount of memory accesses is issued. Precise memory access model can be analyzed via simulators, for example Valgrind, which is capable to run really big workload, for example neural network inference in reasonable time. But simulators and hardware performance analyzers fail to separate the full amount of memory references and cache misses per particular modules as it requires the analysis of program call graph. We are extending Valgrind tool cache simulator, which allows to account memory accesses per software modules and render realistic distribution of hot spot in a program. Additionally the analysis of address sequences in the simulator allows to recover array access patterns and propose effective prefetching schemes. Motivating samples are provided to illustrate the use of Valgrind tool.

Using Human-Agent Teams to Purposefully Design Multi-Agent Systems

With multi-agent teams becoming more of a reality every day, it is important to create a common design model for multi-agent teams. These teams need to be able to function in dynamic environments and still communicate with any humans that may need a problem solved. Existing human-agent research can be used to purposefully create multi-agent teams that are interdependent but can still interact with humans. Rather than creating dynamic agents, the most effective way to overcome the dynamic nature of modern workloads is to create a dynamic team configuration, rather than individual member-agents that can change their roles. Multi-agent teams will require a variety of agents to be designed to cover a diverse subset of problems that need to be solved in the modern workforce. A model based on existing multi-agent teams that satisfies the needs of human-agent teams has been created to serve as a baseline for human-interactive multi-agent teams.

Flexible device compositions and dynamic resource sharing in PCIe interconnected clusters using Device Lending

Modern workloads often exceed the processing and I/O capabilities provided by resource virtualization, requiring direct access to the physical hardware in order to reduce latency and computing overhead. For computers interconnected in a cluser, access to remote hardware resources often requires facilitation both in hardware and specialized drivers with virtualization support. This limits the availability of resources to specific devices and drivers that are supported by the virtualization technology being used, as well as what the interconnection technology supports. For PCI Express (PCIe) clusters, we have previously proposed Device Lending as a solution for enabling direct low latency access to remote devices. The method has extremely low computing overhead, and does not require any application- or device-specific distribution mechanisms. Any PCIe device, such as network cards disks, and GPUs, can easily be shared among the connected hosts. In this work, we have extended our solution with support for a virtual machine (VM) hypervisor. Physical remote devices can be “passed through” to VM guests, enabling direct access to physical resources while still retaining the flexibility of virtualization. Additionally, we have also implemented multi-device support, enabling shortest-path peer-to-peer transfers between remote devices residing in different hosts.Our experimental results prove that multiple remote devices can be used, achieving bandwidth and latency close to native PCIe, and without requiring any additional support in device drivers. I/O intensive workloads run seamlessly using both local and remote resources. With our added VM and multi-device support, Device Lending offers highly customizable configurations of remote devices that can be dynamically reassigned and shared to optimize resource utilization, thus enabling a flexible composable I/O infrastructure for VMs as well as bare-metal machines.

Speeding up Collective Communications Through Inter-GPU Re-Routing

In order to address the vast needs of disparate domains, computing engines are becoming more sophisticated and complex. A typical high-performance computational engine is composed of several accelerator units, in most cases GPUs, plus one or more CPU controllers. All these components are becoming increasingly interconnected to satisfy bandwidth and latency tolerance demands from modern workloads. Due to these constraints, solutions to efficiently interconnect them or to systematically manage their traffic-such as PCIe v3, NVLink v1 and v2 on the hardware side, and NVIDIA Collective Communication Library (NCCL) and AMD ROCM layer on the software side-are becoming more commonplace inside HPC systems and cloud data centers. However, as the number of accelerators increases, workloads (especially machine learning) might not be able to fully exploit the computational substrate due to inefficient use of hardware interconnects. Such scenarios can lead to performance bottlenecks where high-bandwidth links are not used by the underlying libraries and under-performing links are overused. This work proposes Workload Optimization Through Inter-GPU Re-routing (WOTIR), which consists of enhanced NCCL-based collective primitives that aim to boost bandwidth utilization (through more efficient routing) and reduce communication overhead. WOTIR targets GPUs with no direct NVLink communication path (which leads to PCIe communications) and instead re-routes communication through intermediate GPUs to bridge NVLink segments and avoid PCIe communications. Such method allows the maximum possible utilization of the NVLink bandwidth between the GPUs without routing through the PCIe bus. Using this method, we see a reduction of up to 34 percent in execution time for selected machine learning workloads when non-optimal GPU allocations arise.

Large Scale Studies of Memory, Storage, and Network Failures in a Modern Data Center

The workloads running in the modern data centers of large scale Internet service providers (such asAlibaba, Amazon, Baidu, Facebook, Google, and Microsoft) support billions of users and span globallydistributed infrastructure. Yet, the devices used in modern data centers fail due to a variety of causes, fromfaulty components to bugs to misconfiguration. Faulty devices make operating large scale data centerschallenging because the workloads running in modern data centers consist of interdependent programsdistributed across many servers, so failures that are isolated to a single device can still have a widespreadeffect on a workload.In this dissertation, we measure and model the device failures in a large scale Internet service company,Facebook. We focus on three device types that form the foundation of Internet service data centerinfrastructure: DRAM for main memory, SSDs for persistent storage, and switches and backbone linksfor network connectivity. For each of these device types, we analyze long term device failure data brokendown by important device attributes and operating conditions, such as age, vendor, and workload. Wealso build and release statistical models of the failure trends for the devices we analyze.For DRAM devices, we analyze the memory errors in the entire fleet of servers at Facebook over thecourse of fourteen months, representing billions of device days of operation. The systems we examinecover a wide range of devices commonly used in modern servers, with DIMMs that use the modernDDR3 communication protocol, manufactured by 4 vendors in capacities ranging from 2GB to 24GB.We observe several new reliability trends for memory systems that have not been discussed before inliterature, develop a model for memory reliability, show how system design choices such as using lowerdensity DIMMs and fewer cores per chip can reduce failure rates of a baseline server by up to 57.7%.We perform the first implementation and real-system analysis of page offlining at scale, on a cluster ofthousands of servers, identify several real-world impediments to the technique, and show that it canreduce memory error rate by 67%. We also examine the efficacy of a new technique to reduce DRAMfaults, physical page randomization, and examine its potential for improving reliability and its overheads.For SSD devices, we perform a large scale study of flash-based SSD reliability at Facebook. We analyzedata collected across a majority of flash-based solid state drives over nearly four years and manymillions of operational hours in order to understand failure properties and trends of flash-based SSDs.Our study considers a variety of SSD characteristics, including: the amount of data written to and readfrom flash chips; how data is mapped within the SSD address space; the amount of data copied, erased,and discarded by the flash controller; and flash board temperature and bus power. Based on our fieldanalysis of how flash memory errors manifest when running modern workloads on modern SSDs, we make several major observations and find that SSD failure rates do not increase monotonically with flashchip wear, but instead they go through several distinct periods corresponding to how failures emerge andare subsequently detected.For network devices, we perform a large scale, longitudinal study of data center network reliabilitybased on operational data collected from the production network infrastructure at Facebook. Our studycovers reliability characteristics of both intra and inter data center networks. For intra data center networks,we study seven years of operation data comprising thousands of network incidents across twodifferent data center network designs, a cluster network design and a state-of-the-art fabric network design.For inter data center networks, we study eighteen months of recent repair tickets from the field tounderstand the reliability of Wide Area Network (WAN) backbones. In contrast to prior work, we studythe effects of network reliability on software systems, and how these reliability characteristics evolve overtime. We discuss the implications of network reliability on the design, implementation, and operation oflarge scale data center systems and how the network affects highly-available web services.Our key conclusion in this dissertation is that we can gain a deep understanding of why devicesfail—and how to predict their failure—using measurement and modeling. We hope that the analysis,techniques, and models we present in this dissertation will enable the community to better measure,understand, and prepare for the hardware reliability challenges we face in the future.

An Event-Triggered Programmable Prefetcher for Irregular Workloads

Many modern workloads compute on large amounts of data, often with irregular memory accesses. Current architectures perform poorly for these workloads, as existing prefetching techniques cannot capture the memory access patterns; these applications end up heavily memory-bound as a result. Although a number of techniques exist to explicitly configure a prefetcher with traversal patterns, gaining significant speedups, they do not generalise beyond their target data structures. Instead, we propose an event-triggered programmable prefetcher combining the flexibility of a general-purpose computational unit with an event-based programming model, along with compiler techniques to automatically generate events from the original source code with annotations. This allows more complex fetching decisions to be made, without needing to stall when intermediate results are required. Using our programmable prefetching system, combined with small prefetch kernels extracted from applications, we achieve an average 3.0x speedup in simulation for a variety of graph, database and HPC workloads.

Making Huge Pages Actually Useful

The virtual-to-physical address translation overhead, a major performance bottleneck for modern workloads, can be effectively alleviated with huge pages. However, since huge pages must be mapped contiguously, OSs have not been able to use them well because of the memory fragmentation problem despite hardware support for huge pages being available for nearly two decades. This paper presents a comprehensive study of the interaction of fragmentation with huge pages in the Linux kernel. We observe that when huge pages are used, problems such as high CPU utilization and latency spikes occur because of unnecessary work (e.g., useless page migration) performed by memory management related subsystems due to the poor handling of unmovable (i.e., kernel) pages. This behavior is even more harmful in virtualized systems where unnecessary work may be performed in both guest and host OSs. We present Illuminator, an efficient memory manager that provides various subsystems, such as the page allocator, the ability to track all unmovable pages. It allows subsystems to make informed decisions and eliminate unnecessary work which in turn leads to cost-effective huge page allocations. Illuminator reduces the cost of compaction (up to 99%), improves application performance (up to 2.3x) and reduces the maximum latency of MySQL database server (by 30x). Importantly, this work shows the effectiveness of a simple solution for long-standing huge page related problems.

Performance innovations in the IBM z14 platform

The IBM z14 platform brings together numerous innovations that collectively provide considerable performance gains over prior designs for the full spectrum of modern workloads being run on today's enterprise systems. These workloads range from traditional online transaction processing and batch workloads to Linux-based and cloud-based workloads, leveraging advances in analytics, mobile transaction processing, social data mining, cognitive applications, security (blockchain), and other areas. The performance gains in the z14 platform required advancements in the processor hardware microarchitecture, physical design, and instruction-set architecture, as well as in the firmware and hypervisor stacks. The result is a comprehensive platform solution that enables efficient multiprocessor scaling for large single-image partitions and mixed-workload multipartition environments within a single platform, while remaining responsive to the “bursty” demands of batch and transaction-based workloads. The performance gains present in the z14 are a direct result of the collaborative efforts between diverse teams within IBM working together to produce advancements in hardware, firmware, and software that outpace the gains of the individual contributions. In this paper, we discuss these results with a focus on the client value delivered through system performance and workload responsiveness.

Domino Cache

The energy consumption for processing modern workloads is challenging in data centers. Due to the large datasets of cloud workloads, the miss rate of the L1 data cache is high, and with respect to the energy efficiency concerns, such misses are costly for memory instructions because lower levels of memory hierarchy consume more energy per access than the L1. Moreover, large last-level caches are not performance effective, in contrast to traditional scientific workloads. The aim of this article is to propose a large L1 data cache, called Domino, to reduce the number of accesses to lower levels in order to improve the energy efficiency. In designing Domino, we focus on two components that use the on-chip area and are not energy efficient, which makes them good candidates to use their area for enlarging the L1 data cache. Domino is a highly associative cache that extends the conventional cache by borrowing the prefetcher and last-level-cache storage budget and using it as additional ways for data cache. In Domino, the additional ways are separated from the conventional cache ways; hence, the critical path of the first access is not altered. On a miss in the conventional part, it searches the added ways in a mix of parallel-sequential fashion to compromise the latency and energy consumption. Results on the Cloudsuite benchmark suite show that read and write misses are reduced by 30%, along with a 28% reduction in snoop messages. The overall energy consumption per access is then reduced by 20% on average (maximum 38%) as a result of filtering accesses to the lower levels.

Dynamo

Data center power is a scarce resource that often goes underutilized due to conservative planning. This is because the penalty for overloading the data center power delivery hierarchy and tripping a circuit breaker is very high, potentially causing long service outages. Recently, dynamic server power capping, which limits the amount of power consumed by a server, has been proposed and studied as a way to reduce this penalty, enabling more aggressive utilization of provisioned data center power. However, no real at-scale solution for data center-wide power monitoring and control has been presented in the literature. In this paper, we describe Dynamo -- a data center-wide power management system that monitors the entire power hierarchy and makes coordinated control decisions to safely and efficiently use provisioned data center power. Dynamo has been developed and deployed across all of Facebook's data centers for the past three years. Our key insight is that in real-world data centers, different power and performance constraints at different levels in the power hierarchy necessitate coordinated data center-wide power management. We make three main contributions. First, to understand the design space of Dynamo, we provide a characterization of power variation in data centers running a diverse set of modern workloads. This characterization uses fine-grained power samples from tens of thousands of servers and spanning a period of over six months. Second, we present the detailed design of Dynamo. Our design addresses several key issues not addressed by previous simulation-based studies. Third, the proposed techniques and design have been deployed and evaluated in large scale data centers serving billions of users. We present production results showing that Dynamo has prevented 18 potential power outages in the past 6 months due to unexpected power surges; that Dynamo enables optimizations leading to a 13% performance boost for a production Hadoop cluster and a nearly 40% performance increase for a search cluster; and that Dynamo has already enabled an 8% increase in the power capacity utilization of one of our data centers with more aggressive power subscription measures underway.