Storage Hierarchy Research Articles

Reminiscences on Influential Papers

This issue's contributors dive into a history of concurrency control and storage hierarchy for database and data-intensive systems. Enjoy reading!

HDF5 in the exascale era: Delivering efficient and scalable parallel I/O for exascale applications

Accurately modeling real-world systems requires scientific applications at exascale to generate massive amounts of data and manage data storage efficiently. However, parallel input and output (I/O) faces challenges due to new application workflows and the state-of-the-art memory, interconnect, and storage architectures considered in exascale designs. The storage hierarchy has expanded with node-local persistent memory, solid-state storage, and traditional disk and tape-based storage, thus requiring efficiency at each layer and much more efficient data movement among these layers. This paper discusses how the ExaHDF5 project improved the I/O performance and data management for exascale architectures by enhancing HDF5, a widely used parallel I/O library. The team developed an Asynchronous I/O Virtual Object Layer (VOL) connector that allowed overlapping I/O with computation. They also created a Cache VOL to complement asynchronous I/O by incorporating fast storage layers, such as burst buffer and node-local storage, into the parallel I/O workflow through caching and staging data. Additionally, the team enabled data aggregation and I/O at the node level by using a Subfiling Virtual File Driver (VFD). To demonstrate superior I/O performance with HDF5 at exascale, the ExaHDF5 team collaborated with several exascale applications. In this paper, we show I/O performance improvements for three applications: Cabana (a particle-based simulation library), EQSIM (a regional earthquake simulation software), and E3SM (a climate system modeling library).

Data management of scientific applications in a reinforcement learning-based hierarchical storage system

In many areas of data-driven science, large datasets are generated where the individual data objects are images, matrices, or otherwise have a clear structure. However, these objects can be information-sparse, and a challenge is to efficiently find and work with the most interesting data as early as possible in an analysis pipeline. We have recently proposed a new model for big data management where the internal structure and information of the data are associated with each data object (as opposed to simple metadata). There is then an opportunity for comprehensive data management solutions to account for data-specific internal structure as well as access patterns. In this article, we explore this idea together with our recently proposed hierarchical storage management framework that uses reinforcement learning (RL) for autonomous and dynamic data placement in different tiers in a storage hierarchy. Our case-study is based on four scientific datasets: Protein translocation microscopy images, Airfoil angle of attack meshes, 1000 Genomes sequences, and Phenotypic screening images. The presented results highlight that our framework is optimal and can quickly adapt to new data access requirements. It overall reduces the data processing time, and the proposed autonomous data placement is superior compared to any static or semi-static data placement policies.

An Efficient Feature Selection Algorithm for Gene Families Using NMF and ReliefF.

Gene families, which are parts of a genome's information storage hierarchy, play a significant role in the development and diversity of multicellular organisms. Several studies have focused on the characteristics of gene families, such as function, homology, or phenotype. However, statistical and correlation analyses on the distribution of gene family members in the genome have yet to be conducted. Here, a novel framework incorporating gene family analysis and genome selection based on NMF-ReliefF is reported. Specifically, the proposed method starts by obtaining gene families from the TreeFam database and determining the number of gene families within the feature matrix. Then, NMF-ReliefF is used to select features from the gene feature matrix, which is a new feature selection algorithm that overcomes the inefficiencies of traditional methods. Finally, a support vector machine is utilized to classify the acquired features. The results show that the framework achieved an accuracy of 89.1% and an AUC of 0.919 on the insect genome test set. We also employed four microarray gene data sets to evaluate the performance of the NMF-ReliefF algorithm. The outcomes show that the proposed method may strike a delicate balance between robustness and discrimination. Additionally, the proposed method's categorization is superior to state-of-the-art feature selection approaches.

Adaptive Intelligent Tiering for modern storage systems

Enterprise systems routinely use tiered storage consisting of a hierarchy of storage devices that vary in speed and size. One key to obtaining good performance in such a hierarchy is to migrate data elements intelligently to the appropriate tier. For example, moving the most used data towards the fastest tier and the least used data towards the slowest tier. Tiering is typically done based on usage statistics over relatively long time periods. In this paper, we consider a much more agile tiering mechanism called Adaptive Intelligent Tiering (AIT). It can dynamically adapt to the changing behavior of storage accesses by the running applications. The AIT mechanism uses a deep learning model to generate a set of candidate movements and employs a reinforcement learning mechanism to further refine the candidates. Based on extensive simulations in a 3-tier system, we show that the proposed scheme, compared with several other methods, enhances workload performance up to 85% on storage traces with a wide range of characteristics.

A 1.23μJ/Inference, All-Digital Shuffle-Type Group-CNN Accelerator in 28nm CMOS With 85.8% Accuracy on CIFAR-10

To achieve high energy efficiency for edge processing, we implement an all-digital shuffle-type group CNN accelerator with binary weights, which is the first reported shuffle-type GCNN accelerator in CMOS as far as we know. Cross-level optimizations from architecture to hardware level are proposed. The NN architecture is optimized to obtain a hardware-friendly configuration. For hardware-level optimizations, we propose a three-level storage hierarchy with latch-based computing-near-memory to reduce the total power by 25.8%. We also design a novel shuffle hardware to reduce memory access in the original shuffle to 1/3 and a stage-by-stage gated MUX to reduce the power consumption in the original selector circuit to 29%. Fabricated in a 28nm CMOS process, the chip works at 0.48-0.9 V with the best energy efficiency of 1.23 μJ/inf at 0.552 V and 66 MHz with 85.8% accuracy on CIFAR-10 dataset. Its energy efficiency is 1.9- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$34.8\mathbf {\times }$ </tex-math></inline-formula> better than state-of-the-art works.

Gray counters for non-volatile memories

New technologies for non-volatile memories combine the speed and byte addressability of current memory technologies with the low cost, density, and non-volatility of current storage technologies. They use energy only when writing or reading data. While some newer technologies have practically unlimited endurance, others, such as Phase Change Memory do not. However, this limited endurance surpasses that of solid state drives by several orders of magnitude. They can be integrated into the current memory storage hierarchy as a replacement for DRAM. To manage limited endurance, age-based wear leveling divides the memory into pages and counts the number of writes to a page. Pages with high counts can then be relegated to store more stable data. Counters experience “write concentration” as they are incremented as frequently as a page is written, but the changes are concentrated in a much smaller area around the least significant bit. Current designs store page write counters with the page table in the main memory unit.We propose and investigate an alternative counter design based on Gray codes that distributes the bit-flips resulting from the increments over the vast majority of the bits making up the counter. This is a bit-flip aware data structure that can be used in other contexts as well. We show that this design allows us to integrate the page counters into the phase change memory device. We also investigate counters using error control codes.

Consumers' Knowledge and Handling Practices Associated with Fresh-Cut Produce in the United States.

Previous studies have shown that three factors influence fresh-cut produce safety from farm to fork: (1) post-harvest practices in processing facilities, (2) employees’ handling practices in retail facilities, and (3) consumers’ handling practices in domestic kitchens or cooking facilities. However, few studies have examined consumers’ food safety knowledge, risk perceptions, and their handling practices associated with fresh-cut produce. To fill this gap, the present study conducted a nationwide survey to assess U.S. consumers’ food safety knowledge, practices, and risk perception associated with fresh-cut produce among various demographic groups and investigated factors influencing consumers’ food safety practices related to fresh-cut produce. The results showed that consumers lack the knowledge and safe handling practices toward fresh-cut produce regarding storage hierarchy, surface cleaning and sanitizing, and time and temperature control of fresh-cut produce. The men and millennial consumers exhibit a lower level of safe fresh-cut produce handling practices. In addition, a significant interaction was observed between food safety knowledge and risk perceptions on consumers’ fresh-cut produce handling practices, such that food safety knowledge can transfer to practice more effectively for consumers with high levels of risk perception. The results can be utilized to design effective consumer food safety education tools for targeted audiences.

Serving deep learning models with deduplication from relational databases

Serving deep learning models from relational databases brings significant benefits. First, features extracted from databases do not need to be transferred to any decoupled deep learning systems for inferences, and thus the system management overhead can be significantly reduced. Second, in a relational database, data management along the storage hierarchy is fully integrated with query processing, and thus it can continue model serving even if the working set size exceeds the available memory. Applying model deduplication can greatly reduce the storage space, memory footprint, cache misses, and inference latency. However, existing data deduplication techniques are not applicable to the deep learning model serving applications in relational databases. They do not consider the impacts on model inference accuracy as well as the inconsistency between tensor blocks and database pages. This work proposed synergistic storage optimization techniques for duplication detection, page packing, and caching, to enhance database systems for model serving. Evaluation results show that our proposed techniques significantly improved the storage efficiency and the model inference latency, and outperformed existing deep learning frameworks in targeting scenarios.

Transparent Asynchronous Parallel I/O Using Background Threads

Moving toward exascale computing, the size of data stored and accessed by applications is ever increasing. However, traditional disk-based storage has not seen improvements that keep up with the explosion of data volume or the speed of processors. Multiple levels of non-volatile storage devices are being added to handle bursty I/O, however, moving data across the storage hierarchy can take longer than the data generation or analysis. Asynchronous I/O can reduce the impact of I/O latency as it allows applications to schedule I/O early and to check their status later. I/O is thus overlapped with application communication or computation or both, effectively hiding some or all of the I/O latency. POSIX and MPI-I/O provide asynchronous read and write operations, but lack the support for non-data operations such as file open and close. Users also have to manually manage data dependencies and use low-level byte offsets, which requires significant effort and expertise to adopt. In this article, we present an asynchronous I/O framework that supports all types of I/O operations, manages data dependencies transparently and automatically, provides implicit and explicit modes for application flexibility, and error information retrieval. We implemented these techniques in HDF5. Our evaluation of several benchmarks and application workloads demonstrates it effectiveness on hiding the I/O cost from the application.

Programming Abstractions for Managing Workflows on Tiered Storage Systems

Scientific workflows in High Performance Computing ( HPC ) environments are processing large amounts of data. The storage hierarchy on HPC systems is getting deeper, driven by new technologies (NVRAMs, SSDs, etc.) There is a need for new programming abstractions that allow users to seamlessly manage data at the workflow level on multi-tiered storage systems, and provide optimal workflow performance and use of storage resources. In previous work, we introduced a software architecture Managing Data on Tiered Storage for Scientific Workflows (MaDaTS ) that used a Virtual Data Space ( VDS ) abstraction to hide the complexities of the underlying storage system while allowing users to control data management strategies. In this article, we detail the data-centric programming abstractions that allow users to manage a workflow around its data on the storage layer. The programming abstractions simplify data management for scientific workflows on multi-tiered storage systems, without affecting workflow performance or storage capacity. We measure the overheads and effectiveness introduced by the programming abstractions of MaDaTS. Our results show that these abstractions can optimally use the storage capacity in lesser capacity storage tiers, and simplify data management without adding any performance overheads.

Predicting file lifetimes for data placement in multi-tiered storage systems for HPC

The emergence of Exascale machines in HPC will have the foreseen consequence of putting more pressure on the storage systems in place, not only in terms of capacity but also bandwidth and latency. With limited budget we cannot imagine using only storage class memory, which leads to the use of a heterogeneous tiered storage hierarchy. In order to make the most efficient use of the high performance tier in this storage hierarchy, we need to be able to place user data on the right tier and at the right time. In this paper, we assume a 2-tier storage hierarchy with a high performance tier and a high capacity archival tier. Files are placed on the high performance tier at creation time and moved to capacity tier once their lifetime expires (that is once they are no more accessed). The main contribution of this paper lies in the design of a file lifetime prediction model solely based on its path based on the use of Convolutional Neural Network. Results show that our solution strikes a good trade-off between accuracy and under-estimation. Compared to previous work, our model made it possible to reach an accuracy close to previous work (around 98.60% compared to 98.84%) while reducing the underestimations by almost 10x to reach 2.21% (compared to 21.86%). The reduction in underestimations is crucial as it avoids misplacing files in the capacity tier while they are still in use.

Better database cost/performance via batched I/O on programmable SSD

Data should be placed at the most cost- and performance-effective tier in the storage hierarchy. While performance and cost decrease with distance from the CPU, the cost/performance trade-off depends on how efficiently data can be moved across tiers. Log structuring improves this cost/performance by writing batches of pages from main memory to secondary storage using a conventional block-at-a-time I/O interface. However, log structuring incurs overhead in the form of recovery and garbage collection. With computational Solid-State Drives, it is now possible to design a storage interface that minimizes this overhead. In this paper, we offload log structuring from the CPU to the SSD. We define a new batch I/O storage interface and we design a Flash Translation Layer that takes care of log structuring on the SSD side. This removes the CPU computational and I/O load associated with recovery and garbage collection. We compare the performance of the Bw-tree key-value store with its LLAMA host-based log structuring to the same key-value software stack executing on a computational SSD equipped with a batch I/O interface. Our experimental results show the benefits of eliminating redundancies, minimizing interactions across storage layers, and avoiding the CPU cost of providing log structuring.

Chefs' Awareness towards Food Storage Hierarchy in Egyptian Restaurants

This research aims to measure the awareness of chefs towards the hierarchy of food storage and analyze the differences between their awareness based on their demographic data. Literature review includes some topics about food storage in the restaurant such as dry storage, chemical storage, refrigerator storage, and freezer storage, food contamination during food storage, preventing cross-contamination during storage, food contaminations and their impact on health, food hierarchy chart in a restaurant, and chefs’ awareness towards food hierarchy. For achieving the research aim, an electronic questionnaire (as a test) was designed and distributed to a random sample of chefs in Egyptian restaurants, which 447 responses were received and valid to analyze by SPSS V.25. Results indicated that investigated chefs haven’t enough awareness of food hierarchy storage. Some chefs store food in the wrong arrangement that leads to rapid spoilage or cross-contamination. Based on the results, some recommendations have been proposed to increase the awareness of chefs towards food hierarchy storage. One of the most important recommendations was putting guidelines and instructions in clear places within each store that directs chefs on how to arrange food properly.

Designing an Event Store for a\xa0Modern Three-layer Storage Hierarchy

Event stores face the difficult challenge of continuously ingesting massive temporal data streams while satisfying demanding query and recovery requirements. Many of today’s systems deal with multiple hardware-based trade-offs. For instance, long-term storage solutions balance keeping data in cheap secondary media (SSDs, HDDs) and performance-oriented main-memory caches. As an alternative, in-memory systems focus on performance, while sacrificing monetary costs, and, to some degree, recovery guarantees. The advent of persistent memory (PMem) led to a multitude of novel research proposals aiming to alleviate those trade-offs in various fields. So far, however, there is no proposal for a PMem-powered specialized event store.Based on ChronicleDB, we will present several complementary approaches for a three-layer architecture featuring main memory, PMem, and secondary storage. We enhance some of ChronicleDB’s components with PMem for better insertion and query performance as well as better recovery guarantees. At the same time, the three-layer architecture aims to keep the overall dollar cost of a system low. The limitations and opportunities of a PMem-enhanced event store serve as important groundwork for comprehensive system design exploiting a modern storage hierarchy.

LogStore: A Workload-Aware, Adaptable Key-Value Store on Hybrid Storage Systems

Due to recent explosion of data volume and velocity, a new array of lightweight key-value stores have emerged to serve as alternatives to traditional databases. The majority of these storage engines, however, sacrifice their read performance in order to cope with write throughput by avoiding random disk access when writing a record in favor of fast sequential accesses. But, the boundary between sequential versus random access is becoming blurred with the advent of solid-state drives (SSDs). In this work, we propose our new key-value store, LogStore, optimized for hybrid storage architectures. Additionally, introduce a novel cost-based data staging model based on log-structured storage, in which recent changes are first stored on SSDs, and pushed to HDD as it ages, while minimizing the read/write amplification for merging data from SSDs and HDDs. Furthermore, we take a holistic approach in improving both the read and write performance by dynamically optimizing the data layout, such as deferring and reversing the compaction process, and developing an access strategy to leverage the strengths of each available medium in our storage hierarchy. Lastly, in our extensive evaluation, we demonstrate that LogStore achieves up to 6x improvement in throughput/latency over LevelDB, a state-of-the-art key-value store.

Potential of I/O Aware Workflows in Climate and Weather

The efficient, convenient, and robust execution of data-driven workflows and enhanced data management are essential for productivity in scientific computing. In HPC, the concerns of storage and computing are traditionally separated and optimised independently from each other and the needs of the end-to-end user. However, in complex workflows, this is becoming problematic. These problems are particularly acute in climate and weather workflows, which as well as becoming increasingly complex and exploiting deep storage hierarchies, can involve multiple data centres. The key contributions of this paper are: 1) A sketch of a vision for an integrated data-driven approach, with a discussion of the associated challenges and implications, and 2) An architecture and roadmap consistent with this vision that would allow a seamless integration into current climate and weather workflows as it utilises versions of existing tools (ESDM, Cylc, XIOS, and DDN’s IME). The vision proposed here is built on the belief that workflows composed of data, computing, and communication-intensive tasks should drive interfaces and hardware configurations to better support the programming models. When delivered, this work will increase the opportunity for smarter scheduling of computing by considering storage in heterogeneous storage systems. We illustrate the performance-impact on an example workload using a model built on measured performance data using ESDM at DKRZ.

Predicting and Comparing the Performance of Array Management Libraries.

Many applications are increasingly becoming I/O-bound. To improve scalability, analytical models of parallel I/O performance are often consulted to determine possible I/O optimizations. However, I/O performance modeling has predominantly focused on applications that directly issue I/O requests to a parallel file system or a local storage device. These I/O models are not directly usable by applications that access data through standardized I/O libraries, such as HDF5, FITS, and NetCDF, because a single I/O request to an object can trigger a cascade of I/O operations to different storage blocks. The I/O performance characteristics of applications that rely on these libraries is a complex function of the underlying data storage model, user-configurable parameters and object-level access patterns. As a consequence, I/O optimization is predominantly an ad-hoc process that is performed by application developers, who are often domain scientists with limited desire to delve into nuances of the storage hierarchy of modern computers. This paper presents an analytical cost model to predict the end-to-end execution time of applications that perform I/O through established array management libraries. The paper focuses on the HDF5 and Zarr array libraries, as examples of I/O libraries with radically different storage models: HDF5 stores every object in one file, while Zarr creates multiple files to store different objects. We find that accessing array objects via these I/O libraries introduces new overheads and optimizations. Specifically, in addition to I/O time, it is crucial to model the cost of transforming data to a particular storage layout (memory copy cost), as well as model the benefit of accessing a software cache. We evaluate the model on real applications that process observations (neuroscience) and simulation results (plasma physics). The evaluation on three HPC clusters reveals that I/O accounts for as little as 10% of the execution time in some cases, and hence models that only focus on I/O performance cannot accurately capture the performance of applications that use standard array storage libraries. In parallel experiments, our model correctly predicts the fastest storage library between HDF5 and Zarr 94% of the time, in contrast with 70% of the time for a cutting-edge I/O model.

Freezing time emulating new and faster devices with virtual machines

Recent proposals of emerging data storage devices make it necessary to reevaluate all levels of the storage hierarchy to optimize the software stack performance. However, these new devices are not always widely available and therefore early experiments may be impossible. Emulators aim at mimicking as close as possible the behavior of a component, nonetheless, emulating new and fast storage devices is a challenging task due to time perception. In this work, we propose an approach to emulate storage devices using virtual machines (VMs) allowing the evaluation of a new device within a real system. We use a technique called freezing time, which pauses a VM to manipulate its clock and hide the real I/O completion time. Our approach is implemented at the hypervisor level and it is transparent to the guest operating system or application. We evaluate the technique under a real system using regular magnetic disks to emulate faster storage devices. Our method presented a latency error of 6.5% compared to a real device. Moreover, decoupled experiment between two laboratories, at the Barcelona Super Computing Center (BSC) in Spain, and the Center of Computer Science and Free Software (C3SL) in Brazil, demonstrated that our approach is reproducible and promising to allow the virtual evaluation of next-gen storage devices.

I/O Acceleration via Multi-Tiered Data Buffering and Prefetching

Modern High-Performance Computing (HPC) systems are adding extra layers to the memory and storage hierarchy, named deep memory and storage hierarchy (DMSH), to increase I/O performance. New hardware technologies, such as NVMe and SSD, have been introduced in burst buffer installations to reduce the pressure for external storage and boost the burstiness of modern I/O systems. The DMSH has demonstrated its strength and potential in practice. However, each layer of DMSH is an independent heterogeneous system and data movement among more layers is significantly more complex even without considering heterogeneity. How to efficiently utilize the DMSH is a subject of research facing the HPC community. Further, accessing data with a high-throughput and low-latency is more imperative than ever. Data prefetching is a well-known technique for hiding read latency by requesting data before it is needed to move it from a high-latency medium (e.g., disk) to a low-latency one (e.g., main memory). However, existing solutions do not consider the new deep memory and storage hierarchy and also suffer from under-utilization of prefetching resources and unnecessary evictions. Additionally, existing approaches implement a client-pull model where understanding the application’s I/O behavior drives prefetching decisions. Moving towards exascale, where machines run multiple applications concurrently by accessing files in a workflow, a more data-centric approach resolves challenges such as cache pollution and redundancy. In this paper, we present the design and implementation of Hermes: a new, heterogeneous-aware, multi-tiered, dynamic, and distributed I/O buffering system. Hermes enables, manages, supervises, and, in some sense, extends I/O buffering to fully integrate into the DMSH. We introduce three novel data placement policies to efficiently utilize all layers and we present three novel techniques to perform memory, metadata, and communication management in hierarchical buffering systems. Additionally, we demonstrate the benefits of a truly hierarchical data prefetcher that adopts a server-push approach to data prefetching. Our evaluation shows that, in addition to automatic data movement through the hierarchy, Hermes can significantly accelerate I/O and outperforms by more than 2x state-of-the-art buffering platforms. Lastly, results show 10%–35% performance gains over existing prefetchers and over 50% when compared to systems with no prefetching.