Object Storage Research Articles

Boki: Towards Data Consistency and Fault Tolerance with Shared Logs in Stateful Serverless Computing

Bokiis a new serverless runtime that exports a shared log API to serverless functions. Boki shared logs enable stateful serverless applications to manage their state with durability, consistency, and fault tolerance. Boki shared logs achieve high throughput and low latency. The key enabler is the metalog , a novel mechanism that allows Boki to address ordering, consistency and fault tolerance independently. The metalog orders shared log records with high throughput and it provides read consistency while allowing service providers to optimize the write and read path of the shared log in different ways. To demonstrate the value of shared logs for stateful serverless applications, we build Boki support libraries that implement fault-tolerant workflows, durable object storage, and message queues. Our evaluation shows that shared logs can speed up important serverless workloads by up to 4.2×.

Improving Cloud Database Performance Using Storage Technologies

Cloud databases have become indispensable for modern applications due to their scalability, flexibility, and cost-effectiveness. However, optimizing performance in cloud environments can be challenging due to factors such as network latency, data distribution, and resource contention. This paper explores various storage technologies and techniques that can significantly enhance cloud database performance. We delve into the characteristics of different storage types, including solid-state drives (SSDs), hard disk drives (HDDs), and object storage, and discuss their suitability for various workloads. Additionally, we examine caching mechanisms, data compression, and deduplication strategies that can optimize data access and reduce storage costs. Furthermore, we explore the impact of database architecture choices, such as distributed databases and inmemory databases, on performance. We also discuss the importance of effective query optimization and indexing techniques. By understanding and implementing these storage technologies and techniques, organizations can achieve substantial improvements in cloud database performance, leading to better user experiences and increased application efficiency

Efficient security interface for high-performance Ceph storage systems

Ceph portrays a resilient clustered storage solution with supporting object, block, and file storage capabilities with no single point of failure. Despite these qualifications, data confidentiality defines a concern in the system, as authentication and access control are the only data protection security services in Ceph. CephArmor was proposed as a third-party security interface to protect data confidentiality by adding an extra protection layer to data at rest. Despite the added layer, the initial design of the API needed to be more efficient in addressing security and performance simultaneously. In this study, we propose a new architectural design to address the associated issues with the preliminary prototype. Comprehensive performance and security analysis verify the improvement of the proposed method compared to the initial approach. The benchmark result has indicated a 37% improvement on average in IOPS, elapsed time, and bandwidth for the write benchmark compared to the initial model.

Head in the clouds: A Deleuzoguattarian analysis of the environmental impacts of digital memory

Digital memory in the twenty-first century is a complex combination of human and nonhuman elements that interact in various combinations. In an era where environmental devastation is inundating the planet, it is vital to address the digital’s role in this degradation. This article explores various facets of the environmental implications of the digital storage of memory objects using Deleuze and Guattari’s assemblage theory. I identify the ways in which the digital realm is constructed, or ‘territorialised’ in Deleuzoguattarian terms. Beyond identifying the most common myths related to digital memory’s environmental impact, I expand on the work of information theorists such as Clements, Floridi, and Ess by forwarding three ways in which digital memory practices can be deterritorialised. These approaches include (1) knowledge acquisition, (2) taking response-ability, and (3) anti-redundancy movements. I argue that it is vital for cloud storage users, especially those who self-identify as environmentally conscious, to align their online memory practices with accurate understandings of the environmental impacts of the digital.

RefXAS: an open access database of X-ray absorption spectra.

Under DAPHNE4NFDI, the X-ray absorption spectroscopy (XAS) reference database, RefXAS, has been set up. For this purpose, we developed a method to enable users to submit a raw dataset, with its associated metadata, via a dedicated website for inclusion in the database. Implementation of the database includes an upload of metadata to the scientific catalogue and an upload of files via object storage, with automated query capabilities through a web server and visualization of the data and files. Based on the mode of measurements, quality criteria have been formulated for the automated check of any uploaded data. In the present work, the significant metadata fields for reusability, as well as reproducibility of results (FAIR data principles), are discussed. Quality criteria for the data uploaded to the database have been formulated and assessed. Moreover, the usability and interoperability of available XAS data/file formats have been explored. The first version of the RefXAS database prototype is presented, which features a human verification procedure, currently being tested with a new user interface designed specifically for curators; a user-friendly landing page; a full list of datasets; advanced search capabilities; a streamlined upload process; and, finally, a server-side automatic authentication and (meta-) data storage via MongoDB, PostgreSQL and (data-) files via relevant APIs.

Mobilizing underutilized storage nodes via job path: A job-aware file striping approach

Users’ limited understanding of the storage system architecture prevents them from fully utilizing the parallel I/O capability of the storage system, leading to a negative impact on the overall performance of supercomputers. Therefore, exploring effective strategies for utilizing parallel I/O capabilities is imperative. In this regard, we conduct an analysis of the workload on two production supercomputers’ Object Storage Targets (OSTs) and study the potential inefficient I/O patterns for high performance computing jobs. Our research findings indicate that under the traditional stripe settings that most supercomputers use to ensure stability, the real-time load on OSTs is severely unbalanced. This imbalance results in I/O requests that fail to fully utilize the available OSTs. To tackle this issue, we propose a job-aware optimization approach, which includes static and dynamic file striping. Static file striping optimizes all user jobs, whereas dynamic file striping employs clustering of job names and job paths to extract similarities among jobs and predict partially stripe-optimizable jobs for users. Additionally, a stripe recovery mechanism is employed to mitigate the negative impact of stripe misconfigurations. This approach appropriately adjusts the file stripe layout based on the job’s I/O pattern, allowing for better mobilization of underutilized OSTs to enhance parallel I/O capabilities. Through experimental verification, the number of OSTs that jobs can use has been increased, effectively improving the parallel I/O performance of the job without significantly affecting operational stability.

ЕФЕКТИВНІ СХОВИЩА ДАНИХ ДЛЯ РІШЕНЬ МАШИННОГО НАВЧАННЯ

Awareness of the importance of using effective data warehouses to develop machine learning solutions in today's world is becoming increasingly relevant in connection with the growth of data volumes and the growing need for accurate and productive models. The article identifies major vital requirements for data storage systems to ensure effective data management in building machine learning solutions. The study underscores the practical implications of rapid data access and minimal latency, reliable storage systems with robust backup and recovery mechanisms, high-level security, flexibility in handling different data types, and cost-effectiveness. These factors are not just theoretical considerations but directly impact the efficiency and success of machine learning solutions. Together with describing the concepts of block storage and object storage, the article also compares major cloud platforms—AWS (Amazon Web Services), Microsoft Azure, and Google Cloud Platform (GCP)—each providing a comprehensive range of flexible and customizable storage, security, and data processing services. These platforms enable users to manage their information resources flexibly and efficiently, offering unique features that cater to the specific needs of machine learning applications. Understanding the nuances of each provider's offerings empowers engineers and researchers to make informed decisions that align with their unique requirements. The article provides a thorough overview of the key considerations and available options, guiding strategic decisions to support machine learning initiatives effectively. Also the importance of staying up to date with the evolution of major platforms offerings is emphasized. In conclusion, selecting the proper data storage solution is critical for enhancing the performance, security, and cost-efficiency of machine learning models.

Exploring AWS S3: In-Depth Analysis of Bucket Management

Amazon Simple Storage Service (S3) is a cornerstone in modern data management strategies with in Amazon Web Services (AWS). It is pivotal in providing scalable, durable, and accessible object storage infrastructure. A key advantage of S3 is its scalability, enabling seamless storage capacity expansion without upfront investments or provisioning delays. This flexibility empowers organizations to accommodate dynamic data growth while maintaining operational agility, which is crucial in today's rapidly evolving digital landscape. S3's exceptional durability, boasting an impressive 99.999999999% reliability, ensures steadfast data integrity and resilience against potential failures, mitigating concerns about data loss and bolstering confidence in managing mission-critical information effectively. Accessibility is another crucial aspect, with robust access controls and APIs facilitating secure data retrieval, storage, and management from any location. This accessibility fosters collaboration and innovation by enabling seamless data interaction and utilization across geographical boundaries.S3's integration capabilities with other AWS services enhance operational efficiency and optimize data analytics workflows. It is a foundational element in data analytics, facilitating data ingestion, storage, and processing to derive actionable insights from diverse datasets. Furthermore, S3's versatility extends beyond storage, supporting applications such as content delivery, backup and archiving, disaster recovery, and machine learning model training. This versatility drives innovation across industries and underscores S3's role in driving digital transformation and fostering a culture rooted in data-driven decision-making. In conclusion, Amazon S3's unparalleled scalability, durability, and accessibility make it indispensable for organizations navigating the data-centric landscape of modern business environments, underlining the imperative of harnessing its full potential for organizational success.

Modeling of functional language interpreter with metaprogramming

The purpose of the research consists of modeling an interpreter for a functional programming language with metaprogramming capabilities and analyzing ways to implement primitive operators based on macros.Methods. A formal model of a functional language interpreter, which is a subset of Common Lisp, was developed with denotational semantics, which allows you to accurately describe the behavior of the interpreter when calculating language elements such as quoting, accessing variables, sequence of actions, branching, assignment, abstraction, application.Results. Based on denotational semantics, the architecture of a functional language interpreter with metaprogramming capabilities was developed. Numbers, symbols, pairs, strings and arrays were chosen as the basic types of objects. To store objects, a tag architecture was used, where the low-order bits of the object address are always zero, so they can store the object type code and the tag bit. Objects are allocated and freed automatically: a mark and cleanup algorithm is used for garbage collection. Using macros, branching operators, complete and incomplete, selection operator, and block operators of related variables were implemented. Conclusion. As a result of the work, a functional language interpreter with metaprogramming capabilities was implemented. Using macros, primitive operators of condition, selection, and a block of related variables were implemented. Using these operators as an example, it is shown that using metaprogramming, only basic forms and primitives can be built into the interpreter, and the other operators can be implemented using metaprogramming, which makes it possible to simplify and reduce the amount of interpreter code.

Self-Provisioning Infrastructures for the Next Generation Serverless Computing

Serverless computing has ushered in a transformative paradigm, with a promise to alleviate developers from the intricacies of infrastructure management. However, current serverless platforms predominantly offer only serverless compute capabilities. As a consequence, the application developers are once again tasked to explicitly provision and manage the backend services (BaaS), such as object stores or API gateways, the infrastructure, and the configuration models. This violates the main promise of serverless computing and erases much of the practical benefits of the serverless paradigm. It also introduces the challenges of managing the application execution environment, which includes maintaining provisioning and deployment scripts, configuring and managing access permissions, and scaling the services during runtime. To address these challenges, in this paper we introduce a novel paradigm for the next generation of serverless computing, called self-provisioning infrastructure. The self-provisioning infrastructure is an infrastructure that is capable to automatically and autonomously (with zero-configuration and zero-touch) provision serverless functions, their infrastructure, and their supporting BaaS services. To achieve this vision, we introduce novel design principles, models, and mechanisms that are formalized via novel programming, function, and system models. With this novel paradigm, we intend to fortify the core design principles of serverless computing but also extend them to the entire application execution environment. By doing so, we aim to enable the next-generation serverless computing in the Edge-Cloud continuum.

Caching or not: An online cost optimization algorithm for geodistributed data analysis in cloud environments

With the wide application of big data technology, a large number of data geographically stored in data centers across various regions are generated everyday, waiting to be analyzed by big data tasks. Examples of such data analysis tasks include weather prediction and intelligent healthcare applications. Clouds are being used by more and more enterprises due to their nearly infinite resources, ease of scaling, and other characteristics. Organizations often rent StaaS (Storage-as-a-Service) storage products offered by cloud providers, such as OSS (Object Storage Service), for massive data storage, while building a big data cluster in cloud environments to analyze the collected data from various regions. However, when the data to be analyzed are not located in the rented cluster, how to efficiently and economically process the distributed input data stored in clouds becomes an urgent problem to be solved. A simple approach is to only cache frequently accessed data rather than all data from other regions into the cluster to reduce total traffic costs. However, it is generally very hard to predict future data access curves. Thus, a rash caching decision may incur more costs. To address this problem, in this paper we propose an online algorithm for guiding cloud users to make cost-effective caching decisions properly, while not requiring any future information. We prove theoretically that the competitive ratio of our online algorithm is less than 2. Finally we verify the effectiveness of our proposed algorithm through extensive experiments based on the real price of Alibaba’s public IaaS cloud products using both real-world Yahoo S2 data and synthesized datasets.

Analysis-ready VCF at Biobank scale using Zarr.

Variant Call Format (VCF) is the standard file format for interchanging genetic variation data and associated quality control metrics. The usual row-wise encoding of the VCF data model (either as text or packed binary) emphasises efficient retrieval of all data for a given variant, but accessing data on a field or sample basis is inefficient. Biobank scale datasets currently available consist of hundreds of thousands of whole genomes and hundreds of terabytes of compressed VCF. Row-wise data storage is fundamentally unsuitable and a more scalable approach is needed. We present the VCF Zarr specification, an encoding of the VCF data model using Zarr which makes retrieving subsets of the data much more efficient. Zarr is a cloud-native format for storing multi-dimensional data, widely used in scientific computing. We show how this format is far more efficient than standard VCF based approaches, and competitive with specialised methods for storing genotype data in terms of compression ratios and calculation performance. We demonstrate the VCF Zarr format (and the vcf2zarr conversion utility) on a subset of the Genomics England aggV2 dataset comprising 78,195 samples and 59,880,903 variants, with a 5X reduction in storage and greater than 300X reduction in CPU usage in some representative benchmarks. Large row-encoded VCF files are a major bottleneck for current research, and storing and processing these files incurs a substantial cost. The VCF Zarr specification, building on widely-used, open-source technologies has the potential to greatly reduce these costs, and may enable a diverse ecosystem of next-generation tools for analysing genetic variation data directly from cloud-based object stores.

Cloud Storage for Object Detection using ESP32-CAM

Cloud storage services can create an object storage bucket to store our pictures, among them the Cloud Storage FUSE, Scaleway, S3 bucket, Firebase, etc. intelligent IoT systems generate vast amounts of multi-source industrial data, which necessitate a large amount of storage and processing power to enable real-time data processing and analysis. Cloud computing can be intricately linked into intelligent IIoT systems due to its strong computational and storage capabilities. Cloud Storage for Object Detection using ESP32-CAM. Create a workable solution that supports distributed storage bucket and implement it in a real-world setting. Implement the entire system as an addition to the well-known IoT cloud storage and run multiple experiments to evaluate its functionality in scenarios with varying setups and system. The target objects that are used as data sets are the ESP8266, Wemos D1, and Arduino Uno. Figuring out the ideal parameters for training the FOMO (First Object, More Object) model and then putting it into practice. It was necessary to find a balance between learning rate and accuracy, on the other hand, to maintain the highest possible accuracy in the identification of the microcontroller object to minimise the number of false positive reports. Find the value learning rate effective to this object is 0.01 with F1 score 98.7% and accuracy score 89.58%.

X Meta : SSD-HDD-hybrid Optimization for Metadata Maintenance of Cloud-scale Object Storage

Object storage has been widely used in the cloud. Traditionally, the size of object metadata is much smaller than that of object data, and thus existing object storage systems (such as Ceph and Oasis) can place object data and metadata, respectively, on hard disk drives (HDDs) and solid-state drives (SSDs) to achieve high I/O performance at a low monetary cost. Currently, however, a wide range of cloud applications organize their data as large numbers of small objects of which the data size is close to (or even smaller than) the metadata size, thus greatly increasing the cost if placing all metadata on expensive SSDs. This article presents x Meta , an SSD-HDD-hybrid optimization for metadata maintenance of cloud-scale object storage. We observed that a substantial portion of the metadata of small objects is rarely accessed and thus can be stored on HDDs with little performance penalty. Therefore, x Meta first classifies the hot and cold metadata based on the frequency of metadata accesses of upper-layer applications and then adaptively stores the hot metadata on SSDs and the cold metadata on HDDs. We also propose a merging mechanism for hot metadata to further improve the efficiency of SSD storage and optimize range key query and insertion for hot metadata by designing composite keys. We have integrated the x Meta metadata service with Ceph to realize a high-performance, low-cost object store (called xCeph). The extensive evaluation shows that xCeph outperforms the original Ceph by an order of magnitude in the space requirement of SSD storage, while improving the throughput by up to 2.7×.

Comprehensive Analysis of Distributed Object Storage Systems

Distributed object storage systems have emerged as pivotal infrastructures for managing the escalating volumes of unstructured data. This research comprehensively explores the architecture, challenges, advancements, and applications of distributed object storage. The architectural analysis delineates core components, such as metadata servers and storage nodes, emphasizing their role in facilitating scalability and fault tolerance. Challenges encompassing data consistency, security, and performance bottlenecks underscore the need for continual innovation. Advancements, ranging from erasure coding to the integration of machine learning and blockchain, propel the field forward, enhancing resilience and expanding applications. Use cases illustrate the adaptability of distributed object storage across industries, while future directions suggest potential areas for exploration. In conclusion, distributed object storage epitomizes a foundational technology in modern data management, with the research delineating its current significance and future potential.

Design of storage benchmark kit framework for supporting the file storage retrieval

An open-source software framework called the storage benchmark kit (SBK) is used to store the system benchmarking performance framework. The SBK is designed to perform any storage client or device using any data type as a payload. SBK simultaneously helps number of readers as well as writes to the storage system of large amounts of data as well as allows end-to-end latency benchmarking for multiple writers and readers. The SBK uses standardized performance measures for comparing and evaluating various storage systems and their combinations. Distributed file systems, distributed database systems, single or local node databases, systems of object storage, platforms of distributed streaming and messaging, and systems of key-value storage are the storage solutions supported by SBK. The SBK supports various storage systems like XFS, Kafka streaming storage systems, and Hadoop distributed file system (HDFS) performance benchmarking. The experimental results show that a proposed method achieves execution time of 65.530 s, 40.826 s and 30.351 s for the 100k, 500k and 1000k files respectively which ensures better improvement than the existing methods such as simple data interface and distributed data protection system.

Demystifying Ceph: Unveiling the Power of Open-Source Software-Defined Storage: Systematic Literature Review

Ceph, a free and open-source software-defined storage (SDS) platform, offers a highly adaptable, distributed solution, accommodating various applications like cloud storage and virtualization. Deployment requires specific hardware, supported Linux distribution, and network reliability. Each node serves multiple roles, facilitating object storage, monitoring, and management. Configuration entails network setup, key generation, monitor map creation, and daemon configuration. While this abstract introduces Ceph's fundamentals, in-depth exploration mandates consulting official documentation for proper configuration and best practices. It delves into Ceph's architecture, features, and applications. Providing unified storage for object, block, and file data.

Abstract 3558: An integrated data platform to support the international alliance for cancer early detection

Abstract Cancer early detection is one of the most critical areas of cancer research, as it offers the greatest potential for improving patient outcomes. The International Alliance for Cancer Early Detection (ACED) is a global partnership of world-leading cancer research institutions in the UK and US, established in 2019 to accelerate and revolutionize research in this field. ACED brings together the expertise of the Canary Center at Stanford University, the University of Cambridge, the Knight Cancer Institute at Oregon Health and Sciences University, University College London, and the University of Manchester, together with Cancer Research UK, with the goal of catalyzing new collaborations and research in the field of early detection. One of the major challenges to enabling international collaborations across different institutions is the ability to collect, share and discover datasets between researchers and institutions. To enable collaboration on innovative data science, fundamental functions include, 1) controlled data sharing mechanisms; 2) structured metadata to enable data exploration and data discovery; and 3) enabling easy computational access to that data. To advance researchers' collaboration, ACED began development of an Integrated Data Platform (IDP). The ACED-IDP is based on the Gen3 software platform developed by the University of Chicago's Center for Translational Data Science. Based on software systems originally developed for the NCI's Genomic Data Commons, Gen3 has been used in several other data projects including The Blood Profiling Atlas in Cancer (BloodPAC), Australian BioCommons and numerous other research data platforms. The unique nature of ACED required unique innovations to be made for the development of the IDP. Each of the member institutions within the alliance has different existing computer infrastructures, separate authentication platforms and heterogeneous data types forming the basis of their research. The IDP sought a cloud-ready strategy, while still being cognizant of the extreme costs associated with cloud egress fees that hamper researchers' ability to download data. Additions were made to the Gen3 platform to allow for hybrid cloud support, allowing on-premises as well as cloud object storage systems to be linked to the platform. This innovation permits each institution to share files using the mechanisms that they see fit. To unify the various datasets, the standard Gen3 schema was replaced with one derived from the Fast Healthcare Interoperability Resources (FHIR) standard. To support import from clinical data sets, tooling to enable import and export Observational Medical Outcomes Partnership (OMOP) data has been integrated. With this platform in place, we hope to advance international collaborations and accelerate early cancer detection research. Citation Format: Brian Walsh, Liam Beckman, JD Burchett, Matthew Peterkort, Jordan Lee, Michael Fitzsimons, Peter Vassilatos, Binam Bajracharya, Craig Barnes, Jawad Qureshi, Robert Grossman, Carrie Yakura, Yaozhi Lu, Sarah Burge, Daniel Kelberman, Erin Watson, Kyle Ellrott. An integrated data platform to support the international alliance for cancer early detection [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 3558.

SkyPIE: A Fast & Accurate Oracle for Object Placement

Cloud object stores offer vastly different price points for object storage as a function of workload and geography. Poor object placement can thus lead to significant cost overheads. Prior cost-saving techniques attempt to optimize placement policies on the fly, deciding object placements for each object individually. In practice, these techniques do not scale to the size of the modern cloud. In this work, we leverage the static nature and pay-per-use pricing model of cloud environments to explore a different approach. Rather than computing object placements on the fly, we precompute a SkyPIE oracle---a lookup structure representing all possible placement policies and the workloads for which they are optimal. Internally, SkyPIE represents placement policies as a matrix of cost-hyperplanes, which we effectively precompute through pruning and convex optimization. By leveraging a fast geometric algorithm, online queries then are 1 to 8 orders of magnitude faster but as accurate as Integer-Linear-Programming. This makes exact optimization tractable for real workloads and we show >10x cost savings compared to state-of-the-art heuristic approaches.

Public cloud object storage auditing: Design, implementation, and analysis

Cloud storage auditing is a technique that enables a user to remotely check the integrity of the outsourced data in the cloud storage. Although researchers have proposed various protocols for cloud storage auditing, the proposed schemes are theoretical in nature, which are not fit for existing mainstream cloud storage service practices. To bridge this gap, this paper proposes a cloud storage auditing system that works for current mainstream cloud object storage services. We design the proposed system over existing proof of data possession (PDP) schemes and make them practical as well as usable in the real world. Specifically, we propose an architecture that separates the compute and storage functionalities of a storage auditing scheme. Because cloud object storage only provides read and write interfaces, we leverage a cloud virtual machine to implement the user-defined computations that are needed in a PDP scheme. We store the authentication tags of the outsourced data as an independent object to allow existing popular cloud storage applications, e.g., file online previewing. We also present a cost model to analyze the economic cost of a cloud storage auditing scheme. The cost model allows a user to balance security, efficiency, and economic cost by tuning various system parameters. We implemented, open-sourced the proposed system over a mainstream cloud object storage service. Experimental analysis shows that the proposed system is pretty efficient and promising for a production environment usage. Specifically, for a 40 GB sized data, the proposed system only incurs 1.66% additional storage cost, 3796 bytes communication cost, 2.9 seconds maximum auditing time cost, and 0.9 CNY per auditing monetary cost.