Video File Allocation for Wear-Leveling in Distributed Storage Systems With Heterogeneous Solid-State-Disks (SSDs)

Abstract

With the advent of new large-capacity solid-state disks (SSDs) such as quad-level-cells (QLC), SSD arrays can be effectively used in video storage systems that require large-capacity storage space. Typically, SSD manufacturers specify a drive-writes-per-day (DWPD) metric, which is the ratio of bytes written per day to the total capacity in bytes, to ensure an SSD’s specified lifetime; it is important to limit the number of write operations by considering the DWPD for each SSD. We propose a new video file allocation technique to effectively manage the heterogeneous DWPD characteristics of SSDs in distributed storage systems. To express the degree of wear-leveling for heterogeneous SSDs, we first introduce the concept of ADWD, which is the actual number of bytes written per day compared to DWPD. We then propose two algorithms for file placement and migration. The file placement algorithm places files greedily based on the bandwidth-to-space ratio (BSR) of each file and SSD to balance the bandwidth usage and storage of the SSD. The file migration algorithm moves files from overloaded to underloaded SSDs to meet bandwidth limit requirements while minimizing the overall ADWD as a result of migration, and then migrates additional popular files to improve SSD bandwidth utilization. To use these algorithms in actual distributed file systems, we implemented a suite of tools for file placement and migration in the Hadoop distributed file system (HDFS). Experimental results show that the proposed algorithm reduces the mean of ADWD by 35.44% and its standard deviation by 69.78% compared to the benchmark methods on average.