A high‐performance distributed file system for large‐scale concurrent HD video streams

Abstract

SummaryWith the rapid development of intelligent transportation technology, high‐definition data storage, and processing of massive amounts of video surveillance have become key issues. When thousands of high‐bit‐rate video streaming concurrently writes, disk I/O throughput becomes a bottleneck. In addition, this leads to serious energy consumption and disk abrasion. To solve these problems, a new distributed file system for high concurrent and high‐bit‐rate writing is designed. It combines an optimized data storage model, efficient metadata management, and exquisite disk schedule mechanism. The optimized data storage model uses a file pre‐allocation strategy and multiple‐stream input modulating technology to convert the randomly concurrent writes on the disk into sequential writes; the metadata management provides an efficient means of retrieving the specified data; and the dual‐partition schedule mechanism can ensure the disk stability with less abrasion. Through this distributed file system, the disk I/O throughput can be saturated in a high concurrent writing environment. The performance evaluation results demonstrate that the I/O throughput of a normal 7200RPM SATA III disk in our scheme can be stabilized at 150MB/s, easily to support 300 concurrent high‐definition video streams (4Mbit/s each). The distributed file system with eight commodity servers can afford the ability of supporting 8000 high‐definition video streams concurrently writing, which is far greater than the existing video surveillance storage solutions. Copyright © 2015 John Wiley & Sons, Ltd.