Virtualization in Distributed Hot and Cold Storage for IoT Data Retrieval without Caching

Abstract

This paper proposes a data storage method that cost-efficiently manages a massive amount of short lifetime data continuously generated typically by IoT devices (called Live Data, e.g., surveillance camera images) considering real-time data retrieval. Along with the rapid growth of the IoT, many users expect to shift to an Open IoT in which they can share enormous amounts of Live Data. To efficiently use Live Data, it is important to use both hot and cold data storage. Data should be stored in cold storage, which is less expensive but has slow access speeds than hot storage, only if the data are unlikely to be used. The proposed method virtualizes distributed hot storage to provide two data placement mechanisms; (1) an original-data-only optimized allocation according to demand by migrating original data to the appropriate distributed hot storage, and (2) integrated data replacement in the case of capacity overflow by migrating data to the distributed hot storage storing less useful data. As a result of a performance evaluation, compared with a cache algorithm often used for traditional data, the proposed method can reduce operational costs (transmission costs for data migration and retrieval and miss–hit penalty costs) when the ratio of single miss–hit penalty costs to 1-hop transmission costs is more than 6.