Data Placement Algorithm Research Articles

Flimm: Foreground traffic aware data migration manager for distributed storage system

Currently, decentralized distributed storage system is widely used in various scenarios, because decentralized data placement algorithms observably improve scalability and robustness of distributed storage system. However, storage system applied with decentralized data placement algorithm suffers from uncontrollable data migration during expanding the clusters, which lead to dramatic fluctuation of performance. To maintain reliability and stability of storage system, administrators are under great pressure to migrate data during expansion or contraction. Not only the urgent time constraints to expansion but also low interference to foreground application are required during data migration period. It is a difficult challenge for in-experienced administrators to tackle.To address this challenge, this paper introduces Flimm, a fast and low-interference data migration manager that is aware of storage resource utilization and performs data migration with low interference to foreground applications. Flimm mitigates interference by scheduling a subset of migration tasks with careful consideration. Moreover, Flimm adapts to varying foreground application workloads through adaptive migration speed control, providing administrators with convenient and secure parameters for tuning. In our experiments, Flimm demonstrated a 1.2x acceleration in migration speed and achieved a 20% reduction in interference.

Enhancing Scalability through Extended Hybrid Switching with Mixture Path Deployment (EHS-MPD): A Novel Framework

In today's digital landscape, the exponential growth of data has created significant challenges for archiving systems. Existing solutions often struggle to cope with the ever-increasing data volumes, leading to scalability limitations, reduced efficiency, and inflexibility. To address these challenges, this research paper introduces a novel approach that extends hybrid switching with a mixture path deployment framework. By combining the benefits of hybrid switching techniques and the adaptability of the mixture path deployment framework, our proposed approach aims to enhance scalability, efficiency, and flexibility in archiving systems while minimizing the risks of data loss and system failures. This paper provides a comprehensive analysis of the design, implementation, and evaluation of our approach, highlighting its key features, benefits, and potential impact on archiving systems. We review existing archiving approaches, identifying their limitations and shortcomings. Then, we delve into the concept of hybrid switching, exploring its advantages in optimizing network performance. Additionally, we introduce the mixture path deployment framework, emphasizing its benefits in load balancing, fault tolerance, and performance improvement. The proposed approach's architecture is presented, showcasing the integration of hybrid switching techniques and the mixture path deployment framework. Strategies for efficient data placement and management are discussed, including dynamic data placement algorithms and metadata management techniques. The design and implementation of the extended archiving system are described, outlining the interactions among its components. Overall, our novel approach to archiving high scalability through extending hybrid switching with a mixture path deployment framework offers a promising solution to address the limitations of existing archiving systems. It enables improved performance, flexibility, and fault tolerance, paving the way for efficient and scalable archiving solutions in the era of big data.

Parallelism-Optimizing Data Placement for Faster Data-Parallel Computations

Systems performing large data-parallel computations, including online analytical processing (OLAP) systems like Druid and search engines like Elasticsearch, are increasingly being used for business-critical real-time applications where providing low query latency is paramount. In this paper, we investigate an underexplored factor in the performance of data-parallel queries: their parallelism. We find that to minimize the tail latency of data-parallel queries, it is critical to place data such that the data items accessed by each individual query are spread across as many machines as possible so that each query can leverage the computational resources of as many machines as possible. To optimize parallelism and minimize tail latency in real systems, we develop a novel parallelism-optimizing data placement algorithm that defines a linearly-computable measure of query parallelism, uses it to frame data placement as an optimization problem, and leverages a new optimization problem partitioning technique to scale to large cluster sizes. We apply this algorithm to popular systems such as Solr and MongoDB and show that it reduces p99 latency by 7-64% on data-parallel workloads.

Load Balancing Cloud Storage Data Distribution Strategy of Internet of Things Terminal Nodes considering Access Cost.

With the rapid development of Internet of Things (IoT) technology, IoT terminal nodes are facing many challenges in data storage, distribution, and data management. In particular, in the IoT terminal nodes considering access cost, the corresponding data distribution and storage are professional, complex, and miscellaneous. Based on the abovementioned current situation, this article innovatively proposes a complex sensor data placement algorithm based on the cloud storage distribution of IoT terminal nodes. Under this algorithm, the accurate division of IoT data I/O methods is realized through reasonable configuration. Through the adaptive sensing algorithm, while fully considering the access cost of the algorithm, the performance of the IoT data storage system is further optimized. In the corresponding terminal node load balancing problem, this article innovatively proposes the terminal node data sorting and distribution algorithm through the node data. The sorting and distribution algorithm realizes the precise segmentation of the IoT data to be processed, thereby realizing the improvement of data reading and processing speed. Based on the proposed algorithm, this article designs a load balancing cloud storage data distribution optimization system of IoT terminal nodes considering access cost and carries out experimental verification in a real environment. The experimental results show that the data pattern division accuracy corresponding to the proposed distribution strategy is improved to 97.13% and the corresponding data access efficiency is improved to 98.3%, compared with the traditional distribution strategy. Therefore, the data distribution strategy proposed in this article has obvious performance advantages and further promotion value.

RENDA: Resource and Network Aware Data Placement Algorithm for Periodic Workloads in Cloud

The Hadoop enabled cloud platforms are gradually becoming preferred computational environment to execute scientific big data workloads in a periodic manner. However, it is observed that the default data placement approach of such cloud platforms is not the efficient one and often ends up with significant data transfer overhead leading to degradation of the overall job completion time. In this article, a Resource and Network-aware Data Placement Algorithm (RENDA) is proposed to reduce the non-local executions and thereby reduce the overall job completion time for periodic workloads in the cloud environment. The entire job execution is modeled as a two-stage execution characterized as data distribution and data processing. The RENDA reduces the time of the stages as mentioned above by estimating the heterogeneous performance of the nodes on a real-time basis followed by careful allocation of data in several installments to participating nodes. The experimental results show that the proposed RENDA algorithm consistently outperforms over the recent state-of-the-art alternatives with as much as 28 percent reduction in data transfer overhead leading to 16 percent reduction in average job completion time with 27 percent average speedup on average job execution.

A machine learning assisted data placement mechanism for hybrid storage systems

Emerging applications produce massive files that show different properties in file size, lifetime, and read/write frequency. Existing hybrid storage systems place these files onto different storage mediums assuming that the access patterns of files are fixed. However, we find that the access patterns of files are changeable during their lifetime. The key to improve the file access performance is to adaptively place the files on the hybrid storage system using the run-time status and the properties of both files and the storage systems. In this paper, we propose a machine learning assisted data placement mechanism that adaptively places files onto the proper storage medium by predicting access patterns of files. We design a PMFS based tracer to collect file access features for prediction and show how this approach is adaptive to the changeable access pattern. Based on data access prediction results, we present a linear data placement algorithm to optimize the data access performance on the hybrid storage mediums. Extensive experimental results show that the proposed learning algorithm can achieve over 90% accuracy for predicting file access patterns. Meanwhile, this paper can achieve over 17% improvement of system performance for file accesses compared with the state-of-the-art linear-time data placement methods.

Temporal Pattern Classification and Analysis of Large Volume of Data Using Associated Data Placement Algorithm and Online Community Adjustment

Aim: The main motto of the study is to optimize the large volume of data using data placement algorithm and online community adjustment algorithm and comparing their accuracy. Materials and Methods: Data placement algorithm (N=10) and online community adjustment (N=10) was iterated 20 times to optimize the data. Result and Discussion: Data placement algorithm has significantly better accuracy (85%) compared to online community adjustment algorithm(78%). The statistical significance of data placement (p<0.02 independent sample test) is high. Conclusion: With the limits of the study, a data placement algorithm with product manufacturing data offers the best accuracy in data optimization.

Optimizing the data placement and scheduling on multi-port DWM in multi-core embedded system

Applications run in embedded systems should usually be completed within the constrained time to reduce energy consumption. Therefore, the memory of embedded system should satisfy time-efficient and energy-efficient. Domain Wall Memory (DWM) achieves high energy efficiency, high density, non-volatile and low cost advantages. However, the data access on DWM always requires shift operations to align the domain with the port which can access the data in the domain. In the case that there are multi-core processors with multi-port DWM, the data placement and scheduling, therefore, can extremely affect the parallelism, total execution time and performance. In this paper, we handle the data placement and scheduling on multi-port DWM in multi-core system (DPSMDMS), and provide the integer linear programming (ILP) algorithm to solve this problem optimally. What is more, we propose a layering and grouping for data placement and scheduling (LGPS) heuristic algorithm to get the approximate optimal solution in polynomial time. In benchmarks, the experimental results show that ILP and LGPS reduce on average 64.6% and 64.8% of the total execution time, respectively, compared with the non-optimized strategy.

A Consistent Hash Data Placement Algorithm Based on Stripe

A Consistent Hash Data Placement Algorithm Based on Stripe

State Management for Cloud-Native Applications

The stateless cloud-native design improves the elasticity and reliability of applications running in the cloud. The design decouples the life-cycle of application states from that of application instances; states are written to and read from cloud databases, and deployed close to the application code to ensure low latency bounds on state access. However, the scalability of applications brings the well-known limitations of distributed databases, in which the states are stored. In this paper, we propose a full-fledged state layer that supports the stateless cloud application design. In order to minimize the inter-host communication due to state externalization, we propose, on the one hand, a system design jointly with a data placement algorithm that places functions’ states across the hosts of a data center. On the other hand, we design a dynamic replication module that decides the proper number of copies for each state to ensure a sweet spot in short state-access time and low network traffic. We evaluate the proposed methods across realistic scenarios. We show that our solution yields state-access delays close to the optimal, and ensures fast replica placement decisions in large-scale settings.

Novel Data Placement Algorithm for Distributed Storage System Based on Fault-Tolerant Domain

The 3-replica redundancy strategy is widely used to solve the problem of data reliability in large-scale distributed storage systems. However, its storage capacity utilization is only 33%. In this paper, a data placement algorithm based on fault-tolerant domain (FTD) is proposed. Owing to the fine-grained design of the FTD, the data reliability of systems using two replicas is comparable to that of current mainstream systems using three replicas, and the capacity utilization is increased to 50%. Moreover, the proposed FTD provides a new concept for the design of distributed storage systems. Distributed storage systems can take FTDs as the units for data placement, data migration, data repair and so on. In addition, fault detection can be performed independently and concurrently within the FTDs.

ESetStore: An Erasure-Coded Storage System With Fast Data Recovery

Erasure codes have been used extensively in large-scale storage systems to reduce the storage overhead of triplication-based storage systems. One key performance issue introduced by erasure codes is the long time needed to recover from a single failure, which occurs constantly in large-scale storage systems. We present ESetStore, a prototype erasure-coded storage system that aims to achieve fast recovery from failures. ESetStore is novel in the following aspects. We proposed a data placement algorithm named ESet for our ESetStore that can aggregate adequate I/O resources from available storage servers to recover from each single failure. We designed and implemented efficient read and write operations on our erasure-coded storage system via effective use of available I/O and computation resources. We evaluated the performance of ESetStore with extensive experiments on a cluster with 50 storage servers. The evaluation results demonstrate that our recovery performance can obtain linear performance growth by harvesting available I/O resources. With our defined parameter recovery I/O parallelism under some mild conditions, we can achieve optimal recovery performance, in which ESet enables minimal recovery time. Rather than being an alternative to improve recovery performance, our work can be an enhancement for existing solutions, such as Partial-parallel-repair (PPR), to further improve recovery performance.

Data placement in distributed data centers for improved SLA and network cost

Large-scale data-intensive applications provide services to users by routing service requests to geographically distributed data centers interconnected by Internet links. In order to achieve good reliability and data access latency performance, cloud service providers often simultaneously place multiple copies of the data in different data centers. The network communication required for updating the multiple data copies incurs an operational cost. At the same time, the penalty incurred by the Service Level Agreement (SLA) violation for data access from the data centers also imposes an operational cost on the service providers. In this paper, we tackle the problem of data placement in distributed data centers with the aim to minimize the operational cost incurred by delay SLA violation penalty and inter-data center network communication, assuming each data has K data replicas. We propose a K-level Cluster-based Data Placement algorithm (K-CDP) for the problem. The algorithm solves the linear programming relaxation and dual programming problems corresponding to the problem of minimizing SLA violation penalty cost caused by placing a replica of each data in a data center. Based on the obtained solutions, the algorithm clusters the data so that the data with similar placeable data centers form a data cluster. For the data in each cluster, the algorithm selects K data centers to minimize the operational cost. We prove that algorithm K-CDP is 2-approximation to the data placement problem. Our simulation results demonstrate that the proposed algorithm can effectively reduce the penalty cost incurred by delay SLA violation, the network communication cost, and the operational cost of data centers.

Optimal Data Placement for Heterogeneous Cache, Memory, and Storage Systems

New memory technologies are blurring the previously distinctive performance characteristics of adjacent layers in the memory hierarchy. No longer are such layers orders of magnitude different in request latency or capacity. Beyond the traditional single-layer view of caching, we now must re-cast the problem as a data placement challenge: which data should be cached in faster memory if it could instead be served directly from slower memory? We present Chopt, an offline algorithm for data placement across multiple tiers of memory with asymmetric read and write costs. We show that Chopt is optimal and can therefore serve as the upper bound of performance gain for any data placement algorithm. We also demonstrate an approximation of Chopt which makes its execution time for long traces practical using spatial sampling of requests incurring a small 0.2% average error on representative workloads at a sampling ratio of 1%. Our evaluation of Chopt on more than 30 production traces and benchmarks shows that optimal data placement decisions could improve average request latency by 8.2%-44.8% when compared with the long-established gold standard: Belady and Mattson's offline, evict-farthest-in-the-future optimal algorithms. Our results identify substantial improvement opportunities for future online memory management research.

Optimal Data Placement for Heterogeneous Cache, Memory, and Storage Systems

New memory technologies are blurring the previously distinctive performance characteristics of adjacent layers in the memory hierarchy. No longer are such layers orders of magnitude different in request latency or capacity. Beyond the traditional single-layer view of caching, we now must re-cast the problem as a data placement challenge: which data should be cached in faster memory if it could instead be served directly from slower memory? We present CHOPT, an offline algorithm for data placement across multiple tiers of memory with asymmetric read and write costs. We show that CHOPT is optimal and can therefore serve as the upper bound of performance gain for any data placement algorithm. We also demonstrate an approximation of CHOPT which makes its execution time for long traces practical using spatial sampling of requests incurring a small 0.2% average error on representative workloads at a sampling ratio of 1%. Our evaluation of CHOPT on more than 30 production traces and benchmarks shows that optimal data placement decisions could improve average request latency by 8.2%-44.8% when compared with the long-established gold standard: Belady and Mattson's offline, evict-farthest-in-the-future optimal algorithms. Our results identify substantial improvement opportunities for future online memory management research.

A Data Placement Strategy Based on Crow Search Algorithm in Cloud Computing

Background: In cloud computing era, large scale scientific applications process large amount of data in the data centers. Placing of this data onto a data center is a critical issue performed as part of workflow management system and aims to find the best data center to place the data. It has a direct impact on performance, cost and execution time of workflows. Methods: In this paper, a novel data placement strategy is proposed based on Crow Search Algorithm (CSA) that dynamically distributes the data sets to appropriate data centers during runtime stage of workflow. Results: The results obtained b y CSA based run time data placement algorithm are evaluated with the results of other algorithms. Simulation results acknowledge that using CSA based data placement algorithm provides appropriate results in comparison to the other algorithms.

Decoupling NDN caches via CCndnS: Design, analysis, and application

In-network caching is considered to be a vital part of the Internet for future applications (e.g., Internet of Things). One proposal that has attracted interest in recent years, Named Data Networking (NDN), aims to facilitate in-network caching by locating content by name. However, the efficiency of in-network caching has been questioned by experts. Data correlation among caches builds strong dependencies between caches at the edge and in the core. That dependency makes analyzing network performance difficult.This paper proposes CCndnS (Content Caching strategy for NDN with Skip), a caching policy to break the dependencies among caches, thus facilitating the design of an efficient data placement algorithm. Specifically, each cache – regardless of its location in the network – should receive an independent set of requests; otherwise, only misses from downstream caches make their way to the upstream caches, i.e. a filtering effect that induces a correlation among the caches.CCndnS breaks a file into smaller segments and spreads them in the path between requester and publisher in a way that the head of the file (the first segment) should be cached at the edge router close to the requester and the tail far from the requester and towards the content provider. Requests for a segment skip searching caches in its path, to search only the cache with the segment of interest. That reduces the number of futile checks on caches, and thus the delay from memory accesses. This mechanism also decouples the caches, so there is a simple analytical model for cache performance in the network.We illustrate an application of the model to enforce a Service Level Agreement (SLA) between a content provider and the caching system proposed in this paper. The model can be used for cache provisioning for two purposes: (1) To specify the cache size to be reserved for specific contents to reach some desired performance. For instance, if the client of an SLA requires a 50% cache hit for its content at each router, the model can be used to determine the cache size that needs to be reserved to reach the 50% hit rate. (2) To calculate the effect of such reservations on other contents that use the routers covered by the SLA.The design, analysis, and application are tested with extensive simulations.

Sensing Cloud Computing in Internet of Things: A Novel Data Scheduling Optimization Algorithm

In order to addressing the issues of data matching deviation and load imbalance during the data scheduling process of the Internet of Things, Sensing Cloud Computing in IoT: A Novel Data Scheduling Optimization (SCC-DSO) Algorithm is proposed in this paper. First, according to the processing capacity of the IoT working node, a data placement algorithm is designed to reasonably place the input data of the job. Second, the data scheduling queue is optimized based on the data block storage location information to reduce non-local execution of data scheduling. Furthermore, a data prefetching method is designed to pull the data required for non-local data scheduling in advance, and shorten the waiting time of the task for input data. Finally, simulation experiment evaluated by the task localization execution rate and response time is performed. The effectiveness and stability of the algorithm is verified compared with other algorithms.

Data Placement in Era of Cloud Computing: a Survey, Taxonomy and Open Research Issues

In cloud computing, data placement is a critical operation performed as part of workflow management and aims to find the best physical machine to place the data. It has direct impact on performance, cost and execution time of workflows. Number of data placement algorithms is designed in cloud computing environment that aimed to improve various factors affecting the workflows and their execution including the movement of data among data centers. This paper provides a complete survey and analyses of existing data placement schemes proposed in literature for cloud computing. Further, it classifies data placement schemes based on their assess capabilities and objectives. Further objectives and properties of data placement schemes are compared. Finally future research directions are provided with concluding remarks.

Implementation of Effective Data Emplacement Algorithm in Heterogeneous Cloud Environment

This paper is concerned with the study and implementation of effective Data Emplacement Algorithm in large set of databases called Big Data and proposes a model for improving the efficiency of data processing and storage utilization for dynamic load imbalance among nodes in a heterogeneous cloud environment. With the era of explosive information and data receiving, more and more fields need to deal with massive, large scale of data. A method has been proposed with an improved Data Placement algorithm called Effective Data Emplacement Algorithm with computing capacity of each node as a predominant factor that promotes and improves the efficiency in data processing in a short duration time from large set of data. The adaptability of the proposed model can be obtained by minimizing the time with processing efficiency through the computing capacity of each node in the cluster. The proposed solution improves the performance of the heterogeneous cluster environment by effectively distributing data based on the performance oriented sampling as the experimental results made with word count applications.