Dynamic Data Placement Research Articles

Dynamic Data Placement Strategy with Network Security Issues in Distributed Cloud Environment for Medical Issues: An Overview

: The rapid integration of distributed cloud systems in the healthcare industry has profoundly impacted the management of valuable medical data. While this advancement has significantly improved data handling, protecting sensitive healthcare information in such a complex environment remains daunting. This comprehensive study explores the crucial intersection between dynamic data placement strategies and network security concerns in distributed cloud environments, particularly healthcare. After establishing the significance and context of this research, the survey delves into the growing need to safeguard medical data within the everevolving landscape of cloud-based healthcare systems. It lays out fundamental concepts, such as dynamic data placement and network security, highlighting their unique implications in the medical domain. Ultimately, this survey sheds light on the most effective approaches for balancing dynamic data placement and network security in the healthcare sector. This research delves into examining many tactics, evaluating their effectiveness in handling delicate medical information, and presenting tangible use cases. A key focus of this investigation is the fusion of data organization and network safety within the healthcare industry. It investigates the adaptability of dynamic data positioning techniques in fortifying network security and safeguarding against potential threats unique to the healthcare sector. Case studies of the successful implementation of these strategies in healthcare establishments are also included.

SIMPLE: Efficient Temporal Graph Neural Network Training at Scale with Dynamic Data Placement

Dynamic graphs are essential in real-world scenarios like social media and e-commerce for tasks such as predicting links and classifying nodes. Temporal Graph Neural Networks (T-GNNs) stand out as a prime solution for managing dynamic graphs, employing temporal message passing to compute node embeddings at specific timestamps. Nonetheless, the high CPU-GPU data loading overhead has become the bottleneck for efficient training of T-GNNs over large-scale dynamic graphs. In this work, we present SIMPLE, a versatile system designed to address the major efficiency bottleneck in training existing T-GNNs on a large scale. It incorporates a dynamic data placement mechanism, which maintains a small buffer space in available GPU memory and dynamically manages its content during T-GNN training. SIMPLE is also empowered by systematic optimizations towards data processing flow. We compare SIMPLE to the state-of-the-art generic T-GNN training system TGL on four large-scale dynamic graphs with different underlying T-GNN models. Extensive experimental results show that SIMPLE effectively cuts down 80.5% ~ 96.8% data loading cost, and accelerates T-GNN training by 1.8× ~ 3.8× (2.6× on average) compared to TGL.

Data management of scientific applications in a reinforcement learning-based hierarchical storage system

In many areas of data-driven science, large datasets are generated where the individual data objects are images, matrices, or otherwise have a clear structure. However, these objects can be information-sparse, and a challenge is to efficiently find and work with the most interesting data as early as possible in an analysis pipeline. We have recently proposed a new model for big data management where the internal structure and information of the data are associated with each data object (as opposed to simple metadata). There is then an opportunity for comprehensive data management solutions to account for data-specific internal structure as well as access patterns. In this article, we explore this idea together with our recently proposed hierarchical storage management framework that uses reinforcement learning (RL) for autonomous and dynamic data placement in different tiers in a storage hierarchy. Our case-study is based on four scientific datasets: Protein translocation microscopy images, Airfoil angle of attack meshes, 1000 Genomes sequences, and Phenotypic screening images. The presented results highlight that our framework is optimal and can quickly adapt to new data access requirements. It overall reduces the data processing time, and the proposed autonomous data placement is superior compared to any static or semi-static data placement policies.

Enhancing Scalability through Extended Hybrid Switching with Mixture Path Deployment (EHS-MPD): A Novel Framework

In today's digital landscape, the exponential growth of data has created significant challenges for archiving systems. Existing solutions often struggle to cope with the ever-increasing data volumes, leading to scalability limitations, reduced efficiency, and inflexibility. To address these challenges, this research paper introduces a novel approach that extends hybrid switching with a mixture path deployment framework. By combining the benefits of hybrid switching techniques and the adaptability of the mixture path deployment framework, our proposed approach aims to enhance scalability, efficiency, and flexibility in archiving systems while minimizing the risks of data loss and system failures. This paper provides a comprehensive analysis of the design, implementation, and evaluation of our approach, highlighting its key features, benefits, and potential impact on archiving systems. We review existing archiving approaches, identifying their limitations and shortcomings. Then, we delve into the concept of hybrid switching, exploring its advantages in optimizing network performance. Additionally, we introduce the mixture path deployment framework, emphasizing its benefits in load balancing, fault tolerance, and performance improvement. The proposed approach's architecture is presented, showcasing the integration of hybrid switching techniques and the mixture path deployment framework. Strategies for efficient data placement and management are discussed, including dynamic data placement algorithms and metadata management techniques. The design and implementation of the extended archiving system are described, outlining the interactions among its components. Overall, our novel approach to archiving high scalability through extending hybrid switching with a mixture path deployment framework offers a promising solution to address the limitations of existing archiving systems. It enables improved performance, flexibility, and fault tolerance, paving the way for efficient and scalable archiving solutions in the era of big data.

Optimizing data placement and size configuration for morphable NVM based SPM in embedded multicore systems

Embedded multicore systems are widely designed to meet the high-performance requirement. Meanwhile, many embedded multicore systems are equipped with multiple scratchpad memories (SPM) because of their advantages in power efficiency and small area. Whereas, the traditional SRAM-based SPM is limited by its leakage power and capacity. Compared with SRAM, emerging non-volatile memory (NVM) has lower power consumption. Moreover, some morphable NVMs like resistive memory can build a “morphable NVM” to get a balance between capacity and performance, in which a memory cell can be converted between high-density MLC (multi-level cell) and low-latency SLC (single-level cell).Considering the features of NVM, this paper proposes a dynamic data placement and size configuration technique to fully utilize the benefits from morphable NVM based SPM in embedded multicore systems. The core idea is to dynamically convert the memory mode of NVM in each SPM with different workloads, and store the associated data into an optimal storage location for minimizing the total cost of memory access. Therefore, combining with the access pattern of embedded systems, an Integer Linear Programming (ILP) model is first explored for obtaining the optimal data placement and size configuration of SLC/MLC of SPMs. Then, our polynomial-time algorithm (CDA) is provided to obtain a near-optimal result. Finally, our experiments based on the gem5 simulator exhibit the proposed techniques can achieve the performance improvement and the reduction of energy consumption compared with the baseline scheme.

Dynamic data replication and placement strategy in geographically distributed data centers

AbstractWith the evolution of geographically distributed data centers in the Cloud Computing landscape along with the amount of data being processed in these data centers, which is growing at an exponential rate, processing massive data applications become an important topic. Since a given task may require many datasets for its execution and the datasets are spread over several different data centers, finding an efficient way to manage the datasets storage across nodes of a Cloud system is a difficult problem. In fact, the execution time of a task might be influenced by the cost of data transfers, which mainly depends on two criterias. The first one is the initial placement of the input datasets during the build‐time phase, while the second is the replication of the datasets during the runtime phase. The replication is explicitly considered when datasets are being migrated over the data centers in order to make them locally available wherever needed. Data placement and data replication are important challenges in Cloud Computing. Nevertheless, many studies focus on data placement or data replication exclusively. In this paper, a combination of a data placement strategy followed by a dynamic data replication management strategy is proposed, with the purpose of reducing the associated cost of all data transfers between the (distant) data centers. Our proposed data placement approach considers the main characteristics of a data center such as storage capacity and read/write speeds to efficiently store the datasets, while our dynamic data replication management approach considers three parameters: the number of replicas in the system, the dependency between datasets and tasks and the storage capacity of data centers. The decision of when and whether to keep or to delete replicas is determined by the fulfillment of those three parameters. Our approach estimates the total execution time of the tasks as well as the monetary cost, considering the data transfers activity. Our experiments are conducted using Cloudsim simulator. The obtained results show that our proposed strategies produce an efficient data management by reducing the overheads of the data transfers, compared to both a data placement without replication (by 76%) and the selected data replication approach from Kouidri et al. (by 52%), and by improving the financial cost.

Interval Type-2 Fuzzy C-Means Data Placement Optimization in Scientific Cloud Workflow applications

Scientific workflows stand as practical solutions useful for maintaining data intensive applications representation and execution purposes, which entail not only powerful computing resources, but also massive storage. With the emergence of cloud environment, which enhanced the execution of such applications, the study of workflow placement strategies, as targeted to effectively reduce data movements across data centers, has grown into a highly challenging objective. Given the fact that the workflow execution process is implemented in conformity with a task-execution order, and that each task may deal with either a single or multi-dataset, within a unique data center, various data partitioning or clustering methods have been devised in a bid to retrieve the most optimally effective workflow datasets’ distribution among data centers with the aim of remarkably reducing the datasets’ movements. In this work, a fuzzy data-dependencies based partitioning layer is implemented. More specifically, a dynamic massive data placement strategy is advanced through application of an Interval Type-2 Fuzzy C-Means technique. The latter is opted for as a means whereby the cluster related data centers can be rendered more consistent, thereby, making datasets rather closely associated in terms of related dependency, which helps in remarkably influencing the amounts of transferred data. The proposed strategy is evaluated by means of a simulation technique, using both random and real-world scientific workflows. The performed experiments appear to reveal well that our suggested strategy proves to outperform noticeably the relevant state-of-the-art methods, in that it noticeably helps in reducing the number of data movements across data centers.

P.513 Metabolic syndrome in schizophrenia and polymorphic variants of CYP1A2 and GSTP1 genes

Scientific workflows stand as practical solutions useful for maintaining data intensive applications representation and execution purposes, which entail not only powerful computing resources, but also massive storage. With the emergence of cloud environment, which enhanced the execution of such applications, the study of workflow placement strategies, as targeted to effectively reduce data movements across data centers, has grown into a highly challenging objective. Given the fact that the workflow execution process is implemented in conformity with a task-execution order, and that each task may deal with either a single or multi-dataset, within a unique data center, various data partitioning or clustering methods have been devised in a bid to retrieve the most optimally effective workflow datasets’ distribution among data centers with the aim of remarkably reducing the datasets’ movements. In this work, a fuzzy data-dependencies based partitioning layer is implemented. More specifically, a dynamic massive data placement strategy is advanced through application of an Interval Type-2 Fuzzy C-Means technique. The latter is opted for as a means whereby the cluster related data centers can be rendered more consistent, thereby, making datasets rather closely associated in terms of related dependency, which helps in remarkably influencing the amounts of transferred data. The proposed strategy is evaluated by means of a simulation technique, using both random and real-world scientific workflows. The performed experiments appear to reveal well that our suggested strategy proves to outperform noticeably the relevant state-of-the-art methods, in that it noticeably helps in reducing the number of data movements across data centers.

Cost effective dynamic data placement for efficient access of social networks

Social networks boast a huge number of worldwide users who join, connect, and publish various content, often very large, e.g. videos, images etc. For such very large-scale data storage, data replication using geo-distributed cloud services with virtually unlimited capabilities are suitable to fulfill the users’ expectations, such as low latency when accessing their and their friends’ data. However, service providers ideally want to spend as little as possible on replicating users’ data. Moreover, social networks have a dynamic nature and thus replicas need to be adaptable according to the environment, users’ behaviors, social network topology, and workload at runtime. Hence, it is not only crucial to have an optimized data placement and request distribution – meeting individual users’ acceptable latency requirements while incurring minimum cost for service providers – but the data placement must be adapted based on changes in the social network to keep it efficient and effective over time. In this paper, we model data placement as a dynamic set cover problem and propose a novel approach to solve this problem. We have run several experiments using two large-scale, open Facebook and Gowala datasets and real latencies derived from Amazon cloud datacenters to demonstrate our novel strategy’s efficiency and effectiveness.

A novel entropy-based dynamic data placement strategy for data intensive applications in Hadoop clusters

In the last decade, efficient data analysis of data-intensive applications has become an important research issue. Hadoop is the most widely used platform for data intensive application. However, majority of data placement strategies attempt placing related-data close to each other for faster access without considering new datasets, generated or for different MapReduce jobs. This paper deals with improving the map-reduce performance over multi-cluster datasets by means of a novel-entropy-based data placement strategy (EDPS) in three-phases. K-means clustering strategy is employed to extract dependencies among different datasets and group them into data-groups. Then these data-groups are placed in different datacenters while considering heterogeneity. Finally, an entropy-based grouping of the newly generated datasets where these datasets are grouped with most similar existing cluster based on their relative entropy. The experimental results show efficacy of the proposed three-fold dynamic grouping and data placement policy, which significantly reduces the time and improve Hadoop performance.

Performance and impact of dynamic data placement in ATLAS

For high-throughput computing the efficient use of distributed computing resources relies on an evenly distributed workload, which in turn requires wide availability of input data that is used in physics analysis. In ATLAS, the dynamic data placement agent C3PO was implemented in the ATLAS distributed data management system Rucio which identifies popular data and creates additional, transient replicas to make data more widely and more reliably available. This proceedings presents studies on the performance of C3PO and the impact it has on throughput rates of distributed computing in ATLAS. Furthermore, results of a study on popularity prediction using machine learning techniques are presented.

A novel entropy-based dynamic data placement strategy for data intensive applications in Hadoop clusters

In the last decade, efficient data analysis of data-intensive applications has become an important research issue. Hadoop is the most widely used platform for data intensive application. However, majority of data placement strategies attempt placing related-data close to each other for faster access without considering new datasets, generated or for different MapReduce jobs. This paper deals with improving the map-reduce performance over multi-cluster datasets by means of a novel-entropy-based data placement strategy (EDPS) in three-phases. K-means clustering strategy is employed to extract dependencies among different datasets and group them into data-groups. Then these data-groups are placed in different datacenters while considering heterogeneity. Finally, an entropy-based grouping of the newly generated datasets where these datasets are grouped with most similar existing cluster based on their relative entropy. The experimental results show efficacy of the proposed three-fold dynamic grouping and data placement policy, which significantly reduces the time and improve Hadoop performance.

An enhanced hadoop heartbeat mechanism for MapReduce task scheduler using dynamic calibration

MapReduce is a popular programming model for processing large-scale datasets in a distributed environment and is a fundamental component of current cloud computing and big data applications. In this paper, a heartbeat mechanism for MapReduce Task Scheduler using Dynamic Calibration (HMTS-DC) is proposed to address the unbalanced node computation capacity problem in a heterogeneous MapReduce environment. HMTS-DC uses two mechanisms to dynamically adapt and balance tasks assigned to each compute node: 1) using heartbeat to dynamically estimate the capacity of the compute nodes, and 2) using data locality of replicated data blocks to reduce data transfer between nodes. With the first mechanism, based on the heartbeats received during the early state of the job, the task scheduler can dynamically estimate the computational capacity of each node. Using the second mechanism, unprocessed Tasks local to each compute node are reassigned and reserved to allow nodes with greater capacities to reserve more local tasks than their weaker counterparts. Experimental results show that HMTS-DC performs better than Hadoop and Dynamic Data Placement Strategy (DDP) in a dynamic environment. Furthermore, an enhanced HMTS-DC (EHMTS-DC) is proposed by incorporating historical data. In contrast to the slow start property of HMTS-DC, EHMTS-DC relies on the historical computation capacity of the slave machines. The experimental results show that EHMTS-DC outperforms HMTS-DC in a dynamic environment.

C3PO - A Dynamic Data Placement Agent for ATLAS Distributed Data Management

This paper introduces a new dynamic data placement agent for the ATLAS distributed data management system. This agent is designed to pre-place potentially popular data to make it more widely available. It therefore incorporates information from a variety of sources. Those include input datasets and sites workload information from the ATLAS workload management system, network metrics from different sources like FTS and PerfSonar, historical popularity data collected through a tracer mechanism and more. With this data it decides if, when and where to place new replicas that then can be used by the WMS to distribute the workload more evenly over available computing resources and then ultimately reduce job waiting times. This paper gives an overview of the architecture and the final implementation of this new agent. The paper also includes an evaluation of the placement algorithm by comparing the transfer times and the new replica usage.

Cost-Aware Region-Level Data Placement in Multi-Tiered Parallel I/O Systems

Multi-tiered Parallel I/O systems that combine traditional HDDs with emerging SSDs mitigate the cost burden of SSDs while benefiting from their superior I/O performance. While a multi-tiered parallel I/O system is promising for data-intensive applications in high-performance (HPC) domains, placing data on each tier of the system to achieve high I/O performance remains a challenge. In this paper, we propose a cost-aware region-level (CARL) data placement scheme in multi-tiered parallel I/O systems. CARL divides a large file into several small regions, and then places regions on different types of servers based on region access costs. CARL includes a static policy S-CARL and a dynamic policy D-CARL. For applications whose I/O access patterns are completely known, S-CARL calculates the region costs within the entire workload duration, and uses a static data placement scheme to selectively place regions on the proper servers. To adapt to applications whose access patterns are unknown in advance, D-CARL uses a dynamic data placement scheme which migrates data among different servers within each time window. We have implemented CARL under MPI-IO library and OrangeFS parallel file system environment. Our evaluation with representative benchmarks and an application shows that CARL is both feasible and able to improve I/O performance significantly.

Dynamic Data Placement Strategy in MapReduce-styled Data Processing Platform

Data placement is one of the core technologies of the MapReduce-styled data processing platform, which contributes much to the data processing efficiency. A dynamic data placement strategy is proposed in this paper due to that existing data placement techniques are lack of the consideration of the computing load on the data storage node and reduce the ratio of the localized processing of the hot-spot data. The strategy takes the data accessing localization ratio, the remaining computing capacity of nodes as the new factors of the data placement decision. Performance evaluation results show that the proposed data placement strategy outperforms the original strategy in HDFS file system and the average job execution time is reduce by the maximum of 12%.

Predicting dataset popularity for the CMS experiment

The CMS experiment at the LHC accelerator at CERN relies on its computing infrastructure to stay at the frontier of High Energy Physics, searching for new phenomena and making discoveries. Even though computing plays a significant role in physics analysis we rarely use its data to predict the system behavior itself. A basic information about computing resources, user activities and site utilization can be really useful for improving the throughput of the system and its management. In this paper, we discuss a first CMS analysis of dataset popularity based on CMS meta-data which can be used as a model for dynamic data placement and provide the foundation of data-driven approach for the CMS computing infrastructure.

A new data-grouping-aware dynamic data placement method that take into account jobs execute frequency for Hadoop

Recent years have seen an increasing number of scientists employing data parallel computing frameworks, such as Hadoop, in order to run data-intensive applications. Research on data-grouping-aware data placement for Hadoop has become increasingly popular. However, we observe that many data-grouping-aware data placement schemes are static, without taking MapReduce job execution frequency into consideration. Such data placements scheme will lead to severe performance degradation that is way below the potential efficiency of optimal data distribution when executing MapReduce jobs that are executed frequency. In this paper, we propose a new data-grouping-aware dynamic (DGAD) data placement method based on the job execution frequency. Firstly, we build a job access correlation relation model among the data blocks according to the relationships provided by the records about historical data block access. Then we use a clustering algorithm to divide data blocks into clusters according to the job access correlation relation model among the data blocks and propose a data placement algorithm based on data block clusters in order to put correlated data blocks within a cluster on the different nodes. Finally, a series of experiments are carried out in order to verify the method proposed in this paper. Experimental results show that the proposed method can effectively deal with the mass data and can obviously improve the execution efficiency of MapReduce.

A study of dynamic data placement for ATLAS distributed data management

This contribution presents a study on the applicability and usefulness of dynamic data placement methods for data-intensive systems, such as ATLAS distributed data management (DDM). In this system the jobs are sent to the data, therefore having a good distribution of data is significant. Ways of forecasting workload patterns are examined which then are used to redistribute data to achieve a better overall utilisation of computing resources and to reduce waiting time for jobs before they can run on the grid. This method is based on a tracer infrastructure that is able to monitor and store historical data accesses and which is used to create popularity reports. These reports provide detailed summaries about data accesses in the past, including information about the accessed files, the involved users and the sites. From this past data it is possible to then make near-term forecasts for data popularity in the future. This study evaluates simple prediction methods as well as more complex methods like neural networks. Based on the outcome of the predictions a redistribution algorithm deletes unused replicas and adds new replicas for potentially popular datasets. Finally, a grid simulator is used to examine the effects of the redistribution. The simulator replays workload on different data distributions while measuring the job waiting time and site usage. The study examines how the average waiting time is affected by the amount of data that is moved, how it differs for the various forecasting methods and how that compares to the optimal data distribution.

Understanding the T2 traffic in CMS during Run-1

In the run-up to Run-1 CMS was operating its facilities according to the MONARC model, where data-transfers were strictly hierarchical in nature. Direct transfers between Tier-2 nodes was excluded, being perceived as operationally intensive and risky in an era where the network was expected to be a major source of errors. By the end of Run-1 wide-area networks were more capable and stable than originally anticipated. The original data-placement model was relaxed, and traffic was allowed between Tier-2 nodes.Tier-2 to Tier-2 traffic in 2012 already exceeded the amount of Tier-2 to Tier-1 traffic, so it clearly has the potential to become important in the future. Moreover, while Tier-2 to Tier-1 traffic is mostly upload of Monte Carlo data, the Tier-2 to Tier-2 traffic represents data moved in direct response to requests from the physics analysis community. As such, problems or delays there are more likely to have a direct impact on the user community.Tier-2 to Tier-2 traffic may also traverse parts of the WAN that are at the 'edge' of our network, with limited network capacity or reliability compared to, say, the Tier-0 to Tier-1 traffic which goes the over LHCOPN network.CMS is looking to exploit technologies that allow us to interact with the network fabric so that it can manage our traffic better for us, this we hope to achieve before the end of Run-2. Tier-2 to Tier-2 traffic would be the most interesting use-case for such traffic management, precisely because it is close to the users' analysis and far from the 'core' network infrastructure.As such, a better understanding of our Tier-2 to Tier-2 traffic is important. Knowing the characteristics of our data-flows can help us place our data more intelligently. Knowing how widely the data moves can help us anticipate the requirements for network capacity, and inform the dynamic data placement algorithms we expect to have in place for Run-2.This paper presents an analysis of the CMS Tier-2 traffic during Run 1.