Efficient data synchronization method on integrated computing environment

Abstract

To execute scientific applications and simulations of enormous scale, the computing paradigm is evolving into one of cluster computing and cloud computing that can exploit the large number of available computing resources. To maximize the utilization of them, company or research center needs a scheduler engine and its data space to construct a cluster computing environment. However, if certain data space is shared, problems related to the security of node, the network traffic imbalance between nodes, and the data protection could arise. To solve these issues, a manager synchronizing the shared data space for the nodes that constitute a cluster computing environment is designed. The synchronization manager shares data in two ways: First, under the cluster environment, the full synchronization group can mount a specific directory space of the master node via NFS. It is used for the data which can be globally referenced. Second, the partial synchronization group delivers data to assigned workers through rsync. It can be used to locally share data for the isolation. The partial synchronization group is superior to full synchronization group in security and efficiency because data are shared in separate manner. By applying adequate data-sharing method, the designed manager efficiently mediate sharing data as purposed.