Metadata Node Research Articles

Remote Sensing and Landsystems in the Mountain Domain: FAIR Data Accessibility and Landform Identification in the Digital Earth

Satellite imagery has become a major source for identifying and mapping terrestrial and planetary landforms. However, interpretating landforms and their significance, especially in changing environments, may still be questionable. Consequently, ground truth to check training models, especially in mountainous areas, can be problematic. This paper outlines a decimal format, [dLL], for latitude and longitude geolocation that can be used for model interpretation and validation and in data sets. As data have positions in space and time, [dLL] defined points, as for images, can be associated with metadata as nodes. Together with vertices, metadata nodes help build ‘information surfaces’ as part of the Digital Earth. This paper examines aspects of the Critical Zone and data integration via the FAIR data principles, data that are; findable, accessible, interoperable and re-usable. Mapping and making inventories of rock glacier landforms are examined in the context of their geomorphic and environmental significance and the need for geolocated ground truth. Terrestrial examination of rock glaciers shows them to be predominantly glacier-derived landforms and not indicators of permafrost. Remote-sensing technologies used to track developing rock glacier surface features show them to be climatically melting glaciers beneath rock debris covers. Distinguishing between glaciers, debris-covered glaciers and rock glaciers over time is a challenge for new remote sensing satellites and technologies and shows the necessity for a common geolocation format to report many Earth surface features.

An Adaptive Metadata Management Scheme Based on Deep Reinforcement Learning for Large-Scale Distributed File Systems

A major challenge confronting today’s distributed metadata management schemes is how to meet the dynamic requirements of various applications through effectively mapping and migrating metadata nodes to different metadata servers (MDS’s). Most of the existing works dynamically reallocate nodes to different servers adopting history-based coarse-grained methods, failing to make a timely and efficient update on the distribution of nodes. In this paper, we present the first deep reinforcement learning-leveraged distributed metadata management scheme, AdaM, to address the aforementioned dilemma. AdaM is an adaptive fine-grained metadata management scheme that trains an actor-critic network to migrate “hot” metadata nodes to different MDS’s based on its observations of the current “states” (i.e., access pattern, the structure of namespace tree and current distribution of nodes on MDS’s). Adaptive to varying access patterns, AdaM can automatically migrate hot metadata nodes among servers to keep load balancing while maintaining metadata locality. Besides, we propose a self-adaptive metadata cache policy, which dynamically combines the two strategies of managing caches on the server side and the client side to gain better query performance. Last but not least, we design a distributed metadata processing 2PC Protocol called MST-based 2PC to ensure data consistency. Experiments on a real-world dataset demonstrate the superiority of our proposed method over other schemes.

An End-to-End Learning-Based Metadata Management Approach for Distributed File Systems

Current distributed file systems are designed to support PB-scale even EB-scale data storage. Metadata service, which manages file attribute information and the global namespace tree, is crucial to system performance. Distributed metadata management, using multiple metadata servers (MDS's) to store metadata, provides effective approaches to alleviate the workload of a single server. However, maintaining good metadata locality and keeping load balancing among MDS's is a nontrivial problem. In this paper, we present the first machine learning based model called DeepHash, which leverages the neural network to learn a locality preserving hashing (LPH) mapping scheme. DeepHash first converts the metadata nodes to feature vectors by the network embedding technology. Due to the absence of training labels, i.e., the hash values of metadata nodes, we design a pair loss function with distinctive characters to train DeepHash, and introduce the sampling strategy to improve the training efficiency. Besides, we propose an efficient algorithm to dynamically balance the workload and adopt the cache model to improve query efficiency. The experiments on the Amazon EC2 platform demonstrate that the DeepHash can preserve the metadata locality meanwhile maintaining a high load balancing, which denotes the effectiveness and efficiency of DeepHash compared with traditional and state-of-the-art schemes.

ColdStore: A Storage System for Archival Data

With the astounding growth of the information era, the amount of data increases at high speed. However, a significant fraction of the data residing in storage is rarely accessed, which is referred to as cold data. It has become a looming problem of how to optimally manage and maintain this data at a low cost. In this paper, we present a large-scale, reliable, energy- and cost-efficient storage system called ColdStore. Different from traditional file systems, ColdStore aims to store cold data efficiently in the aspects of energy and cost. It is designed as a high-available distributed system, which is a cluster composed of a number of nodes with different roles: Metadata Nodes maintain the metadata of all files; Transfer Nodes play a role in encoding, decoding and caching; and Storage Nodes store encoded data for durability. Dedicated hardware is designed for Storage Nodes to reduce the power consumption. Each Storage Node is equipped with a low-power CPU and has a power supply control unit of hard disk drives. Due to the architecture we proposed, the vast majority, as many as 93.75%, of the hard disk drives can be powered off under the archival workload, while the system still provides reasonable performance and high reliability.

Keeping up with storage: Decentralized, write-enabled dynamic geo-replication

Large-scale applications are ever-increasingly geo-distributed. Maintaining the highest possible data locality is crucial to ensure high performance of such applications. Dynamic replication addresses this problem by dynamically creating replicas of frequently accessed data close to the clients. This data is often stored in decentralized storage systems such as Dynamo or Voldemort, which offer support for mutable data. However, existing approaches to dynamic replication for such mutable data remain centralized, thus incompatible with these systems. In this paper we introduce a write-enabled dynamic replication scheme that leverages the decentralized architecture of such storage systems. We propose an algorithm enabling clients to locate tentatively the closest data replica without prior request to any metadata node. Large-scale experiments on various workloads show a read latency decrease of up to 42% compared to other state-of-the-art, caching-based solutions.

A modern service-oriented distributed storage solution

This paper presents a distributed storage system which is built by lots of low-cost nodes. Low-cost node is divided into two parts: metadata nodes and storage nodes. The storage system will use a group of low-cost metadata nodes instead of the Google File System's (GFS's) single master server machine. These metadata nodes are linked by high-speed and high-density local peer-to-peer (P2P) network. We also designed the method organized metadata in the metadata cluster. The storage nodes are also linked by high-speed local P2P network. Metadata cluster and storage cluster are linked through high-speed reliable network. User files are organized into a tree structure by using a virtual storage component called “unit”. Storage system is built on the top of every node's operating system. This paper also gives the specific way of file addressing.