Exabytes Of Data Research Articles

Automated Network Services for Exascale Data Movement

The Large Hadron Collider (LHC) experiments distribute data by leveraging a diverse array of National Research and Education Networks (NRENs), where experiment data management systems treat networks as a “blackbox” resource. After the High Luminosity upgrade, the Compact Muon Solenoid (CMS) experiment alone will produce roughly 0.5 exabytes of data per year. NREN Networks are a critical part of the success of CMS and other LHC experiments. However, during data movement, NRENs are unaware of data priorities, importance, or need for quality of service, and this poses a challenge for operators to coordinate the movement of data and have predictable data flows across multi-domain networks. The overarching goal of SENSE (The Software-defined network for End-to-end Networked Science at Exascale) is to enable National Labs and universities to request and provision end-to-end intelligent network services for their application workflows leveraging SDN (Software-Defined Networking) capabilities. This work aims to allow LHC Experiments and Rucio, the data management software used by CMS Experiment, to allocate and prioritize certain data transfers over the wide area network. In this paper, we will present the current progress of the integration of SENSE, Multi-domain end-to-end SDN Orchestration with QoS (Quality of Service) capabilities, with Rucio, the data management software used by CMS Experiment.

Performance Enhancement of Distributed System Using HDFS Federation and Sharding

In today's world, 2.5 exabytes of data are generated and processed by the IT industry and different organizations. Processing and managing such a massive volume of Big data is challenging. Hadoop(HDFS) is widely used framework for processing Bigdata. The limitation of HDFS is it can scale only the datanodes and not namenode where the metadata is managed. There is no scaling of name nodes in the present Hadoop architecture, so managing the metadata of the exabyte scale becomes an essential and challenging leading to tightly coupled block storage, namespace scalability, and performance bottleneck issues. The proposed approach dynamic federated metadata management (DFMM) architecture manages the metadata in the name node. DFMM configures multiple name nodes using HDFS federation technique, and metadata is partitioned into numerous name nodes using sharding technique. Sharding manages the metadata using locality-preserving hashing and consistent hashing methods. Locality-preserving hashing stores metadata in the most appropriate name node using the file's parent directory, and metadata is dynamically distributed among Name nodes to maintain load balancing. DFMM also maintains replicas for the high availability and reliability of data. Results showcase the superiority of the proposed DFMM architecture combined with the sharding technique is highly scalable and manages metadata efficiently, and can store up to 1 billion files with a namespace size of 400 GB, enhancing the throughput by 21% and is highly scalable as compared to existing techniques. The comparative study analysis of existing HDFS, and proposed DFMM architecture combined with sharding enhances the performance and manages the metadata more efficiently.

The Emerging Trends of Multi-Label Learning.

Exabytes of data are generated daily by humans, leading to the growing needs for new efforts in dealing with the grand challenges for multi-label learning brought by big data. For example, extreme multi-label classification is an active and rapidly growing research area that deals with classification tasks with extremely large number of classes or labels; utilizing massive data with limited supervision to build a multi-label classification model becomes valuable for practical applications, etc. Besides these, there are tremendous efforts on how to harvest the strong learning capability of deep learning to better capture the label dependencies in multi-label learning, which is the key for deep learning to address real-world classification tasks. However, it is noted that there have been a lack of systemic studies that focus explicitly on analyzing the emerging trends and new challenges of multi-label learning in the era of big data. It is imperative to call for a comprehensive survey to fulfil this mission and delineate future research directions and new applications.

Analysis of English Cultural Teaching Model Based on Machine Learning

According to the world population, nearly five billion people use mobile phones in their daily lives, and this has increased by 20% in the last twelve months compared to the previous report. An average survey conducted by researchers to find the amount of data consumed in a month by every mobile phone in the world has finally resulted in 45 exabytes of data being collected from a single user within a month. In today's world, data consumption and data analytics are being considered as one of the most important necessities for e-commerce companies. With the help of such collected data from a person, it is possible to predict the future signature or activity of the person. If 45 terabytes of data can be stored for a single user, determining the average calculation and amount of data to be collected for five billion users appears to be much more difficult. More than the human working concept, it looks like it would be difficult for a traditional computer system to handle this amount of data. To study and understand a concept from machine learning and artificial intelligence requires quite a collection of data to predict according to a person's activity. This article explains the roles of faculty and students, as well as the requirements for academic evaluation. Even before the pandemic, most people did not have any idea about the online teaching model. It is only after the disability of conducting direct (offline) classes that people are forced to get into the online world of teaching. Nearly 60% of countries are trying to convert their education systems to such online models, which improve communication between students and teachers and also enable different schemes for students. Big data can be considered as one of the technological revolutions in information technology companies that became popular after the crisis of cloud computing. A support vector machine (SVM) is proposed for analyzing English culture teaching and is compared with the traditional fuzzy logic. The results show the proposed model achieves an accuracy of 98%, which is 5% higher than the existing algorithm.

Exploitation of HPC Resources for data intensive sciences

The Large Hadron Collider (LHC) will enter a new phase beginning in 2027 with the upgrade to the High Luminosity LHC (HL-LHC). The increase in the number of simultaneous collisions coupled with a more complex structure of a single event will result in each LHC experiment collecting, storing, and processing exabytes of data per year. The amount of generated and/or collected data greatly outweighs the expected available computing resources. In this paper, we discuss effcient usage of HPC resources as a prerequisite for data-intensive science at exascale. First, we discuss the experience of porting CMS Hadron and Electromagnetic calorimeters reconstruction code to utilize Nvidia GPUs within the DEEP-EST project; second, we look at the tools and their adoption in order to perform benchmarking of a variety of resources available at HPC centers. Finally, we touch on one of the most important aspects of the future of HEP - how to handle the flow of petabytes of data to and from computing facilities, be it clouds or HPCs, for exascale data processing in a flexible, scalable and performant manner. These investigations are a key contribution to technical work within the HPC collaboration among CERN, SKA, GEANT and PRACE.

On the road to a scientific data lake for the High Luminosity LHC era

The experiments at CERN’s Large Hadron Collider use the Worldwide LHC Computing Grid, the WLCG, for its distributed computing infrastructure. Through the distributed workload and data management systems, they provide seamless access to hundreds of grid, HPC and cloud based computing and storage resources that are distributed worldwide to thousands of physicists. LHC experiments annually process more than an exabyte of data using an average of 500,000 distributed CPU cores, to enable hundreds of new scientific results from the collider. However, the resources available to the experiments have been insufficient to meet data processing, simulation and analysis needs over the past five years as the volume of data from the LHC has grown. The problem will be even more severe for the next LHC phases. High Luminosity LHC will be a multiexabyte challenge where the envisaged Storage and Compute needs are a factor 10 to 100 above the expected technology evolution. The particle physics community needs to evolve current computing and data organization models in order to introduce changes in the way it uses and manages the infrastructure, focused on optimizations to bring performance and efficiency not forgetting simplification of operations. In this paper we highlight a recent R&D project related to scientific data lake and federated data storage.

Enabling Data Intensive Science on Supercomputers for High Energy Physics R&D Projects in HL-LHC Era

The ATLAS experiment at CERN’s Large Hadron Collider uses theWorldwide LHC Computing Grid, the WLCG, for its distributed computing infrastructure. Through the workload management system PanDA and the distributed data management system Rucio, ATLAS provides seamless access to hundreds of WLCG grid and cloud based resources that are distributed worldwide, to thousands of physicists. PanDA annually processes more than an exabyte of data using an average of 350,000 distributed batch slots, to enable hundreds of new scientific results from ATLAS. However, the resources available to the experiment have been insufficient to meet ATLAS simulation needs over the past few years as the volume of data from the LHC has grown. The problem will be even more severe for the next LHC phases. High Luminosity LHC will be a multiexabyte challenge where the envisaged Storage and Compute needs are a factor 10 to 100 above the expected technology evolution. The High Energy Physics (HEP) community needs to evolve current computing and data organization models in order to introduce changes in the way it uses and manages the infrastructure, focused on optimizations to bring performance and efficiency not forgetting simplification of operations. In this paper we highlight recent R&D projects in HEP related to data lake prototype, federated data storage and data carousel.

BigPanDA: PanDA Workload Management System and its Applications beyond ATLAS

Modern experiments collect peta-scale volumes of data and utilize vast, geographically distributed computing infrastructure that serves thousands of scientists around the world. Requirements for rapid, near real-time data processing, fast analysis cycles and need to run massive detector simulations to support data analysis pose special premium on efficient use of available computational resources. A sophisticated Workload Management System (WMS) is needed to coordinate the distribution and processing of data and jobs in such environment. The ATLAS experiment at CERN uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. While PanDAcurrently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, it runs around 2 million jobs per day on hundreds of Grid sites and serving thousands of ATLAS users. In 2017 about 1.5 exabytes of data were processed with PanDA.In 2012 BigPanDA project project was started with aim to introduce new types of computing resources into ATLAS computing infrastructure, but also to offering PanDA features to different data-intensive applications for projects and experiments outside of ATLAS and High-Energy and Nuclear Physics. In this article we will present accomplishments and discuss possible directions for future work.

Developing Regional Primary Health Analytic Capability

Introduction : Brisbane South PHN, in partnership with Primary Health Providers (PHP) in the region, have built a rich trusted source of de-identified population health data. Analytics outputs from the dataset are used by PHP partners to improve patient management, resulting in improved healthcare outcomes. Description : A 2014 Dell EMC report [1] estimated that in 2013 healthcare generated 153 Exabytes of data, which by 2020 will likely increase by 150%. We see the enormous benefits in using analytics on de-identified data from our PHP partners’ electronic medical records to identify barriers to care, help improve prevention, diagnosis, management and follow-up care. Informed by whitepaper [2] and international literature [3], a health data analytics framework (Figure 1) has been developed and used to implement an incremental, methodical pathway towards advanced analytics. Our current efforts focus on : Prescriptive analytics identifying: the “what happened”, “how many”, “how often” questions? For example, population health data analytics that show prevalence of chronic co-morbidity in our region’s cardiovascular disease patient population and their gender distributions (Figure 2); how many patients with chronic conditions don’t have management plans, or no health check in 6 or 12 months. Level 1 predictive analytics to consider: “why did this happen”, “what can/should be done”? For example, analytics that show risk factors which may explain differences in prevalence by gender (Figure 3), and what risk-stratified interventions can be considered. Analytics results are used by PHP partners to improve care and interventions, resulting in improved outcomes. We use similar analytics tools and models deployed by our acute care partners in the region. The methodologies are transferrable to other healthcare sectors. We are working on strategies to address challenges, e.g., sustaining analytics momentum, improving data quality, interoperability, and tools (through design, use, collaboration), keeping pace with analytic science and technology maturity. Our intermediate term goal is to advance our capabilities, including effective predictive risk models [4] application to achieve accurate population risk stratification, optimise management and outcomes. Longer term goal is to progress towards advance predictive and applied AI analytics through collaborations and capability building. Conclusion : We are at an early stage on the analytic maturity curve. Our current aim is to optimise healthcare data and analytics output values. Future success will depend on our ability to improve data quality and interoperability; grow analytics capabilities; and to provide quality analytics outputs as catalysts for our PHP partners to improve their intervention strategies and patient outcomes. Discussion : Current analytics techniques are either question-driven – the “what”, “”why”, and “how”; or hypothesis driven (Figure 1). As more data are generated, more complex questions will be considered and hypotheses generated. Healthcare analytics is still a maturing discipline. The ability to ask useful questions; generate accurate hypotheses; convert hypotheses into effective analytics questions; and to produce high value outputs is dependent on expert knowledge, predictive models and technology maturity. Over time, our strategy is to improve analytics capabilities; work with technology suppliers and experts to improve tools, knowledge and skills; and initiate pathways towards achieving applied AI analytics.

TRIFECTA: Security, Energy Efficiency, and Communication Capacity Comparison for Wireless IoT Devices

The widespread proliferation of sensor nodes in the era of the Internet of Things (IoT), coupled with increasing sensor fidelity and data acquisition modality, is expected to generate 30+ Exabytes of data per month by 2020. Since most of these IoT devices will be wirelessly connected at the last few feet, wireless communication is an integral part of the future IoT scenario. The ever-shrinking size of unit computation (Moores law) and continued improvements in efficient communication (Shannons law) are expected to harness the true potential of the IoT revolution and produce a dramatic societal impact. However, reducing the size of IoT nodes and the slow improvement in energy storage density leads to reduced energy availability. Moreover, smaller size and less energy means fewer resources available for securing IoT nodes, making the energy-sparse low-cost leaf nodes of the network prime targets for attackers. In this article, we survey six prominent wireless technologies with respect to three dimensions: security, energy efficiency, and communication capacity. We point out the state of the art, open issues, and the road ahead for promising research directions.

Modeling and Simulation of Extreme-Scale Fat-Tree Networks for HPC Systems and Data Centers

As parallel and distributed systems are evolving toward extreme scale, for example, high-performance computing systems involve millions of cores and billion-way parallelism, and high-capacity storage systems require efficient access to petabyte or exabyte of data, many new challenges are posed on designing and deploying next-generation interconnection communication networks in these systems. Fat-tree networks have been widely used in both data centers and high-performance computing (HPC) systems in the past decades and are promising candidates of the next-generation extreme-scale networks. In this article, we present FatTreeSim, a simulation framework that supports modeling and simulation of extreme-scale fat-tree networks with the goal of understanding the design constraints of next-generation HPC and distributed systems and aiding the design and performance optimization of the applications running on these systems. We have systematically experimented FatTreeSim on Emulab and Blue Gene/Q and analyzed the scalability and fidelity of FatTreeSim with various network configurations. On the Blue Gene/Q Mira, FatTreeSim can achieve a peak performance of 305 million events per second using 16,384 cores. Finally, we have applied FatTreeSim to simulate several large-scale Hadoop YARN applications to demonstrate its usability.

Cloud Storage Providers and its Metrics

Cloud Storage Providers (CSPs) delivered over an Exabyte of data under contract. This amazing growth announcers a new period for how storage is delivered and consumed. The CSP market has seen new providers, new functionality, improved platform maturity, increased acceptance in the enterprise and significant price reductions. Amazon’s S3 service crossed the 1-trillion-object mark1, a significant indication of platform maturity, and all major CSPs dropped their prices multiple times. As public cloud storage becomes critical to enterprise infrastructure, independent testing and monitoring of these systems is becoming all the more critical to IT. It publishes the report to share the information that we assemble in order to properly evaluate CSPs for our own use. Cloud storage is a key factor of our storage-as-a-service, much like the traditional system vendors leverage commodity hard drives in their storage arrays. Keywords: Cloud Storage Providers (CSPs), Functionality, Price, Presentation, Availability, Scalability, Benchmark.

Networkmetrics: multivariate big data analysis in the context of the internet

Multivariate problems are found in all areas of knowledge. In chemistry and related disciplines. The chemometric community was developed in a joint effort to understand and solve problems mainly from a multivariate and exploratory perspective. This perspective is, indeed, of broader applicability, even in areas of knowledge far from chemistry. In this paper, we focus on the Internet: the net of devices that allow an interconnected world where all types of data can be shared and unprecedented communication services can be provided. Problems in the Internet, or, in general, in networking, are not very different from chemometric problems. Yet, they also have their specificities, mainly their typical Big Data nature and the presence of unstructured data. Exabytes of data in the format of text, video, audio, etc. are daily generated in the Internet. Part of these data need to be analyzed for disparate purposes, ranging from the optimization of the communications to cybersecurity and cyberwalfare. We argue that exploratory multivariate analysis is, indeed, useful to tackle these problems and has a role to play in the Big Data arena, currently limited to data mining techniques and visualization analytics. In absence of a better name, we call Networkmetrics the approach of treating Internet problems from the multivariate and exploratory perspective.

Exploring the Design Space of an Energy-Efficient Accelerator for the SKA1-Low Central Signal Processor

The Square Kilometre Array (SKA) will be the biggest radio telescope ever built, with unprecedented sensitivity, angular resolution, and survey speed. Collectively, the SKA’s antennas are expected to gather exabytes of data per second and store one petabyte of data every day, requiring exa operations per second for the processing. This paper focuses on the SKA1-Low, the SKA’s aperture-array instrument consisting of 131,072 antennas that will be built in the first phase of the deployment of the project. In particular, our work explores the design of a custom architecture for the central signal processor (CSP) of the SKA1-Low. The CSP processes digitized samples sent by antennas receiving extra-terrestrial radio-frequency signals between 50 and 350 MHz. We describe the challenges in building the CSP, and present a quantitative study for the implementation of a custom hardware architecture for executing the main CSP algorithms. By taking advantage of emerging 3D-stacked-memory devices and by exploring the design space for a 14-nm implementation, we estimate a power consumption of 9.62 W for processing all channels of a sub-band and an energy efficiency at application level of up to 312 GFLOPS/W for our architecture.

PanDA: Exascale Federation of Resources for the ATLAS Experiment at the LHC

After a scheduled maintenance and upgrade period, the world’s largest and most powerful machine – the Large Hadron Collider(LHC) – is about to enter its second run at unprecedented energies. In order to exploit the scientific potential of the machine, the experiments at the LHC face computational challenges with enormous data volumes that need to be analysed by thousand of physics users and compared to simulated data. Given diverse funding constraints, the computational resources for the LHC have been deployed in a worldwide mesh of data centres, connected to each other through Grid technologies.The PanDA (Production and Distributed Analysis) system was developed in 2005 for the ATLAS experiment on top of this heterogeneous infrastructure to seamlessly integrate the computational resources and give the users the feeling of a unique system. Since its origins, PanDA has evolved together with upcoming computing paradigms in and outside HEP, such as changes in the networking model, Cloud Computing and HPC. It is currently running steadily up to 200 thousand simultaneous cores (limited by the available resources for ATLAS), up to two million aggregated jobs per day and processes over an exabyte of data per year. The success of PanDA in ATLAS is triggering the widespread adoption and testing by other experiments. In this contribution we will give an overview of the PanDA components and focus on the new features and upcoming challenges that are relevant to the next decade of distributed computing workload management using PanDA.

The future of PanDA in ATLAS distributed computing

Experiments at the Large Hadron Collider (LHC) face unprecedented computing challenges. Heterogeneous resources are distributed worldwide at hundreds of sites, thousands of physicists analyse the data remotely, the volume of processed data is beyond the exabyte scale, while data processing requires more than a few billion hours of computing usage per year. The PanDA (Production and Distributed Analysis) system was developed to meet the scale and complexity of LHC distributed computing for the ATLAS experiment. In the process, the old batch job paradigm of locally managed computing in HEP was discarded in favour of a far more automated, flexible and scalable model. The success of PanDA in ATLAS is leading to widespread adoption and testing by other experiments. PanDA is the first exascale workload management system in HEP, already operating at more than a million computing jobs per day, and processing over an exabyte of data in 2013. There are many new challenges that PanDA will face in the near future, in addition to new challenges of scale, heterogeneity and increasing user base. PanDA will need to handle rapidly changing computing infrastructure, will require factorization of code for easier deployment, will need to incorporate additional information sources including network metrics in decision making, be able to control network circuits, handle dynamically sized workload processing, provide improved visualization, and face many other challenges. In this talk we will focus on the new features, planned or recently implemented, that are relevant to the next decade of distributed computing workload management using PanDA.

Abstract 4743: A population-based approach to address clinical cancer care: The national genomics platform

Abstract Our understanding of the genetic basis for cancer is advancing at a rapid rate due to the application of next generation sequencing (NGS) technologies. It is becoming common to sequence tumors and patients in an attempt to find actionable mutations that could offer the patient a more targeted and effective treatment. However, the genomics of cancer is very complex and only a handful of actionable mutations have been characterized. Larger-scale studies are needed to understand the pathogenic spectrum of cancer variants and deliver reliable clinical decision-making support to providers. Several large-scale sequencing projects are underway to collect NGS data on tumors (e.g. TCGA, ICGC, and TARGET) but a national platform for analysis, interpretation, and reporting does not exist. We propose that a consolidated informatics platform for the collection of outcomes data with genomics and clinical data would accelerate research and provide patients with the opportunity for personalized cancer treatment. In addition, with 1,665,540 new cancer cases predicted in the US for 2014, a national-scale genomics platform is needed, capable of sequencing 3 million genomes per year, storing exabytes of data, while supporting over 20,000 oncologists, researchers, and analysts. At Lockheed Martin, we deliver highly scalable and reliable information systems for a variety of missions and citizen services. Here, we will present our vision for a national cancer genomics platform to include NGS data collection, cost-efficient storage, scalable and modular processing pipelines, and collaborative analytics and data sharing capabilities, within a compliant privacy and security framework. In conclusion, by leveraging the scale of clinical cancer sequencing and capturing these data into a case management system for translational research, this platform provides data at-scale needed for finding actionable mutations, designing effective treatments and implementing prevention strategies, faster. Citation Format: Ogan Abaan, Amrita Basu, Noah Brown, Bret Light, David Deal, Michael Hultner. A population-based approach to address clinical cancer care: The national genomics platform. [abstract]. In: Proceedings of the 106th Annual Meeting of the American Association for Cancer Research; 2015 Apr 18-22; Philadelphia, PA. Philadelphia (PA): AACR; Cancer Res 2015;75(15 Suppl):Abstract nr 4743. doi:10.1158/1538-7445.AM2015-4743

Public Auditing of Dynamic Big Data Storagewith Efficient High Memory Utilization andECC Algorithm

Big data is an evolving term that describes any voluminous amount of structured, semi-structured and unstructured data. The term is often used when speaking about peta-bytes and exa-bytes of data. The security and privacy is the static and huge challenging issue in big data storage. There are many ways to compromise data because of insufficient authentication, authorization, and audit (AAA) controls, such as deletion or alteration of records without a backup of the original content. The existing research work showed that it can fully support authorized auditing and fine-grained update requests. However, such schemes in existence suffer from several common drawbacks. First maintaining the storages can be a difficult task and second it requires high resource costs for the implementation. This paper, Propose a formal analysis technique called full grained updates. It includes the efficient searching for downloading the uploaded file and also focuses on designing the auditing protocol to improve the server-side protection for the efficient data confidentiality and data availability.

Distilling Big Data: Refining Quality Information in the Era of Yottabytes.

Big Data is the buzzword of the modern century. With the invasion of pervasive computing, we live in a data centric environment, where we always leave a track of data related to our day to day activities. Be it a visit to a shopping mall or hospital or surfing Internet, we create voluminous data related to credit card transactions, user details, location information, and so on. These trails of data simply define an individual and form the backbone for user-profiling. With the mobile phones and their easy access to online social networks on the go, sensor data such as geo-taggings and events and sentiments around them contribute to the already overwhelming data containers. With reductions in the cost of storage and computational devices and with increasing proliferation of Cloud, we never felt any constraints in storing or processing such data. Eventually we end up having several exabytes of data and analysing them for their usefulness has introduced new frontiers of research. Effective distillation of these data is the need of the hour to improve the veracity of the Big Data. This research targets the utilization of the Fuzzy Bayesian process model to improve the quality of information in Big Data.

Key Technologies of Distributed Storage for Cloud Computing

PDF HTML阅读 XML下载 导出引用 引用提醒 云计算环境下的分布存储关键技术 DOI: 10.3724/SP.J.1001.2012.04175 作者: 作者单位: 作者简介: 通讯作者: 中图分类号: 基金项目: 国家重点基础研究发展计划(973)(2011CB302601); 国家自然科学基金(60873215); 湖南省自然科学杰出青年基金(S2010J5050); 高等学校博士学科点专项科研基金(200899980003) Key Technologies of Distributed Storage for Cloud Computing Author: Affiliation: Fund Project: 摘要 | 图/表 | 访问统计 | 参考文献 | 相似文献 | 引证文献 | 资源附件 | 文章评论 摘要:云计算作为下一代计算模式,在科学计算和商业计算领域均发挥着重要作用,受到当前学术界和企业界的广泛关注.云计算环境下的分布存储主要研究数据在数据中心上的组织和管理,作为云计算环境的核心基础设施,数据中心通常由百万级以上节点组成,存储其上的数据规模往往达到PB 级甚至EB 级,导致数据失效成为一种常态行为,极大地限制了云计算的应用和推广,增加了云计算的成本.因此,提高可扩展性和容错性、降低成本,成为云计算环境下分布存储研究的若干关键技术.针对如何提高存储的可扩展性、容错性以及降低存储的能耗等目标,从数据中心网络的设计、数据的存储组织方式等方面对当前分布存储的关键技术进行了综述.首先,介绍并对比了当前典型的数据中心网络结构的优缺点;其次,介绍并对比了当前常用的两种分布存储容错技术,即基于复制的容错技术和基于纠删码的容错技术;第三,介绍了当前典型的分布存储节能技术,并分析了各项技术的优缺点;最后指出了当前技术面临的主要挑战和下一步研究的方向. Abstract:Considered as the next generation computing model, cloud computing plays an important role in scientific and commercial computing area and draws great attention from both academia and industry fields. Under cloud computing environment, data center consist of a large amount of computers, usually up to millions, and stores petabyte even exabyte of data, which may easily lead to the failure of the computers or data. The large amount of computers composition not only leads to great challenges to the scalability of the data center and its storage system, but also results in high hardware infrastructure cost and power cost. Therefore, fault-tolerance, scalability, and power consumption of the distributed storage for a data center becomes key part in the technology of cloud computing, in order to ensure the data availability and reliability. In this paper, a survey is made on the state of art of the key technologies in cloud computing in the following aspects: Design of data center network, organization and arrangement of data, strategies to improve fault-tolerance, methods to save storage space, and energy. Firstly, many kinds of classical topologies of data center network are introduced and compared. Secondly, kinds of current fault-tolerant storage techniques are discussed, and data replication and erasure code strategies are especially compared. Thirdly, the main current energy saving technology is addressed and analyzed. Finally, challenges in distributed storage are reviewed as well as future research trends are predicted. 参考文献 相似文献 引证文献