Open Source Distributed File System Research Articles

TCM Constitution Analysis Method Based on Parallel FP-Growth Algorithm in Hadoop Framework.

This work is devoted to establishing a comparatively accurate classification model between symptoms, constitutions, and regimens for traditional Chinese medicine (TCM) constitution analysis to provide preliminary screening and decision support for clinical diagnosis. However, for the analysis of massive distributed medical data in a cloud platform, the traditional data mining methods have the problems of low mining efficiency and large memory consumption, and long tuning time, an association rules method for TCM constitution analysis (ARA-TCM) is proposed that based on FP-growth algorithm and the open-source distributed file system in Hadoop framework (HDFS) to make full use of its powerful parallel processing capability. Firstly, the proposed method was used to explore the association rules between the 9 kinds of TCM constitutions and symptoms, as well as the regimen treatment plans, so as to discover the rules of typical clinical symptoms and treatment rules of different constitutions and to conduct an evidence-based medical evaluation of TCM effects in constitution-related chronic disease health management. Secondly, experiments were applied on a self-built TCM clinical records database with a total of 30,071 entries and it is found that the top three constitutions are mid constitution (42.3%), hot and humid constitution (31.3%), and inherited special constitution (26.2%), respectively. What is more, there are obvious promotions in the precision and recall rate compared with the Apriori algorithm, which indicates that the proposed method is suitable for the classification of TCM constitutions. This work is mainly focused on uncovering the rules of “disease symptoms constitution regimen” in TCM medical records, but tongue image and pulse signal are also very important to TCM constitution analysis. Therefore, this additional information should be considered into further studies to be more in line with the actual clinical needs.

A hybrid disaster-tolerant model with DDF technology for MooseFS open-source distributed file system

In a distributed file system (DFS), a disaster-tolerant model provides high data security in the data disaster event by geographically isolates the backup data, while the hot-backup model provides high availability when the main server is offline. Therefore, the combined model is widely concerned in multiple fields. However, previous investigations have neglected to the disaster tolerance for open-source distributed file systems. When a server run into a fault status and performs backup switch, the remote data center cannot receive any data replicas. We proposed a hybrid model in this work to provide disaster-tolerance especially for open-source DFS by build geographically isolate replicas. In addition, we optimize the model with direct data fetch (DDF) technology. DDF can bypass the main server, and retrieve data directly from the data node. The model was implemented on an open-source DFS named MooseFS. Compared with HDFS and IBM/PPRC, the hybrid model with DDF provides a strong performance boost to the hybrid model during the switch procedure, and this hybrid model performs best for storage of large amount of files in small size.

Performance Optimization for Managing Massive Numbers of Small Files in Distributed File Systems

The processing of massive numbers of small files is a challenge in the design of distributed file systems. Currently, the combined-block-storage approach is prevalent. However, the approach employs the traditional file systems such as ExtFS and may cause inefficiency when accessing small files randomly located in the disk. This paper focuses on optimizing the performance of data servers in accessing massive numbers of small files. We present a Flat Lightweight File System (iFlatLFS) to manage small files, which is based on a simple metadata scheme and a flat storage architecture. iFlatLFS is designed to substitute the traditional file system on data servers and can be deployed underneath distributed file systems that store massive numbers of small files. iFlatLFS can greatly simplify the original data access procedure. The new metadata proposed in this paper occupies only a fraction of the metadata size based on traditional file systems. We have implemented iFlatLFS in CentOS 5.5 and integrated it into an open source Distributed File System (DFS), called Taobao FileSystem (TFS), which is developed by a top B2C service provider, Alibaba, in China and is managing over 28.6 billion small photos. We have conducted extensive experiments to verify the performance of iFlatLFS. The results show that when the file size ranges from 1 to 64 KB, iFlatLFS is faster than Ext4 by 48 and 54 percent on average for random read and write in the DFS environment, respectively. Moreover, after iFlatLFS is integrated into TFS, iFlatLFS-based TFS is faster than the existing Ext4-based TFS by 45 and 49 percent on average for random read access and hybrid access (the mix of read and write accesses), respectively.

AN INTERACTIVE WEB-BASED ANALYSIS FRAMEWORK FOR REMOTE SENSING CLOUD COMPUTING

Abstract. Spatiotemporal data, especially remote sensing data, are widely used in ecological, geographical, agriculture, and military research and applications. With the development of remote sensing technology, more and more remote sensing data are accumulated and stored in the cloud. An effective way for cloud users to access and analyse these massive spatiotemporal data in the web clients becomes an urgent issue. In this paper, we proposed a new scalable, interactive and web-based cloud computing solution for massive remote sensing data analysis. We build a spatiotemporal analysis platform to provide the end-user with a safe and convenient way to access massive remote sensing data stored in the cloud. The lightweight cloud storage system used to store public data and users’ private data is constructed based on open source distributed file system. In it, massive remote sensing data are stored as public data, while the intermediate and input data are stored as private data. The elastic, scalable, and flexible cloud computing environment is built using Docker, which is a technology of open-source lightweight cloud computing container in the Linux operating system. In the Docker container, open-source software such as IPython, NumPy, GDAL, and Grass GIS etc., are deployed. Users can write scripts in the IPython Notebook web page through the web browser to process data, and the scripts will be submitted to IPython kernel to be executed. By comparing the performance of remote sensing data analysis tasks executed in Docker container, KVM virtual machines and physical machines respectively, we can conclude that the cloud computing environment built by Docker makes the greatest use of the host system resources, and can handle more concurrent spatial-temporal computing tasks. Docker technology provides resource isolation mechanism in aspects of IO, CPU, and memory etc., which offers security guarantee when processing remote sensing data in the IPython Notebook. Users can write complex data processing code on the web directly, so they can design their own data processing algorithm.

Metadata Indexing Sub-System for Distributed File System

The efficiency of metadata indexing is important to the performance of distributed file system. Time and space spending of current metadata management algorithms are unstable. In this paper, we use B-tree to index the metadata of distributed file system. Lustre is an open source distributed file system in which Hash function is used to manage the metadata. We implement the prototype of metadata indexing sub-system on Lustre and use Iozone to test the I/O performance of Lustre with and without the metadata indexing sub-system respectively. The simulation results show that Lustre with the metadata indexing sub-system has higher adaptability than Lustre with Hash-based metadata management algorithm.

Commodity hardware and open source solutions in FTU data management

Abstract Frascati Tokamak Upgrade (FTU) data management system underwent several developments in the last year, mainly due to the availability of huge amount of open source software and cheap commodity hardware. First of all, we replaced the old and expensive four SUN/SOLARIS servers running AFS (Andrew File System) fusione.it cell with three SuperServer Supermicro SC-742. Secondly Linux 2.4 OS has been installed on our new cell servers and OpenAFS 1.2.8 open source distributed file system has replaced the commercial IBM/Transarc AFS. A pioneering solution—SGI’s XFS file system for Linux—has been adopted to format one terabyte of FTU storage system on which the AFS volumes are based. Benchmark tests have shown the good performances of XFS compared to the classical ext3 Linux file system. Third, the data access software has been ported to Linux, together with the interfaces to Matlab and IDL, as well as the locally developed data display utility, SHOX. Finally a new Object-Oriented Data Model (OODM) has been developed for FTU shots data to build and maintain a FTU data warehouse (DW). FTU OODM has been developed using ROOT, an object oriented data analysis framework well-known in high energy physics. Since large volumes of data are involved, a parallel data extraction process, developed in the ROOT framework, has been implemented taking advantage of the AFS distributed environment of FTU computing system.