Traditional File Systems Research Articles

DudeTM

Emerging non-volatile memory (NVM) offers non-volatility, byte-addressability and fast access at the same time. To make the best use of these properties, it has been shown by empirical evidence that programs should access NVM directly through CPU load and store instructions, so that the overhead of a traditional file system or database can be avoided. Thus, durable transactions become a common choice of applications for accessing persistent memory data in a crash consistent manner. However, existing durable transaction systems employ either undo logging , which requires a fence for every memory write, or redo logging , which requires intercepting all memory reads within transactions. This paper presents DUDETM, a crash-consistent durable transaction system that avoids the drawbacks of both undo logging and redo logging. DUDETM uses shadow DRAM to decouple the execution of a durable transaction into three fully asynchronous steps. The advantage is that only minimal fences and no memory read instrumentation are required. This design also enables an out-of-the-box transactional memory (TM) to be used as an independent component in our system. The evaluation results show that DUDETM adds durability to a TM system with only 7.4 ~ 24.6% throughput degradation. Compared to the existing durable transaction systems, DUDETM provides 1.7times to 4.4times higher throughput. Moreover, DUDETM can be implemented with existing hardware TMs with minor hardware modifications, leading to a further 1.7times speedup.

A New Design of In-Memory File System Based on File Virtual Address Framework

The emerging technologies of persistent memory, such as PCM, MRAM, provide opportunities for preserving files in memory. Traditional file system structures may need to be re-studied. Even though there are several file systems proposed for memory, most of them have limited performance without fully utilizing the hardware at the processor side. This paper presents a framework based on a new concept, “File Virtual Address Space”. A file system, Sustainable In-Memory File System (SIMFS), is designed and implemented, which fully utilizes the memory mapping hardware at the file access path. First, SIMFS embeds the address space of an open file into the process’ address space. Then, file accesses are handled by the memory mapping hardware. Several optimization approaches are also presented for the proposed SIMFS. Extensive experiments are conducted. The experimental results show that the throughput of SIMFS achieves significant performance improvement over the state-of-the-art in-memory file systems.

Coral: A Cloud-Backed Frugal File System

With simple access interfaces and flexible billing models, cloud storage has become an attractive solution to simplify the storage management for both enterprises and individual users. However, traditional file systems with extensive optimizations for local disk-based storage backend can not fully exploit the inherent features of the cloud to obtain desirable performance. In this paper, we present the design, implementation, and evaluation of Coral, a cloud based file system that strikes a balance between performance and monetary cost. Unlike previous studies that treat cloud storage as just a normal backend of existing networked file systems, Coral is designed to address several key issues in optimizing cloud-based file systems such as the data layout, block management, and billing model. With carefully designed data structures and algorithms, such as identifying semantically correlated data blocks, kd-tree based caching policy with self-adaptive thrashing prevention, effective data layout, and optimal garbage collection, Coral achieves good performance and cost savings under various workloads as demonstrated by extensive evaluations.

Performance Optimization for Managing Massive Numbers of Small Files in Distributed File Systems

The processing of massive numbers of small files is a challenge in the design of distributed file systems. Currently, the combined-block-storage approach is prevalent. However, the approach employs the traditional file systems such as ExtFS and may cause inefficiency when accessing small files randomly located in the disk. This paper focuses on optimizing the performance of data servers in accessing massive numbers of small files. We present a Flat Lightweight File System (iFlatLFS) to manage small files, which is based on a simple metadata scheme and a flat storage architecture. iFlatLFS is designed to substitute the traditional file system on data servers and can be deployed underneath distributed file systems that store massive numbers of small files. iFlatLFS can greatly simplify the original data access procedure. The new metadata proposed in this paper occupies only a fraction of the metadata size based on traditional file systems. We have implemented iFlatLFS in CentOS 5.5 and integrated it into an open source Distributed File System (DFS), called Taobao FileSystem (TFS), which is developed by a top B2C service provider, Alibaba, in China and is managing over 28.6 billion small photos. We have conducted extensive experiments to verify the performance of iFlatLFS. The results show that when the file size ranges from 1 to 64 KB, iFlatLFS is faster than Ext4 by 48 and 54 percent on average for random read and write in the DFS environment, respectively. Moreover, after iFlatLFS is integrated into TFS, iFlatLFS-based TFS is faster than the existing Ext4-based TFS by 45 and 49 percent on average for random read access and hybrid access (the mix of read and write accesses), respectively.

Clustered page-level mapping for flash memory-based storage devices

Recent consumer devices such as smartphones, smart TVs and tablet PCs adopt NAND flash memory as storage device due to its advantages of small size, reliability, low power consumption, and high performance. The unique characteristics of NAND flash memory require an additional software layer, called flash translation layer (FTL), between traditional file systems and flash memory. In order to reduce the garbage collection cost, FTLs generally try to separate hot and cold data. Previous hot and cold separation techniques monitor the storage access patterns within storage device, or exploit file system hints from host system. This paper proposes a novel clustered page-level mapping, called CPM, which can separate hot and cold data efficiently by allocating different flash memory block groups to different logical address regions. CPM can reduce the FTL map loading overhead during garbage collection and it does not require any high-cost monitoring overhead or host hint. This paper also proposes a K-associative version of CPM, called K-CPM, which allows different logical address regions to share a physical block group in order to achieve high block utilizations. Experimental results show that CPM improves the storage I/O performance by about 54% compared with a previous page-level mapping FTL, and K-CPM further improves the performance by about 19.4% compared with CPM.

Protecting data in cloud environment with attribute-based encryption

Traditional file systems cannot satisfy the recent requirements in the cloud environment. Meanwhile, current file systems designed for cloud computing cannot provide enough flexibility, fine-grained access control and access security. In this paper, a novel secure file sharing scheme based on attribute control is presented. In order to design a practical cloud file system with attribute based encryption, we give a systematic definition of attribute computing in cloud computing environment. Based on the definition, we present a secure and practical attribute based encryption scheme without pairings CP-ABE-WP under cloud computing scenarios. Then we design a secure cloud file system applying our CP-ABE-WP to provide data management and secure data sharing. According to our analysis and test, the scheme is chosen plaintext secure in selective ID model with acceptable performance and can satisfy the file sharing application in cloud computing.

Research on data pre-deployment in information service flow of digital ocean cloud computing

Data pre-deployment in the HDFS (Hadoop distributed file systems) is more complicated than that in traditional file systems. There are many key issues need to be addressed, such as determining the target location of the data prefetching, the amount of data to be prefetched, the balance between data prefetching services and normal data accesses. Aiming to solve these problems, we employ the characteristics of digital ocean information service flows and propose a deployment scheme which combines input data prefetching with output data oriented storage strategies. The method achieves the parallelism of data preparation and data processing, thereby massively reducing I/O time cost of digital ocean cloud computing platforms when processing multi-source information synergistic tasks. The experimental results show that the scheme has a higher degree of parallelism than traditional Hadoop mechanisms, shortens the waiting time of a running service node, and significantly reduces data access conflicts.

Design and Implementation of a Log-Structured File System for Flash-Based Solid State Drives

Even in modern SSDs, the disparity between random and sequential write bandwidth is more than 10-fold. Moreover, random writes can shorten the limited lifespan of SSDs because they incur more NAND block erases per write. To overcome the problems of random writes, we propose a new file system, SFS, for SSDs. SFS is similar to the traditional log-structured file system (LFS) in that it transforms all random writes at the file system level to sequential ones at the SSD level, as a way to exploit the maximum write bandwidth of the SSD. But, unlike the traditional LFS, which performs hot/cold data separation on segment cleaning, SFS takes a new on writing data grouping strategy. When data blocks are to be written, SFS puts those with similar update likelihood into the same segment for sharper bimodal distribution of segment utilization, and thus aims at minimizing the inevitable segment cleaning overhead that occurs in any log-structured file system. We have implemented a prototype SFS by modifying Linux-based NILFS2 and compared it with three state-of-the-art file systems using several realistic workloads. Our experiments on SSDs show that SFS outperforms LFS by up to 2.5 times in terms of throughput. In comparison to modern file systems, SFS drastically reduces the block erase count inside SSDs by up to 23.3 times. Although SFS was targeted for SSDs, its data grouping on writing would also work well in HDDs. To confirm this, we repeated the same set of experiments over HDDs, and found that SFS is quite promising in HDDs: although the slow random reads in HDDs make SFS slightly less effective, SFS still outperforms LFS by 1.7 times.

A Log-Structured File System Based on LevelDB

Traditional file systems have some shortages in storing small files, such as randomness of data layout, waste of disk space and lack of inode resources. In this thesis, a log-structured file system named LevelFS based on LevelDB is presented. By setting the write buffer, it can make disk randomized writes of small files into disk sequential writes, and reduce the distance of related data, so as to improve the read and write performance of file system. Experiments show that LevelFS can greatly improve read and write performance of small files without affect the large ones.

The Design of an Embedded File System Based on the Flash

Analyzed the unique characteristics of reading and writing the flash memory, and researched the ways of designing a file system. The target embedded file system has the advantages of the traditional file system and the sequential file system. It is able to adapt to the unique characteristics of reading and writing the flash memory. It can solve the inconsistent data problem in the embedded device when power down. And it can solve the bad blocks of the flash which are generated by erase operations also.

Interview

Dr Paulino Ruiz-de-Clavijo from Universidad de Sevilla in Spain talks about the research behind his Letter ‘NanoFS: hardware-oriented file system’ on page 1216. Dr Paulino Ruiz-de-Clavijo During my higher education as a computer engineer I have always been interested in low-level programming and digital electronic circuits. In 1999 I started my PhD research at the University of Seville and the National Microelectronics Centre - CSIC, which I completed at the ID2 research group of the University of Seville. That period allowed me to start working with reprogrammable devices; FPGAs in particular. This technology fascinated me because it opened up the possibility to reuse my knowledge and experience in software programming to develop hardware using hardware description languages (HDLs), and I found the hardware-software codesign a great field to explore. The research in our group involves the wide field of hardware-software codesign and reconfigurable hardware. Our work is mainly focused on the design of digital embedded systems and SoC (system on chip) on FPGAs and the design of application-specific peripherals for SoC. Relating to SoCs, we have worked with open platforms like the LEON and OpenRISC soft-core microprocessors. In our group we have also developed hardware for industrial projects like a SNTP synchronisation platform based on FPGA and a frequency synchroniser for OFDM systems. We have developed a new file system called ‘nano file system’ (NanoFS) which is specially tailored to be implemented in hardware. It has been designed from scratch keeping in mind the hardware implementation limitations. The main objective is to be able to do file system operations just with digital circuits while keeping a minimum hardware footprint. The approach keeps the high level file system features and it can be used in both low level embedded systems and high-level operating systems. An all-hardware file system reader core is presented in our Letter which has been successfully implemented and tested in a low-range FPGA device. It can be included in custom designs to read data from files stored in external storage media. Most file systems are designed with two overall objectives: to be optimal for the storage medium and/or to be optimal when implemented in software. Our proposal is a hardware-friendly file system suitable for embedded systems. The file system can be used without an operating system or before the operating system is booted. File systems like FAT or EXT2 are widely used in electronic devices and SoC. The layouts of these traditional file systems are intended to be managed with software algorithms. When these algorithms are translated to HDL, a large amount of resources are necessary. There have been a few initiatives to develop file systems for embedded systems to solve these problems that simplifies data structures and improves some software routines. These file systems are written in C and supply an API for some microcontrolers but are not designed to achieve a hardware implementation. Some custom designs in hardware require dynamic configuration, such as the network configuration, calibration parameters, etc. In previous work we developed a SNTP synchronisation server designed as a full hardware solution to improve the precision. To try different configurations of these servers, we had to reprogram the FPGA device. We thought that a data file in a file system that could be accessed directly from the device hardware would be a faster and more flexible method to achieve that. We created NanoFS to solve the problems we found when we tried to access other file systems from hardware. Currently, from a software perspective, a dedicated file manager application is used to access the file system. We are working to improve a Linux native driver for NanoFS so we can access the files right from the operating system. At this moment the driver works in read-only mode. We are also migrating the file manager to the MS Windows platform. From the hardware perspective, we are developing some new hardware cores with write support. We plan to test the write support module with current solid state disks to measure the performance. Further applications we have planned to use the new file system are: data acquisition systems; boot loaders for embedded systems; and FPGA dynamic reconfiguration.

Scalable Multimedia Archive Storage System Research

In the traditional file system, The lack of contact between the file name and the data content led to waste a lot of availability storage space, especially in large data archive storage system. This paper designs and implements a scalable multimedia archive storage system, called IMCAS. IMCAS use CAS store the archive data and use real-time monitor program automatically extract metadata information. through dynamic loading extraction module, the system can support variety of multimedia form.

Delegation-Based I/O Mechanism for High Performance Computing Systems

Massively parallel applications often require periodic data checkpointing for program restart and post-run data analysis. Although high performance computing systems provide massive parallelism and computing power to fulfill the crucial requirements of the scientific applications, the I/O tasks of high-end applications do not scale. Strict data consistency semantics adopted from traditional file systems are inadequate for homogeneous parallel computing platforms. For high performance parallel applications independent I/O is critical, particularly if checkpointing data are dynamically created or irregularly partitioned. In particular, parallel programs generating a large number of unrelated I/O accesses on large-scale systems often face serious I/O serializations introduced by lock contention and conflicts at file system layer. As these applications may not be able to utilize the I/O optimizations requiring process synchronization, they pose a great challenge for parallel I/O architecture and software designs. We propose an I/O mechanism to bridge the gap between scientific applications and parallel storage systems. A static file domain partitioning method is developed to align the I/O requests and produce a client-server mapping that minimizes the file lock acquisition costs and eliminates the lock contention. Our performance evaluations of production application I/O kernels demonstrate scalable performance and achieve high I/O bandwidths.

Retrieving relevant information: traditional file systems versus tagging

Purpose – The aim is to determine, in a business context, if tagging is a more effective method of discovering relevant information when compared to traditional hierarchical filing systems.

Techniques for improving performance of the FAST (fully-associative sector translation) flash translation layer

A Flash Translation Layer (FTL) emulates a block device interface on top of flash memory to support traditional disk-based file systems. Due to the erase-beforewrite feature of flash memory, an FTL usually performs outof- place updates and uses a cleaning procedure to reclaim obsolete data. The FAST (Fully-Associative Sector Translation) FTL is one of the most well-known FTLs and has been used in many previous studies. It reserves a small portion of the flash storage as the log area and uses the blocks in this area (i.e., log blocks) to accommodate data overwrites. Among all the log blocks, one (called the SW log block) is used to accommodate sequential overwrites, while the others (called RW log blocks) are used to accommodate random overwrites. In this paper, two techniques are proposed to reduce the cleaning cost and hence to improve the performance of FAST. The first technique reduces the cost of RW log block reclamation by utilizing efficiency-driven cleaning policies, and the second technique replaces high-cost SW log block reclamation with low-cost RW log block reclamation. We have implemented the proposed techniques on the FAST FTL as well as the HFTL, an FTL based on FAST. The simulation results on six real/benchmark-based workloads show that the proposed techniques significantly reduce the cleaning cost, by up to 97.64% in FAST and 66.80% in HFTL, demonstrating the effectiveness of the proposed techniques1.

Implementing a genomic data management system using iRODS in the Wellcome Trust Sanger Institute

BackgroundIncreasingly large amounts of DNA sequencing data are being generated within the Wellcome Trust Sanger Institute (WTSI). The traditional file system struggles to handle these increasing amounts of sequence data. A good data management system therefore needs to be implemented and integrated into the current WTSI infrastructure. Such a system enables good management of the IT infrastructure of the sequencing pipeline and allows biologists to track their data.ResultsWe have chosen a data grid system, iRODS (Rule-Oriented Data management systems), to act as the data management system for the WTSI. iRODS provides a rule-based system management approach which makes data replication much easier and provides extra data protection. Unlike the metadata provided by traditional file systems, the metadata system of iRODS is comprehensive and allows users to customize their own application level metadata. Users and IT experts in the WTSI can then query the metadata to find and track data.The aim of this paper is to describe how we designed and used (from both system and user viewpoints) iRODS as a data management system. Details are given about the problems faced and the solutions found when iRODS was implemented. A simple use case describing how users within the WTSI use iRODS is also introduced.ConclusionsiRODS has been implemented and works as the production system for the sequencing pipeline of the WTSI. Both biologists and IT experts can now track and manage data, which could not previously be achieved. This novel approach allows biologists to define their own metadata and query the genomic data using those metadata.

Digital teaching files – a useful teaching tool for the modern radiologist

Radiologists have always collected copies of model examples and interesting cases encountered in daily practice to use for teaching purposes. A collection of teaching files is an important resource for medical education and the dissemination of knowledge in radiology. Furthermore, the presence of a radiological teaching file is also a requirement at several universities and tertiary institutions in South Africa and abroad, where radiology postgraduate training is conducted.
 
 The advent of digital radiology and the more widespread use of picture archiving and communication systems (PACS) have led to a unique opportunity to change the way that radiology teaching cases are collected, stored and managed. Traditional teaching file systems are rapidly becoming obsolete and incompatible with the digital environment of modern radiology departments.

A semantic file system for integrated product data management

A mechatronic system is a synergistic integration of mechanical, electrical, electronic and software technologies into electromechanical systems. Unfortunately, mechanical, electrical, and software data are often handled in separate Product Data Management (PDM) systems with no automated sharing of data between them or links between their data. Presently, this is a significant drawback with regard to supporting the collaborative and integrated design of mechatronic systems. One underlying research question to be addressed is: “ How can such domain-specific PDM systems be integrated and what are the standards needed for this?” Our starting point for addressing this question is to look into how engineering data is stored today. We believe that the more intelligent and sophisticated the mechanisms for file management are, the more Computer-aided Engineering systems will benefit from them in terms of making integrated cross-disciplinary product development more efficient. However, while Computer-Aided Design and Engineering (CAD/CAE) systems and related tools such as, for example, Product Data Management (PDM), Engineering Data Management (EDM), etc. have significantly matured over the past ten years, most of their associated database systems are still based on traditional file systems and hierarchical directory structures. In light of this context, we will initially discuss a number of disadvantages of current file management systems. In the body of the paper our main contribution is presented. That is, a formal mathematical model of a new semantic file system, SIL ( Semantics Instead of Location), that allows engineers to access data based on semantic information rather than storage location is proposed. A major difference between our approach and previous related work is that we do not aim at yet another point solution and, instead, propose an approach that may be employed by next generation engineering data processing systems on a larger scale. In addition, a corresponding programming interface along with a graphical user interface used as a file browser is presented and the benefits of utilizing the proposed semantic file system for product data management in the field of integrated design of mechatronic systems are discussed.

Files are Siles

With the increasing storage capacity of personal computing devices, the problems of information overload and information fragmentation are apparent on users’ desktops. For the Web, semantic technologies solve this problem by adding a machine-interpretable information layer on top of existing resources. It has been shown that the application of these technologies to desktop environments is helpful for end users. However, certain characteristics of the Semantic Web architecture that are commonly accepted in the Web context are not desirable for desktops. To overcome these limitations, the authors propose the sile model, which combines characteristics of the Semantic Web and file systems. This model is a conceptual foundation for the Semantic Desktop and serves as underlying infrastructure on which applications and further services can be built. The authors present one service, a virtual file system based on siles, which allows users to semantically annotate files and directories and keeps full compatibility to traditional hierarchical file systems. The authors also discuss how Semantic Web vocabularies can be applied for meaningful annotation of files and present a prototypical implementation of the model and analyze the performance of typical access operations, both on the file system and metadata level.

A Real-Time File System for Constrained Quality of Service Applications

The prevalence of multi-tuners, high-definition digital video recorder systems and home networking is increasing the number of simultaneous streams that must be processed by recorder storage devices. Whereas recent hard-disk drives provide enough performance to theoretically handle such workloads, general purpose file systems and I/O schedulers used by operating systems such as Linux do not satisfy the quality of service (QoS) requirements necessary for efficient processing of real-time video streams. In this paper, we introduce the Audio/Video File System (AVFS) composed of a file system and a disk I/O scheduler optimized for handling simultaneous high bit-rate real-time streams. Using a precise QoS measurement method, evaluation results of a Linux implementation of AVFS show that, compared to traditional file systems such as ext3 and JFS, AVFS provides QoS guarantees for real-time streams and more stable performance.