NFS Server Research Articles

Everyone Loves File

Oracle File Storage Service (FSS) is an elastic filesystem provided as a managed NFS service. A pipelined Paxos implementation underpins a scalable block store that provides linearizable multipage limited-size transactions. Above the block store, a scalable B-tree holds filesystem metadata and provides linearizable multikey limited-size transactions. Self-validating B-tree nodes and housekeeping operations performed as separate transactions allow each key in a B-tree transaction to require only one page in the underlying block transaction. The filesystem provides snapshots by using versioned key-value pairs. The system is programmed using a nonblocking lock-free programming style. Presentation servers maintain no persistent local state making them scalable and easy to failover. A non-scalable Paxos-replicated hash table holds configuration information required to bootstrap the system. An additional B-tree provides conversational multi-key minitransactions for control-plane information. The system throughput can be predicted by comparing an estimate of the network bandwidth needed for replication to the network bandwidth provided by the hardware. Latency on an unloaded system is about 4 times higher than a Linux NFS server backed by NVMe, reflecting the cost of replication. FSS has been in production since January 2018 and holds tens of thousands of customer file systems comprising many petabytes of data.

Tuning and Optimizing Network File System Server Performance

Network file system is one of distributed file systems that are used over network to provide remotely access to data on the servers. The last version of NFS has a number of features that help in increasing server performance. However, the provided features are limited to a specific type of operations on the server. Therefore, finding more options may work alongside to enhance server performances. This research is discussing the provided features in NFSv4 and activates one of the tuning options to tests NFS server performance.

Storage-aware Algorithms for Scheduling of Workflow Ensembles in Clouds

This paper focuses on data-intensive workflows and addresses the problem of scheduling workflow ensembles under cost and deadline constraints in Infrastructure as a Service (IaaS) clouds. Previous research in this area ignores file transfers between workflow tasks, which, as we show, often have a large impact on workflow ensemble execution. In this paper we propose and implement a simulation model for handling file transfers between tasks, featuring the ability to dynamically calculate bandwidth and supporting a configurable number of replicas, thus allowing us to simulate various levels of congestion. The resulting model is capable of representing a wide range of storage systems available on clouds: from in-memory caches (such as memcached), to distributed file systems (such as NFS servers) and cloud storage (such as Amazon S3 or Google Cloud Storage). We observe that file transfers may have a significant impact on ensemble execution; for some applications up to 90 % of the execution time is spent on file transfers. Next, we propose and evaluate a novel scheduling algorithm that minimizes the number of transfers by taking advantage of data caching and file locality. We find that for data-intensive applications it performs better than other scheduling algorithms. Additionally, we modify the original scheduling algorithms to effectively operate in environments where file transfers take non-zero time.

Building an organic block storage service at CERN with Ceph

Emerging storage requirements, such as the need for block storage for both OpenStack VMs and file services like AFS and NFS, have motivated the development of a generic backend storage service for CERN IT. The goals for such a service include (a) vendor neutrality, (b) horizontal scalability with commodity hardware, (c) fault tolerance at the disk, host, and network levels, and (d) support for geo-replication. Ceph is an attractive option due to its native block device layer RBD which is built upon its scalable, reliable, and performant object storage system, RADOS. It can be considered an "organic" storage solution because of its ability to balance and heal itself while living on an ever-changing set of heterogeneous disk servers.This work will present the outcome of a petabyte-scale test deployment of Ceph by CERN IT. We will first present the architecture and configuration of our cluster, including a summary of best practices learned from the community and discovered internally. Next the results of various functionality and performance tests will be shown: the cluster has been used as a backend block storage system for AFS and NFS servers as well as a large OpenStack cluster at CERN. Finally, we will discuss the next steps and future possibilities for Ceph at CERN.

Glitz

We propose Glitz, a system to integrate multiple file server federation regimes. NFS version 4 is a significant advance over prior versions of NFS, in particular specifying how NFS clients can navigate a large, multi-server namespace whose constituent parts may be replicated or moved while in use, as specified by NFS servers. This capability is essentially the same as that of previous distributed file systems such as AFS [7]. Sophisticated as this NFS capability is, it does not address the larger problem of building a usable system atop this basic capability. Multiple single-architecture solutions have been proposed, but each of these is based on an architecture for server federation that does not easily admit other members [3, 16, 18]. Glitz allows those multiple-server federations to interoperate and collaborate in a vendor-independent fashion. We give a history of file system federation efforts as well as a detailed tour of Glitz and its benefits for vendors.

The ATLAS Tier-3 in Geneva and the Trigger Development Facility

The ATLAS Tier-3 farm at the University of Geneva provides storage and processing power for analysis of ATLAS data. In addition the facility is used for development, validation and commissioning of the High Level Trigger of ATLAS [1]. The latter purpose leads to additional requirements on the availability of latest software and data, which will be presented. The farm is also a part of the WLCG [2], and is available to all members of the ATLAS Virtual Organization. The farm currently provides 268 CPU cores and 177 TB of storage space. A grid Storage Element, implemented with the Disk Pool Manager software [3], is available and integrated with the ATLAS Distributed Data Management system [4]. The batch system can be used directly by local users, or with a grid interface provided by NorduGrid ARC middleware [5]. In this article we will present the use cases that we support, as well as the experience with the software and the hardware we are using. Results of I/O benchmarking tests, which were done for our DPM Storage Element and for the NFS servers we are using, will also be presented.

Using Firewalls to Enforce Enterprise-wide Policies over Standard Client-Server Interactions

We propose and evaluate a novel framework for enforcing global coordination and control policies over message passing software components in enterprise computing environments. This framework combines the use of firewalls, both per-node software and dedicated firewalls, with an existing coordination and control system to enforce policies that, among other properties, are stateful and communal. The firewalls act as a set of distributed reference monitors that filter messages exchanged between the interacting software components. The coordination and control system coordinates the firewalls to enforce a specific set of policies, passing only messages allowed by these policies. Filtering decisions may be based on credentials presented to the coordination and control system as well as system state accumulated over time. This filtering approach decouples coordination and control from application implementation, allowing the coordination and control mechanism and application implementations to evolve independently of each other. We demonstrate the power of our framework by using it to specify and enforce an RBAC policy with delegation, revocation, and separation-of-duty over accesses to a cluster of NFS and SMB file servers without changing any client or server implementations. Measurements show that our framework imposes acceptable overheads when enforcing this policy.

Separating agreement from execution for byzantine fault tolerant services

We describe a new architecture for Byzantine fault tolerant state machine replication that separates agreement that orders requests from execution that processes requests. This separation yields two fundamental and practically significant advantages over previous architectures. First, it reduces replication costs because the new architecture can tolerate faults in up to half of the state machine replicas that execute requests. Previous systems can tolerate faults in at most a third of the combined agreement/state machine replicas. Second, separating agreement from execution allows a general privacy firewall architecture to protect confidentiality through replication. In contrast, replication in previous systems hurts confidentiality because exploiting the weakest replica can be sufficient to compromise the system. We have constructed a prototype and evaluated it running both microbenchmarks and an NFS server. Overall, we find that the architecture adds modest latencies to unreplicated systems and that its performance is competitive with existing Byzantine fault tolerant systems.

The Design of the Expand Parallel File System

This article describes an implementation of MPI-IO using a new parallel file system, called Expand (Expandable Parallel File System), which is based on NFS servers. Expand combines multiple NFS servers to create a distributed partition where files are striped. Expand requires no changes to the NFS server and uses RPC operations to provide parallel access to the same file. Expand is also independent of the clients, because all operations are implemented using RPC and NFS protocols. Using this system, we can join heterogeneous servers (Linux, Solaris, Windows 2000, etc.) to provide a parallel and distributed partition. The article describes the design, implementation and evaluation of Expand with MPI-IO. This evaluation has been made in Linux clusters and compares Expand and PVFS.

Prefetching over a network

We discuss CTIP, an implementation of a network filesystem extension of the successful TIP informed prefetching and cache management system. Using a modified version of TIP in NFS client machines (and unmodified NFS servers). CTIP takes advantage of application-supplied hints that disclose the application's future read accesses. CTIP uses these hints to aggressively prefetch file data from an NFS file server and to make better local cache replacement decisions. This prefetching hides disk latency and exposes storage parallelism. Preliminary measurements that show CTIP can reduce execution time by a ratio comparable to that obtained with local TIP over a suite of I/O-intensive hinting applications. (For four disks, the reductions in execution time range from 17% to 69%). If local TIP execution requires that data first be loaded from remote storage into a local scratch area, then CTIP execution is significantly faster than the aggregate time for loading the data and executing. Additionally, our measurements show that the benefit of CTIP for hinting applications improves in the face of competition from other clients for server resources. We conclude with an analysis of the remaining problems with using unmodified NFS servers.

Analysis of caching algorithms for distributed file systems

When picking a cache replacement policy for file systems, LRU (Least Recently Used) has always been the obvious choice, because of the temporal locality found in programs and data. However, in the case of Sun NFS servers much of the locality is filtered out by the client cache. It has been conjectured that this filtering of locality by the client caches render LRU ineffective as a server cache replacement policy. This study disproves the conjecture by simulating a NFS server cache with real world traces. Traces were taken of NFS read and write requests sent to an NFS server. These traces were then run through a cache simulator using six different cache replacement policies: Least Recently Used (LRU), Least Frequently Used (LFU), Frequency Based Replacement (FBR), First In First Out (FIFO), Random (RAND), and Optimal (OPT). RAND and OPT were used to provide lower and upper bounds on performance. Results show that LRU is an effective NFS server cache replacement policy and frequency based tend to exhibit erratic behavior in the presence of temporal locality and sequentially accessed files.