Develop a Model to Secure and Optimize Distributed File Systems for ISP Log Management

Abstract

Distributed file system (DFS) is a transformation of traditional file systems to perform file read, write and execution of petabyte or larger-sized datasets with high-velocity and different structures. In order to process these large amounts of data in an inexpensive and efficient way, Hadoop Distributed File System (HDFS) is used and designed to scale up from a single server to hundreds of servers, with a very high degree of fault tolerance. Secured and optimized distributed file system for ISP user log management is a centralized log analytics platform that supports collaboration in a distributed environment. DFS for ISP log provides a uniform name space for all ISP along with location independence and location transparency. Any user’s internet access log can be accessed and analyzed using apache sparks data brick file system on stored log in DFS. In this paper, at first, limitation of popular DFS like NFS and CIFS is explored. ISP log management, DFS file architecture and security is considered for designing proposed ISP log management system. Finally, simulation on proposed distributed file system and query processing is performed to obtain the desired result related to optimization and security of the proposed distributed log analysis system.