Big Data: Does it Call for Distributed File System

Abstract

Today, in order to support decision for strategic advantages alignment, companies‟ have started to realize the importance of using large data. It is being observed through different study cases that “Large data usually demands for faster processing”. As a result, companies are now investing more in processing larger sets of data rather than investing in expensive algorithms. A larger amount of data gives a better inference for decision making but also working with it can create challenge due to processing limitations. In order to easily manage and use this large amount of content in a proper systematic manner, Big Data, HDFS & other file systems were being introduced. Big data is used for „larger data sets having more varied and complex structure, having problems in analyzing, visualizing and storing for further processing‟. The process of examining such large amounts of data inorderto reveal hidden patterns and secret correlations is named as Big Data Analytics. The useful information for companies or organizations will help them in gaining richer and deeper insights and getting advantages over the competition. Implementation of this Big Data needs to be analyzed and executed as accurately as possible. This term paper give an overview about what Big Data is, its classification, challenges it faces, need for Distributed File System, Hadoop and its components i.e. Hadoop Distributed file System and Map Reduce, and application of HDFS in Cloud Computing