Examine Heuristic Data Lake Management Using AWS: A Big Data Handling Approach

Abstract

In this era of technology, the most valued asset can be ‘Data’. With the increasing number of data, the value of it keeps increasing.  Data storage and data manipulate for to achieve some particular goals or business requirements increasing in number and storing it has become a complex and tedious task. With the use of some advanced technologies like hadoop, it simplified the data storing process, but due to rapid development and excessive use of AI and ML, tons of data is collected. The quintessence is to ascertain an extra cost effective storage alternative. This paper provides with an effective solution to store data over the cloud with numerous benefits over traditional data storage methods by developing a data lake using AWS a Cost Effective Data Lake Management algorithm (CEDLMA). Furthermore, the functionalities of data lake include managing and storing sorted as well as unsorted data, gathering various analytics from the data lake as per business requirements.  Proposed work is evaluated with AWS’s IAM and S3 services.