Reference Data Architecture for Extreme Data Processing in Real Time

Abstract

Data which is known to be key asset for any enterprise has pivotal role in driving businesses decisions. ii) The technology advancement has open new avenues which has transformed the way we generate, consume or use data to derive value from it. iii) The speed at which data is generated is growing rapidly, which is quite complex, leading to advancement in the field of big data. “Big data” is a term used for massive data sets which have complex structure and cannot be handled by standard software or platforms. iv) The category of data which has huge potential to become part of this big data journey, huge computational needs and usage over the coming years will be due to introduction of IoT (internet of things) based devices, digital economies, digital devices and connected devices. v) This category of data has possibilities due to Digital Transformations and will be empirical to look for possibilities where it can be effectively used with a faster turn-around to consumers for their analytical or reporting needs. vi) This paper will try to explore alternative frameworks which we can use for evolving data giving more computational power to existing Big Data Platforms highlighting why they are important given the current challenges with available platforms, tools or technologies. Keywords — Analytics Platforms, Artificial Intelligence, Big Data, Blockchain, Cloud Analytical Platforms, Data Architecture, Data Engineering, Data Lake, Digital, Ensembling, Hyperparameters, In-Memory, Internet of Things (IoT), Machine or Deep Learning, Master Data Management, Statistical Modelling