Chapter 4 - Frameworks for Big Data Integration, Warehousing, and Analytics

Abstract

Big data is a term for large and complex datasets that traditional processing approaches are not suitable to deal with them. Big data usually comes from the Internet, enterprise systems, Internet of Things, and other information systems. Data collection and preparation, storage management, data processing, data analysis, and knowledge presentation would generate new insights to support decision-making and business intelligent operation. Smart grid is a developing trend of electrical power energy industry. The core message is to implement the next generation of cyber-physical systems shaping future energy industry that is based on a deep merge of operational technology and internet information technology. The growth of smart grid is dependent on the availability of high performance computing (HPC) and analytics technology to process a massive amount of data set. Deployment of advanced technologies within smart grid and usage of state-of-the-art computing systems provide utility companies with innovative capabilities. These advances lead to unprecedented explosion of data volumes. As smart grid operations will leverage advanced metering infrastructure to drive more real time decision-making and operational activities, complex event processing and stream computing are needed for the modern smart grid. The chapter discusses one core technique that would support the growth of smart grid, big data with HPC, with a focus on the platform, data integration, warehousing, and analytics that are particularly adaptive to handle a variety of characteristics of energy industry data. Finally, the chapter summarizes and proposes a comprehensive, technical solution for smart grid platform with applications focusing on complementary operation of multiform energy system that supports all aspects within a data lifetime cycle, e.g., acquisition, storage, analytics, and visualization.