Smart Grid Big Data Research Articles

Real-time Access and Processing of Massive Measurement Data in Smart Power Grids

Abstract This paper introduces a comprehensive smart grid big data solution, focusing on the processing and analysis of vast grid data to facilitate critical applications such as data resource management, real-time monitoring of grid conditions, and predictive load forecasting. Specifically, grid monitoring data are routed to distributed message queues, enhancing the indexing speed of real-time data access via the implementation of a B+ tree indexing algorithm. Furthermore, an optimized ant colony algorithm enhances the integration of big data with other advanced technologies, enabling efficient classification of diverse power information from multiple metering data sources. For empirical validation, data from national grid power meters were analyzed. Correlation analysis revealed that the correlation coefficients among smart meters 1, 5, and 15 are predominantly higher than 0.9. These coefficients tend to become more pronounced with time, delineating clearer connections and distinctions among the data from these meters. Additionally, the correlation between temperature and load values ranged between 0.91 and 0.98, significantly influencing daily load forecasts. The year 2023 saw an increase in the detection of online monitoring faults by 236 compared to 2020, underscoring the enhanced capabilities of smart grid condition maintenance. Moreover, monitoring data from various nodes of the national grid, with the exception of node 1#, exhibited deviation values ranging from 0.01 to 0.05, indicating high monitoring precision. In conclusion, the big data-driven approach to smart grid management presented in this study not only predicts load and performs state inspections efficiently but also holds significant practical value, suggesting a robust framework for future smart grid applications.

Canonical Correlation Analysis and Visualization for Big Data in Smart Grid

Canonical Correlation Analysis and Visualization for Big Data in Smart Grid

Adaptive Energy Management of Big Data Analytics in Smart Grids

The smart grid (SG) ensures the flow of electricity and data between suppliers and consumers. The reliability and security of data also play an important role in the overall management. This can be achieved with the help of adaptive energy management (AEM). This research aims to highlight the big data issues and challenges faced by AEM employed in SG networks. In this paper, we will discuss the most commonly used data processing methods and will give a detailed comparison between the outputs of some of these methods. We consider a dataset of 50,000 instances from consumer smart meters and 10,000 attributes from previous fault data and 12 attributes. The comparison will tell us about the reliability, stability, and accuracy of the system by comparing the output of the various graphical plots of these methods. The accuracy percentage of the linear regression method is 98%; for the logistic regression method, it is 96%; and for K-Nearest Neighbors, it is 92%. The results show that the linear regression method applied gives the highest accuracy compared to logistic regression and K-Nearest Neighbors methods for prediction analysis of big data in SGs. This will ensure their use in future research in this field.

Grid data asset relationship and intelligent classification integrating knowledge graph and Internet of Things

Abstract With the development of smart grids, power grids have accumulated massive amounts of data in various links such as power generation, transmission, substation, distribution, power consumption, and dispatch. More and more big data applications are beginning to be applied in various professional fields of the power grid. Promote the application and value discovery of smart grid big data through data fusion inside and outside the grid. Grid data has become an important asset for enterprise development, but power grid enterprises lack effective technical means to solve the whole life cycle monitoring and relationship of power grid data assets. Aiming at the relationship between power grid data assets, this paper proposes a set of grid data asset relationship and intelligent classification framework that integrates knowledge graph and Internet of Things. First, the grid knowledge graph extraction relationship is carried out by ProjE algorithm. Then, the relationship between power grid data assets and intelligent classification framework that integrates knowledge graph and Internet is proposed. Finally, the corresponding classification application is proposed by using intelligent classification algorithm. Experimental results show that the intelligent classification accuracy rate can reach 93.12% under the relationship between the knowledge graph and the Internet data assets, which has a new idea for the future development of the relationship between power grid data assets.

Big Data Architecture Design of Smart Grid Based On Big Data Platform

A novel big data-driven approach has been proposed to address the challenges of extensive data and user application difficulties in the smart grid. A smart grid big data architecture based on a big data platform has been designed. The random matrix theory is applied to establish a random matrix algorithm model, which samples data from the smart grid big database and undergoes training and learning to construct a data model for user requirements. Through calculations, the macroscopic data hidden in the smart grid big data is transformed into microscopic data for users' reference, enabling the identification of parameters that affect the normal operation of the smart grid at its core. The computed data can be displayed locally and also uploaded remotely to the SG186 marketing system for various user applications. Additionally, the data can be permanently stored in the cloud through wireless communication. By incorporating the random matrix theory algorithm into the smart grid big data architecture, the proposed solution not only enhances the intuitive visualization of smart grid big data but also provides technical references for future work.

BBNP: A Blockchain-Based Novel Paradigm for Fair and Secure Smart Grid Communications

As the future energy infrastructure, smart grid aims to overcome the disadvantages of traditional power grid, e.g., low efficiency and unstable service. However, the frequent collection and analysis of the user’s electricity data may bring various security and privacy threats. Besides, the traditional centralized data storage model in the smart grid is prone to the single point of failure. To address these challenges, in this article, for fair and secure smart grid communication, a blockchain-based novel paradigm, named BBNP, is proposed. Specifically, based on the pseudorandom function and auxiliary information generation and sharing technology, a lightweight data aggregation protocol is designed first to protect the user’s data privacy and ensure communication confidentiality. Then, a novel efficient authentication mechanism is proposed to generate and share session keys in a noninteractive way, which is leveraged for MAC authentication to achieve data integrity of the transmitted data. After that, based on the subjective logic reputation model, a blockchain node consensus mechanism is studied to efficiently store smart grid big data and effectively solve the single point failure problem. By constructing the long-term reputation model for consensus nodes (CNs) and integrating batch verification technology, the problems of CN fair selection and scalability of large-scale nodes are solved simultaneously. Finally, the performance evaluation indicates that BBNP outperforms the state-of-the-art similar schemes in computing complexity, communication cost, system availability, and fairness of block generation.

The design of power grid data management system based on blockchain technology and construction of system security evaluation model

The construction of smart grid has triggered comprehensive changes in power equipment. At the same time, massive data are waiting to be screened, analyzed and processed. However, there are many data sources and complex data structure in smart grid, and smart grid requires higher security, reliability, traceability, non-repudiation and real-time performance of data processing, which poses unprecedented challenges to the traditional data management. Blockchain technology has the advantages of decentralization, autonomy, leakage prevention and openness. It can match the demands of power data management in terms of operation mode, security protection, complement power data, and better meet the security and credibility requirements of energy Internet. Therefore, this paper designs the smart grid data management system based on blockchain, and constructs the system security evaluation model by using AHP and expert consultation method. Firstly, the characteristics of smart grid big data and blockchain technology are analyzed; Then the technical architecture of data management system is designed by using alliance chain, and the technical architecture is analyzed in detail; Finally, the index factors affecting system security are combed, and the system security evaluation model is constructed by AHP. Distributed storage, asymmetric encryption, consensus mechanism, and other technologies of blockchain technology can well solve the problems of power grid data storage and application, and better ensure the security of smart grid data.

Architecture design of big data analysis platform for smart grid in rural areas

The power grid is facing more and more challenges with the development of new energy. Thus, uncoordinated network load and local heavy overload problems always affect the development of rural power grid, especially the impact of voltage quality and power supply reliability on the rural power grid. According to the characteristics of rural power grid, firstly, the work establishes the overall architecture of smart grid big data analysis platform and then analyzes the software functions and hardware configuration of the platform, finally, discusses the deployment mode of rural big data platform and power grid big data analysis platform. Furthermore, it introduces the technical principle and data transmission principle of smart grid big data analysis platform. Finally, the paper emphasizes that we should build a rural intelligent big data analysis platform according to building a green and transparent smart platform area and building an integrated and open intelligent cockpit.

Application Research of Big Data Real-time Processing Technology in Smart Grid

Big data is valuable, and big data processing technology can efficiently process massive data and get the value in big data, so the research and application of big data technology become popular nowadays. Smart grid is one of the most potential field for application of big data technology, applying big data technology to smart grid can improve the reliability and economy of smart grid. The operating conditions of power system changes rapidly, therefore, real-time processing technology became the focus of application field. This paper reviewed the data sources, data characteristics and data application scenarios of big data in smart grid, analyzed data processing modes of big data processing technology: batch processing and stream processing, and compared the realization methods of real-time big data processing, summarized the application of big data real-time processing technology in smart grid, and the challenges in the application, and finally, prospected the application of real-time big data processing technology in smart grid.

Research on Anti-Interference Screening of On-line Measurement of Battery Ageing under Smart Grid Big Data

By analyzing the traditional operation and maintenance mode of substation batteries, the paper proposes a new remote online maintenance and management system for batteries based on smart grids, which changes the traditional maintenance method of using discharge load meters to check offline battery discharge packs. Single battery online voltage inspection, internal resistance test, and dynamic equalization, as well as battery pack remote online verification discharge and equalization charging, real-time monitoring of battery aging, promote intelligent, efficient, and unmanned maintenance battery status in substations, Provide strong technical and program guarantees.

Privacy Protection of the Smart Grid System Based on Blockchain

With the development of new technologies for smart grid construction, the smart grid is gradually developing. Smart grid construction is an important field of next generation information technology, and its value and significance have been widely recognized. Smart grid construction not only provides development opportunities, but also brings great challenges. Security and privacy have become serious problems in the development of smart grid. Based on this, this paper discusses the system structure of the block chain, and puts forward the security diagnosis scheme of the smart grid according to the existing problems in the smart grid. Based on the theoretical basis of smart grid big data technology, this paper combines the grid data encryption process with the privacy protection process under the premise of protecting privacy and security, and develops the latest network security protocol. Then use block chain distributed storage to reduce the connection protection process in privacy, and use multi-level encryption. The experimental results show that the technology can improve the rate of sensitive data hiding and has good anti-cracking ability, which provides a theoretical reference for the construction of enterprise power grid.

Distributed Tree-Based Machine Learning for Short-Term Load Forecasting With Apache Spark

Machine learning algorithms have been intensively applied to perform load forecasting to obtain better accuracies as compared to traditional statistical methods. However, with the huge increase in data size, sophisticated models have to be created which require big data platforms. Optimal and effective use of the available computational resources can be attained by maximizing the effective utilization of the cluster nodes. Parallel computing is demanded to allow for optimal resource utilization in dealing with smart grid big data. In this paper, a master-slave parallel computing paradigm is utilized and experimented with for load forecasting in a multi-AMI environment. The paper proposes a concurrent job scheduling algorithm in a multi-energy data source environment using Apache Spark. An efficient resource utilization strategy is proposed for submitting multiple Spark jobs to reduce job completion time. The optimal value of clustering is used in this paper to cluster the data into groups to be able to reduce the computational time additionally. Multiple tree-based machine learning algorithms are tested with parallel computation to evaluate the performance with tunable parameters on a real-world dataset. One thousand distribution transformers’ real data from Spain for three years are used to demonstrate the performance of the proposed methodology with a trade-off between accuracy and processing time.

Big Data Management in Smart Grids: Technologies and Challenges

Smart grids are re-engineering the electricity transmission and distribution system throughout the world. It is an amalgam of increased digital information with the electrical power grids. Managing the data generated from the grid efficiently is the key to successful knowledge extraction from the smart grid big data. Most of the scientific advancements are becoming data-driven and becoming an interesting area of research for data scientists. It is challenging the world computationally enough to develop new storage methods and data processing technologies. Managing big data involves data cleaning, integration of varied data sources, and decision-making applications. This paper focuses on the study of big data management and proposes a management process to help manage the data in the grid. Data management tools and techniques have been leveraged in understanding the sources and data types in the grid. The paper emphasizes the limitations of the existing solutions inclined towards applications of the smart grid big data.

Analysis of the Development Status and Strategy of Smart Grid Big Data Technology

Analysis of the Development Status and Strategy of Smart Grid Big Data Technology

Smart Grid Big Data Analytics: Survey of Technologies, Techniques, and Applications

Smart grids have been gradually replacing the traditional power grids since the last decade. Such transformation is linked to adding a large number of smart meters and other sources of information extraction units. This provides various opportunities associated with the collected big data. Hence, the triumph of the smart grid energy paradigm depends on the factor of big data analytics. This includes the effective acquisition, transmission, processing, visualization, interpretation, and utilization of big data. The paper provides deep insights into various big data technologies and discusses big data analytics in the context of the smart grid. The paper also presents the challenges and opportunities brought by the advent of machine learning and big data from smart grids.

Application and Analysis of Big Data Technology in Smart Grid

In order to make the modern power system more intelligent, various data acquisition equipment and information management systems are required, such as smart meters, remote terminal units (RTU), phasor measurement units(PMU), power distribution management systems(DMS), and energy management systems(EMS), customer management system(CMS), power generation management system(GMS) have been widely installed. These systems create large amounts of data, which is the main source of smart grid big data. Researching advanced data analysis technology is the most urgent task for the development of smart grid. It can make the best decision for the control and operation of the grid, and can improve the flexibility, safety, reliability and efficiency of the power system. This article analyses the cause of smart grid big data and the importance of data analysis, and discusses the development trend of big data technology. Selectively analysed the source and type of smart grid big data, summarized the characteristics of smart grid big data, gave various application scenarios, and introduced the smart grid big data analysis platform and the functions being developed.

Research on Lean Analysis Algorithm for Equipment Centralized Monitoring in Big Data Era

The equipment monitoring brought by the smart grid big data is difficult to effectively supervise the operation status of the whole network substation, and the typical defects (familial defects) are difficult to classify and locate. This paper proposes the substation operation state evaluation algorithm and typical defect classification algorithm. The operating state evaluation algorithm of the substation is based on different operation data generated by the substation. By normalizing the data of different dimensions, the substation is divided into different operating state levels. The typical defect classification algorithm establishes and maintains the historical experience database, and calculates the conditional probability of each information item to realize the correlation between the signal and the defect, and finally judge whether the signal is from a typical defect. These two algorithms are effective means for equipment monitoring professionals to realize intelligent supervision of substations and equipment in the era of big data.

Evaluation of big data frameworks for analysis of smart grids

With the rapid development of smart grids and increasing data collected in these networks, analyzing this massive data for applications such as marketing, cyber-security, and performance analysis, has gained popularity. This paper focuses on analysis and performance evaluation of big data frameworks that are proposed for handling smart grid data. Since obtaining large amounts of smart grid data is difficult due to privacy concerns, we propose and implement a large scale smart grid data generator to produce massive data under conditions similar to those in real smart grids. We use four open source big data frameworks namely Hadoop-Hbase, Cassandra, Elasticsearch, and MongoDB, in our implementation. Finally, we evaluate the performance of different frameworks on smart grid big data and present a performance benchmark that includes common data analysis techniques on smart grid data.

Edge computing enabled non-technical loss fraud detection for big data security analytic in Smart Grid

With the fast development of Smart Grid globally, the security issues arise sharply and Non-Technical Loss (NTL) fraud is one of the major security issues. There are some existing NTL fraud detectors, however, when big data security challenges emerge in Smart Grid, none of them can detect NTL fraud for big data in Smart Grid. In this paper, we propose ENFD, an NTL detection scheme enabled by Edge Computing and big data analytic tools to address big data NTL fraud detection problem in Smart Grid. The research work provides us with experience of developing big data security solutions in Smart Grid. The experimental results show that ENFD can efficiently detect big data NTL frauds which cannot be detected by the state-of-the-art detectors. ENFD can detect small data NTL frauds as well and the average detection speed is about six to seven times that of the fastest detector exists in the literature.

Guest Editorial: Introduction to Special Section on Big Data Computing for the Smart Grid

The five papers in this special section focuses on the improvement of smart grid operations and applications with emphasis on big data computing, aiming to solicit and publish original research papers on the technologies, algorithms, and methodologies that highlight emerging computation technologies for smart grid big data.