Big Data Volume Research Articles

Applied Digital Documentation with Parametric 3D Modeling and Internet of Trees Functionality for Smart Forests and Monument Landscapes

The Problem: The classic traditional digital documentation (i.e. study, design, development, and maintenance of forests, parks, and monument landscapes) shows significant problems of functionality, adaptability, sustainability, velocity, and viability.
 Target & Research Objectives: In the framework of forests, parks, and monument landscapes, the massive volumes of Big Data (greater variety, arriving in increasing volumes, and with more velocity) can be used to address documentation problems that wouldn’t have been able to tackle before. Smart forest, as an Internet- enabled “product”, requires Big Data because it operates in real-time and requires real-time evaluation and action. Also, the recent research and practice advances in Blockchain data structures and Distributed Ledger Technologies (DLT) support generic structures with many services (e.g. parametric functionality). Hence, a DLT smart digital documentation can address the documentation problems.
 The research objectives, for digital documentation with blockchain functionality (DLT smart documentation), are defined by grouping in user-friendly palettes generic 3D CAD modeling tools which could be parameterized (e.g. trees modeling) in order to support adaptability; and by designing a new flexible and customized GUI (Graphical User Interface).
 Methods: The proposed methodology is based on parameterized 3D modeling of trees usually found in forests and urban parks (width and height parameters). Also, a personalized GUI operates as an interface between the end-user and the parameterized CAD (GUI palettes with many 3D modeling tools). Hence, in this paper, a parameterized 3D design is examined, analyzed, and presented in the context of digital documentation with internet of trees functionality for smart forests, urban parks, monument landscapes, and cultural heritage.
 Results: From the research conducted the results are: (i) a personalized, innovative, and flexible graphical interface (GUI) that could be incorporated in any commercial CAD environment; (ii) many parameterized 3D design tools specialized in the development of forms, plans, and modules (e.g., parametric tree 3D models) of objects and entities found in forests, urban parks, and monument landscapes; and (iii) the introduced internet of trees operativity (software routine), ideal for smart forests, smart monument landscapes, and landscape architecture digital documentation applications with blockchain functionality.
 Application: Possibility to support foresters, engineers, and landscape architects in development studies and documentation of peri-urban forests, recreational parks, pocket parks, monument landscapes, and cultural heritage projects. Increased integration functionality in blockchain knowledge databases.

Distributed dynamic online learning with differential privacy via path-length measurement

Dynamic online learning has been given great concerns as real-time and non-stationary systems develop and it can be used for solving many practical sequential decision problems like dynamic recommendation systems for online advertisement. Due to the large volume of big data in online learning, dynamic online learning is required to be deployed under the distributed framework where the communication among agents is time-varying and real-time. Meanwhile, since the information sharing among agents is vulnerable to interception by the adversary, the privacy protection problem is significant, but to the best of our knowledge, this work is the first to consider privacy-preserving distributed online learning under dynamic circumstances. Specifically, based on a differentially private distributed online gradient descent algorithm, we minimize the regret of the network. Moreover, via the rigorous mathematical analysis, we achieve the sublinear regret bound under ϵ-differential privacy, that is, we respectively obtain O(T+VT) and O(log⁡T+VT) for convex and strongly convex function, where T is the time horizon and the path-length VT reflects the variation of the minimizer sequence. The experiments performed on the real datasets validate that the presented algorithm achieves acceptable optimization performance even when privacy protection is relatively strict.

Securing Big Data Integrity for Industrial IoT in Smart Manufacturing Based on the Trusted Consortium Blockchain (TCB)

The smart manufacturing ecosystem enhances the end-to-end efficiency of the mine-to-market lifecycle to create the value chain using the big data generated rapidly by edge computing devices, third-party technologies, and various stakeholders connected via the industrial Internet of things. In this context, smart manufacturing faces two serious challenges to its industrial IoT big data integrity: real-time transaction monitoring and peer validation due to the volume and velocity dimensions of big data in industrial IoT infrastructures. Modern blockchain technologies as an embedded layer substantially address these challenges to empower the capabilities of the IIoT layer to meet the integrity requirements of the big data layer. This paper presents the trusted consortium blockchain (TCB) framework to provide an optimal solution for big data integrity through a secure and verifiable hyperledger fabric modular (HFM). The TCB leverages trustworthiness in heterogeneous IIoT networks of governing end-point peers to achieve strong integrity for big data and support high transaction throughput and low latency of HFM contents. Our proposed framework drives the fault-tolerant properties and consensus protocols to monitor malicious activities of tunable peers if compromised and validates the signed evidence of big data recorded in real-time HFM operated over different smart manufacturing environments. Experimentally, the TCB has been evaluated and reached tradeoff results of throughput and latency better than the comparative consortium blockchain frameworks.

Mango Pest Detection Using Entropy-ELM with Whale Optimization Algorithm

Image processing, agricultural production, and field monitoring are essential studies in the research field. Plant diseases have an impact on agricultural production and quality. Agricultural disease detection at a preliminary phase reduces economic losses and improves the quality of crops. Manually identifying the agricultural pests is usually evident in plants; also, it takes more time and is an expensive technique. A drone system has been developed to gather photographs over enormous regions such as farm areas and plantations. An atmosphere generates vast amounts of data as it is monitored closely; the evaluation of this big data would increase the production of agricultural production. This paper aims to identify pests in mango trees such as hoppers, mealybugs, inflorescence midges, fruit flies, and stem borers. Because of the massive volumes of large-scale high-dimensional big data collected, it is necessary to reduce the dimensionality of the input for classifying images. The community-based cumulative algorithm was used to classify the pests in the existing system. The proposed method uses the Entropy-ELM method with Whale Optimization to improve the classification in detecting pests in agriculture. The Entropy-ELM method with the Whale Optimization Algorithm (WOA) is used for feature selection, enhancing mango pests’ classification accuracy. Support Vector Machines (SVMs) are especially effective for classifying while users get various classes in which they are interested. They are created as suitable classifiers to categorize any dataset in Big Data effectively. The proposed Entropy-ELM-WOA is more capable compared to the existing systems.

The effect of big data on financial reporting quality

The current manuscript aimed to explain the impact of big data on the financial reporting quality of the industrial sector in Jordan. To achieve the manuscript goals and validate hypotheses, a field study was conducted by distributing a questionnaire to 325 financial managers in industrial companies listed on the Amman Stock Exchange during a specific period. Gathered data were analyzed using structural equation modeling (SEM). The manuscript concluded that the big data dimensions, including variety, volume, and velocity, had a positive impact on financial reporting quality. Therefore, a set of recommendations were provided to managers of the industrial companies in Jordan to put in place an extensive data governance system to as-sure data quality, security, privacy, and compliance. To ensure the integrity and dependability of financial reporting, define data ownership, create data quality standards, and develop processes for data access, use, and preservation.

Performance Enhancement of Distributed System Using HDFS Federation and Sharding

In today's world, 2.5 exabytes of data are generated and processed by the IT industry and different organizations. Processing and managing such a massive volume of Big data is challenging. Hadoop(HDFS) is widely used framework for processing Bigdata. The limitation of HDFS is it can scale only the datanodes and not namenode where the metadata is managed. There is no scaling of name nodes in the present Hadoop architecture, so managing the metadata of the exabyte scale becomes an essential and challenging leading to tightly coupled block storage, namespace scalability, and performance bottleneck issues. The proposed approach dynamic federated metadata management (DFMM) architecture manages the metadata in the name node. DFMM configures multiple name nodes using HDFS federation technique, and metadata is partitioned into numerous name nodes using sharding technique. Sharding manages the metadata using locality-preserving hashing and consistent hashing methods. Locality-preserving hashing stores metadata in the most appropriate name node using the file's parent directory, and metadata is dynamically distributed among Name nodes to maintain load balancing. DFMM also maintains replicas for the high availability and reliability of data. Results showcase the superiority of the proposed DFMM architecture combined with the sharding technique is highly scalable and manages metadata efficiently, and can store up to 1 billion files with a namespace size of 400 GB, enhancing the throughput by 21% and is highly scalable as compared to existing techniques. The comparative study analysis of existing HDFS, and proposed DFMM architecture combined with sharding enhances the performance and manages the metadata more efficiently.

Big data analytics techniques and their impacts on reducing information asymmetry: Evidence from Jordan

This study aimed to demonstrate the impact of big data analytics techniques on reducing information asymmetry in industrial companies listed on the Amman Stock Exchange from the point of view of workers in Jordanian financial intermediation companies. Two approaches have been adopted to achieve the target of this research. The first approach is the analytical descriptive approach through a survey to collect primary data that measures the elements of the independent variable related to big data analytics techniques (Volume, Velocity, Variety, and Veracity). The second approach is an applied approach that measures the dependent variable of information asymmetry based on the financial statements of industrial companies listed on Amman Stock Exchange for the period (2015-2021). The statistical program (SPSS) has been used to analyze data and test the hypotheses through multiple regression testing. Based on the results of the statistical analysis of the data and the opinions of the research community, it was found that the huge volume of big data has become difficult to process using traditional data processing applications. Furthermore, there is a statistically significant relationship between big data analytics techniques and the reduction of information asymmetry from the point of view of employees in intermediation firms in Jordan. Consequently, it is necessary for those in charge of the industrial companies listed on the Amman Stock Exchange to develop modern techniques capable of analyzing big data with high efficiency. It can also assist in providing target groups including investors, stakeholders, and other beneficiaries with reliable and efficient data required to make rational decisions, as well as to reduce the risks of information asymmetry.

Filter and Embedded Feature Selection Methods to Meet Big Data Visualization Challenges

This study focuses on meeting the challenges of big data visualization by using of data reduction methods based the feature selection methods. To reduce the volume of big data and minimize model training time (Tt) while maintaining data quality. We contributed to meeting the challenges of big data visualization using the embedded method based “Select from model (SFM)” method by using “Random forest Importance algorithm (RFI)” and comparing it with the filter method by using “Select percentile (SP)” method based chi square “Chi2” tool for selecting the most important features, which are then fed into a classification process using the logistic regression (LR) algorithm and the k-nearest neighbor (KNN) algorithm. Thus, the classification accuracy (AC) performance of LR is also compared to the KNN approach in python on eight data sets to see which method produces the best rating when feature selection methods are applied. Consequently, the study concluded that the feature selection methods have a significant impact on the analysis and visualization of the data after removing the repetitive data and the data that do not affect the goal. After making several comparisons, the study suggests (SFMLR) using SFM based on RFI algorithm for feature selection, with LR algorithm for data classify. The proposal proved its efficacy by comparing its results with recent literature.

ПОРІВНЯННЯ АЛГОРИТМІВ КЛАСИФІКАЦІЇ BIG DATA МЕТОДАМИ ІМІТАЦІЙНОГО МОДЕЛЮВАННЯ

With the development of information transmission and storage technologies, the volumes of data that require processing and analysis are growing rapidly. Therefore, the task of developing algorithms for solving various artificial intelligence problems for Big Data volumes is urgent. In our works, this informal term "Big Data" refers to situations when known processing algorithms do not allow solving a problem in a practically acceptable time. With regard to classification tasks, such conditions are possible when the first place is not even high reliability (that is, the minimum number of errors), but productivity (classification speed). The well-known method of nearest neighbors is one of the most productive. However, the indicator of the order of growth (the number of typical operations) for it is K x M x N, where K is the number of nearest neighbors, M is the number of classes, N is the typical number of class elements. Along with this, we propose to consider algorithms based on the principles of M-means, where classes are replaced by only a small number of their characteristics. Among such algorithms, the article considers: the algorithm of class centers and the algorithm of adaptive rules. The order of growth for these algorithms is only M according to the number of classes. The comparative analysis of these algorithms is performed by the method of simulation modeling. Simulation models are implemented by the Adaptive Metrics program, developed at the Department of Software Engineering at DUITZ. In this program, the classification problem is solved using the example of the dichotomy problem for classes A and B. The program has the possibility of very flexible setting of models. Problems can be solved in 1-dimensional, 2-dimensional,..., 6-dimensional spaces. The distribution of factor values for classes A and B can have quite different statistical characteristics - from uniform and triangular distribution functions to functions approaching a normal distribution. The graphical interface of the program allows you to dynamically observe the solution of the classification problem in one-dimensional, two-dimensional and 6-dimensional projections. As a result of multiple runs of the program, it was established that the algorithms of the nearest neighbors slightly outperform the algorithms of class centers and adaptive rules according to the criterion of reliability, and also comply with the principle of compactness (concentration of the largest number of erroneous solutions in the hypercube of errors). Algorithms based on M-means principles significantly outperform this algorithm in terms of performance. Also, the algorithm of adaptive rules best corresponds to the principle of equality of classes and is the most productive of the considered ones.

An intelligent big data analytics method for two-dimensional non-residential building energy forecasting

The wide use of sensor and information technologies in buildings resulted in the massive generation of data related to its operation. Thus, there is a need for identifying patterns in data that they may use for the optimal operation of the buildings. The case of non-residential building characteristically provides a big volume of data whose analysis requires computational efficient methods. In this paper, we introduce a new big data analytic method that is applicable to forecasting energy demand in non-residential buildings. The goal is to make energy forecasts in a two-dimensional (2D) space defined by i) the electricity load and ii) gas demand. The proposed method combines the matrix profile (MP) method with a Long-Short Term Memory (LSTM) neural network. The combination of the above tools provides an efficient method in 2D hour ahead forecasting with the big data environment of smart buildings. In specific, with respect to mean average percentage error (MAPE) the combined MP-LSTM method provides a concurrent forecast of electricity and gas around 3% and 4%, respectively.

A federated collaborative recommendation model for privacy-preserving distributed recommender applications based on microservice framework

The wide adoption of IoT technologies has accelerated the accumulation of big data. Recommender systems (RS) is one of the most effective methods to extract user interested items from the huge volume of big data. However, implementing a recommender system over the distributed error-prone IoT devices faces two challenges. On the one hand, the distributed IoT devices may randomly fail to deliver its local data due to hardware malfunction, which may cause unavailability of the recommender system. Moreover, collecting the raw data from the distributed IoT devices may cause data privacy leakage issue, since the privacy data of user-item interaction records may be abused by vicious parties. In view of these challenges, we propose a federated collaborative recommendation model based on microservice framework in this paper to implement privacy preserving distributed recommendation applications. Firstly, we utilize the federated learning framework to train the collaborative recommendation model, where the raw data on each distributed device is kept locally and only item related model parameters are exposed to train the federated recommendation model. Moreover, we adopt the microservice framework to encapsulate different functions of the federated recommender model. Each distributed device can participate in the federated training process via service registration and service discovery function of the microservice framework. Furthermore, we enhance the typical Neural Collaborative Filtering model with the proposed FedNeuMF model by fusing auxiliary user profiles and item attributes to improve the recommendation accuracy. Finally, we conduct a set of experiments on three real-world datasets to check the performance of our proposal.

Revealing underlying factors of absenteeism: A machine learning approach.

The basis of support is understanding. In machine learning, understanding happens through assimilated knowledge and is centered on six pillars: big data, data volume, value, variety, velocity, and veracity. This study analyzes school attendance problems (SAP), which encompasses its legal statutes, school codes, students' attendance behaviors, and interventions in a school environment. The support pillars include attention to the physical classroom, school climate, and personal underlying factors impeding engagement, from which socio-emotional factors are often the primary drivers. This study asked the following research question: What can we learn about specific underlying factors of absenteeism using machine learning approaches? Data were retrieved from one school system available through the proprietary Building Dreams (BD) platform, owned by the Fight for Life Foundation (FFLF), whose mission is to support youth in underserved communities. The BD platform, licensed to K-12 schools, collects student-level data reported by educators on core values associated with in-class participation (a reported-negative or positive-behavior relative to the core values) based on Social-Emotional Learning (SEL) principles. We used a multi-phased approach leveraging several machine learning techniques (clustering, qualitative analysis, classification, and refinement of supervised and unsupervised learning). Unsupervised technique was employed to explore strong boundaries separating students using unlabeled data. From over 20,000 recorded behaviors, we were able to train a classifier with 90.2% accuracy and uncovered a major underlying factor directly affecting absenteeism: the importance of peer relationships. This is an important finding and provides data-driven support for the fundamental idea that peer relationships are a critical factor affecting absenteeism. The reported results provide a clear evidence that implementing socio-emotional learning components within a curriculum can improve absenteeism by targeting a root cause. Such knowledge can drive impactful policy and programming changes necessary for supporting the youth in communities overwhelmed with adversities.

Real Time Feature Influence Analysis Based ETL Framework for Efficient Big Data Management

The increased volume of big data introduces various challenges for its maintenance and analysis. There exist various approaches to the problem, but they fail to achieve the expected results. To improve the big data management performance, an efficient real time feature influence analysis based Extraction, Transform, and Loading (ETL) framework is presented in this article. The model fetches the big data and analyses the features to find noisy records by preprocessing the data set. Further, the method performs feature extraction and applies feature influence analysis to various data nodes and the data present in the data nodes. The method estimates Feature Specific Informative Influence (FSII) and Feature Specific Supportive Influence (FSSI). The value of FSII and FSSI are measured with the support of a data dictionary. The class ontology belongs to various classes of data. The value of FSII is measured according to the presence of a concrete feature on a tuple towards any data node, whereas the value of FSSI is measured based on the appearance of supportive features on any data point towards the data node. Using these measures, the method computes the Node Centric Transformation Score (NCTS). Based on the value of NCTS the method performs map reduction and merging of data nodes. The NCTS_FIA method achieves higher performance in the ETL process. By adapting feature influence analysis in big data management, the ETL performance is improved with the least amount of time complexity.

Refinery 4.0, a Review of the Main Challenges of the Industry 4.0 Paradigm in Oil & Gas Downstream

Industry 4.0 concept has become a worldwide revolution that has been mainly led by the manufacturing sector. Continuous Process Industry is part of this global trend where there are aspects of the "fourth industrial revolution" that must be adapted to the particular context and needs of big continuous processes such as oil refineries that have evolved to control paradigms supported by sector-specific technologies where big volumes of operation-driven data are continuously captured from a plethora of sensors. The introduction of Artificial Intelligence techniques can overcome the current limitations of Advanced Control Systems (mainly MPCs) by providing better performance on highly non-linear and complex systems and by operating with a broader scope in terms of signals/data and sub-systems. Moreover, the state of the art of traditional PID/MPC based solutions is showing an asymptotic improvement that requires a disruptive approach in order to reach relevant improvements in terms of efficiency, optimization, maintenance, etc. This paper shows the key aspects in oil refineries to successfully adopt Big Data and Machine Learning solutions that can significantly improve the efficiency and competitiveness of continuous processes.

Methodology for the Surveillance the Voltage Supply in Public Buildings Using the ITIC Curve and Python Programming

This paper proposes an easy-to-implement method for detecting and assessing two of the most frequent PQ (Power Quality) problems: voltage sags and swells. These can affect sensitive equipment such as computers, programmable logic controllers, contactors, etc. Therefore, it is of great interest to implement it in any laboratory, not only for protection reasons but also as a safeguard for claims against the supply company. Thanks to the actual context, in which it is possible to manage big volumes of data, connect multiple devices with IoT (Internet of Things), etc., it is feasible and of great interest to monitor the voltage at specific points of the network. This makes it possible to detect voltage sags and swells and diagnose which points are more prone to this type of problems. For the detection of sags and swells, a program written in Python is in charge of crawling all the files in the database and target those RMS values that fall outside the established limits. Compared to LabVIEW, which might have been the most logical alternative, being the acquisition hardware from the same company (National Instruments), Python has a higher computational performance and is also free of charge, unlike LabVIEW. Thanks to the libraries available in Python, it allows a hardware control close to what is possible using LabVIEW. Implemented in MATLAB, the ITIC (Information Technology Industry Council) power acceptability curve reflects the impact of these power quality disturbances in electrical power systems. The results showed that the combined action of Python and MATLAB performed well on a conventional desktop computer.

An Overview of Data Warehouse and Data Lake in Modern Enterprise Data Management

Data is the lifeblood of any organization. In today’s world, organizations recognize the vital role of data in modern business intelligence systems for making meaningful decisions and staying competitive in the field. Efficient and optimal data analytics provides a competitive edge to its performance and services. Major organizations generate, collect and process vast amounts of data, falling under the category of big data. Managing and analyzing the sheer volume and variety of big data is a cumbersome process. At the same time, proper utilization of the vast collection of an organization’s information can generate meaningful insights into business tactics. In this regard, two of the popular data management systems in the area of big data analytics (i.e., data warehouse and data lake) act as platforms to accumulate the big data generated and used by organizations. Although seemingly similar, both of them differ in terms of their characteristics and applications. This article presents a detailed overview of the roles of data warehouses and data lakes in modern enterprise data management. We detail the definitions, characteristics and related works for the respective data management frameworks. Furthermore, we explain the architecture and design considerations of the current state of the art. Finally, we provide a perspective on the challenges and promising research directions for the future.

Comprehensive Analysis of Various Big Data Classification Techniques: A Challenging Overview

Data over the internet has been increasing everyday, and automatic mining of essential information from an enormous amount of data has become a challenging task today for an organisation with a huge dataset. In recent years, the prominent technology in the domain of Information Technology (IT) is big data, which is unstructured data that solves the computational complexity of classical database systems. The data is fast and big and typically derived from multiple and independent sources. The three main challenges are data accessing, semantics, and domain knowledge for various big data utilisations and complexities raised by big data volumes. One of the major limitations is the classification of big data. This paper introduces well-defined classification methodologies employed for big data classification. This paper reviews 50 research papers based on classification methods of big data, and such methodologies are primarily categorised into six different categories, namely K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Fuzzy-based method, Bayesian-based method, Random Forest, and Decision Tree. In addition, detailed analysis and discussion are carried out by considering classification techniques, dataset utilised, evaluation metrics, semantic similarity measures, and publication year. In addition, research gaps and issues for several traditional big data classification techniques are explained to expand investigators’ works to provide effective big data management.

Astoundingly Smart System Furnishing Ranking of Big Data in Search Engines

The abruptly escalating internet is sensational. It inculcates a humungous volume of big data, which is obsolete and tedious to manage, scrutinize, analyze and perform operations upon them in conventional ways. Big data has thus expedited the search and retrieval of information, necessitating the development of contemporary search algorithms to aid this process. However, the primary hindrance for the first and second generations of conventional search engines was the syntax of keywords devoid of semantic meaning and the lack of a knowledge base that linked disparate web material. This article presents a framework based on trendy technologies, specifically Extracting, Transforming and Integrating (ETI) processes, ontology graphs, and indexing Resource Description Framework (RDF) using the wide-column Not only Structured Query Language (NoSQL) method. The most significant contribution in this regard is developing a mathematical model to compute the similarity score between a query and stored RDF documents using semantic relations. Numerous operations were carried out to evaluate the effectiveness of the proposed methodology in installing data sources, such as DBpedia and YAGO dataset. Insofar as experimental results are concerned, the suggested model achieves greater precision than other comparable systems.

System Identification at the Extreme Edge for Network Load Reduction in Vibration-Based Monitoring

Mechanical complexity, wide dimensions, and big data volume may hamper the implementation of Internet of Things (IoT)-enabled structural health monitoring (SHM) systems. In particular, one of the most important challenges is the reduction of the data payload to be transmitted over the monitoring network. Addressing the problem in the context of vibration-based SHM, this work explores system identification (SysId) as an innovative strategy for data compression at the extreme edge. Indeed, SysId is a signal processing technique aiming at finding a very reduced (i.e., less then one tenth of the total signal length) set of meaningful parameters, which can provide an alternative, but yet completely equivalent, frequency characterization of the structure. In the proposed approach, an embedded system-oriented adaptation of the sequential tall-skinny QR decomposition (eS-TSQR) from the dense linear algebra domain has been exploited to tackle both the memory and computational complexity of the involved algorithms. This yielded to the embodiment of input–output and output-only SysId models into a resource constrained device (i.e., an STM32L5 microcontoller unit), targeted on low-power and low-cost SHM applications, proving high effectiveness for the structural assessment of civil and industrial plants. Besides, a cost-benefit analysis is also presented, in which the energy saving brought by SysId running in a sensor-near manner is comprehensively measured against the power consumption due to data transmission, as implied by state-of-the-art communication protocols for IoT. Results demonstrate that SysId is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.19\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.78\times $ </tex-math></inline-formula> less energy demanding (with a payload reduction of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$9\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$45\times $ </tex-math></inline-formula> ) w.r.t. compressed sensing-driven and compression-free solutions, respectively.

Deep Learning Algorithms and Parallel Distributed Computing Techniques for High-Resolution Load Forecasting Applying Hyperparameter Optimization

Electrical load forecasting is one of the critical tasks that helps power utility companies in planning and operation as well as the energy managementsystem (EMS) in controlling and optimizing the power grid’s performance. It guarantees system adequacy and reliability by reducing the gap between supply and demand, and it bolsters cybersecurity by enabling false data detection and system security by predicting system peak load. This article aims to build an accurate long-term load forecasting model based on deep learning (DL) algorithms. Initially, big-data analytics and feature engineering are applied to the power load and weather historical data. Next, three different load forecasting models are developed based on three corresponding deep neural network models (DNN): the simple RNN, the long short term memory (LSTM), and the gated recurrent unit (GRU), respectively. To find an accurate DNN model, a two-step hyperparameter optimization is performed consisting of cross-validation. In the first step, the random search (RS) algorithm is employed to narrow the parameter search space to a smaller local one. In the second step, the grid search (GS) algorithm is used to find the best hyperparameters in the local area located by RS. However, the computational burden of the DNN training process is too high and may lead to out-of-memory (OOM) issues, especially when we are dealing with a big volume of data. To overcome this problem, we developed our Python program based on Tensorflow framework and implemented the framework on a high-performance computer cluster (HPCC). Then, different jobs are defined to employ parallel distributed computing (PDC) capability on the SIU BigDawg HPCC. To this end, all DNN hyperparameters (e.g., number of layers, neurons, dropout, batch size, epoch, learning rate, etc.) can be optimized and result in a comprehensive and accurate load forecasting model. Our proposed method, along with the programming framework, is evaluated on a real-world dataset. Among the three models developed and tested, the GRU, with load and weather as features, has demonstrated excellent performance and outperformed the other two models for long-term forecasting with the best metrics (i.e., MSE, MAPE, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^2$</tex-math></inline-formula> ) and daily error almost less than 1%. Also, our best model (optimized GRU) is compared with the other reported state-of-the-art models such as DBN, SVR, RM-LSTM, DM-GCNN, TCMS-CNN. It is demonstrated that the proposed framework yields a superior performance than the others.