New York Independent System Operator Research Articles

ATTnet: An explainable gated recurrent unit neural network for high frequency electricity price forecasting

The primary contribution of this study is the proposal of an explainable deep-learning neural network (ATTnet) that employs an attention mechanism to achieve accurate electricity spot price forecasting and an explainable model pipeline. The concise, single-stream network consists of a 5-head attention mechanism and gated recurrent units, which have been developed to model the temporal dependencies of the volatile market data. In addition to introducing a novel neural network architecture for volatile time series data, this study makes a substantial contribution by investigating prediction factors in two ways: temporally via the attention scores from the input sequences and globally via feature Shapely values. In real-time electricity price prediction, historical prices, temperature, hour, and zonal load are found to be the most important variables. The deep learning model was tested on real-time price profiles from eight generators within the New York Independent System Operator (NYISO) network. The proposed model achieves performance gains of 21% in MAE and 22% in MAPE over the state-of-the-art benchmark methods.

Foreseeing the worst: Forecasting electricity DART spikes

Statistical learning models are proposed for the prediction of the probability of a spike in the electricity DART (day-ahead minus real-time price) spread. Assessing the likelihood of DART spikes is of paramount importance for virtual bidders, among others. The model’s performance is evaluated on historical data for the Long Island zone of the New York Independent System Operator (NYISO). A tailored feature set encompassing novel engineered features is designed. Such a set of features makes it possible to achieve excellent predictive performance and discriminatory power. Results are shown to be robust to the choice of the predictive algorithm. Lastly, the benefits of forecasting the spikes are illustrated through a trading exercise, confirming that trading strategies employing the model predicted probabilities as a signal generate consistent profits.

Cyber-Attack Detection Using Principal Component Analysis and Noisy Clustering Algorithms: A Collaborative Machine Learning-Based Framework

This paper proposes a collaborative machine learning-based framework to detect cyber-attacks in a power system, leading to deviation in the state variable behavior. Based on the proposed architecture, three different machine learning-based methods, i.e., visualization, classification, and clustering, are employed and compared to find the best one in the FDIA detection process. To this end, pre-processing is employed in the first stage. In the second stage, the patterns of the state vectors are transferred into features. Hence, 24 statistical features, including measures of central tendency, variability, measures of shape, and position, are extracted to find various properties. Then, in the third stage, a supervised algorithm is employed to rank and find the most crucial features in FDIA. In the fourth stage, an unsupervised dimensionality reduction technique (PCA) is applied to reduce the feature space. In the fifth and last stage, visualization, classification, and clustering-based methods are developed to detect FDIA. To simulate an attack, it is assumed that an intruder decreases or increases the state vectors at different buses with various attack parameters (i.e., 0.90, 0.95, 0.96, 0.97, 0.98, 1, 1.02, 1.03, 1.04, 1.05, and 1.10). The proposed method effectiveness is assessed on the New York Independent System Operator (NYISO) data applied to the IEEE 14-bus system. The results presented in the paper from different scenarios (i.e., phase angle ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\theta $ </tex-math></inline-formula> ), voltage magnitude ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$V_{m}$ </tex-math></inline-formula> ), measurements, and multiple attacks) on a real-world dataset demonstrate that the collaborative optimized PCA-Density-based machine learning technique can detect most of the attack samples with good performance scores (i.e., recall, precision, F1) and outperforms the other investigated methods. Moreover, it is general and adaptable enough to cover the situation where either the system characteristics or the attack behavior changes.

Adaptive Load Forecasting Methodology Based on Generalized Additive Model with Automatic Variable Selection.

For decentralized energy management in a smart grid, there is a need for electric load forecasting at different places in the grid hierarchy and for different levels of aggregation. Load forecasting functionality relies on the load time series prediction model, which provides accurate forecasts. Complex and heterogeneous multi-source load time series in a smart grid require flexible modeling approaches to meet the accuracy demand. This work proposes an adaptive load forecasting methodology based on the generalized additive model (GAM) with the big data estimation method. It is based on a set of GAM terms, constructed for a specific multi-source load forecasting application in the grid and a procedure that dynamically selects the most relevant terms and generates forecasts for particular load time series. Data from publicly available New York Independent System Operator (NYISO) databases are used for testing. The 24-hour-ahead forecasting results for eleven New York City zones, of different sizes and types, indicate the applicability of the proposed methodology.

A market mechanism for truthful bidding with energy storage

This paper proposes a market mechanism for multi-interval electricity markets with generator and storage participants. Drawing ideas from supply function bidding, we introduce a novel bid structure for storage participation that allows storage units to communicate their cost to the market using energy-cycling functions that map prices to cycle depths. The resulting market-clearing process – implemented via convex programming – yields corresponding schedules and payments based on traditional energy prices for power supply and per-cycle prices for storage utilization. We illustrate the benefits of our solution by comparing the competitive equilibrium – assuming price-taker participants – of the resulting mechanism to that of an alternative solution that uses prosumer-based bids. Our solution shows several advantages over the standard prosumer-based approach that prices energy per slot. It does not require a priori price estimation. It also incentivizes participants to reveal their truthful cost, thus leading to an efficient, competitive equilibrium. Numerical experiments using New York Independent System Operator (NYISO) data validate our findings.

QCAE: A quadruple branch CNN autoencoder for real-time electricity price forecasting

Real-time electricity market data is highly volatile and very noisy. The properties of such data make forecasting models difficult to develop, with traditional statistical models in particular affected by the “curse of dimensionality” for such data. However, autoencoders, or neural networks specifically designed to reduce the noise and dimensions of input data, may prove useful to advance the accuracy of real-time price forecasting models. This paper studies the optimal design of such an autoencoder, developing a quadruple branch, CNN-based autoencoder (QCAE) which is pre-trained and then directly linked to a forecasting model. The QCAE compresses the input data in both time and feature directions. Ablation analyses verify the architecture of the QCAE, and its integration with the forecasting model is tested and validated on fifty generators in the New York Independent System Operator (NYISO) power grid. The QCAE forecasting framework outperforms benchmark and state-of-the-art models with an average improvement of 6.3% in sMAPE and 3.10% in MAE.

A Stochastic Optimization Framework for Realizing Combined Value Streams From Customer-Side Resources

Due to numerous supporting policies aimed at decarbonizing electricity infrastructures in different regions of the world, customer-side resources are becoming increasingly valuable. Consequently, load serving entities (LSEs) which typically have access to these customer-side resources can use them for multiple services simultaneously. In this paper, we discuss a stochastic optimization framework for using clusters of residential HVACs, electric water heaters (EWH) and behind-the-meter (BTM) batteries, spread around the LSE’s distribution network, for energy arbitrage, peak shaving and market based frequency regulation simultaneously. Our framework captures the effects of controlling the consumption of the customer-side resources on the voltages in the LSE’s distribution network. We also discuss two real-time dispatch algorithms capable of eliciting fast response from the resources to derive regulation signals from the market operator with minimal voltage violations. We evaluate the optimization models and dispatch algorithms using a HELICS-based co-simulation platform and real-world data from New York Independent System Operator (NYISO).

GHTnet: Tri-Branch deep learning network for real-time electricity price forecasting

A highly accurate electricity price prediction model is of the utmost importance for multiple power systems tasks, such as generation dispatch and bidding. Due to the liberalization of the electricity market, as well as high renewable penetration, the properties of electricity price time series are becoming more stochastic and complex. Traditional statistical methods and machine learning algorithms cannot model such volatile market conditions with high fidelity. In this paper, we propose a data-driven deep learning network (GHTnet) to capture the temporal distribution of real-time price data. A new CNN module, based on GoogLeNet, is developed to capture the high-frequency features of this data, while inclusion of time series summary statistics is shown to improve the forecasting of volatile price spikes. The deep learning model is developed and validated on real-time price time series from 49 generators in the New York Independent System Operator (NYISO), achieving significant performance improvements over that of state-of-the-art benchmark methods, with an average 17.34% improvement in MAPE.

Short-Term Load Forecasting Using an Attended Sequential Encoder-Stacked Decoder Model with Online Training

The paper presents a new approach for the prediction of load active power 24 h ahead using an attended sequential encoder and stacked decoder model with Long Short-Term Memory cells. The load data are owned by the New York Independent System Operator (NYISO) and is dated from the years 2014–2017. Due to dynamics in the load patterns, multiple short pieces of training on pre-filtered data are executed in combination with the transfer learning concept. The evaluation is done by direct comparison with the results of the NYISO forecast and additionally under consideration of several benchmark methods. The results in terms of the Mean Absolute Percentage Error range from 1.5% for the highly loaded New York City zone to 3% for the Mohawk Valley zone with rather small load consumption. The execution time of a day ahead forecast including the training on a personal computer without GPU accounts to 10 s on average.

Cyber Attacks and Faults Discrimination in Intelligent Electronic Device-Based Energy Management Systems

Intelligent electronic devices (IEDs) along with advanced information and communication technology (ICT)-based networks are emerging in the legacy power grid to obtain real-time system states and provide the energy management system (EMS) with wide-area monitoring and advanced control capabilities. Cyber attackers can inject malicious data into the EMS to mislead the state estimation process and disrupt operations or initiate blackouts. A machine learning algorithm (MLA)-based approach is presented in this paper to detect false data injection attacks (FDIAs) in an IED-based EMS. In addition, stealthy construction of FDIAs and their impact on the detection rate of MLAs are analyzed. Furthermore, the impacts of natural disturbances such as faults on the system are considered, and the research work is extended to distinguish between cyber attacks and faults by using state-of-the-art MLAs. In this paper, state-of-the-art MLAs such as Random Forest, OneR, Naive Bayes, SVM, and AdaBoost are used as detection classifiers, and performance parameters such as detection rate, false positive rate, precision, recall, and f-measure are analyzed for different case scenarios on the IEEE benchmark 14-bus system. The experimental results are validated using real-time load flow data from the New York Independent System Operator (NYISO).

Economic and environmental impacts of a proposed ‘Carbon adder' on New York’s energy market

ABSTRACT The New York Independent System Operator (NYISO) has developed a carbon-pricing proposal to reduce carbon intensive electricity generation in anticipation of future clean energy goals in the state. The proposed measure is a so-called ‘carbon adder’ on CO2 emissions from the power sector that targets the social cost of carbon amidst existing overlapping policies. The carbon adder is set as the difference between the targeted social cost of carbon and the prevailing RGGI price for CO2 emission allowances. We investigate the economic and environmental impacts from the imposition of a carbon adder on New York’s power sector. Our analysis indicates that the carbon adder gives the ‘right’ price signal for New York’s power generation to turn into a greener one and is shown to be more cost-effective than clean energy standards. Requirements for permit price floors in the RGGI market induces carbon permit retirements across RGGI states leading to small reductions in region- and country-wide emissions levels.The proposed border carbon adjustments on electricity trade are shown to further mitigate emission leakage. Key policy insights NYISO’s proposed carbon adder provides strong incentives for decarbonization and renewable power generation in the New York region. The carbon adder is shown to be more cost-effective than New York's clean energy standard measures in achieving an equivalent level of emissions reductions. This is because the carbon adder targets energy carriers by their specific CO2 emissions while the clean energy standard implicitly targets energy carriers by their specific CO2 emissions while the clean energy standard implicitly provides a uniform subsidy to all clean energy carriers. The existing RGGI price floor mechanism assures that NYISO’s unilateral use of carbon adder is environmentally effective by generating overall emission reductions in the RGGI region in order to maintain the existing price floor. Border carbon adjustments on the embodied carbon of electricity imports to New York, and an emission containment reserve, which dynamically adjusts RGGI cap as a response to RGGI price change, will likely mitigate leakage further and improve competitiveness of the New York’s energy sector.

How does new energy storage affect the operation and revenue of existing generation?

As energy storage is integrated into grids through policies or market forces, it has an effect on the dispatch, economics, and retirement of other generators. While the complementary relationship between storage and renewables is well-known, the effect of storage additions is not necessarily limited to renewables. This work models the system effects of new storage on the generation, operating income, and retirement of power plants at three levels of increasing complexity. First, we evaluate the marginal effects of storage on generation sources without any effects on market prices or dispatch. Second, we use a dispatch model to study bulk storage in New York Independent System Operator (NYISO), Midcontinent ISO (MISO), and California ISO (CAISO), allowing storage to shift dispatch patterns and affect the operation/income of existing generators. Third, we examine the mid- and long-term effects on the generation fleet by accounting for the retirement of power plants that lose sufficient annual revenue from new storage. Results suggest that marginal new storage increases coal generation and decreases natural gas generation in the West and Midwest, and does the opposite in New England and California. With bulk storage additions, the operating income of all other generating units is reduced unless retirement is included. With retirements considered, the least flexible generation—coal, nuclear, and solar—gain the most operating income with storage. In all cases, simple cycle gas turbines lose the most operating income as they are offset by storage during the discharge phase and then retired in the long term.

On the Feasibility of Load-Changing Attacks in Power Systems During the COVID-19 Pandemic.

The electric power grid is a complex cyberphysical energy system (CPES) in which information and communication technologies (ICT) are integrated into the operations and services of the power grid infrastructure. The growing number of Internet-of-things (IoT) high-wattage appliances, such as air conditioners and electric vehicles, being connected to the power grid, together with the high dependence of ICT and control interfaces, make CPES vulnerable to high-impact, low-probability load-changing cyberattacks. Moreover, the side-effects of the COVID-19 pandemic demonstrate a modification of electricity consumption patterns with utilities experiencing significant net-load and peak reductions. These unusual sustained low load demand conditions could be leveraged by adversaries to cause frequency instabilities in CPES by compromising hundreds of thousands of IoT-connected high-wattage loads. This article presents a feasibility study of the impacts of load-changing attacks on CPES during the low loading conditions caused by the lockdown measures implemented during the COVID-19 pandemic. The load demand reductions caused by the lockdown measures are analyzed using dynamic mode decomposition (DMD), focusing on the March-to-July 2020 period and the New York region as the most impacted time period and location in terms of load reduction due to the lockdowns being in full execution. Our feasibility study evaluates load-changing attack scenarios using real load consumption data from the New York Independent System Operator (NYISO) and shows that an attacker with sufficient knowledge and resources could be capable of producing frequency stability problems, with frequency excursions going up to 60.5 Hz and 63.4 Hz, when no mitigation measures are taken.

From Firm Solar Power Forecasts to Firm Solar Power Generation an Effective Path to Ultra-High Renewable Penetration a New York Case Study

We introduce firm solar forecasts as a strategy to operate optimally overbuilt solar power plants in conjunction with optimally sized storage systems so as to make up for any power prediction errors, and hence entirely remove load balancing uncertainty emanating from grid-connected solar fleets. A central part of this strategy is the plant overbuilding that we term implicit storage. We show that strategy, while economically justifiable on its own account, is an effective entry step to achieving least-cost ultra-high solar penetration where firm power generation will be a prerequisite. We demonstrate that in the absence of an implicit storage strategy, ultra-high solar penetration would be vastly more expensive. Using the New York Independent System Operator (NYISO) as a case study, we determine current and future costs of firm forecasts for a comprehensive set of scenarios in each ISO electrical region, comparing centralized vs. decentralized production and assessing load flexibility’s impact. We simulate the growth of the strategy from firm forecast to firm power generation. We conclude that ultra-high solar penetration enabled by the present strategy, whereby solar would firmly supply the entire NYISO load, could be achieved locally at electricity production costs comparable to current NYISO wholesale market prices.

High Frequency Evidence on the Effects of Wind Power Generation on the Electricity Price

Exploiting the hourly data from the New York Independent System Operator (NYISO) database, we model the impact of wind generation on electricity spot prices and on their dynamic volatility behavior while taking into account the variations in zonal wind output. We find that the growing share of wind generation not only leads to a decline in electricity prices, but also results in an increased price volatility. The results indicate that a 1% rise in the aggregate wind power generation leads to a 0.012% decline in the average NYISO prices while the mentioned price effect in the wind producing zones is estimated to be a 0.055% to 0.085% downward adjustment of the zonal prices for a 1% growth in wind generation within the zone. Our findings have two implications: First, the increasing downward pressure on electricity prices due to the growing share of wind generation can challenge the financial sustainability of the electricity markets in the long-run, which highlights the need to proceed towards an effective market structure to ensure the recovery of fixed generation costs. Second, the induced price volatility implies that effective use of price risk hedging strategies should be considered by all actors in the power markets.

An integrated method based on relevance vector machine for short-term load forecasting

Short-term electricity load forecasting has become increasingly important due to the privatization and deregulation in the energy market. This study proposes a probabilistic learning method to predict hour-ahead and day-ahead load demand. Unlike methods in previous studies, the proposed method integrates wavelet transform and feature selection as key preprocessing steps. Features are divided into current state related features and historical information related features. Current state related features are forecasted by the regression model before being added into the load prediction model. The entire learning and prediction process is based on the relevance vector machine (RVM) that utilizes load data characteristics. A number of test cases are presented using benchmark datasets from the New York Independent System Operator (NYISO) and ISO New England. Based on the detailed empirical comparison, the proposed RVM-based integrated method outperforms classical time series approaches and state-of-the-art artificial intelligence methods on short-term load forecasting.

Social sensing: towards social media as a sensor for resilience in power systems and other critical infrastructures

ABSTRACT Improving Critical Infrastructure (CI) resilience is a key challenge facing modern society. Increased integration of sensors into infrastructure systems, combined with modern computational capabilities, provides an opportunity to develop novel data-driven methods for improving resilience.  Social media serves as a promising data source for such methods, as it has become widely used for information dissemination. This paper aims to demonstrate the value of social media for CI resilience by using this novel data source to model CI behaviors at higher spatiotemporal resolutions than previously shown. We apply this approach, which focuses on statistical analysis and forecasting methods, to a case study of Hurricane Sandy using publicly available Twitter data and power system data for the New York Independent System Operator (NYISO). We find evidence of statistically significant correlations between Twitter and power system data, and develop models for forecasting future behaviors in the NYISO power system using these data.

Measurement-driven optimal control of utility-scale power systems: A New York State grid perspective

This paper focuses on designing and testing a supplementary controller for an ultra-large, utility-scale power system, namely the New York State (NYS) Power Grid from a completely measurement-based perspective. We present the control design using the Flexible AC Transmission System (FACTS) facility at the NYS grid. We use the utility-scale Eastern Interconnection (EI) model consisting of over 70,000 buses in the PSS/E platform for this research. The coherency structure of the NYS grid is analyzed by performing Principal Component Analysis (PCA) on frequency measurements obtained from multiple contingency simulations. Thereafter, we use the frequency measurements from PMU-enabled buses to identify a reduced-order state space model of the grid such that it matches the input-output characteristics along with identifying the inter-area modes. This model is then used to design the Linear Quadratic Gaussian (LQG) based optimal FACTS controller. Then the controller is implemented in the PSS/E model of the Eastern Interconnection (EI) as a PSS/E-FORTRAN based user defined module. The effectiveness of the control performance is shown using the non-linear simulations under different contingencies provided by the New York Independent System Operator (NYISO).

Optimization of Aggregate Capacity of PEVs for Frequency Regulation Service in Day-Ahead Market

An aggregator can coordinate plug-in electric vehicles (PEVs) to provide frequency regulation service to an independent system operator (ISO). The aggregator can participate in the electricity markets of ISOs which provide economic incentives for PEV frequency regulation service. While the ISOs typically use forward market [e.g., day-ahead market (DAM)] to trade frequency regulation service, the available regulation capacity of an aggregator is subject to the random arrival and departure of the PEVs. In the DAM, the aggregator submits a bid to indicate its available capacity on the next day. This motivates us to study the problem of how an aggregator determines its bid in the DAM, given the uncertainty of the available regulation capacity of the PEVs. The DAM is used to trade the frequency regulation capacity in California ISO (CAISO) and New York ISO (NYISO). We consider two types of DAMs based on the market rules of CAISO and NYISO. For the first type, the exact amount of regulation capacity submitted in the DAM needs to be fulfilled on the next day. For the second type, a market participant can settle a shortage of capacity by paying a penalty to the ISO. In both cases, the aggregator can participate in the real-time market to sell extra capacity on the next day. We formulate the problem for determining the bid using stochastic programming. As PEVs have uncertain arrival and departure times, our problem formulation incorporates risk management using the conditional value at risk. Efficient algorithms are proposed for solving the formulated problem. PEV charging data collected in Vancouver, BC, Canada, is used in our simulations. We compare the profit of the aggregator when it participates in the markets of CAISO and NYISO. Our simulation results show that the uncertainty of the PEVs’ available capacity has less effect on the profit and financial risk as the number of PEVs increases.

Applications of Hybrid EMD with PSO and GA for an SVR-Based Load Forecasting Model

Providing accurate load forecasting plays an important role for effective management operations of a power utility. When considering the superiority of support vector regression (SVR) in terms of non-linear optimization, this paper proposes a novel SVR-based load forecasting model, namely EMD-PSO-GA-SVR, by hybridizing the empirical mode decomposition (EMD) with two evolutionary algorithms, i.e., particle swarm optimization (PSO) and the genetic algorithm (GA). The EMD approach is applied to decompose the load data pattern into sequent elements, with higher and lower frequencies. The PSO, with global optimizing ability, is employed to determine the three parameters of a SVR model with higher frequencies. On the contrary, for lower frequencies, the GA, which is based on evolutionary rules of selection and crossover, is used to select suitable values of the three parameters. Finally, the load data collected from the New York Independent System Operator (NYISO) in the United States of America (USA) and the New South Wales (NSW) in the Australian electricity market are used to construct the proposed model and to compare the performances among different competitive forecasting models. The experimental results demonstrate the superiority of the proposed model that it can provide more accurate forecasting results and the interpretability than others.