Australian National Electricity Market Research Articles

Crises in Texas and Australia: Failures of energy-only markets or unforeseen consequences of price caps?

The Australian National Electricity Market (NEM) and the Texas wholesale electricity market (ERCOT) are two of the most prominent examples of energy-only wholesale electricity markets in the world. In early 2021 and mid 2022 these markets suffered from unprecedented crises, prompting some commentators to question the energy-only market design. Both markets are now considering implementing a form of capacity mechanism. Are these crises evidence of a fundamental flaw in the energy-only market design, or something else? We argue that, although the crises were very different in form, both crises arose in part from the effect of price caps in the wholesale market. We set out a model of the optimal mix of generation technologies in a framework in which generators can invest to protect against extreme weather events. We show that a price cap (below VoLL) reduces incentives for investment in hardening generation. Rather than further lowering the price caps – as was done in Texas – we propose a range of reforms to strengthen and enhance the confidence in the wholesale market.

Constructing Interval Forecasts for Solar and Wind Energy Using Quantile Regression, ARCH and Exponential Smoothing Methods

The research reported in this article focuses on a comparison of two different approaches to setting error bounds, or prediction intervals, on short-term forecasting of solar irradiation as well as solar and wind farm output. Short term in this instance relates to the time scales applicable in the Australian National Electricity Market (NEM), which operates on a five-minute basis throughout the year. The Australian Energy Market Operator (AEMO) has decided in recent years that, as well as point forecasts of energy, it is advantageous for planning purposes to have error bounds on those forecasts. We use quantile regression as one of the techniques to construct the bounds. This procedure is compared to a method of forecasting the conditional variance by use of either ARCH/GARCH or exponential smoothing, whichever is more appropriate for the specific application. The noise terms for these techniques must undergo a normalising transformation before their application. It seems that, for certain applications, quantile regression performs better, and the other technique for some other applications.

AC–DC security-constrained optimal power flow for the Australian National Electricity Market

This paper presents a new methodology to solve the integrated ac–dc security-constrained optimal power flow (SCOPF) problem for large-scale power systems, with a focus on the Australian National Electricity Market (NEM). The proposed SCOPF is based on a decomposition scheme that significantly reduces the dimensionality of the problem and thus improves the computational performance. It further supports a contingency ranking, evaluation, and non-dominated filtering algorithm, cyclic selection of filtered contingencies, careful choice of slack variables, and a tailored warm start to the nonlinear problem. The methodology is validated against the two-stage mathematical programming model that uses an explicit nonlinear model of both the base case and the contingency cases, at a slightly higher cost but significantly faster execution. Moreover, the NEM power system, is studied addressing different scenarios including load variations, renewable energy zone integration, and additional reinforcements like point-to-point and meshed HVDC. The proposed approaches provides preventive solutions in less than 5 min thus affirming the model’s effectiveness in real-time applications and planning studies.

Derivatives and hedging practices in the Australian National Electricity Market

This article expands on the seminal policy work by Anderson et al. (2007) and explores the impacts of a changing energy mix on derivatives and hedging practices 15 years on from their work in the Australian National Electricity Market (NEM). The NEM has undergone vast change since 2007 as variable renewable energy (VRE) has entered the market. This can have a significant impact on the contracting patterns and derivatives’ use of participants. A survey was undertaken to determine how market participants in the NEM conduct derivatives trading and hedging of their portfolios. Additionally, participants shared their views on the current state of energy policy and operation of the market. The results showed participants continue to use derivatives, as was previously documented, with increasing focus on power purchase agreements (PPA) for VRE contracting and a sharp drop in the use of traditional peak contracts. Nonetheless, there arose apprehension regarding participants' future capacity to hedge portfolios, given the diminishing liquidity within the derivative market due to the departure of dispatchable plants. This poses a difficult path for policy makers and regulators as a functioning and active derivative market is integral for the operation of an energy only market such as the NEM.

Forecasting individual bids in real electricity markets through machine learning framework

With the increasing uncertainty caused by the complexity of the world’s energy environment and the increasing penetration rate of renewable energy, it is significant to estimate the future operation of power markets in advance. Forecasting individual bids in spot electricity markets is a promising new method for achieving so, but it has not been fully studied due to the difficulty of forecasting a bid function. The idealization of existing optimization-based models decreases their practical effects in real markets. Thus, we propose a scalable forecasting framework that incorporates several customized state-of-art machine learning methods according to the characteristics of the bidding data. First, several low-rank approximation algorithms are customized to encode the high-dimensional bidding curves into low-dimensional feature spaces and reconstruct them from the predicted feature space. Second, a transformer-based multidimensional time series prediction algorithm is proposed to predict the bidding feature based on both related factors and historical bidding records. To appropriately evaluate the performances of the forecasting methods, we introduce a dynamic criterion based on the economic implications of bids. The comprehensive framework is tested based on actual market data from the Australian national electricity market, and in the empirical example, the feasibility and effectiveness of the proposed framework are demonstrated.

A probabilistic forecast methodology for volatile electricity prices in the Australian National Electricity Market

The South Australia region of the Australian National Electricity Market (NEM) displays some of the highest levels of price volatility observed in modern electricity markets. This paper outlines an approach to probabilistic forecasting under these extreme conditions, including spike filtration and several post-processing steps. We propose using quantile regression as an ensemble tool for probabilistic forecasting, with our combined forecasts achieving superior results compared to all constituent models. Within our ensemble framework, we demonstrate that averaging models with varying training-length periods leads to a more adaptive model and increased prediction accuracy. The applicability of the final model is evaluated by comparing our median forecasts with the point forecasts available from the Australian NEM operator, with our model outperforming these NEM forecasts by a significant margin.

An Affordable Sophisticated Frequency Control Ancillary Services for Australian National Electricity Market System Considering Industry and Infrastructure Challenges

An Affordable Sophisticated Frequency Control Ancillary Services for Australian National Electricity Market System Considering Industry and Infrastructure Challenges

Navigating the crisis: Fuel price caps in the Australian national wholesale electricity market

Fuel price caps are one of the potential regulatory tools for controlling wholesale electricity prices when fuel prices are volatile. In this paper, we introduce a theoretical model to study the effects of such caps on firms’ bidding behaviour and clearing prices in spot market auctions. We then use data from the Australian National Electricity Market (NEM), which recently implemented such caps, to empirically test and compare their effectiveness in three different states. Our theoretical findings suggest that fuel price caps can be binding, especially when electricity demand is lower and competition among generators is higher. When demand is high, alternative policy tools, such as market price caps, may be more effective in controlling auction prices. Our empirical analysis employs various techniques, such as Generalized Additive Models (GAM) and machine learning algorithms, to test the effectiveness of price caps in the NEM. We find mixed results regarding the effectiveness of fuel price caps in different states. Specifically, fuel price caps reduced wholesale electricity prices in Queensland and New South Wales, while they were not effective in controlling wholesale prices in Victoria.

Emission intensities in the Australian National Electricity Market – An econometric analysis

This study examines the evolution of CO2 emission intensities in the wholesale electricity sector for the National Electricity Market (NEM) in Australia. Using daily data, we examine the impact of demand, changes in the generation mix, as well as the closure of major coal-fired power plants on emission intensities in the four largest regional markets of the NEM. Particular emphasis is also placed on the assessment of any potential association between the examined driving factors and major Australian climate and energy policies such as the 2012–2014 Carbon Pricing Mechanism (CPM) or the Renewable Energy Target (RET) scheme. Our results suggest that changes in renewable deployment can be considered as the principal factor affecting emission intensities of all regional markets. Moreover, the fossil fuel mixture effect, more precisely the reduction in coal-fired generation combined with the increase in generation from gas and renewables, also played a significant role in reducing emission intensities in the Australian power sector. Interestingly, our findings suggest that the influence of the implemented CPM was not too substantial and had only temporary effects on emission intensities.

Firming 100% renewable power: Costs and opportunities in Australia's National Electricity Market

The decreasing cost of solar photovoltaic (PV) and wind power technologies makes 100% renewable energy systems economically viable. Building more capacity to mitigate periods of low solar and wind supply results in systems that produce surplus energy at other times. As these surpluses vary across time and space, it becomes crucial to map their geographic distribution and quantify their magnitude. This paper presents the largest geographic and longest time series analysis of oversizing solar PV and wind in the context of the Australian National Electricity Market. We examine the additional solar, wind and battery capacity required to meet 100% of demand between 2010 and 2020. Without battery storage, this is achieved by generating approximately four times demand at an average production cost 28% lower than recent wholesale electricity prices. The addition of 1–8 h of storage reduces the average production cost by 55% compared to recent prices. Our analysis also investigates the spatiotemporal characteristics of surplus energy generation under different scenarios. We identify which regions are most suitable for the expansion of industrial, commercial, and business operations using surplus renewable generation.

Stochastic bidding for VPPs enabled ancillary services: A case study

Strategic bidding which aims to optimally harvest the price difference in the wholesale electricity market can efficiently allocate VPPs’ aggregated resources to provide large flexibility for energy and frequency regulation (FR), aiding in the real-time rebalancing of supply and demand. However, the uncertainty of renewable energy results in substantial imbalances between supply and demand, which causes high randomness of system frequency deviation and significant price fluctuations in the electricity market, making bidding challenging. To tackle these issues, a risk-averse optimal bidding strategy is proposed for VPPs to participate in both energy and FR markets. Moreover, a payment recovery mechanism is designed to recover the cost of FR undersupply. Specifically, each VPP submits bids to maximize profit, while the market operator clears the market and penalizes undersupply through the payment recovery mechanism. The existence of Nash equilibrium is proved, and a distributed best response algorithm is implemented to calculate the risk-averse optimal bidding strategy, taking into account Denial-of-Service attacks. The mean-convergence and variance-convergence of our proposed algorithm are derived. A case study of Australian National Electricity Market (NEM) validates the effectiveness of the proposed method in reducing overbidding risk, strengthening FR service reliability and improving profits for VPPs.

Receding horizon dispatch of multi-period look-ahead market for energy storage integration

Although noteworthy efforts towards energy storage participation in electricity markets have been witnessed worldwide during the last decades, market mechanisms for optimal storage integration are yet to be comprehensively studied. This paper presents a novel approach wherein a multi-period look-ahead market is proposed to progressively solve the optimisation problem over a receding horizon, incorporating intertemporal constraints to enable forward planning. The focus is specifically on modelling the bidding behaviour of energy storage, and to address this, we develop two distinct models for the participation of energy storage in the market. The integrated model aims to schedule the future dispatch of the storage to minimise total operational cost, taking into account the fixed costs of the storage, while the strategic model pursues the dual ambition to maximise storage profit by formulating a bilevel optimisation problem. We validate the viability of our proposed market mechanisms through case studies on the Australian National Electricity Market. We conduct case studies which provide evidence that both the strategic and integrated bidding models effectively address the requirement for forward planning, resulting in reduced market costs and volatility. Notably, the strategic model outperforms in terms of generating higher storage revenue, while the integrated model exhibits a slight advantage in terms of lower market costs compared to the strategic model.

On the use of time varying marginal and time varying average emissions factors in building energy analysis

Calculation of greenhouse gas emissions from grid consumed electricity requires use of grid emissions intensity factors. In current building emissions regulations annual fixed factors are used, though there is growing recognition of the benefits of time-varying factors that take account of the varying emissions intensity as different generators ramp up/down. There are two main types of time varying emissions factors: marginal (MEF) and average (XEF) emissions intensity. Regulators seeking to draft policy around use of time varying emissions factors will need to evaluate the most appropriate methodology. Several studies in the scientific literature state MEF should be used to calculate savings from flexible demand strategies since it is the marginal generator that is the first to respond to small changes in demand. Here we suggest that XEF should in fact be the preferred approach in most scenarios. This recommendation is based on three factors; (i) alignment with historical emissions accounting methods; (ii) a more consistent basis from which to evaluate the alternative (baseline) scenario; and (iii) greater clarity over the incentive that it provides for investment in renewable generation. These factors are explored using examples and data from the Australian National Electricity Market.

Flexible specification testing in quantile regression models

AbstractWe propose three novel consistent specification tests for quantile regression models which generalize former tests in three ways. First, we allow the covariate effects to be quantile‐dependent and nonlinear. Second, we allow parameterizing the conditional quantile functions by appropriate basis functions, rather than parametrically. We are thereby able to test for general functional forms, while retaining linear effects as special cases. In both cases, the induced class of conditional distribution functions is tested with a Cramér–von Mises type test statistic for which we derive the theoretical limit distribution and propose a bootstrap method. Third, a modified test statistic is derived to increase the power of the tests. We highlight the merits of our tests in a detailed MC study and two real data examples. Our first application to conditional income distributions in Germany indicates that there are not only still significant differences between East and West but also across the quantiles of the conditional income distributions, when conditioning on age and year. The second application to data from the Australian national electricity market reveals the importance of using interaction effects for modeling the highly skewed and heavy‐tailed distributions of energy prices conditional on day, time of day and demand.

A Hyperbolic Bid Stack Approach to Electricity Price Modelling

Modelling the energy price in the Australian National Electricity Market (NEM) requires features that are not well reflected in existing models. We present a semi-structural, multi-regional model wherein bidding is not required to be cost-based, renewable fuels and storage technology are structurally integrated, and network constraints are often binding in optimal dispatch. Available fuel capacity then does not necessarily sum to registered bid capacity, as-bid fuel costs do not dependably follow input fuel prices, and cross-regional interconnectedness requires modelling trade. Furthermore, modelling the NEM spot price path must admit price negativity and price spikes. Extending previous work in the literature, the present paper proposes a hyperbolic bid stack approach to price modelling under these conditions.

Electricity price spike clustering: A zero-inflated GARX approach

Sudden dramatic rises in electricity prices, known as electricity “price spikes”, are ubiquitous in electricity spot markets worldwide. These price spikes often cluster. Energy retailers in many countries, including Australia, purchase electricity in an unregulated spot market and sell it to consumers at a heavily regulated price. As a result, the occurrence and clustering of spikes in the spot electricity price is particularly hazardous for retailers, and effective modelling and forecasting of these spikes is of foremost importance for effective risk management. This paper uses a data-dependent two-regime threshold autoregression to determine the electricity price spikes and proposes a zero-inflated generalized autoregression with exogenous covariates (ZIGARX) for modelling the time series count of price spikes. This model is shown to well capture the salient features of electricity price spikes, and can provide effective rolling probability forecasts of price spike occurrence. We apply the proposed approach to analyze the spot prices in four major regions in the Australian National Electricity Market (NEM) over the period April 2016 to February 2022. Results show that price spike clustering is captured very well using our model, and the persistence in price spike occurrence remains even after controlling for the relevant exogenous covariates. We also find that, in most of our sample period, the increasing use of renewable generation has mixed effects on the occurrence of price spikes.

The impact of rooftop solar on wholesale electricity demand in the Australian National Electricity Market

Solar energy from rooftop photovoltaic (PV) systems in Australia’s National Electricity Market (NEM) has been continuously increasing during the last decade. How much this change has affected power demand from electricity networks is an important question for both regulators and utility investors. This study aims to quantify the impact of rooftop solar energy generation on spot electricity demand and also to forecast power system load in the post-covid-19 era. Using half-hourly data from 2009 to 2019, we develop a novel approach to estimate rooftop solar energy generation before building regression models for wholesale electricity demand of each state. We find that the adoption of solar PV systems has significantly changed the levels and intra-day patterns of power demand, especially by reducing daytime power consumption from the grid and creating a “duck curve”. The results also show that most states in the NEM would see decreased electricity demand during 2019–2034.

Green hydrogen is a necessary component of a grid with an increasing non-dispatchable renewable energy supply

It is very well-known that wind and solar energy are supplied when there are resources, which are continuously changing, and not when demanded by the grid. Thus, in a grid without the supply of dispatchable electricity burning carbon and hydrocarbon combustion fuels, it is indispensable to include very large energy storage and given the variability not only short term within the day but from day to day following seasonality additional to weather, the large energy storage has to be mostly hydrogen. Extremely misleading are the cost assessment of prices of electricity to consumers through a Levelized Cost of Energy (LCOE) without factoring in dispatchability, as dispatchable and non-dispatchable electricity has a different value. Without energy storage, the production by wind and solar in the defect of the grid demand has to be supplied by other energy sources. Similarly, the production in excess would be wasted. This inconvenient truth has already produced increasing prices of electricity for customers. The Australian National Electricity Market data shows as phased with the uptake of wind and solar, in the absence of adequate energy storage, the price of energy supply has grown enormously. Without addressing the energy storage issue, mission net zero is incompatible with having cheap and abundant electricity.

Point-on-Wave Voltage Phase-Angle Jump Sensitivity Analysis of Grid-Connected Single-Phase Inverters

An extensive experimental analysis of the behavior of thirty-one off-the-shelf distributed photovoltaic (DPV) inverters to voltage phase angle jump (VPAJ) disturbance is done in this paper. Classification of undesirable behavior from DPV inverters is carried out based on the responses to voltage phase angle jump and identified the cause of such behavior through simulation and experimental evidence. Grid-connected inverters can respond undesirably to grid disturbance, such as disconnection and power curtailment, which can put <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$ 3-5$</tex-math></inline-formula> % of demand at risk and a cumulative <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$ 3-4$</tex-math></inline-formula> % grid at risk. This is a much larger contingency than before. Further, statistical analysis demonstrates the impact of such disturbances on DPV inverters in the Australian National Electricity Market. A sensitivity analysis is carried out to identify the vulnerability of inverters for VPAJ that occur at different points-on-wave (POW). This research aims to determine the scale of at-risk generation, understand how they operate, and what had to be performed to stop or reduce the risk. The ultimate impact is how to model this effect properly to guide the responses of DPV inverters, also making a significant study case internationally for systems with high penetration of DPV inverters.

Day ahead carbon emission forecasting of the regional National Electricity Market using machine learning methods

Accurate forecasting of regional electrical grid carbon emissions is an essential part of demand response programs for emissions reduction. Most existing research for short-term (day ahead) forecasting focuses on the use of variations of recurrent neural networks, in particular, the Long–Short​ Term Memory (LSTM) cells. This research proposes a Particle Swarm Optimised-extremely randomised trees (PSO-ERT) regressor model for day ahead forecasting of the emissions intensity of the Australian National Electricity Market (NEM). The PSO-ERT model is compared with LSTM, another method known as Extreme Learning Machine (ELM), and also with two classic machine learning algorithms, Multi-layer perceptron (MLP) and Decision Tree (DT). Models are trained using 1.5 years of weather, demand and generation data collected from the five NEM regions. Results show that the ERT model performs better than both LSTM and ELM with Root Mean Squared Error (RMSE) of 42.9, 109.8, 165.7, 25.4 and 74.1 for New South Wales, Tasmania, South Australia, Queensland and Victoria, respectively. For all models, forecasting performance was better for regions with less renewable generation and for times when renewable generation was a smaller percentage of demand suggesting that the addition of weather forecasts in the predictor feature set would further improve performance.