Correlations Of Forecast Errors Research Articles

Energy and reserve procurement in integrated electricity and heating system: A high-dimensional covariance matrix approach based on stochastic differential equations

The integration of stochastic renewable energy sources into energy systems presents challenges due to flexibility insufficiency and uncertainty modeling inaccuracy. To address these challenges, this paper explores a high-dimensional covariance matrix approach based on stochastic differential equations (SDEs) for energy and reserve co-dispatch in integrated electricity and heating systems. By capturing the long-term correlation of wind power forecast errors, the SDE model simultaneously models point forecasts and high-dimensional positive-definite covariance matrices, resulting in a compact uncertainty set. To ensure computational efficiency, we employ duality theory to transform the stochastic co-dispatch problem with the high-dimensional uncertainty set into deterministic convex programming. Additionally, a virtual heat storage model is introduced for the district heating network, leveraging the thermal inertia of the network to enhance reserve capacity. Simulations validate the calibration and sharpness of the proposed high-dimensional uncertainty set. The results demonstrate that co-dispatch using the SDE-based uncertainty set reduces operational costs by 4.9% compared to a set that does not consider the long-term correlation of wind power, under a 95% coverage rate. Moreover, the storage capabilities of district heating pipelines and smart buildings provide cost-free reserves, saving the operational costs of the two test systems by 2.8% and 1.3%, respectively.

Predicting the Equity Premium with Combination Forecasts: A Reappraisal

Abstract This paper reappraises the usefulness of combining individual forecasts for predicting the U.S. equity premium. For comparison, we also consider penalized regression and dimension reduction approaches. We fail to find evidence of predictive ability in recent decades, regardless of the forecasting method used. Further analysis shows that an increase in the correlation of individual forecast errors is an important factor in the declining performance of combination forecasts.()

Optimal reconciliation of hierarchical wind energy forecasts utilizing temporal correlation

Independent wind energy forecasts of a wind farm at different time horizons have limited accuracy, and they show disagreement despite relating to the same wind farm. The limited forecast accuracy is attributable to the insufficient information at a particular time horizon of the wind energy time series, whereas applying distinct forecasting methods to several time series of non-identical patterns at different time scales causes disagreement among forecasts. Mutual disagreement among less accurate forecasts negatively impacts the decision-making capabilities in associated power systems activities at distinct time scales. The configuration of time series expressing different time horizons at different levels of a non-overlapped hierarchically aggregated framework manifests a temporal hierarchy. Forecast combination through reconciliation of time series forecasts drawn at different hierarchical levels of temporal hierarchy using any state-of-the-art method facilitates the sharing of diverse information across the hierarchy; consequently, accuracy and mutual agreement of forecasts improve. Such benefits may be further enhanced by embedding intra- and inter-level forecast error correlations in the forecast reconciliation process. However, the forecast error covariance matrix of temporal hierarchy becomes a complex high-dimensional structure while accommodating intra- and inter-level correlations. Estimating such a matrix is challenging since the high-dimensional structure severely impedes the identifiability of model parameters. Besides, in the hierarchical forecast reconciliation process, the number of predictor variables is generally higher than the number of samples. This condition gives rise to a singular covariance matrix, making it non-invertible, and thus obstructs its parameter estimation. This work employs the MinT(shrinkage) covariance matrix estimator that considers all correlations and shrinks the non-diagonal components of the matrix toward zero to avert the complexity and, therefore, the non-identifiability. Additionally, the shrinkage parameter λ of MinT(shrinkage) conveniently obtains the invertible matrix. The case study validates that while incorporating the intra- and inter-level forecast error correlations, MinT(shrinkage) provides competitively accurate and mutually agreed forecasts over other reconciliation methods.

Analysis of spatially coherent forecast error structures

AbstractUnderstanding error properties is an essential part in numerical weather prediction. Predictable relationship between errors of different regions due to some underlying systematic or random process can give rise to correlated errors. Estimation of error correlation is crucial for improvement of forecasts. However, the size of the corresponding correlation matrix is larger than what is possible to represent on geographical maps in order to diagnose its full spatial variation. Here, we propose a complex network‐based approach to analyse forecast error correlations that enables us to estimate the spatially varying component of the error. A quantitative study of the spatio‐temporal coherent structures of medium‐range forecast errors of different climate variables using network measures can reveal common sources of errors. Such information is crucial, especially in cases such as the outgoing long‐wave radiation, in which errors are correlated across very long distances, indicating an underlying climate mechanism as the source of the error. We show that the spatial patterns of forecast error co‐variability may not be the same as that of the corresponding climate variable itself, thereby implying that the mechanisms behind the correlated errors may be different from the climate processes responsible for the spatio‐temporal interactions of the climate variable. Our results highlight the importance of diagnosing the full spatial variation of error correlations to understand the origin and propagation of forecast errors, and demonstrate complex networks to be a promising diagnostic tool in this regard.

Spatial correlation in weather forecast accuracy: a functional time series approach

A functional time series approach is proposed for investigating spatial correlation in daily maximum temperature forecast errors for 111 cities spread across the U.S. The modelling of spatial correlation is most fruitful for longer forecast horizons, and becomes less relevant as the forecast horizon shrinks towards zero. For 6-day-ahead forecasts, the functional approach uncovers interpretable regional spatial effects, and captures the higher variance observed in inland cities versus coastal cities, as well as the higher variance observed in mountain and midwest states. The functional approach also naturally handles missing data through modelling a continuum, and can be implemented efficiently by exploiting the sparsity induced by a B-spline basis. The temporal dependence in the data is modeled through temporal dependence in functional basis coefficients. Independent first order autoregressions with generalized autoregressive conditional heteroskedasticity [AR(1)+GARCH(1,1)] and Student-t innovations work well to capture the persistence of basis coefficients over time and the seasonal heteroskedasticity reflecting higher variance in winter. Through exploiting autocorrelation in the basis coefficients, the functional time series approach also yields a method for improving weather forecasts and uncertainty quantification. The resulting method corrects for bias in the weather forecasts, while reducing the error variance.

Multi-objective risk analysis for flood control operation of a complex reservoir system under multiple time-space correlated uncertainties

Many uncertainties are involved in flood control operation of a multi-reservoir system and result in risks for the system. Risk analysis for reservoir flood control operation is essential for decision making. Traditional flood control risk analysis only takes flood forecast error as the main risk source, ignoring the influence of dynamic control of flood-limited water level on reservoir initial water level for flood regulation. In this study, the spatial correlation of reservoir initial water level errors and the spatiotemporal correlation of flood forecast errors are identified via copula function. A risk analysis model considering both upstream and downstream is established for multi-objective flood control operation of a complex reservoir system, coupling multi-dimensional uncertainties of reservoir initial water levels, flood forecast errors, reservoir capacity curve errors, reservoir discharge curve errors and river flood routing errors. The impact of multiple risk sources with spatiotemporal correlations on reservoir flood control operation is then evaluated and the competitive tradeoff between flood control risks in upstream and downstream analyzed. The model is applied to a mixed four-reservoir system in Pi River Basin in China. Results indicated that 1) there is a strong correlation between the reservoir initial water level errors in space and also between the flood forecast errors in space and time, which can be effectively described by copula function, and ignoring the correlations will underestimate the flood control risk; 2) the two objective values of upstream and downstream show a clear competitive relationship, and the solution that prefers one objective has low risk marginal benefit, thus choosing a compromise solution can balance risks; 3) coupling the initial water level uncertainty reduces the flood control risk caused by the flood forecast errors to a certain extent, but also lead to the increase of extreme risk loss.

Reliability sensitivity of wind power system considering correlation of forecast errors based on multivariate NSTPNT method

The impact of wind power forecast errors (WPFEs) on power system reliability can be quantified by a sensitivity model, which helps to determine the importance of different wind farms. However, the unknown distribution and correlation of WPFEs make it difficult to calculate the reliability sensitivity. The existing univariate non-standard third-order polynomial normal transformation (NSTPNT) expresses the reliability sensitivity of WPFEs by a normal random variable with explicit distribution, and is not suitable for multiple wind farms with correlated forecast errors. In this paper, the univariate NSTPNT method is extended to the multivariate by deriving the analytical expression of the correlation coefficients before and after the transformation, to establish the transformation between the WPFEs and a normal random vector (RV) with the specific correlation. A reliability sensitivity model to the WPFEs expressed to the normal RV is then proposed. The numerical results validate the accuracy of the proposed multivariate NSTPNT and the sensitivity model. The maximum relative error for using the sensitivity to approximate the change of reliability with distribution parameters of the WPFEs is less than 2.42%. The necessity of considering the correlation of WPFEs is analyzed. The maximum relative error of the sensitivity reaches 83% when the correlation is ignored.

Two stage unit commitment considering multiple correlations of wind power forecast errors

When the correlation of wind power output among wind farms is not considered, the integrated stochastic characteristics of wind power will not be captured accurately. Using this inaccurate feature may lead to an impractical even a failing result of unit commitment (UC). Therefore, this paper proposes a multiple correlations model for wind power forecast errors (WPFEs), and to capture this multiple correlation feature in UC problem, a two-stage chance-constrained interval UC (CIUC) model is proposed. First, an analytical expression of multiple correlations, including spatial, temporal and conditional correlations, is presented to improve the description accuracy of stochastic WPFEs. To strike a balance between risk and operational cost, a chance-constrained decision method is developed to optimize the time-varying interval of wind power output in the first stage. Subsequently, an interval UC model is established to determine the optimal operational schedule in the second stage. Finally, the proposed CIUC model is solved using a solution strategy that combines column-and-constraint generation and sample average approximation. The effectiveness and practicality of the proposed method are verified via the numerical results for IEEE 39-bus and 118-bus systems.

Economic implications of forecasting electricity generation from variable renewable energy sources

Short-term forecasting of electricity generation from variable renewable energy sources is not an end in itself but should provide some net benefit to its user. In the case of electricity trading, which is in the focus of this paper, the benefit can be quantified in terms of an improved economic outcome. Although some effort has been made to evaluate and to improve the profitability of electricity forecasts, the understanding of the underlying effects has remained incomplete so far. In this paper, we develop a more comprehensive theoretical framework of the connection between the statistical and the economic properties of day-ahead electricity forecasts. We find that, apart from the accuracy and the bias, which have already been extensively researched, the correlation between the forecast errors and the market price spread determines the economic implications - a phenomenon which we refer to as ‘correlation effect’. Our analysis is completed by a case study on solar electricity forecasting in Germany which illustrates the relevance and the limits of both our theoretical framework and the correlation effect. • The economics of short-term electricity forecasts are studied analytically and empirically. • The value and the risk of forecast-based electricity trading are evaluated. • We find that the economics depend on the correlation of forecast errors with prices. • Hence, the most accurate forecast is not necessarily the most accurate.

Analyst forecasts: sales and profit margins

Sales and profit margins are two popular earnings components discussed in the media. We study properties of one-year-ahead analyst forecasts of these two components. As sales are in dollar amounts and profit margin is a ratio, we propose robust statistical methods to assess and contrast their forecast properties. We find that four performance properties associated with earnings forecasts—optimism, relative accuracy with respect to benchmark model forecasts, forecast suboptimality, and serial correlation of forecast errors—apply to both sales and profit margins. Sales forecasts, in general, perform better than profit margin forecasts. Further evidence also shows that sales forecasts perform better than profit margin forecasts in terms of how their forecast errors explain earnings forecast errors and how realized surprises affect adjustments of the respective forecasts. We also find that a better information environment, surrogated by size, improves sales forecasts more than profit margin forecasts. All of these findings suggest that forecasting profit margins is inherently more difficult than forecasting sales.

A Deficit of Seasonal Temperature Forecast Skill over West Coast Regions in NMME

Abstract This study investigates the forecast skill of seasonal-mean near-surface (2 m) air temperature in the North American Multimodel Ensemble (NMME) Phase 2, with a focus on the West Coast of the United States. Overall, 1-month lead time NMME forecasts exhibit skill superior or similar to persistence forecasts over many continental regions, and skill is generally higher over the ocean than the continent. However, forecast skill along most West Coast regions is markedly lower than in the adjacent ocean and interior, especially during the warm seasons. Results indicate that the poor forecast skill along the West Coast of the United States reflects deficiencies in their representation of multiple relevant physical processes. Analyses focusing on California find that summer forecast errors are spatially coherent over the coastal region and the inland region individually, but the correlation of forecast errors between the two regions is low. Variation in forecast performance over the coastal California region is associated with anomalous geopotential height over the lower middle latitudes and subtropics of the eastern Pacific, North America, and the western Atlantic. In contrast, variation in forecast performance over the inland California region is associated with the atmospheric circulation over the western United States. Further, it is found that forecast errors along the California coast are linked to anomalies of low cloudiness (stratus clouds) along the coastal region.

Optimal sizing of energy storage considering the spatial‐temporal correlation of wind power forecast errors

Energy storage is considered as an effective approach to deal with the power deviation that caused by the stochastic wind power forecast error. Because of the spatial-temporal correlation of forecast errors for wind farms, which locate close to each other and integrate into the same regional power grid, energy storage could be deployed effectively and economically. Therefore, this study proposes an optimisation method to size the capacity of energy storage system (ESS) considering the spatial-temporal correlation of forecast errors for multiple nearby wind farms. The copula theory based correlation model of high-dimensional forecast errors is built to capture the spatial-temporal characteristics of forecast error. Then, according to multiple scenarios technique, an optimal ESS sizing model is established to minimise the investment and operation costs of the ESS. Meanwhile, an operation strategy considering the trend of prediction and correlation of errors is used to implement rolling operation strategy of ESS. The case studies demonstrate the effectiveness of the new method and illustrate the significant impact of the correlation of forecast error on the capacity of ESS.

The business cycle implications of fluctuating long run expectations

I consider a DSGE model where consumption depends on the present discounted value of wage and capital income. The agent is uncertain if these variables are stationary or non-stationary and puts positive probability on both representations. The agent uses Bayesian learning to update his probability weights on each model and these weights vary over time according to how well each model fits the data. The model exhibits an improved fit to the data relative to a no-learning benchmark. It requires half the standard deviation of exogenous shocks to match the volatility of output and still matches the relative volatilities of key business cycle variables. The model lowers the contemporaneous correlation of consumption and wages with output and generates positive autocorrelation in growth rates. Impulse responses exhibit persistent responses and forecast errors are positively serially correlated. Finally, in contrast to the existing literature, the model endogenously generates observed time varying volatility and long run predictability of business cycle variables, especially for investment, without generating counterfactually high serial correlation of forecast errors.

CorClustST—Correlation-based clustering of big spatio-temporal datasets

Increasing amounts of high-velocity spatio-temporal data reinforce the need for clustering algorithms which are effective for big data processing and data reduction. As currently applied spatio-temporal clustering algorithms have certain drawbacks regarding the comparability of the results, we propose an alternative spatio-temporal clustering technique which is based on empirical spatial correlations over time. As a key feature, CorClustST makes it easily possible to compare and interpret clustering results for different scenarios such as multiple underlying variables or varying time frames. In a test case, we show that the clustering strategy successfully identifies increasing spatial correlations of wind power forecast errors in Europe for longer forecast horizons. An extension of the clustering algorithm is finally presented which allows for a large-scale parallel implementation and helps to circumvent memory limitations. The proposed method will especially be helpful for researchers who aim to preprocess big spatio-temporal datasets and who intend to compare clustering results and spatial dependencies for different scenarios.

International Diversification, SFAS 131 and Post‐Earnings‐Announcement Drift

AbstractUsing 1990 through 2013 data of U.S. firms with foreign operations, we show that (i) the serial correlation of analyst forecast errors increases in the extent of international diversification, (ii) post‐earnings‐announcement drift (PEAD) based on analyst forecast errors increases in the extent of international diversification, and (iii) the impact of international diversification on the serial correlation of analyst forecast errors and its associated drift is significantly reduced after the implementation of SFAS 131 on segment disclosures. When we replicate our tests using seasonally differenced earnings, we fail to observe similar patterns. Overall, our results suggest that investors’ underreaction to announced earnings is a likely explanation for PEAD. Our findings also indicate that disclosures required under SFAS 131 are useful to analysts in forming efficient earnings expectations, thereby helping capital market participants in the pricing of internationally diversified firms’ earnings.

International Diversification, SFAS 131, and Post-Earnings Announcement Drift

Using 1990 through 2013 data of U.S. firms with foreign operations, we show that (1) the serial correlation of analyst forecast errors increases to the degree that firms diversify internationally, (2) post-earnings-announcement drift (PEAD) based on analyst forecast errors increases to the degree that firms increase international operations, and (3) international diversification’s impact on the serial correlation of analyst forecast errors and its associated drift is significantly reduced after the implementation of SFAS 131, which requires firms to report disaggregated information of operating segments based on the way management organizes the segments within the enterprise for making operating decisions and assessing performance. When we replicate our tests using seasonally differenced earnings, we fail to observe patterns similar to those using analyst forecast errors. Overall, the results of our study suggest that investors’ underreaction to announced earnings is a likely explanation for PEAD. Our findings also indicate that disclosures required under SFAS 131 are informative in helping analysts more accurately predict earnings, which can help capital markets price internationally diversified firms’ earnings.

Refining Financial Analystss Forecasts by Predicting Earnings Forecast Errors

Prior research on financial analyst’ quarterly earnings forecasts has documented serial correlation in forecast errors. This paper examines the way serial correlation in quarterly earnings forecast errors varies with firm and analyst attributes such as the firm’s industry and the analyst’s experience and brokerage house affiliation. Finding that serial correlation in forecast errors is significant and seemingly independent of firm and analyst attributes, I model consensus forecast errors as an autoregressive process. I demonstrate that the model of forecast errors that best fits the data is AR(1), and use the obtained autoregressive coefficients to predict consensus forecast errors. Modeling the consensus forecast errors as an autoregressive process, the present study predicts future consensus forecast errors, and proposes a series of refinements to the consensus. These refinements were not presented in prior literature, and can be useful to financial analysts and investors.

Refining financial analysts’ forecasts by predicting earnings forecast errors

PurposeThe purpose of this paper is to examine the way serial correlation in quarterly earnings forecast errors varies with firm and analyst attributes such as the firm’s industry and the analyst’s experience and brokerage house affiliation. Prior research on financial analysts’ quarterly earnings forecasts has documented serial correlation in forecast errors.Design/methodology/approachFinding that serial correlation in forecast errors is significant and seemingly independent of firm and analyst attributes, the consensus forecast errors are modeled as an autoregressive process. The model of forecast errors that best fits the data is AR(1), and the obtained autoregressive coefficients are used to predict consensus forecast errors.FindingsModeling the consensus forecast errors as an autoregressive process, the present study predicts future consensus forecast errors and proposes a series of refinements to the consensus.Originality/valueThese refinements were not presented in prior literature and can be useful to financial analysts and investors.

Assimilation of MODIS Dark Target and Deep Blue observations in the dust aerosol component of NMMB-MONARCH version 1.0

Abstract. A data assimilation capability has been built for the NMMB-MONARCH chemical weather prediction system, with a focus on mineral dust, a prominent type of aerosol. An ensemble-based Kalman filter technique (namely the local ensemble transform Kalman filter – LETKF) has been utilized to optimally combine model background and satellite retrievals. Our implementation of the ensemble is based on known uncertainties in the physical parametrizations of the dust emission scheme. Experiments showed that MODIS AOD retrievals using the Dark Target algorithm can help NMMB-MONARCH to better characterize atmospheric dust. This is particularly true for the analysis of the dust outflow in the Sahel region and over the African Atlantic coast. The assimilation of MODIS AOD retrievals based on the Deep Blue algorithm has a further positive impact in the analysis downwind from the strongest dust sources of the Sahara and in the Arabian Peninsula. An analysis-initialized forecast performs better (lower forecast error and higher correlation with observations) than a standard forecast, with the exception of underestimating dust in the long-range Atlantic transport and degradation of the temporal evolution of dust in some regions after day 1. Particularly relevant is the improved forecast over the Sahara throughout the forecast range thanks to the assimilation of Deep Blue retrievals over areas not easily covered by other observational datasets. The present study on mineral dust is a first step towards data assimilation with a complete aerosol prediction system that includes multiple aerosol species.

A case study involving single observation experiments performed over snowy Siberia using a coupled atmosphere‐land modelling system

AbstractThrough a series of single observation experiments, we investigate forecast error covariance and correlation structures by applying the Maximum Likelihood Ensemble Filter (MLEF) data assimilation method with a coupled atmosphere–land surface model. We consider a coupled approach that includes the full complexity of the cross‐component covariance and correlation structures, specifically between 2‐m air temperature and snow temperature. We show that the error covariance and correlation are complex and are strongly dependent on both weather conditions and the land surface scheme, indicating a need for further investigation of the relevance of flow‐dependent ensemble covariance in coupled systems. We also demonstrate that the use of coupled error covariance methods improves the efficiency of information transfer between the atmosphere and the land surface by allowing the well‐observed atmosphere to influence land surface variables.