Real-world Market Data Research Articles

Interpretable Transformer Model for Capturing Regime Switching Effects of Real-Time Electricity Prices

Real-time electricity prices are economic signals incentivizing market players to support real-time system balancing. These price signals typically switch between low- and high-price regimes depending on whether the power system is in surplus or shortage of generation, which is hard to capture. In this context, we propose a new Transformer-based model to assist the short-term trading strategies of market players. The proposed model offers high-performance probabilistic forecasts of real-time prices while providing insights into its inner decision-making process. Transformers rely on attention mechanisms solely computed via feed-forward networks to explicitly learn temporal patterns, which allows them to capture complex dependencies such as regime switching. Here, we augment Transformers with subnetworks dedicated to endogenously quantify the relative importance of each input feature. Hence, the resulting forecaster intrinsically provides the temporal attribution of each input feature, which foster trust and transparency for subsequent decision makers. Our case study on real-world market data of the Belgian power system demonstrates the ability of the proposed model to outperform state-of-the-art forecasting techniques, while shedding light on its most important drivers.

Foreign Exchange Options on Heston-CIR Model Under Lévy Process Framework

In this paper, we consider the Heston-CIR model with Lévy process for pricing in the foreign exchange (FX) market by providing a new formula that better fits the distribution of prices. To do that, first, we study the existence and uniqueness of the solution to this model. Second, we examine the strong convergence of the Lévy process with stochastic domestic short interest rates, foreign short interest rates and stochastic volatility. Then, we apply Least Squares Monte Carlo (LSMC) method for pricing American options under our model with stochastic volatility and stochastic interest rate. Finally, by considering real-world market data, we illustrate numerical results for the four-factor Heston-CIR Lévy model.

A reinforcement and imitation learning method for pricing strategy of electricity retailer with customers’ flexibility

The effective pricing of retail broker in competitive electricity market constitutes a key problem toward four goals: (1) the maximization of the broker’s economic benefits; (2) the balance between customers’ energy supply and demand; (3) the realization of the energy supply and demand flexibility potential of customers; (4) the constraint that prevents the retail prices from too high or too low. Unfortunately, few studies can achieve four goals simultaneously. Moreover, the complicated electricity trading environment with continuous states and actions also increases the difficulty of learning optimal pricing strategy. To solve these problems, a reinforcement and imitation learning approach is proposed to develop the optimal pricing strategy of retail broker in this paper. Specifically, the proposed approach consists of a demand prediction method to predict customers’ energy demand and supply volume, a self-generated expert knowledge imitation learning mechanism to instruct the agent to imitate given expert policy with generated expert knowledge, and an action policy learning method. Different from existing studies, our approach achieves all four goals and exploits the generated transition tuples fully to learn a more effective pricing strategy. The proposed scheme is verified by experiments using real-world market data, the experimental results illustrate our proposed approach gains 9.71%, 3.32%, and 15.94% higher economic profits than three state-of-the-art pricing strategies, respectively. Meanwhile, the total needed computation time for our method to learn an effectiveness pricing strategy is only 4102 s. The results show that our method gains the highest economic profits for the broker with acceptable computation cost. Moreover, the changing curves of customers’ consumption/production habits demonstrate that the proposed method could achieve demand/supply response of customers.

Quantitative study of candlestick pattern & identifying candlestick patterns using deep learning for the Indian stock market

The stock market is an integral aspect of any country’s economic infrastructure. Analyzing and attempting to play the markets to maximize profits is an endeavor a large fraction of the population aspires to. Candlestick patterns are the backbone of Technical Analysis, used for trading in the stock market. There are a number of candlestick patterns in the market, each with its own benefits and downsides. Due to this, the task that befalls the hands of analysts is deciding which patterns provide the most effective gauge of the current market situation. Due to the large level of noise and widely recognized semi-strong form of market efficiency, analyzing and forecasting the stock market is infamously difficult. For traders that use Technical Analysis to trade, it's critical to be able to recognize candlestick patterns quickly. We will be attempting to determine their respective effectiveness with respect to the Indian Stock Market via exploratory analysis conducted on real-world market data. Also, we'll use candlestick charts to train neural networks and subsequently find patterns. Deep Learning will be used to recognize candlestick patterns in large-cap Indian equities.

Beating the market with a bad predictive model

It is a common misconception that in order to make consistent profits as a trader, one needs to possess some extra information leading to an asset value estimation that is more accurate than that reflected by the current market price. While the idea makes intuitive sense and is also well substantiated by the widely popular Kelly criterion, we prove that it is generally possible to make systematic profits with a completely inferior price-predicting model. The key idea is to alter the training objective of the predictive models to explicitly decorrelate them from the market. By doing so, we can exploit inconspicuous biases in the market maker’s pricing, and profit from the inherent advantage of the market taker. We introduce the problem setting throughout the diverse domains of stock trading and sports betting to provide insights into the common underlying properties of profitable predictive models, their connections to standard portfolio optimization strategies, and the commonly overlooked advantage of the market taker. Consequently, we prove the desirability of the decorrelation objective across common market distributions, translate the concept into a practical machine learning setting, and demonstrate its viability with real-world market data.

Understanding Convergence Bids During Blackouts: Analytical Results and Real-World Implications

Convergence bidding is a financial instrument that is widely adopted in recent years in two-settlement electricity markets to reduce the price gap between the day-ahead market (DAM) and the real-time market (RTM). This paper, for the first time, investigates the operation and impact of convergence bids (CBs) during <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">blackouts</i> . First, the amount of load shedding in the RTM is modeled as a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">function</i> of the amount of the cleared CBs in the DAM. The sign of the slope of this function is proposed as a metric to determine if a CB <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">exacerbates</i> or <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">heals</i> the power outages. Next, a series of mathematical theorems are developed to obtain and characterize this new metric under different network conditions. It is proved that, when there is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">no</i> congestion in the DAM, the metric is always greater than or equal to zero. When there <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">is</i> congestion in the DAM, the metric can be positive or negative. Using numerical case studies, we show that, not only when there is no congestion, but also most often when there is congestion, the introduced metric is positive. Therefore, supply CBs almost always hurt the system during blackouts while demand CBs almost always help the system. Furthermore, the impact of load shedding on the profit of CBs is also analyzed. It is shown that, load shedding usually creates advantage for supply CBs and disadvantage for demand CBs in terms of their profit. The implications of these results are discussed. We also analyze the real-world market data from the California Independent System Operator (ISO) during the blackouts in August 2020. It is shown that, the decision by the California ISO to suspend CBs during this event matches the mathematical and numerical results that are obtained and discussed in this paper.

Uplift modeling with value-driven evaluation metrics

Measuring the success of targeted marketing actions is challenging. Research on prescriptive analytics recommends uplift models to guide targeting decisions. Uplift models predict how much a marketing action will change customers' behavior, known as the individual treatment effect (ITE). Marketers can then solicit customers in decreasing order of their estimated ITE. We argue that the ITE-based targeting policy is not fully consistent with a business value maximization objective. We propose business-centric evaluation metrics that integrate estimates of the ITE and the expected business value and validate their benefits relative to the ITE-based targeting baseline using real-world marketing data. The new metrics yield remarkably higher profit across different uplift models, targeting depths, profit functions, and data sets. They further contribute to the growing field of interpretable data science by uncovering interdependencies between covariates, ITE, and profit and by clarifying whether customers are worth targeting because of high responsiveness or high value.

Coordinated Bidding Strategy of Wind Farms and Power-to-Gas Facilities Using a Cooperative Game Approach

Power-to-gas (P2G) technology is an effective way to provide additional flexibility to electricity systems with high penetration of renewable energy. Effective P2G bidding strategies in electricity markets can bring in arbitrage opportunities. This article proposes a coordinated bidding strategy for wind farms and P2G facilities using a cooperative game approach. A coordination framework for optimizing the profit of wind farms and P2G facilities is presented, in which wind power participates in day-ahead, real-time and reserve markets, and the P2G facilities participate in day-ahead and reserve markets for arbitrage. Wind farms strategically purchase reserve to avoid high imbalance penalties. As a coalition, wind farms can provide energy to P2G facilities and receive reserve capacity in return. The coordinated bidding strategy is modeled on stochastic programming to maximize the expected joint profit of wind farms and P2G facilities. Nucleolus and Shapley-value based methods are proposed to allocate profits to each player in the coalition. Numerical results based on real world market data demonstrate that the proposed bidding strategy can increase the profits of both wind farms and P2G facilities. Strategically purchasing reserve can reduce the imbalance penalties of wind farms, and participating in reserve markets of P2G facilities can increase profit.

A risk-constrained decision support tool for EV aggregators participating in energy and frequency regulation markets

The growing trend of electric vehicles (EVs) in recent years has led to the emergence of EV aggregators in electricity markets. Generally, the main goal of an EV aggregator is to buy electricity from the wholesale market in a cost-effective manner while satisfying the charging requirements of EV owners. Accordingly, this paper presents a decision support tool for EV aggregators which enables them to determine the optimal bidding strategy to effectively participate in the day-ahead and real-time energy, and frequency regulation markets. Indeed, the aggregator mainly obtains profit by selling energy during the high-price hours (via vehicle-to-grid (V2G) capability) and providing primary frequency regulation service to the system operator. The proposed approach is based on a two-stage stochastic programming method, where risk aversion is modeled through the conditional value-at-risk (CVaR). The underlying uncertainties including the real-time energy prices, real-time regulation service deployments, and the uncertainties associated with the EV owners (i.e., arrival time, departure time, and initial battery state-of-charge (SOC)) are modeled as stochastic processes that are represented by different sets of scenarios. The cardinality of the combined scenario set is reduced using backward probability distance algorithm to make the resulting mixed-integer linear programming (MILP) problem tractable in large-scale and real-world cases. Extensive numerical analysis with one thousand EVs and real-world market data are conducted to validate the efficiency of the proposed approach in terms of solution optimality, robustness, and computation efficiency.

A valence asymmetry in predecisional distortion of information: Evidence from an eye tracking study with incentivized choices.

Existing research shows that the order in which evidence arrives can bias its evaluation and the resulting decision in favor of information encountered early on. We used eye-tracking to study the underlying cognitive mechanisms in the context of incentivized financial choices based on real world market data. Subjects learned about the presence/absence of a transaction fee, before seeing expert opinions regarding an investment prospect and deciding whether to invest. Although the fee had no effect on the processing of negative opinions, we found that positive ones were processed more effortlessly (with lower gaze duration and pupil dilation) when it was absent, that is, when they were congruent with the positive initial information in the shape of the lack of fees. Despite their more effortless processing in the absence of fees, positive opinions then had a greater impact on the subjects' beliefs. In addition to an initial study with 100 subjects, these findings were replicated in a second, preregistered experiment with 103 subjects, in which a positive premium was paid in the event of no fee. Thus, we argue that the valence asymmetry in favor of positive information observed in evaluative priming, person perception, and related tasks (the density hypothesis) also plays a crucial role in incentivized economic choice. In fact, rather than being a detrimental bias, the overweighting of initial evidence often observed in decisions could be seen as an adaptive heuristic aimed at reducing the cost of processing later, similar information. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Effects of correlations and fees in random multiplicative environments: Implications for portfolio management.

Geometric Brownian motion (GBM) is frequently used to model price dynamics of financial assets, and a weighted average of multiple GBMs is commonly used to model a financial portfolio. Diversified portfolios can lead to an increased exponential growth compared to a single asset by effectively reducing the effective noise. The sum of GBM processes is no longer a log-normal process and has a complex statistical properties. The nonergodicity of the weighted average process results in constant degradation of the exponential growth from the ensemble average toward the time average. One way to stay closer to the ensemble average is to maintain a balanced portfolio: keep the relative weights of the different assets constant over time. To keep these proportions constant, whenever assets values change, it is necessary to rebalance their relative weights, exposing this strategy to fees (transaction costs). Two strategies that were suggested in the past for cases that involve fees are rebalance the portfolio periodically and rebalance it in a partial way. In this paper, we study these two strategies in the presence of correlations and fees. We show that using periodic and partial rebalance strategies, it is possible to maintain a steady exponential growth while minimizing the losses due to fees. We also demonstrate how these redistribution strategies perform in a phenomenal way on real-world market data, despite the fact that not all assumptions of the model hold in these real-world systems. Our results have important implications for stochastic dynamics in general and to portfolio management in particular, as we show that there is a superior alternative to the common buy-and-hold strategy, even in the presence of correlations and fees.

Switching to Non-Affine Stochastic Volatility: A Closed-Form Expansion for the Inverse Gamma Model

This paper introduces the Inverse Gamma (IGa) stochastic volatility model with time-dependent parameters, defined by the volatility dynamics dVt = κt.(θt − Vt).dt λt.Vt.dBt. This non-affine model is much more realistic than classical affine models like the Heston stochastic volatility model, even though both are as parsimonious (only four stochastic parameters). Indeed, it provides more realistic volatility distribution and volatility paths, which translate in practice into more robust calibration and better hedging accuracy, explaining its popularity among practitioners.In order to price vanilla options with IGa volatility, we propose a closed-form volatility-of-volatility expansion. Specifically, the price of a European put option with IGa volatility is approximated by a Black-Scholes price plus a weighted combination of Black-Scholes greeks, where the weights depend only on the four time-dependent parameters of the model. This closed-form pricing method allows for very fast pricing and calibration to market data. The overall quality of the approximation is very good, as shown by several calibration tests on real-world market data where expansion prices are compared favorably with Monte Carlo simulation results.This paper shows that the IGa model is as simple, more realistic, easier to implement and faster to calibrate than classical transform-based affine models. We therefore hope that the present work will foster further research on non-affine models like the Inverse Gamma stochastic volatility model, all the more so as this robust model is of great interest to the industry.

Segmenting Customers from Population to Individuals: Does 1-to-1 Keep Your Customers Forever?

There have been various claims made in the marketing community about the benefits of 1-to-1 marketing versus traditional customer segmentation approaches and how much they can improve understanding of customer behavior. However, few rigorous studies exist that systematically compare these approaches. In this paper, we conducted such a study and compared the predictive performance of aggregate, segmentation, and 1-to-1 marketing approaches across a broad range of experimental settings, such as multiple segmentation levels, multiple real-world marketing data sets, multiple dependent variables, different types of classifiers, different segmentation techniques, and different predictive measures. Our experiments show that both 1-to-1 and segmentation approaches significantly outperform aggregate modeling. Reaffirming anecdotal evidence of the benefits of 1-to-1 marketing, our experiments show that the 1-to-1 approach also dominates the segmentation approach for the frequently transacting customers. However, our experiments also show that segmentation models taken at the best granularity levels dominate 1-to-1 models when modeling customers with little transactional data using effective clustering methods. In addition, the peak performance of segmentation models are reached at the finest granularity levels, skewed towards the 1-to-1 case. This finding adds support for the microsegmentation approach and suggests that 1-to-1 marketing may not always be the best solution

Establishing impacts of the inputs in a feedforward neural network

Artificial neural network models are now being widely used in various areas of statistical research. Nevertheless, there is a certain degree of reluctance amongst members of the business profession in applying neural networks to business analysis. One of the major causes of scepticism is the inability of the models to provide explanation on how they reach their decisions. The current experiment is concerned with solving this problem by developing a framework for establishing the impacts of the input variables on the network output. The framework was tested on a feedforward neural network model for turnover forecasting which was developed in co-operation with a British retailer using real world marketing data. The results obtained are compared with those from a sensitivity analysis.

Information-theoretic estimation of individual consideration set

Generally, in the choice modeling literature, each individual decision-maker is assumed to be aware of a set of alternatives at the start of the decision process. This initial set defined as the awareness set usually includes a mixture of alternatives, some seriously and some casually under contemplation. The decision-making process leads to a partitioning of the awareness set into two disjoint subsets: consideration set and non-consideration set. The consideration set consists of the more seriously considered alternatives. This paper proposes an information-theoretic algorithm for estimating the consideration set. The proposed algorithm estimates the choice probabilities in the awareness sets according to the maximum entropy principle which by use of a multinomial logit model, computes an individual information index for each set, identifies the weak or unacceptable alternatives for each individual, and reduces each awareness set to a consideration set. The average of the selected alternative probabilities is proposed as a statistic by which the predicmtive quality of various consideration sets can be compared. We empirically illustrate the proposed method by estimating consideration sets for a sample of industrial decision-makers using a real world marketing data set. The analysis leads to an important finding; that the predictive power of the multinomial logit is in identifying the weak (non-considered) alternatives rather than predicting the chosen alternative among the choices in the consideration set. The proposed algorithm enables us to empirically implement choice set reduction for a given data set using an information criterion. Thus, the stage is set for advancing choice theory by investigating the impact of using additional data and/or other choice models on the derived consideration sets.