Financial Prediction Problems Research Articles

A data mining framework for financial prediction

In the financial markets, because real-time transactions directly relate to profit, it is important to process and analyze data on a real-time basis. In practice, decisions influenced by experts’ experiences from fundamental and technical analysis occur frequently compared to decisions using prediction algorithms. A domain-specific data mining framework was proposed recently to reduce related cost. Therefore, this study proposes a novel data mining framework suitable for financial markets according to expert knowledge. The proposed framework predominantly considers the following three perspectives as the standards for the effectiveness of research: interpretability, proper prediction metrics, and reporting methods. We applied our framework to the real-world financial prediction problems, such as the 3–10 year treasury spread forecasts. Consequently, we achieved an 84% prediction performance on the spread prediction and used hierarchical information to provide additional insight. In addition, we obtained practical knowledge and synergies through extraction of critical variables that can be used as a quick and accurate data-driven decision making support tool by active agents in the real world.

Fusion in stock market prediction: A decade survey on the necessity, recent developments, and potential future directions

Investment in a financial market is aimed at getting higher benefits; this complex market is influenced by a large number of events wherein the prediction of future market dynamics is challenging. The investors’ etiquettes towards stock market may demand the need of studying various associated factors and extract the useful information for reliable forecasting. Fusion can be considered as an approach to integrate data or characteristics, in general, and enhance the prediction based on the combinational approach that can aid each other. We conduct a systematic approach to present a survey for the years 2011–2020 by considering articles that have used fusion techniques for various stock market applications and broadly categorize them into information fusion, feature fusion, and model fusion. The major applications of stock market include stock price and trend prediction, risk analysis and return forecasting, index prediction, as well as portfolio management. We also provide an infographic overview of fusion in stock market prediction and extend our survey for other finely addressed financial prediction problems. Based on our surveyed articles, we provide potential future directions and concluding remarks on the significance of applying fusion in stock market.

Predictive intraday correlations in stable and volatile market environments: Evidence from deep learning

Standard methods and theories in finance can be ill-equipped to capture highly non-linear interactions in financial prediction problems based on large-scale datasets, with deep learning offering a way to gain insights into correlations in markets as complex systems. In this paper, we apply deep learning to econometrically constructed gradients to learn and exploit lagged correlations among S&P 500 stocks to compare model behaviour in stable and volatile market environments, and under the exclusion of target stock information for predictions. In order to measure the effect of time horizons, we predict intraday and daily stock price movements in varying interval lengths and gauge the complexity of the problem at hand with a modification of our model architecture. Our findings show that accuracies, while remaining significant and demonstrating the exploitability of lagged correlations in stock markets, decrease with shorter prediction horizons. We discuss implications for modern finance theory and our work’s applicability as an investigative tool for portfolio managers. Lastly, we show that our model’s performance is consistent in volatile markets by exposing it to the environment of the recent financial crisis of 2007/2008.

Deep learning for finance: deep portfolios

We explore the use of deep learning hierarchical models for problems in financial prediction and classification. Financial prediction problems – such as those presented in designing and pricing securities, constructing portfolios, and risk management – often involve large data sets with complex data interactions that currently are difficult or impossible to specify in a full economic model. Applying deep learning methods to these problems can produce more useful results than standard methods in finance. In particular, deep learning can detect and exploit interactions in the data that are, at least currently, invisible to any existing financial economic theory. Copyright © 2016 John Wiley & Sons, Ltd.

Deep Learning for Finance: Deep Portfolios

We explore the use of deep learning hierarchical models for problems in financial prediction and classification. Financial prediction problems – such as those presented in designing and pricing securities, constructing portfolios, and risk management – often involve large data sets with complex data interactions that currently are difficult or impossible to specify in a full economic model. Applying deep learning methods to these problems can produce more useful results than standard methods in finance. In particular, deep learning can detect and exploit interactions in the data that are, at least currently, invisible to any existing financial economic theory.

Evaluation of co-evolutionary neural network architectures for time series prediction with mobile application in finance

The fusion of soft computing methods such as neural networks and evolutionary algorithms have given a very promising performance for time series prediction problems. In order to fully harness their strengths for wider impact, effective real-world implementation of prediction systems must incorporate the use of innovative technologies such as mobile computing. Recently, co-evolutionary algorithms have shown to be very promising for training neural networks for time series prediction. Cooperative coevolution decomposes a problem into subcomponents that are evolved in isolation and cooperation typically involves fitness evaluation. The challenge has been in developing effective subcomponent decomposition methods for different neural network architectures. In this paper, we evaluate the performance of two problem decomposition methods for training feedforward and recurrent neural networks for chaotic time series problems. We further apply them for financial prediction problems selected from the NASDAQ stock exchange. We highlight the challenges in real-time implementation and present a mobile application framework for financial time series prediction. The results, in general, show that recurrent neural networks have better generalisation ability when compared to feedforward networks for real-world time series problems.

FINDING PROBLEM SPECIFIC SHANNON INFORMATION IN HIGH DIMENSIONAL INPUT SPACE FOR ARTIFICIAL NEURAL NETWORKS

Artificial neural networks, due to their ability to find the underlying model even in complex highly nonlinear and highly coupled problems, have found significant use as prediction engines in many domains. However, in problems where the input space is of high dimensionality, there is the unsolved problem of reducing dimensionality in some optimal way such that Shannon information important to the prediction is preserved. The important Shannon information may be a subset of total information with an unknown partition, unknown coupling and linear or nonlinear in nature. Solving this problem is an important step in classes of machine learning problems and many data mining applications. This paper describes a semi-automatic algorithm that was developed over a 5-year period while solving problems with increasing dimensionality and difficulty in (a) flow prediction for a magnetically levitated artificial heart (13 dimensions), (b) simultaneous chemical identification/concentration in gas chromatography (22 detection dimensions with wavelet compressed time series of 180,000 points), and finally in (c) financial analytics portfolio prediction in credit card and sub-prime debt problems (80 to 300 dimensions of sparse data with a portfolio value of approximately US$300,000,000.00). The algorithm develops a map of input space combinations and their importance to the prediction. This information is used directly to construct the optimal neural network topology for a given error performance. Importantly, the algorithm also produces information that shows whether the space between input nodes is linear or nonlinear; an important parameter in determining the number of training points required in the reduced dimensionality of the training set. Software was developed in the MatLAB environment using the Artificial Neural Network Toolbox, Parallel and Distributed Computing toolboxes, and runs on Windows or Linux based supercomputers. Trained neural networks can be compiled and linked to server applications and run on normal servers or clusters for transaction or web based processing. In this paper, application of the algorithm to two separate financial analytics prediction problems with large dimensionality and sparse data sets are shown. The algorithm is an important development in machine learning for an important class of problems in prediction, clustering, image analysis, and data mining. In the first example application for subprime debt portfolio analysis, performance of the neural network provided a 98.4% prediction rate, compared to 33% rate using traditional linear methods. In the second example application regarding credit card debt, performance of the algorithm provided a 95% accurate prediction (in terms of match rate), and is 10% better than other methods we have compared against, primarily logistic regression.

Forecasting Business Efficiency by Using Classification Techniques: A Comparative Analysis Based on a Spanish Case

A comparative, simulation-based study of the performance of a number of classificatory devices, both parametric (LDA and Logit) and non-parametric (perceptron neural nets and fuzzy-rule-based classifiers) is conducted. The paper uses as a benchmark the problem of forecasting the level of efficiency of Spanish commercial and industrial companies upon the basis of a set of financial ratios. This case illustrates well some distinctive features/complexities of many financial prediction problems, namely that of being characterized by a high dimension feature space and low degree of separability. Results indicate a higher performance of model-free classifiers, even for moderate sample sizes. The average effects of sample size variations on the predictive of each classifier are also measured by using Monte Carlo simulations, and response surfaces are estimated in order to summarize these effects.

A constraint-guided method with evolutionary algorithms for economic problems

This paper presents an evolutionary algorithms based constrain-guided method (CGM) that is capable of handling both hard and soft constraints in optimization problems. While searching for constraint-satisfied solutions, the method differentiates candidate solutions by assigning them with different fitness values, enabling favorite solutions to be distinguished more likely and more effectively from unfavored ones. We illustrate the use of CGM in solving two economic problems with optimization involved: (1) searching equilibriums for bargaining problems; (2) reducing the rate of failure in financial prediction problems. The efficacy of the proposed CGM is analyzed and compared with some other computational techniques, including a repair method and a penalty method for the problem (1), a linear classifier and three neural networks for the problem (2), respectively. Our studies here suggest that the evolutionary algorithms based CGM compares favorably against those computational approaches.

Forecasting business profitability by using classification techniques: A comparative analysis based on a Spanish case

A comparative study of the performance of a number of classificatory devices, both parametric (LDA and Logit) and non-parametric (perceptron neural nets and fuzzy-rule-based classifiers) is conducted, and a Monte Carlo simulation-based approach is used in order to measure the average effects of sample size variations on the predictive performance of each classifier. The paper uses as a benchmark the problem of forecasting the level of profitability of Spanish commercial and industrial companies upon the basis of a set of financial ratios. This case illustrates well a distinctive feature of many financial prediction problems, namely that of being characterized by a high dimension feature space as well as a low degree of separability. Response surfaces are estimated in order to summarize the results. A higher performance of model-free classifiers is generally observed, even for fairly moderate sample sizes.

Forecasting business profitability by using classification techniques: A comparative analysis based on a Spanish case

A comparative study of the performance of a number of classificatory devices, both parametric (LDA and Logit) and non-parametric (perceptron neural nets and fuzzy-rule-based classifiers) is conducted, and a Monte Carlo simulation-based approach is used in order to measure the average effects of sample size variations on the predictive performance of each classifier. The paper uses as a benchmark the problem of forecasting the level of profitability of Spanish commercial and industrial companies upon the basis of a set of financial ratios. This case illustrates well a distinctive feature of many financial prediction problems, namely that of being characterized by a high dimension feature space as well as a low degree of separability. Response surfaces are estimated in order to summarize the results. A higher performance of model-free classifiers is generally observed, even for fairly moderate sample sizes.

Discovering Interesting Patterns for Investment Decision Making with GLOWER x-A Genetic Learner Overlaid with Entropy Reduction

Prediction in financial domains is notoriously difficult for a number of reasons. First, theories tend to be weak or non-existent, which makes problem formulation open ended by forcing us to consider a large number of independent variables and thereby increasing the dimensionality of the search space. Second, the weak relationships among variables tend to be nonlinear, and may hold only in limited areas of the search space. Third, in financial practice, where analysts conduct extensive manual analysis of historically well performing indicators, a key is to find the hidden interactions among variables that perform well in combination. Unfortunately, these are exactly the patterns that the greedy search biases incorporated by many standard rule learning algorithms will miss. In this paper, we describe and evaluate several variations of a new genetic learning algorithm (GLOWER) on a variety of data sets. The design of GLOWER has been motivated by financial prediction problems, but incorporates successful ideas from tree induction and rule learning. We examine the performance of several GLOWER variants on two UCI data sets as well as on a standard financial prediction problem (S&P500 stock returns), using the results to identify one of the better variants for further comparisons. We introduce a new (to KDD) financial prediction problem (predicting positive and negative earnings surprises), and experiment with GLOWER, contrasting it with tree- and rule-induction approaches. Our results are encouraging, showing that GLOWER has the ability to uncover effective patterns for difficult problems that have weak structure and significant nonlinearities.