Popular Performance Metrics Research Articles

Stock Market Price Prediction Using Machine Learning Models

Stock forecasting has historically been a popular and lucrative field of study. It has been demonstrated that machine learning applications improve accuracy and return in the area of finance forecasting and prediction. This study chose data from the Yahoo Finance database that represented Apple's (AAPL) close price for research. This study categorized articles using a series of machine learning models, encompassing Linear Regression, Random Forest and so on. This paper also examines each article's dataset, variable, model, and findings. The survey in use showcases the findings using the most popular performance metrics. Recent models that combine LSTM with other techniques, For instance, RF has received a lot of study. Deep learning techniques like reinforcement learning and others produced excellent results. In conclusion, the use of deep learning-based techniques for financial modeling has become growing in popularity over the past few years.

Spatial Prediction of Fluvial Flood in High-Frequency Tropical Cyclone Area Using TensorFlow 1D-Convolution Neural Networks and Geospatial Data

Fluvial floods endure as one of the most catastrophic weather-induced disasters worldwide, leading to numerous fatalities each year and significantly impacting socio-economic development and the environment. Hence, the research and development of new methods and algorithms focused on improving fluvial flood prediction and devising robust flood management strategies are essential. This study explores and assesses the potential application of 1D-Convolution Neural Networks (1D-CNN) for spatial prediction of fluvial flood in the Quang Nam province, a high-frequency tropical cyclone area in central Vietnam. To this end, a geospatial database with 4156 fluvial flood locations and 12 flood indicators was considered. The ADAM algorithm and the MSE loss function were used to train the 1D-CNN model, whereas popular performance metrics, such as Accuracy (Acc), Kappa, and AUC, were used to measure the performance. The results indicated remarkable performance by the 1D-CNN model, achieving high prediction accuracy with metrics such as Acc = 90.7%, Kappa = 0.814, and AUC = 0.963. Notably, the proposed 1D-CNN model outperformed benchmark models, including DeepNN, SVM, and LR. This achievement underscores the promise and innovation brought by 1D-CNN in the realm of susceptibility mapping for fluvial floods.

Forecasting Sugar Price Fluctuation In Malaysia Using Geometric Brownian Motion Modelling

A mathematical model known as Geometric Brownian motion has proven to be an effective tool that can be deployed to forecast the price of goods in the future due to the presence of random terms, which represent the stochastic or random fluctuation of prices over a given period of time. The success of this model revolves around the estimation of its governing parameters. To efficiently predict the price of goods, using the Geometric Brownian Motion model (GBM), one needs to determine the value of returns and use the calculated returns to estimate the value of drift as well as volatility. This research used the value of volatility and drift terms obtained from real data of sugar in the months under review. The method has shown to be very reliable in capturing the intelligent trend in the price of sugar in Malaysia. The model was able to stimulate the pattern of the predicted price that shares a great resemblance to the actual price of sugar. The result obtained is very encouraging and places this study with a good note now that the country is just returning to normalcy from the pandemic that has crippled the economies of most developing nations in the world. We used the model to predict the price of sugar for a period of 20 months and the result of our prediction as well as the graph of the predicted prices confirmed the practicability of the model. The obtained values of MAPE and MSE, two of the popular performance metrics, also justified the effectiveness of the GBM in capturing the trend in the data. This model can be classified as a good model that can be deployed to forecast the price of goods that exhibit high volatility.

Evaluating classifier performance with highly imbalanced Big Data

Using the wrong metrics to gauge classification of highly imbalanced Big Data may hide important information in experimental results. However, we find that analysis of metrics for performance evaluation and what they can hide or reveal is rarely covered in related works. Therefore, we address that gap by analyzing multiple popular performance metrics on three Big Data classification tasks. To the best of our knowledge, we are the first to utilize three new Medicare insurance claims datasets which became publicly available in 2021. These datasets are all highly imbalanced. Furthermore, the datasets are comprised of completely different data. We evaluate the performance of five ensemble learners in the Machine Learning task of Medicare fraud detection. Random Undersampling (RUS) is applied to induce five class ratios. The classifiers are evaluated with both the Area Under the Receiver Operating Characteristic Curve (AUC), and Area Under the Precision Recall Curve (AUPRC) metrics. We show that AUPRC provides a better insight into classification performance. Our findings reveal that the AUC metric hides the performance impact of RUS. However, classification results in terms of AUPRC show RUS has a detrimental effect. We show that, for highly imbalanced Big Data, the AUC metric fails to capture information about precision scores and false positive counts that the AUPRC metric reveals. Our contribution is to show AUPRC is a more effective metric for evaluating the performance of classifiers when working with highly imbalanced Big Data.

Trust-based co-operative routing for secure communication in mobile ad hoc networks

The working of a Mobile Ad hoc NETwork (MANET) relies on the supportive cooperation among the network nodes. But due to its intrinsic features, a misbehaving node can easily lead to a routing disorder. This paper presents two trust-based routing schemes, namely Trust-based Self-Detection Routing (TSDR) and Trust-based Co-operative Routing (TCOR) designed with an Ad hoc On-demand Distance Vector (AODV) protocol. The proposed work covers a wide range of security challenges, including malicious node identification and prevention, accurate trust quantification, secure trust data sharing, and trusted route maintenance. This brings a prominent solution for mitigating misbehaving nodes and establishing efficient communication in MANET. It is empirically validated based on a performance comparison with the current Evolutionary Self-Cooperative Trust (ESCT) scheme, Generalized Trust Model (GTM), and the conventional AODV protocol. The extensive simulations are conducted against three different varying network scenarios. The results affirm the improved values of eight popular performance metrics overcoming the existing routing schemes. Among the two proposed works, TCOR is more suitable for highly scalable networks; TSDR suits, however, the MANET application better with its small size. This work thus makes a significant contribution to the research community, in contrast to many previous works focusing solely on specific security aspects, and results in a trade-off in the expected values of evaluation parameters and asserts their efficiency.

Deep learning methods may not outperform other machine learning methods on analyzing genomic studies.

Deep Learning (DL) has been broadly applied to solve big data problems in biomedical fields, which is most successful in image processing. Recently, many DL methods have been applied to analyze genomic studies. However, genomic data usually has too small a sample size to fit a complex network. They do not have common structural patterns like images to utilize pre-trained networks or take advantage of convolution layers. The concern of overusing DL methods motivates us to evaluate DL methods’ performance versus popular non-deep Machine Learning (ML) methods for analyzing genomic data with a wide range of sample sizes. In this paper, we conduct a benchmark study using the UK Biobank data and its many random subsets with different sample sizes. The original UK Biobank data has about 500k participants. Each patient has comprehensive patient characteristics, disease histories, and genomic information, i.e., the genotypes of millions of Single-Nucleotide Polymorphism (SNPs). We are interested in predicting the risk of three lung diseases: asthma, COPD, and lung cancer. There are 205,238 participants have recorded disease outcomes for these three diseases. Five prediction models are investigated in this benchmark study, including three non-deep machine learning methods (Elastic Net, XGBoost, and SVM) and two deep learning methods (DNN and LSTM). Besides the most popular performance metrics, such as the F1-score, we promote the hit curve, a visual tool to describe the performance of predicting rare events. We discovered that DL methods frequently fail to outperform non-deep ML in analyzing genomic data, even in large datasets with over 200k samples. The experiment results suggest not overusing DL methods in genomic studies, even with biobank-level sample sizes. The performance differences between DL and non-deep ML decrease as the sample size of data increases. This suggests when the sample size of data is significant, further increasing sample sizes leads to more performance gain in DL methods. Hence, DL methods could be better if we analyze genomic data bigger than this study.

Bi-Objective Colored Traveling Salesman Problems

As a generalization of the well-known multiple traveling salesman problem, a colored traveling salesman problem (CTSP) utilizes colors to describe the accessibility of individual cities to salesmen. To expand its application scope, this work presents a bi-objective CTSP (BCTSP) over hypergraphs, taking into account the balance of workload among salesmen. To solve it, a bi-objective variable neighborhood search (BVNS) is proposed as a solution framework. In BVNS, we exploit a two-stage initialization and a probability-based insertion to produce feasible solutions and generate their neighborhood. Next, population-based multi-insertion, color-preserving exchange and 2-opt constitute a powerful local search procedure, where color-preserving exchange improves the solutions by modifying the route intersections among distinct salesmen. Besides, Delaunay triangulation is utilized to prepare candidates for multi-insertion, thereby increasing the possibility to find an optimal route by shortening links among vertices. Extensive experiments are conducted on 20 cases. To make a comprehensive comparison, four existing methods, i.e., two genetic algorithms and two variable neighborhood search methods, are adapted by exploiting an elitist non-dominated sorting for solving BCTSP instances. The experimental results show that BVNS is superior to its peers in achieving Pareto optima in terms of three popular performance metrics, i.e., hypervolume, inverted generational distance, and C-metric. In addition, the study of four BVNS variants reveals that probability-based insertion, population-based multi-insertion, and color-preserving exchange play significant roles in BVNS’s high performance.

Distributed Denial-of-Service (DDoS) Attacks and Defense Mechanisms in Various Web-Enabled Computing Platforms

The demand for Internet security has escalated in the last two decades because the rapid proliferation in the number of Internet users has presented attackers with new detrimental opportunities. One of the simple yet powerful attack, lurking around the Internet today, is the Distributed Denial-of-Service (DDoS) attack. The expeditious surge in the collaborative environments, like IoT, cloud computing and SDN, have provided attackers with countless new avenues to benefit from the distributed nature of DDoS attacks. The attackers protect their anonymity by infecting distributed devices and utilizing them to create a bot army to constitute a large-scale attack. Thus, the development of an effective as well as efficient DDoS defense mechanism becomes an immediate goal. In this exposition, we present a DDoS threat analysis along with a few novel ground-breaking defense mechanisms proposed by various researchers for numerous domains. Further, we talk about popular performance metrics that evaluate the defense schemes. In the end, we list prevalent DDoS attack tools and open challenges.

Machine learning models predicting returns: Why most popular performance metrics are misleading and proposal for an efficient metric

Numerous machine learning models have been developed to achieve the ‘real-life’ financial objective of optimising the risk/return profile of investment strategies. In the current article: (a) we present and classify the most popular performance metrics used in 190 articles analysed. We noticed that, in most articles, no attention is devoted to the criteria used to compare the algorithms. (b) We evaluate the ability of the metrics used in the literature to assess the efficiency of algorithms to improve investments results. We demonstrate that many of the most popular metrics, like mean squared error (MSE) or root mean squared error (RMSE), are inappropriate for this purpose while others, like accuracy or F1, are just weak. We explain why risk-adjusted return-based metrics are best-in-class, although they suffer from statistical limitations and do not allow easy comparison of algorithms across assets or over time. (c) We propose a new discriminant metric that measures the efficiency of AI models to optimize the risk-adjusted return, which is statistically more robust, and which can test the effectiveness and the stability of models over time and across assets.

Measurement Invariance Investigation for Performance of Deep Learning Architectures

Models can be compared against each other. Such comparison helps to find strengths and weaknesses of different approaches and gives additional possibilities for model validation. Multi-class classification is a classification task where each image is assigned to one and only one label. Confusion matrix, Precision, Recall, and F1 Score are popular performance metrics. The common sense is that the performance of any architecture is dependent on the sizes of data set. The goodness of the architecture of deep learning models for different data sets is critical issue. This paper implements Pearson correlation coefficient and the multivariate linear regression method to assess the accuracy of deep learning architectures with five performance indicators. The five performance indicators are training loss rate, robustness, training time, number of model parameters and computation complexity. There are five image datasets used to test four deep learning models: Alexnet, GoogLeNet, ResNet, MobileNet to obtain the values of accuracy and other five indicators. The most important contribution of the article is to show that the accuracy indicator related to training loss rate and training time indicators are not dependent on the selection of the data group. According to the definition of Measurement Invariance (MI), the measurement invariance is demonstrated by the linear regression analysis and inner product of the unit normal vector of the linear regression planes.

The Abuse of Popular Performance Metrics in Hydrologic Modeling

AbstractThe goal of this commentary is to critically evaluate the use of popular performance metrics in hydrologic modeling. We focus on the Nash‐Sutcliffe Efficiency (NSE) and the Kling‐Gupta Efficiency (KGE) metrics, which are both widely used in hydrologic research and practice around the world. Our specific objectives are: (a) to provide tools that quantify the sampling uncertainty in popular performance metrics; (b) to quantify sampling uncertainty in popular performance metrics across a large sample of catchments; and (c) to prescribe the further research that is, needed to improve the estimation, interpretation, and use of popular performance metrics in hydrologic modeling. Our large‐sample analysis demonstrates that there is substantial sampling uncertainty in the NSE and KGE estimators. This occurs because the probability distribution of squared errors between model simulations and observations has heavy tails, meaning that performance metrics can be heavily influenced by just a few data points. Our results highlight obvious (yet ignored) abuses of performance metrics that contaminate the conclusions of many hydrologic modeling studies: It is essential to quantify the sampling uncertainty in performance metrics when justifying the use of a model for a specific purpose and when comparing the performance of competing models.

An Ensemble-Based Scalable Approach for Intrusion Detection Using Big Data Framework.

In this study, we set up a scalable framework for large-scale data processing and analytics using the big data framework. The popular classification methods are implemented, tuned, and evaluated by using intrusion datasets. The objective is to select the best classifier after optimizing the hyper-parameters. We observed that the decision tree (DT) approach outperforms compared with other methods in terms of classification accuracy, fast training time, and improved average prediction rate. Therefore, it is selected as a base classifier in our proposed ensemble approach to study class imbalance. As the intrusion datasets are imbalanced, most of the classification techniques are biased toward the majority class. The misclassification rate is more in the case of the minority class. An ensemble-based method is proposed by using K-Means, RUSBoost, and DT approaches to mitigate the class imbalance problem; empirically investigate the impact of class imbalance on classification approaches' performance; and compare the result by using popular performance metrics such as Balanced Accuracy, Matthews Correlation Coefficient, and F-Measure, which are more suitable for the assessment of imbalanced datasets.

Machine learning models predicting returns: why most popular performance metrics are misleading and proposal for an efficient metric

Numerous machine learning models have been developed to achieve the ‘real-life’ financial objective of optimising the risk/return profile of investment strategies. In the current article: (a) we present and classify the most popular performance metrics used in 190 articles analysed. We noticed that, in most articles, no attention is devoted to the criteria used to compare the algorithms. (b) We evaluate the ability of the metrics used in the literature to assess the efficiency of algorithms to improve investments results. We demonstrate that many of the most popular metrics, like mean squared error (MSE) or root mean squared error (RMSE), are inappropriate for this purpose while others, like accuracy or F1, are just weak. We explain why risk-adjusted return-based metrics are best-in-class, although they suffer from statistical limitations and do not allow easy comparison of algorithms across assets or over time. (c) We propose a new discriminant metric that measures the efficiency of AI models to optimize the risk-adjusted return, which is statistically more robust, and which can test the stability of models over time and across assets.

Empowered AODV Protocol in Wireless Sensor Network using Three Variable RSA Cryptosystem

Background: The Wireless Sensor Network (WSN) is a type of networks which primarily designed for the purpose of monitoring in remote areas. It consists of communicating nodes (called sensor's) which communicate each other to share their data and passing the information to the central node. In many applications like defence requires the secure communication of information. However, due to the numerous characteristics of WSN such as open shared communication channel, limited memory, and processing power of sensors, etc. these networks are vulnerable to various attacks such as black hole, gray hole, etc. Objective: The objective of the paper is to secure the AODV routing protocol in WSN using cryptography techniques. Methods: In this paper, the Ad hoc On-demand Distance Vector (AODV) routing protocol has been chosen for information routing because of their lightweight processing capability. To provide secure communication in WSN, the AODV routing protocol is secured by utilizing the RSA key generation algorithm. Here, RSA with three variables (three prime numbers) is employed instead of two variables. Results: The effectiveness of the proposed approach in handling black hole attack is being verified through the simulation results obtained from the experiments conducted using Network Simulator tool (NS2). The three popular performance metrics namely Average End-to-End Delay, Packet Delivery Ratio, and Average Throughput are used for evaluation purpose. These results are observed under different pose time and varying number of malicious nodes. Conclusion: In this paper, a new three variable RSA cryptosystem-based security model is proposed to protect the communication against the Black Hole (BH) attack in wireless sensor networks. The use of three variables instead of two variables allows our model to provide more security as compared to other methods. Simulation results obtained from the experiments carried out using NS2 tool evident the performance of the proposed model over original AODV and other previous models.

Scholarly publication venue recommender systems

PurposeThe purpose of this investigation is to identify, evaluate, integrate and summarize relevant and qualified papers through conducting a systematic literature review (SLR) on the application of recommender systems (RSs) to suggest a scholarly publication venue for researcher's paper.Design/methodology/approachTo identify the relevant papers published up to August 11, 2018, an SLR study on four databases (Scopus, Web of Science, IEEE Xplore and ScienceDirect) was conducted. We pursued the guidelines presented by Kitchenham and Charters (2007) for performing SLRs in software engineering. The papers were analyzed based on data sources, RSs classes, techniques/methods/algorithms, datasets, evaluation methodologies and metrics, as well as future directions.FindingsA total of 32 papers were identified. The most data sources exploited in these papers were textual (title/abstract/keywords) and co-authorship data. The RS classes in the selected papers were almost equally used. DBLP was the main dataset utilized. Cosine similarity, social network analysis (SNA) and term frequency–inverse document frequency (TF–IDF) algorithm were frequently used. In terms of evaluation methodologies, 24 papers applied only offline evaluations. Furthermore, precision, accuracy and recall metrics were the popular performance metrics. In the reviewed papers, “use more datasets” and “new algorithms” were frequently mentioned in the future work part as well as conclusions.Originality/valueGiven that a review study has not been conducted in this area, this paper can provide an insight into the current status in this area and may also contribute to future research in this field.

Energy Efficient Task Scheduling for Real-Time Embedded Systems in a Fuzzy Uncertain Environment

During the designing phase of real-time embedded systems (RTESs), available information on task characteristics is either incomplete or imprecise. So, task timing constraints are mostly approximated estimations by designers. This indicates that there are underlying uncertainties in these timing constraints, which can be appropriately modeled using fuzzy numbers. Moreover, feasible scheduling of tasks and energy efficiency are two essential requirements for better utilization and durability of RTESs. Thus, justifiable performance of this kind of systems warrants energy savings amid timely production of the computational outputs, although these two issues are mutually contradictory. This article reports a novel formulation of the energy efficient real-time scheduling problem in a fuzzy uncertain environment and proposes a novel solution approach called “ε-constraint coupled energy efficient genetic algorithm (ε-EEGA).” The working of the proposed approach is demonstrated taking a real-life example. Also, a thorough comparative analysis is provided considering well-known existing approaches including multiobjective evolutionary algorithms. Results are compared using popular performance metrics, which suggests that the proposed ε-EEGA is efficient in giving better energy savings with faster computations than its existing counterparts. Standard statistical tests such as analysis of variance and Kruskal-Wallis are performed.

A Question-Based Approach to the Design of a Successful “Finance for Non-Financial Managers” Executive Education Program

Many successful non-financial managers aspire to contribute at the larger table of management decision making. To do so necessitates broadening their skills to include financial acumen. For non-financial managers, learning new financial constructs can be daunting, and knowing when to use which tool is challenging. We describe a three-questions-based approach underlying the design and delivery of our successful one-week “Financial Management for Non-Financial Executives” program at the University of Virginia’s Darden School of Business. We use a three-questions-based approach to facilitate the learning process in each of the following four financial arenas that comprise the overarching, larger financial acumen agenda.
 
 
 Modeling the financial effects associated with typical internal operating decision alternatives
 Assessing the impact of operating decisions on the financial statements produced for external constituents
 Assessing the impact of operating decisions on popular financial performance metrics used to compare and contrast companies
 Recognizing and incorporating the basic tax implications applicable to internal operating decision alternatives
 
 
 For each of these four financial arenas, we outline three key questions tailored for each, using one comprehensive example to illustrate the application of our questions-based approach.

On evaluating the resource usage effectiveness of multi-tenant cloud storage

Abstract As multi-tenant cloud storage services are becoming increasingly popular in commercial clouds, there is a growing need for evaluating their resource usage effectiveness. Although tail latency and resource utilization are the most important and popular performance metrics, it is biased to use only one of them on evaluating the resource usage effectiveness. In this paper, we present a unified framework, Stochastic Model-based Effectiveness Analyzer (SMEA), to evaluate the resource usage effectiveness of multi-tenant cloud storage systems. Instead of evaluating tail latency and resource utilization separately, we propose a new metric called resource-productivity to combine both performance. Since evaluating tail latency or resource utilization solely is challenging due to contention, queueing and burstiness of workloads, a Markov-modulated Poisson process (MMPP) is adopted in SMEA to properly characterize the behavior of workloads. Moreover, the Stochastic Network Calculus (SNC)-based network latency analysis model is extended to evaluate the end-to-end tail latency over all system components, with the principle “treating the computer as a network”. After obtaining the analyzed tail latency and resource utilization, we derive the resource-productivity mathematically. We have implemented SMEA on a small-scale testbed. Extensive trace-driven experiments demonstrate that SMEA is efficient in evaluating the resource usage effectiveness of the multi-tenant cloud storage system with relative error less than 13%.

An intelligent hybridization of ARIMA with machine learning models for time series forecasting

The development of accurate forecasting systems can be challenging in real-world applications. The modeling of real-world time series is a particularly difficult task because they are generally composed of linear and nonlinear patterns that are combined in some form. Several hybrid systems that combine linear and nonlinear techniques have obtained relevant results in terms of accuracy in comparison with single models. However, the best combination function of the forecasting of the linear and nonlinear patterns is unknown, which makes this modeling an open question. This work proposes a hybrid system that searches for a suitable function to combine the forecasts of linear and nonlinear models. Thus, the proposed system performs: (i) linear modeling of the time series; (ii) nonlinear modeling of the error series; and (iii) a data-driven combination that searches for: (iii.a) the most suitable function, between linear and nonlinear formalisms, and (iii.b) the number of forecasts of models (i) and (ii) that maximizes the performance of the combination. Two versions of the hybrid system are evaluated. In both versions, the ARIMA model is used in step (i) and two nonlinear intelligent models – Multi-Layer Perceptron (MLP) and Support Vector Regression (SVR) – are used in steps (ii) and (iii), alternately. Experimental simulations with six real-world complex time series that are well-known in the literature are evaluated using a set of popular performance metrics. Our results show that the proposed hybrid system attains superior performance when compared with single and hybrid models in the literature.

Adaptive multi-threshold based de-noising filter for medical image applications

Medical image processing is the emerging research area and many researchers contributed to medical image processing by proposing new techniques for medical image enhancement and abnormality detection. Interpretation of medical images is a challenging problem because of the unavoidable noise produced by the medical imaging devices and interference. In this work, a new framework is proposed for noise detection and reduction. This framework comprises two phases. First phase is the noise detection phase which is performed using the newly proposed adaptive multi-threshold scheme (AMT). In second phase, modification of noisy pixel is done using edge preserving median filter (EPM), which conserves the edge component and controls the blurring effect with preservation of fine details of interior region. The proposed work is tested with benchmark images and few medical images. It produces promising result and the results are compared with existing two-stage noise reduction techniques. Popular performance metrics such PSNR and SSIM are used for evaluation. Quantitative analysis and experimental results demonstrate that the proposed method is more efficient and suitable for medical image pre-processing.