Popular Metrics Research Articles

PyNeval: A Python Toolbox for Evaluating Neuron Reconstruction Performance.

Quality assessment of tree-like structures obtained from a neuron reconstruction algorithm is necessary for evaluating the performance of the algorithm. The lack of user-friendly software for calculating common metrics motivated us to develop a Python toolbox called PyNeval, which is the first open-source toolbox designed to evaluate reconstruction results conveniently as far as we know. The toolbox supports popular metrics in two major categories, geometrical metrics and topological metrics, with an easy way to configure custom parameters for each metric. We tested the toolbox on both synthetic data and real data to show its reliability and robustness. As a demonstration of the toolbox in real applications, we used the toolbox to improve the performance of a tracing algorithm successfully by integrating it into an optimization procedure.

Predictive performance of ensemble hydroclimatic forecasts: Verification metrics, diagnostic plots and forecast attributes

AbstractPredictive performance is one of the most important issues for practical applications of ensemble hydroclimatic forecasts. While different forecasting studies tend to use different combinations of verification metrics and diagnostic plots, this paper presents a literature survey of popular verification metrics, diagnostic plots and their corresponding forecast attributes. Twenty metrics to quantify predictive performance of ensemble, deterministic and categorical forecasts are detailed; six types of diagnostic plots to visualize the relationship between ensemble forecasts and observations are presented; and the correspondence relationships of verification metrics and diagnostic plots with ten forecast attributes are illustrated. Numerical experiments are devised for raw ensemble precipitation forecasts generated by the Climate Forecast System version 2 (CFSv2) to illustrate the verification metrics, diagnostic plots and forecast attributes. The results suggest that verification metrics that pay attention to some similar attributes can be related to one another. Meanwhile, the metrics are not necessarily in full accordance as they tend to reflect different attributes. Time series plots visualize forecasts over a period of interest and graphically illustrate forecast bias, reliability and skill. Quantile range plots provide an overall diagnosis for the association of ensemble forecasts with observations. Overall, verification metrics can be illustrated by diagnostic plots for individual grid cells and also be screened by spatial plots at regional and global scales. The survey presented in this article can serve as a roadmap for the selection of verification metrics, diagnostic plots and forecast attributes for verifying ensemble hydroclimatic forecasts.This article is categorized under: Science of Water > Hydrological Processes

FakeNewsIndia: A benchmark dataset of fake news incidents in India, collection methodology and impact assessment in social media

Online Social Media platforms (OSMs) have become an essential source of information. The high speed at which OSM users submit data makes moderation extremely hard. Consequently, besides offering online networking to users, the OSMs have also become carriers for spreading fake news. Knowingly or unknowingly, users circulate fake news on OSMs, adversely affecting an individual’s offline activity. To counter fake news, several dedicated websites (referred to as fact-checkers) have sprung up whose sole purpose is to identify and report fake news incidents. There are well-known datasets of fake news; however, not much work has been done regarding credible datasets of fake news in India. Therefore, we design an automated data collection pipeline to collect fake incidents reported by fact-checkers in this work. We gather 4,803 fake news incidents from June 2016 to December 2019 reported by six popular fact-checking websites in India and make this dataset (FakeNewsIndia) available to the research community. We find 5,031 tweets on Twitter and 866 videos on YouTube mentioned in these 4,803 fake news incidents. Further, we evaluate the impact of fake new incidents on the two prominent OSM platforms, namely, Twitter and YouTube. We use popularity metrics based on engagement rate and likes ratio to measure impact and categorize impact into three levels — low, medium, and high. Our learning models use features extracted from text, images, and videos present in the fake news incident articles written by fact-checking websites. Experiments show that we can predict the impact (popularity) of videos (appearing on fake news incident articles) on YouTube more accurately (with baseline accuracy ranging from 86% to 92%) as compared to the impact (popularity) of tweets on Twitter (with baseline accuracy of 37% to 41%). We need to build more intelligent models that predict tweets’ impact, appearing in fact-checking incident articles on Twitter as future work.

A Bias-Free Time-Aware PageRank Algorithm for Paper Ranking in Dynamic Citation Networks

The process of ranking scientific publications in dynamic citation networks plays a crucial rule in a variety of applications. Despite the availability of a number of ranking algorithms, most of them use common popularity metrics such as the citation count, h-index, and Impact Factor (IF). These adopted metrics cause a problem of bias in favor of older publications that took enough time to collect as many citations as possible. This paper focuses on solving the problem of bias by proposing a new ranking algorithm based on the PageRank (PR) algorithm; it is one of the main page ranking algorithms being widely used. The developed algorithm considers a newly suggested metric called the Citation Average rate of Change (CAC). Time information such as publication date and the citation occurrence’s time are used along with citation data to calculate the new metric. The proposed ranking algorithm was tested on a dataset of scientific papers in the field of medical physics published in the Dimensions database from years 2005 to 2017. The experimental results have shown that the proposed ranking algorithm outperforms the PageRank algorithm in ranking scientific publications where 26 papers instead of only 14 were ranked among the top 100 papers of this dataset. In addition, there were no radical changes or unreasonable jump in the ranking process, i.e., the correlation rate between the results of the proposed ranking method and the original PageRank algorithm was 92% based on the Spearman correlation coefficient.

Ограничения применимости дифференцируемых эталонных показателей качества изображений

Subjective image-quality measurement plays a critical role in the development of image-processing applications. The purpose of a visual-quality metric is to approximate the results of subjective assessment. In this regard, more and more metrics are under development, but little research has considered their limitations. This paper addresses that deficiency: we show how image preprocessing before compression can artificially increase the quality scores provided by the popular metrics DISTS, LPIPS, HaarPSI, and VIF as well as how these scores are inconsistent with subjective-quality scores. We propose a series of neural-network preprocessing models that increase DISTS by up to 34.5%, LPIPS by up to 36.8%, VIF by up to 98.0%, and HaarPSI by up to 22.6% in the case of JPEG-compressed images. A subjective comparison of preprocessed images showed that for most of the metrics we examined, visual quality drops or stays unchanged, limiting the applicability of these metrics.

SPICE: Semantic Pseudo-Labeling for Image Clustering.

The similarity among samples and the discrepancy among clusters are two crucial aspects of image clustering. However, current deep clustering methods suffer from inaccurate estimation of either feature similarity or semantic discrepancy. In this paper, we present a Semantic Pseudo-labeling-based Image ClustEring (SPICE) framework, which divides the clustering network into a feature model for measuring the instance-level similarity and a clustering head for identifying the cluster-level discrepancy. We design two semantics-aware pseudo-labeling algorithms, prototype pseudo-labeling and reliable pseudo-labeling, which enable accurate and reliable self-supervision over clustering. Without using any ground-truth label, we optimize the clustering network in three stages: 1) train the feature model through contrastive learning to measure the instance similarity; 2) train the clustering head with the prototype pseudo-labeling algorithm to identify cluster semantics; and 3) jointly train the feature model and clustering head with the reliable pseudo-labeling algorithm to improve the clustering performance. Extensive experimental results demonstrate that SPICE achieves significant improvements (~10%) over existing methods and establishes the new state-of-the-art clustering results on six balanced benchmark datasets in terms of three popular metrics. Importantly, SPICE significantly reduces the gap between unsupervised and fully-supervised classification; e.g. there is only 2% (91.8% vs 93.8%) accuracy difference on CIFAR-10. Our code is made publicly available at https://github.com/niuchuangnn/SPICE.

A multi-channel convolutional neural network driven computer vision system towards identification of species and maturity stage of banana fruits: case studies with Martaman and Singapuri banana

Banana is a widely consumed fruit worldwide due to its high nutritional values. The nutritional impact of banana considerably varies with the species and ripening stages. The existing practices of both species and ripening stage detection of banana are performed majorly by human experience and chemical analysis. This paper presents a computer vision-based system development to identify banana species and corresponding ripening stages simultaneously. The proposed system is less expensive, non-invasive and faster with potential to provide up to 98% accuracy. The work utilises multi-channel convolutional neural network (CNN) architecture to classify the species and ripening stage in a single go. The results of the proposed system was assessed against different popular metrics and found competitive to the existing techniques. This technique can be also integrated to a handheld device or mobile app in future for firsthand assessment by the consumers and sellers in the market.

Measurement Invariance Investigation for Performance of Deep Learning Architectures

Models can be compared against each other. Such comparison helps to find strengths and weaknesses of different approaches and gives additional possibilities for model validation. Multi-class classification is a classification task where each image is assigned to one and only one label. Confusion matrix, Precision, Recall, and F1 Score are popular performance metrics. The common sense is that the performance of any architecture is dependent on the sizes of data set. The goodness of the architecture of deep learning models for different data sets is critical issue. This paper implements Pearson correlation coefficient and the multivariate linear regression method to assess the accuracy of deep learning architectures with five performance indicators. The five performance indicators are training loss rate, robustness, training time, number of model parameters and computation complexity. There are five image datasets used to test four deep learning models: Alexnet, GoogLeNet, ResNet, MobileNet to obtain the values of accuracy and other five indicators. The most important contribution of the article is to show that the accuracy indicator related to training loss rate and training time indicators are not dependent on the selection of the data group. According to the definition of Measurement Invariance (MI), the measurement invariance is demonstrated by the linear regression analysis and inner product of the unit normal vector of the linear regression planes.

A comprehensive understanding of popular machine translation evaluation metrics

A comprehensive understanding of popular machine translation evaluation metrics

Impact Of Load Balancing Interference Routing Metric On Multi Channel Multi Radio Wireless Mesh Network

Wireless Mesh Network (WMNs) is a promising technology that has been accepted favorably by the Internet Service Providers (ISPs). ISPs use it to provide last mile broadband Internet connectivity to the end users. This popularity arises from the fact, that it inter operates well with diverse wireless systems and provides robust fault tolerance with a high degree of redundancy and reliability. Routing metrics plays a crucial part in the performance of WMN. When routing protocols are implemented, the routing metric is assigned to different paths. It calculates the optimum path to predict the best routing path. These are integrated in routing protocols to improve WMNs efficiency in terms of reliability, latency, throughput, error rate and cost. This paper compares the impact of using load balancing interference aware routing metric (LBIARM) on the performance of Multi-Channel Multi Radio WMNs (MCMR WMNs). The performance of LBIARM was evaluated by comparing it with popular standard metrics weighted cumulative expected transmission time metric and Hop Count metric. OPNET Modeler 17.5 PL1 was used as a simulation tool for implementation. Simulation results show the effectiveness of using LBIARM metric on MCMR WMNs.

Automatic Math Word Problem Generation With Topic-Expression Co-Attention Mechanism and Reinforcement Learning

Taking several topic words and a math expression as input, the aim of math word problem generation is to generate a problem that can be answered by the given expression and related to these topic words. Considerable progress has been achieved by sequence-to-sequence neural network models in many natural language generation tasks, but these models do not effectively consider the characteristics of the math word problem generation task. They may generate problems that are unrelated to the topic words and expressions, and problems that cannot be solved. In this paper, we propose a new model, MWPGen, for automatically generating math word problems. MWPGen has a topic-expression co-attention mechanism to extract relevant information between topic words and expressions. Further, we fine-tune MWPGen with the solving result of the generated problem as the reward for reinforcement learning. MWPGen shows improved performance in popular automatic evaluation metrics and improves the solvability of generated problems.

Deep Learning-based Face Super-resolution: A Survey

Face super-resolution (FSR), also known as face hallucination, which is aimed at enhancing the resolution of low-resolution (LR) face images to generate high-resolution face images, is a domain-specific image super-resolution problem. Recently, FSR has received considerable attention and witnessed dazzling advances with the development of deep learning techniques. To date, few summaries of the studies on the deep learning-based FSR are available. In this survey, we present a comprehensive review of deep learning-based FSR methods in a systematic manner. First, we summarize the problem formulation of FSR and introduce popular assessment metrics and loss functions. Second, we elaborate on the facial characteristics and popular datasets used in FSR. Third, we roughly categorize existing methods according to the utilization of facial characteristics. In each category, we start with a general description of design principles, present an overview of representative approaches, and then discuss the pros and cons among them. Fourth, we evaluate the performance of some state-of-the-art methods. Fifth, joint FSR and other tasks, and FSR-related applications are roughly introduced. Finally, we envision the prospects of further technological advancement in this field.

Compressed Video Quality Index Based on Saliency-Aware Artifact Detection.

Video coding technology makes the required storage and transmission bandwidth of video services decrease by reducing the bitrate of the video stream. However, the compressed video signals may involve perceivable information loss, especially when the video is overcompressed. In such cases, the viewers can observe visually annoying artifacts, namely, Perceivable Encoding Artifacts (PEAs), which degrade their perceived video quality. To monitor and measure these PEAs (including blurring, blocking, ringing and color bleeding), we propose an objective video quality metric named Saliency-Aware Artifact Measurement (SAAM) without any reference information. The SAAM metric first introduces video saliency detection to extract interested regions and further splits these regions into a finite number of image patches. For each image patch, the data-driven model is utilized to evaluate intensities of PEAs. Finally, these intensities are fused into an overall metric using Support Vector Regression (SVR). In experiment section, we compared the SAAM metric with other popular video quality metrics on four publicly available databases: LIVE, CSIQ, IVP and FERIT-RTRK. The results reveal the promising quality prediction performance of the SAAM metric, which is superior to most of the popular compressed video quality evaluation models.

A new measure of financial texts’ readability

We define and construct the ‘semantic complexity index’ (SCI) which captures the marginal contribution of multi-clausal phrases (e.g., ‘slowdown in business activity’); and adjectives, adverbs and adversative conjunctions (e.g., ‘greater’, ‘slightly’, ‘however’) which alter the connotation of financial texts. More semantic complexity makes text difficult to interpret, which leads to poor readability, increased ambiguity and more investor uncertainty. We show that during 1994-2018, yearly SCIs of US firms' MD&A section display significantly positive association with firms' subsequent volatility and standardized unexpected earnings. We find that firms downplay negative information and exaggerate positive information, but markets react to such firms by associating higher subsequent volatility to them. We also show that the SEC Plain English Rule (October 1998) has reduced the semantic complexity of US firms' 10-Ks, and hence has improved readability observations at odds with other popular readability metrics.

CANet: A Combined Attention Network for Remote Sensing Image Change Detection

Change detection (CD) is one of the essential tasks in remote sensing image processing and analysis. Remote sensing CD is a process of determining and evaluating changes in various surface objects over time. The impressive achievements of deep learning in image processing and computer vision provide an innovative concept for the task of CD. However, existing methods based on deep learning still have problems detecting small changed regions correctly and distinguishing the boundaries of the changed regions. To solve the above shortcomings and improve the efficiency of CD networks, inspired by the fact that an attention mechanism can refine features effectively, we propose an attention-based network for remote sensing CD, which has two important components: an asymmetric convolution block (ACB) and a combined attention mechanism. First, the proposed method extracts the features of bi-temporal images, which contain two parallel encoders with shared weights and structures. Then, the feature maps are fed into the combined attention module to reconstruct the change maps and obtain refined feature maps. The proposed CANet is evaluated on the two publicly available datasets for challenging remote sensing image CD. Extensive empirical results with four popular metrics show that the designed framework yields a robust CD detector with good generalization performance. In the CDD and LEVIR-CD datasets, the F1 values of the CANet are 3.3% and 1.3% higher than those of advanced CD methods, respectively. A quantitative analysis and qualitative comparison indicate that our method outperforms competitive baselines in terms of both effectiveness and robustness.

Automatic glottis segmentation for laryngeal endoscopic images based on U-Net

The glottis’s morphology not only reflects vocal and respiratory information, but also plays an important role in the diagnosis of laryngeal diseases. The glottis segmentation is a primary step in computer-aided diagnostic system, however is challenging due to various shapes of glottis, low contrast with surrounding tissues, the existence of laryngeal diseases and so on. In this paper, a deep attention network based on U-Net with color normalization operation (CN-DA-Unet) is proposed to achieve an end-to-end segmentation of the glottal area for the first time. The original images are first processed by color normalization to reduce the adverse effects of low contrast and large differences in colors between different images. The normalized images are then sent to the proposed DA-Unet for feature extraction. In this network, residual structure is incorporated to extract rich features from deep neural networks. After extracting features, a feature pyramid attention (FPA) module is applied to enhance the semantic information of the glottal area. These features are up-sampled and added to the features from the corresponding encoding layer for several times to obtain the final segmented image. The proposed approach is tested on laryngeal images of an in–house dataset including images from healthy subjects and pathologic subjects. Its performance is evaluated by several reliable and popular evaluation metrics, achieving the dice coefficient of 92.9%, sensitivity of 93.5% and precision of 92.6%. These results demonstrate the effectiveness of our proposed approach and the better performance comparing with several popular networks.

The Abuse of Popular Performance Metrics in Hydrologic Modeling

AbstractThe goal of this commentary is to critically evaluate the use of popular performance metrics in hydrologic modeling. We focus on the Nash‐Sutcliffe Efficiency (NSE) and the Kling‐Gupta Efficiency (KGE) metrics, which are both widely used in hydrologic research and practice around the world. Our specific objectives are: (a) to provide tools that quantify the sampling uncertainty in popular performance metrics; (b) to quantify sampling uncertainty in popular performance metrics across a large sample of catchments; and (c) to prescribe the further research that is, needed to improve the estimation, interpretation, and use of popular performance metrics in hydrologic modeling. Our large‐sample analysis demonstrates that there is substantial sampling uncertainty in the NSE and KGE estimators. This occurs because the probability distribution of squared errors between model simulations and observations has heavy tails, meaning that performance metrics can be heavily influenced by just a few data points. Our results highlight obvious (yet ignored) abuses of performance metrics that contaminate the conclusions of many hydrologic modeling studies: It is essential to quantify the sampling uncertainty in performance metrics when justifying the use of a model for a specific purpose and when comparing the performance of competing models.

Protecting Ownership Rights of Videos Against Digital Piracy: An Efficient Digital Watermarking Scheme

Violation of one’s intellectual ownership rights by the others is a common problem which entertainment industry frequently faces now-a-days. Sharing of information over social media platforms such as Instagram, WhatsApp and twitter without giving credit the owner causes huge financial losses to the owner and hence needs an immediate attention. Digital watermarking is a promising technique to protect owners’ right against digital piracy. Most of the state-of-the-art techniques does not provides adequate level of resilience against majority of video specific attacks and other commonly applied attacks. Therefore, this paper proposes a highly transparent and robust video watermarking solution to protect the owners rights by first convert each video frame into YCbCr color components and then select twenty five strongest speeded-up robust features (SURF) points of the normalized luminance component as points for both watermark embedding and extraction. After applying variety of geometric, simple signal processing and video specific attacks on the watermarked video meticulous analysis is performed using popular metrics which reveals that the proposed scheme possesses high correlation value which makes it superior for practical applications against these attacks. The scheme also proposes a novel three-level impairment scale for subjective analysis which gives stable results to derive correct conclusions.

More Trust in Fewer Followers: Diverging Effects of Popularity Metrics and Green Orientation Social Media Influencers

Digital marketing campaigns increasingly utilize social media influencers. Research in influencer marketing has investigated popularity metrics but found conflicting results on how the number of followers and likes on posts might influence consumers' behaviors. The present research investigates green living orientation of influencers as a moderating characteristic that leads to differential interpretation of popularity metrics in the context of green advertising messages. Specifically, lower popularity metrics seem to benefit green influencers or “greenfluencers.” An increased perception of trust in greenfluencers with lower popularity results in enhanced attitudes toward a sponsored product and increased purchase intentions. Additionally, consumers are willing to donate higher amounts to a related charity after exposure to a promotional post by a green influencer with low following. We present the results of three lab studies and discuss theoretical and practical implications.

An Ensemble-Based Scalable Approach for Intrusion Detection Using Big Data Framework.

In this study, we set up a scalable framework for large-scale data processing and analytics using the big data framework. The popular classification methods are implemented, tuned, and evaluated by using intrusion datasets. The objective is to select the best classifier after optimizing the hyper-parameters. We observed that the decision tree (DT) approach outperforms compared with other methods in terms of classification accuracy, fast training time, and improved average prediction rate. Therefore, it is selected as a base classifier in our proposed ensemble approach to study class imbalance. As the intrusion datasets are imbalanced, most of the classification techniques are biased toward the majority class. The misclassification rate is more in the case of the minority class. An ensemble-based method is proposed by using K-Means, RUSBoost, and DT approaches to mitigate the class imbalance problem; empirically investigate the impact of class imbalance on classification approaches' performance; and compare the result by using popular performance metrics such as Balanced Accuracy, Matthews Correlation Coefficient, and F-Measure, which are more suitable for the assessment of imbalanced datasets.