Global Performance Metrics Research Articles

Assessing Deep and Shallow Learning Methods for Quantitative Prediction of Acute Chemical Toxicity.

Animal-based methods for assessing chemical toxicity are struggling to meet testing demands. In silico approaches, including machine-learning methods, are promising alternatives. Recently, deep neural networks (DNNs) were evaluated and reported to outperform other machine-learning methods for quantitative structure-activity relationship modeling of molecular properties. However, most of the reported performance evaluations relied on global performance metrics, such as the root mean squared error (RMSE) between the predicted and experimental values of all samples, without considering the impact of sample distribution across the activity spectrum. Here, we carried out an in-depth analysis of DNN performance for quantitative prediction of acute chemical toxicity using several datasets. We found that the overall performance of DNN models on datasets of up to 30 000 compounds was similar to that of random forest (RF) models, as measured by the RMSE and correlation coefficients between the predicted and experimental results. However, our detailed analyses demonstrated that global performance metrics are inappropriate for datasets with a highly uneven sample distribution, because they show a strong bias for the most populous compounds along the toxicity spectrum. For highly toxic compounds, DNN and RF models trained on all samples performed much worse than the global performance metrics indicated. Surprisingly, our variable nearest neighbor method, which utilizes only structurally similar compounds to make predictions, performed reasonably well, suggesting that information of close near neighbors in the training sets is a key determinant of acute toxicity predictions.

A Survey on Algorithmic Aspects of Virtual Optical Network Embedding for Cloud Networks

An important challenge in network virtualization is the process of assigning physical resources to virtual network requests, where virtual node resources represent the demands of IT capacities while virtual links are connected between pairs of virtual nodes. Due to their special characteristics, physical optical network infrastructures requires specific strategies to solve this problem, called virtual optical network embedding (VONE). This paper presents a survey on most relevant VONE schemes found in the literature. A discussion of important algorithmic aspects of this problem is presented, such as optical-layer restrictions, global performance metrics, and possible research directions. In addition, we propose a new taxonomy of VONE approaches, considering the type of optical networks, dynamic or static scenarios, and the optimization method.

Joint Transceiver, Data Streams, and User Ordering Optimization for Nonlinear Multiuser MIMO Systems

We consider nonlinear signal processing algorithms for downlink multiple-input multiple-output (MIMO) systems with multiple-antenna users. The design goal is to improve bit error rate (BER) performance by jointly optimizing the transceiver, data streams, and user ordering, under given total transmit power. We consider a general global objective function, whose elements are Schur-convex functions of the mean square error (MSE) of each user. With nonlinear Tomlinson-Harashima precoding combined with block successive zero-forcing precoding at the transmitter, we show that the optimal nonlinear transceiver leads to favorable diagonal and parallel structures for all users with the general global performance metric. The number of data streams of each user is allowed to be an arbitrary number no more than the rank of the effective channel. We then provide closed-form expressions of the transceiver matrices and optimal power allocation of each user, and analytically characterize the optimal number of data streams of each user for the minimax and average BER metrics. The user ordering is also optimized to further improve the system performance. We also analytically investigate the impact of channel spatial correlation on the performance of our scheme and illustrate why our proposed adaptive strategy can mitigate the performance degradation caused by channel spatial correlation. The superiority of our proposed framework is demonstrated through numerical results.

Globally Optimal Cooperation in Dense Cognitive Radio Networks

In cooperative spectrum sensing, local sensing at different sensing nodes is done either using soft decisions or hard decisions. The hard decision-based sensing has the advantage of using only one bit to report the local decision. In the literature, the hard decisions are combined at the fusion center using AND, OR, or MAJORITY rules. Although the problem of finding the fusion rule was addressed and solved for the soft decisions fusion, it was not solved in the hard-decisions sensing. The problem of calculating the local and global decision thresholds in hard decisions-based cooperative spectrum sensing is known for its mathematical intractability. Hence, previous studies relied on simple suboptimal counting rules for decision fusion in order to avoid the exhaustive numerical search required for obtaining the optimal thresholds. These simple rules are not globally optimal as they do not maximize the overall global detection probability by jointly selecting local and global thresholds. Instead, they try to maximize the detection probability for a specific global threshold. In this paper, a globally optimal decision fusion rule for Primary User signal detection based on the Neyman-Pearson (NP) criterion is derived. The algorithm is based on a novel representation for the global performance metrics in terms of the regularized incomplete beta function. Based on this mathematical representation, it is shown that the globally optimal NP hard decision fusion test can be put in the form of a conventional one dimensional convex optimization problem. A binary search for the global threshold can be applied yielding a complexity of $$\mathcal {O}(\log _{2}(N))$$O(log2(N)), where $$N$$N represents the number of cooperating users. The logarithmic complexity is appreciated because we are concerned with dense networks, and thus $$N$$N is expected to be large. The proposed optimal scheme outperforms conventional counting rules, such as the OR, AND, and MAJORITY rules. It is shown via simulations that, although the optimal rule tends to the simple OR rule when the number of cooperating secondary users is small, it offers significant SNR gain in dense cognitive radio networks with large number of cooperating users.

Unmanned aerial vehicle trajectory tracking algorithm comparison

PurposeThe purpose of this paper is to analyze and compare the performance of several different UAV trajectory tracking algorithms in normal and abnormal flight conditions to investigate the fault‐tolerant capabilities of a novel immunity‐based adaptive mechanism.Design/methodology/approachThe evaluation of these algorithms is performed using the West Virginia University (WVU) UAV simulation environment. Three types of fixed‐parameter algorithms are considered as well as their adaptive versions obtained by adding an immunity‐based mechanism. The types of control laws investigated are: position proportional, integral, and derivative control, outer‐loop nonlinear dynamic inversion (NLDI), and extended NLDI. Actuator failures on the three channels and increased turbulence conditions are considered for several different flight paths. Specific and global performance metrics are defined based on trajectory tracking errors and control surface activity.FindingsThe performance of all of the adaptive controllers proves to be better than their fixed parameter counterparts during the presence of a failure in all cases considered.Research limitations/implicationsThe immunity inspired adaptation mechanism has promising potential to enhance the fault‐tolerant capabilities of autonomous flight control algorithms and the extension of its use at all levels within the control laws considered and in conjunction with other control architectures is worth investigating.Practical implicationsThe WVU UAV simulation environment has been proved to be a valuable tool for autonomous flight algorithm development, testing, and evaluation in normal and abnormal flight conditions.Originality/valueA novel adaptation mechanism is investigated for UAV control algorithms with fault‐tolerant capabilities. The issue of fault tolerance of UAV control laws has only been addressed in a limited manner in the literature, although it becomes critical in the context of imminent integration of UAVs within the commercial airspace.

An agent-based framework for performance modeling of an optimistic parallel discrete event simulator

Abstract Purpose The performance of an optimistic parallel discrete event simulator (PDES) in terms of the total simulation execution time of an experiment depends on a large set of variables. Many of them have a complex and generally unknown relationship with the simulation execution time. In this paper, we describe an agent-based performance model of a PDES kernel that is typically used to simulate large-sized complex networks on multiple processors or machines. The agent-based paradigm greatly simplifies the modeling of system dynamics by representing a component logical process (LP) as an autonomous agent that interacts with other LPs through event queues and also interacts with its environment which comprises the processor it resides on. Method We model the agents representing the LPs using a “base” class of an LP agent that allows us to use a generic behavioral model of an agent that can be extended further to model more details of LP behavior. The base class focuses only on the details that most likely influence the overall simulation execution time of the experiment. Results We apply this framework to study a local incentive based partitioning algorithm where each LP makes an informed local decision about its assignment to a processor, resulting in a system akin to a self organizing network. The agent-based model allows us to study the overall effect of the local incentive-based cost function on the simulation execution time of the experiment which we consider to be the global performance metric. Conclusion This work demonstrates the utility of agent-based approach in modeling a PDES kernel in order to evaluate the effects of a large number of variable factors such as the LP graph properties, load balancing criteria and others on the total simulation execution time of an experiment.

Multiscale Markov models with random transitions for energy demand management

A stochastic model is proposed for fluctuations in electricity demand that are associated with individual user's consumption choices. Electricity consumption is modeled as a function of social activities of consumers. The dynamics of these activities are modeled as a Markov chain. Markov models are simplified models that capture the stochasticity to the unmodeled dynamics typically attributed to white noise disturbances. Additional uncertainties are also accrued in the process of calibrating the transition rates of these chains from finite samples. In this paper, these uncertainties are accounted for by considering random transition matrices. Such formalism can also reflect the fluctuations in the environment in which the chain evolves. We also discuss a third interpretation where uncertain transitions, in a multiscale setting, reflect the fine-resolution information that is lost in the process of state aggregation. As numerical demonstration, we study the activity modeling of a heterogeneous population. Activity uncertainties are propagated onto the energy demand. Demand uncertainties, in turn, are propagated onto a global performance metric. Such uncertainty management framework bridges between the actual drivers of the energy consumption and the system health. Subsequent decisions can be robustly supported based on the results of the quantitative model proposed in this paper.

Analytical Modeling of Medium-Access Delay for Cooperative Wireless Networks Over Rayleigh Fading Channels

CoopMAC has been recently proposed as an efficient medium-access-control (MAC)-layer protocol for cooperative wireless networks. This protocol allows high-data-rate terminals to assist low-data-rate terminals by relaying their data. The backward compatibility of this protocol with well-known distributed local area networks (LANs), its adaptable functionality, and its capability to flatten the received signal-to-noise ratio (SNR) level over time make CoopMAC a desirable choice for cooperative communications. In spite of the prevalence and importance of this protocol, not enough attention has been paid to this protocol so far. Therefore, in this paper, we develop an analytical model to evaluate the performance of this protocol. Our proposed model is centered around a 2-D Markov chain, similar to that introduced by Bianchi for IEEE 802.11. This model empowers us to derive the probability density function (pdf) of medium-access delay in CoopMAC from which any higher statistical moments can be obtained. Using this obtained pdf, both local and global network performance metrics can be improved.

Load or concentration, logged or unlogged? Addressing ten years of uncertainty in neural network suspended sediment prediction

AbstractThis paper addresses current inconsistencies in methodological approaches for neural network modelling of suspended sediment. An expansion in the number of case studies being published over the last decade has yet to result in agreed guidelines on whether suspended sediment load or concentration should be modelled, and whether log‐transformation of data is either necessary or potentially beneficial. This contrasts with the well‐recognized guidelines that direct traditional sediment rating curve studies. The paper reports a comprehensive set of single‐input single‐output neural network suspended sediment modelling experiments performed on two catchments in Puerto Rico. It examines the impact of internal complexity, input variable choice and data transformation on the form, consistency and physical rationality of model outputs, the existence of localized overfitting and the usefulness of global performance metrics. Sound guidance on whether to model sediment load or concentration, and whether to model log‐transformed data is provided. Copyright © 2011 John Wiley & Sons, Ltd.

Structural influence of gene networks on their inference: analysis of C3NET

BackgroundThe availability of large-scale high-throughput data possesses considerable challenges toward their functional analysis. For this reason gene network inference methods gained considerable interest. However, our current knowledge, especially about the influence of the structure of a gene network on its inference, is limited.ResultsIn this paper we present a comprehensive investigation of the structural influence of gene networks on the inferential characteristics of C3NET - a recently introduced gene network inference algorithm. We employ local as well as global performance metrics in combination with an ensemble approach. The results from our numerical study for various biological and synthetic network structures and simulation conditions, also comparing C3NET with other inference algorithms, lead a multitude of theoretical and practical insights into the working behavior of C3NET. In addition, in order to facilitate the practical usage of C3NET we provide an user-friendly R package, called c3net, and describe its functionality. It is available from https://r-forge.r-project.org/projects/c3net and from the CRAN package repository.ConclusionsThe availability of gene network inference algorithms with known inferential properties opens a new era of large-scale screening experiments that could be equally beneficial for basic biological and biomedical research with auspicious prospects. The availability of our easy to use software package c3net may contribute to the popularization of such methods.ReviewersThis article was reviewed by Lev Klebanov, Joel Bader and Yuriy Gusev.

FlexDCP

Current multicore architectures offer high throughput by increasing hardware resource utilization. As the number of cores in a multicore system increases, providing Quality of Service (QoS) to applications in addition to throughput is becoming an important problem. In this work, we present FlexDCP, a framework that allows the Operating System (OS) to guarantee a QoS for each application running in a chip multiprocessor. FlexDCP directly estimates the performance of applications for different cache configurations instead of using indirect measures of performance like the number of misses. This information allows the OS to convert QoS requirements into resource assignments. Consequently, it offers more flexibility to the OS as it can optimize different QoS metrics like per-application performance or global performance metrics such as fairness, weighted speed up or throughput. Our results show that FlexDCP is able to force applications in a workload to run at a certain percentage of their maximum performance in 94% of the cases considered, being on average 1:48% under the objective for remaining cases. When optimizing a global QoS metric like fairness, FlexDCP consistently outperforms traditional eviction policies like LRU, pseudo LRU and previous dynamic cache partitioning proposals for two-, four- and eightcore configurations. In an eight-core architecture FlexDCP obtains a fairness improvement of 10:1% over Fair, the best policy in the literature optimizing fairness.

Bias adjustment techniques for improving ozone air quality forecasts

In this study, we apply two bias adjustment techniques to help improve forecast accuracy by postprocessing air quality forecast model outputs. These techniques are applied to modeled ozone (O3) forecasts over the continental United States during the summer of 2005. The first technique, referred to as the hybrid forecast (HF), combines the most recent observed ozone with model‐predicted ozone tendency for the subsequent forecast time period. The second technique applied here is the Kalman filter (KF), which is a recursive, linear, and adaptive method that takes into account the temporal variation of forecast errors at a specific location. Two modifications to the Kalman filter are investigated. A key parameter in the KF approach is the error ratio, which determines the relative weighting of observed and forecast ozone values. This parameter is optimized to improve the prediction of ozone based on time series data at individual monitoring sites in the Aerometric Information Retrieval Now network. The optimal error ratios inherent in the KF algorithm implementation are found to vary across space; however, comparisons of the resultant KF‐adjusted forecasts using a single fixed value of this parameter with those using the optimal values determined for each individual site reveal similar results, suggesting that the uncertainty in the estimation of this parameter does not have a significant impact on the final bias‐adjusted predictions. The KF postprocessing is also combined with the Kolmogorov‐Zurbenko filter, which extracts the intraday variability in the observed ozone time series; the results indicate a significant improvement in the ozone forecasts at locations in the Pacific coast region, but not as much at locations across the rest of the continental U.S. domain. Both HF and KF bias adjustment techniques help significantly reduce the systematic errors in ozone forecasts. For most of the global performance metrics examined, the KF approach performed better than the HF method. For the given model applications, both methods are effective in reducing biases at low ozone mixing ratio levels, but not as well at the high mixing ratio levels. This is due in part to the fact that high ambient ozone levels occur much less frequently than low to moderate levels for which the current model exhibits a systematic high bias. Additionally, the 12 km model grid structure is often unable to adequately capture the magnitude of peak O3 levels. Thus, these extreme events at discrete monitor locations are more difficult to predict.

Quality of light path with ASE noise accumulation in nonlinear optical propagation using global performance metrics

In this paper, we have targeted our investigations to the inclusion in performance evaluation of light path on the effect of amplified spontaneous emission (ASE) noise accumulated in nonlinear optical propagation. Quality of light-path transmission feasibility has been reported by using a global performance metrics, called Q-factor, optical signal-to-noise ratio (OSNR), BER, extinction ratio and nonlinear frequency shift, which takes into account the interaction between the parameters aforementioned and returns a unique value corresponding to the performance metrics.

Modeling and performance analysis of single-bus tightly-coupled multiprocessors

A hierarchical stochastic queuing model is presented that consists of a set of processing elements (PEs) and a single queue/server pair representing the shared memory. The model takes into account not only global system behavior but also behavior of tasks on each processing element and probabilistic task migration among the PEs. It is shown that through the use of repeated aggregation each PE can be treated as a queue/state-dependent server pair. This reduction results in an immense simplification of the model. A CPU sensitivity analysis of this reduced model shows that at high CPU service rates, varying the CPU speed has a negligible effect on the server's state-dependent service rate, whereas at low CPU service rates, varying the CPU speed has an effect that depends on the local branching probabilities. Global performance metrics are then obtained on the basis of this reduction. >