Multiple Performance Metrics Research Articles

Investigation on communication aspects of multiple swarm networked robotics

Swarm robotics is an emerging field of robotics and is envisioned to play a vital role in surveillance and search/rescue operations. Most of the existing works on swarm networked robotics address the problem of formation movement or the communication aspects within a swarm. However, none of the existing works consider multiple swarms. Even for the case of single swarms, researchers use unrealistic assumptions with respect to communication, leading to unrealistic results. In this paper, we evaluate the performance of multiple swarms considering realistic assumptions with respect to communication. To the best of our knowledge, it will be the first time where the performance is evaluated for the case of multiple swarms while considering realistic assumptions with respect to communication. Our simulation results shed light on the roles of three different types of communication associated with multiple swarms with respect to multiple performance metrics.

Performing Under Pressure: Varsity Athletes Excel in Medical School.

The admission process for medical school relies on objective and subjective measures of personal achievement, and selecting successful medical students is a daunting task for admission committees. While there is a significant body of literature examining MCAT scores and undergraduate grade point average (GPA) with medical school performance, there is a paucity of research on impact of specific student accomplishments on future performance. We hypothesize participation in a varsity collegiate sport will correlate with higher performance during medical school. A retrospective review of four medical school classes at a single institution was completed. Each student's admission application was examined for at least 1year of participation in a varsity-level sport at their respective undergraduate institution. A total of 62 athletes (16.36%) were identified out of 441 total students. Multiple medical school performance metrics were obtained for each student. There was no difference in MCAT scores between athletes and non-athlete medical students. There was a significant difference in step 1, step 2 CK, NBME shelf exams, cumulative year 3 performance, and AOA status with the athletes outperforming their peers. Students who participate in collegiate varsity athletics excel in medical school. One explanation for this finding may be participation in high-stakes athletic training and competition results in development of specific attributes beneficial in medical school. These attributes may include receptiveness to criticism, time management, resiliency, team participation, and performing under pressure. Additional research is needed to elucidate the attributes that determine improved medical school performance, such that medical educators can utilize this knowledge to better prepare all students for the rigors of medical school.

The Virtual-Environment-Foraging Task enables rapid training and single-trial metrics of rule acquisition and reversal in head-fixed mice

Behavioural flexibility is an essential survival skill, yet our understanding of its neuronal substrates is still limited. While mouse research offers unique tools to dissect the neuronal circuits involved, the measurement of flexible behaviour in mice often suffers from long training times, poor experimental control, and temporally imprecise binary (hit/miss) performance readouts. Here we present a virtual-environment task for mice that tackles these limitations. It offers fast training of vision-based rule reversals (~100 trials per reversal) with full stimulus control and continuous behavioural readouts. By generating multiple non-binary performance metrics per trial, it provides single-trial estimates not only of response accuracy and speed, but also of underlying processes like choice certainty and alertness (discussed in detail in a companion paper). Based on these metrics, we show that mice can predict new task rules long before they are able to execute them, and that this delay varies across animals. We also provide and validate single-trial estimates of whether an error was committed with or without awareness of the task rule. By tracking in unprecedented detail the cognitive dynamics underlying flexible behaviour, this task enables new investigations into the neuronal interactions that shape behavioural flexibility moment by moment.

The multi-objective deployment optimization of UAV-mounted cache-enabled base stations

The deployment of unmanned aerial vehicle (UAV)-mounted base stations is emerging as an effective solution for providing wireless communication service to ground terminals (GTs) which have failed to be associated with ground base stations for some reason. Meanwhile, with the propose of reducing the transmission latency and easing the load of backhaul links between UAVs and the core network, UAVs are equipped with the ability of caching popular contents in the storage of base stations. In this paper, we investigate the efficient deployment problem of UAVs (such as transmitting power, number of UAVs, locations and caching) while guaranteeing the quality of service requirements. In this case, the UAV plays the role of a coordinator to provide high-quality communication service for GTs as well as maximize the benefit of caching. However, there exists an intractable issue that UAVs need to consider the optimization problem of multiple performance metrics with various types of optimization variables. To tackle the challenge, we propose a reinforcement learning-based approach to solve the multi-objective deployment problem while maintaining the optimal tradeoff between power consumption and backhaul saving. Numerical results evaluate the performance of the proposed algorithm.

Multilevel split of high-dimensional water quality data using artificial neural networks for the prediction of dissolved oxygen in the Danube River

In this study, a self-organizing network-based monitoring location similarity index (LSI) was coupled with Ward neural networks (WNNs) with the aim to create a more accurate, but less complex, multiple sites model for the prediction of dissolved oxygen (DO) content. This multilevel splitting approach comprises the LSI-based grouping of monitoring locations according to their similarity, and virtual splitting of processed data based on their features using WNN. The values of 18 water quality parameters monitored for 12 years at 17 sites on the Danube River flow thought Serbia were used. The optimal input combinations were selected using partial mutual information algorithm with termination based on the Akaike information criterion. LSI-based splitting has yielded two groups of monitoring sites that were modeled with separate WNN models. The number and types of selected inputs differed between those two groups of sites, which was in agreement with possible pollution sources. Multiple performance metrics have revealed that the WNN models perform similar or better than multisite DO prediction models published in the literature, while using two to four times less inputs and data patterns.

Training-Free Non-Intrusive Load Extracting of Residential Electric Vehicle Charging Loads

Extracting the charging loads of residential electric vehicle (EV) clusters and identifying their charging patterns can help grid operators develop effective regulation strategies. The duration of the power consumption event (PCE) and the interval between adjacent events are used to characterize the difference in the stochastic behavior of the load pattern between the EV cluster and the air conditioner (AC) cluster. An event detection method based on skipping power difference is proposed, which can effectively identify changing edges of the PCE. A training-free non-intrusive load extracting (NILE) algorithm based on bounding-box fitting and load signatures is proposed, which can automatically identify the start time, the end time and the power amplitude of the charging event. The validity of the NILE algorithm is verified by multiple performance metrics on the real data set.

A Novel Approach for Breast Cancer Investigation and Recognition Using M-Level Set-Based Optimization Functions

Breast cancer identification is the first and foremost step in the journey of proper diagnosis and treatment of this disease. Therefore, many medical examinations and applications are devised and used including different approaches for breast imaging. For medical image analysis, nonparametric algorithm has been used. This work focuses on improving breast cancer recognition in mammogram images using a nonparametric approach based on image pixel intensities (IBCNP). A nonparametric approach is a new expression to deal with uncertainty. In this paper, the nonparametric approach is used to solve the uncertainty in breast cancer detection; uncertainty means no one knows for sure how to detect breast cancer without imaging, which creates an uncertainty for the oncologists. This paper investigates the feasibility of applying the nonparametric approach in mammography (x-ray breast imaging) to detect and hence diagnose and treat breast cancer. Analysing breast images is a very complicated process and requires distinguishing normal and malignant pixel intensities. Maximizing the shared data between image pixel intensities and the desired areas (infected region) is defined as the identification problem, subject to a constraint on the total size of the desired area borders. As part of this research, a comparative study was conducted on the proposed algorithm, Breast Cancer Cells Infestation based on a set of partial differential equations (PDEs), Incoherent Motion in Breast Cancer (IMBC), Global Dynamics of a Breast Cancer Competition Model using five simple differential equations (SDFs), and Estimation of Intensity Using Nonparametric Method (EINP) techniques. These five techniques of medical image segmentation are applied to 156 different mammogram images to obtain the exact measurement of the efficiency of the identification process. This process uses multiple performance metrics such as the overall mammogram image quality (Q), the dice similarity coefficient (DSC), Peak signal-to-noise ratio (PSNR) metric, Haussdorff Distance (HD), and probability of tumor detection (PTD). This paper assumes lack of knowledge of the probability densities in relation to pixel intensities of an image within a given area. It proposes an estimation of non-parametric density that formulates the theoretical data optimization problem by applying curve evolution methods and deriving the correlated gradient flows. It uses level-set techniques to achieve the resulting evolution. The algorithm was applied to 156 sets of different mammogram images from twelve groups with both malignant and/or normal. The experimental findings showed that the proposed algorithm is 94.937% accurately effective in breast cancer detection.

A MCDM-Based Evaluation Approach for Imbalanced Classification Methods in Financial Risk Prediction

Various classifiers have been proposed for financial risk prediction. The traditional practice of using a singular performance metric for classifier evaluation is not sufficient for imbalanced classification. This paper proposes a multi-criteria decision making (MCDM)-based approach to evaluate imbalanced classifiers in credit and bankruptcy risk prediction by considering multiple performance metrics simultaneously. An experimental study is designed to provide a comprehensive evaluation of imbalanced classifiers using the proposed evaluation approach over seven financial imbalanced data sets from the UCI Machine Learning Repository. The TOPSIS, a well-known MCDM method, was applied to rank three categories of imbalanced classifiers using six popular evaluation criteria. The rankings results indicate that: 1) the rankings generated by the TOPSIS, which combine the results of six evaluation criteria, provide a more reasonable evaluation of imbalanced classifiers over any single performance criterion; and 2) Synthetic Minority Oversampling Technique (SMOTE)-based ensemble techniques outperform other groups of imbalanced learning approaches. Specifically, SMOTEBoost-C4.5, SMOTE-C4.5, and SMOTE-MLP were ranked as the top three classifiers based on their performances on the six criteria.

A framework for generating and evaluating façade designs using a multi-agent system approach

Digital design paradigms in architecture have been rooted in representational models which are geometry centered and therefore fail to capture building complexity holistically. Due to a lack of computational design methodologies, existing digital design workflows do little in predicting design performance in the early design stage and in most cases analysis and design optimization are done after a design is fixed. This work proposes a new computational design methodology, intended for use in the area of conceptual design of building design. The proposed methodology is implemented into a multi-agent system design toolkit which facilitates the generation of design alternatives using stochastic algorithms and their evaluation using multiple environmental performance metrics. The method allows the user to probabilistically explore the solution space by modeling the design parameters’ architectural design components (i.e. façade panel) into modular programming blocks (agents) which interact in a bottom-up fashion. Different problem requirements (i.e. level of daylight inside a space, openings) described into agents’ behavior allow for the coupling of data from different engineering fields (environmental design, structural design) into the a priori formation of architectural geometry. In the presented design experiment, a façade panel is modeled into an agent-based fashion and the multi-agent system toolkit is used to generate and evolve alternative façade panel configurations based on environmental parameters (daylight, energy consumption). The designer can develop the façade panel geometry, design behaviors, and performance criteria to evaluate the design alternatives. The toolkit relies on modular and functionally specific programming modules (agents), which provide a platform for façade design exploration by combining existing three-dimensional modeling and analysis software.

The significance of periodic parameters for ANN modeling of daily SO2 and NOx concentrations: A case study of Belgrade, Serbia

In recent decades, artificial neural networks (ANNs) have been used for the prediction of concentration of air pollutants in urban areas. Beside meteorological variables, periodic parameters, such as hour of the day or month of the year, have been frequently used to improve the performance of ANN models by representing variations of emission sources. In this paper, different forms of periodic parameters, i.e. smoothed cosines based approximation and normalized historical mean values, were combined with meteorological variables in order to analyze the sensitivity of the ANN model to them. Ward neural network and general regression neural network were used and compared for the prediction of daily average concentrations of SO2 and NOx in Belgrade, Serbia. Multiple performance metrics have demonstrated that models based on periodic parameters outperform the corresponding models that used only meteorological variables as inputs. Also, a newly proposed normalized historical mean MOYnmv (month of the year) proved to be more appropriate in majority of cases than the traditional cosines based approximation (MOYcos). A simple rule for the selection of the most efficient MOY form was defined depending on their mutual correlation (r). Results have shown that if MOYnmv is correlated with MOYcos with r > 0.8, then ANN models what uses MOYnmv provide more accurate predictions.

The Neglected “R” in the Risk-Needs-Responsivity Model: A New Approach for Assessing Responsivity to Correctional Interventions

Prevailing correctional practice holds that offenders should be assigned to interventions on the basis of assessments for risk, needs, and responsivity. Assessments of responsivity, however, typically consist of little more than a checklist of items such as motivation, gender, language, or culture. We introduce a new actuarial approach for assessing responsivity, which focuses on predicting whether individuals will desist after participating in an intervention. We assess responsivity by using multiple classification methods and predictive performance metrics to analyze various approaches for prioritizing individuals for correctional interventions. The results suggest that adding an actuarial responsivity assessment to the existing risk and needs assessments would likely improve treatment assignments and further enhance the effectiveness of an effective intervention. We conclude by discussing the implications of more rigorous responsivity assessments for correctional research, policy, and practice.

Benchmarking optimization methods for parameter estimation in large kinetic models

MotivationKinetic models contain unknown parameters that are estimated by optimizing the fit to experimental data. This task can be computationally challenging due to the presence of local optima and ill-conditioning. While a variety of optimization methods have been suggested to surmount these issues, it is difficult to choose the best one for a given problem a priori. A systematic comparison of parameter estimation methods for problems with tens to hundreds of optimization variables is currently missing, and smaller studies provided contradictory findings.ResultsWe use a collection of benchmarks to evaluate the performance of two families of optimization methods: (i) multi-starts of deterministic local searches and (ii) stochastic global optimization metaheuristics; the latter may be combined with deterministic local searches, leading to hybrid methods. A fair comparison is ensured through a collaborative evaluation and a consideration of multiple performance metrics. We discuss possible evaluation criteria to assess the trade-off between computational efficiency and robustness. Our results show that, thanks to recent advances in the calculation of parametric sensitivities, a multi-start of gradient-based local methods is often a successful strategy, but a better performance can be obtained with a hybrid metaheuristic. The best performer combines a global scatter search metaheuristic with an interior point local method, provided with gradients estimated with adjoint-based sensitivities. We provide an implementation of this method to render it available to the scientific community.Availability and implementationThe code to reproduce the results is provided as Supplementary Material and is available at Zenodo https://doi.org/10.5281/zenodo.1304034.Supplementary information Supplementary data are available at Bioinformatics online.

Real-Time Nowcasting of Microbiological Water Quality at Recreational Beaches: A Wavelet and Artificial Neural Network-Based Hybrid Modeling Approach.

The number of beach closings caused by bacterial contamination has continued to rise in recent years, putting beachgoers at risk of exposure to contaminated water. Current approaches predict levels of indicator bacteria using regression models containing a number of explanatory variables. Data-based modeling approaches can supplement routine monitoring data and provide highly accurate short-term forecasts of beach water quality. In this paper, we apply the nonlinear autoregressive network with exogenous inputs (NARX) method with explanatory variables to predict Escherichia coli concentrations at four Lake Michigan beach sites. We also apply the nonlinear input-output network (NIO) and nonlinear autoregressive neural network (NAR) methods in addition to a hybrid wavelet-NAR (WA-NAR) model and demonstrate their application. All models were tested using 3 months of observed data. Results revealed that the NARX models provided the best performance and that the WA-NAR model, which requires no explanatory variables, outperformed the NIO and NAR models; therefore, the WA-NAR model is suitable for application to data scarce regions. The models proposed in this paper were evaluated using multiple performance metrics, including sensitivity and specificity measures, and produced results comparable or superior to those of previous mechanistic and statistical models developed for the same beach sites. The relatively high R2 values between data and the NARX models ( R2 values of ∼0.8 for the beach sites and ∼0.9 for the river site) indicate that the new class of models shows promise for beach management.

A case-study based framework for assessing the multi-sector performance of green infrastructure

Green infrastructure is emerging as a holistic stormwater management strategy that can also provide multi-sector benefits. Robust demonstration of project success can help leverage the appeal of green infrastructure to different sectors and open the door to a variety of funding opportunities. Yet comprehensively assessing the performance of these natural systems can be challenging, especially when communicating the benefits to a wide variety of stakeholders. A cohesive, well-described assessment structure may promote a higher degree of investor confidence by more comprehensively monitoring and measuring green infrastructure success. This paper develops a conceptual framework that incorporates a robust assessment component for communicating with potential investors through the inclusion of multiple evaluation methods, performance metrics, and risk categories. The applied performance of this framework is then validated using fourteen U.S. and international case studies. We found that our framework fit a wide range of projects while maintaining a degree of flexibility that did not sacrifice specificity when applied to individual case studies. This suggests that: 1) some GI projects already incorporate one or more evaluation methods; 2) a number of highly specific metrics—particularly social and economic performance metrics—exist that are capable of capturing a wide-range of benefits that can be easily integrated into a framework; 3) the incorporation of risk and risk management technique identification could be emphasized to increase investor confidence; 4) at least some degree of standardization across projects exists already which can help future project implementers design GI strategies that best fit their needs.

Model Predictive Control as a Simulator for Studying the Operation Quality of Semiconductor Supply Chains in the Perspective of Integrated Device Manufacturers

The nature of competition in the semiconductor industry has shifted from inter-firm to inter-supply chain. For integrated device manufacturers (IDMs), effective supply chain management (SCM) has become their competitive advantage in the market with intensive rivalry. Model predictive control (MPC) is a convincingly robust and flexible technique for dynamically controlling material release, managing inventories, and meeting customer demand through supply chains. This study proposed an MPC framework for programming a nonstationary semiconductor manufacturing supply chain network in the perspective of IDMs. The supply chain network is characterized by multiple echelons from raw material supply to finished product shipment, multiple factories, manufacturing stochasticity, diverse product, and product substitution mechanisms. The MPC model was employed as a simulator to obtain the optimal operations of supply chain network under various combinations of three two-level scenario variables: high/low demand variation, push/pull supply chain strategy, and normal/quick raw material supply policy. The operation quality of supply chain and the effects of scenario variables were assessed based on multiple performance metrics.

The First Comparison Between Swarm‐C Accelerometer‐Derived Thermospheric Densities and Physical and Empirical Model Estimates

AbstractThe first systematic comparison between Swarm‐C accelerometer‐derived thermospheric density and both empirical and physics‐based model results using multiple model performance metrics is presented. This comparison is performed at the satellite's high temporal 10‐s resolution, which provides a meaningful evaluation of the models' fidelity for orbit prediction and other space weather forecasting applications. The comparison against the physical model is influenced by the specification of the lower atmospheric forcing, the high‐latitude ionospheric plasma convection, and solar activity. Some insights into the model response to thermosphere‐driving mechanisms are obtained through a machine learning exercise. The results of this analysis show that the short‐timescale variations observed by Swarm‐C during periods of high solar and geomagnetic activity were better captured by the physics‐based model than the empirical models. It is concluded that Swarm‐C data agree well with the climatologies inherent within the models and are, therefore, a useful data set for further model validation and scientific research.

Evaluating Performance of Regression Machine Learning Models Using Multiple Error Metrics in Azure Machine Learning Studio

Data driven companies effectively use regression machine learning methods for making predictions in many sectors. Cloud-based Azure Machine Learning Studio (MLS) has a potential of expediting machine learning experiments by offering a convenient and powerful integrated development environment. The process of evaluating machine learning models in Azure MLS has certain limitations, e.g. small number of performance metrics and lack of functionality to evaluate custom built regression models with R language. This paper reports the results of an effort to build an Enhanced Evaluate Model (EEM) module which facilitates and accelerates Azure experiments development and evaluation. The EEM combines multiple performance metrics allowing for multisided evaluation of the regression models. EEM offers 4 times more metrics than built-in Azure Evaluate Model module. EEM metrics include: CoD, GMRAE, MAE, MAPE, MASE, MdAE, MdAPE, MdRAE, ME, MPE, MRAE, MSE, NRMSE_mm, NRMSE_sd, RAE, RMdSPE, RMSE, RMSPE, RSE, sMAPE, SMdAPE, SSE. Also, EEM supports evaluation of the R language based regression models. The operational Enhanced Evaluate Model module has been published to the web and openly available for experiments and extensions.

A multimetric and multideme multiagent system for multiobjective optimization

AbstractThis article proposes a multiagent system consisting of a number of multiobjective metaheuristic agents (namely, multiobjective genetic algorithm, strength Pareto evolutionary algorithm, differential evolution, simulated annealing, and particle swarm optimization) working toward to extract optimal or very close‐to‐optimal Pareto fronts using multiple performance metrics in a sessionwise manner. At the beginning of each session, the main population is divided into a number of subpopulations, and each of them is assigned to a particular agent. The system runs in consecutive sessions such that, at the beginning of a session, agents start running after being assigned with a subpopulation and return the optimized subpopulations together with the corresponding set of nondominated solutions at the end of the session. There are 3 multiobjective assessment metrics in use, and a different metric is considered for each session to measure the success of each metaheuristic agent. The evaluation of individual agents using a particular assessment metric is used in 2 ways: first, the number of fitness evaluations for each agent is adjusted based on their performance; second, the subpopulation improved by an individual agent might be rejected on the basis of its evaluation score. At the end of each session, individual subpopulations are merged to get the updated main population, whereas individual sets of nondominated solutions are combined to form the global Pareto front. In addition to the individual multiobjective metaheuristic agents, the system also contains a number of coordination and synchronization agents that run the whole system toward its objectives. The proposed system is tested using real‐valued multiobjective benchmark problems in 2009 IEEE Congress on Evolutionary Computation. Experimental results and statistical evaluations exhibited that the achieved success is better than many of state‐of‐the‐art algorithms.

Misfire and valve clearance faults detection in the combustion engines based on a multi-sensor vibration signal monitoring

Fault diagnosis of the rotary machines is investigated through different kinds of signals. However, the literature shows that the vibration signal analysis is the most commonly used and effective approach. This research investigates the engine faults, including the misfire and valve clearance faults, using the vibration data captured by four sensors placed in different locations of the automobile engine and under different experimental circumstances. The application of the Fast Fourier Transform (FFT) is proposed as a feature extraction methodology which leads to the extraction of 16 features. In addition, four features are extracted using the acquired signals eigenvalues. The statistical approach is proposed to select features for classification of the engine’s state. The Artificial Neural Networks (ANN), Support Vector Machines (SVM), and k Nearest Neighbor (kNN) classification algorithms are employed to predict if the motor works healthily based on the selected features and, if not, what kind of faults is in the engine. The performance of ANN, SVM, and kNN in fault diagnosis is analyzed considering different scenarios, features, and based on multiple performance metrics. Comparing the results with the similar efforts in the literature proves the validity of the proposed methods and highlights their superiorities.

Ensemble classifier for misuse detection using N-gram feature vectors through operating system call traces

This paper presents development and performance evaluation of a host-based misuse intrusion detection system. The misuse detection system employs an ensemble design for classification and N-gram feature extraction methodology for preprocessing the raw ADFA-LD system call trace data. Inputs to the e nsemble classifier are fixed size patterns whose attributes are N-grams derived from operating system call traces. The ADFA-LD data set entails system call traces generated by a number of concurrently executing applications on the Linux operating system environment. In addition to the normal or attack-free operating mode, six different attacks are modeled in the data set. Two filters are used to capture the unique signatures of each class and also to reduce the dimensionality of input patterns. The most frequent unique features in the form of N-grams from each class are extracted from the raw trace data. Then, to eliminate the effect of noise, most frequent patterns regardless of uniqueness are extracted based on the statistical attributes of their occurrence frequencies. The SMOTE algorithm is used to balance the classes in terms of pattern counts. The classifier design is based on majority voting ensemble with base classifiers including naïve Bayes, support vector machine, PART, decision tree and random forest. The study considers both binary and multi class problems. In the case of binary class problem, the two classes are the “normal” and the “attack” where the latter is formed by merging all of the trace data for all six different attack signatures into a single one. In the multiclass case, normal and six attack classes are each considered separately. A simulation study was performed to evaluate the performance of the proposed host-based misuse detection system. Multiple performance indicators and metrics including confusion matrix, true positive/negative, and false positive/negative rates were recorded. The proposed host-based misuse detection system demonstrated very high performance for detecting the attacks for the binary class problem although its performance for the multi-class case will benefit from further improvement.