Global Performance Measures Research Articles

PROTOCOL: Small Class Sizes for Improving Student Achievement in Primary and Secondary Schools: A Systematic Review

PROTOCOL: Small Class Sizes for Improving Student Achievement in Primary and Secondary Schools: A Systematic Review

Combining Multiple Measures into a Single Figure of Merit

Researchers often collect a number of dependent measures in a study, each of which may reflect some aspect of overall performance. In psychology, in particular response time and accuracy are frequently used measures. Trade-offs between speed and accuracy are often observed, and can occur between other measures. How should overall performance be characterized in the presence of trade-offs? In this paper, we consider how multiple measures can be combined into a single, overall measure of performance. We consider two different case studies, one involving speed and accuracy, and the other involving the relationship between sampling of visual information and resulting accuracy in a pedal-tracking task. We define a global measure of performance using summated z-scores. We discuss the behavior of this measure and contrast it with other approaches to creating linear combinations of variables.

Wavelet‐based multiscale performance analysis: An approach to assess and improve hydrological models

AbstractThe temporal dynamics of hydrological processes are spread across different time scales and, as such, the performance of hydrological models cannot be estimated reliably from global performance measures that assign a single number to the fit of a simulated time series to an observed reference series. Accordingly, it is important to analyze model performance at different time scales. Wavelets have been used extensively in the area of hydrological modeling for multiscale analysis, and have been shown to be very reliable and useful in understanding dynamics across time scales and as these evolve in time. In this paper, a wavelet‐based multiscale performance measure for hydrological models is proposed and tested (i.e., Multiscale Nash‐Sutcliffe Criteria and Multiscale Normalized Root Mean Square Error). The main advantage of this method is that it provides a quantitative measure of model performance across different time scales. In the proposed approach, model and observed time series are decomposed using the Discrete Wavelet Transform (known as theà trouswavelet transform), and performance measures of the model are obtained at each time scale. The applicability of the proposed method was explored using various case studies––both real as well as synthetic. The synthetic case studies included various kinds of errors (e.g., timing error, under and over prediction of high and low flows) in outputs from a hydrologic model. The real time case studies investigated in this study included simulation results of both the process‐based Soil Water Assessment Tool (SWAT) model, as well as statistical models, namely the Coupled Wavelet‐Volterra (WVC), Artificial Neural Network (ANN), and Auto Regressive Moving Average (ARMA) methods. For the SWAT model, data from Wainganga and Sind Basin (India) were used, while for the Wavelet Volterra, ANN and ARMA models, data from the Cauvery River Basin (India) and Fraser River (Canada) were used. The study also explored the effect of the choice of the wavelets in multiscale model evaluation. It was found that the proposed wavelet‐based performance measures, namely the MNSC (Multiscale Nash‐Sutcliffe Criteria) and MNRMSE (Multiscale Normalized Root Mean Square Error), are a more reliable measure than traditional performance measures such as the Nash‐Sutcliffe Criteria (NSC), Root Mean Square Error (RMSE), and Normalized Root Mean Square Error (NRMSE). Further, the proposed methodology can be used to: i) compare different hydrological models (both physical and statistical models), and ii) help in model calibration.

A resilient global supplier selection strategy—a case study of an automotive company

Disaster studies have recently received considerable attention. Determining how to build a resilient supply chain to mitigate uncertainty is the priority for automotive companies. Supplier selection strategies have been identified as vital for mitigating this uncertainty. This study sought to verify the criteria for selecting suppliers by using global performance measurements to identify optimal supply resources and locations in an uncertain disaster environment. The study focuses on an automotive company case study and evaluates the results through weighted goal programming (WGP) and preemptive goal programming (PGP) methods. The findings are as follows: (1) The investigated automotive company must arrange additional supply resources in the involved region by using a global supplier selection strategy, (2) an appropriate location in the involved region is identified through a global performance measurement and evaluation, and (3) disaster damage mitigation is determined to be a priority in long-term business strategies. This case study improves the preparedness of the automotive industry to manage unpredictable disasters in the supply chain through the effective use of a global selection strategy.

Observer design for generalized sector-bounded noisy nonlinear systems

This article presents a new observer for a class of nonlinear systems, defined as a “generalized sector-bounded” nonlinear system, in the presence of both sensor and input disturbances. The generalized sector-bounded nonlinearity is shown to be a super-set of Lipschitz, bounded Jacobian, one-sided Lipschitz, monotonically increasing and dissipative nonlinearities. This article presents necessary and sufficient conditions for this observer to guarantee a desired minimum performance. The conditions for the observer are presented as a linear matrix inequality that can be solved offline using commercial solvers, and the solution to the linear matrix inequality is used to explicitly compute the observer gain. This article then extends these results to case where an additive nonlinearity appears in the sensor output. The use of the methodology developed in this article is demonstrated through illustrative examples. Compared to previous results on nonlinear observers, the proposed observer guarantees a global performance measure for a very general class of nonlinear systems and does not require online computation of the observer gain.

Combining Speed and Accuracy into a Global Measure of Performance

Response time and accuracy are two of the most frequently collected dependent measures. Tradeoffs between speed and accuracy are often observed, both between people, and between experimental conditions. In this paper we consider how speed, and accuracy, can be combined into a single, overall measure of performance. We consider two different approaches that adjust accuracy scores based on observed speed of responding and we examine how well those measures work with different data sets. We then present a third approach that combines standardized speed and accuracy scores. We show how this latter approach can represent the data fairly well regardless of which (if any) speed-accuracy tradeoff occurs in the data. We also show how this measure can be further generalized by applying differential weightings to the standardized scores of speed, and accuracy, respectively. We conclude by discussing the value of the measure for use in analyzing human performance data where continuous indicators of accuracy or error can be collected or constructed relatively easily. Our goal in developing the global measure of performance is not to accurately model the speed-accuracy relationship, but rather to create a measure that is more sensitive to experimental differences and causal relationships than either speed or accuracy alone.

Systematic review and bivariate/HSROC random-effect meta-analysis of immunochemical and guaiac-based fecal occult blood tests for colorectal cancer screening.

Current literature evidences higher accuracy of immunological (iFOBT) vis-à-vis guaiac-based (gFOBT) fecal occult blood tests for colorectal cancer (CRC) screening. Few well-designed head-to-head comparisons exist. This meta-analysis assesses the performances of two iFOBTs compared with an established gFOBT using colonoscopy as the gold standard. We mobilized a bivariate and a hierarchical summary receiver operating characteristic (HSROC) model. Positive likelihood ratio (LR) and negative likelihood ratio (LR) and diagnostic odds ratios were back-calculated. We constructed bivariate credibility ellipses in the HSROC space and calculated areas under the curve to obtain a global measure of test performance. Estimates are presented at 95% credibility levels. We included and analyzed 21 studies. OC-Sensor was the best test for CRC screening, with high sensitivity (0.87; 95% credibility interval: 0.73-0.95) and specificity (0.93; 95% credibility interval: 0.84-0.96), optimal LR (12.01) and LR (0.14), and a high diagnostic odds ratio (88.05). Bivariate credibility ellipses showed OC-Sensor's dominance over Hemoccult (sensitivity: 0.47; 95% credibility interval: 0.37-0.58; specificity: 0.93; 95% credibility interval: 0.91-0.95). Our findings support the use of OC-Sensor for CRC detection. The diagnostic estimates obtained may be extended to derive model parameters for economic decision models and to offer insight for future clinical and public health decision making. Our findings could influence the future of FOBTs within the CRC screening arsenal.

Towards a Holistic Approach for Mutual Fund Performance Appraisal

There is a growing literature that employs nonparametric frontier methods in order to evaluate the performance of investment funds. This paper proposes an integrated approach for analyzing the efficiency and performance of mutual funds. The methodology combines data envelopment analysis (DEA) with a multicriteria decision aid methodology. DEA is employed to assess the relative efficiency of mutual funds in terms of their return, capital flow, gross expense ratio, turnover rate, and risk. In a second stage, a multicriteria approach is employed to develop an overall performance measure on the basis of the DEA efficiency results. The resulting model evaluates all mutual funds in a common basis and enables comparisons over time. The methodology is applied to a sample of more than 500 US equity mutual funds over the period 2003---2010. The analysis is implemented under three different time-window periods (one, three, and five year evaluations). The results obtained for portfolios constructed on the basis of our global performance measure, Sharpe ratio and Morningstar rating system are useful since they provide significant economic evidence in favor of our global performance measure. Our results entail practical implications for both investors and fund managers in terms of fund selection and efficient portfolio management respectively.

Monetary rewards modulate inhibitory control.

The ability to override a dominant response, often referred to as behavioral inhibition, is considered a key element of executive cognition. Poor behavioral inhibition is a defining characteristic of several neurological and psychiatric populations. Recently, there has been increasing interest in the motivational dimension of behavioral inhibition, with some experiments incorporating emotional contingencies in classical inhibitory paradigms such as the Go/NoGo and Stop Signal Tasks (SSTs). Several studies have reported a positive modulatory effect of reward on performance in pathological conditions such as substance abuse, pathological gambling, and Attention Deficit Hyperactive Disorder (ADHD). However, experiments that directly investigate the modulatory effects of reward magnitudes on the performance of inhibitory tasks are scarce and little is known about the finer grained relationship between motivation and inhibitory control. Here we probed the effect of reward magnitude and context on behavioral inhibition with three modified versions of the widely used SST. The pilot study compared inhibition performance during six blocks alternating neutral feedback, low, medium, and high monetary rewards. Study One compared increasing vs. decreasing rewards, with low, high rewards, and neutral feedback; whilst Study Two compared low and high reward magnitudes alone also in an increasing and decreasing reward design. The reward magnitude effect was not demonstrated in the pilot study, probably due to a learning effect induced by practice in this lengthy task. The reward effect per se was weak but the context (order of reward) was clearly suggested in Study One, and was particularly strongly confirmed in study two. In addition, these findings revealed a “kick start effect” over global performance measures. Specifically, there was a long lasting improvement in performance throughout the task when participants received the highest reward magnitudes at the beginning of the protocol. These results demonstrate a dynamical behavioral inhibition capacity in humans, as illustrated by the reward magnitude modulation and initial reward history effects.

EPA-1326 - Smoking and stimulants drugs use as variables related to more severe psychosis

Introduction The relationship between neuropsychological and overall performance in people with schizophrenia is known. Smoking and stimulant drugs use can improve neuropsychological outcomes, however the existence of drugs use may be a more severe illness marker. Objective The purpose of this study is to investigate which clinical and epidemiological variables, including stimulant drugs use and smoking, influence on neuropsychological performance in patients with psychosis. Material and methods 92 patients with different psychosis were assessed with a battery that included SCIP, to assess neuropsychological performance, PANSS, to evaluate psychotalogy, GAF and SIX as global performance measures. We also explore clinic and sociodemographic data. A binary logistic regresion model was applied on scores on the task ‘words’ of the SCIP (memory and learning), dichotomized at the median. The model included: sex, age, onset age, family history, negative scale, positive scale, global psychopatology and PANSS subtypes, estimated premorbid IQ using Barona Index, stimulant drugs use, smoking and funcionality. Results The absence of stimulating drugs use and smoking (trials 2 and 3 of SCIP), was associated with better memory and learning in patients with psychosis. Younger age (trials 1-4 and total of words), higher functionality (trials 1,2 and total of words) and premorbid IQ (trials 2 and 4) were also positively associated with better neuropsychological performance. Conclusions Stimulant drugs use and smoking could be markers of poorer previous neuropsychological function in psychotic patients. It is necessary to do longitudinal studies evaluating these variables as markers, risk or protective factors of cognitive performance

Match-to-match variation in physical activity and technical skill measures in professional Australian Football

To determine the match-to-match variability in physical activity and technical performance measures in Australian Football, and examine the influence of playing position, time of season, and different seasons on these measures of variability. Longitudinal observational study. Global positioning system, accelerometer and technical performance measures (total kicks, handballs, possessions and Champion Data rank) were collected from 33 players competing in the Australian Football League over 31 matches during 2011-2012 (N=511 observations). The global positioning system data were categorised into total distance, mean speed (mmin(-1)), high-speed running (>14.4 kmh(-1)), very high-speed running (>19.9 kmh(-1)), and sprint (>23.0 kmh(-1)) distance while player load was collected from the accelerometer. The data were log transformed to provide coefficient of variation and the between subject standard deviation (expressed as percentages). Match-to-match variability was increased for higher speed activities (high-speed running, very high-speed running, sprint distance, coefficient of variation %: 13.3-28.6%) compared to global measures (speed, total distance, player load, coefficient of variation %: 5.3-9.2%). The between-match variability was relativity stable for all measures between and within AFL seasons, with only few differences between positions. Higher speed activities (high-speed running, very high-speed running, sprint distance), but excluding mean speed, total distance and player load, were all higher in the final third phase of the season compared to the start of the season. While global measures of physical performance are relatively stable, higher-speed activities and technical measures exhibit a large degree of between-match variability in Australian Football. However, these measures remain relatively stable between positions, and within and between Australian Football League seasons.

Evidence Based Laboratory Medicine

Patients and society expect physicians to base their approach to any type of clinical problem on informed diagnostic reasoning. Informed diagnostics means that clinicians understand and readily apply the principles of diagnostic decision making, which include an estimate of the pre-test probability (prevalence) of diseases and information about the characteristics and discriminatory power of the applied investigations. Despite the crucial importance of the appropriate use of diagnostic tools in clinical decision-making, many diagnostic tests have not been subjected to rigorous evaluation to establish the diagnostic accuracy and clinical utility of laboratory investigations using modern standards of clinical epidemiology. The lack of good quality research in the field not only contributes to inappropriate utilization of laboratory services but also to wasting significant resources. Evidence-based laboratory medicine tries to combat this problem by combining methods from epidemiology, biostatistics, clinical and social sciences with basic sciences to evaluate the role of investigations in clinical decision making and outcomes for patients. Evidence-based laboratory medicine aims to advance clinical diagnosis and management of diseases through systematic researching and disseminating generalisable new knowledge which meets the standards of critical review on clinically effective practice of laboratory investigations. The main phases of practicing evidence-based laboratory medicine are 1. question formulation, 2. systematic literature search, 3. critical appraisal of literature, 4. implementation of evidence into practice, and 5. evaluation of impact in a clinical audit cycle. The use of evidence in laboratory medicine requires systematically compiled databases of standardised and critically appraised information on the test characteristics and diagnostic accuracy of laboratory investigations. Such an approach is the Bayes Library that provides information on the global and specific measures of test performance in different patient groups and settings, in addition to prevalence data of common diseases. The Bayes Library will support informed diagnostic decisions and improve patient outcome by integrating evidence-based medicine into the diagnostic service, education and training of laboratories. The process of constant questioning and reviewing the evidence for rational diagnosis of diseases provides a practical tool to identify gaps in our knowledge and thus it generates new research ideas in laboratory medicine.

Using negative feedback to guide behavior: Impairments on the first 4 cards of the Wisconsin Card Sorting Test predict negative symptoms of schizophrenia

Research has demonstrated that individuals with schizophrenia fail to appropriately use negative feedback to guide learning. These learning deficits are thought to arise from abnormalities in midbrain dopamine activity. Primary and enduring negative symptoms are also associated with abnormal dopamine activity and are expected to produce more severe deficits in learning when they present in individuals with schizophrenia. The current study examines this matter by comparing individuals with deficit syndrome schizophrenia, which is characterized by primary and enduring negative symptoms, to individuals with nondeficit syndrome schizophrenia and to normal controls in their use of positive feedback and negative feedback to guide learning on the first four cards of the WCST. Participants included 67 individuals with schizophrenia (15 deficit; 52 nondeficit syndrome) and 51 healthy controls. Accuracy data from the first 4 cards of the WCST and measures of global test performance were examined. Individuals with schizophrenia were significantly less accurate than controls in their performance on early (pre-shift) WCST trials, and this impairment was significantly greater in patients with deficit than nondeficit schizophrenia. Additionally, accuracy across the first 4 WCST cards significantly predicted the number of categories completed and percentage of perseverative errors across the entire test. These findings suggest that negative symptoms of schizophrenia are associated with difficulty using negative feedback to adaptively guide behavior, and are consistent with the notion that abnormal DA signaling contributes to the higher-order executive functioning impairments seen in schizophrenia with severe negative symptoms.

Evaluating geostatistical methods of blending satellite and gauge data to estimate near real-time daily rainfall for Australia

Rain gauges provide valuable information about the amount and frequency of rainfall. In Australia, the majority of rain gauges are located in populated, wet coastal regions. Approximately 2000 gauges reporting within 24 h of a target day were used to make near real-time (NRT) estimates of daily precipitation. The remaining approximate to 4000 gauges for the same target day were used to evaluate bias and estimation performance using several traditional statistics. There is considerable potential to improve the estimation of rainfall in Australia using related ancillary data, particularly in sparsely gauged areas. The Tropical Rainfall Measuring Mission (TRMM) Multisatellite Precipitation Analysis (TMPA-RT) near real-time product (3B42RT) provided images (0.25 degrees resolution) of precipitation across Australia. Daily precipitation was estimated in 2009110 approximately every 5 km across Australia. This study evaluated selected geostatistical methods for estimating daily rainfall maps across Australia. It tackled the change of support problem and spatial intermittency of daily rainfall data in blending satellite and gauge data. Dissension occurred amongst traditional global statistical measures of performance which were compounded by extremes of gauge density. Overall, our assessment is that blending the 3B42RT satellite and rain gauge data was not worthwhile. However, the blending considerably reduced the estimation variance indicating that uncertainty of the map estimates was a neglected property necessary to detect change and difference in patterns. (C) 2013 Elsevier B.V. All rights reserved.

Cardiopoietic Stem Cell Therapy in Heart Failure: The C-CURE (Cardiopoietic stem Cell therapy in heart failURE) Multicenter Randomized Trial With Lineage-Specified Biologics

This study sought to evaluate the feasibility and safety of autologous bone marrow-derived and cardiogenically oriented mesenchymal stem cell therapy and to probe for signs of efficacy in patients with chronic heart failure. In pre-clinical heart failure models, cardiopoietic stem cell therapy improves left ventricular function and blunts pathological remodeling. The C-CURE (Cardiopoietic stem Cell therapy in heart failURE) trial, a prospective, multicenter, randomized trial, wasconducted in patients with heart failure of ischemic origin who received standard of care or standard of care plus lineage-specified stem cells. In the cell therapy arm, bone marrow was harvested and isolated mesenchymal stem cells were exposed to a cardiogenic cocktail. Derived cardiopoietic stem cells, meeting release criteria underGood Manufacturing Practice, were delivered by endomyocardial injections guided by left ventricular electromechanical mapping. Data acquisition and analysis were performed in blinded fashion. The primary endpointwas feasibility/safety at 2-year follow-up. Secondary endpoints included cardiac structure/function and measures of global clinical performance 6 months post-therapy. Mesenchymal stem cell cocktail-based priming was achieved for each patient with the dose attained in 75% and delivery without complications in 100% of cases. There was no evidence of increased cardiac or systemic toxicity induced by cardiopoietic cell therapy. Left ventricular ejection fraction was improved by cell therapy (from 27.5 ± 1.0% to 34.5 ± 1.1%) versus standard of care alone (from 27.8 ± 2.0% to 28.0 ± 1.8%, p< 0.0001) and was associated with a reduction in left ventricular end-systolic volume (-24.8 ± 3.0 ml vs.-8.8 ± 3.9 ml, p< 0.001). Cell therapy also improved the 6-min walk distance (+62 ± 18 m vs.-15 ± 20 m, p< 0.01) and provided a superior composite clinical score encompassing cardiac parameters in tandem with New York Heart Association functional class, quality of life, physical performance, hospitalization, and event-free survival. The C-CURE trial implements the paradigm of lineage guidance in cell therapy. Cardiopoietic stem cell therapy wasfound feasible and safe with signs of benefit in chronic heart failure, meriting definitive clinical evaluation. (C-Cure Clinical Trial; NCT00810238).

Limit state capacities for global performance assessment of bridges exposed to hurricane surge and wave

Past failure of coastal bridges during hurricanes has revealed the vulnerability of these structures to wave and storm surge loading, thus signifying the need for reliability assessment of coastal bridges to provide a basis for hurricane risk assessment and mitigation. Such a reliability assessment requires probabilistic estimation of the capacity and demand of the structure, which are often conveniently decoupled to support risk assessment of existing bridges as well as performance based design or retrofit. This paper presents a new approach for probabilistic analysis of limit state capacities for bridge systems, in which local response quantities from numerical simulations are mapped to global performance measures, such as system strength and stiffness loss. Four different configurations of a case study bridge are analyzed to derive probabilistic models of the limit state capacities, revealing the significant bridge component demand parameters that contribute to the global (system) damage levels and the mathematical form of the limit state capacities, along with uncertainty quantification. Such models can be used in reliability assessment of coastal bridges and extended to evaluate the effectiveness of alternative retrofit measures proposed for unseating prevention. The proposed approach for global performance assessment complements emerging interests in multi-hazard analysis and system reliability of bridges by providing a new perspective on identifying the joint local damages which affect global behaviors that relate to practical decision making on post-event bridge functionality.

On identifying sparse representations of consensus networks

We consider the problem of identifying optimal sparse graph representations of dense consensus networks. The performance of the sparse representation is characterized by the global performance measure which quantifies the difference between the output of the sparse graph and the output of the original graph. By minimizing the sum of this performance measure and a sparsity-promoting penalty function, the alternating direction method of multipliers identifies sparsity structures that strike a balance between the performance measure and the number of edges in the graph. We then optimize the edge weights of sparse graphs over the identified topologies. Two examples are provided to illustrate the utility of the developed approach.

Pencina et al. Respond to "The Incremental Value of New Markers" and "Clinically Relevant Measures? A Note of Caution"

Pencina et al. Respond to "The Incremental Value of New Markers" and "Clinically Relevant Measures? A Note of Caution"

Variability in performance: Identifying early signs of future cognitive impairment.

The current study examined whether year-to-year variability in cognitive performance differ between individuals cognitively unimpaired and individuals who subsequently develop dementia. Analyses included a case-control sample of Baltimore Longitudinal Study of Aging (BLSA; mean [M] age = 69.90, standard deviation [SD] = 8.92) participants. One hundred and 35 clinically diagnosed demented participants were matched with 135 nondemented participants based on age at initial testing and sex. Cognitive performance was examined using measures of memory, executive function, attention, language, and global mental status performance. Cognitive performance was examined from baseline to 5 years before cognitive impairment (M, assessments = 3.03, SD = 2.80). As compared with unimpaired individuals, individuals diagnosed with dementia had greater variability on measures of attention, executive function, language, and semantic memory at least 5 years before the estimated onset of cognitive impairment, which may be indicative of maladaptive cognitive functioning. The dementia cases, however, had less variability on visual memory than the unimpaired group, which may suggest that these cases had more difficulty learning. These results demonstrate that performance variability indexed over annual or biennial visits may be useful in identifying early signs of subsequent cognitive impairment.

A model for supply chain design considering the cost of quality

Recent studies have shown that the cost of quality (COQ) is of more strategic and economic importance than previously conceived. Whereas previous works have applied COQ as an internal performance measure within companies, the purpose of this paper is to present a model for supply chain design that computes the COQ as a global performance measure for the entire supply chain. In addition, rather than assume an exogenously given COQ curve, our model computes COQ in terms of internal operational decisions such as the error rate at inspection and fraction defective at manufacturing. The model can be used to design a logistic route that achieves a minimum total cost while maintaining an overall quality level and to evaluate the impact of investment in quality to increase overall profits. The behaviour of the model is illustrated with numerical examples that show how the COQ function changes depending on various parameters.