Categorical Metrics Research Articles

Extended contingency table: Performance metrics for satellite observations and climate model simulations

[1] Validation of gridded satellite observations and climate model simulations are fundamental to future improvements in retrieval algorithms and model developments. Among the metrics, the contingency table, which includes a number of categorical indices, is extensively used in evaluation studies. While the categorical indices offer invaluable information, they do not provide any insight into the volume of the variable detected correctly/incorrectly. In this study, the contingency table categorical metrics are extended to volumetric indices for evaluation of gridded data. The suggested indices include (a) Volumetric Hit Index (VHI): volume of correctly detected simulations relative to the volume of the correctly detected simulations and missed observations; (b) Volumetric False Alarm Ratio (VFAR): volume of false simulations relative to the sum of simulations; (c) Volumetric Miss Index (VMI): volume of missed observations relative to the sum of missed observations and correctly detected simulations; and (d) the Volumetric Critical Success Index (VCSI). The latter provides an overall measure of volumetric performance including volumetric hits, false alarms, and misses. First, using two synthetic time series, the volumetric indices are evaluated against the contingency table categorical metrics. Then, the volumetric indices are used to evaluate a gridded data set at the continental scale. The results show that the volumetric indices provide additional information beyond the commonly used categorical metrics that can be useful in evaluating gridded data sets.

Phi-square Lexical Competition Database (Phi-Lex): An online tool for quantifying auditory and visual lexical competition

A widely agreed-upon feature of spoken word recognition is that multiple lexical candidates in memory are simultaneously activated in parallel when a listener hears a word, and that those candidates compete for recognition (Luce, Goldinger, Auer, & Vitevitch, Perception 62:615-625, 2000; Luce & Pisoni, Ear and Hearing 19:1-36, 1998; McClelland & Elman, Cognitive Psychology 18:1-86, 1986). Because the presence of those competitors influences word recognition, much research has sought to quantify the processes of lexical competition. Metrics that quantify lexical competition continuously are more effective predictors of auditory and visual (lipread) spoken word recognition than are the categorical metrics traditionally used (Feld & Sommers, Speech Communication 53:220-228, 2011; Strand & Sommers, Journal of the Acoustical Society of America 130:1663-1672, 2011). A limitation of the continuous metrics is that they are somewhat computationally cumbersome and require access to existing speech databases. This article describes the Phi-square Lexical Competition Database (Phi-Lex): an online, searchable database that provides access to multiple metrics of auditory and visual (lipread) lexical competition for English words, available at www.juliastrand.com/phi-lex .

Multitrial evaluation of longitudinal tumor measurement (TM)-based metrics for predicting overall survival (OS) using the RECIST 1.1 data warehouse.

6520 Background: We previously reported that alternative cutpoints and alternate categorical metrics to RECIST standards provided no meaningful improvement in OS prediction. Here, we sought to identify and validate continuous, longitudinal TM-based metrics for predicting OS. Methods: Data from 13 trials (1739 patients with sufficient imaging-based measurements from breast, lung or colon cancer) were randomly split (60:40) into training and validation datasets and landmarked at 12-weeks (w). Multiple candidate metrics that capture tumor curve trajectory were considered: first slope (or % change) from baseline to 6 w, last slope (or % change) from 6 w to 12 w, and indicator of increase (<5% (includes no increase or a decrease) vs. ≥5%) from the prior assessment (baseline to 6 w, 6 w to 12 w). Multivariable Cox models stratified by study and number of baseline target lesions (<3 vs. >3) and adjusted for baseline tumor burden (ratio of the sum of lesions/number of lesions) were fit for each tumor type. Predictive ability was assessed via concordance (c)-index (0.5: no association to 1.0: perfect association), and hazard ratios/p-values (HR/p). Results: Patients whose tumor size increased by at least 5% from the prior assessment, as well as whose tumors showed a positive slope (or % change) from 6 to 12 w (indicating tumor growth) had worse OS. The first slope (or % change) metric was less consistent, with an initial decrease in slope suggesting worse OS in some cases. Predictive ability of these metrics varied across tumor type with poorest prediction in Breast (Table). Conclusions: Continuous TM-based metrics performed no better than categorical RECIST metrics for OS prediction. [Table: see text]

Abstract A72: Occupational sunlight exposure, race, and risk of renal cancer

Abstract Introduction: Vitamin D, obtained primarily from ultraviolet (UV) exposure, is hydroxylated to its biologically active form within the kidneys. Epidemiological evidence supporting a relationship between vitamin D and kidney cancer risk has been inconsistent despite experimental data indicating that vitamin D and its metabolites may inhibit carcinogenesis. Previously we reported an inverse association between renal cell carcinoma (RCC) risk and occupational UV exposure among men in Europe. In the current study, we examined the association between occupational UV exposure and RCC risk among U.S. residents and investigated whether this association varies by race and sex. Methods: Lifetime occupational data for 1,217 RCC cases and 1,235 controls in a large population-based case-control study were blindly assessed by an industrial hygiene expert to create a job exposure matrix that categorized frequency, duration, confidence, and intensity of occupational UV exposure. Categorical exposure metrics used to evaluate exposure-response relationships with occupational exposure were based on tertiles of exposure levels among controls. Unconditional logistic regression was used to calculate odds ratios (ORs) and 95% confidence intervals (CIs) to examine the association between RCC risk and UV exposure by sex and race. Results: Across all exposure-response matrices, women in the highest tertile of occupational UV exposure had a 40% reduction in RCC risk compared to those in the lowest tertile; risk decreased significantly as occupational UV exposure increased (p-trend=0.01 to 0.004). A marginally significant, non-linear inverse association was observed among men. Further stratification by race revealed similar findings for black and white women. Significantly decreased associations between RCC risk and occupational UV exposure were observed among black men (p-trend=0.05), but not among white men, although results for blacks were based on small numbers. Our findings suggest that occupational UV exposure may protect against RCC. We caution any inferences made using race, given power limitations for stratified analyses, until additional well-powered studies can replicate and extend these findings. Citation Information: Cancer Epidemiol Biomarkers Prev 2011;20(10 Suppl):A72.

Preservation of Spatial Information in Rasterization Processes: A Practical Approach Using Real Categorical Data and Landscape Metrics

Rasterization (i.e., vector-to-raster conversion) is a common process in geographic information systems. As it implies redistribution of map information, it is important to know how much data are lost when a particular resolution is chosen, in order to use data successfully. This study explores the potential of landscape metrics to reflect sensitivity to rasterization processes. A vectorial land use map was rasterized to an array of cell sizes. Deviation from original data was estimated, and landscape metrics were calculated for each land cover class. Results allowed grouping of classes—with respect to how information was preserved—depending on their landscape characteristics.

Evaluation of Trial Outcomes in Acute Kidney Injury by Creatinine Modeling

Clinical trials of acute kidney injury (AKI) use changes in creatinine as outcome metrics. This study investigated how outcome metrics and baseline creatinine affect trial outcome. A one-compartment pharmacokinetic model of creatinine change resulting from a decrease in GFR was applied to a population of 10,000 simulated virtual inpatients. Treatment was simulated as an amelioration of GFR decrease by a specified percentage, the treatment efficacy, in 50%. Three categorical and two continuous outcome metrics were calculated and compared. Outcomes were compared for measured and estimated baseline creatinine levels that were back-calculated assuming a GFR of 100 or 75 ml/min. The continuous metrics, the average value of creatinine and the average value of creatinine relative to baseline decreased approximately linearly with increase in treatment efficacy. The categorical metrics displayed a sigmoidal decrease and erroneously suggested perfect treatment when GFR decrease was ameliorated by only 60 to 80%. Using an estimate of baseline creatinine increased the number of patients who were classified as having AKI. When used to determine clinical trial outcome, continuous metrics correctly detected the extent of intervention. At low treatment efficacy, categorical metrics underestimated and at high treatment efficacy overestimated the effect of treatment. These effects were exaggerated when the population contained a high proportion of patients with more severe AKI. An estimated baseline creatinine level will overestimate AKI prevalence compared with a measured baseline value. Clinical trials of AKI should use a continuous outcome metric and a measured baseline and report baseline median and interquartile range.

Performance evaluation of NOAA-EPA developmental aerosol forecasts

To aid air quality model development and assess air quality forecasts, the Meteorological Development Laboratory (MDL) provided categorical verification metrics for developmental aerosol predictions. The National Air Quality Forecasting Capability (NAQFC) generated 48 h (of) gridded hourly developmental predictions for the lower 48 states (CONUS) domain in 12 km horizontal spacing. The NAQFC uses the North American Mesoscale (NAM) model with EPA’s Community Multiscale Air Quality (CMAQ) model to produce predictions of ground level aerosol concentrations. We used bilinear interpolation to calculate predicted daily maximum values at the location of the observation sites. We compared these interpolated predicted values to the observed daily maximum to produce 2 × 2 contingency tables, with a threshold of 40 μg/m3 during the months of March–August, 2007. The model showed some degree of skill in predicting aerosol exceedances. These results are preliminary as the NAQFC model for aerosol prediction is in the developmental stage. A more comprehensive performance evaluation will be accomplished in 2008, when more data become available. Our verification metrics included categorical analyses for Fraction Correct (FC) or percent correct (FC × 100), Threat Score (TS) or Critical Success Index (CSI), Probability of Detection (POD), and the False Alarm Rate (FAR), Mean Absolute Error (MAE) and mean algebraic error or bias, where bias is forecast minus observation. Graphic products included weekly statistics for the CONUS displayed in the form of bar charts, scatterplots, and graphs. In addition, we split the CONUS into six geographic regions and provided regional statistics on a monthly basis. MDL produced spatial maps of daily 1-h maximum predicted aerosol values overlaid with the corresponding point observations. MDL also provided spatial maps of the daily maximum of the 24-h running average. We derived the 24-h running average from the 1-h average predicted aerosol values and observations.

P.3.c.059 Post-injection delirium/sedation syndrome observed with olanzapine long-acting injection: review of the first 25 events

Bipolar depression is more pervasive than mania, but has fewer evidence-based treatments.Using data from multicenter, randomized, double-blind, placebo-controlled trials and meta-analyses, we assessed the number needed to treat (NNT) for response and the number needed to harm (NNH) for selected side effects for older and newer acute bipolar depression treatments.The 2 older FDA-approved treatments for bipolar depression, olanzapine-fluoxetine combination (OFC) and quetiapine (QTP) monotherapy, were efficacious (response NNT=4 for OFC, NNT=6 for QTP), but similarly likely to yield harms (OFC weight gain NNH=6; QTP sedation/somnolence NNH=5). Commonly used unapproved agents (lamotrigine monotherapy and adjunctive antidepressants) tended to be well-tolerated (with double-digit NNHs), although this advantage was at the cost of inadequate efficacy (response NNT=12 for lamotrigine, NNT=29 for antidepressants). In contrast, the newly approved agent lurasidone was not only efficacious (response NNT=5 for monotherapy, NNT=7 as adjunctive therapy), but also had enhanced tolerability (NNH=15 for akathisia [monotherapy], NNH=16 for nausea [adjunctive]). Although adjunctive armodafinil appeared well tolerated, its efficacy in bipolar depression has not been consistently demonstrated in randomized controlled trials.NNT and NNH are categorical metrics; only selected NNHs were assessed; limited generalizability of efficacy (versus effectiveness) studies.For acute bipolar depression, older approved treatments may have utility in high-urgency situations, whereas lamotrigine and antidepressants may have utility in low-urgency situations. Newly approved lurasidone may ultimately prove useful in diverse situations. New drug development needs to focus on not only efficacy but also on tolerability.

New Categorical Metrics for Air Quality Model Evaluation

Abstract Traditional categorical metrics used in model evaluations are “clear cut” measures in that the model’s ability to predict an “exceedance” is defined by a fixed threshold concentration and the metrics are defined by observation–forecast sets that are paired both in space and time. These metrics are informative but limited in evaluating the performance of air quality forecast (AQF) systems because AQF generally examines exceedances on a regional scale rather than a single monitor. New categorical metrics—the weighted success index (WSI), area hit (aH), and area false-alarm ratio (aFAR)—are developed. In the calculation of WSI, credits are given to the observation–forecast pairs within the observed exceedance region (missed forecast) or the forecast exceedance region (false alarm), depending on the distance of the points from the central line (perfect observation–forecast match line or 1:1 line on scatterplot). The aH and aFAR are defined by matching observed and forecast exceedances within an area (i.e., model grid cells) surrounding the observation location. The concept of aH and aFAR resembles the manner in which forecasts are usually issued. In practice, a warning is issued for a region of interest, such as a metropolitan area, if an exceedance is forecast to occur anywhere within the region. The application of these new categorical metrics, which are supplemental to the traditional counterparts (critical success index, hit rate, and false-alarm ratio), to the Eta Model–Community Multiscale Air Quality (CMAQ) forecast system has demonstrated further insight into evaluating the forecasting capability of the system (e.g., the new metrics can provide information about how the AQF system captures the spatial variations of pollutant concentrations).

Applying Cathegorical Metrics on Fuzzy Systems

This work analyzes the categorical metrics usage on a very specific subset of Intelligent Systems: Fuzzy Systems. Several characteristics for such systems must be carefully evaluated when metrics and indicators are defined, in order to consider the fuzzy essence as part of the evaluation result. A set of metrics and indicators are defined and applied to the classical inverted pendulum problem. The paper does not intend to provide an exhaustive analysis of the quality evaluation on soft computing problem. It just presents a way to start the study of quality measure in that area.

An operational evaluation of the Eta–CMAQ air quality forecast model

The National Oceanic and Atmospheric Administration (NOAA), in partnership with the United States Environmental Protection Agency (EPA), are developing an operational, nationwide Air Quality Forecasting (AQF) system. An experimental phase of this program, which couples NOAA's Eta meteorological model with EPA's Community Multiscale Air Quality (CMAQ) model, began operation in June of 2004 and has been providing forecasts of ozone (O 3) concentrations over the northeastern United States. An important component of this AQF system has been the development and implementation of an evaluation protocol. Accordingly, a suite of statistical metrics that facilitates evaluation of both discrete- and categorical-type forecasts was developed and applied to the system in order to characterize its performance. The results reveal that the AQF system performed reasonably well in this inaugural season (mean domain wide correlation coefficient=0.59), despite anomalously cool and wet conditions that were not conducive to the formation of O 3. Due in part to these conditions, the AQF system overpredicted concentrations, resulting in a mean bias of +10.2 ppb (normalized mean bias=+22.8%). In terms of error, the domain-wide root mean square error averaged 15.7 ppb (normalized mean error=28.1%) for the period. Examination of the discrete and categorical metrics on a daily basis revealed that the AQF system's level of performance was closely related to the synoptic-scale meteorology impacting the domain. The model performed very well during periods when anticyclones, characterized by clear skies, dominated. Conversely, periods characterized by extensive cloud associated with fronts and/or cyclones, resulted in poor model performance. Subsequent analysis revealed that factors associated with CMAQ's cloud cover scheme contributed to this overprediction. Accordingly, changes to the cloud schemes are currently underway that are expected to significantly improve the AQF system's performance in anticipation of its second year of operation.