Summary Metrics Research Articles

Evaluating a novel Test Site Analysis (TSA) bioassessment approach

AbstractWe compared the performance of Test Site Analysis (TSA) to 2 existing bioassessment methods: the BEnthic Assessment of SedimenT (BEAST), and the multimetric approach. In TSA, simple spreadsheet calculations are used to: 1) derive an overall multivariate measure of dissimilarity between a potentially impaired test site and a reference benchmark using a number of biological summary metrics, 2) classify each site as impaired, potentially impaired, or in reference condition with formal hypothesis tests, and 3) identify the metrics important in distinguishing a significantly impaired test site from the reference benchmark. We used data collected as part of the Fraser River Action Plan (British Columbia, Canada) to compare the biological condition of 15 potentially impaired test sites known to be exposed to agriculture, logging, or mining to the condition of 61 minimally disturbed reference sites with comparable habitat. When TSA was used, the false-positive (i.e., Type-1 error) and false-negative (i.e....

Accurately Defining the Reference Condition for Summary Biotic Metrics: A Comparison of Four Approaches

Protocols for bioassessment often relate changes in summary metrics that describe aspects of biotic assemblage structure and function to environmental stress. Biotic assessment using multimetric indices now forms the basis for setting regulatory standards for stream quality and a range of other goals related to water resource management in the USA and elsewhere. Biotic metrics are typically interpreted with reference to the expected natural state to evaluate whether a site is degraded. It is critical that natural variation in biotic metrics along environmental gradients is adequately accounted for, in order to quantify human disturbance-induced change. A common approach used in the IBI is to examine scatter plots of variation in a given metric along a single stream size surrogate and a fit a line (drawn by eye) to form the upper bound, and hence define the maximum likely value of a given metric in a site of a given environmental characteristic (termed the 'maximum species richness line' - MSRL). In this paper we examine whether the use of a single environmental descriptor and the MSRL is appropriate for defining the reference condition for a biotic metric (fish species richness) and for detecting human disturbance gradients in rivers of south-eastern Queensland, Australia. We compare the accuracy and precision of the MSRL approach based on single environmental predictors, with three regression-based prediction methods (Simple Linear Regression, Generalised Linear Modelling and Regression Tree modelling) that use (either singly or in combination) a set of landscape and local scale environmental variables as predictors of species richness. We compared the frequency of classification errors from each method against set biocriteria and contrast the ability of each method to accurately reflect human disturbance gradients at a large set of test sites. The results of this study suggest that the MSRL based upon variation in a single environmental descriptor could not accurately predict species richness at minimally disturbed sites when compared with SLR's based on equivalent environmental variables. Regression-based modelling incorporating multiple environmental variables as predictors more accurately explained natural variation in species richness than did simple models using single environmental predictors. Prediction error arising from the MSRL was substantially higher than for the regression methods and led to an increased frequency of Type I errors (incorrectly classing a site as disturbed). We suggest that problems with the MSRL arise from the inherent scoring procedure used and that it is limited to predicting variation in the dependent variable along a single environmental gradient.

Temporal response of stream benthic macroinvertebrate communities to the synergistic effects of anthropogenic acidification and natural drought events

Summary1. We evaluated whether the surprisingly weak biological recovery associated with declines in acid deposition were related to drought‐induced re‐acidification of streams. We used test site analysis (TSA) to characterise temporal changes (1995–2003) in the degree and nature of impairments to stream benthic macroinvertebrate (BMI) communities influenced by acid deposition and drought.2. The BMI communities in four historically impacted test streams were compared with communities in 30 minimally impacted reference streams. Six multivariate (e.g. ordination axes scores) and four traditional (e.g. % Diptera) summary metrics were used to describe BMI communities. Using all metrics simultaneously (i.e. Mahalanobis or generalised distance), the TSA provided a single probability that a test community was impaired. If a test community was significantly impaired, a further analysis was done to identify the metric(s) important in distinguishing the test community from reference condition.3. Results of the TSAs indicated that the generalised distances between test communities and the reference condition were inversely related to stream water pH (n = 36). The TSAs also indicated ordination metrics based on BMI abundance were important in distinguishing significantly impaired communities from reference conditions. Temporal trends indicated that there has been short‐term recovery of these BMI communities, but that overall improvements have been hampered by acid or metal toxicity associated with drought‐induced re‐acidification of the streams.4. Our use of a variety of summary metrics to obtain a single statistical test of significance within the context of the reference‐condition approach provided a simple and unambiguous framework for evaluating the biological condition of test sites.

The impact of risk status, preexisting morbidity, and polypharmacy on treatment decisions concerning menopausal symptoms

Because each menopausal treatment has a unique benefit–risk profile, deciding whether menopausal hormone therapy (HT) or another menopausal treatment is appropriate for an individual requires assessing the risks and benefits of each treatment as well as determining the patient's risk factors, comorbidities, and current medications. Simple summary metrics for assessing net treatment effects on multiple outcomes, such as the Women's Health Initiative's (WHI) global index, are not necessarily generalizable beyond the study population in which they were derived. However, trial evidence can be translated to individual decisions by transforming relative risks into absolute risks. The validity of generalizing relative risks found in clinical trials to groups with differing baseline risks depends on the homogeneity of relative risks across various risk strata. Subgroup analyses from the major clinical studies of HT found its relative risks on cardiovascular disease, hip fracture, breast cancer, colorectal cancer, and ovarian cancer to be relatively uniform across strata defined by age, race/ethnicity, antecedent risk status, or prior disease. The number of significant subgroup findings in these studies tended to match the number expected by chance alone. However, data are limited for many subgroups, especially those with comorbidities. Strata defined by the concurrent use of drugs that could interact with HT would not be expected to be homogeneous, yet data on drug interactions are limited. More data are needed about the effects of menopausal treatments in diverse populations, and more attention is needed to translate this evidence into clinical practice and to develop tools to support informed decision making.

The impact of risk status, preexisting morbidity, and polypharmacy on treatment decisions concerning menopausal symptoms

Because each menopausal treatment has a unique benefit-risk profile, deciding whether menopausal hormone therapy (HT) or another menopausal treatment is appropriate for an individual requires assessing the risks and benefits of each treatment as well as determining the patient's risk factors, comorbidities, and current medications. Simple summary metrics for assessing net treatment effects on multiple outcomes, such as the Women's Health Initiative's (WHI) global index, are not necessarily generalizable beyond the study population in which they were derived. However, trial evidence can be translated to individual decisions by transforming relative risks into absolute risks. The validity of generalizing relative risks found in clinical trials to groups with differing baseline risks depends on the homogeneity of relative risks across various risk strata. Subgroup analyses from the major clinical studies of HT found its relative risks for cardiovascular disease, hip fracture, breast cancer, colorectal cancer, and ovarian cancer to be relatively uniform across strata defined by age, race/ethnicity, antecedent risk status, or prior disease. The number of significant subgroup findings in these studies tended to match the number expected by chance alone. However, data are limited for many subgroups, especially those with comorbidities. Strata defined by the concurrent use of drugs that can interact with HT are not expected to be homogeneous, yet data on drug interactions are limited. More data are needed about the effects of menopausal treatments in diverse populations, and more attention is needed to translate this evidence into clinical practice and to develop tools to support informed decision making.

Assessing the learning curve for the acquisition of laparoscopic skills on a virtual reality simulator

The aim of this study was to develop summary metrics and assess the construct validity for a virtual reality laparoscopic simulator (LapSim) by comparing the learning curves of three groups with different levels of laparoscopic expertise. Three groups of subjects ('expert', 'junior', and 'naïve') underwent repeated trials on three LapSim tasks. Formulas were developed to calculate scores for efficiency ('time-error') and economy of 'motion' ('motion') using metrics generated by the software after each drill. Data (mean +/- SD) were evaluated by analysis of variance (ANOVA). Significance was set at p < 0.05. All three groups improved significantly from baseline to final for both 'time-error' and 'motion' scores. There were significant differences between groups in time error performances at baseline and final, due to higher scores in the 'expert' group. A significant difference in 'motion' scores was seen only at baseline. We have developed summary metrics for the LapSim that differentiate among levels of laparoscopic experience. This study also provides evidence of construct validity for the LapSim.

FLORISTIC QUALITY INDICES FOR BIOTIC ASSESSMENT OF DEPRESSIONAL MARSH CONDITION IN FLORIDA

Evaluation of wetland ecological condition requires quantitative biological indices for measuring anthropogenic impairment. We implemented a modified floristic quality assessment index (FQAI) protocol for 75 isolated, depressional herbaceous wetland systems, exploring refinements of FQAI standard methods. Species encountered during sampling (n = 397) were assigned coefficients of conservatism (CC) by ten expert botanists working independently. A quantitative summary metric of adjacent site buffer (up to 100 m) land use intensity, called the landscape development intensity (LDI) index, was calculated for each wetland system to quantify expected anthropogenic impairment. The association between LDI and wetland community mean CC scores was strong and conditionally independent of ecoregion. Weaker associations with LDI were observed for other community summary metrics, including richness‐weighted FQAI. We inverted LDI to compute an intensity coefficient (IC), which quantifies observed buffer development intensity tolerated by each species. IC scores were significantly associated with CC scores on a species basis and strongly associated on a site mean basis. Growing interest in floristic quality assessment for regulatory purposes provides opportunities for formally linking expert opinion and ground observations of species‐specific disturbance tolerance.

What accuracy statistics really measure

Provides the software estimation research community with a better understanding of the meaning of, and relationship between, two statistics that are often used to assess the accuracy of predictive models: the mean magnitude relative error (MMRE) and the number of predictions within 25% of the actual, pred(25). It is demonstrated that MMRE and pred(25) are, respectively, measures of the spread and the kurtosis of the variable z, where z=estimate/actual. Thus, z is considered to be a measure of accuracy, and statistics such as MMRE and pred(25) to be measures of properties of the distribution of z. It is suggested that measures of the central location and skewness of z, as well as measures of spread and kurtosis, are necessary. Furthermore, since the distribution of z is non-normal, non-parametric measures of these properties may be needed. For this reason, box-plots of z are useful alternatives to simple summary metrics. It is also noted that the simple residuals are better behaved than the z variable, and could also be used as the basis for comparing prediction systems.