Best Professional Judgment Research Articles

Reference values of the physico-chemical parameters of the water streams of Tai national Park (Côte d'Ivoire)

water in West Africa is based on European standards which are inadequate. This study was carried out in order to determine the threshold values of physico-chemical parameters for the monitoring of water quality in tropical forest rivers in West Africa. Thus, physico-chemical parameters were measured in 324 surface waters collected in three typical rivers of Taï National Park (Côte d’Ivoire) from May 2016 to April 2017. Subsequently, the sampling stations were categorised as natural, transitional and impacted based on a principal component analysis (PCA). The results confirmed the typology of sampling stations reported in the literature. Based on best professional judgment (BPJ) and trisection methods, the reference values for Tai National Park are Temperature: 25.10-25.33°C; pH: 6.86 - 7.04; DO: 6.33 - 6.40 mg/L; EC: 37. 13 - 38.20 μS/cm; TSD: 27.50 - 29.03 mg/L; NO2- : 0.005 - 0.009 mg/L; NO3- : 0.450 - 0.625 mg/L; PO42- : 0.053 - 0.054 mg/L; TP: 0.123 - 0.134 mg/L; FCl: 0.03 - 0.04 mg/L; TCl: 0.04 - 0.05 mg/L. These data could be used as threshold values for monitoring rivers and rainforest wetlands in West Africa.

Developing ocean climate change indicators for the north-central California coast and ocean

The Ocean Climate Indicators Project, developed for the Greater Farallones National Marine Sanctuary (GFNMS), yielded the first set of physical and biological ocean climate indicators specifically developed for the north-central California coast and ocean region, which extends from Point Arena to Point Año Nuevo and includes the ocean shorelines of the San Francisco metropolitan area. This case study produced a series of physical and biological indicator categories through a best professional judgment (BPJ) process with an interdisciplinary group of over 50 regional research scientists and marine resource managers from a wide range of state and federal agencies, NGOs, and universities. A working group of research scientists and marine resource managers used this set of ocean climate indicators to develop the Ocean Climate Indicators Monitoring Inventory and Plan. The Plan includes monitoring goals and objectives common for eight physical and four biological indicators; specific goals for each indicator; monitoring strategies and activities; an inventory of available monitoring data; opportunities for expanding or improving existing or new monitoring approaches; and case studies with specific examples of the indicators’ utility for natural resource management and basic scientific research. Beyond developing indicators that support effective science-based management decisions, this scalable process established and strengthened mutually beneficial connections between scientists and managers, resulting in indicators that had broad support of project participants, were quickly adopted by the GFNMS, and could be used by managers and scientists from this region and beyond.

Evaluation of Four Parameterization Strategies for the APEX Model

Abstract.State-of-the-art model parameterization consists of calibrating and validating the model using monitored data. When data are not available, many studies have relied on alternate strategies, including parameter sets obtained for eco-hydrologically similar watersheds and best professional judgment (BPJ). The objectives of this study were to (1) test the accuracy of four APEX parameterization strategies to predict runoff quantity and quality in a small agricultural watershed and (2) compare the effects of the different parameter sets on relative and absolute water quality outcomes for different conservation practice scenarios. A BPJ and three parameter sets obtained on nearby and more distant sites, including two fully calibrated parameter sets, i.e., for runoff, sediment, and total phosphorus (TP), and one partially calibrated parameter set, i.e., for runoff and TP only, were evaluated based on edge-of-field runoff, sediment loss, and TP loss, as well as for long-term annual predictions and relative changes for six conservation practice scenarios. Only the parameter sets that were fully calibrated met the model performance criteria on the test watershed and produced similar 30-year average annual predictions for the conservation practices. Relative changes in conservation outcomes were similar for the fully and partially calibrated parameter sets. They were different for the BPJ parameter set for at least one conservation practice. In the absence of site-specific data, the best parameterization strategy was to use parameter sets from a model calibrated for runoff, sediment, and nutrient losses from an eco-hydrologically similar site. Partial calibration of the model was sufficient to obtain consistent relative effects of the conservation practices. Keywords: Conservation practices, Hydrologic modeling, Model parameterization, Phosphorus, Sediment.

Multisite Evaluation of APEX for Water Quality: I. Best Professional Judgment Parameterization.

The Agricultural Policy Environmental eXtender (APEX) model is capable of estimating edge-of-field water, nutrient, and sediment transport and is used to assess the environmental impacts of management practices. The current practice is to fully calibrate the model for each site simulation, a task that requires resources and data not always available. The objective of this study was to compare model performance for flow, sediment, and phosphorus transport under two parameterization schemes: a best professional judgment (BPJ) parameterization based on readily available data and a fully calibrated parameterization based on site-specific soil, weather, event flow, and water quality data. The analysis was conducted using 12 datasets at four locations representing poorly drained soils and row-crop production under different tillage systems. Model performance was based on the Nash-Sutcliffe efficiency (NSE), the coefficient of determination () and the regression slope between simulated and measured annualized loads across all site years. Although the BPJ model performance for flow was acceptable (NSE = 0.7) at the annual time step, calibration improved it (NSE = 0.9). Acceptable simulation of sediment and total phosphorus transport (NSE = 0.5 and 0.9, respectively) was obtained only after full calibration at each site. Given the unacceptable performance of the BPJ approach, uncalibrated use of APEX for planning or management purposes may be misleading. Model calibration with water quality data prior to using APEX for simulating sediment and total phosphorus loss is essential.

Evaluating the APEX Model for Simulating Streamflow and Water Quality on Ten Agricultural Watersheds in the U.S.

To better understand the Agricultural Policy Environmental Extender (APEX) model’s strengths and limitations in simulating streamflow and water quality, we evaluated its performance under a diverse range of climatic, topographic, soil, cover, and land management conditions using three parameter settings: best professional judgment (BPJ), partially calibrated, and fully calibrated. A total of 18 calibration parameters governing streamflow, crop yield, and sediment, N, and P were adjusted. Hydrologic and water quality responses were simulated on ten USDA-ARS watersheds with heterogeneous forested, pasture/range, and/or corn-soybean cropping systems. Based on percent bias and Nash-Sutcliffe coefficient of efficiency test statistics, 0%, 0%, 15%, and 9% of the data sets were considered satisfactory or better in simulating monthly streamflow, sediment, N, and P, respectively, in the BPJ mode. However, when used with the fully calibrated settings, 73%, 43%, 38%, and 27% of the data sets were satisfactory or better. Findings indicate that when fully calibrated, APEX estimates streamflow very well, sediment and nitrogen moderately well, and phosphorus marginally well at a monthly time scale. However, when using the BPJ approach, APEX does not estimate watershed-level streamflow, sediment, or nutrients accurately at a monthly time step. Additionally, APEX lacks an element of robustness in simulating streamflow and water quality constituents when applied to the same watershed with a different period of record or to a nearby watershed with a similar period of record. Based on these findings, it is recommended that users exercise caution when employing APEX with the BPJ approach or in validation mode at watershed scales.

Evaluating ecological states of rocky intertidal communities: A Best Professional Judgment exercise

A Best Professional Judgment (BPJ) exercise was performed to determine the level of agreement among experts in evaluating the ecological states of western North American rocky intertidal communities. Species-abundance and environmental data from 12 central and 11 southern California sites were provided to 14 experts who independently ranked communities from best to worst and assigned each to one of five categories based on the degree of deviation from an expected natural biological state. Experts achieved Spearman correlations of 0.49 (central California) and 0.30 (southern California) in their rankings and averaged 75.4% and 70.0% Euclidean Similarity (ES) in their community evaluations. These ES values compare favorably with agreement levels found for similar exercises with soft bottom macroinvertebrate assemblages. The experts emphasized macrophytes with functional characteristics related to morphology and sessile macroinvertebrates in their assessments. Several challenges were noted in interpreting rocky intertidal data sets, the most prominent of which are high spatial and temporal variation and site-to-site differences in natural disturbance regimes, features that lead to multiple, expected community states. Experts required detailed, physical habitat descriptions to develop community composition expectations that differed for different shore types, and expressed concern about evaluating rocky intertidal communities based on only a single sampling event. Distinguishing natural from anthropogenic disturbance without information on the sources and magnitudes of anthropogenic perturbation was also found to be challenging because the biological responses to these stressors are often similar. This study underscores the need for long-term data sets that describe the dynamics of populations and communities and rigorous testing of expert judgments to firmly establish broadly applicable and consistent links between community states and anthropogenic stressors on rocky shores.

Testing the use of best professional judgment to create biological benchmarks for habitat assessment of wetlands and oak savannas in northwestern Indiana

The Tolleston Strandplain at the southern end of Lake Michigan offers a unique “dune and swale” topography supporting oak savannas on the dunes and a mosaic of wetland communities in the swales. Following years of human degradation, sites in this area are now being restored. In this effort, assessments of vegetative quality have been necessary for proper management decisions. However, it is poorly understood what indices best reflect the vegetative quality of these oak savannas and wetlands. A potential method for determining the best indices for these community types is to use metric benchmarks that employ expert best professional judgment (BPJ). In order to confirm the viability and consistency of BPJ for creating benchmarks, Kappa analysis was used to determine the level of agreement among seven experts. They placed each of 63 transects from this unique landscape into one of four quality categories: (1) “good to very good,” (2) “medium,” (3) “poor,” and (4) “very poor.” While experts had good agreement about the quality of severely degraded riverine wetlands, they had fair agreement when assessing the swales, and poor agreement when assessing the oak savannas. Using discriminant analysis, follow-up questions, and a comparison of each expert's quality categories with remnant oak savanna metrics, varying perspectives of quality also were discovered which may have influenced the experts’ assessments of the sites. Therefore, BPJ must be used with caution when creating metric benchmarks.

Validation of Rapid Assessment Methods to Determine Streamflow Duration Classes in the Pacific Northwest, USA

United States Supreme Court rulings have created uncertainty regarding U.S. Clean Water Act (CWA) authority over certain waters, and established new data and analytical requirements for determining CWA jurisdiction. Thus, rapid assessment methods are needed that can differentiate between ephemeral, intermittent, and perennial streams. We report on the validation of several methods. The first (Interim Method) was developed through best professional judgment (BPJ); an alternative (Revised Method) resulted from statistical analysis. We tested the Interim Method on 178 study reaches in Oregon, and constructed the Revised Method based on statistical analysis of the Oregon data. Next, we evaluated the regional applicability of the methods on 86 study reaches across a variety of hydrologic landscapes in Washington and Idaho. During the second phase, we also compared the Revised Method with a similar approach (Combined Method) based on combined field data from Oregon, Washington, and Idaho. We further compared field-based methods with a GIS-based approach (GIS Method) that used the National Hydrography Dataset and a synthetic stream network. Evaluations of all methods compared results with actual streamflow duration classes. The Revised Method correctly determined known streamflow duration 83.9% of the time, versus 62.3% accuracy of the Interim Method and 43.6% accuracy for the GIS-based approach. The Combined Method did not significantly outperform the Revised Method. Analysis showed biological indicators most accurately discriminate streamflow duration classes. While BPJ established a testable hypothesis, this study illustrates the importance of quantitative field testing of rapid assessment methods. Results support a consistent method applicable across the Pacific Northwest.

Defining the reference condition for wadeable streams in the Sand Hills subdivision of the Southeastern Plains ecoregion, USA.

The Sand Hills subdivision of the Southeastern Plains ecoregion has been impacted by historical land uses over the past two centuries and, with the additive effects of contemporary land use, determining reference condition for streams in this region is a challenge. We identified reference condition based on the combined use of 3 independent selection methods. Method 1 involved use of a multivariate disturbance gradient derived from several stressors, method 2 was based on variation in channel morphology, and method 3 was based on passing 6 of 7 environmental criteria. Sites selected as reference from all 3 methods were considered primary reference, whereas those selected by 2 or 1 methods were considered secondary or tertiary reference, respectively. Sites not selected by any of the methods were considered non-reference. In addition, best professional judgment (BPJ) was used to exclude some sites from any reference class, and comparisons were made to examine the utility of BPJ. Non-metric multidimensional scaling indicated that use of BPJ may help designate non-reference sites when unidentified stressors are present. The macroinvertebrate community measures Ephemeroptera, Plecoptera, Trichoptera richness and North Carolina Biotic Index showed no differences between primary and secondary reference sites when BPJ was ignored. However, there was no significant difference among primary, secondary, and tertiary reference sites when BPJ was used. We underscore the importance of classifying reference conditions, especially in regions that have endured significant anthropogenic activity. We suggest that the use of secondary reference sites may enable construction of models that target a broader set of management interests.

Is there a significant relationship between the benthic status of an area, determined by two broadly-used indices, and best professional judgment?

Two benthic indices to assess the quality status (the AZTI's Marine Biotic Index (AMBI) and multivariate-AMBI (M-AMBI)) are being used extensively in different habitats worldwide. We try to interpret what is behind these indices making them suitable for different habitats. To demonstrate that, we used best professional judgment (BPJ), applying it to a dataset from southern Chile, to determine the criteria proposed by 12 experts in assessing the status. The experts were provided with raw species abundance data, from 12 stations within a gradient of disturbance, from unaffected to severely affected. There was a very good agreement among experts (kappa values 0.72–0.77), with highly significant (p<0.001) correlation between BPJ and AMBI and M-AMBI classifications, and an agreement of 76.4% and 81.9%, respectively. When comparing BPJ in Chile with other results in Europe, USA and northern Africa, similar patterns can be identified: (i) the number of criteria identified for classification is very high (range 7–12); (ii) the experts use several criteria together in the BPJ assessment; and (iii) the rank of the most important criteria is indicator species, richness, and diversity/dominance. These criteria are included in indices such as AMBI and M-AMBI. Hence, although experts are classifying samples subjectively when applying BPJ, they are corroborated in their opinions when using such indices. This fact can explain why these indices are so widely used.

Tolerance values of benthic macroinvertebrates for stream biomonitoring: assessment of assumptions underlying scoring systems worldwide

Tolerance values (TVs) based on benthic macroinvertebrates are one of the most widely used tools for monitoring the biological impacts of water pollution, particularly in streams and rivers. We compiled TVs of benthic macroinvertebrates from 29 regions around the world to test 11 basic assumptions about pollution tolerance, that: (1) Arthropoda are < tolerant than non-Arthropoda; (2) Insecta < non-Insecta; (3) non-Oligochaeta < Oligochaeta; (4) other macroinvertebrates < Oligochaeta + Chironomidae; (5) other macroinvertebrate taxa < Isopoda + Gastropoda + Hirudinea; (6) Ephemeroptera + Plecoptera + Trichoptera (EPT) < Odonata + Coleoptera + Heteroptera (OCH); (7) EPT < non-EPT insects; (8) Diptera < Insecta; (9) Bivalvia < Gastropoda; (10) Baetidae < other Ephemeroptera; and (11) Hydropsychidae < other Trichoptera. We found that the first eight of these 11 assumptions were supported despite regional variability. In addition, we examined the effect of Best Professional Judgment (BPJ) and non-independence of TVs among countries by performing all analyses using subsets of the original dataset. These subsets included a group based on those systems using TVs that were derived from techniques other than BPJ, and groups based on methods used for TV assignment. The results obtained from these subsets and the entire dataset are similar. We also made seven a priori hypotheses about the regional similarity of TVs based on geography. Only one of these was supported. Development of TVs and the reporting of how they are assigned need to be more rigorous and be better described.

Framework for interpreting sediment quality triad data

Integration of multiple lines of evidence (MLOE) data in a sediment quality triad assessment can be accomplished by means of numerous approaches, with most relying on some form of expert best professional judgment. Best professional judgment (BPJ) can be problematic in application to large data sets or in a regulatory setting where the assessment protocol needs to be transparent and consistently reproducible. We present a quantitative, objective framework for integrating the results of triad-based assessments and test its efficacy by applying it to 25 California sites and comparing the resulting classifications with those of 6 experts who were provided the same data. The framework is based on integrating the answers to 2 questions: 1) is there biological degradation, and 2) is chemical exposure high enough to potentially result in a biological response? The framework produced results that matched the median classifications of the experts better than did 5 of the 6 experts. Moreover, the framework was unbiased, with samples that differed from the median expert response evenly divided between those classified as more or less impacted. The framework was also evaluated by application to a set of sites from known degraded and reference areas, which the framework distinguished well. Although any framework needs to be flexible to supplemental data when they are available, the framework presented provides an objective means for using a triad-based approach in large-scale assessments for which relying on expert input for every sample is impractical.

Assessing coastal benthic macrofauna community condition using best professional judgement – Developing consensus across North America and Europe

Benthic indices are typically developed independently by habitat, making their incorporation into large geographic scale assessments potentially problematic because of scaling inequities. A potential solution is to establish common scaling using expert best professional judgment (BPJ). To test if experts from different geographies agree on condition assessment, sixteen experts from four regions in USA and Europe were provided species-abundance data for twelve sites per region. They ranked samples from best to worst condition and classified samples into four condition (quality) categories. Site rankings were highly correlated among experts, regardless of whether they were assessing samples from their home region. There was also good agreement on condition category, though agreement was better for samples at extremes of the disturbance gradient. The absence of regional bias suggests that expert judgment is a viable means for establishing a uniform scale to calibrate indices consistently across geographic regions.

A Weight-of-Evidence Framework for Assessing Sediment (Or Other) Contamination: Improving Certainty in the Decision-Making Process

A basic framework is presented for the ecological weight-of-evidence (WOE) process for sediment assessment that clearly defines its essential elements and will improve the certainty of conclusions about whether or not impairment exists due to sediment contamination, and, if so, which stressors and biological species (or ecological responses) are of greatest concern. The essential “Certainty Elements” are addressed in a transparent best professional judgment (BPJ) process with multiple lines-of-evidence (LOE) ultimately quantitatively integrated (but not necessarily combined into a single value). The WOE Certainty Elements include: (1) Development of a conceptual model (showing linkages of critical receptors and ecosystem quality characteristics); (2) Explanation of linkages between measurement endpoint responses (direct and indirect with associated spatial/temporal dynamics) and conceptual model components; (3) Identification of possible natural and anthropogenic stressors with associated exposure dynamics; (4) Evaluation of appropriate and quantitatively based reference (background) comparison methods; (5) Consideration of advantages and limitations of quantification methods used to integrate LOE; (6) Consideration of advantages and limitations of each LOE used; (7) Evaluation of causality criteria used for each LOE during output verification and how they were implemented; and (8) Combining the LOE into a WOE matrix for interpretation, showing causality linkages in the conceptual model. The framework identifies several statistical approaches for integrating within LOE, the suitability of which depends on physical characteristics of the system and the scale/nature of impairment. The quantification approaches include: (1) Gradient (regression methods); (2) Paired reference/test (before/after control impact and ANOVA methods); (3) Multiple reference (ANOVA and multivariate methods); and 4) Gradient with reference (regression, ANOVA and multivariate methods). This WOE framework can be used for any environmental assessment and is most effective when incorporated into the initial and final study design stages (e.g., the Problem Formulation and Risk Characterization stages of a risk assessment) with reassessment throughout the project and decision-making process, rather than in a retrospective data analysis approach where key certainty elements cannot be adequately addressed.