Inter-agency Comparisons Research Articles

Reproducibility of river water quality measurements: inter-agency comparisons for quality assurance

ABSTRACTWe assessed the reproducibility of river state-of-environment (SoE) water quality measurements in the Wellington Region, New Zealand (NZ). Field staff from GWRC and NIWA conducted 29 side-by-side water sampling and in-situ measurements at six river sites of diverse water quality for 12 variables measured routinely in river SoE monitoring across NZ. Field measurements of water temperature, dissolved oxygen, electrical conductivity and visual clarity agreed closely with strong numerical similarity (within 10%). Numerical similarity ranged widely for laboratory measurements, from strong for nitrate-nitrite-nitrogen to weak for turbidity, dissolved reactive phosphorus, and ammoniacal-nitrogen. Numerical agreement was very weak for laboratory pH (which is problematic) and E. coli–which is ‘tolerable’ for many applications given good correlation (R = 0.94) over a 2000-fold concentration range. The findings of our inter-agency comparison have contributed to quality assurance recommendations in the NZ National Environmental Monitoring Standard (NEMS) for water quality.

Reduced-dynamic and kinematic baseline determination for the Swarm mission

The Swarm mission of the European Space Agency was launched in November 2013 with the objective of performing measurements of the earth’s magnetic field with unprecedented accuracy. At the beginning of data collection, two satellites started to fly in orbits with a separation in ascending nodes of 1°–1.5° at an altitude of about 480 km, and a third satellite has been placed in a higher orbit with an altitude of 530 km. The three spacecraft are equipped with dual-frequency eight-channel GPS receivers for the generation of precise orbits. Although such orbits support the fulfillment of the primary objectives of the mission, precise space baselines may be helpful for studying the earth’s gravity field, a spin-off application of the Swarm mission. Hitherto, a particular challenge for the computation of precise baselines from Swarm has been the presence of half-cycle ambiguities in GPS carrier phase observations, which complicate the implementation of integer ambiguity resolution methods. The present study shows the feasibility of generating carrier phase observations with full-cycle ambiguities, which in turn has been used to improve the performance of reduced-dynamic and kinematic precise baseline determination schemes. The implemented strategies have been tested in a period of 90 days in 2016. The obtained reduced-dynamic and kinematic baseline products were evaluated by inter-product and inter-agency comparisons using two independent software tools.

Inter-agency comparison of TanDEM-X baseline solutions

TanDEM-X (TerraSAR-X add-on for Digital Elevation Measurement) is the first Synthetic Aperture Radar (SAR) mission using close formation flying for bistatic SAR interferometry. The primary goal of the mission is to generate a global digital elevation model (DEM) with 2m height precision and 10m ground resolution from the configurable SAR interferometer with space baselines of a few hundred meters. As a key mission requirement for the interferometric SAR processing, the relative position, or baseline vector, of the two satellites must be determined with an accuracy of 1mm (1D RMS) from GPS measurements collected by the onboard receivers. The operational baseline products for the TanDEM-X mission are routinely generated by the German Research Center for Geosciences (GFZ) and the German Space Operations Center (DLR/GSOC) using different software packages (EPOS/BSW, GHOST) and analysis strategies. For a further independent performance assessment, TanDEM-X baseline solutions are generated at the Astronomical Institute of the University of Bern (AIUB) on a best effort basis using the Bernese Software (BSW).Dual-frequency baseline solutions are compared for a 1-month test period in January 2011. Differences of reduced-dynamic baseline solutions exhibit a representative standard deviation (STD) of 1mm outside maneuver periods, while biases are below 1mm in all directions. The achieved baseline determination performance is close to the mission specification, but independent SAR calibration data takes acquired over areas with a well known DEM from previous missions will be required to fully meet the 1mm 1D RMS target. Besides the operational solutions, single-frequency baseline solutions are tested. They benefit from a more robust ambiguity fixing and show a slightly better agreement of below 1mm STD, but are potentially affected by errors caused by an incomplete compensation of differential ionospheric path delays.

An Interagency Comparison of Screening‐Level Risk Assessment Approaches

Approaches to risk assessment have been shown to vary among regulatory agencies and across jurisdictional boundaries according to the different assumptions and justifications used. Approaches to screening-level risk assessment from six international agencies were applied to an urban case study focusing on benzo[a]pyrene (B[a]P) exposure and compared in order to provide insight into the differences between agency methods, assumptions, and justifications. Exposure estimates ranged four-fold, with most of the dose stemming from exposure to animal products (8-73%) and plant products (24-88%). Total cancer risk across agencies varied by two orders of magnitude, with exposure to air and plant and animal products contributing most to total cancer risk, while the air contribution showed the greatest variability (1-99%). Variability in cancer risk of 100-fold was attributed to choices of toxicological reference values (TRVs), either based on a combination of epidemiological and animal data, or on animal data. The contribution and importance of the urban exposure pathway for cancer risk varied according to the TRV and, ultimately, according to differences in risk assessment assumptions and guidance. While all agency risk assessment methods are predicated on science, the study results suggest that the largest impact on the differential assessment of risk by international agencies comes from policy and judgment, rather than science.

Reliability of assigning correct current procedural terminology—4 E/M codes

Study objectives: We determine the degree of interagency and intra-agency agreement on Current Procedural Terminology-4 (CPT-4) coding levels for emergency department medical records. We hypothesized that the level of agreement among coding agencies would be poor, but the distribution of codes would be similar and not significantly different. Methods: We performed a prospective observational study consisting of 3 separate randomized, blinded trials: 2 interagency comparisons and 1 intra-agency comparison. The setting was 2 suburban academic EDs. In the 2 interagency audits, 4 coding agencies were used. In the intra-agency comparison, we used 5 individual coders from within our own internal agency. The main outcome measures were the level of agreement not due to chance estimated with the multiple-rater weighted κ statistic and the Kendall τ-b. We measured the distribution of codes by agency or by individual coders with the χ2 test. Results: Our sample for the 2 interagency audits consisted of 194 and 195 records, respectively. We observed poor agreement in the level of coding assigned to individual charts among the 4 coding agencies, with κ values of 0.28 (95% confidence interval [CI] 0.275 to 0.285) and 0.287 (95% CI 0.283 to 0.291). Our intra-agency comparison consisted of 100 records. The agreement in the intra-agency review was significantly better but still fair (κ=0.436; 95% CI 0.428 to 0.444). The distribution of CPT-4 codes was significantly different for all 3 comparisons (P <.001). Conclusion: For our patient population, group of physicians, and methods of documentation, we identified poor-to-fair agreement in coding of emergency charts between coding agencies. Only fair agreement was measured in the intra-agency sample. The distribution of assigned CPT-4 codes was significantly different in each comparison. These findings have important financial and legal implications regarding the reliability of coding methods. [Ann Emerg Med. 2002;40:269-274.]

Lessons from comparisons of ultraviolet-monitoring spectroradiometers

A series of three interagency comparisons of solar-ultraviolet-monitoring spectroradiometers was conducted at Table Mountain outside Boulder, Colorado. The primary purpose of these comparisons was to compare the solar-ultraviolet irradiances measured by all the instruments during synchronized spectral scans. In addition, selected instrument parameters important for these measurements were characterized. Experimental techniques were developed to perform all these characterizations in the field. The comparisons demonstrated that the spectral responsivity of an instrument must be determined at its field location with standard lamps calibrated from a common scale of spectral irradiance. Using such a common scale and appropriate convolution techniques, the measured irradiances from synchronized scans agreed to within 2 %.

Inside the standard industrial classification codes: How many paper mills are there in Washington?

Comparison of enterprise data from different government agencies yields a surprising conclusion: there is no consensus on the number of enterprises in particular SIC codes, even in a concentrated, highly visible industry such as Washington paper mills. Reasons for the disagreements include differing treatment of multi-product firms and specialized purposes of data-gathering agencies, but not all of the observed disagreements can be explained by such identifiable factors. A 1990 industry trade publication and 1988 U.S. Census Bureau data are in complete agreement, finding 5 pulp mills (SEC 2611), 13 paper mills (SIC 2621) and 4 paperboard mills (SIC 2631), for a combined total of 22 mills. However, for 1989 Washington state agencies report as many as 42 and as few as 10 mills in these categories combined. U.S. EPA data for 1990 finds 21 mills total, but none in SIC 2631. Some agencies classify an enterprise into SIC categories on the basis of primary product shipped; others allocate fractions of an enterprise to different SIC codes in proportion to the product mix. Environmental agencies often have specialized enterprise counts focused on number of discharge points, which may differ from counts based on economic activity. Further work is evidently needed on standardization and interpretation of SIC codes, to provide a universal standard of classification, and to allow meaningful inter-agency comparison of industry data. For now, the number of paper mills in Washington is probably 22, but depends on who you ask.

The Disposition of Juvenile Records: An Interagency Comparison

This paper was presented at the Academy of Criminal Justice Sciences' Annual Meetings, St. Louis, Missouri, March, 1987. We also wish to thank the Administrative Office of the Courts of the State of California for their support, and all the agencies that participated in the survey.

Measuring efficiency in the public sector

One method of pursuing efficiency in the public sector has been the publication of performance indicators for individual agencies. It is often unclear how these indicators should be interpreted in isolation. This paper uses data envelopment analysis to show how the data underlying performance indicators can be used to generate a single measure of efficiency for an agency. The method systematically adjusts for differences in the environment that different agencies face. The potential limitations of the technique for the purpose of inter-agency comparison are then discussed.

Measuring productivity for home health nurses.

Home health managers are under increasing pressure to look at the operations of their agencies with critical eyes to determine whether they are providing the best quality services in the most efficient manner. A primary objective in this agency review is to increase the visit time available while satisfactorily performing the necessary maintenance and administrative services. Because nurses comprise a large portion of agency budgets, a useful starting point is nurses' productivity. An approach is presented which sequentially breaks down nursing time into a number of components. At each step, the resulting figures can be examined to evaluate the existence and nature of problems. Within an agency, this approach allows agencies to identify problems and suggest options for change. In inter-agency comparisons, differences in agency policy which affect productivity are made explicit, making the comparisons more useful. Three measures are also presented to evaluate aspects of overall agency productivity.

Evaluating community awareness of a community mental health center.

To assess community awareness of a community mental health center, a telephone survey was conducted, using a random stratified sample. In order to provide comparative analyses, two other mental health agencies were included in the study. Five specific issues were addressed: (a) the percentage of the population that was aware of the agencies, (b) inter-agency comparisons, (c) heterogeneity of awareness levels among geographic areas, (d) the relative merit of different sources of information, and (e) the relationship between awareness and demographic factors. The results indicated relatively high levels of awareness for the three agencies (X = 57%). There were, however, significant differences in awareness levels between agencies and between geographic areas. The source of information cited most often by respondents was the newspaper; in addition, there was an effect for the primary agency's storefront office. Sex, education, and family size had significant but weak correlations with the amount of accurate information a respondent possessed. The results provided substantive answers to the agency's concerns as well as methodological guidelines for future evaluations of community awareness.