How effective is score-based data quality assessment? An illustration with fish BCF data

Abstract

Increasingly rigorous data quality (DQ) evaluations and/or screening practices are being applied to environmental and ecotoxicological datasets. DQ is predominantly evaluated by scoring given data against preselected criteria. This study provides the first examination on the effectiveness of score-based DQ evaluation in providing statistically meaningful differentiation of measurements using fish bioconcentration factor (BCF) dataset as an illustration. This is achieved by inspecting how log BCF differs with the built-in overall-DQ and specific-DQ evaluations, and how it is influenced by interactive effects and hierarchy of DQ criteria. Approximately 80–90% of analyzable chemicals show no statistical difference in log BCF between low-quality (LQ) and high-quality (HQ) measurements in overall evaluation (n = 183) or in individual evaluation of 6 DQ criteria (n = 53 to 101). Further examination shows that log BCF may/may not change with different combinations or total number of criteria violations. Tree analysis and nodal structures of deviation in log BCF also reveal the absence of common structural dependence on the criteria violated. Finally, simple averaging of all measurements without DQ differentiation yields comparable log BCFs as those derived using strictly HQ data with ≤0.5 log unit difference in over 93% of the chemicals (n = 158) and no dependence on number of measurements, fraction of LQ measurements, or bioaccumulation potential of the chemicals. For accurate log BCF, DQ appears no more important than having more independent measurements irrespective of their individual DQ statuses. This work concludes by calling for: (i) re-documentation of experimental details in legacy environmental and ecotoxicological datasets, (ii) examination of other DQ-categorized datasets using the tests and tools applied here, and (ii) a thorough and systematic reflection on how DQ should be assessed for modeling, benchmarking, and other data-based analyses or applications.