Anticipating data-induced bias

Abstract

In their Report “Prevalence-induced concept change in human judgment” (29 June, p. [1465][1]), D. E. Levari and coauthors show that perception is influenced by contrast. In a sequence of experiments, they demonstrate that when people are accustomed to a certain number of blue dots, threatening faces, or examples of unethical behavior, they will expand their definition of blue, threatening, and unethical once the prevalence of each decreases. Levari et al. 's finding that judgment depends on context may be applicable to any field in which humans categorize continuous distributions. Geologists, for example, identify rock structures based on satellite images. Individuals may vary by 10% in their identification of structures from multiple images of the same area, which differ in contrast ([ 1 ][2]). Biologists, in an effort to deliver prognoses in the face of climate change, score the heat tolerance of animals by classifying their behavior. The intraresearcher measurement error in such work may amount to 29% ([ 2 ][3]). Medical doctors diagnose millions of new skin cancer cases per year ([ 3 ][4]), starting from visual evaluation ([ 4 ][5]). Error rates of this evaluation may be as high as 50% ([ 4 ][5]). The contrast problem detected by Levari et al. could contribute to the variation inherent in these procedures across fields. Identifying this researcher bias would open the door to addressing it. The problems in rock structure identification, behavior classification, and skin diagnostics could be solvable by advanced automated quantitative analysis, although current attempts need further refinement ([ 1 ][2], [ 4 ][5]). There may also be ways to maintain contrast at a standardized frequency throughout analysis. This could be achieved based on prior knowledge about new data or by adding many known individuals to a few unknown before performing the identification task. Identifying counter-measurements and quality-control tools against this context dependency in science may be limited by lack of awareness rather than lack of options. 1. [↵][6]1. T. Scheiber et al ., GFF 137, 362 (2015). [OpenUrl][7] 2. [↵][8]1. L. E. Castaneda, 2. G. Calabria, 3. L. A. Betancourt, 4. E. L. Rezende, 5. M. Santos , J. Therm. Biol. 37, 432 (2012). [OpenUrl][9][CrossRef][10] 3. [↵][11]WHO, “INTERSUN: The Global UV Project: A guide and compendium” (World Health Organization, Geneva, 2003). 4. [↵][12]1. J. Dinnes et al ., Cochrane Database Syst. Rev. 2015, 1 (2015). [OpenUrl][13] [1]: /lookup/doi/10.1126/science.aap8731 [2]: #ref-1 [3]: #ref-2 [4]: #ref-3 [5]: #ref-4 [6]: #xref-ref-1-1 View reference 1 in text [7]: {openurl}?query=rft.jtitle%253DGFF%26rft.volume%253D137%26rft.spage%253D362%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [8]: #xref-ref-2-1 View reference 2 in text [9]: {openurl}?query=rft.jtitle%253DJ.%2BTherm.%2BBiol.%26rft.volume%253D37%26rft.spage%253D432%26rft_id%253Dinfo%253Adoi%252F10.1016%252Fj.jtherbio.2012.03.005%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [10]: /lookup/external-ref?access_num=10.1016/j.jtherbio.2012.03.005&link_type=DOI [11]: #xref-ref-3-1 View reference 3 in text [12]: #xref-ref-4-1 View reference 4 in text [13]: {openurl}?query=rft.jtitle%253DCochrane%2BDatabase%2BSyst.%2BRev.%26rft.volume%253D2015%26rft.spage%253D1%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx