Observer agreement on the morphology of porous cranial lesions: Results from a workshop at the 2019 meeting of the Paleopathology Association

Abstract

ObjectiveThis paper presents the results of a 2019 Paleopathology Association workshop that tested observer agreement on porous cranial lesion morphology and presence using multiple sets of existing guidelines for data collection. MaterialsSixteen conference attendees of varying osteological experience served as observers. Three crania were assigned to each of four published guidelines for identifying and categorizing lesion morphology, for a total of twelve well-preserved human crania from the National Museum of Natural History Biological Anthropology Collections. MethodsObservers assessed each cranium macroscopically according to its assigned set of guidelines. ResultsObserver concordance was higher using scoring guidelines with higher-quality photographs, such as the 2019 guidelines from Rinaldo and colleagues. ConclusionsData collection guidelines with high-quality color photos may support greater reliability of researcher-generated data on macroscopic skeletal features. SignificanceThe conclusions of any research study are only as reliable as the data on which they are based. This work highlights the need for ongoing practices of quality control in a field in which much data results from individual judgement calls. LimitationsObserver concordance is not a measure of observer accuracy. Sample size is insufficient to draw broadly generalizable conclusions on the reliability of data collected using the guidelines tested, and conference environments are not a facsimile of research settings. Suggestions for further researchIterative testing of methodological consistency using larger sample sizes and more non-pathological crania is advised to identify the factors that influence observer discordance and to improve guidelines for qualitative assessments.