How We Developed a Data Exchange Format: Lessons Learned from Camera Trap Data Package (Camtrap DP)

Abstract

Camera trapping has revolutionized wildlife ecology and conservation by enabling automated data acquisition, leading to the accumulation of massive amounts of camera trap data worldwide (Steenweg et al. 2016, Kays et al. 2020, Delisle et al. 2021). Although management and processing of camera trap-derived big data are becoming increasingly solvable with the help of scalable infrastructures, harmonization and exchange of the data remain limited, hindering its full potential. We therefore developed a new data exchange format, Camera Trap Data Package (Camtrap DP), to facilitate the exchange, harmonization and archiving of camera trap data at local and global scales (Camtrap DP Development Team 2023). Camtrap DP was developed with two guiding principles. It should: allow easy and interoperable data exchange and be developed openly and collaboratively. allow easy and interoperable data exchange and be developed openly and collaboratively. Camtrap DP structures the data in a simple model consisting of three tables (Deployments, Media, and Observations), which supports a wide range of camera deployment designs, classification techniques and analytical use cases. To describe these tables and the accompanying metadata, we adopted the Frictionless Standards, a collection of open specifications developed by the Frictionless Data project (Fowler et al. 2018), which offer a standardized way to describe datasets, data files and tabular data. Doing so, we did not have to reinvent how to express generic properties such as licenses, contributors, file formats, field names, data types, required values, controlled values, and relationships between tables. We expanded upon those to describe the necessary properties related to camera trapping, relying on existing standards such as Darwin Core (Wieczorek et al. 2012), Audiovisual Core (Audiovisual Core Maintenance Group 2023) and Data Cite Metadata Schema (DataCite Metadata Working Group 2021) where possible. This approach is not only efficient, but also facilitates interoperability: since a Camtrap DP is in essence a Frictionless Data Package, existing software tools can be used to read and validate data. We developed Camtrap DP openly, collaboratively, and with version control from the start. It is licensed under the permissive MIT license, allowing anyone to use it. Suggestions for change were and continue to be discussed in a public issue tracker on GitHub. These are incorporated only after review and automated testing. Once a number of changes have been adopted, a new version of the standard is released using semantic versioning. This allows Camtrap DP to evolve over time, while making sure that software and datasets referring to older versions of the standard are still valid. Equally important to the success of a data exchange format is community-wide adoption, which requires trust and implementation by existing systems. From the start, we have involved researchers as well as maintainers of software tools and management platforms from the camera trapping community. Using an iterative approach, they tested and provided feedback on Camtrap DP to make sure it met their requirements. To aid their understanding of the format, we provided a website, an example dataset that is versioned with the format, and an R package to read, explore and visualize Camtrap DP datasets (Oldoni et al. 2023). Through open development and outreach, we also managed to get the support from trusted and well-recognized organizations such as Biodiversity Information Standards (TDWG) and the Global Biodiversity Information Facility (GBIF). As a result, Camtrap DP can now be used as a data publication format in the Integrated Publishing Toolkit (Robertson et al. 2014).