Investigating open data portals automatically

Abstract

Deploying a suitable open data platform is one of the most important requirements for succeeding in the provision of open data. Currently, there are several platforms available in the market ranging from the commercial ecosystem to free and open source software. However, we know less about the extent to which they are adopted and what they offer. This paper aims to provide a methodology to investigate this. The methodology is illustrated through studying adoption and use of open data software platforms through a comprehensive survey of 3,152 open data portals worldwide. We have identified 1,104 installations relying on the main existing platforms CKAN, Socrata, ArcGIS Open Data, and OpenDataSoft. To support our analysis, we have automatically fetched metadata about 1,921,636 stored datasets. Our findings indicated that there is a gap between the adoption and the effective use of open data platforms, particularly in terms of technology choice. These data are both from a descriptive and analytical point, non-trivial and showcase the relevance of the methodology. This work makes contributions regarding the development of methods to automatically survey open data platforms and provides insights about availability of open data portals based on the utilization of software platforms, organized by country and frequency of dataset updates.