Bigger, Better, Faster, Transparent: Using Crowds and Algorithms to Build the Databases of the Future

Abstract

This article reviews available text analysis approaches and their limited success generating rich, transparent databases at scale. After elucidating the precise difficulties faced by previous projects and methods, the article introduces a new approach to text analysis featuring innovative procedures and open source tools to optimally combine the efforts of researchers, crowds, and algorithms. The article describes the first substantive project deploying the approach, imagines other projects on theoretical/empirical frontiers enabled by the approach, and closes with promising implications for social science when researchers are able to generate larger, more complex, transparent databases faster.