Obtaining Data from the Internet: A Guide to Data Crawling in Management Research

Abstract

The increasing availability of data on the Internet opens new opportunities for management research and the method of data crawling can be used for automated large-scale data extraction. We show that data crawling has quickly gained popularity and is used for a wide variety of purposes, but has so far gained less traction in the field of management. We argue that we could use many data sets used in other disciplines for answering questions in management research and show that setting up a data crawler does not require advanced programming skills. However, a lot of pitfalls can challenge the success of using crawled data for research. We develop a guideline for crawling projects and address how many of the regularly occurring challenges can be addressed.