Abstract
Abstract I propose a solution to content removal bias in statistics from web scraped data. Content removal bias occurs when data is removed from the web before a scraper is able to collect it. The solution I propose is based on inverse probability weights, derived from the parameters of a survival function with complex forms of data censoring. I apply this solution to the calculation of the proportion of newly built dwellings with web scraped data on Luxembourg, and I run a counterfactual experiment and a Montecarlo simulation to confirm the findings. The results show that the extent of content removal bias is relatively small if the scraping occurs frequently compared with the online permanence of the data; and that it grows larger with less frequent scraping.
Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
