Implementation of Biased Big Data to the Japanese Official Labor Statistics Using Supervised Learning under Covariate Shift

Abstract

Recently, the National Statistical Institutes (NSIs) have started to use new data sources to produce official statistics. These new data sources often referred to as "Big Data" are not directly related to statistical production purposes. An advantage of using Big Data for official statistics is the speed of publication. Surveys designed to produce official statistics are time-consuming. On the other hand, we can obtain such new data sources with almost no time lag as the previous day ’s information is available the next day. However, the new data sources have a problem. They are likely selective with respect to the target population. This selection bias needs to be corrected when using these data sources to produce official statistics. In this study, we implemented Big Data to the Japanese official labor statistics, wage changes of hired career-changing employees. This economic indicator can potentially be indispensable for public policymakers and recruiters in private firms. If the indicator had been published quickly, they could have identified the degree of pressure on wage by the external labor market and could have used the information for their decision-making. The problem is the indicator is not published quickly. The time lag ranges from six to thirteen months. This means the indicator cannot be used to make decisions. To address this problem, we used transaction data of private employment agency and the idea of supervised learning under covariate shift to correct the selection bias. The proposed method can achieve early publication if a certain margin of error is allowed. The range of time lag reduces from a few days to five months, and policymakers and hiring managers could use the indicator for their decision-making.