Big Data In Official Statistics Research Articles

Quality framework for combining survey, administrative and big data for official statistics

Creating statistics by combining data sources allows for the production of new, more timely and/or more detailed statistics. With an intended statistical output in mind, and various potentially useful data sources, there is a need to assess the potential of each source to contribute to the intended statistic. Quality frameworks provide tools for such tasks. This paper proposes a quality framework that includes dimensions applicable to survey, administrative and big data to support the assessment of the potential of each source to contribute to the intended statistic. The framework is applied to a case study of mobility data and a case study of virus particle detection in sewage data.

Big Data in Official Statistics: A View of the Problem

This article addresses the use of Big Data in official statistics, a much-discussed topic in the statistical community. The idea of replacing official statistics' indicators with a new information source, Big Data, was actively promoted in an overwhelming number of publications by foreign and domestic researchers several years ago. A more realistic perspective of this issue has gained ground in recent years; the issue is how to incorporate this unconventional source into statistical production and use Big Data for official statistical evaluations. The UN Statistical Commission's proactive stance greatly contributed to the discussion's shift in focus. The UN Statistical Commission was able to make the conversation about issues with Big Data application in official statistics a constructive process, leveraging its authority and opportunities. This process included all regional statistical platforms of the UN, specialized organizations and programs and other international agencies. The creation, testing, and implementation of novel approaches in official statistical surveys also involved national statisticians working for the government in universities, research centers and business. The article analyzes the practical application of Big Data in national statisticians' line of work, outlines the benefits and drawbacks of Big Data regarding its potential application in official statistics, outlines the directions of this activity, and formulates suggestions for the involvement of the Russian statistical community in this process. The author argues the need to avoid contrasting Big Data as information on society and economy versus official statistics based on traditional sources. In his opinion, Big Data is currently not a complete alternative to data collected on the basis of classical natural phenomena but can be built into national systems for collecting information on specific characteristics of socio-economic phenomena. The expansion of their use largely depends on the level of development of information and communication technologies as well as ICT integration in public life.

Overview of the use of big data for official statistics in Latin America and the Caribbean

In 2020 and 2021, the challenges related to the decline in the financing of statistical production and the cooperation of respondents was exacerbated by the COVID-19 pandemic. This scenario led national statistical offices (NSOs) to accelerate consideration of alternative data sources to complement or even replace traditional survey data. In this context, the use of big data to produce statistics has become promising. The use of big data for statistics is already in practice in many parts of the Global North and has also been spreading rapidly in the South. Part of the success of this trend is due to the support of the United Nations Committee of Experts on Big Data and Data Science for Official Statistics (UNCEBD), in particular its four Regional Hubs for Big Data. To learn the extent of the use of big data for official statistics in Latin America and the Caribbean, the United Nations Regional Hub for Big Data in Brazil conducted a study of the practices of NSOs in the region. A very promising scenario was found regarding the use of big data from satellite imagery, web scraping and other big data sources, for applications such as the production of price statistics, land use and cover patterns and migration.

Some Thoughts on Official Statistics and its Future (with discussion)

Abstract In this article, we share some reflections on the state of statistical science and its evolution in the production systems of official statistics. We first try to make a synthesis of the evolution of statistical thinking. We then examine the evolution of practices in official statistics, which had to face very early on a diversification of sou rces: first with the use of censuses, then sample surveys and finally administrative files. At each stage, a profound revision of methods was necessary. We show that since the middle of the 20th century, one of the major challenges of statistics has been to produce estimates from a variety of sources. To do this, a large number of methods have been proposed which are based on very different f oundations. The term “big data” encompasses a set of sources and new statistical methods. We first examine the potential of valorization of big data in official statistics. Some applications such as image analysis for agricultural prediction are very old and will be further developed. However, we report our skepticism towards web-scrapping methods. Then we examine the use of new deep learning methods. With access to more and more sources, the great challenge will remain the valorization and harmonization of these sources.

Big Data in the Philippines: How do we actually use them?

With the advancement of technology, digitalization, and the internet of things, large amounts of complex data are being produced daily. This vast quantity of various data produced at high speed is referred to as Big Data. The utilization of Big Data is being implemented with success in the private sector, yet the public sector seems to be falling behind despite the many potentials Big Data has already presented. In this regard, this paper explores ways in which the government can recognize the use of Big Data for official statistics. It begins by gathering and presenting Big Data-related initiatives and projects across the globe for various types and sources of Big Data implemented. Further, this paper discusses the opportunities, challenges, and risks associated with using Big Data, particularly in official statistics. This paper also aims to assess the current utilization of Big Data in the country through focus group discussions and key informant interviews. Based on desk review, discussions, and interviews, the paper then concludes with a proposed framework that provides ways in which Big Data may be utilized by the government to augment official statistics.

Transition from survey to sensor-enhanced official statistics: Road freight transport as an example

Capture-recapture (CRC) is currently considered a promising method to integrate big data in official statistics. We previously applied CRC to estimate road freight transport with survey data (as the first capture) and road sensor data (as the second capture), using license plate and time-stamp to identify re-captured vehicles. A considerable difference was found between the single-source, design-based survey estimate, and the multiple-source, model-based CRC estimate. One possible explanation is underreporting in the survey, which is conceivable given the response burden of diary questionnaires. In this paper, we explore alternative explanations by quantifying their effect on the estimated amount of underreporting. In particular, we study the effects of 1) reporting errors, including a mismatch between the reported day of loading and the measured day of driving, 2) measurement errors, including false positives and OCR failure, 3) considering vehicles reported not owned as nonresponse error instead of frame error, and 4) response mode. We conclude that alternative hypotheses are unlikely to fully explain the difference between the survey estimate and the CRC estimate. Underreporting, therefore, remains a likely explanation, illustrating the power of combining survey and sensor data.

Towards big data as official statistics: Case study of the use of mobile positioning data to delineate metropolitan areas in Indonesia

Nowadays, the use of so-called big data as a new data source to complement official statistics has become an opportunity for organizations focusing on statistics. The use of big data can lead to a more efficient data collection. However, currently, there has not been any standard business process for big data collection and processing in BPS-Statistics Indonesia. Meanwhile, the adoption of technologies alone cannot determine the success of big data use. It is widely known that big data use can be challenging, since there are issues regarding data access, quality, and methodology, as well as the development of required skillsets. This paper proposes a framework for a business process that is specifically designed to support the use of big data for official statistics at BPS-Statistics Indonesia along with how existing technology will support it. The development of this framework is based on the wider Statistical Business Process Framework and Architecture (SBFA) developed by BPS-Statistics Indonesia to describe and manage its overall statistical business processes. The paper uses the example of the use of Mobile Positioning Data (MPD) as a big data source to delineate Metropolitan Areas in Indonesia as a way to explain the implementation of the framework.

Innovations from Iran: Resolving quality issues in the integration of administrative and big data in official statistics

In a modernized statistical production process, non-traditional data sources such as ‘big’ data are increasingly being considered either as the main or as a supplementary source for official statistics. Their use however has brought new sorts of challenges: messy datasets, duplicate entries, missing information and misspellings to name a few. In many cases, there is also no unique identifier which can be used to unambiguously identify a record for the purpose of data integration. These challenges can be compounded by a non-English based alphabet like the Persian/Farsi alphabet used in Iran. In this paper, two innovative methods have been elaborated to address such data challenges. More specifically, the application of probabilistic record linkage using an ACSII coding system is an innovative way to deal with both data challenges and lack of unique identifier simultaneously. Moreover, text mining is an innovative way to address categorization and grouping systems that are not suitable for statistical purposes. Both innovative approaches can improve the accuracy and coherency of datasets and for data integration result in higher quality datasets. Results of research undertaken by the authors show the innovations lead to more effective data integration and improve the quality of the resulting official statistics. The innovations have wide applicability especially in non-English alphabet countries.

Utilization and Development of Big Data for Official Statistics

Utilization and Development of Big Data for Official Statistics

Big Data: Potential, Challenges and Statistical Implications

Big data are part of a paradigm shift that is significantly transforming statistical agencies, processes, and data analysis. While administrative and satellite data are already well established, the statistical community is now experimenting with structured and unstructured human-sourced, process-mediated, and machine-generated big data. The proposed SDN sets out a typology of big data for statistics and highlights that opportunities to exploit big data for official statistics will vary across countries and statistical domains. To illustrate the former, examples from a diverse set of countries are presented. To provide a balanced assessment on big data, the proposed SDN also discusses the key challenges that come with proprietary data from the private sector with regard to accessibility, representativeness, and sustainability. It concludes by discussing the implications for the statistical community going forward.

The opportunities, challenges and risks of big data for official statistics

The development of big data is set to be a significant disruptive innovation in the production of official statistics offering a range of opportunities, challenges and risks to the work of National Statistical Institutions (NSIs). This paper provides a synoptic overview of these issues in detail, mapping out the various pros and cons of big data for producing official statistics, examining the work to date by NSIs in formulating a strategic and operational response to big data, and plotting some suggestions with respect to on-going change management needed to address the use of big data for official statistics.

Big data in official statistics

The concept of "big data" stands to change the face of official statistics over the coming years, having an impact on almost all aspects of data production. The tasks of future statisticians will not necessarily be to produce new data, but rather to identify and make use of existing data to adequately describe social and economic phenomena. Until big data can be used correctly in official statistics, a lot of questions need to be answered and problems solved: the quality of data, data protection, privacy, and the sustainable availability are some of the more pressing issues to be addressed. The essential skills of official statisticians will undoubtedly change, and this implies a number of challenges to be faced by statistical education systems, in universities, and inside the statistical offices. The national statistical offices of the European Union have concluded a concrete strategy for exploring the possibilities of big data for official statistics, by means of the Big Data Roadmap and Action Plan 1.0. This is an important first step and will have a significant influence on implementing the concept of big data inside the statistical offices of Germany.

International collaboration to understand the relevance of Big Data for official statistics

This paper describes international collaboration activities to help the official statistical community to better understand the phenomenon of “Big Data”, and how it might impact on statistical production. This work is taking the form of annual projects, overseen by the High-Level Group for the Modernisation of Statistical Production and Services, a group of ten heads of national and international statistical organisations. The results obtained during 2014 are presented, and the paper looks forward to the planned activities in 2015.

Big Data and Official Statistics: Opportunities, Challenges and Risks

The development of big data is set to be a significant disruptive innovation in the production of official statistics offering a range of opportunities, challenges and risks to the work of national statistical institutions (NSIs). This paper provides a synoptic overview of these issues in detail, mapping out the various pros and cons of big data for producing official statistics, examining the work to date by NSIs in formulating a strategic and operational response to big data, and plotting some suggestions with respect to on-going change management needed to address the use of big data for official statistics.