Big data and semantic technology: A future for data integration, exploration and visualisation

Abstract

In a world of ever increasing data availability and user expectations, National Statistical Offices face mounting chal- lenges to produce relevant and timely statistics. They need to transform their business practice to take advantage of big data - especially administrative data - by integrating non-traditional and survey data sources to maximise value, and utilising new technology to enable enhanced analysis. An example of a response to these challenges is the prototype GLIDE (Graphically Linked Information Discovery Environment) the Australian Bureau of Statistics (ABS) is currently developing using semantic web technology. This environment includes as a test case a prototype semantic linked employer-employee database (LEED) which integrates administrative tax data and ABS business register data to enable detailed microeconomic analysis. However, as data structures become more complex and multi-dimensional, data integration and exploration encounters challenges within traditional relational databases, prompting the exploration of alternatives. Semantic web technology allows for a flexible data structure, machine reasoning and inference on the dataset, a shared understanding of the data's meaning, reusable classifications and standards, easy exploration of many dimensions, and network analysis. The possible advantages of such an approach for official statistics are demonstrated through two practical examples, showing how the prototype GLIDE supports effective data exploration and visualisation, and enables network analysis, to solve real business problems.