CULTIVATING DATA OBSERVABILITY AS THE NEXT FRONTIER OF DATA ENGINEERING: A PATH TO ENHANCED DATA QUALITY, TRANSPARENCY, AND DATA GOVERNANCE IN THE DIGITAL AGE

Abstract

In the age of increasing process automation and data-driven decision-making, ensuring the reliability, transparency and usability of data is of paramount importance. In this context, the concept of "data observability" has aroused the interest of practitioners and there is also a lot of grey content on it. On the other hand, there is a lack of academic effort to define and build on the concept. This conference paper will therefore examine the importance of "data observability" in modern data ecosystems. The focus is on the definition and characterisation of the concept, the differentiation from other concepts (e.g. data quality, data monitoring, data discovery, data operations) and why this concept appears to be so important in an increasingly data-driven world. In addition, the concept of "data observability" is categorised in the dynamically developing research field of data governance. For this purpose, a multivocal literature review (MLR) was conducted, a form of systematic literature review (SLR) which, in addition to the published (formal) literature (e.g. journal and conference papers), also includes and brings together the grey literature (e.g. blog posts, videos and white papers). The results show that the concept of "data observability" has the potential to revolutionise the way companies manage, analyse and derive insights from their data, ultimately leading to more informed and confident decision-making. Nevertheless, there is still plenty of room for further research into the specific contribution to better data and therefore better business processes and decisions.