Automated Data Harmonization (ADH) using Artificial Intelligence (AI)

Abstract

Organizations in the business of Information Services deal with very large volumes of data which is collected from a variety of proprietary, as well as, public sources in multiple languages with different formats, naming conventions, and context. Mapping such data into enterprise master data for reporting and prediction is an effort-intensive, time-consuming process which is prone to errors. Machines cannot match these sources and map to master data accurately. Enterprises are eager to automate the human intensive tasks of data harmonization so that their resources can focus on finding the insights to drive the business. We undertook one such automation initiative for a global Market Research Major (MRM) and achieved a significant level of success leveraging Artificial Intelligence (AI) techniques. The Automated Data Harmonization (ADH) solution has been a multi-step approach of Dictionary Matching, Fuzzy Text Similarity, and different Machine Learning techniques. It has been implemented on the Big Data stack for better performance and scalability. In order to streamline the overall business process, runtime rules and workflow has been implemented. The Proof of Concept has yielded an overall F-Score within the range of 82–93% depending on the variation of the dataset. The deployed version is continuing to deliver high accuracy and gained adoption as a core micro-service across the organization. The Business as Usual (BAU) cycle time has been reduced by 80% (from 14 days to 3 days). While the solution is unique and tailored to meet a set of specific business requirements, it can be extended for media metadata standardization across multiple devices, author name and citation resolution in scholarly journals, leads resolution in multi-channel marketing and ad campaigns etc.