Combining the Power of IoT and Big Data to Unleash the Potential of Digital Oil Field

Abstract

Abstract The objective of this paper is to demonstrate the process of unleashing the potential of digital oil fields by combining the power of Big Data platform with the Internet of Things (IoT). This new method enables efficient machine learning training utilizing Big Data and real-time scoring against mathematical models for predicting future outcomes. Digital oil fields have a diverse set of IoT devices that measure important field metrics in real-time, such as downhole pressure, temperature and oil rate. A typical digital oil well is equipped with many equipment such as Multiphase Flow Meter, Electrical Submersible Pump, and Permanent Downhole Monitoring Systems. Those equipments have several sensors generating a huge volume of data every second. In order to enable data scientists to analyze this huge amount of data streaming from various data sources, a data engineering pipeline was built. This pipeline combines data from various real-time and historical data repositories along with a master relational database in order to provide a consistent and clean analytics database for data scientists. This method saves data scientists the trouble of manually preparing and cleaning data from different datasets. Furthermore, by utilizing the analytics database cluster for machine learning, data scientists were able to use bigger data sets for training their models which can improve the accuracies of the models. As part of the solution, a scoring engine was built which consumes real-time data feed from the digital oil fields and performs real-time predictions and scoring utilizing machine learning models. The new architecture significantly improved the productivity of data scientists by allowing them to focus on building models and not to have to worry about data plumbing and deployment of the model to the field. Moreover by utilizing bigger data sets, models accuracies was improved considerably. Finally by integrating the models with IoT real-time data stream, field engineers can see and act on the models’ predictions in a timely manner. This architecture and methodology combines different technology domains (IoT and Big Data) with unique solution to bring value to the Oil and Gas producing & production business function.