Big Data Ingestion Research Articles

Integrating IoT with Big Data Platforms: Challenges, Technologies and Strategic Business Insights

The proliferation of Internet of Things (IoT) devices has ushered in an era where vast amounts of data are generated at the edge of network architectures, presenting both unprecedented challenges and opportunities for big data ingestion systems. This paper explores the complexities and strategic advantages associated with the integration of IoT data into big data platforms, highlighting how this convergence is reshaping business analytics and decision-making processes.

Enhancing Data Quality in Big Data Ingestion with Apache Spark's DataFrame APIs

Enhancing Data Quality in Big Data Ingestion with Apache Spark's DataFrame APIs

An Integrated Framework for Health State Monitoring in a Smart Factory Employing IoT and Big Data Techniques

With the rapid growth in the use of various smart digital sensors, the Internet of Things (IoT) is a swiftly growing technology, which has contributed significantly to Industry 4.0 and the promotion of IoT-based smart factories, which gives rise to the new challenges of big data analytics and the implementation of machine learning techniques. This article proposes a practical framework that combines IoT techniques, a data lake, data analysis, and cloud computing for manufacturing equipment health-state monitoring and diagnostics in smart manufacturing. It addresses all the required aspects in the realization of such a system and allows the seamless interchange of data and functionality. Due to the specific characteristics of IoT sensor data (low quality, redundant multisources, partial labeling), we not only provide a promising framework but also give detailed insights and pay considerable attention to data quality issues. In the proposed framework, an ingestion procedure is designed to manage data collection, data security, data transformation and data storage issues. To improve the quality of IoT big data, a high-noise feature filter is proposed for automated preliminary sensor selection to suppress noisy features, followed by a noisy data cleaning module to provide good quality data for unbiased diagnosis modeling. The proposed framework can achieve seamless integration between IoT big data ingestion from the physical factory and machine learning-based data analytics in the virtual systems. It is built on top of the Apache Spark processing engine, being capable of working in both big data and real-time environments. One case study has been conducted based on a four-stage syngas compressor from real industries, which won the Best Industry Application of IoT at the BigInsights Data & AI Innovation Awards. The experimental results demonstrate the effectiveness of both the proposed IoT-architecture and techniques to address the data quality issues.

A Scalable framework for data lakes ingestion

In the age of big data, the way we store and analyze heterogeneous data has changed. The complexity of various data inputs in the lakes indicates the significant importance of data ingestion that aids companies in making sense and getting more value from the variety and volume of data that is growing exponentially. In this paper, we propose a scalable and robust data lake architecture based on Apache NiFi for managing big data ingestion from various data sources. To assess the usefulness of our proposal, we evaluated its performance through a benchmark study with the architectures proposed in the literature. The analysis of this comparisons has proven that our contribution exceeds the existing one in terms of the features it supports.

Big data and remote sensing: A new software of ingestion

Currently, remote sensing is widely used in environmental monitoring applications, mostly air quality mapping and climate change supervision. However, satellite sensors occur massive volumes of data in near-real-time, stored in multiple formats and are provided with high velocity and variety. Besides, the processing of satellite big data is challenging. Thus, this study aims to approve that satellite data are big data and proposes a new big data architecture for satellite data processing. The developed software is enabling an efficient remote sensing big data ingestion and preprocessing. As a result, the experiment results show that 86 percent of the unnecessary daily files are discarded with a data cleansing of 20 percent of the erroneous and inaccurate plots. The final output is integrated into the Hadoop system, especially the HDFS, HBase, and Hive, for extra calculation and processing.

Learning analytics for higher education: proposal of big data ingestion architecture

Research background: Higher education institutions are generating multiple formats of data from diverse sources across the globe. The data ingestion layer is responsible for collecting data and transform for analysis. Learning analytics plays a vital role in providing decision-making support and selection of suitable timely intervention. The lack of tailored big-data ingestion architectures for academics led to several implementation challenges. Purpose of the article: The purpose of this article is to propose data ingestion architecture enabled for big data learning analytics. Methods: The study reviews existing literature to examine big-data ingestion tools and frameworks; and identify big-data ingestion challenges. An optimized framework for the real world learning analytics application was not yet in place at global higher educations. Consequently, the big-data ingestion pipeline is experiencing challenges of inefficient and complex data access, slow processing time, and security issues associated with transferring data to the system. The proposed data ingestion architecture is based on review of recent literature and adapts best international practices, guidelines, and techniques to meet the demand of current big-data ingestion issues. Findings & value added: This study identifies the current global challenges in implementing learning analytics projects. Review of recent big data ingestion techniques has been done based on defined metrics tuned for learning analytics purposes. The proposed data ingestion framework would increase the effectiveness of collecting, importing, processing and storing of learning data. Besides, the proposed architecture contributes to the construction of full-fledged big-data learning analytics ecosystem of higher educations.

Big Data Ingestion and Preparation Tools

Developing in Big Data applications become very important in the last few years, many organizations and industries are aware that data analysis is becoming an important factor to be more competitive and discover new trends and insights. Data ingestion and preparation step is the starting point for developing any Big Data project. This paper is a review for some of the most widely used Big Data ingestion and preparation tools, it discusses the main features, advantages and usage for each tool. The purpose of this paper is to help users to select the right ingestion and preparation tool according to their needs and applications’ requirements.

Ensuring Data Integrity in Big Data Ingestion: Techniques and Best Practices for Data Quality Assurance

Ensuring Data Integrity in Big Data Ingestion: Techniques and Best Practices for Data Quality Assurance

SAT-ETL-Integrator: an extract-transform-load software for satellite big data ingestion

Satellite data are used in several environmental applications, particularly in air quality supervising, climate change monitoring, and natural disaster predictions. However, remote sensing (RS) data occur in huge volume, in near-real time, and are stored inside complex structures. We aim to prove that satellite data are big data (BD). Accordingly, we propose a software as an extract-transform-load tool for satellite data preprocessing. We focused on the ingestion layer that will enable an efficient RSBD integration. As a result, the developed software layer receives data continuously and removes ∼86 % of the unused files. This layer also eliminates nearly 20% of erroneous datasets. Thanks to the proposed approach, we successfully reduced storage space consumption, enhanced the RS data accuracy, and integrated preprocessed datasets into a Hadoop distributed file system.

A Global Manufacturing Big Data Ecosystem for Fault Detection in Predictive Maintenance

Artificial intelligence, big data, machine learning, cloud computing, and Internet of Things (IoT) are terms which have driven the fourth industrial revolution. The digital revolution has transformed the manufacturing industry into smart manufacturing through the development of intelligent systems. In this paper, a big data ecosystem is presented for the implementation of fault detection and diagnosis in predictive maintenance with real industrial big data gathered directly from large-scale global manufacturing plants, aiming to provide a complete architecture which could be used in industrial IoT-based smart manufacturing in an industrial 4.0 system. The proposed architecture overcomes multiple challenges including big data ingestion, integration, transformation, storage, analytics, and visualization in a real-time environment using various technologies such as the data lake, NoSQL database, Apache Spark, Apache Drill, Apache Hive, OPC Collector, and other techniques. Transformation protocols, authentication, and data encryption methods are also utilized to address data and network security issues. A MapReduce-based distributed PCA model is designed for fault detection and diagnosis. In a large-scale manufacturing system, not all kinds of failure data are accessible, and the absence of labels precludes all the supervised methods in the predictive phase. Furthermore, the proposed framework takes advantage of some of the characteristics of PCA such as its ease of implementation on Spark, its simple algorithmic structure, and its real-time processing ability. All these elements are essential for smart manufacturing in the evolution to Industry 4.0. The proposed detection system has been implemented into the real-time industrial production system in a cooperated company, running for several years, and the results successfully provide an alarm warning several days before the fault happens. A test case involving several outages in 2014 is reported and analyzed in detail during the experiment section.

Big Data Analytics: Analysis of Features and Performance of Big Data Ingestion Tools

The purpose of this study was to analyze the features and performance of some of the most widely used big data ingestion tools. The analysis is made for three data ingestion tools, developed by Apache: Flume, Kafka and NiFi. The study is based on the information about tool functionalities and performance. This information was collected from different sources such as articles, books and forums, provided by people who really used these tools. The goal of this study is to compare the big data ingestion tools, in order to recommend that tool which satisfies best the specific needs. Based on the selected indicators, the results of the study reveal that all tools consistently assure good results in big data ingestion, but NiFi is the best option from the point of view of functionalities and Kafka, considering the performance.

Bluff Forwarding: A Practical Protocol for Delivering Refreshed Symmetric Keys on a Multi-Path Big Data Ingestion System

In this paper, we present a clean-slate design of a novel and practical protocol for transporting refreshed symmetric keys for multi-path big data ingestion systems. Our objective is to securely and reliably deliver the refreshed keys from data sources to the data collection servers even in the presence of malicious attackers. To satisfy this objective, we first adapt the secret sharing algorithm. We split symmetric keys into multiple secrets in such a way that partial retrieval of the secrets does not warrant the reconstruction of the original key. Then, we hide the updated secret keys by shuffling them into a group of randomly generated fake keys. These keys are encapsulated in a message container called a bluff. We develop a protocol that makes it computationally infeasible for the attackers to distinguish between bluffs and normal data. We implement and test this protocol on Apache Flume, which is a widely used, state-of-the-art data ingestion system. We analyze the security aspects of our protocol and observe the effects of various configuration settings on the data ingestion performance.

Real Time Product Feedback Review and Analysis Using Apache Technologies and NOSQL Database

Whenever a feedback system comes into mind, it’s always a demand of the e-commerce organizations to get the customer feedbacks in real time and to build some strong dashboards on top of these feedbacks/ratings. So that they can easily know the performance of any product at any point of time as well as they could able to take a decision, on what to do with the products those are getting very poor feedbacks. Which will result in a minimum impact on the tangible and intangible assets of the organizations. For achieving the above goal it is very necessary for these organizations to adopt the right tool and implement the required environment which can deal with the real time big data ingestion, enrichment, indexing and have the power to perform simple as well as complex analysis algorithm on the stored data. In this paper, we have collected Amazon Product Ratings for doing analysis and used Apache NiFi for ingesting real-time data into Apache Solr and have taken help of Banana Dashboard to show the real time analysis results in the form of attractive and user-friendly dashboards.

Hue Application for Big Data Ingestion

Hue Application for Big Data Ingestion