Data Ingestion Research Articles

Microplastics and Trash Cleaning and Harmonization (MaTCH): Semantic Data Ingestion and Harmonization Using Artificial Intelligence.

With the rapid expansion of microplastic research and reliance on semantic descriptors, there is an increasing need for plastic pollution data harmonization. Data standards have been developed but are seldom implemented across research sectors, geographic regions, environmental media, or size classes of plastic pollution. Harmonization of existing data is currently hindered by increasingly large datasets using thousands of different categorical variable descriptors, as well as various metrics used to describe particle abundance and differing size ranges studied across groups. For this study, we used manually developed relational databases to build an algorithm utilizing artificial intelligence capable of automatically curating harmonized, more usable datasets describing micro to macro plastic pollution in the environment. The study algorithm MaTCH (microplastics and trash cleaning and harmonization) can harmonize datasets with different formats, nomenclature, methods, and measured particle characteristics with an accuracy of 71-94% when matching semantically. All other non-semantic corrections are reported within a 95% confidence interval and with model uncertainty. All steps of the algorithm are integrated in an open-source software tool for the benefit of the scientific community and ease of integration for all plastic pollution data.

Web Traffic Anomaly Detection Using Isolation Forest

As companies increasingly undergo digital transformation, the value of their data assets also rises, making them even more attractive targets for hackers. The large volume of weblogs warrants the use of advanced classification methodologies in order for cybersecurity specialists to identify web traffic anomalies. This study aims to implement Isolation Forest, an unsupervised machine learning methodology in the identification of anomalous and non-anomalous web traffic. The publicly available weblogs dataset from an e-commerce website underwent data preparation through a systematic pipeline of processes involving data ingestion, data type conversion, data cleaning, and normalization. This led to the addition of derived columns in the training set and manually labeled testing set that was then used to compare the anomaly detection performance of the Isolation Forest model with that of cybersecurity experts. The developed Isolation Forest model was implemented using the Python Scikit-learn library, and exhibited a superior Accuracy of 93%, Precision of 95%, Recall of 90% and F1-Score of 92%. By appropriate data preparation, model development, model implementation, and model evaluation, this study shows that Isolation Forest can be a viable solution for close to accurate web traffic anomaly detection.

Adapting the open-source Gen3 platform and kubernetes for the NIH HEAL IMPOWR and MIRHIQL clinical trial data commons: Customization, cloud transition, and optimization

ObjectiveThis study aims to provide the decision-making framework, strategies, and software used to successfully deploy the first combined chronic pain and opioid use data clinical trial data commons using the Gen3 platform. Materials and MethodsThe approach involved adapting the open-source Gen3 platform and Kubernetes for the needs of the NIH HEAL IMPOWR and MIRHIQL networks. Key steps included customizing the Gen3 architecture, transitioning from Amazon to Google Cloud, adapting data ingestion and harmonization processes, ensuring security and compliance for the Kubernetes environment, and optimizing performance and user experience. ResultsThe primary result was a fully operational IMPOWR data commons built on Gen3. Key features include a modular architecture supporting diverse clinical trial data types, automated processes for data management, fine-grained access control and auditing, and researcher-friendly interfaces for data exploration and analysis. DiscussionThe successful development of the Wake Forest IDEA-CC data commons represents a significant milestone for chronic pain and addiction research. Harmonized, FAIR data from diverse studies can be discovered in a secure, scalable repository. Challenges remain in long-term maintenance and governance, but the commons provides a foundation for accelerating scientific progress. Key lessons learned include the importance of engaging both technical and domain experts, the need for flexible yet robust infrastructure, and the value of building on established open-source platforms. ConclusionThe WF IDEA-CC Gen3 data commons demonstrates the feasibility and value of developing a shared data infrastructure for chronic pain and opioid use research. The lessons learned can inform similar efforts in other clinical domains.

Using Time-Series Databases for Energy Data Infrastructures

The management of energy market data, such as load, production, forecasts, and prices, is critical for energy market participants, who develop in-house energy data infrastructure services to aggregate data from many sources to support their business operations. Energy data management frequently involves time sensitive operations, including rapid data ingestion, real-time querying, filling in gaps from missing or delayed data, and updating large volumes of timestamped and loosely structured data, all of which demand high processing power. Traditional relational database management systems (RDBMSs) often struggle with these operations, whereas time series databases (TSDBs) appear to be a more efficient solution, providing enhanced scalability, reliability, real-time data availability and superior performance. This paper examines the advantages of TSDBs over RDBMS for energy data management, demonstrating that TSDBs can either replace or complement RDBMSs. We present quantitative improvements in digestion, integration, architecture, and performance, demonstrating that operations such as importing and querying time-series energy data, along with the overall system’s efficiency, can be significantly improved, achieving up to 100 times faster operations compared to relational databases, all without requiring extensive modifications to the existing information system’s architecture.

Mastering Data Pipelines for Al: A Beginner's Guide to Building Efficient Workflows

This comprehensive article explores the critical role of data pipelines in Artificial Intelligence (AI) development, emphasizing their importance in ensuring high-quality, relevant data for machine learning models. It covers key stages of the data pipeline process, including data ingestion, cleaning and preprocessing, transformation and feature engineering, and storage and loading. The article discusses various tools, techniques, and best practices for each stage, addressing challenges in handling diverse data sources, scalability issues, and the need for efficient data management. It highlights the significance of robust data pipelines in enhancing AI model performance and reliability, while also considering the dynamic nature of the AI field and the necessity for continuous learning and adaptation.

Mastering Real-Time Data Processing Applications : Optimization Strategies for Peak Performance

This comprehensive article explores strategies for optimizing real-time data processing applications in the era of big data and distributed systems. It examines the growing demand for real-time analytics across industries and organizations' challenges in maintaining low latency and scalability. The article discusses key optimization techniques, including leveraging the Akka framework's actor model and supervision strategies, implementing efficient data ingestion through stream processing and partitioning, utilizing caching strategies, employing robust monitoring and auto-scaling mechanisms, ensuring fault tolerance through checkpointing and graceful degradation, and adopting asynchronous communication patterns. Throughout the article, real-world case studies and performance metrics demonstrate the significant improvements these strategies can bring to system throughput, latency, and resilience in various sectors such as finance, e-commerce, telecommunications, and manufacturing.

Building Scalable MLOps: Optimizing Machine Learning Deployment and Operations

As machine learning (ML) models become increasingly integrated into mission-critical applications and production systems, the need for robust and scalable MLOps (Machine Learning Operations) practices has grown significantly. This paper explores key strategies and best practices for building scalable MLOps pipelines to optimize the deployment and operation of machine learning models at an enterprise scale. It delves into the importance of automating the end-to-end lifecycle of ML models, from data ingestion and model training to testing, deployment, and monitoring. Approaches for implementing continuous integration and continuous deployment (CI/CD) pipelines tailored for ML workflows are discussed, enabling efficient and repeatable model updates and deployments. The paper emphasizes the criticality of implementing comprehensive monitoring and observability mechanisms to track model performance, detect drift, and ensure the reliability and trustworthiness of deployed models. The paper also addresses the challenges of managing model versioning and governance at scale, including techniques for maintaining a centralized model registry, enforcing access controls, and ensuring compliance with regulatory requirements. The paper aims to provide a comprehensive guide for organizations seeking to establish scalable and robust MLOps practices, enabling them to unlock the full potential of machine learning while mitigating risks and ensuring responsible AI deployment. Keywords—Machine Learning Operations (MLOps), Scalable AI Deployment, Continuous Integration and Continuous Deployment (CI/CD) for ML, ML Monitoring and Observability, Model Reproducibility, Model Versioning and Governance, Centralized Model Registry, Responsible AI Deployment, Ethical AI Practices, Enterprise MLOps

Data Integrity Problems in High-Volume High-Velocity Data Ingestion

In the era of bigdata, and never ending data push from IoT devices, the IT infrastructure are built to be scalable to handle the huge batch loads or continuous streaming live data. However, the big question is how can be establish the data integrity. How can we make sure no data is lost from Origin till the destination passing through numerous touch points enroute ? How can we ensure the quality with continuous inflow ? Should the inflow be suspended to perform the DQ checks? Or should it be a totally independent parallel activity. Let’s explore. Key words: Quality Data Management, Data Pipelines, AutoScalling, Large Scale Streaming Data, Performance, IoT, Data Cleansing

Real Time Data Ingestion and Transformation in Azure Data Platforms

Real Time Data Ingestion and Transformation in Azure Data Platforms

End-to-end AI pipeline optimization: Benchmarking and performance enhancement techniques for recommendation systems

End-to-end AI pipeline optimization is critical for improving the efficiency and performance of recommendation systems, which play a pivotal role in personalizing user experiences across various domains. This review explores benchmarking and performance enhancement techniques tailored to recommendation systems within AI pipelines. The objective is to streamline the processes involved in data ingestion, feature engineering, model training, and deployment to achieve optimal system performance and user satisfaction. Recommendation systems typically involve complex workflows that require continuous optimization. Benchmarking serves as a foundational step, enabling the identification of bottlenecks and inefficiencies within the pipeline. By establishing clear performance metrics, such as precision, recall, and latency, benchmarking allows for the comparative analysis of different algorithms, data processing methods, and system configurations. These metrics guide the selection of the most suitable models and techniques, thereby enhancing overall system effectiveness. Performance enhancement techniques are then applied to various stages of the AI pipeline. Advanced methods in feature engineering, such as automated feature selection and dimensionality reduction, can significantly improve model accuracy while reducing computational overhead. In the model training phase, techniques like hyperparameter tuning, gradient-based optimization, and distributed training are employed to accelerate convergence and improve model generalization. Additionally, optimization strategies at the deployment stage, including model compression, quantization, and the use of specialized hardware, are crucial for minimizing latency and resource consumption. This review also highlights the importance of continuous monitoring and feedback loops to maintain the effectiveness of recommendation systems in dynamic environments. By integrating real-time analytics and adaptive algorithms, systems can adjust to changing user behaviors and preferences, ensuring sustained performance improvements. In conclusion, optimizing end-to-end AI pipelines for recommendation systems involves a multifaceted approach that includes benchmarking, feature engineering, model training, and deployment enhancements. These efforts collectively contribute to more efficient, scalable, and accurate recommendation systems, ultimately leading to better user experiences and operational efficiencies. This paper will focus on optimizing AI pipelines for recommendation systems, covering the entire process from data extraction and feature engineering to model deployment and performance benchmarking. It will discuss techniques for identifying and mitigating performance bottlenecks in different computing environments, providing valuable insights for enhancing the efficiency of recommendation systems, which are crucial for various applications.

Optimizing Data Ingestion and Manipulation for Sports Marketing Analytics

In the dynamic landscape of sports marketing, the ability to effectively ingest and manipulate data is critical for driving decision-making and enhancing competitive advantage. This paper explores strategies for optimizing data ingestion and manipulation processes tailored to sports marketing analytics. We begin by analyzing the complexities of data sources, including social media, ticket sales, fan engagement metrics, and market trends. The diverse nature of these data streams necessitates robust ingestion frameworks that can accommodate real-time processing and historical data analysis. We propose a hybrid approach that leverages modern data engineering techniques, such as ETL (Extract, Transform, Load) processes, data warehousing, and cloud-based solutions. This approach enables seamless integration of structured and unstructured data, ensuring that marketers can access comprehensive insights into consumer behavior and campaign effectiveness. Furthermore, we discuss the application of advanced analytics and machine learning models to refine data manipulation techniques, enabling organizations to derive actionable insights that drive marketing strategies. By implementing these optimized processes, sports marketing professionals can enhance their understanding of audience preferences, optimize resource allocation, and ultimately improve return on investment (ROI) for marketing initiatives. This paper contributes to the existing body of knowledge by providing a framework for optimizing data ingestion and manipulation in sports marketing analytics, thus empowering organizations to make data-driven decisions that resonate with fans and stakeholders alike.

Model development for bespoke large language models for digital triage assistance in mental health care

Contemporary large language models (LLMs) may have utility for processing unstructured, narrative free-text clinical data contained in electronic health records (EHRs) — a particularly important use-case for mental health where a majority of routinely-collected patient data lacks structured, machine-readable content.A significant problem for the United Kingdom’s National Health Service (NHS) are the long waiting lists for specialist mental healthcare. According to NHS data (NHS Digital, 2024), in each month of 2023, there were between 370,000 and 470,000 individual new referrals into secondary mental healthcare services. Referrals must be triaged by clinicians, using clinical information contained in the patient’s EHR to arrive at a decision about the most appropriate mental healthcare team to assess and potentially treat these patients.The ability to efficiently recommend a relevant team by ingesting potentially voluminous clinical notes could help services both reduce referral waiting times and with the right technology, improve the evidence available to justify triage decisions.We present and evaluate three different approaches for LLM-based, end-to-end ingestion of variable-length clinical EHR data to assist clinicians when triaging referrals. Our model is able to deliver triage recommendations consistent with existing clinical practices and its architecture was implemented on a single GPU, making it practical for implementation in resource-limited NHS environments where private implementations of LLM technology will be necessary to ensure confidential clinical data are appropriately controlled and governed. Code available at: https://github.com/NtaylorOX/BespokeLLM_Triage.

A Framework for Automated Big Data Analytics in Cybersecurity Threat Detection

This research presents a novel framework designed to enhance cybersecurity through the integration of Big Data analytics, addressing the critical need for scalable and real-time threat detection in large-scale environments. Utilizing technologies such as Apache Kafka for efficient data ingestion, Apache Flink for stream processing, and advanced machine learning models like LSTM and Autoencoders, the framework offers robust anomaly detection capabilities. It also includes automated response mechanisms using SOAR and XDR systems, significantly improving response times and accuracy in threat mitigation. The proposed solution not only addresses current challenges in handling vast and complex data but also paves the way for future advancements, such as the integration of more sophisticated AI techniques and application across various domains, including IoT and cloud security. This research contributes to the field by providing a comprehensive, adaptive, and scalable framework that meets the demands of modern cybersecurity landscapes.

Microservices architecture to enable an open platform for realizing zero defects in cyber-physical manufacturing

Abstract The market’s demand for high-quality products necessitates innovative manufacturing approaches that emphasize flexibility, adaptability, and the reduction of defects. Traditional systems are currently evolving towards embracing I4.0 technologies, including data collection, processing, analytics and digital twin, aiming for zero-defect manufacturing quality. This paper introduces an open platform, compliant with RAMI4.0 standards, designed to improve manufacturing quality. The platform integrates data using Asset Administration Shells with microservices adaptation for data ingestion and advanced analytics. Additionally, it incorporates Non-Destructive Inspection tools, demonstrating a seamless integration of measurement and quality assurance. This study details the architectural specification and validation of the openZDM platform, employing microservices to ensure flexibility, modularity and scalability, aligning with the RAMI4.0 model. The platform was validated through deployment in an automotive assembly line, highlighting its effectiveness in integrating inspection scenarios and early defect detection tools. The architecture employs a choreographed approach to manage loosely coupled microservices, enabling efficient data lifecycle management from collection through analytics to visualization.

Differences among the total electron content derived by radio occultation, global ionospheric maps and satellite altimetry

In recent years, significant progress has been in ionospheric modeling research through data ingestion and data assimilation from a variety of sources, including ground-based global navigation satellite systems, space-based radio occultation and satellite altimetry (SA). Given the diverse observing geometries, vertical data coverages and intermission biases among different measurements, it is imperative to evaluate their absolute accuracies and estimate systematic biases to determine reasonable weights and error covariances when constructing ionospheric models. This study specifically investigates the disparities among the vertical total electron content (VTEC) derived from SA data of the Jason and Sentinel missions, the integrated VTEC from the Constellation Observing System for Meteorology, Ionosphere and Climate (COSMIC) and global ionospheric maps (GIMs). To mitigate the systematic bias resulting from differences in satellite altitudes, the vertical ranges of various VTECs are mapped to a standardized height. The results indicate that the intermission bias of SA-derived VTEC remains relatively stable, with Jason-1 serving as a benchmark for mapping other datasets. The mean bias between COSMIC and SA-derived VTEC is minimal, suggesting good agreement between these two space-based techniques. However, COSMIC and GIM VTEC exhibit remarkable seasonal discrepancies, influenced by the solar activity variations. Moreover, GIMs demonstrate noticeable hemispheric asymmetry and a degradation in accuracy ranging from 0.7 to 1.7 TECU in the ocean-dominant Southern Hemisphere. While space-based observations effectively illustrate phenomena such as the Weddell Sea anomaly and longitudinal ionospheric characteristics, GIMs tend to exhibit a more pronounced mid-latitude electron density enhancement structure.

Technology Focus: Reservoir Surveillance (September 2024)

In today’s oil and gas operations, surveillance technologies have undergone a revolution, reshaping how facilities, wells, and reservoirs are monitored. These advancements not only have increased the scale at which these technologies are deployed but also have led to an unprecedented influx of data. The sheer volume of data, however, poses a significant challenge to traditional analytical methods, overwhelming their capacity to derive actionable insights effectively. To address this challenge, the industry is rapidly advancing toward automated solutions powered by artificial-intelligence (AI) -driven analytics. These systems automate data ingestion and use machine-learning algorithms to sift through massive data sets, identifying anomalies and prioritizing actionable insights. By automating routine surveillance tasks, engineers can focus on critical actions that deliver substantial operational benefits. For example, in paper IPTC 23912, operators successfully optimized production operations by harnessing real-time field data through smart systems, effectively managing operations with complex near-critical fluids. Similarly, in paper SPE 218470, researchers proposed a novel workflow integrating virtual flowmetering and permanent downhole gauge data for pattern recognition to enhance real-time monitoring and decision-making in petroleum and geothermal industries. Nevertheless, ensuring effective surveillance of operational assets requires a strategic approach. A valuable tool in crafting such strategies is the value of information (VOI) assessment. This method systematically evaluates how acquiring specific information can influence decision-making and operational outcomes. For instance, paper SPE 215318 highlights a field operator’s systematic approach to VOI assessment, aiming to optimize daily operations and guide future development activities. In essence, while surveillance technologies have inundated operators with unprecedented data flows, advancements in automation and AI-driven analytics offer the promise of unlocking this data’s true potential. By embracing these technologies, the oil and gas industry can navigate the complexities of the modern energy landscape with greater agility, precision, and cost-effectiveness. Recommended additional reading at OnePetro: www.onepetro.org. SPE 215119 Surveillance, Analysis, and Optimization During Active Drilling Campaign by Yanfen Zhang, Chevron, et al. OTC 35413 New Opportunities in Well and Reservoir Surveillance Using Multiple Downhole Pressure Gauges in Deepwater Injector Wells by Piyush Pankaj, ExxonMobil, et al. OTC 34863 Digital Twin for Oil-Rim Management Using Early Warning System and Exception-Based Surveillance, Offshore Malaysia by M. Mahamad Amir, Petronas, et al.

Analyzing AWS Edge Computing Solutions to Enhance IoT Deployments

This paper explores integrating Internet of Things (IoT) deployments with edge computing, focusing on Amazon Web Services (AWS) as a key facilitator. It provides an analysis of AWS IoT services and their integration with edge computing technologies, addressing challenges, and practical applications across industries, and outlining future research directions. IoT and edge computing revolutionize data processing by enabling real-time analytics, reduced latency, and enhanced operational efficiency. IoT involves interconnected devices autonomously gathering and exchanging data, while edge computing processes data near its source, decentralizing data processing and minimizing data transmission to centralized servers. AWS facilitates scalable and secure infrastructures for IoT and edge computing. AWS IoT Core manages IoT device connectivity and data ingestion, AWS Greengrass extends AWS capabilities to edge devices, and AWS Lambda enables serverless computing, empowering efficient deployment and scaling of IoT applications. Centralized cloud architectures often struggle with vast IoT data. Edge computing decentralizes data processing, reducing latency, enhancing real-time capabilities, and minimizing bandwidth. AWS ensures secure device connectivity through AWS IoT Core, supporting various protocols for seamless integration with IoT devices. AWS Greengrass allows local data processing and machine learning at the edge, vital for environments with limited connectivity or stringent latency requirements. AWS Lambda supports serverless computing, enabling scalable, event-driven architectures without server management, crucial for fluctuating IoT workloads. In conclusion, AWS advances IoT capabilities at the edge, with practical implementations across industries. As IoT evolves, AWS remains pivotal, innovating to meet dynamic IoT deployment demands.

A holonic approach to clinical pathway data analysis

BackgroundHealthcare faces challenges due to the advancements of Industry 4.0 as large volumes of data are generated within healthcare facilities that, combined with the complex nature of healthcare environments, make it difficult to utilise and interpret this data effectively. PurposeA novel holonic approach to clinical pathway data analysis is presented and implemented as a clinical pathway digital twin. A holon is here taken to be an autonomous and co-operative building block of a software system for transforming, transporting, storing and/or validating information. The digital twin's aim is to ingest, structure and analyse the information associated with a clinical pathway to support healthcare professionals in making informed decisions, for example monitoring and predicting the duration from admission to discharge for individual patients. MethodReal world observations and a review of literature led to the identification of a generic set of clinical pathway analysis needs and, derived therefrom, a set of design requirements. A proof-of-concept clinical pathway analysis digital twin was implemented using a holonic approach derived from the ARTI reference architecture. The holonic approach is evaluated in a hip and knee replacement pathway case study. The evaluation includes automated statistical analyses and machine learning predictions. ResultsThe evaluation demonstrates that the holonic approach provides an intuitive and extensible means to aggregate and disaggregate information tactically, and to derive context-tailored analysis features. The holonic approach enhances checking for data completion and handling data anomalies. The evaluation also demonstrates on-demand report generation, which reduces repetitive manual tasks for healthcare professionals. ConclusionThe novel holonic data analysis approach facilitates context-rich analyses tailored to specific clinical pathway activities, with effective tailoring of data ingestion and analysis. Healthcare professionals can use the data analysis approach to extract valuable insights for decision-making related to clinical pathways.

Creating and leveraging bespoke large-scale knowledge graphs for comparative genomics and multi-omics drug discovery with SocialGene.

The rapid expansion of multi-omics data has transformed biological research, offering unprecedented opportunities to explore complex genomic relationships across diverse organisms. However, the vast volume and heterogeneity of these datasets presents significant challenges for analyses. Here we introduce SocialGene, a comprehensive software suite designed to collect, analyze, and organize multi-omics data into structured knowledge graphs, with the ability to handle small projects to repository-scale analyses. Originally developed to enhance genome mining for natural product drug discovery, SocialGene has been effective across various applications, including functional genomics, evolutionary studies, and systems biology. SocialGene's concerted Python and Nextflow libraries streamline data ingestion, manipulation, aggregation, and analysis, culminating in a custom Neo4j database. The software not only facilitates the exploration of genomic synteny but also provides a foundational knowledge graph supporting the integration of additional diverse datasets and the development of advanced search engines and analyses. This manuscript introduces some of SocialGene's capabilities through brief case studies including targeted genome mining for drug discovery, accelerated searches for similar and distantly related biosynthetic gene clusters in biobank-available organisms, integration of chemical and analytical data, and more. SocialGene is free, open-source, MIT-licensed, designed for adaptability and extension, and available from github.com/socialgene.

Overview on Data Ingestion and Schema Matching

This overview traced the evolution of data management, transitioning from traditional ETL processes to addressing contemporary challenges in Big Data, with a particular emphasis on data ingestion and schema matching. It explored the classification of data ingestion into batch, real-time, and hybrid processing, underscoring the challenges associated with data quality and heterogeneity. Central to the discussion was the role of schema mapping in data alignment, proving indispensable for linking diverse data sources. Recent advancements, notably the adoption of machine learning techniques, were significantly reshaping the landscape. The paper also addressed current challenges, including the integration of new technologies and the necessity for effective schema matching solutions, highlighting the continuously evolving nature of schema matching in the context of Big Data