Data Pipeline Research Articles

Reinforcement Learning for Intra- & Inter-Node Recommender Data Pipeline Optimization

Deep learning-based recommender models (DLRMs) have become an essential component of many modern recommender systems. Several companies are now building large compute clusters reserved for DLRM training, driving new interest in cost- & time- saving optimizations. The systems challenges faced in this setting are unique; while typical deep learning (DL) training jobs are dominated by model execution times, the most important factor in DLRM training performance is often online data ingestion. In this paper, we study real-world DLRM data processing pipelines taken from our compute cluster at Netflix to observe the performance impacts of online ingestion and identify shortfalls in existing data pipeline optimizers. Our studies lead us to design a new solution for data pipeline optimization, InTuneX . InTuneX is designed for production-scale multi-node recommender data pipelines. It unifies & tackles the challenges of both intra- and inter- node pipeline optimization. We achieve this with a multi-agent reinforcement learning (RL) design, simultaneously optimizing node assignments at the cluster level & CPU assignments within nodes. Our experiments show that InTuneX can build optimized data pipeline configurations within minutes. We apply InTuneX to our cluster, and find that it increases single-node data ingestion throughput by as much as 2.29X versus state-of-the-art optimizers, while improving the cost-efficiency of multi-node pipelines by 15-25%.

From data to nutrition: the impact of computing infrastructure and artificial intelligence

This article explores the significant impact that artificial intelligence (AI) could have on food safety and nutrition, with a specific focus on the use of machine learning and neural networks for disease risk prediction, diet personalization, and food product development. Specific AI techniques and explainable AI (XAI) are highlighted for their potential in personalizing diet recommendations, predicting models for disease prevention, and enhancing data-driven approaches to food production. The article also underlines the importance of high-performance computing infrastructures and data management strategies, including data operations (DataOps) for efficient data pipelines and findable, accessible, interoperable, and reusable (FAIR) principles for open and standardized data sharing. Additionally, it explores the concept of open data sharing and the integration of machine learning algorithms in the food industry to enhance food safety and product development. It highlights the METROFOOD-IT project as a best practice example of implementing advancements in the agri-food sector, demonstrating successful interdisciplinary collaboration. The project fosters both data security and transparency within a decentralized data space model, ensuring reliable and efficient data sharing. However, challenges such as data privacy, model interoperability, and ethical considerations remain key obstacles. The article also discusses the need for ongoing interdisciplinary collaboration between data scientists, nutritionists, and food technologists to effectively address these challenges. Future research should focus on refining AI models to improve their reliability and exploring how to integrate these technologies into everyday nutritional practices for better health outcomes.

PopFL: A scalable Federated Learning model in serverless edge computing integrating with dynamic pop-up network

With the rapid increase in the number of Internet-of-Things (IoT) devices, the massive volume of data creates significant challenges for traditional cloud-based solutions. These solutions often lead to high latency, increased operational costs, and limited scalability, making them unsuitable for real-time applications and resource-constrained environments. As a result, edge, and fog computing have emerged as viable alternatives, reducing latency and costs by processing data closer to its source. However, managing the flow of such vast and distributed data streams requires well-structured data pipelines to control the complete lifecycle—from data acquisition at the source to processing at the edge and fog layers, and finally storage and analytics in the cloud. To dynamically handle data analytics at varying distances from the source, often on heterogeneous hardware devices with collaborative learning techniques such as Federated Learning (FL). FL enables decentralized model training by leveraging the local data on Edge Devices (EDs), thereby preserving data privacy and reducing communication overhead with the cloud. However, FL faces critical challenges, including data heterogeneity, where the non-independent and identically distributed (non-IID) nature of data degrades model performance, and resource limitations on EDs, which lead to inefficiencies in training and biases in the aggregated models.To address these issues, we propose a novel FL solution, called Pop-Up Federated Learning (PopFL) in edge networks. This solution introduces hierarchical aggregation to reduce network congestion by distributing the aggregation tasks across multiple Fog Servers (FSs), rather than relying solely on centralized cloud aggregation. To further enhance participation and resource utilization at the edge, we incorporate the Stackelberg game model, which incentivizes EDs based on their contribution and resource availability. Additionally, PopFL employs a pop-up ad-hoc network for scalable and efficient communication between EDs and FSs, ensuring robust data transmission in dynamic network conditions. Extensive experiments conducted on three diverse datasets highlight the superior performance of PopFL compared to state-of-the-art FL techniques. The results show significant improvements in model accuracy, robustness, and fairness across various scenarios, effectively addressing the challenges of data heterogeneity and resource limitations. Through these innovations, PopFL paves the way for more reliable and efficient distributed learning systems, unlocking the full potential of FL in real-world applications where low latency and scalable solutions are critical.

Secure Supply Chain Information Interchange using Distributed Trust Backbone

International trade requires transparent visibility of the goods transportation. High-quality data related to containers is essential for container movement across the border speed. However, customs and port authorities face information incorrectness and inconsistency, which are significant determinants that decrease the performance of container clearance in supply chain activities. The Seamless Integrated Data Pipeline principle has been proposed to overcome the mentioned data quality shortcomings and enhance supply chain visibility. Based on the Data Pipeline idea, we proposed the Distributed Trust Backbone (DTB) as a model of secure information exchange between parties within the supply chain activity. However, the supply chain data is highly dynamic. Access control on dynamic resources is the key to enabling secure data exchange and clear visibility. We take this challenge up in this paper. We propose an access control mechanism based on the supply chain Data Pipeline concept and apply it to the DTB model. The elaboration on the concrete detail of the system is presented in this paper. The prototype has been developed and performed in the simulation tests. It reduces 58% of requesting data for supply chain activities. The results of the experiments show that our proposed method performs 100% access control to data with BigO(1) accessing the Access Control List. It can ensure that the information for decision-making in the supply chain is of high quality. The supply chain visibility is clearer and speeds up a modern information exchange system of supply chains.

Designing secure data pipelines for medical billing fraud detection using homomorphic encryption and federated learning

Medical billing fraud imposes significant financial and operational challenges on healthcare systems, highlighting the need for robust, privacy-preserving fraud detection solutions. This study presents a secure data pipeline that integrates homomorphic encryption (HE) and federated learning (FL) to enable decentralized fraud detection while maintaining patient confidentiality. Homomorphic encryption ensures data remains protected throughout the analytical process, while federated learning facilitates collaborative model training across healthcare institutions without requiring data centralization. Key findings reveal that increasing privacy levels via differential privacy effectively reduces data leakage risks, though it introduces minor computational overhead and a slight reduction in model accuracy. Scalability tests show that larger datasets considerably increase encryption time and memory usage, underscoring the need for optimized encryption algorithms. Additionally, secure communication protocols, while essential for data integrity, result in increased latency, which may impact real-time detection capabilities. The proposed pipeline achieves a balance between security and fraud detection accuracy, demonstrating its potential for real-world applications. However, further optimization of encryption methods and secure communication protocols is essential for broader scalability. This work advances privacy-centric approaches in healthcare fraud detection, setting a foundation for developing secure, scalable fraud detection systems.

Painting the Future: Leveraging Generative AI's Power in the Financial Ecosystem

Generative Artificial Intelligence (GenAI) stands out as a transformative force reshaping how institutions operate and innovate. This whitepaper traces the historical evolution of GenAI from conceptual stages to its current significance, highlighting key milestones and applications in the financial sector. Presently, GenAI is integrated into banking, particularly for fraud detection and personalized marketing. The case studies exemplify successful implementations by institutions like J. P Morgan and Microsoft showcasing benefits such as reduced fraud levels and improved customer experiences. The paper delves into the specific use cases across loan processing, portfolio management, asset management, regulatory compliance, and customer service automation. Despite GenAI's potential, the implementation and integration raise challenges related to data privacy, bias, ethical concerns, and technical complexities. The paper stresses the need for governance frameworks and compliance with global standards to tackle challenges. It recommends financial institutions prepare for future advancements by establishing efficient data pipelines, leveraging the cloud, building on existing frameworks, and adopting a human - led, tech - powered approach. It summarizes the critical importance of responsible innovation, urging institutions to embrace GenAI ethically to contribute to a more inclusive and sustainable financial ecosystem.

Hadoop Ecosystem Enhances Data Analytics for Music Streaming: A Case Study of User Behavior in the Last FM Dataset

This paper proposed a big data pipeline to analyze user behavior on Last.fm which aims to make data-driven recommendations for improving user engagement and attracting new users. The comprehensive analysis of user behavior in the music streaming industry using the Hadoop ecosystem and data analytics techniques. Specifically, the study focuses on Last.fm, a popular music streaming platform that collects large amounts of user activity data. The article proposes a new data pipeline utilizing Hadoop Distributed File System (HDFS) for data storage and Apache Pig for data transformation, leading to improved data preprocessing and analysis. Various analyses are conducted, including identifying the most listened to artists, top users based on song consumption and social connections, artist popularity by tags, and the most recently tagged artists. The findings provide valuable insights into user preferences, current trends, and opportunities for enhancing the recommendation algorithm and user engagement. The article concludes by offering recommendations for personalized marketing strategies and curated playlists to increase user satisfaction and revenue.

Systematic construction of composite radiation therapy dataset using automated data pipeline for prognosis prediction

BackgroundExisting research on medical data has primarily focused on single time-points or single-modality data. This study aims to collect all data generated during radiotherapy comprehensively to improve the treatment and prognosis of patients with malignant tumors. MethodsThe data collected from each medical institution were transmitted to the lead organization, where they underwent a file integrity check and were processed using a data pipeline. The key metadata of the collected data were compiled into a database, which were examined by data analysts to identify outliers based on theoretical and institution-specific characteristics. Appropriate filters were applied and the filtered data were subsequently reviewed by artificial intelligence (AI)-based models and researchers for radiotherapy organ slides. Finally, they were annotated by specialists. ResultsThe final dataset included 30,136 three-dimensional cone-beam computed tomography scans and 5,019 tabular data entries collected from 5,019 patients. It comprised 2,043,162 Digital Imaging and Communications in Medicine-format files with a total file size of 832 GB. Quality verification of the data using AI models revealed high classification performance for most organs, with relatively poor performance for the rectum. Overall, the macro AUROC value was 0.947. ConclusionsThis study implemented an automated data pipeline and AI-based verification to enhance the quality of collected radiotherapy data. The constructed dataset can be utilized for various types of future research and is expected to contribute to the improvement of radiotherapy efficiency.

Next-Generation Data Pipeline Designs for Modern Analytics : A Comprehensive Review

This comprehensive article examines the transformative evolution of data pipeline architectures in modern analytics, focusing on the integration of real-time and batch-processing methodologies to meet contemporary data processing demands. The article investigates how advanced frameworks like Apache Spark and Databricks, coupled with innovative technologies such as Delta Lake, are reshaping traditional data processing paradigms to accommodate increasing data volumes and complexity. Through a detailed article of hybrid pipeline architectures, data quality mechanisms, and observability practices, this paper demonstrates the critical role of next-generation pipeline designs in enabling organizations to build scalable, reliable, and maintainable data infrastructures. The article explores the implementation of ACID-compliant data lake technologies, automated monitoring systems, and sophisticated quality assurance methods that collectively ensure data integrity and processing efficiency. Key findings highlight the significance of emerging technologies, including edge computing and serverless architectures, in shaping future data pipeline designs. The article provides valuable insights into architectural patterns, best practices, and future trends that organizations can leverage to optimize their data processing capabilities and maintain competitive advantage in an increasingly data-driven business landscape.

Processing, evaluating, and understanding FMRI data with afni_proc.py.

FMRI data are noisy, complicated to acquire, and typically go through many steps of processing before they are used in a study or clinical practice. Being able to visualize and understand the data from the start through the completion of processing, while being confident that each intermediate step was successful, is challenging. AFNI's afni_proc.py is a tool to create and run a processing pipeline for FMRI data. With its flexible features, afni_proc.py allows users to both control and evaluate their processing at a detailed level. It has been designed to keep users informed about all processing steps: it does not just process the data, but also first outputs a fully commented processing script that the users can read, query, interpret, and refer back to. Having this full provenance is important for being able to understand each step of processing; it also promotes transparency and reproducibility by keeping the record of individual-level processing and modeling specifics in a single, shareable place. Additionally, afni_proc.py creates pipelines that contain several automatic self-checks for potential problems during runtime. The output directory contains a dictionary of relevant quantities that can be programmatically queried for potential issues and a systematic, interactive quality control (QC) HTML. All of these features help users evaluate and understand their data and processing in detail. We describe these and other aspects of afni_proc.py here using a set of task-based and resting-state FMRI example commands.

How to Staff When Customers Arrive in Batches

In many different settings, requests for service can arrive in near or true simultaneity with one another. This creates batches of arrivals to the underlying queueing system. In this paper, we study the staffing problem for the batch arrival queue. We show that batches place a dangerous and deceptive stress on services, requiring a high amount of resources and exhibiting a fundamentally larger tail in those demands. This uncovers a service regime in which a system with large batch arrivals may have low utilization but will still have nontrivial waiting. Methodologically, these staffing results follow from novel large batch and large batch-and-rate limits of the multiserver queueing model. In the large batch limit, we establish the first formal connection between general multiserver queues and storage processes, another family of stochastic models. By consequence, we show that the batch scaled queue length process is not asymptotically normal and that, in fact, the fluid- and diffusion-type limits coincide. Hence, the (safety) staffing of this system must be directly proportional to the batch size just to achieve a nondegenerate probability of wait. In exhibition of the existence and insights of this large batch regime, we apply our results to data on Covid-19 contact tracing in New York City. In doing so, we identify significant benefits produced by the tracing agency’s decision to staff above national recommendations, and we also demonstrate that there may have been an opportunity to further improve the operation by optimizing the arrival pattern in the public health data pipeline. This paper was accepted by Baris Ata, stochastic models and simulation. Funding: We acknowledge the generous support of the National Science Foundation (NSF) through Andrew Daw’s Graduate Research Fellowship under grant DGE-1650441, received during the initial phase of this research as part of his doctoral studies at Cornell University. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2021.03979 .

A Novel Spatial Data Pipeline for Streaming Smart City Data

Point cloud data from light detection and ranging (LiDAR) is often used for its spatial qualities, particularly in smart city projects involving vehicles and pedestrians. In this paper, the authors introduce a streaming and an on-demand pipeline for capturing LiDAR data from Velodyne Ultra Pucks placed along nine northern Nevada intersections known as the Living Lab within smart city project for the City of Reno. This pipeline is an iteration of a previously proposed pipeline with several feature enhancements. A streaming point cloud service was implemented to stream Point Cloud Data (PCD), LASzip (LAZ), and Google Draco. Also, two web services were built for the packet capture (PCAP) and ROS 2 bag files that enables acquisition of these formats for LiDAR data. A metadata service tracks edge device states and a GraphQL service interfaces with multiple services across the Living Lab. Draco provided the best processing time and had more options that affected the quality of the point cloud. To evaluate this pipeline, a discussion is provided, with an analysis of the point cloud formats.

Event-Driven Data Pipelines : A Cloud-Based Approach to Real-Time Data Processing

This article comprehensively analyzes event-driven data pipelines in cloud computing environments, examining their architecture, implementation considerations, and real-world applications. The article explores the fundamental components of event-driven systems, from event sources and message brokers to processing layers, while evaluating their performance characteristics and reliability mechanisms. Through detailed analysis of system architectures, we investigate the integration of various cloud services and their role in enabling scalable, real-time data processing. The article demonstrates how modern event-driven architectures achieve sub-millisecond processing times and handle millions of events per second while maintaining system resilience. Our findings reveal significant improvements in operational efficiency across various industries, including financial services, marketing automation, and IoT monitoring solutions. The article also addresses critical implementation aspects, including security frameworks, integration protocols, and best practices for system optimization. These insights provide valuable guidance for organizations leveraging event-driven architectures in their digital transformation initiatives. The article concludes that event-driven data pipelines represent a crucial advancement in data processing technology, offering unprecedented capabilities in handling real-time data streams while maintaining scalability, reliability, and cost-effectiveness.

Research on oil and gas pipeline leak detection method based on 1DCNN-DBO-LSTM

Abstract Detecting leaks in oil and gas pipelines is crucial for the safe operation of these systems. Current techniques for identifying oil and gas pipeline breaches are insufficient and fail to utilize the temporal attributes of the leak signal. This research proposes a fusion recognition model that integrates a long short-term memory network (LSTM) optimized using a one-dimensional convolutional neural network (1DCNN) with the dung beetle optimization algorithm (DBO). The model initially tackles the issue of excessive depth in the 1DCNN network, which paradoxically results in decreased accuracy and increased computational load. It develops a lightweight 1DCNN network via structural experiments that adaptively extracts data features. Subsequently, to mitigate the effects of the LSTM model's high stochasticity and the challenges associated with manual hyperparameter selection on detection accuracy, an enhanced LSTM model is proposed. This model is optimized through the integration of the Dung Beetle Optimization Algorithm (DBO) to refine the hyperparameters in LSTM. A 1DCNN-DBO-LSTM recognition model is developed, whereby the data features retrieved adaptively from the 1DCNN are fed into the enhanced DBO-LSTM for classification, and the T-SNE dimensionality reduction technique is utilized to visualize the network at each output layer. The model is assessed utilizing the precision and duration measures. The suggested model enhances recognition accuracy and reduces detection time compared to existing advanced models. The approach presented in this work can extract pipeline data features more swiftly and accurately, thus enhancing classification precision.

"I inherently just trust that it works": Investigating Mental Models of Open-Source Libraries for Differential Privacy

Differential privacy (DP) is a promising framework for privacy-preserving data science, but recent studies have exposed challenges in bringing this theoretical framework for privacy into practice. These tensions are particularly salient in the context of open-source software libraries for DP data analysis, which are emerging tools to help data stewards and analysts build privacy-preserving data pipelines for their applications. While there has been significant investment into such libraries, we need further inquiry into the role of these libraries in promoting understanding of and trust in DP, and in turn, the ways in which design of these open-source libraries can shed light on the challenges of creating trustworthy data infrastructures in practice. In this study, we use qualitative methods and mental models approaches to analyze the differences between conceptual models used to design open-source DP libraries and mental models of DP held by users. Through a two-stage study design involving formative interviews with 5 developers of open-source DP libraries and user studies with 17 data analysts, we find that DP libraries often struggle to bridge the gaps between developer and user mental models. In particular, we highlight the tension DP libraries face in maintaining rigorous DP implementations and facilitating user interaction. We conclude by offering practical recommendations for further development of DP libraries.

Advancing Fraud Detection in Banking: Integration of Data Pipelines, Machine Learning, and Cloud Computing

This article examines the transformation of fraud detection in the banking sector through the integration of advanced data pipelines, machine learning (ML), artificial intelligence (AI), and cloud computing technologies. We analyze how modern data pipelines enable real-time processing of vast transactional datasets, significantly improving the timeliness and accuracy of fraud detection. The article explores the application of ML models in identifying suspicious patterns and the role of AI-driven systems in continuously adapting to evolving fraud schemes while reducing false positives. We evaluate the impact of cloud platforms such as AWS, Azure, and Google Cloud in providing scalable, cost-efficient infrastructures for processing massive datasets and supporting seamless integration with ML models. The article presents case studies of successful implementations by major banks, demonstrating substantial reductions in processing times and improved detection efficiency. Additionally, we address key challenges including data quality maintenance, model interpretability, and false positive mitigation. The article concludes by discussing future innovations such as Federated Learning and Explainable AI (XAI), which promise to enhance cross-institutional collaboration and decision-making transparency in fraud detection. This comprehensive analysis provides valuable insights for financial institutions seeking to enhance their fraud detection capabilities in an increasingly complex digital landscape.

An intelligent monitoring approach for urban natural gas pipeline leak using semi-supervised learning generative adversarial networks

Traditional gas pipeline leak monitoring methods are subjected to the long response times and high false alarm rates. Deep learning can enhance the accuracy and real-time performance of pipeline leak monitoring. This paper develops an intelligent monitoring approach for urban gas pipeline leaks based on a semi-supervised learning Generative Adversarial Network (SGAN). First, the Isolation Forest algorithm is used to classify anomalies in the collected process parameter data of urban natural gas pipelines. One-Hot Encoding is used to label a small amount of sample data of pipeline leak. Second, both the labeled and unlabeled data are input into SGAN model for semi-supervised learning and classification to monitor the state of urban gas pipeline leak. The methodology addresses the imbalance between pipeline leak status data and normal data. The comparison with GAN and MLP shows that the methodology reaches the highest values in all evaluation metrics (precision = 94.1%, accuracy = 95.63%, recall = 93.93%, F1 score = 94.32%). The superior performance and accuracy make it more effective for urban natural gas pipeline leak monitoring.

An automated computational data pipeline to rapidly acquire, score, and rank toxicological data for ecological hazard assessment.

Biological Evaluations support Endangered Species Act (ESA) consultation with the US Fish and Wildlife Service and National Marine Fisheries Service by federal action agencies, such as the USEPA, regarding impacts of federal activities on threatened or endangered species. However, they are often time-consuming and challenging to conduct. The identification of pollutant benchmarks or guidance to protect taxa for states and tribes when USEPA has not yet developed criteria recommendations is also of importance to ensure a streamlined approach to Clean Water Act program implementation. Due to substantial workloads, tight regulatory timelines, and the often-protracted length of ESA consultations, there is a need to streamline the development of biological evaluation toxicity assessments for determining the impact of chemical pollutants on ESA-listed species. Moreover, there is limited availability of species-specific toxicity data for many contaminants, further complicating the consultation process. New approach methodologies are being increasingly used in toxicology and chemical safety assessment to rapidly and cost-effectively provide data that can fill gaps in hazard and/or exposure characterization. Here, we present the development of an automated computational pipeline-RASRTox (Rapidly Acquire, Score, and Rank Toxicological data)-to rapidly extract and categorize ecological toxicity benchmark values from curated data sources (ECOTOX, ToxCast) and well-established quantitative structure-activity relationships (TEST, ECOSAR). As a proof of concept, points-of-departure (PODs) generated in RASRTox for 13 chemicals were compared against benchmark values derived using traditional methods-toxicity reference values (TRVs) and water quality criteria (WQC). The RASRTox PODs were generally within an order of magnitude of corresponding TRVs, though less concordant compared with WQC. The greatest utility of RASRTox, however, lies in its ability to quickly and systematically identify critical studies that may serve as a basis for screening value derivation by toxicologists as part of an ecological hazard assessment. As such, the strategy described in this case study can potentially be adapted for other risk assessment contexts and stakeholder needs. Integr Environ Assess Manag 2024;20:2203-2217. © 2024 Society of Environmental Toxicology & Chemistry (SETAC). This article has been contributed to by U.S. Government employees and their work is in the public domain in the USA.

Mastering Data Pipelines for Al: A Beginner's Guide to Building Efficient Workflows

This comprehensive article explores the critical role of data pipelines in Artificial Intelligence (AI) development, emphasizing their importance in ensuring high-quality, relevant data for machine learning models. It covers key stages of the data pipeline process, including data ingestion, cleaning and preprocessing, transformation and feature engineering, and storage and loading. The article discusses various tools, techniques, and best practices for each stage, addressing challenges in handling diverse data sources, scalability issues, and the need for efficient data management. It highlights the significance of robust data pipelines in enhancing AI model performance and reliability, while also considering the dynamic nature of the AI field and the necessity for continuous learning and adaptation.

Advantages of Big Data Analysis and AI Technology in Data Collecting And Processing of Preventive Maintenance

In recent years, the emergence of Cyber-Physical systems (CPS), big data, cloud computing, and industrial wireless networks have promoted the development of Industry 4.0. Intelligent manufacturing systems are the leaders of this change. In smart manufacturing, the emphasis will be placed on using technologies such as big data analytics, cloud computing, edge computing, and artificial intelligence (AI). The advantages of new technologies are significantly reflected in the comparison of preventive maintenance and traditional maintenance based on intelligent technology: preventive maintenance significantly reduces the cost of prevention and, at the same time, dramatically improves the accuracy and timeliness of maintenance, bringing inestimable value to intelligent manufacturing. This paper will analyze new technologies’ innovative advantages and technical prospects through big data analysis and specific application scenarios of AI technology in preventive maintenance. Firstly, in data acquisition, this paper examines the architecture of preventive maintenance systems based on big data. Then, it analyzes the composition of the data pipeline: in the realization of data collection and processing in the field of cloud computing and edge computing, and the aspect of data processing, it discusses the participation of AI technology in data understanding and processing and the unprecedented changes it brings.