Real-time Data Processing Research Articles

Accelerating data acquisition with FPGA-based Edge Machine Learning: a case study with LCLS-II

Abstract New scientific experiments and instruments generate vast amounts of data that need to be transferred for storage or further processing, often overwhelming traditional systems. Edge Machine Learning (EdgeML) addresses this challenge by integrating
Machine Learning (ML) algorithms with edge computing, enabling real-time data processing directly at the point of data generation. EdgeML is particularly beneficial for environments where immediate decisions are required, or where bandwidth and
storage are limited. In this paper, we demonstrate a high-speed configurable ML model in a fully customizable EdgeML system using a Field Programmable Gate Array (FPGA). Our demonstration focuses on an angular array of electron spectrometers,
referred to as the ’CookieBox,’ developed for the Linac Coherent Light Source II (LCLS-II) project. The EdgeML system captures 51.2 Gbps from a 6.4 GS/s analog to digital converter and is designed to integrate data pre-processing and ML inside an
FPGA. Our implementation achieves an inference latency of 0.2 μs for the ML model, and a total latency of 0.4 μs for the complete EdgeML system, which includes preprocessing, data transmission, digitization, and ML inference. The modular design of
the system allows it to be adapted for other instrumentation applications requiring low-latency data processing.

Enhancing IoT data acquisition efficiency via FPGA-based implementation with OpenCL framework

The increasing demand for real-time data processing in Internet of Things (IoT) applications necessitates the development of efficient and flexible data acquisition systems capable of receiving and processing data from various sensor types. In conjunction with OpenCL, field-programmable gate arrays (FPGAs) have recently emerged as powerful platforms for accelerating data-intensive tasks. This study explored the implementation of an FPGA for data acquisition using OpenCL, aiming to design and implement an efficient data acquisition system tailored for IoT applications. Utilizing OpenCL for FPGA-based data acquisition offers several advantages that contribute to system efficiency, particularly in hardware interfaces between FPGA and external devices used in IoT applications. OpenCL abstracts the complexity of the FPGA hardware interface to external DDR memory for storing temporary data and a communication interface to the host CPU for transferring the collected data and enabling remote access, enabling developers to focus on algorithm design and functionality. To enable data reading from an external analog-to-digital converter (ADC) chip for IoT applications, we developed a component module that utilizes the Avalon-streaming interface and can stream the data to the OpenCL kernel. An experiment was conducted to demonstrate the performance of our proposed design. According to the findings of the experiments, a data acquisition implementation based on an FPGA and OpenCL can simultaneously read analog signals via a multichannel ADC. The proposed design provides a foundation for designing efficient data acquisition solutions, addressing the increasing needs of FPGA-based data acquisition in various IoT environments.

Adaptive FPGA-Based Accelerators for Human–Robot Interaction in Indoor Environments

This study addresses the challenges of human–robot interactions in real-time environments with adaptive field-programmable gate array (FPGA)-based accelerators. Predicting human posture in indoor environments in confined areas is a significant challenge for service robots. The proposed approach works on two levels: the estimation of human location and the robot’s intention to serve based on the human’s location at static and adaptive positions. This paper presents three methodologies to address these challenges: binary classification to analyze static and adaptive postures for human localization in indoor environments using the sensor fusion method, adaptive Simultaneous Localization and Mapping (SLAM) for the robot to deliver the task, and human–robot implicit communication. VLSI hardware schemes are developed for the proposed method. Initially, the control unit processes real-time sensor data through PIR sensors and multiple ultrasonic sensors to analyze the human posture. Subsequently, static and adaptive human posture data are communicated to the robot via Wi-Fi. Finally, the robot performs services for humans using an adaptive SLAM-based triangulation navigation method. The experimental validation was conducted in a hospital environment. The proposed algorithms were coded in Verilog HDL, simulated, and synthesized using VIVADO 2017.3. A Zed-board-based FPGA Xilinx board was used for experimental validation.

Fault-Tolerant Scheduling Mechanism for Dynamic Edge Computing Scenarios Based on Graph Reinforcement Learning

With the proliferation of Internet of Things (IoT) devices and edge nodes, edge computing has taken on much of the real-time data processing and low-latency response tasks which were previously managed by cloud computing. However, edge computing often encounters challenges such as network instability and dynamic resource variations, which can lead to task interruptions or failures. To address these issues, developing a fault-tolerant scheduling mechanism is crucial to ensure that a system continues to operate efficiently even when some nodes experience failures. In this paper, we propose an innovative fault-tolerant scheduling model based on asynchronous graph reinforcement learning. This model incorporates a deep reinforcement learning framework built upon a graph neural network, allowing it to accurately capture the complex communication relationships between computing nodes. The model generates fault-tolerant scheduling actions as output, ensuring robust performance in dynamic environments. Additionally, we introduce an asynchronous model update strategy, which enhances the model’s capability of real-time dynamic scheduling through multi-threaded parallel interactions with the environment and frequent model updates via running threads. The experimental results demonstrate that the proposed method outperformed the baseline algorithms in terms of quality of service (QoS) assurance and fault-tolerant scheduling capabilities.

Harnessing big data for national security: A review of information systems and emerging technologies in the U.S

The rapid growth of big data and emerging technologies has fundamentally transformed the national security landscape, enabling governments to leverage vast amounts of information for decision-making, threat detection, and operational effectiveness. This review explores the integration of big data analytics into U.S. national security systems, focusing on the technological innovations that facilitate real-time data processing, secure communications, and predictive analytics. Key emerging technologies, such as artificial intelligence, blockchain, and the Internet of Things, are evaluated for their role in enhancing intelligence gathering, cybersecurity, and surveillance operations. Furthermore, the paper examines the foundational principles of big data in national security, detailing its data sources, types, and uses in both structured and unstructured forms. While these advancements present significant opportunities, they also pose challenges related to privacy, ethics, policy, and legal frameworks, which are critical for ensuring that the implementation of such technologies aligns with democratic values and security objectives. The paper concludes by highlighting future research directions, emphasizing the need for continued development in big data infrastructure, governance, and ethical considerations to fully harness its potential in protecting national interests.

Mastering Real-Time Data Processing Applications : Optimization Strategies for Peak Performance

This comprehensive article explores strategies for optimizing real-time data processing applications in the era of big data and distributed systems. It examines the growing demand for real-time analytics across industries and organizations' challenges in maintaining low latency and scalability. The article discusses key optimization techniques, including leveraging the Akka framework's actor model and supervision strategies, implementing efficient data ingestion through stream processing and partitioning, utilizing caching strategies, employing robust monitoring and auto-scaling mechanisms, ensuring fault tolerance through checkpointing and graceful degradation, and adopting asynchronous communication patterns. Throughout the article, real-world case studies and performance metrics demonstrate the significant improvements these strategies can bring to system throughput, latency, and resilience in various sectors such as finance, e-commerce, telecommunications, and manufacturing.

From Movement to Meaning: Real-time Detection of Behavioural Patterns in Human Psychology using Deep Learning and Time Series Sequence

Understanding human behaviour and psychology is complex and involves various fields, including cognitive science, neurology, and artificial intelligence (AI). In this study, we introduce a cutting-edge system that detects both manual (like hand gestures) and non-manual (such as facial expressions or body movements) features in real time, using advanced deep learning techniques and time-series analysis. This system translates these human movements into text, providing a powerful tool to analyse subtle behavioural and psychological signals. By processing real-time data, our system significantly advances psychology by detecting small expressions, involuntary gestures, and behaviour patterns often associated with psychological and neurological conditions. For instance, it can help identify repetitive actions or atypical gestures in people with autism spectrum disorder (ASD), which are critical for early diagnosis. It can also track signs of neurodegenerative diseases like Parkinson’s or Alzheimer’s, detecting tremors, posture changes, or facial expressions, and offer insights for early intervention

Enhancing System Efficiency through AI, Edge Computing, and Resource Optimization in Modern Infrastructure

This paper explores innovative strategies for enhancing system efficiency in modern infrastructure by integrating artificial intelligence (AI), edge computing, and resource optimization techniques. As the complexity of infrastructure systems increases, traditional methods often fall short in addressing the evolving demands of operational efficiency and reliability. By leveraging AI algorithms for predictive analytics and resource allocation, and utilizing edge computing for real-time data processing, organizations can significantly improve performance and responsiveness. The study examines case studies that highlight successful implementations of these technologies across various sectors, including infrastructure monitoring, and grid maintenance. Insights from this research provide a framework for practitioners to adopt these advanced methodologies, ultimately leading to more resilient and efficient infrastructure systems.

Decentralized Machine Learning Framework for the Internet of Things: Enhancing Security, Privacy, and Efficiency in Cloud-Integrated Environments

The Internet of things (IoT) presents unique challenges for the deployment of machine learning (ML) models, particularly due to constraints on computational resources, the necessity for decentralized processing, and concerns regarding security and privacy in interconnected environments such as the Internet of cloud. In this paper, a novel decentralized ML framework is proposed for IoT environments characterized by wireless communication, dynamic data streams, and integration with cloud services. The framework integrates incremental learning algorithms with a robust decentralized model exchange protocol, ensuring that data privacy is preserved, while enabling IoT devices to participate in collaborative learning from distributed data across cloud networks. By incorporating a gossip-based communication protocol, the framework ensures energy-efficient, scalable, and secure model exchange, fostering effective knowledge sharing among devices, while addressing the potential security threats inherent in cloud-based IoT ecosystems. The framework’s performance was evaluated through simulations, demonstrating its ability to handle the complexities of real-time data processing in resource-constrained IoT environments, while also mitigating security and privacy risks within the Internet of cloud.

Optimization of artificial intelligence in localized big data real-time query processing task scheduling algorithm

Optimization of artificial intelligence in localized big data real-time query processing task scheduling algorithm

A Deep Learning-Based Car Accident Detection Framework Using Edge and Cloud Computing

The evolving technological landscape has seen a pivotal shift with the advent of edge computing, transforming various sectors, particularly accident detection. Edge computing enhances road safety by enabling realtime data processing from onboard sensors, cameras, and connected devices, addressing limitations in traditional cloudbased systems. This paper introduces a deep learning-based accident detection framework within an edge-cloud setup. Utilizing a CNN model, accidents are detected at the edge node near the data source, ensuring low latency, reduced network usage, and faster execution times. The model achieves a remarkable 95.91% accuracy.

Enhancing financial reporting and management efficiency through enterprise resource planning (ERP) systems: A theoretical review for large-scale energy operations

Enterprise Resource Planning (ERP) systems have emerged as essential tools in streamlining financial reporting and improving management efficiency across various industries, including large-scale energy operations. This theoretical review explores how ERP systems enhance financial reporting by integrating real-time data, automating financial processes, and fostering transparency. In energy operations, where financial activities are complex and data-intensive, ERP systems centralize financial data, reducing errors, improving compliance, and supporting decision-making through enhanced visibility and accuracy. By automating tasks such as budgeting, forecasting, and financial statement generation, ERP systems enable energy companies to efficiently manage their financial resources, thus optimizing cost control and profitability. Additionally, ERP systems provide significant benefits for management efficiency, particularly in large-scale operations. The integration of financial, operational, and managerial processes within a single platform allows for better resource allocation, improved communication across departments, and faster response times to market fluctuations. These capabilities are crucial in the energy sector, where companies face volatile market conditions, regulatory pressures, and high operational risks. ERP systems enhance managerial decision-making by providing real-time insights into key performance indicators (KPIs), enabling executives to make data-driven decisions that support long-term strategic goals. However, implementing ERP systems in large-scale energy operations presents challenges, including high costs, lengthy implementation periods, and the need for employee training. This review also examines these barriers and suggests strategies for successful ERP adoption, including phased implementation, stakeholder engagement, and customization to industry-specific needs. In conclusion, ERP systems hold the potential to revolutionize financial reporting and management efficiency in large-scale energy operations. By leveraging ERP's real-time data integration and process automation, energy companies can significantly improve their financial management, enhance decision-making, and adapt more quickly to market changes. Keywords: ERP Systems, Financial Reporting, Management Efficiency, Large-Scale Energy Operations, Process Automation, Real-Time Data Integration, Decision-Making, Cost Control, Market Fluctuations, Energy Sector.

Automatic Indoor Thermal Comfort Monitoring Based on BIM and IoT Technology

Building Information Modeling (BIM) and Internet of Thing (IoT) integration technologies can improve operational efficiency in the operational phase of construction projects. Currently, research on the integration of BIM and IoT has yet to ensure secure data transmission and lacks real-time data processing capabilities. This study builds a framework to collect and analyze BIM and IoT data in real time. The framework is verified to be effective through a case study in an office building. The monitoring system can automatically calculate the Predicted Mean Vote (PMV) value, upload and update real-time temperature and humidity data, and visualize thermal comfort through heat maps. The proposed integration approach offers building management strategies to enhance thermal comfort in office environments, fostering a more inclusive and accommodating workspace that acknowledges the diverse cultural backgrounds of occupants.

Optimizing Business Operations through Edge Computing: Advancements in Real-Time Data Processing for the Big Data Era

The question of big data brought new opportunities and threats to business environment primarily in terms of processing increasingly large amounts of data in realtime. This paper examines the way in which edge computing can /should benefit business processes by offering improvements to real-time data analysis beyond what is provided by cloud-based systems while overcoming the challenge of latency, bandwidth and scalability. The study uses both primary and secondary data collected from 50 companies that use edge computing for processing of big data in industries including manufacturing, health, and retail. It involves cases where the factors are manifested, practices involving changes in numerical values of operations and other models involving the management of real-time data. Studies show that edge computing cuts down data processing delay by 40-60% on average, cuts data utilization and improves decision-making. This work also examines the use-cases of edge computing in order to demonstrate its potential across industries that must analyze data in near realtime, including self-driving automobiles, IoT gadgets, and logistics. Toward this end, this research provides an analysis of the ethical issues of decentralized data handling especially focusing on data security and data privacy. In response to the research questions of the paper, this study provides practical recommendations to help the companies in pursuing edge computing in the big data environment for the sake of sustaining market competitiveness. The research enriches the existing knowledge in business technology by presenting the possibility of change that edge computing brings.

Research on Intelligent Agricultural Monitoring and Control System Based on Neural Networks in the Internet of Things Environment

In today's rapidly advancing information technology landscape, the Internet of Things (IoT) has profoundly impacted traditional agriculture. IoT technology, as an intelligent solution, enables real-time data collection, processing, and analysis, significantly enhancing the efficiency and intelligence of agricultural production. Among these advancements, the neural network-based intelligent agricultural monitoring system stands out. This system not only monitors the agricultural production environment in real-time but also employs deep learning algorithms to predict future trends, guiding agricultural practices to achieve precision farming. Therefore, exploring the intelligent agricultural monitoring and control system based on neural networks in the IoT environment holds significant importance.

Development of an Intelligent Coal Production and Operation Platform Based on a Real-Time Data Warehouse and AI Model

Smart mining solutions currently suffer from inadequate big data support and insufficient AI applications. The main reason for these limitations is the absence of a comprehensive industrial internet cloud platform tailored for the coal industry, which restricts resource integration. This paper presents the development of an innovative platform designed to enhance safety, operational efficiency, and automation in fully mechanized coal mining in China. This platform integrates cloud edge computing, real-time data processing, and AI-driven analytics to improve decision-making and maintenance strategies. Several AI models have been developed for the proactive maintenance of comprehensive mining face equipment, including early warnings for periodic weighting and the detection of common faults such as those in the shearer, hydraulic support, and conveyor. The platform leverages large-scale knowledge graph models and Graph Retrieval-Augmented Generation (GraphRAG) technology to build structured knowledge graphs. This facilitates intelligent Q&A capabilities and precise fault diagnosis, thereby enhancing system responsiveness and improving the accuracy of fault resolution. The practical process of implementing such a platform primarily based on open-source components is summarized in this paper.

Broadband and parallel multiple-order optical spatial differentiation enabled by Bessel vortex modulated metalens

Optical analog image processing technology is expected to provide an effective solution for high-throughput and real-time data processing with low power consumption. In various operations, optical spatial differential operations are essential in edge extraction, data compression, and feature classification. Unfortunately, existing methods can only perform low-order or selectively perform a particular high-order differential operation. Here, we propose and experimentally demonstrate a Bessel vortex modulated metalens composed of a single complex amplitude metasurface, which can perform multiple-order radial differential operations over a wide band by presetting the order of the corresponding Bessel vortex. This architecture further enables angle multiplexing to create multiple information channels that synchronously perform multi-order spatial differential operations, indicating the superiority of the proposed devices in parallel processing. Our approach may find various applications in artificial intelligence, machine vision, autonomous driving, and advanced biomedical imaging.

Designing Personalized Learning Paths for Foreign Language Acquisition Using Big Data: Theoretical and Empirical Analysis

This study introduces the Data-Driven Personalized Learning Model (DDPLM), a sophisticated framework designed to enhance foreign language acquisition through the integration of big data analytics. Implemented within the educational platforms Edmodo and Duolingo, DDPLM utilizes real-time data processing to tailor learning paths and content dynamically to individual learner needs. Our findings indicate significant improvements in language learning efficiency, engagement, and retention. The model’s adaptability across different digital environments showcases its potential scalability and effectiveness in various educational contexts. Additionally, the research addresses the critical role of personalized feedback and adaptive challenges in maintaining learner motivation and promoting deeper linguistic comprehension. The outcomes suggest that DDPLM significantly transforms traditional language education, making it more personalized, efficient, and aligned with individual learning preferences.

Enhancing the Heat Dissipation Efficiency of Computing Units Within Autonomous Driving Systems and Electric Vehicles

With the rapid development of autonomous driving technology and electric vehicles, the demand for computing units in vehicles has surged, particularly for high-density computing related to artificial intelligence, sensor fusion, and real-time data processing. This increase has posed significant heat dissipation challenges for automotive electronic systems. Effective heat dissipation is essential for ensuring system stability, safety, and extending the lifespan of these systems. This study explores the thermal design of computing units in current autonomous driving systems and electric vehicles. The electronic components within the computing units generate significant heat, but due to environmental constraints, fans can not be used for internal cooling, as their operation can draw in dust from the environment, contaminating the internal electronic components. Therefore, computing units are often designed with a sealed structure, utilizing the thermal conductivity of metal casings to quickly dissipate internal heat sources. The results of this study show that comparing embedded cooling modules within the casing to commonly used thermal pads reveals that changing the materials of the cooling module can significantly enhance cooling performance, noticeably reducing the temperature of the internal electronic components and minimizing the risk of overheating. Additionally, this can improve the overall reliability and performance of the system, providing new ideas and application prospects for future automotive thermal design.

What does e-invoice data bring to SNA and real-time economy?

This paper explores the potential of electronic invoice data to enhance the System of National Accounts (SNA) and facilitate a Real-Time Economy (RTE). The rise of data science and advancements in information and communication technologies have increased the use of big data in economic statistics. E-invoice data is being digitized in Europe to detect VAT fraud, but it also contains a vast amount of information on economic entities that make up the SNA. This study focuses on integrating electronic accounting information to improve the coverage, granularity, and update speed of SNA. The paper examines the concept of RTE, which emphasizes real-time data processing and automation, particularly in Estonia and Finland. It also highlights the benefits and challenges of using invoice data for economic statistics, including issues related to data storage formats, legislative systems, and the coordination of data linkages. The research demonstrates invoice data’s theoretical and practical applications for constructing Input–Output Tables (IOTs) and monitoring financial situations. Despite the technical and institutional hurdles, the study suggests integrating invoice data can significantly transform economic statistics.