Processing Latency Research Articles

Deep Learning-Based Real-Time Data Quality Assessment and Anomaly Detection for Large-Scale Distributed Data Streams

Time delay and data quality degradation pose significant challenges in large-scale distributed data streams processing. This paper proposes a deep learning-based realtime data quality assessment and anomaly detection method for distributed streaming data environments. The proposed approach integrates quality-aware feature extraction with adaptive deep neural networks to enable real-time quality monitoring and anomaly detection. A multi-dimensional quality assessment framework is developed, incorporating temporal-spatial correlations and stream characteristics for comprehensive quality evaluation. The system implements a distributed architecture with parallel processing capabilities, enabling scalable operations across multiple nodes while maintaining low-latency responses. A novel online learning mechanism is introduced to adapt model parameters dynamically, ensuring robust performance under evolving data patterns. Experimental evaluation conducted on three large-scale datasets, including industrial IoT sensors (2.5TB), network traffic (1.8TB), and financial transactions (3.2TB), demonstrates superior performance compared to traditional methods. The system achieves 97.8% detection accuracy while maintaining processing latency below 10ms, with linear scalability up to 128 nodes. Results show consistent performance improvement across different operational scenarios, with 95% precision in anomaly detection and throughput exceeding 1.2 million events per second.

A Cost-Minimized Task Migration Assignment Mechanism in Blockchain Based Edge Computing System

Background: Cloud computing is usually introduced to execute computing intensive tasks for data processing and data mining. As a supplement to cloud computing, edge computing is provided as a new paradigm to effectively reduce processing latency, energy consumption cost and bandwidth consumption for time-sensitive tasks or resource-sensitive tasks. To better meet such requirements during task assignment in edge computing systems, an intelligent task migration assignment mechanism based on blockchain is proposed, which jointly considers the factors of resource allocation, resource control and credit degree. Methods: In this paper, an optimization problem is firstly constructed to minimize the total cost of completing all tasks under constraints of delay, energy consumption, communication, and credit degree. Here, the terminal node mines computing resources from edge nodes to complete task migration. An incentive method based on blockchain is provided to mobilize the activity of terminal nodes and edge nodes, and to ensure the security of the transaction during migration. The designed allocation rules ensure the fairness of rewards for successfully mining resource. To solve the optimization problem, an intelligent migration algorithm that utilizes a dual “actor-reviewer” neural network on inverse gradient update is proposed which makes the training process more stable and easier to converge. Results: Compared to the existing two benchmark mechanisms, the extensive simulation results indicate that the proposed mechanism based on neural network can converge at a faster speed and achieve the minimal total cost. Conclusion: To satisfy the requirements of delay and energy consumption for computing intensive tasks in edge computing scenarios, an intelligent, blockchain based task migration assignment mechanism with joint resource allocation and control is proposed. To realize this mechanism effectively, a dual “actor-reviewer” neural network algorithm is designed and executed.

A Hybrid LSTM-KNN Framework for Detecting Market Microstructure Anomalies:

This paper proposes a novel hybrid LSTM-KNN framework for detecting market microstructure anomalies in high-frequency credit default swap (CDS) markets. The framework integrates the temporal learning capabilities of Long Short-Term Memory networks with the pattern recognition strengths of K-Nearest Neighbors classification to identify price jumps and market anomalies. Through analysis of high-frequency CDS market data spanning from 2020 to 2023, encompassing over 2.5 million data points from five major CDS indices, the research demonstrates significant improvements in jump detection accuracy. The hybrid model achieves a 92.8% accuracy rate, representing a 15.2% improvement over traditional statistical methods and an 8.5% enhancement compared to standalone deep learning approaches. The framework maintains computational efficiency with an average processing latency of 48.2 milliseconds, enabling real-time market applications. The empirical analysis reveals strong correlations between detected jumps and market liquidity conditions, with bid-ask spreads and order book imbalances identified as critical predictive indicators. The research contributes to both theoretical understanding of market microstructure dynamics and practical applications in risk management and market surveillance.

Automating Data Pipelines with AI for Scalable, Real-Time Process Optimization in the Cloud

Modern data processing environments demand efficient, scalable solutions for handling massive data streams in real-time, yet traditional Extract, Transform, Load (ETL) pipelines face significant limitations in processing speed and adaptability. This article presents an AI-Enhanced Cloud Data Pipeline (AECDP) framework that combines Deep Learning-based Stream Processing (DLSP) with Adaptive Resource Management (ARM) for real-time data optimization. The framework introduces novel algorithms for stream processing, resource allocation, and quality assurance, including the Adaptive Stream Processing Algorithm (ASPA) and Anomaly Detection and Correction (ADC) system. The implementation utilizes a multi-cloud architecture with containerized microservices, enabling independent scaling and maintenance of pipeline components. Experimental results demonstrate the framework's effectiveness across various industry applications, including e-commerce, financial services, and manufacturing sectors. The system achieves consistent sub-second latency for real-time processing, linear throughput scaling, and optimal resource utilization across cloud instances. Additionally, the framework incorporates advanced security features and automated quality monitoring systems, ensuring robust and reliable data processing. The AECDP framework represents a significant advancement in data pipeline automation, providing organizations with a comprehensive solution for managing complex data processing requirements while maintaining high performance and reliability standards.

A Systematic Review of Fast, Scalable, and Efficient Hardware Implementations of Elliptic Curve Cryptography for Blockchain

Blockchain technology entered the enterprise domain under the name of permissioned blockchains and hybrid or verifiable database systems, as they provide a distributed solution that allows multiple distrusting parties to share common information. One drawback of these systems is the overhead added by the cryptographic functions which impacts the throughput in terms of transactions per second and increases the latency of transaction processing. Many of the cryptographic functions and protocols used in blockchains are based on Elliptic Curve Cryptography (ECC). Unfortunately, ECC operations such as modulo inverse or scalar point multiplication have considerable latency which causes the slowdown of the entire system. In such situations, reconfigurable computing architectures, such as FPGAs, can be used to offload these tasks to overcome the performance loss. This survey analyzes the current state-of-the-art designs and implementations of ECC from a hardware perspective. We use a PRISMA-based approach to filter recent publications and to reduce their number from over 16,000 to only 43 highly relevant designs. In the end, we show that very few designs are able to fulfill all three properties of high performance, scalability, and efficiency.

Emotional expressions, but not social context, modulate attention during a discrimination task

ABSTRACT Investigating social context effects and emotional modulation of attention in a laboratory setting is challenging. Electroencephalography (EEG) requires a controlled setting to avoid confounds, which goes against the nature of social interaction and emotional processing in real life. To bridge this gap, we developed a new paradigm to investigate the effects of social context and emotional expressions on attention in a laboratory setting. We co-registered eye-tracking and EEG to assess gaze behavior and brain activity while participants performed a discrimination task followed by feedback. Video clips of one second in which a confederate displayed either positive, neutral or negative expressions were presented as feedback to the discrimination task. Participants’ belief was manipulated by telling them that the videos were selected either by the computer (non-social condition) or by the experimenter in the adjacent room that observed them via videochat (social condition). We found that emotional expressions modulated late attention processing in the brain (EPN and LPC), but neither early processing (P1) nor saccade latency. Social context did not influence any of the variables studied. We conclude this new paradigm serves as a stepping stone to the development of new paradigms to study social interaction within EEG experiments.

Robust neural networks using stochastic resonance neurons

Various successful applications of deep artificial neural networks are effectively facilitated by the possibility to increase the number of layers and neurons in the network at the expense of the growing computational complexity. Increasing computational complexity to improve performance makes hardware implementation more difficult and directly affects both power consumption and the accumulation of signal processing latency, which are critical issues in many applications. Power consumption can be potentially reduced using analog neural networks, the performance of which, however, is limited by noise aggregation. Following the idea of physics-inspired machine learning, we propose here a type of neural network using stochastic resonances as a dynamic nonlinear node and demonstrate the possibility of considerably reducing the number of neurons required for a given prediction accuracy. We also observe that the performance of such neural networks is more robust against the impact of noise in the training data compared to conventional networks.

High throughput edit distance computation on FPGA-based accelerators using HLS

Edit distance is a computational grand challenge problem to quantify the minimum number of editing operations required to modify one string of characters to the other, finding many applications of natural language processing. In recent years, relevant and increasing interest has also emerged from deoxyribonucleic acid (DNA) applications, like Next Generation Sequencing and DNA storage technologies. Both applications share two crucial features: i) the information is coded into the four bases of DNA and ii) the level of operational noise is still high causing errors in the data, requiring inclusion in the workflow of the computation of algorithms such as the edit distance for finding similarities between sequences. To boost this computation many solutions are available in the literature. Among them, the FPGAs are largely used since the data domain of those applications is strings of 4 characters represented as two-bit values, inconveniently fitting the basic data types of ordinary CPUs and GPUs, with additional benefits of providing a high level of parallelism and low processing latency. This contribution presents a computing- and energy-efficient design implementing the edit distance algorithm combining metaprogramming and High-Level Synthesis. We also assess the performance of our design targeting recent FPGA-based accelerators. Our solution uses nearly 90% of FPGA basic-block hardware resources achieving about 90% of computing efficiency delivering a maximum throughput of 16.8 TCUPS and an energy efficiency of 46 Mpair/Joule, enabling the use of FPGAs as a new class of accelerators for High Performance Computing in DNA applications.

Enhancing Rate-Splitting-Based Distributed Edge Computing via Multi-Group Information Recycling

To address the straggling effect in distributed edge computing, existing methods often introduce extra computation or communication costs. Recently, information recycling has emerged as an efficient solution that avoids such extra overhead. Nonetheless, the performance of the existing information recycling-assisted rate-splitting-based distributed edge computing scheme significantly hinges on the single common stream, whose rate is limited by the edge node (EN) with the worst channel quality. To this end, a multi-group information recycling scheme for rate-splitting-based distributed edge computing is proposed, where the ENs are clustered into multiple groups, each with its own common stream. In this way, the proposed scheme alleviates the communication limitation of a single common stream, thereby boosting the information recycling mechanism for promoted computation collaboration. A K-medoids-based grouping algorithm is designed for the general multi-antenna case, and a more efficient continuous partitioning-based grouping algorithm is proposed for the special case of single-antenna. Besides, a convex–concave procedure-based algorithm is developed to solve the corresponding latency optimization problem. Simulations demonstrate that the proposed scheme offers significant and robust advantages across various communication and computation conditions. In the considered scenarios, the proposed scheme can substantially reduce the processing latency by up to 51.0% compared to the conventional information recycling scheme.

GTBTL-IoT: An Approach of Curtailing Task Offloading Time for Improved Responsiveness in IoT-MEC Model

INTRODUCTION: The Internet of Things (IoT) has transformed daily life by interconnecting digital devices via integrated sensors, software, and connectivity. Although IoT devices excel at real-time data collection and decision-making, their performance on complex tasks is hindered by limited power, resources, and time. To address this, IoT is often combined with cloud computing (CC) to meet time-sensitive demands. However, the distance between IoT devices and cloud servers can result in latency issues. OBJECTIVES: To mitigate latency challenges, Mobile Edge Computing (MEC) is integrated with IoT. MEC offers cloud-like services through servers located near network edges and IoT devices, enhancing device responsiveness by reducing transmission and processing latency. This study aims to develop a solution to optimize task offloading in IoT-MEC environments, addressing challenges like latency, uneven workloads, and network congestion. METHODS: This research introduces the Game Theory-Based Task Latency (GTBTL-IoT) algorithm, a two-way task offloading approach employing Game Matching Theory and Data Partitioning Theory. Initially, the algorithm matches IoT devices with the nearest MEC server using game-matching theory. Subsequently, it splits the entire task into two halves and allocates them to both local and MEC servers for parallel computation, optimizing resource usage and workload balance. RESULTS: GTBTL-IoT outperforms existing algorithms, such as the Delay-Aware Online Workload Allocation (DAOWA) Algorithm, Fuzzy Algorithm (FA), and Dynamic Task Scheduling (DTS), by an average of 143.75 ms with a 5.5 s system deadline. Additionally, it significantly reduces task transmission, computation latency, and overall job offloading time by 59%. Evaluated in an ENIGMA-based simulation environment, GTBTL-IoT demonstrates its ability to compute requests in real-time with optimal resource usage, ensuring efficient and balanced task execution in the IoT-MEC paradigm. CONCLUSION: The Game Theory-Based Task Latency (GTBTL-IoT) algorithm presents a novel approach to optimize task offloading in IoT-MEC environments. By leveraging Game Matching Theory and Data Partitioning Theory, GTBTL-IoT effectively reduces latency, balances workloads, and optimizes resource usage. The algorithm's superior performance compared to existing methods underscores its potential to enhance the responsiveness and efficiency of IoT devices in real-world applications, ensuring seamless task execution in IoT-MEC systems.

“Can’t Remember What I Forgot:” Investigating Organizational Forgetting Within a Project-Based Organization

This article is based on the study of an engineering organization that is experiencing major difficulties on its first new project in 10 years. It contributes to the literature on organizational forgetting in project-based organizations, examining how this phenomenon can be perceived and assessed. The analysis of the data provides a better understanding of the conditions under which a situation of accidental forgetting can be (mis)identified. The case reveals that, due to five sources of ambiguity linked to the identification process—latency, novelty, multiplicity, complexity, and credibility—the forgetting phenomenon may be underestimated or the subject of contradictory assessments within the organization.

Robust deadline-aware network function parallelization framework under demand uncertainty

The orchestration of Service Function Chains (SFCs) in Mobile Edge Computing (MEC) becomes crucial for ensuring efficient service provision, especially under dynamic and uncertain demand. Meanwhile, the parallelization of Virtual Network Functions (VNFs) within an SFC can further optimize resource usage and reduce the risk of deadline violations. However, most existing works formulate the SFC orchestration problem in MEC with deterministic demands and costly runtime resource reprovisioning to handle dynamic demands. This paper introduces a Robust Deadline-aware network function Parallelization framework under Demand Uncertainty (RDPDU) designed to address the challenges posed by unpredictable fluctuations in user demand and resource availability within MEC networks. RDPDU to consider end-to-end latency for SFC assembly by modeling load-dependent processing latency and load-independent propagation latency. Also, RDPDU formulates the problem assuming uncertain demand by Quadratic Integer Programming (QIP) to be resistant to dynamic service demand fluctuations. By discovering dependencies between VNFs, the RDPDU effectively assembles multiple sub-SFCs instead of the original SFC. Finally, our framework uses Deep Reinforcement Learning (DRL) to assemble sub-SFCs with guaranteed latency and deadline. By integrating DRL into the SFC orchestration problem, the framework adapts to changing network conditions and demand patterns, improving the overall system's flexibility and robustness. Experimental evaluations show that the proposed framework can effectively deal with demand fluctuations, latency, deadline, and scalability and improve performance against recent algorithms.

CUAHN-VIO: Content-and-uncertainty-aware homography network for visual-inertial odometry

Learning-based visual ego-motion estimation is promising yet not ready for navigating agile mobile robots in the real world. In this article, we propose CUAHN-VIO, a robust and efficient monocular visual-inertial odometry (VIO) designed for micro aerial vehicles (MAVs) equipped with a downward-facing camera. The vision frontend is a content-and-uncertainty-aware homography network (CUAHN). Content awareness measures the robustness of the network toward non-homography image content, e.g. 3-dimensional objects lying on a planar surface. Uncertainty awareness refers that the network not only predicts the homography transformation but also estimates the prediction uncertainty. The training requires no ground truth that is often difficult to obtain. The network has good generalization that enables “plug-and-play” deployment in new environments without fine-tuning. A lightweight extended Kalman filter (EKF) serves as the VIO backend and utilizes the mean prediction and variance estimation from the network for visual measurement updates. CUAHN-VIO is evaluated on a high-speed public dataset and shows rivaling accuracy to state-of-the-art (SOTA) VIO approaches. Thanks to the robustness to motion blur, low network inference time (∼23 ms), and stable processing latency (∼26 ms), CUAHN-VIO successfully runs onboard an Nvidia Jetson TX2 embedded processor to navigate a fast autonomous MAV.

A Comprehensive of CPU and GPU Performance and Applications in Autonomous Vehicles

The rapid advancement of autonomous vehicle technology over the past decade has significantly increased the complexity of intelligent transportation systems. This need is clearly reflected in the requirements for high-performance computing in autonomous vehicles — a requirement that artificial intelligence, machine learning and big data analytic have all come to meet through their integration. We need the CPU and GPU power to handle processing in real-time data, sensor fusion and decision-making. We investigate how the CPU and GPU perform in an autonomous driving scenario, concluding that these tasks are complementary to one another describing where each can be best used within a heterogeneous computing architecture. The research delves into how these computing chips can be best used under the demand for real-time processing, reliability and efficiency. The study is to better understand the techniques for improving CPU-GPU collaboration, and applies these new findings on performance-intensive tasks like image recognition or track construction. The technical part involves conducting driving scenarios in the real world on three different types of CPU and several GPU, with key metrics such as processing latency, power consumption and accuracy. The actual experimental results show the advantages of GPU parallel computing and deep learning, while CPU still has a good advantage in multi-tasking ability and logical calculation. The results are important for the implementation of future autonomous driving systems, highlighting heterogeneous computing architectures as a means to optimize and support safe, effective vehicle operations.

Edge Computing vs. Cloud Computing: A Comparative Analysis for Real-Time AI Applications

AbstractReal-time AI has been evidenced to be supported by two models, namely edge computing and cloud computing. This research seeks to compare the two methods: Latency, security features, and data processing capabilities will be among the most critical aspects of comparison. Real-time AI applications are envisioned to be robust through 2020, most of which will be used in applications such as autonomous automobile systems and IoT devices that demand low latency for data processing. As data processing is outsourced in clouds, it is elastic and centralized, but this deal faces latency problems when the distance between the source of data and the processing center is large. On the other hand, edge computing refers to conducting computation closer to the data source, which could reduce latency and enhance everyday real-time performance.This research assesses the above models based on the literature review, technical papers, and case studies from industries that depend more on real-time AI. According to the research, edge computing is typically more effective for latency-sensitive workloads, while cloud computing outperforms throughput-intensive applications. Security concerns, however, present themselves as having dual effects; while advancing the privacy of handling data through edge computing, it elevates new risks. As such, this study concludes that no clear winner depends on the necessary application, therefore recommending a symmetric mode for the requirements, where both latency time and computational needs are required. Further studies should be conducted to establish the coordinated implementation approach of these models.

A node deployment and resource optimization method for CPDS based on cloud‐fog‐edge collaboration

AbstractWith the development of the Internet of Things (IoT) in power distribution and the advancement of energy information integration technologies, the explosive growth in network data volume caused by massive terminal devices connecting to the power distribution network has become a significant challenge. Multi‐terminal collaborative computing is a key approach to addressing issues such as high latency and high energy consumption. In this article, fog computing is introduced into the computing network of the power distribution system, and a cloud‐fog‐edge collaborative computing architecture for intelligent power distribution networks is proposed. Within this framework, an improved weighted K‐means method based on information entropy theory is presented for node partitioning. Subsequently, an improved multi‐objective particle swarm optimization algorithm (MWM‐MOPSO) is employed to solve the task resource allocation problem. Finally, the effectiveness of the proposed architecture and allocation strategy is validated through simulations on the OPNET and PureEdgeSim platforms. The results demonstrate that, compared to traditional cloud‐edge service architectures, the proposed architecture and task offloading scheme achieve better performance in terms of processing latency and energy consumption.

AI For Real-Time Detection And Mitigation Of Fake News Via Multi-Source Verification

The drought of fake news in the cover of the digital age makes information quality and its influence by senses. To address this issue, this paper introduces an innovative process to counter the misinformation spread by designing a real-time detection and mitigation model which is based on an AI Environment. Existing solutions do postpublication fact-checking, but our proposed system performs cross-verification of news sources for extracting facts from publications, it observes language patterns and compares content with trusted databases, analyzing the reliability among them in real-time. We empirically show that this approach achieves a significant improvement in performance over the state-of-the-art on both the LIAR dataset and real-time social media analysis. Our results demonstrate that the system is able to attain 89% of accuracy on detecting fake news, with a processing latency lower than 1.5s per article. It provides a real-time scalable solution to detect and combat fake news, assisting in the continued fight to keep online information accurate

Evaluating the Possibilities and Difficulties of Edge Computing's Real-time Data Processing

This study examines real-time data processing within the framework of edge computing, evaluating its importance, difficulties, and uses. It incorporates a thorough methodology that includes critical analysis, experimental findings, and a review of the literature. The goals are to comprehend the fundamentals of edge computing, assess its applicability to real-time data processing, and pinpoint its advantages and disadvantages. Results highlight how important edge computing is for lowering latency in data processing, saving bandwidth on networks, and improving scalability—especially for AR/VR and driverless car applications.

Optimization model for vehicular network data queries in edge environments

As the Internet of Vehicles advances, the demand for timely data acquisition by vehicle users continues to escalate, albeit confronted with the challenge of excessive data retrieval latency. The emergence of edge computing provides technical support for the development of vehicular networks by caching data in advance to reduce data acquisition latency. Therefore, how to effectively cache and query data becomes a key issue in addressing the timeliness of data acquisition in vehicular networks. In this paper, we investigate an efficient query optimization model to minimize data acquisition latency. Firstly, based on the distribution of data query frequencies across different servers, we propose an edge collaborative caching strategy using a tabu search algorithm. This strategy prioritizes high-traffic data by finding two optimal storage nodes for each high-traffic data in descending order of data popularity, ensuring a backup within the collaborative domain for each data segment. This not only reduces data transmission latency between nodes during task execution but also prevents single-point failures. Secondly, we deploy cuckoo filters on edge nodes to enable rapid localization of cached data nodes when users query data, thus reducing data processing latency. Finally, simulation results demonstrate that the proposed query optimization model outperforms other schemes in terms of average data query latency.

Performance Testing Techniques for Live TV Streaming on STBs

In the realm of live TV streaming, performance testing is critical to ensure seamless delivery and optimal user experience on Set-Top Boxes (STBs). This paper presents a comprehensive exploration of performance testing techniques tailored for live TV streaming on STBs, focusing on methodologies, challenges, and best practices. With the growing demand for high-quality, uninterrupted live TV content, effective performance testing becomes imperative to address issues related to latency, buffering, and overall system reliability. The study begins with an overview of the unique performance challenges associated with live TV streaming on STBs. Unlike on-demand streaming, live TV requires real-time data processing and minimal latency to deliver a continuous viewing experience. The paper categorizes performance testing into several key areas: network performance, system resource utilization, and user experience. Network performance testing evaluates the efficiency of data transmission across various network conditions. Techniques such as bandwidth measurement, latency analysis, and jitter assessment are employed to simulate different network scenarios and determine their impact on streaming quality. These tests help in identifying bottlenecks and optimizing network configurations to support smooth streaming.