Data Processing Requirements Research Articles

A Diffusion–Attention-Enhanced Temporal (DATE-TM) Model: A Multi-Feature-Driven Model for Very-Short-Term Household Load Forecasting

With the proliferation of smart home devices and the ever-increasing demand for household energy management, very-short-term load forecasting (VSTLF) has become imperative for energy usage optimization, cost saving and for sustaining grid stability. Despite recent advancements, VSTLF in the household scenario still poses challenges. For instance, some characteristics (e.g., high-frequency, noisy and non-stationary) exacerbate the data processing and model training procedures, and the heterogeneity in household consumption patterns causes difficulties for models with the generalization capability. Further, the real-time data processing requirement calls for both the high forecasting accuracy and improved computational efficiency. Thus, we propose a diffusion–attention-enhanced temporal (DATE-TM) model with multi-feature fusion to address the above issues. First, the DATE-TM model could integrate residents’ electricity consumption patterns with climatic factors. Then, it extracts the temporal feature using an encoder and meanwhile models the data uncertainty through a diffusion model. Finally, the decoder, enhanced with the attention mechanism, creates the precise prediction for the household load forecasting. Experimental results reveal that DATE-TM significantly surpasses classical neural networks such as BiLSTM and DeepAR, especially in handling the data uncertainty and long-term dependency.

The prevalence of gas exchange data processing methods: a semi-automated scoping review.

Cardiopulmonary exercise testing involves collecting variable breath-by-breath data, sometimes requiring data processing of outlier removal, interpolation, and averaging before later analysis. These data processing choices, such as averaging duration, affect calculated values such as VO2max. However, assessing the implications of data processing without knowing popular methods worth comparing is difficult. In addition, such details aid study reproduction. We conducted a semi-automated scoping review of articles with exercise testing that collected data breath-by-breath from three databases. Of the 8,344 articles, 376 (mean 4.5%, 95% CI: 4.1-5.0%) and 581 (7.0%, 6.4-7.5%) described outlier removal and interpolation, respectively. A random subset of 1,078 articles revealed 60.9% (57.9-63.7%) reported averaging methods. Commonly documented outlier cutoffs were ± 3 or 4 SD (39.1% and 51.6%, respectively). The dominating interpolation duration and procedure were one second (93.9%) and linear interpolation (92.5%). Averaging methods commonly described were 30 (30.9%), 60 (12.4%), 15 (11.6%), 10 (11.0%), and 20 (8.1%) second bin averages. This shows that studies collecting breath-by-breath data often lack detailed descriptions of data processing methods, particularly for outlier removal and interpolation. While averaging methods are more commonly reported, improved documentation across all processing steps will enhance reproducibility and facilitate future research comparing data processing choices.

ОРГАНІЗАЦІЯ СИСТЕМИ ЗАХИСТУ ЦИФРОВОЇ ІНФОРМАЦІЇ У ФІНАНОВОМУ ОБЛІКУ

Abstract. Introduction. The constant development of technologies, changing forms of threats and the need to adapt to new conditions will require organizations to constantly improve their security systems. The purpose of this work is to systematize theoretical provisions and substantiate practical recommendations regarding the organization of protection of digital financial accounting data. Methods. In the process of research, the method of induction was used at the stage of collecting, processing and systematization of the received information, the method of analysis and synthesis - to combine the constituent elements of economic phenomena in a single process. Results. The essential characteristics of the concepts of information security, digital security and data cyber security in the context of economic security of the enterprise are revealed. A set of measures in the system of organizing digital security of financial accounting data has been identified. National and international legal acts, standards and recommendations define mandatory requirements for data storage, processing and transmission, thus providing a legal basis for the implementation of information security policies. The main directions of state policy in the field of information security have been determined. The state in which business entities currently work is accompanied by constant cyber-attacks both on the enterprises themselves and on the totality of their digital data in the field of accounting and taxation. The main measures to ensure cyber security for critical infrastructure enterprises have been determined. They singled out the main problems of digital information protection in accounting, namely: data leakage; software attacks; phishing and social engineering; malware. The main methods of preventing cyber threats are summarized. Recommendations on the organization of the system for ensuring the security of digital data in financial accounting have been formed. The actions of the accounting staff of the enterprise or the enterprise of the partner, which may create a threat of information leakage, are singled out. A set of information sources in need of protection has been identified. Conclusions. The digital information protection system should become an element of the company's accounting policy, and its implementation will contribute to the economic security of the business entity.

Research Progress in the Field of MapReduce Framework Based on Serverless Computing

In recent years, big data technology has presented a rapid development trend with each passing day, and its application in all walks of life is becoming more and more extensive and in-depth. At the same time, traditional computing frameworks are gradually showing serious challenges in performance bottlenecks and scalability in the face of massive data processing requirements, making it difficult to meet the growing needs of data processing and highly concurrent access scenarios. Firstly, this paper introduces MapReduce framework and serverless platforms, and then introduces the MapReduce framework model based on serverless platforms from several parts of implementation, application scenarios and technical principles. This paper argues that the serverless-based MapReduce framework will provide a more efficient and flexible solution for big data processing and analyzing, which is of great significance in promoting the development of data-driven decision-making and intelligent applications. The main content of this paper will be introduced from the basic overview of MapReduce computing framework and serverless platform, the implementation of serverless-based MapReduce computing framework, technical principles and application scenarios.

Human-AI Enabled Edge Computing for Data Processing: A Comprehensive Analysis

The exponential growth in data generation and processing requirements has driven the need for more efficient and intelligent computational approaches at the edge of networks. This comprehensive article investigates the integration of human expertise with AI-enabled edge computing systems, focusing on optimization strategies, implementation frameworks, and real-world applications. Through extensive analysis of implementation cases and performance metrics, the research demonstrates significant improvements in processing efficiency, with systems achieving a reduction in latency and improvement in decision accuracy through human-AI collaboration. The article presents a detailed framework for implementing these hybrid systems, addressing critical aspects including technical architecture requirements, integration guidelines, and risk mitigation strategies. Results indicate substantial benefits in operational efficiency, resource utilization, and decision-making capabilities across various industrial applications. While identifying implementation challenges and limitations, the article provides strategic recommendations for successful deployment and highlights opportunities for future advancement in the field. The article contributes significantly to understanding how organizations can effectively leverage human-AI collaboration in edge computing environments to address the growing demands of data processing in modern digital infrastructure.

Automating Data Pipelines with AI for Scalable, Real-Time Process Optimization in the Cloud

Modern data processing environments demand efficient, scalable solutions for handling massive data streams in real-time, yet traditional Extract, Transform, Load (ETL) pipelines face significant limitations in processing speed and adaptability. This article presents an AI-Enhanced Cloud Data Pipeline (AECDP) framework that combines Deep Learning-based Stream Processing (DLSP) with Adaptive Resource Management (ARM) for real-time data optimization. The framework introduces novel algorithms for stream processing, resource allocation, and quality assurance, including the Adaptive Stream Processing Algorithm (ASPA) and Anomaly Detection and Correction (ADC) system. The implementation utilizes a multi-cloud architecture with containerized microservices, enabling independent scaling and maintenance of pipeline components. Experimental results demonstrate the framework's effectiveness across various industry applications, including e-commerce, financial services, and manufacturing sectors. The system achieves consistent sub-second latency for real-time processing, linear throughput scaling, and optimal resource utilization across cloud instances. Additionally, the framework incorporates advanced security features and automated quality monitoring systems, ensuring robust and reliable data processing. The AECDP framework represents a significant advancement in data pipeline automation, providing organizations with a comprehensive solution for managing complex data processing requirements while maintaining high performance and reliability standards.

Increasing Security and Capacity in SDNFV Structures Via Edge Node Extension

The rapid proliferation of IoT devices and corresponding requirements for efficient data processing, Software-Defined Networking Function Virtualization (SDNFV) has come to be a key vehicle for agile management of network resources. The advanced SDNFV model proposed in this study is intended to resolve the two main challenges of security and scalability. The sensitivity of the data transmitted through IoT networks as they grow in size and intricacy requires improved security procedures to hold ill-suited access to their information and ensure its integrity. Encryption & Authentication Protocols: Integration of the encryption and authentication protocol together in a model, that secures data streams against potential cyber threats and threats, enhancing security paradigm. Additionally, the model tackles the scalability challenge by implementing multi-edge node support for distributed processing to better manage high volumes of IoT data. Such expansion is especially notable since it solves latency issues and bottlenecks in the network so a more resilient network structure. The current study compares the simulation results of this improved SDNFV model with existing models and showcases that amidst numerous architectures, the suggested model provides higher efficiency in terms of data privacy and processing capability. This latest development in SDNFV may be fundamental to future IoT platforms that are capable of providing a custom non-functioning IoT backbone to cope with the big data requirements of today's ever-growing network surrounding such devices.

Ensemble learning based on bi-directional gated recurrent unit and convolutional neural network with word embedding module for bioactive peptide prediction.

Bioactive peptides, as small protein fragments, are essential mediators of diverse physiological activities, such as antimicrobial, anti-inflammatory, anticancer, antioxidant, and immunomodulatory functions. Despite their substantial potential in pharmaceuticals and the food industry, conventional methods for peptide classification and activity prediction are limited by high costs, time-intensive procedures, and extensive data processing requirements. Here, we present BioPepPred-DLEmb, a novel computational model integrating Convolutional Neural Networks (CNNs) and Bidirectional Gated Recurrent Units (BiGRUs), augmented with natural language processing to encode amino acids into information-dense vectors. Evaluated across nine bioactive peptide datasets, BioPepPred-DLEmb demonstrates superior predictive accuracy (0.909) and sensitivity (0.911) compared to traditional methods. Through UMAP visualization and Kplogo analysis, the model effectively differentiates peptide activity states and identifies key biomarkers. The predicted antimicrobial peptides (Pred-AMPs) exhibit potent efficacy in vitro, achieving low micromolar inhibitory concentrations (2-16μmol/L) against pathogens such as Escherichia coli and Acinetobacter baumannii. These findings establish a robust foundation for bioactive peptide development, with implications for advancements in precision medicine, personalized therapies, and functional food innovations.

Development of Self-Powered Energy-Harvesting Electronic Module and Signal-Processing Framework for Wearable Healthcare Applications.

A battery-operated biomedical wearable device gradually assists in clinical tasks to monitor patients' health states regarding early diagnosis and detection. This paper presents the development of a self-powered portable electronic module by integrating an onboard energy-harvesting facility for electrocardiogram (ECG) signal processing and personalized health monitoring. The developed electronic module provides a customizable approach to power the device using a lithium-ion battery, a series of silicon photodiode arrays, and a solar panel. The new architecture and techniques offered by the developed method include an analog front-end unit, a signal processing unit, and a battery management unit for the acquiring and processing of real-time ECG signals. The dynamic multi-level wavelet packet decomposition framework has been used and applied to an ECG signal to extract the desired features by removing overlapped and repeated samples from an ECG signal. Further, a random forest with deep decision tree (RFDDT) architecture has been designed for offline ECG signal classification, and experimental results provide the highest accuracy of 99.72%. One assesses the custom-developed sensor by comparing its data with those of conventional biosensors. The onboard energy-harvesting and battery management circuits are designed with a BQ25505 microprocessor with the support of silicon photodiodes and solar cells which detect the ambient light variations and provide a maximum of 4.2 V supply to enable the continuous operation of an entire module. The measurements conducted on each unit of the proposed method demonstrate that the proposed signal-processing method significantly reduces the overlapping samples from the raw ECG data and the timing requirement criteria for personalized and wearable health monitoring. Also, it improves temporal requirements for ECG data processing while achieving excellent classification performance at a low computing cost.

Cloud-Native Microservices for Real-Time Data Systems: A Technical Deep Dive

Cloud-native microservices have emerged as a transformative solution for organizations facing the challenges of real-time data processing and analysis in today's digital landscape. This comprehensive article explores the fundamental components of cloud-native architectures, from their foundational principles to implementation best practices. The article examines how containerization, orchestration, and event-driven architectures enable unprecedented scalability and efficiency in handling modern data processing requirements. Through a detailed examination of microservices as building blocks, the research demonstrates how these architectural patterns significantly improve system reliability, performance, and maintainability. The article encompasses advanced monitoring and observability practices, highlighting their crucial role in maintaining robust cloud-native systems. By investigating real-world implementations across various industries, this study provides valuable insights into the practical benefits and challenges of adopting cloud-native approaches.

Data Processing of Used Car Export Service Flow Based on Zookeeper + Kafka

The used car export industry has shown a trend of rapid growth in recent years. The export of used cars involves data pertaining to vehicle information, market demand, logistics, customs, and numerous other aspects. These data are constantly generated in the form of streams, posing challenges to the data processing capabilities of service platforms. Traditional data processing methods struggle to meet the requirements of real-time and large-scale data processing, thus necessitating an efficient streaming data processing architecture. Based on this, this paper proposes a streaming data processing scheme for a used car export service platform utilizing Zookeeper and Kafka. It describes the architecture design and data processing flow, analyzes the advantages of the scheme in terms of reliability, scalability, and real-time performance, and demonstrates its effectiveness through experiments and case studies.

TOMOMAN: a software package for large-scale cryo-electron tomography data preprocessing, community data sharing and collaborative computing.

Cryo-electron tomography (cryo-ET) and subtomogram averaging (STA) are becoming the preferred methodologies for investigating subcellular and macromolecular structures in native or near-native environments. Although cryo-ET is amenable to a wide range of biological problems, these problems often have data-processing requirements that need to be individually optimized, precluding the notion of a one-size-fits-all processing pipeline. Cryo-ET data processing is also becoming progressively more complex due to the increasing number of packages for each processing step. Though each package has its strengths and weaknesses, independent development and different data formats make them difficult to interface with one another. TOMOMAN (TOMOgram MANager) is an extensible package for streamlining the interoperability of packages, enabling users to develop project-specific processing workflows. TOMOMAN does this by maintaining an internal metadata format and wrapping external packages to manage and perform preprocessing, from raw tilt-series data to reconstructed tomograms. TOMOMAN can also export these metadata between various STA packages. TOMOMAN includes tools for archiving projects to data repositories, allowing subsequent users to download TOMOMAN projects and directly resume processing. By tracking essential metadata, TOMOMAN streamlines data sharing, which improves the reproducibility of published results, reduces computational costs by minimizing reprocessing, and enables the distribution of cryo-ET projects between multiple groups and institutions. TOMOMAN provides a way for users to test different software packages in order to develop processing workflows that meet the specific needs of their biological questions and to distribute their results to the broader scientific community.

Comparison of Hadoop Mapreduce and Apache Spark in Big Data Processing with Hgrid247-DE

In today’s rapidly evolving information technology landscape, managing and analyzing big data has become one of the most significant challenges. This paper explores the implementation of two major frameworks for big data processing: Hadoop MapReduce and Apache Spark. Both frameworks were tested in three scenarios sorting, summarizing, and grouping using HGrid247-DE as the primary tool for data processing. A diverse set of datasets sourced from Kaggle, ranging in size from 3 MB to 260 MB, was employed to evaluate the performance of each framework. The findings reveal that Apache Spark generally outperforms Hadoop MapReduce in terms of processing speed due to its in-memory data handling capabilities. However, Hadoop MapReduce proved to be more efficient in specific scenarios, particularly when dealing with smaller tasks or when memory resources are limited. This is largely because Apache Spark can experience overhead when initializing tasks for smaller jobs. Furthermore, Hadoop MapReduce's reliance on disk I/O makes it more suitable for tasks involving vast amounts of data that surpass available memory. In contrast, Spark excels in situations where quick iterative processing and real-time data analysis are essential. This study provides valuable insights into the strengths and limitations of each framework, offering guidance for practitioners and researchers when selecting the appropriate tool for specific big data processing requirements, particularly with respect to speed, memory usage, and task complexity.

Shadow Testing in Autonomous Vehicles : A Novel Approach to Validating Full Self-Driving AI Systems

As autonomous vehicles progress towards widespread deployment, ensuring the reliability and safety of Full Self-Driving (FSD) AI systems remains a critical challenge. This article presents a comprehensive analysis of shadow testing, an innovative approach to validating new FSD AI models in real-world conditions without compromising user safety. We examine the methodology of deploying new AI models in "shadow mode," where they process live sensory data alongside the operational system but do not control the vehicle. This approach enables the collection of valuable performance metrics and the identification of edge cases while mitigating risks associated with direct deployment. Our study demonstrates how shadow testing facilitates a continuous feedback loop for iterative improvement, enhancing model reliability through data-driven refinement. We further explore the challenges of implementing shadow testing at scale, including data processing requirements and ethical considerations. Our findings suggest that shadow testing not only accelerates the development of robust FSD systems but also plays a crucial role in building public trust in autonomous vehicle technology. This article contributes to the growing body of knowledge on AI safety validation techniques and provides insights for standardizing testing protocols in the autonomous vehicle industry.

Cybermycelium: a reference architecture for domain-driven distributed big data systems.

The ubiquity of digital devices, the infrastructure of today, and the ever-increasing proliferation of digital products have dawned a new era, the era of big data (BD). This era began when the volume, variety, and velocity of data overwhelmed traditional systems that used to analyze and store that data. This precipitated a new class of software systems, namely, BD systems. Whereas BD systems provide a competitive advantage to businesses, many have failed to harness the power of them. It has been estimated that only 20% of companies have successfully implemented a BD project. This study aims to facilitate BD system development by introducing Cybermycelium, a domain-driven decentralized BD reference architecture (RA). The artifact was developed following the guidelines of empirically grounded RAs and evaluated through implementation in a real-world scenario using the Architecture Tradeoff Analysis Method (ATAM). The evaluation revealed that Cybermycelium successfully addressed key architectural qualities: performance (achieving <1,000 ms response times), availability (through event brokers and circuit breaking), and modifiability (enabling rapid service deployment and configuration). The prototype demonstrated effective handling of data processing, scalability challenges, and domain-specific requirements in a large-scale international company setting. The results highlight important architectural trade-offs between event backbone implementation and service mesh design. While the domain-driven distributed approach improved scalability and maintainability compared to traditional monolithic architectures, it requires significant technical expertise for implementation. This contribution advances the field by providing a validated reference architecture that addresses the challenges of adopting BD in modern enterprises.

Using cross-recurrence quantification analysis to compute similarity measures for time series of unequal length with applications to sleep stage analysis

Comparing time series of unequal length requires data processing procedures that may introduce biases. This article describes, validates, and applies Cross-Recurrence Quantification Analysis (CRQA) to detect and quantify correlation and coupling among time series of unequal length without prior data processing. We illustrate and validate this application using continuous and discrete data from a model system (study 1). Then we use the method to re-analyze the Sleep Heart Health Study (SHHS), a rare large dataset comprising detailed physiological sleep measurements acquired by in-home polysomnography. We investigate whether recurrence patterns of ultradian NREM/REM sleep cycles (USC) predict mortality (study 2). CRQA exhibits better performance compared with traditional approaches that require trimming, stretching or compression to bring two time series to the same length. Application to the SHHS indicates that recurrence patterns linked to stability of USCs are associated with all-cause mortality even after controlling for other sleep parameters, health, and sociodemographics. We suggest that CRQA is a useful tool for analyzing categorical time series, where the underlying structure of the data is unlikely to result in matching data points—such as ultradian sleep cycles.

The RAS Paradigm in Mainframe Systems: A Comprehensive Analysis of Design Principles and Operational Benefits

This article examines the critical role of Reliability, Availability, and Serviceability (RAS) in mainframe computing systems. We explore how RAS principles have become fundamental design features in modern mainframes, enhancing their ability to meet the demanding requirements of enterprise-level data processing. The article provides an in-depth analysis of each RAS component, detailing how reliability is achieved through robust hardware self-checking and extensive software testing, availability is maintained via seamless component failover and layered error recovery, and serviceability is ensured through advanced diagnostic capabilities and modular replacement units. We argue that the holistic integration of RAS principles in mainframe architecture not only maximizes system uptime and operational continuity but also significantly impacts application design and overall system efficiency. Through a comprehensive review of current mainframe technologies and industry practices, this article highlights the enduring importance of RAS in an era of increasing computational complexity and datadriven decision making. Our findings suggest that RAS principles will continue to evolve, playing a crucial role in shaping the future of high-performance, mission-critical computing systems.

Dual-polarized transmissive metasurfaces for near-unitary-NA analog spatial computing empowered by impedance matching and mismatching.

In recent years, due to the increasing requirement for real-time and massive data processing, optical analog computation has arisen as a promising alternative to digital computation. Optical spatial differentiation plays a fundamentally important role in various emerging technologies, including augmented reality, autonomous driving, and object recognition. However, previous demonstrations encountered several limitations, such as the dependence on polarization states and a typically limited numerical aperture (NA) of about 0.5, especially in the transmission mode. Here, a new, to our knowledge, design strategy based on the evolution between impedance matching and mismatching in a metasurface is proposed to fill this gap, which can perform dual-polarized second-order derivative for image processing. Our scheme enables high transmission under dual polarization over an 85° incident angle range (NA = 0.996), resulting in more than twofold spatial resolution. Our work paves the way for polarization-insensitive high-resolution signal and image processing in the terahertz region.

LLM-Twin: mini-giant model-driven beyond 5G digital twin networking framework with semantic secure communication and computation

Beyond 5G networks provide solutions for next-generation communications, especially digital twins networks (DTNs) have gained increasing popularity for bridging physical and digital space. However, current DTNs pose some challenges, especially when applied to scenarios that require efficient and multimodal data processing. Firstly, current DTNs are limited in communication and computational efficiency, since they require to transmit large amounts of raw data collected from physical sensors, as well as to ensure model synchronization through high-frequency computation. Second, current models of DTNs are domain-specific (e.g. E-health), making it difficult to handle DT scenarios with multimodal data processing requirements. Finally, current security schemes for DTNs introduce additional overheads that impair the efficiency. Against the above challenges, we propose a large language model (LLM) empowered DTNs framework, LLM-Twin. First, based on LLM, we propose digital twin semantic networks (DTSNs), which enable more efficient communication and computation. Second, we design a mini-giant model collaboration scheme, which enables efficient deployment of LLM in DTNs and is adapted to handle multimodal data. Then, we designed a native security policy for LLM-twin without compromising efficiency. Numerical experiments and case studies demonstrate the feasibility of LLM-Twin. To our knowledge, this is the first to propose an LLM-based semantic-level DTNs.

TOMOMAN: a software package for large scale cryo-electron tomography data preprocessing, community data sharing, and collaborative computing.

Cryo-electron tomography (cryo-ET) and subtomogram averaging (STA) are becoming the preferred methodologies for investigating subcellular and macromolecular structures in native or near-native environments. While cryo-ET is amenable to a wide range of biological problems, these problems often have data processing requirements that need to be individually optimized, precluding the notion of a one-size-fits-all processing pipeline. Cryo-ET data processing is also becoming progressively more complex due to an increasing number of packages for each processing step. Though each package has its own strengths and weaknesses, independent development and different data formats makes them difficult to interface with one another. TOMOMAN (TOMOgram MANager) is an extensible package for streamlining the interoperability of packages, enabling users to develop project-specific processing workflows. TOMOMAN does this by maintaining an internal metadata format and wrapping external packages to manage and perform preprocessing, from raw tilt-series data to reconstructed tomograms. TOMOMAN can also export this metadata between various STA packages. TOMOMAN also includes tools for archiving projects to data repositories; allowing subsequent users to download TOMOMAN projects and directly resume processing where it was previously left off. By tracking essential metadata, TOMOMAN streamlines data sharing, which improves reproducibility of published results, reduces computational costs by minimizing reprocessing, and enables distributed cryo-ET projects between multiple groups and institutions. TOMOMAN provides a way for users to test different software packages to develop processing workflows that meet the specific needs of their biological questions and to distribute their results with the broader scientific community.