Data Streams Research Articles

A Novel Deep Learning Model for Stock Prediction Integrating Historical Prices, Core Volatility Trends and Financial News

Accurate stock price forecasting is crucial for investors and analysts aiming to make informed decisions and mitigate risks in financial markets. While deep learning models have proven effective in identifying complex patterns in quantitative data, they often struggle to account for non-quantitative factors such as corporate events, market sentiment, and regulatory changes that can cause unpredictable stock price movements. To address this challenge, an ensemble model is introduced that integrates historical price data, financial news, and stock fundamentals to predict next-day stock prices. The model combines one-dimensional Convolutional Neural Networks (1D CNNs), Long Short-Term Memory networks (LSTMs), and Gated Recurrent Units (GRUs), applied independently to each data stream and concurrently as a unified ensemble, to capture both short-term and long-term market behaviors. NVIDIA (NVDA) stock was selected for testing due to its high volatility and rapid price fluctuations, presenting a challenging case for predictive modeling. Using data spanning from April 2014 to March 2024, the model achieved a coefficient of determination (R²) of 0.983 and a mean absolute error (MAE) of $12.72. These results suggest that the ensemble approach effectively captures both quantitative and qualitative factors, making it a promising tool for improving stock price prediction in volatile market conditions.

6.E. Practice session: Using European environmental data in public health: datasets and applications

Abstract The European Environment Agency collects, quality-assures and quality-checks data on environment, climate and sustainability. It does so through its network and other institutional partners across 38 European countries. Most of these validated environmental data are publicly available and many can be used by practitioners to assess environmental risks to the health of their communities, as well as for designing and evaluating policies’ effectiveness. The objective of this workshop is to provide public health practitioners and scientists with an overview of the EEA data streams of public health relevance, including on air pollution, noise, water pollution, industrial emissions and climate impacts, among others. It will also highlight existing examples of local and national assessments and communication products based on these data, as well as demonstrating how specifically data on air pollution can be used for a quick health impact evaluation. The format of the workshop will be a practice session of 60 minutes with four presentations, the last being a live example demonstration, and an ample session of questions and answers. Key messages • The EEA collects and publishes large, validated datasets of public health from across 38 member countries, including on air and water pollution, industrial emissions, noise and climate change. • These data can be directly used by practitioners to assess environmental risks to the health of their communities, and to design and evaluate policies, with publicly available models and tools.

Environmental Public Health Tracking in England- support for Health Impact Assessments

Abstract Environmental epidemiology is a key function for the public health service with regards to investigating the public health effects of environmental hazards, monitoring the public health efforts to control the environmental risks, and providing the evidence that informs decisions making and preparedness. At the United Kingdom Health Security Agency (UKHSA), the Environmental Public Health Tracking (EPHT) Programme, initiated in 2010, has developed a number of data streams to help link environment and health data, to support Health Impact Assessment (HIA). The main aims of the EPHT programme in England are to: 1. Identify populations at risk from exposure to significant environmental and chemical hazards 2. Investigate relationships between hazard and disease 3. Inform public health policy making 4. Optimise intervention and prevention strategies 5. Generate hypotheses for further research. Such EPHT functions are based on three key components: a) by building a national EPHT program that integrates surveillance of environmental hazards, exposures and related health outcomes, to allow data linkage and assessment of environmental and health information; b) strengthening the epidemiology science, by conducting research and training of the workforce and c) developing strategies for response to immediate risks, incidents and preparing for future risks. Examples to support HIA include a) developing an air pollution exposure surveillance work programme, that has designed a pilot population vulnerability indicator; b) developing indicators for climate change impacts for public health, c) reporting regularly on the health impacts of public fluoridation water supply schemes (as mandated by Government) and d) exploring risk factors associated with elevated blood lead concentration in children, through laboratory reporting to surveillance systems. The talk will highlight some examples of practice and outline how the EPHT programme in England has supported HIA practices.

EEA data streams of public health relevance: the examples of air pollution, environmental noise and industrial emissions

Abstract The knowledge work of the European Environment Agency relies on high-quality data from our network and other institutional partners across 38 European countries. We collect, quality-assure and quality-check data on the environment, climate and sustainability, many of which are of high public health relevance. This presentation will summarize the most importance EEA datasets of public health relevance, including on air pollution, noise, industrial emissions, and others.

Decentralized Machine Learning Framework for the Internet of Things: Enhancing Security, Privacy, and Efficiency in Cloud-Integrated Environments

The Internet of things (IoT) presents unique challenges for the deployment of machine learning (ML) models, particularly due to constraints on computational resources, the necessity for decentralized processing, and concerns regarding security and privacy in interconnected environments such as the Internet of cloud. In this paper, a novel decentralized ML framework is proposed for IoT environments characterized by wireless communication, dynamic data streams, and integration with cloud services. The framework integrates incremental learning algorithms with a robust decentralized model exchange protocol, ensuring that data privacy is preserved, while enabling IoT devices to participate in collaborative learning from distributed data across cloud networks. By incorporating a gossip-based communication protocol, the framework ensures energy-efficient, scalable, and secure model exchange, fostering effective knowledge sharing among devices, while addressing the potential security threats inherent in cloud-based IoT ecosystems. The framework’s performance was evaluated through simulations, demonstrating its ability to handle the complexities of real-time data processing in resource-constrained IoT environments, while also mitigating security and privacy risks within the Internet of cloud.

An Ultrawide Bandwidth Digital Backend System Based on PFB Algorithm for QTT

For the planning of the QiTai radio Telescope ultrawide bandwidth low-frequency pulsar receiving system, we designed and implemented a Field-Programmable Gate Array (FPGA)+CPU/GPU hybrid architecture digital backend system based on the Polyphase FilterBank (PFB) channeling algorithm. We used the FPGA signal acquisition and processing platform to implement ultrawide bandwidth signal sampling and designed the PFB algorithm to realize the digital channelization of multiple analog bandwidth signals. We also developed data encapsulation and multichannel parallel distribution firmware algorithms and realized the real-time parallel transmission of high-speed astronomical data streams based on the VLBI Data Interchange Format. We developed the Ultra Wide bandwidth Low-frequency pulsar data process PIPEline, which realized the real-time processing and data packaging of massive pulsar signals. Using the L-band (964–1732 MHz bandwidth) receiving system of the Nanshan 26 m radio telescope, we conducted a systematic test on the designed digital backend system and obtained high-quality observation data. By using the professional pulsar data processing software DSPSR to process the observation data, we obtained high signal-to-noise ratio pulse profiles.

Multi-task photonic reservoir computing: wavelength division multiplexing for parallel computing with a silicon microring resonator

Nowadays, as the ever-increasing demand for more powerful computing resources continues, alternative advanced computing paradigms are under extensive investigation. Significant effort has been made to deviate from conventional Von Neumann architectures. In-memory computing has emerged in the field of electronics as a possible solution to the infamous bottleneck between memory and computing processors, which reduces the effective throughput of data. In photonics, novel schemes attempt to collocate the computing processor and memory in a single device. Photonics offers the flexibility of multiplexing streams of data not only spatially and in time, but also in frequency or, equivalently, in wavelength, which makes it highly suitable for parallel computing. Here, we numerically show the use of time and wavelength division multiplexing (WDM) to solve four independent tasks at the same time in a single photonic chip, serving as a proof of concept for our proposal. The system is a time-delay reservoir computing (TDRC) based on a microring resonator (MRR). The addressed tasks cover different applications: Time-series prediction, waveform signal classification, wireless channel equalization, and radar signal prediction. The system is also tested for simultaneous computing of up to 10 instances of the same task, exhibiting excellent performance. The footprint of the system is reduced by using time-division multiplexing of the nodes that act as the neurons of the studied neural network scheme. WDM is used for the parallelization of wavelength channels, each addressing a single task. By adjusting the input power and frequency of each optical channel, we can achieve levels of performance for each of the tasks that are comparable to those quoted in state-of-the-art reports focusing on single-task operation. We also quantify the memory capacity and nonlinearity of each parallelized RC and relate these properties to the performance of each task. Finally, we provide insight into the impact of the feedback mechanism on the performance of the system.

Unified rules of renewable weighted sums for various online updating estimations

This paper establishes unified frameworks of renewable weighted sums (RWS) for various online updating estimations in the models with streaming data sets. The newly defined RWS lays the foundation of online updating likelihood, online updating loss function, online updating estimating equation and so on. The idea of RWS is intuitive and heuristic, and the algorithm is computationally simple. The RWS applies to various types of parametric, nonparametric and semiparametric estimators from likelihood, quasi-likelihood and least squares. The consistency and asymptotic normality of the proposed renewable estimator are established, and the oracle property is obtained. Moreover, these properties are always satisfied, without the familiar constraints on the number of data batches, which means that the new method is adaptive to the situation where streaming data sets arrive perpetually. The performance of the method is further illustrated by various numerical examples from simulation experiments and real data analysis.

Dynamics of real-time forecasting failure and recovery due to data gaps: A study using EnKF-based assimilation with the Lorenz model

Data assimilation-based real-time forecasting is widely used in meteorological and hydrological applications, where continuous data streams are employed to update forecasts and maintain accuracy. However, the reliability of the data source can be compromised due to sensor and communication failures or physical or cyber-attacks, and the impact of data stream failures on the accuracy of the forecasting system is not well understood. This study aims to systematically investigate the process of data stream failure and recovery for the first time. To achieve this, data gaps with varying lengths and timings are introduced to EnKF-based data assimilation system on the Lorenz model operating in both chaotic and periodic modes. Results show that the forecasting error grows exponentially in the chaotic mode but was limited in the periodic mode from the start of the data gap. For chaotic mode, the recovery of the system depends on the length of the data gap if the model error is not saturated; after saturation, the timing of the data stream recovery is important. Moreover, even long after restarting the data assimilation in the chaotic mode, the forecasting system cannot fully restore the original accuracy, while the periodic mode is generally resilient to disruption. This research introduces new metrics for quantifying system resilience and provides crucial insights into the long-term implications of data gaps, advancing our understanding of forecasting system behavior and reliability.

Machine learning-based intelligent localization technique for channel classification in massive MIMO

Multiple-input multiple-output (MIMO) technology has been widely adopted in wireless communications, which enables the simultaneous transmission of multiple data streams via multiple transmitting and receiving antennas. In a MIMO system with non-line-of-sight (NLOS), transmitted signals are reflected by various obstacles along the path, reaching the antenna at different angles and times. In 5G networks, the NLOS problem is a major challenge for massive MIMO localization, significantly reducing positioning accuracy. In this work, an intelligent localization technique based on NLOS identification and mitigation is proposed to address this problem. In this solution, a Convolutional Neural Network (CNN) based hybrid Archimedes-based Salp Swarm Algorithm (HASSA) technique is proposed to detect NLOS or the line of sight (LOS) and estimate the location. The accuracy can be analyzed by considering the angle of arrival of signals, threshold-based time of arrival, and time difference of arrival from different antennas. A novel reinforcement learning-based optimization approach is used for the mitigation of NLOS in the radio wave propagation path, which in turn reduces the computational complexity. We use the Ensemble Deep Deterministic Policy Gradient-Based Approach (EDDPG)-based Honey Badger algorithm (HBA) for the aforementioned process. The simulation of this approach assesses diverse scenarios and considers different parameters, and the approach is compared with various state-of-the-art works. From the simulation results, our proposed approach can be used for the identification and detection of LOS and NLOS components and can precisely enhance the localization compared with other approaches.

Rapid likelihood free inference of compact binary coalescences using accelerated hardware

Abstract We report a gravitational-wave parameter estimation algorithm, AMPLFI, based on likelihood-free inference using normalizing flows. The focus of AMPLFI is to perform real-time parameter estimation for candidates detected by machine-learning based compact binary coalescence search, Aframe. We present details of our algorithm and optimizations done related to data-loading and pre-processing on accelerated hardware. We train our model using binary black-hole (BBH) simulations on real LIGO-Virgo detector noise. Our model has ∼6 million trainable parameters with training times ≦ 24 hours. Based on online deployment on a mock data stream of LIGO-Virgo data, Aframe+AMPLFI is able to pick up BBH candidates and infer parameters for real-time alerts from data acquisition with a net latency of ∼6 seconds.

Near Real-Time Responsive Flood Event Representation: An Open-Source Interactive Web Application Architecture

Abstract. The adoption of accessible Digital Twin technology in flood applications has been impeded by a notable lack of practical examples. Arguably, the most distinguishing feature of Digital Twin technology is the integration of near real-time data analytics through IoT sensor connectivity, which has remained underutilised in flood applications and therefore indicates a critical research gap. This project endeavoured to develop a comprehensive and replicable open-source architecture framework, specifically tailored for near real-time responsive flood event representation. The resultant interactive web application integrated near real-time river height and rain fall data streams and performed on-demand data analytics. Key results include the establishment of a back-end spatial database, a coupled physical city and digital space model, and a functional holistic front-end user interface. Additionally, this research aligned with the Gemini Principles by emphasising data interoperability and maintaining an information feedback loop. The study also aimed to configure near real-time IoT sensor connections, implement event triggers, and deliver interactive visualisations in an easily accessible format. The research addressed key questions surrounding the effective integration of IoT sensor data, identification of crucial flood indicators and parameters, and rapid quantification of flood event impacts. Ultimately, this research project demonstrates how Digital Twin technology can swiftly provide decision-makers with crucial insights during flood disaster events.

Data Stream Protocols for Real-Time Messaging : Optimizing Performance and Reliability

This article explores the design and implementation of data stream protocols for real-time messaging, focusing on key aspects that contribute to their performance and reliability. It examines protocol architectures, data formatting and serialization techniques, error handling mechanisms, and quality of service features. The article compares popular protocols such as WebSocket and MQTT, discusses performance optimization techniques, and addresses security considerations. Emerging trends like WebTransport, HTTP/3, and edge computing integration are also explored, highlighting their potential to reshape the landscape of real-time messaging. By providing a comprehensive overview of current practices and future directions, this article aims to equip developers and system architects with the knowledge needed to design efficient and reliable real-time communication systems.

Data Integrity Problems in High-Volume High-Velocity Data Ingestion

In the era of bigdata, and never ending data push from IoT devices, the IT infrastructure are built to be scalable to handle the huge batch loads or continuous streaming live data. However, the big question is how can be establish the data integrity. How can we make sure no data is lost from Origin till the destination passing through numerous touch points enroute ? How can we ensure the quality with continuous inflow ? Should the inflow be suspended to perform the DQ checks? Or should it be a totally independent parallel activity. Let’s explore. Key words: Quality Data Management, Data Pipelines, AutoScalling, Large Scale Streaming Data, Performance, IoT, Data Cleansing

A Survey of Multimodal Perception Methods for Human–Robot Interaction in Social Environments

Human–robot interaction (HRI) in human social environments (HSEs) poses unique challenges for robot perception systems, which must combine asynchronous, heterogeneous data streams in real time. Multimodal perception systems are well-suited for HRI in HSEs and can provide more rich, robust interaction for robots operating among humans. In this article, we provide an overview of multimodal perception systems being used in HSEs, which is intended to be an introduction to the topic and summary of relevant trends, techniques, resources, challenges, and terminology. We surveyed 15 peer-reviewed robotics and HRI publications over the past 10+ years, providing details about the data acquisition, processing, and fusion techniques used in 65 multimodal perception systems across various HRI domains. Our survey provides information about hardware, software, datasets, and methods currently available for HRI perception research, as well as how these perception systems are being applied in HSEs. Based on the survey, we summarize trends, challenges, and limitations of multimodal human perception systems for robots, then identify resources for researchers and developers and propose future research areas to advance the field.

Online kernel sliced inverse regression

Online dimension reduction techniques are widely utilized for handling high-dimensional streaming data. Extensive research has been conducted on various methods, including Online Principal Component Analysis, Online Sliced Inverse Regression (OSIR), and Online Kernel Principal Component Analysis (OKPCA). However, it is important to note that the exploration of online supervised nonlinear dimension reduction techniques is still limited. This article presents a novel approach called Online Kernel Sliced Inverse Regression (OKSIR), which specifically tackles the challenge of dealing with the increasing dimension of the kernel matrix as the sample size grows. The proposed method incorporates two key components: the approximate linear dependence condition and dictionary variable sets. These components enable a reduced-order approach for online variable updates, improving the efficiency of the process. To solve the OKSIR problem, we formulate it as an online generalized eigen-decomposition problem and employ stochastic optimization techniques to update the dimension reduction directions. Theoretical properties of this online learner are established, providing a solid foundation for its application. Through extensive simulations and real data analysis, we demonstrate that the proposed OKSIR method achieves performance comparable to that of batch processing kernel sliced inverse regression. This research significantly contributes to the advancement of online dimension reduction techniques, enhancing their effectiveness in practical applications.

Modeling Enzyme Kinetics: Current Challenges and Future Perspectives for Biocatalysis.

Biocatalysis is becoming a data science. High-throughput experimentation generates a rapidly increasing stream of biocatalytic data, which is the raw material for mechanistic and novel data-driven modeling approaches for the predictive design of improved biocatalysts and novel bioprocesses. The holistic and molecular understanding of enzymatic reaction systems will enable us to identify and overcome kinetic bottlenecks and shift the thermodynamics of a reaction. The full characterization and modeling of reaction systems is a community effort; therefore, published methods and results should be findable, accessible, interoperable, and reusable (FAIR), which is achieved by developing standardized data exchange formats, by a complete and reproducible documentation of experimentation, by collaborative platforms for developing sustainable software and for analyzing data, and by repositories for publishing results together with raw data. The FAIRification of biocatalysis is a prerequisite to developing highly automated laboratory infrastructures that improve the reproducibility of scientific results and reduce the time and costs required to develop novel synthesis routes.

An Identical Code–Data Based Nonorthogonal Multiple Access

ABSTRACTNonorthogonal multiple access (NOMA) is a promising technique operating with limited resources. NOMA can be executed in both power and code domains. Cyclic interleaved NOMA strategies are debated in recent literature, combining power and code domains. Identical code cyclic shift (ICCS) transmission is viewed to be a bridge between code division access and NOMA. We propose identical code–data cyclic shift NOMA (ICDCS‐NOMA) in which the code and data are cyclically shifted in distinction to simple cyclic interleaved transmission introduced in our earlier work and the ICCS transmission. The cell‐edge user in a NOMA pair engages its signal transmission using ICDCS; in contrast, the cell‐center user prefers simple cyclic interleaved transmission. The spread data stream and its cyclic shifted version are blended in ICDCS, followed by cyclic interleaving. A distinct and extreme fading channel pair is assumed with the same number of paths for the edge and center users. Diversity combining multipath signals is embraced at the edge user terminal, and frequency domain equalization (FDE)/matched filter (MF) receiver is employed at the center user. The simulation indicates the dominance of the proposed ICDCS‐NOMA transmission in terms of the attained diversity gain over the conventional, ICCS, and simple cyclic interleaved transmissions.

Building Biodiversity Digital Twins

A digital twin is a replica of a system, process, or object realized as a digital version. Digital twins are widespread in engineering and aim to can help many kinds of organizations simulate real situations and their outcomes, ultimately allowing them to make better decisions. Digital twins combine cutting-edge modelling approaches with powers of supercomputers with data streams. These data resources, coming through the research infrastructures, depend on data standardization, efficient data models and workflows, and on FAIRness (Findability, Accessibility, Interoperability, Reusability) of these elements. Digital twin concepts have been applied to biodiversity data since 2022, most notably through the Biodiversity Digital Twin project. The outcomes of dynamic modeling based on the biodiversity data (which availability changes in volumes and quality) against the globally available changing environmental and human impact data are expected to form the basis for management of natural resources and aims to support biodiversity recovery. The BioDT consortium brings together a dynamic team of experts in biodiversity data and standards, high performance computing, and FAIR data to realize the first set of biodiversity Digital Twin prototypes. Biodiversity evidence generated, standardized and shared from collections, systematic and long term surveys, eDNA and metabaroding, citizen science and other communities representing a range of biodiversity evidence types, can to be applied in the dynamic and interactive modelling solutions, known as digital tiwins. The Biodiversity Digital Twin project (and the twin approach in general) offers advanced simulation and prediction models to tackle global biodiversity challenges. This project is designed to address biodiversity dynamics through a set of use cases, providing insights and tools for conservation and restoration efforts. The BioDT project has revealed similarities and differences of applying Biodiversity Information Standards (TDWG) thinking and principles of data standardization to the communities with different levels of familiarity with standards and FAIR principles. The presentation at SPNCH-TDWG 2024 covered the process of building biodiversity digital twins by the second project year and exposed the work along the gradient of data intensity, of data FAIRness, and of complience to the international data standards. The details of ten BioDT project stories are covered in the special issue of the Pensoft RIO journal on BioDT project (Afsar et al. 2024, Chala et al. 2024, Frøslev et al. 2024, Ingenloff et al. 2024, Groeneveld et al. 2024, Khan et al. 2024, Mikryukov et al. 2024, Ovaskainen et al. 2024, Rolph et al. 2024, Taubert et al. 2024).

Best practice guide for patellofemoral pain based on synthesis of a systematic review, the patient voice and expert clinical reasoning

ObjectiveDefine a best practice guide for managing people with patellofemoral pain (PFP).MethodsA mixed-methods convergent segregated synthesis of meta-analysed data with a thematic analysis of semistructured interviews and focus groups. Agreement...