Real-time Analysis Framework Research Articles

Building Flexible, Data-driven Framework for Real-time Analysis

In the contemporary business landscape, the escalating demand for real-time predictive analytics is driven by the imperative for dynamic decision-making. Traditional analytics models often prove inadequate in addressing the need for agility required to respond swiftly to rapidly changing circumstances. Real-time predictive analytics, however, offers a transformative solution, empowering organizations to make informed and timely decisions in fast-paced environments. This capability proves invaluable in industries were staying ahead of emerging trends is critical, fostering a proactive approach to decision-making that can significantly impact competitiveness. The sheer volume and diversity of data require sophisticated solutions for processing and analysis. Real-time predictive analytics becomes an indispensable tool, offering the capability to promptly extract valuable insights from massive datasets. This not only enhances decision-making but also allows organizations to stay ahead by uncovering trends and patterns in real time. Scalability is a fundamental consideration for organizations on a growth trajectory. Real-time predictive analytics frameworks provide a scalable foundation, allowing businesses to seamlessly expand their analytical capabilities. This adaptability ensures that the framework can handle the increasing demands for processing power and storage, aligning with the evolving needs of a growing organization.

An IoT-Fog-Cloud Integrated Framework for Real-Time Remote Cardiovascular Disease Diagnosis

Recently, it has proven difficult to make an immediate remote diagnosis of any coronary illness, including heart disease, diabetes, etc. The drawbacks of cloud computing infrastructures, such as excessive latency, bandwidth, energy consumption, security, and privacy concerns, have lately been addressed by Fog computing with IoT applications. In this study, an IoT-Fog-Cloud integrated system, called a Fog-empowered framework for real-time analysis in heart patients using ENsemble Deep learning (FRIEND), has been introduced that can instantaneously facilitate remote diagnosis of heart patients. The proposed system was trained on the combined dataset of Long-Beach, Cleveland, Switzerland, and Hungarian heart disease datasets. We first tested the model with eight basic ML approaches, including the decision tree, logistic regression, random forest, naive Bayes, k-nearest neighbors, support vector machine, AdaBoost, and XGBoost approaches, and then applied ensemble methods including bagging classifiers, weighted averaging, and soft and hard voting to achieve enhanced outcomes and a deep neural network, a deep learning approach, with the ensemble methods. These models were validated using 16 performance and 9 network parameters to justify this work. The accuracy, PPV, TPR, TNR, and F1 scores of the experiments reached 94.27%, 97.59%, 96.09%, 75.44%, and 96.83%, respectively, which were comparatively higher when the deep neural network was assembled with bagging and hard-voting classifiers. The user-friendliness and the inclusion of Fog computing principles, instantaneous remote cardiac patient diagnosis, low latency, and low energy consumption, etc., are advantages confirmed according to the achieved experimental results.

Development of a deep learning based automated data analysis for step-filter x-ray spectrometers in support of high-repetition rate short-pulse laser-driven acceleration experiments.

We present a deep learning based framework for real-time analysis of a differential filter based x-ray spectrometer that is common on short-pulse laser experiments. The analysis framework was trained with a large repository of synthetic data to retrieve key experimental metrics, such as slope temperature. With traditional analysis methods, these quantities would have to be extracted from data using a time-intensive and manual analysis. This framework was developed for a specific diagnostic, but may be applicable to a wide variety of diagnostics common to laser experiments and thus will be especially crucial to the development of high-repetition rate (HRR) diagnostics for HRR laser systems that are coming online.

Deep-Framework: A Distributed, Scalable, and Edge-Oriented Framework for Real-Time Analysis of Video Streams.

Edge computing is the best approach for meeting the exponential demand and the real-time requirements of many video analytics applications. Since most of the recent advances regarding the extraction of information from images and video rely on computation heavy deep learning algorithms, there is a growing need for solutions that allow the deployment and use of new models on scalable and flexible edge architectures. In this work, we present Deep-Framework, a novel open source framework for developing edge-oriented real-time video analytics applications based on deep learning. Deep-Framework has a scalable multi-stream architecture based on Docker and abstracts away from the user the complexity of cluster configuration, orchestration of services, and GPU resources allocation. It provides Python interfaces for integrating deep learning models developed with the most popular frameworks and also provides high-level APIs based on standard HTTP and WebRTC interfaces for consuming the extracted video data on clients running on browsers or any other web-based platform.

A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring.

In recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical ‘batch’ processing—extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework.

A general framework for real-time analysis of massive multimedia streams

Big Data platforms provide opportunities for the management and analysis of large quantities of information, but the services they provide are often too raw, since they focus on issues of fault-tolerance, increased parallelism, and so on. An additional software layer is, therefore, needed to effectively use such architectures for advanced applications in several important real-world domains, such as scientific and health care sensors, user-generated data, supply chain systems and financial companies, to name a few. In this paper, we present RAM $$^3$$ S, a framework for the real-time analysis of massive multimedia streams, where data come from multiple data sources (such as sensors and cameras) that are widely located on the territory, with the final goal to discovery new and hidden information from the output of data sources as they occur, thus with very limited latency. We apply RAM $$^3$$ S to the use case of automatic detection of “suspect” people from several concurrent video streams, and instantiate it on top of three different open source engines for the analysis of streaming Big Data (i.e., Apache Spark, Apache Storm, and Apache Flink). The effectiveness and scalability of RAM $$^3$$ S instantiation is experimentally evaluated on real data, also comparing the performance of the three considered Big Data platforms. Such comparison is performed both on a cluster of physical machines in our datalab and on the Google Cloud Platform.

2254

OBJECTIVES/SPECIFIC AIMS: The iobio project enables anyone (eg, diagnosticians, MDs, genetic counselors, citizen scientists) to perform useful analysis of genomic data, without a need to rely on bioinformaticians. iobio uses a novel real-time analysis framework, coupled with powerful visualizations delivered in a standard web browser. The project successfully supports free academic/nonprofit users, but occasions exist where it is more applicable for the project to be delivered commercially. Frameshift Genomics is developing commercial applications and functionality, which will exist alongside and in coordination with the academic project. These products will be marketed to large institutions including genome institutes, hospitals, diagnostic labs etc., but also to individual users who do not have access to large compute resources, or bioinformatic analysts, and everything in between. METHODS/STUDY POPULATION: The commercial iobio project under Frameshift Genomics aims to develop applications and features that cannot be successfully supported by an academic model. For example, when analyses are scaled up to processing of extremely large data sets, a commercial product with access to compute resources makes more sense than an academic tool. Bam.iobio is an application that samples data from sequencing alignment files, taking seconds to generate and visualize statistics representative of the entire file. This app is offered for free academically. When analysis involves thousands of such files, however, the commercial application, multibam.iobio, is more suitable. Other examples, including support for licensed third-party software and permitting extensive computation via cloud platforms, can also only be reasonably be supported via commercial software. Finally, development of commercial applications is driving adoption of more rigorous testing platforms, delivering more robust products. A particular strength of the iobio platform is allowing non-bioinformaticians to understand their data, for example providing quality control functionality providing confidence in data sets and the conclusions drawn from them. Such analyses are critical to all users of genomic data, and the iobio platform is ideally suited to provide an intuitive, integrated framework for performing them. RESULTS/ANTICIPATED RESULTS: The iobio project has been readily adopted by many in the community and shows significant promise for democratizing genomic analysis. Work is ongoing, supported by NIH small business grants, to develop commercial applications that will be marketed to analysts and medical professionals from large genome institutes and universities, to individual project users and citizen scientists. DISCUSSION/SIGNIFICANCE OF IMPACT: There are currently a number of iobio tools available academically, and they have been embraced by many in the genomics community. In fact, a number of popular platforms (eg, Galaxy, the International Cancer Genome Consortium (ICGC) data portal, mygene2 at the University of Washington) have incorporated iobio tools into their own platforms. To date, the gene.iobio variant interrogation tool has been used in a number of diagnostic projects, aiding identification of putative causative variants, and the pre-release version of the commercial multibam.iobio tool has been critical in unearthing data quality problems in project level data.

The IceCube realtime alert system

Although high-energy astrophysical neutrinos were discovered in 2013, their origin is still unknown. Aiming for the identification of an electromagnetic counterpart of a rapidly fading source, we have implemented a realtime analysis framework for the IceCube neutrino observatory. Several analyses selecting neutrinos of astrophysical origin are now operating in realtime at the detector site in Antarctica and are producing alerts for the community to enable rapid follow-up observations. The goal of these observations is to locate the astrophysical objects responsible for these neutrino signals. This paper highlights the infrastructure in place both at the South Pole site and at IceCube facilities in the north that have enabled this fast follow-up program to be implemented. Additionally, this paper presents the first realtime analyses to be activated within this framework, highlights their sensitivities to astrophysical neutrinos and background event rates, and presents an outlook for future discoveries.

High-performance meteorological data processing framework for real-time analysis and visualization

The current level of geoscience advancements enables analysis of the Earth as a highly complex system with many components, like hydrosphere, lithosphere, and atmosphere, which interact within and with others. Large amounts of observational data and simulation data generated by numerical models need to be analyzed in order to extract valuable knowledge from them. Visualization is a very important part of data analysis since it provides an easy and fast way to assess the data and understand, which parts of it are of particular interest and whether there are any errors in the dataset. The most efficient type of tools for meteorological data visualization are geographic information systems (GIS) because they provide great functionality for manipulating geospatial data. In this paper, we describe the processing framework, which can be efficiently used as a backend for GIS by providing rapid access to the data located at the remote storage nodes. Experimental results demonstrate that developed framework allows real-time data access and can successfully handle a large number of simultaneous requests.

Partitioned k-Exclusion Real-Time Locking Protocol Motivated by Multicore Multi-GPU Systems

Graphic processing units (GPUs) have been widely recognized as cost-efficient co-processors with acceptable size, weight, and power consumption. However, adopting GPUs in real-time systems is still challenging, due to the lack in framework for real-time analysis. In order to guarantee real-time requirements while maintaining system utilization in modern heterogeneous systems, such as multicore multi-GPU systems, a novel suspension-based k-exclusion real-time locking protocol and the associated suspension-aware schedulability analysis are proposed. The proposed protocol provides a synchronization framework that enables multiple GPUs to be efficiently integrated in multicore real-time systems. Comparative evaluations show that the proposed methods improve upon the existing work in terms of schedulability.

A Machine Intelligence Approach to Virtual Ballet Training

This article presents a framework for real-time analysis and visualization of ballet dance movements performed within a Cave Virtual Reality Environment (CAVE). A Kinect sensor captures and extracts dance-based movement features, from which a topology preserved "posture space" is constructed using a spherical self-organizing map (SSOM). Recordings of dance movements are parsed into gestural elements by projection onto the SSOM to form unique trajectories in posture space. Dependencies between postures in a trajectory are modeled using a Markovian empirical transition matrix, which is then used to recognize attempted movements. This allows for quantitative assessment and feedback of a student's performance, delivered using concurrent, localized visualizations together with a performance score based on incremental dynamic time warping (IDTW).

Analysing AFDX Networks Using End-to-end Response Time Analysis

In this paper, we present a novel real-time analysis framework for AFDX (Avionics Full Duplex Switched Ethernet) networks. The framework, based on end-to-end response time analysis, calculates not only delay bounds, but also the maximum jitter for each VL (Virtual Link) at each hop, which is necessary according to the AFDX standard. Moreover, the framework supports multicasting, i.e., VLs with several paths, and VLs with arbitrary delay bounds, i.e., shorter, longer, or equal to their periods. An analysis method to calculate the worst-case buffer population is included in the framework, as it is important to guarantee that no buffer-overflow occurs. With a performance surpassing that of Network Calculus and comparable with Trajectory Approach, our framework presents a good choice due to its many features and its foundation in well-accepted analysis methods.

Real-time imaging of human brain function by near-infrared spectroscopy using an adaptive general linear model

Near-infrared spectroscopy is a non-invasive neuroimaging method which uses light to measure changes in cerebral blood oxygenation associated with brain activity. In this work, we demonstrate the ability to record and analyze images of brain activity in real-time using a 16-channel continuous wave optical NIRS system. We propose a novel real-time analysis framework using an adaptive Kalman filter and a state–space model based on a canonical general linear model of brain activity. We show that our adaptive model has the ability to estimate single-trial brain activity events as we apply this method to track and classify experimental data acquired during an alternating bilateral self-paced finger tapping task.