Near Real-time System Research Articles

Leveraging past information and machine learning to accelerate land disturbance monitoring

Near real-time (NRT) monitoring of land disturbances holds great importance for delivering emergency aid, mitigating negative social and ecological impacts, and distributing resources for disaster recovery. Many past NRT techniques were built upon examining the overall change magnitude of a spectral anomaly with a predefined threshold, namely the unsupervised approach. However, their lack of fully considering spectral change direction, change date, and pre-disturbance conditions often led to low detection sensitivity and high commission errors, especially when only a few satellite observations were available at the early disturbance stage, eventually resulting in a longer lag to produce a reliable disturbance map. For this study, we developed a novel supervised machine learning approach guided by historical disturbance datasets to accelerate land disturbance monitoring. This new approach consisted of two phases. For the first phase, the supervised approach applied retrospective analysis on historical Harmonized Landsat Sentinel-2 (HLS) datasets from 2015 to 2021, combined with several open disturbance products. The disturbance model was constructed for each condition of consecutive anomaly number, with the aim of enhancing the specificity for delineating early-stage disturbance regions. Then, these stage-based models were applied for an NRT scenario to predict disturbance probabilities with 2022 HLS images incrementally on a weekly basis. To demonstrate the capability of this new approach, we developed an operational NRT system incorporating both the unsupervised and supervised approach. Latency and accuracy were evaluated against 3000 samples that were randomly selected from the five most influential disturbance events of the United States in 2022, based on labels and disturbance dates interpreted from daily PlanetScope images. The evaluation showed that the supervised approach required 15 days (since the start of the disturbance event) to reach the plateau of its F1 curve (where most disturbance pixels are detected with high confidence), seven days earlier with roughly 0.2 F1 score improvement compared to the unsupervised approach (0.733 vs. 0.546 F1 score). Further analysis showed the improvement was mainly due to the substantial decrease in commission errors (17.7% vs 44.4%). The latency component analysis illustrated that the supervised approach only took an average of 4.1 days to yield the first disturbance alert at its fastest daily updating speed, owing to its decreased sensitivity lag. This finding highlighted the importance of using past knowledge and machine learning to reduce detection delays for an NRT monitoring task.

State estimation for large power distribution systems

This article presents the modifications and improvements that have been included over the classic formulation of the state estimation to be able to run in real networks with large power distribution systems. Improvements include a new penalty term that keeps bus consumption or power generation within an allowed band and the inclusion of dynamic elements such as voltage regulators and capacitor batteries. A complete algorithm is presented that is capable of operating on large power systems in near real time. This algorithm and the results have been tested with numerous cases of real distribution networks covering large geographical areas, obtaining very good results and demonstrating the enormous usefulness of this approach.

Towards Deep-Learning-Driven Intrusion Detection for the Internet of Things

Cyber-attacks on the Internet of Things (IoT) are growing at an alarming rate as devices, applications, and communication networks are becoming increasingly connected and integrated. When attacks on IoT networks go undetected for longer periods, it affects availability of critical systems for end users, increases the number of data breaches and identity theft, drives up the costs and impacts the revenue. It is imperative to detect attacks on IoT systems in near real time to provide effective security and defense. In this paper, we develop an intelligent intrusion-detection system tailored to the IoT environment. Specifically, we use a deep-learning algorithm to detect malicious traffic in IoT networks. The detection solution provides security as a service and facilitates interoperability between various network communication protocols used in IoT. We evaluate our proposed detection framework using both real-network traces for providing a proof of concept, and using simulation for providing evidence of its scalability. Our experimental results confirm that the proposed intrusion-detection system can detect real-world intrusions effectively.

CancerLinQ: Origins, Implementation, and Future Directions.

Rapid-learning health systems have been proposed as a potential solution to the problem of quality in medicine, by leveraging data generated from electronic health systems in near-real time to improve quality and reduce cost. Given the complex, dynamic nature of cancer care, a rapid-learning health system offers large potential benefits to oncology practice. In this article, we review the rationale for developing a rapid-learning health system for oncology and describe the sequence of events that led to the development of ASCO's CancerLinQ (Cancer Learning Intelligence Network for Quality) initiative, as well as the current state of CancerLinQ, including its importance to efforts such as the Beau Biden Cancer Moonshot. We then review the considerable challenges facing optimal implementation of a rapid-learning health system such as CancerLinQ, including integration of rapidly expanding multiomic data, capturing big data from a variety of sources, an evolving competitive landscape, and implementing a rapid-learning health system in a way that satisfies many stakeholders, including patients, providers, researchers, and administrators.

Near-real-time non-obstructed flood inundation mapping using synthetic aperture radar

In the event of a flood disaster, first response agencies need inundation maps produced in near real time (NRT). Such maps can be generated using satellite-based information. In this study, we developed mapping techniques that rely on synthetic aperture radar (SAR) on-board earth-orbiting platforms. SAR provides valid ground surface measurements through cloud cover with high resolution and sampling frequency that has recently increased through multiple missions. Despite numerous efforts, automatic processing of SAR data to derive accurate inundation maps still poses challenges. To address them, we have developed an NRT system named RAdar-Produced Inundation Diary (RAPID). RAPID integrates four processing steps: classification based on statistics, morphological processing, multi-threshold-based compensation, and machine-learning correction. Besides SAR data, the system integrates multisource remote-sensing data products, including land cover classification, water occurrence, hydrographical, water type, and river width products. In comparison to expert handmade flood maps, the fully-automated RAPID system exhibited “overall,” “producer,” and “user” accuracies of 93%, 77%, and 75%, respectively. RAPID accommodates commonly encountered over- and under-detections caused by noise-like speckle, water-like radar response areas, strong scatterers, and isolated inundation areas—errors that are in common practice to ignore, mask out, or be filtered out by coarsening the effective resolution. RAPID can serve as the kernel algorithm to derive flood inundation products from satellites—both existing and to be launched—equipped with high-resolution SAR sensors, including Envisat, Radarsat, NISAR, Advanced Land Observation Satellite (ALOS)-1/2, Sentinel-1, and TerraSAR-X.

An Energy-Efficient Task Scheduling for Near-Realtime Systems with Execution Time Variation

An Energy-Efficient Task Scheduling for Near-Realtime Systems with Execution Time Variation

Leveraging a Statewide Clinical Data Warehouse to Expand Boundaries of the Learning Health System.

Learning Health Systems (LHS) require accessible, usable health data and a culture of collaboration—a challenge for any single system, let alone disparate organizations, with macro- and micro-systems. Recently, the National Science Foundation described this important setting as a cyber-social ecosystem. In 2004, in an effort to create a platform for transforming health in South Carolina, Health Sciences South Carolina (HSSC) was established as a research collaboration of the largest health systems, academic medical centers and research intensive universities in South Carolina. With work beginning in 2010, HSSC unveiled an integrated Clinical Data Warehouse (CDW) in 2013 as a crucial anchor to a statewide LHS. This CDW integrates data from independent health systems in near-real time, and harmonizes the data for aggregation and use in research. With records from over 2.7 million unique patients spanning 9 years, this multi-institutional statewide clinical research repository allows integrated individualized patient-level data to be used for multiple population health and biomedical research purposes. In the first 21 months of operation, more than 2,800 de-identified queries occurred through i2b2, with 116 users. HSSC has developed and implemented solutions to complex issues emphasizing anti-competitiveness and participatory governance, and serves as a recognized model to organizations working to improve healthcare quality by extending the traditional borders of learning health systems.

Insights into the possible seismic damage of residential buildings in Bucharest, Romania, at neighborhood resolution

Seismic loss estimation analysis for large cities is a very demanding yet necessary task; the modelling of such complex systems requires first of all insightful input data at good resolution, referring to local effects, buildings and socio-economic aspects. Also, the implementation of less empirical estimation methods is needed. Until recently, these requirements could not be fulfilled for Bucharest, the capital city in the European Union which is most at risk due to earthquakes. Based on 2011 and 2002 census data, standardized according to the framework of the Near-real time System for Estimating the Seismic Damage in Romania (SeisDaRo) through a unique approach, and on relevant hazard scenarios, we estimate for the first time the building damage at census tract scale. The methodology applied relies on 48 vulnerability curves for buildings, on the Improved Displacement Coefficient Analytical Method included in the SELENA software for computing damage probabilities and on deterministic seismic hazard scenarios, including the maximum possible earthquake. By using overlay analysis with satellite imagery and a new methodology integrated in GIS we show how results can be enhanced, reflecting more localized characteristics. Best practices for seismic risk mapping are also expressed. The results are promising and contribute to mitigation efforts in Bucharest.

Adoption of Geospatial Systems towards evolving Sustainable Himalayan Mountain Development

Abstract. Natural resources dependence of mountain communities, rapid social and developmental changes, disaster proneness and climate change are conceived as the critical factors regulating sustainable Himalayan mountain development. The Himalayan region posed by typical geographic settings, diverse physical and cultural diversity present a formidable challenge to collect and manage data, information and understands varied socio-ecological settings. Recent advances in earth observation, near real-time data, in-situ measurements and in combination of information and communication technology have transformed the way we collect, process, and generate information and how we use such information for societal benefits. Glacier dynamics, land cover changes, disaster risk reduction systems, food security and ecosystem conservation are a few thematic areas where geospatial information and knowledge have significantly contributed to informed decision making systems over the region. The emergence and adoption of near-real time systems, unmanned aerial vehicles (UAV), board-scale citizen science (crowd-sourcing), mobile services and mapping, and cloud computing have paved the way towards developing automated environmental monitoring systems, enhanced scientific understanding of geophysical and biophysical processes, coupled management of socio-ecological systems and community based adaptation models tailored to mountain specific environment. There are differentiated capacities among the ICIMOD regional member countries with regard to utilization of earth observation and geospatial technologies. The region can greatly benefit from a coordinated and collaborative approach to capture the opportunities offered by earth observation and geospatial technologies. The regional level data sharing, knowledge exchange, and Himalayan GEO supporting geospatial platforms, spatial data infrastructure, unique region specific satellite systems to address trans-boundary challenges would go a long way in evolving sustainable Himalayan livelihoods.

Parallel Programming Techniques Applied to Water Pump Scheduling Problems

Most of the energy consumed by a water company is used to operate pumping systems. Identifying the optimal schedule for such systems in near real time will drastically reduce energy costs. The pump scheduling problem comprises three main elements: the pumping system, the tank, and the water demand to be satisfied. In this paper, a mathematical programming model and techniques used to solve this problem are presented. This study analyzed a parallel programming paradigm to solve this problem by introducing stochastic programming techniques (scenario tree evaluation) and multisite problems. Numerical experiments were designed and completed on parallel computers combining classical mathematical programming techniques and parallel tools. As a result, the parallel programming strategy was experimentally proven to be a useful technique for near-real-time pump scheduling applications.

Assessment of Hydrological Database for Automated Flood-alert System—Case Study: Shonai River, Kasugai

This study uses database infrastructure, hydrologic models, and geospatial analysis to assess flood inundation areas and their distribution in the Shonai sub-basin, Kasugai City, in western Japan. The proposed method utilizes near-real time data; hence, efficient database processing is an important requirement for the assessment tool. The river hydrological models utilize geometric input flow characteristics and river discharge volume to generate an inundation map. Discharge volume is estimated owing to the lack of data on discharge measurements. Distributions are presented in maps of inundation-susceptibility using an interactive web-based system. The simulated results from the hydrological model indicate that the topographic characteristics (slope and elevation) of the watershed significantly affect the depth of inundation. Hydrological factors (e.g., rainfall and water level) also significantly affect changes in the rate of river discharge. The centralized geo-database can be processed on-the-fly, and it is believed that this approach is reliable and well suited to such a flood alert system. The significant points of this work are to determine a necessary but unknown parameter (discharge volume) via existing data resources, and—ultimately—to make informative mapping data available in a timely manner. The use of a near-real time system ensures that the public and relevant agencies can assess emergency information and initiate appropriate and prompt responses. 

Fast estimation and analysis of the inter-frequency clock bias for Block IIF satellites

The inter-frequency bias of PRN25 was noticed by the scientific community and considered to be caused by thermal variations. The inter-frequency bias leads to an apparent inter-frequency clock bias (IFCB), which could be obtained using the difference of two ionosphere-free phase combinations (L1/L2 and L1/L5). We present an efficient approach derived from the epoch-differenced strategy for fast estimation of IFCBs for Block IIF satellites. For the analysis, data from 32 stations from the IGS network spanning 10 months (DOY 213, 2011---153, 2012) are processed. The processing times show that the epoch-differenced method is more efficient than the undifferenced one. In order to study the features of IFCB, a harmonic analysis is performed by using a FFT (fast Fourier transformation), and significant periodic variations with the periods of 12, 6 and 8 h are noticed. The fourth-order period is determined by comparing the performances of the model with different periods. After determination, a harmonics-based function of order 4 is used to model the IFCB, and the single-day amplitudes and phases are estimated for the 10 months from a least squares fit. Based on the estimated results, the characterization of IFCB is discussed. The algorithm is incorporated into the MGPSS software developed at SHAO (Shanghai Astronomical Observatory, Chinese Academy of Sciences) and used to monitor the IFCB variations of GPS and COMPASS systems in near real time.

Workflow management in large distributed systems

The MonALISA (Monitoring Agents using a Large Integrated Services Architecture) framework provides a distributed service system capable of controlling and optimizing large-scale, data-intensive applications. An essential part of managing large-scale, distributed data-processing facilities is a monitoring system for computing facilities, storage, networks, and the very large number of applications running on these systems in near realtime. All this monitoring information gathered for all the subsystems is essential for developing the required higher-level services—the components that provide decision support and some degree of automated decisions—and for maintaining and optimizing workflow in large-scale distributed systems. These management and global optimization functions are performed by higher-level agent-based services. We present several applications of MonALISA's higher-level services including optimized dynamic routing, control, data-transfer scheduling, distributed job scheduling, dynamic allocation of storage resource to running jobs and automated management of remote services among a large set of grid facilities.

Influence of rainfall estimation error and spatial variability on sewer flow prediction at a small urban scale

Legislative drivers for water quality and urban flood risk are driving a growing need to accurately determine the performance of urban drainage systems in near real time. Rainfall data are clearly a key input to urban drainage system models. Historically rain gauge data have been used, however radar rainfall data are now widely available and benefits from significantly higher spatial coverage than rain gauges in most UK urban catchments. This paper describes a detailed study based on a small (11km2) urban catchment in West Yorkshire, England. Radar and rain gauge data have been compared and used as the input to hydrodynamic sewer flow simulations, and the results of these simulations have been compared with measured flows in the sewer system. The results showed that for this size of catchment, there can be significant differences in simulated peak flows and combined sewer overflow spill volumes due to inherent uncertainties between the two rainfall estimates.

Informatics in Radiology

Radiology departments are a rich source of information in the form of digital radiology reports and images obtained in patients with a wide spectrum of clinical conditions. A free text radiology report and image search application known as Render was created to allow users to find pertinent cases for a variety of purposes. Render is a radiology report and image repository that pools researchable information derived from multiple systems in near real time with use of (a) Health Level 7 links for radiology information system data, (b) periodic file transfers from the picture archiving and communication system, and (c) the results of natural language processing (NLP) analysis. Users can perform more structured and detailed searches with this application by combining different imaging and patient characteristics such as examination number; patient age, gender, and medical record number; and imaging modality. Use of NLP analysis allows a more effective search for reports with positive findings, resulting in the retrieval of more cases and terms having greater relevance. From the retrieved results, users can save images, bookmark examinations, and navigate to an external search engine such as Google. Render has applications in the fields of radiology education, research, and clinical decision support.

Near Real-Time Monitoring of Bead Area Damage in Rolling Tires Using a Rotating Wheel Model and Multi-Directional Vibration Data

In current tire durability tests, technicians must stop tests periodically to inspect each tire, resulting in a time consuming, expensive, and relatively subjective process. Online tire monitoring would thus be a dramatic improvement over the current methodology. If such a system could be extended to on-line vehicle use, it could dramatically increase safety, reduce downtime, and lead to better fuel efficiency in commercial and passenger cars and trucks. A near real-time system to monitor the initiation of bead area damage in rolling tires was developed using vibration data collected from the wheel end spindle. To generate an accurate model of the forced response of the wheel end spindle, a tire on a fixed spindle was impacted with a modal punch while the tire was preloaded against a plate. The frequency response functions acquired from this system were then used to develop a rotating tire model consisting of time-delayed forcing functions applied to the tire patch at the rolling frequency; this model was used to assist in interpreting the results from the near real-time monitoring system. The near real-time continuous monitoring system has been deployed in a manufacturing test environment and utilizes frequency data from three mutually orthogonal acceleration measurements. These data are combined to create a highly sensitive composite index that identifies when the initiation of bead area damage has occurred.

Computer-assisted three-dimensional reconstruction and motion analysis of living, crawling cells

A computer-assisted three-dimensional dynamic image analysis system (3D-DIAS) has been developed for reconstructing and motion analyzing living, crawling cells. The system simultaneously reconstructs the cell surface, the nucleus and pseudopodia, both expanding and retracting. Although this system has been developed for single cell analysis, it can be used for the dynamic reconstruction and motion analysis of cells in early embryos, the human heart and any other cell, organ or object changing shape over time. Ongoing development of a dynamic analysis system with a confocal front-end, a high speed reconstruction system, a near-real time system and a virtual reality system are described.

The IDA Near Real Time System

Introduction Since 1992, IRIS/IDA stations of the Global Seismographic Network have been configured as nodes on the Internet for the purpose of near realtime data access. The IDA Station Processor (ISP), a UNIX workstation connected to the IDA Data Acquisition System (IDAS), provides rapid data access to local and remote users. Additionally, it is used to monitor station state-of-health, and to unpack and copy data tapes for the station operators. The ISP may be installed at the same location as the IDAS or, should conditions require, at a different location connected via a dedicated telecommunication circuit. The software for real-time data collection and transfer, called the IDA Near Real Time System (NRTS), is designed to provide both continuous and occasional data telemetry from IRIS/IDA Stations to remote (from the station) users with the following goals in mind: 1. To support real-time seismology 2. To...