Real-time Situational Awareness Research Articles

Real-Time Perception Enhancement in Obscured Environments for Underground Mine Search and Rescue Teams

First responders in underground mines face a myriad of challenges when searching for personnel in a disaster scenario. Possibly, the most acute challenge is the complete lack of visibility owing to a combination of dust, smoke, and pitch-black conditions. Moreover, the complex environment compounds the difficulty of navigating and searching the area as well as identifying hazardous conditions until in close proximity. Enhanced perception and localization technologies that enable rapid and safe disaster response could mitigate the mine rescue team’s risk and reduce response times. We developed a proof of concept perception enhancement tool for mine rescue personnel in pitch-black conditions by leveraging LiDAR, thermal camera, and data fusion to reconstruct a 3D representation of the space in real-time. This enhancement is visualized on HoloLens, allowing the responders real-time situational awareness of personnel, walls, obstacles, or fires in otherwise opaque environments. The technology is a first step towards faster, safer, and more effective disaster response for mine rescue operations, including detection of unexpected hazards before they become imminent threats.

A novel unmanned aerial vehicle driven real-time situation awareness for fire accidents in chemical tank farms

A large number of flammable hazardous materials are stored in chemical tank farms, where fire-induced domino accidents can be easily triggered. In this study, a novel real-time fire situation awareness (FSA) approach based on UAV is proposed to capture spatio-temporal evolution characteristics and predict development trends of fire accidents. Firstly, fire images are acquired by UAV, and the key parameters of fire are extracted in real time based on YOLOv8 network. Then, the thermal radiation and impact on surrounding equipment are predicted by combining LSTM network, solid flame model and improved probit model. The proposed method is verified by small-scale tank fire experiments, which demonstrate its superiority in terms of physical consistency and prediction accuracy. The results show that the mean absolute percentage error (MAPE) of fire parameter extraction is not higher than 5.43%, the MAPE of thermal radiation prediction is not higher than 25%, and the dynamic time to failure (dttf) for the model tank at different location is predicted. This work has the potential to provide a novel solution for real-time assessment of fire size and trend prediction to support firefighting, emergency rescue and decision making in fire accident scenarios.

Efficient multi-branch segmentation network for situation awareness in autonomous navigation

Real-time and high-precision situational awareness technology is critical for autonomous navigation of unmanned surface vehicles (USVs). In particular, robust and fast obstacle semantic segmentation methods are essential. However, distinguishing between the sea and the sky is challenging due to the differences between port and maritime environments. In this study, we built a dataset that captured perspectives from USVs and unmanned aerial vehicles in a maritime port environment and analysed the data features. Statistical analysis revealed a high correlation between the distribution of the sea and sky and row positional information. Based on this finding, a three-branch semantic segmentation network with a row position encoding module (RPEM) was proposed to improve the prediction accuracy between the sea and the sky. The proposed RPEM highlights the effect of row coordinates on feature extraction. Compared to the baseline, the three-branch network with RPEM significantly improved the ability to distinguish between the sea and the sky without significantly reducing the computational speed.

A secure and efficient UAV network defense strategy: Convergence of blockchain and deep learning

Unmanned Aerial Vehicles (UAVs) are highly versatile and efficient tools utilized across diverse industries for data collection. However, they are vulnerable to wireless communication and data exchange risks, including unauthorized access, data theft, and network attacks. To address these problems, we introduce a secure and reliable UAV network service architecture that incorporates blockchain and deep learning to provide more secure and efficient network services for UAVs. We propose a UAV cluster identity management module by combining blockchain, encryption algorithms, and digital signatures to enhance the security of UAV communication data transmission. Then, based on machine learning, deep learning, and malicious process detection technology, we propose a real-time secure situational awareness system for UAV cluster terminal devices to enhance the security of the operating environment for UAVs. Finally, we propose a data-trustworthy interconnection platform based on blockchain, smart contracts, and consensus algorithms to realize secure and efficient sharing and transmission of terminal data. The results of the experiments demonstrate the feasibility and effectiveness of our UAV network service architecture.

Techniques for Enhancing Security in Industrial Control Systems

Increasingly Industrial Control Systems (ICS) systems are being connected to the Internet to minimise the operational costs and provide additional flexibility. These control systems such as the ones used in power grids, manufacturing and utilities operate continually and have long lifespans measured in decades rather than years as in the case of Information Technology (IT) systems. Such industrial control systems require uninterrupted and safe operation. However, they can be vulnerable to a variety of attacks, as successful attacks on critical control infrastructures could have devastating consequences to the safety of human lives as well as a nation’s security and prosperity. Furthermore, there can be a range of attacks that can target ICS and it is not easy to secure these systems against all known attacks let alone unknown ones. In this paper, we propose a software enabled security architecture using Software Defined Networking (SDN) and Network Function Virtualisation (NFV) that can enhance the capability to secure industrial control systems. We have designed such an SDN/NFV enabled security architecture and developed a Control System Security Application (CSSA) in SDN Controller for enhancing security in ICS by achieving real time situational awareness and dynamic policy-driven decision making across the network infrastructure. In particular, CSSA can be used for establishing secure path for end-to-end communication between devices and also deal against certain specific attacks namely denial of service attacks, from unpatched vulnerable control system components and securing the communication flows from the legacy devices that do not support any security functionality. We also discuss how CSSA provides reliable paths for safety critical messages in control systems. We discuss the prototype implementation of the proposed architecture and the results obtained from our analysis.

Optimizing Timely Coverage in Communication Constrained Collaborative Sensing Systems

We consider a collection of distributed sensor nodes periodically exchanging information to achieve real-time situational awareness in a communication constrained setting, e.g., collaborative sensing amongst vehicles to improve safety-critical decisions. Nodes may be both consumers and producers of sensed information. Consumers express interest in information about particular locations, e.g., obstructed regions and/or road intersections, whilst producers broadcast updates on what they are currently able to see. Accordingly, we introduce and explore optimizing trade-offs between the coverage and the space-time interest weighted average “age” of the information available to consumers. We consider two settings that capture the fundamental character of the problem. The first addresses selecting a subset of producers that maximizes the coverage of the consumers preferred regions and minimizes the average age of these regions given that producers provide updates at a fixed rate. The second addresses the minimization of the interest weighted average age achieved by a fixed subset of producers with possibly overlapping coverage by optimizing their update rates. The first problem is shown to be submodular and thus amenable to greedy optimization while the second has a non-convex/non-concave cost function which is amenable to effective optimization using the Frank-Wolfe algorithm. Numerical results exhibit the benefits of context dependent optimization information sharing among obstructed sensing nodes.

Detecting Global Events with Bayesian Changepoint Detection on Flight Data

As a means of enhancing the intelligence community’s ability to be prepared and prompt in the detection of global events, we present a Bayesian changepoint detection methodology to detect anomalous activity in open-source flight data. We demonstrate a computationally inexpensive methodology to monitor a geographic location for changes in flight activity over time and demonstrate its utility in a case study of the 2022 Russian invasion of Ukraine. Beyond flight data, the methodology described in this paper shows the potential to generate near real-time situational awareness using open-source time series data streams. This paper was previously published and presented in the Donald R. Keith Memorial Capstone Conference at USMA in May of 2023.

An underwater sound event detection algorithm based on Temporal Graph Convolutional Network

Accurate underwater sound event detection (SED) and real-time underwater situation awareness plays an important role in intelligent sonar system. However, little research on underwater SED has been done because underwater acoustics have very complex and dynamic spectral and temporal characteristics which make underwater SED difficult to solve. The goal of SED is to recognize type and time of the input audio signal and most of SED research have been focused on airborne sound so far. To address underwater SED problem, we adopt temporal graph convolutional network (T-GCN), which is originally designed for traffic prediction and is composed of GCN and gated recurrent unit (GRU) To capture spectral and temporal information simultaneously, the GCN is used to learn topological structures to capture spectral correlation and the GRU is used to learn changes of input sound to capture temporal correlation. The proposed model utilizes sound spectrogram and its annotation with temporal information as input and then the GCN and GRU of the model is employed to solve underwater SED problem. Experiments using urban SED dataset and underwater sound dataset, demonstrate that our proposed model expresses connection of spectral and temporal information effectively and shows reliable underwater SED performance accordingly.

Event Detection, Localization, and Classification Based on Semi-Supervised Learning in Power Grids

Real-time situational awareness and event analysis are crucial to the security of the modern power grid, which is a complicated nonlinear system and hard to be completely modeled. Massive data is collected but the information hasn't been sufficiently leveraged. To effectively extract the event features, this paper proposes a framework for event detection, localization, and classification in power grids based on semi-supervised learning. Specifically, event detection is realized by invertible neural network (INN), hence to learn complex distributions of real-world measurements in a flexible way. Abundant normal measurements are learned by INN and explicit log-likelihoods then serve as the indicator to distinguish events with adequate sensitivity. Moreover, risks induced by events are assessed and spatial locations are determined. Since the majority of power system events are recorded without labels in practice, a pseudo label (PL) technique is leveraged to classify events with limited labels. The PL-based approach has an enhanced separating capability for events and outperforms other approaches under a low labeling rate. Case studies with simulated data in the IEEE 39-bus system and real-world measurements verify the effectiveness of the proposed framework.

Riverside: A design study on visualization for situation awareness in cybersecurity

Real-time situation awareness is a key challenge of cybersecurity defense. Visual analytics has been utilized for this purpose, but existing tools tend to require detailed knowledge about the network, which can be challenging in large-scale, production networks. We conducted an interview study involving 24 security professionals to gather requirements for the design, development, and evaluation of visualization to aid situation awareness in cybersecurity. Using these findings, we designed a visualization tool – called RIVERSIDE – for providing a real-time view of the dynamically changing computer network to support situation awareness. We evaluated Riverside in a user study involving 10 participants. Participants were placed in an incident response scenario that tasked them to identify malicious activity on a network. 20% of the users identified all attack component, while an additional 40% only missed one component.

Africa CDC's blueprint to enhance early warning surveillance: accelerating implementation of event-based surveillance in Africa.

Event-based surveillance (EBS) is a core component of early warning surveillance. In 2018, Africa CDC developed the first edition of an event-based surveillance framework to guide African Union Member States in implementing EBS. Country experiences during the COVID-19 pandemic demonstrated the value of data from non-traditional sources for real time situational awareness; at the same time revealed the huge gaps in strengthening this arm of surveillance. Learning from these lessons and to begin to close those gaps, Africa CDC convened subject matter experts from African Union Member States and technical partners to develop the second edition of the EBS framework, 2023 and its training materials. The revised version includes additional sections such as, the multi-sectoral one health collaboration in EBS, monitoring and evaluation, cross border EBS, and use of event management systems. The current manuscript provides an overview of the 2023 Africa CDC EBS framework and highlights experience in two countries that have successfully employed this resource in their implementation efforts.

Digital technology enhanced situation awareness for construction safety: Systematic review and future research directions

The dynamics of construction processes pose significant challenges for workers to perceive, understand, and anticipate hazards on sites. Maintaining an adequate level of situation awareness (SA) is critical for safety. Recent years have seen increasing research interests in SA and digital technology (DT) in the construction safety field. However, there lacks a systematic review that elucidates the impacts of DT on workers’ cognitive processes involved in hazard identification. To fill the research gap, the paper aims to conceptualize the role of SA and DT in construction safety management and provide a theoretical basis for the future development of technology-enabled situation-aware construction safety management systems. This study employed a five-step systematic review approach with a scientometric analysis to provide an unbiased and comprehensive overview of research concerning SA in construction safety. The results suggested that current research on the cognitive processes related to hazard identification has been fragmented and has caused theoretical confusion. There is a need to develop a cognitive framework for hazard identification. Four specific research directions were recommended: (1) develop and validate a real-time SA measurement, (2) identify SA information requirements, (3) develop a cognitive model of hazard identification, and (4) develop a technology-enhanced SA system. By synthesizing the current literature, this review paper identified and analyzed the theoretical roles of SA and DT in worker hazard identification, shedding light on the potential for developing a technology-enabled situation-aware construction safety management system. It also offered a valuable perspective to review and re-design safety management practices.

Deep-Learning-Based Denial-of-Service Resilient Framework for Wide Area Situational Awareness of Power Systems

Vulnerability of the wide area measurement system (WAMS) to Denial of Service (DoS) attacks can hinder the real-time situational awareness of the grid. In this regard, a DoS cyber-attack resilient WAMS framework is presented, which utilizes deep-learning based overcomplete denoising autoencoder (ODAE) architecture. The proposed framework is able to reconstruct PMU data during DoS attacks on PMU-PDC connections for the entire period of power system contingencies, which helps the operator to be aware of the occurrence of any such disturbance and assist them in decision making related to the stability of the grid. The performance of the proposed framework has been validated on WSCC 9 bus system, which is simulated on the developed cyber-physical WAMS testbed involving real time digital simulator (RTDS), hardware PMUs, Hardware PDC, network switches etc. Subsequently, the response of the suggested framework has also been evaluated on the real field PDC data gathered from the Northern Region of the Indian Power Grid (NRIPG). The experimental results indicate that the reconstructed PMU measurements during DoS attacks help in maintaining the dynamic visualization of system-wide information thereby ensuring the resiliency of wide area monitoring applications against DoS attacks.

Soldier Tracking and Health Monitoring System

The Soldier Tracking and Health Monitoring System is a comprehensive solution designed to enhance the safety, efficiency, and well-being of soldiers in military operations. This system combines advanced tracking technologies with health monitoring capabilities to provide real-time situational awareness and vital sign monitoring for individual soldiers. Overall, this system provides a comprehensive solution for tracking soldiers' locations and monitoring their health in real-time. By combining advanced tracking technologies with health monitoring capabilities, it aims to enhance the effectiveness and safety of military operations while ensuring the well-being of individual soldiers.

CyPhERS: A cyber-physical event reasoning system providing real-time situational awareness for attack and fault response

Cyber–physical systems (CPSs) constitute the backbone of critical infrastructures such as power grids or water distribution networks. Operating failures in these systems can cause serious risks for society. To avoid or minimize downtime, operators require real-time awareness about critical incidents. However, online event identification in CPSs is challenged by the complex interdependency of numerous physical and digital components, requiring to take cyber attacks and physical failures equally into account. The online event identification problem is further complicated through the lack of historical observations of critical but rare events, and the continuous evolution of cyber attack strategies. This work introduces and demonstrates CyPhERS, a Cyber-Physical Event Reasoning System. CyPhERS provides real-time information pertaining the occurrence, location, physical impact, and root cause of potentially critical events in CPSs, without the need for historical event observations. Key novelty of CyPhERS is the capability to generate informative and interpretable event signatures of known and unknown types of both cyber attacks and physical failures. The concept is evaluated and benchmarked on a demonstration case that comprises a multitude of attack and fault events targeting various components of a CPS. The results demonstrate that the event signatures provide relevant and inferable information on both known and unknown event types.

Operationalizing crop model data assimilation for improved on-farm situational awareness

The ability of ‘digital agriculture’ to support on-farm decision making is predicated on the real-time combination of observations and prior knowledge into an integrated digital environment. The mathematical discipline that seeks to provide this integration is known as model data assimilation (DA), with demonstrated benefits including improved predictive reliability, and the capacity to identify unexpected changes in field conditions and potential measurement errors. Despite routine adoption in other fields, the delayed adoption of DA in agriculture is due to the need to express end-of-season outcomes such as yield, update forecasts of these outcomes throughout the growing season as data become available, and enhance forecast reliability. To overcome these challenges, three guiding principles are introduced, providing a means to operationalize crop model DA for robust on-farm decision support. We apply the guiding principles using a South Australian viticulture case study. Our case study involves application of an iterative form of a widely used DA algorithm (ensemble Kalman filter) to dynamically update both static parameters and states associated with a grapevine simulation model. Daily weather data as well as fortnightly ground-based leaf area index (LAI) data are used for assimilation. It is shown how crop model DA can lead to not only significant improvements in forecasts of LAI but also to forecasts of end-of-season yield. The guiding principles also enable observations of greatest value to be identified throughout the season. This study highlights the role that formal crop model DA can play in agricultural decision support through enhancing situational awareness in real time.

Military Support and Rescue Robot

Abstract: Military support and rescue robots are becoming increasingly important in modern warfare and disaster response efforts. These robots can perform tasks that are too dangerous or difficult for human soldiers or first responders. They can also gather and transmit crucial information in real-time to help commanders make informed decisions. This abstract will discuss the key features and capabilities of military support and rescue robots, as well as their potential applications. One of the primary functions of military support and rescue robots is to assist soldiers and first responders in dangerous situations. These robots can be used to search for and extract wounded soldiers, identify and disarm explosive devices, and provide cover fire for advancing troops. They can also be equipped with sensors and cameras to gather intelligence and provide real-time situational awareness to commanders. In disaster response scenarios, these robots can assist in search and rescue efforts, locate and extract survivors from collapsed buildings, and provide aid and medical assistance to those in need. They can also be used to survey damaged infrastructure and assess the extent of the damage. Military support and rescue robots are typically equipped with advanced sensors and communication systems to enable them to operate in a variety of environments. They can be designed to operate on land, sea, or air, and can be adapted to handle different terrains and weather conditions. Some robots are also capable of autonomous operation, allowing them to navigate and complete tasks without human intervention. Overall, military support and rescue robots have the potential to greatly enhance the effectiveness of soldiers and first responders in dangerous and challenging environments.

Supporting Humanitarian Crisis Decision Making with Reliable Intelligence Derived from Social Media Using AI

Abstract Recent advances in the field of artificial intelligence bring promising new capabilities that can substantially improve our ability to manage complex and evolving situations in the face of uncertainty. Humanitarian crises exemplify such situations, and the pervasiveness of social media renders it one of the most abundant sources of real-time information available. However, it is quite a difficult task to condense a body of social media posts into useful information quickly. In this paper we consider the challenge of using social media reports to provide a reliable, real-time situational awareness in the management of humanitarian crises. Effectively addressing this challenge requires extracting only the relevant information out of text and images in individual social media posts, fusing this information together into actionable information points for decision makers, and providing an assessment of the trustworthiness of this information. We propose a general solution framework and discuss a system developed in collaboration with NATO which combines state-of-the-art deep learning, natural language processing, computer vision, and information fusion models to provide a reliable, actionable, real-time situational awareness for supporting decision making in humanitarian crisis logistics. In addition to the technical approach, we also discuss important practical aspects of this project including the development and validation process, challenges encountered along the way, and key lessons learned.

Improved space weather observations and modeling for aviation radiation

In recent years there has been a growing interest from the aviation community for space weather radiation forecasts tailored to the needs of the aviation industry. In 2019 several space weather centers began issuing advisories for the International Civil Aviation Organization alerting users to enhancements in the radiation environment at aviation flight levels. Due to a lack of routine observations, radiation modeling is required to specify the dose rates experienced by flight crew and passengers. While mature models exist, support for key observational inputs and further modeling advancements are needed. Observational inputs required from the ground-based neutron monitor network must be financially supported for research studies and operations to ensure real-time data is available for forecast operations and actionable end user decision making. An improved understanding of the geomagnetic field is required to reduce dose rate uncertainties in regions close to the open/closed geomagnetic field boundary, important for flights such as those between the continental US and Europe which operate in this region. Airborne radiation measurements, which are crucial for model validation and improvement, are lacking, particularly during solar energetic particle events. New measurement campaigns must be carried out to ensure progress and in situ atmospheric radiation measurements made available for real-time situational awareness. Furthermore, solar energetic particle forecasting must be improved to move aviation radiation nowcasts to forecasts in order to meet customer requirements for longer lead times for planning and mitigation.

Smart Mobility in the Cloud: Enabling Real-Time Situational Awareness and Cyber-Physical Control Through a Digital Twin for Traffic

This article presents the design, implementation, and use cases of the Chattanooga Digital Twin (CTwin) towards the vision for next-generation smart city applications for urban mobility management. CTwin is an end-to-end web-based platform that incorporates various aspects of the decision-making process for optimizing urban transportation systems in Chattanooga, Tennessee, to reduce traffic congestion, incidents, and vehicle fuel consumption. The platform serves as a cyberinfrastructure to collect and integrate multi-domain urban mobility data from various online repositories and Internet of Things (IoT) sensors, covering multiple urban aspects (e.g., traffic, natural hazards, weather, and safety) that are relevant to urban mobility management. The platform enables advanced capabilities for: (a) real-time situational awareness on traffic and infrastructure conditions on highways and urban roads, (b) cyber-physical control for optimizing traffic signal timing, and (c) interactive visual analytics on big urban mobility data and various metrics for traffic prediction and transportation performance evaluation. The platform is designed using a multi-level componentization paradigm and is implemented using modular and adaptive architecture, rendering it as a generalizable and extendable prototype for other urban management applications. We present several use cases to demonstrate CTwin’s core capabilities for supporting decision-making in smart urban mobility management.