Partial Observations Research Articles

Data-Driven Pathwise Sampling Approaches for Online Anomaly Detection

Moving vehicle-based sensors (MVSs) have been increasingly used for real-time sensing and anomaly detection in various applications such as the detection of wildfires and oil spills. In this article, we propose data-driven sampling strategies using MVSs to quickly identify abrupt changes in an area of interest in real time considering their pathwise movement constraints. To tackle challenges due to variability and partial observability of online observations, we integrate instruments of statistical process control and mathematical optimization to monitor the global status of the area of interest and adaptively adjust paths of MVSs to sample from suspicious locations based on real-time data. We provide theoretical investigations and conduct simulations to validate the superior performance of the proposed methods. In a numerical study based on real-world wildfire data, we illustrate that our proposed strategies are able to detect wildfires much earlier than benchmark methods and can significantly reduce wildfire-related costs.

Optimizing pathfinding for goal legibility and recognition in cooperative partially observable environments

In this paper, we perform a joint design of goal legibility and recognition in a cooperative, multi-agent pathfinding setting with partial observability. More specifically, we consider a set of identical agents (the actors) that move in an environment only partially observable to an observer in the loop. The actors are tasked with reaching a set of locations that need to be serviced in a timely fashion. The observer monitors the actors' behavior from a distance and needs to identify each actor's destination based on the actor's observable movements. Our approach generates legible paths for the actors; namely, it constructs one path from the origin to each destination so that these paths overlap as little as possible while satisfying budget constraints. It also equips the observer with a goal-recognition mapping between unique sequences of observations and destinations, ensuring that the observer can infer an actor's destination by making the minimum number of observations (legibility delay). Our method substantially extends previous work, which is limited to an observer with full observability, showing that optimizing pathfinding for goal legibility and recognition can be performed via a reformulation into a classical minimum cost flow problem in the partially observable case when the algorithms for the fully observable case are appropriately modified. Our empirical evaluation shows that our techniques are as effective in partially observable settings as in fully observable ones.

Inverse stable reconstruction of 3 coefficients for the heterogeneous Maxwell equations by finite number of partial interior observations

We consider an inverse problem of determining the isotropic inhomogeneous electromagnetic coefficients of the non-stationary Maxwell’s equations in a bounded domain of R3 by means of a finite number of interior data of as less as possible components of the solutions. Our main result is a Lipschitz stability estimate for the inverse problem and our proof relies on a Carleman estimate for the heterogeneous Maxwell’s equations.

Deep Recurrent Reinforcement Learning for Intercept Guidance Law under Partial Observability

ABSTRACT Nowadays, the rapid development of hypersonic vehicles brings great challenges to the missile defense system. As achieving successful interception depends highly on terminal guidance laws, research on guidance laws for intercepting highly maneuvering targets has aroused increasing attention. Artificial intelligence technologies, such as deep reinforcement learning (DRL), have been widely applied to improve the performance of guidance laws. However, the existing DRL guidance laws rarely consider the partial observability problem of onboard sensors, resulting in the limitations of their engineering applications. In this paper, a deep recurrent reinforcement learning (DRRL)-based guidance method is investigated to address the intercept guidance problem against maneuvering targets under partial observability. The sequence consisting of previous state observations is utilized as the input of the policy network. A recurrent layer is introduced into the networks to extract hidden information behind the temporal sequence to support policy training. The guidance problem is formulated as a partially observable Markov decision process model, and then a range-weighted reward function that considers the line-of-sight rate and energy consumption is designed to guarantee convergence of policy training. The effectiveness of the proposed DRRL guidance law is validated by extensive numerical simulations.

Building Category Graphs Representation with Spatial and Temporal Attention for Visual Navigation

Given an object of interest, visual navigation aims to reach the object’s location based on a sequence of partial observations. To this end, an agent needs to (1) acquire specific knowledge about the relations of object categories in the world during training and (2) locate the target object based on the pre-learned object category relations and its trajectory in the current unseen environment. In this article, we propose a Category Relation Graph (CRG) to learn the knowledge of object category layout relations and a Temporal-Spatial-Region attention (TSR) architecture to perceive the long-term spatial-temporal dependencies of objects, aiding navigation. We establish CRG to learn prior knowledge of object layout and deduce the positions of specific objects. Subsequently, we propose the TSR architecture to capture relationships among objects in temporal, spatial, and regions within observation trajectories. Specifically, we implement a Temporal attention module (T) to model the temporal structure of the observation sequence, implicitly encoding historical moving or trajectory information. Then, a Spatial attention module (S) uncovers the spatial context of the current observation objects based on CRG and past observations. Last, a Region attention module (R) shifts the attention to the target-relevant region. Leveraging the visual representation extracted by our method, the agent accurately perceives the environment and easily learns a superior navigation policy. Experiments on AI2-THOR demonstrate that our CRG-TSR method significantly outperforms existing methods in both effectiveness and efficiency. The supplementary material includes the code and will be publicly available.

Unravelling uncertainty in trajectory prediction using a non-parametric approach

Predicting the trajectories of road agents is fundamental for self-driving cars. Trajectory prediction contains many sources of uncertainty in data and modelling. A thorough understanding of this uncertainty is crucial in a safety-critical task like auto-piloting a vehicle. In practice, it is necessary to distinguish between the uncertainty caused by partial observability of all factors that may affect a driver’s near-future decisions, the so-called aleatoric uncertainty, and the uncertainty of deploying a model in new scenarios that are possibly not present in the training set, the so-called epistemic uncertainty. They reflect the trade-off between data collection and model improvement In this paper, we propose a new framework to systematically quantify both sources of uncertainty. Specifically, to approximate the spatial distribution of an agent’s future position, we propose a 2D histogram-based deep learning model combined with deep ensemble techniques for measuring aleatoric and epistemic uncertainty by entropy-based quantities. The proposed Uncertainty Quantification Network (UQnet) employs a causal part to enhance its generalizability so rare driving behaviours can be effectively identified. Experiments on the INTERACTION dataset show that UQnet is able to give more robust predictions in generalizability tests compared to the correlation-based models. Further analysis presents that high aleatoric uncertainty cases are mainly caused by heterogeneous driving behaviours and unknown intended directions. Based on this aleatoric uncertainty component, we estimate the lower bounds of mean-square-error and final-displacement-error as indicators for the predictability of trajectories. Furthermore, the analysis of epistemic uncertainty illustrates that domain knowledge of speed-dependent driving behaviour is essential for adapting a model from low-speed to high-speed situations. Our paper contributes to motion forecasting with a new framework, that recasts the problem of accuracy improvement in a way that focuses on differentiating between unpredictable components and rare cases for which more and different data should be collected.

Updating Nonlinear Stochastic Dynamics of an Uncertain Nozzle Model Using Probabilistic Learning With Partial Observability and Incomplete Dataset

Abstract This article introduces a methodology for updating the nonlinear stochastic dynamics of a nozzle with uncertain computational model. The approach focuses on a high-dimensional nonlinear computational model constrained by a small target dataset. Challenges include the large number of degrees-of-freedom, geometric nonlinearities, material uncertainties, stochastic external loads, underobservability, and high computational costs. A detailed dynamic analysis of the nozzle is presented. An updated statistical surrogate model relating the observations of interest to the control parameters is constructed. Despite small training and target datasets and partial observability, the study successfully applies probabilistic learning on manifolds (PLoM) to address these challenges. PLoM captures geometric nonlinear effects and uncertainty propagation, improving conditional mean statistics compared to training data. The conditional confidence region demonstrates the ability of the methodology to accurately represent both observed and unobserved output variables, contributing to advancements in modeling complex systems.

Report on Landsat 8 and Sentinel-2B observations of the Nord Stream 2 pipeline methane leak

Abstract. In late September 2022, explosions of the Nord Stream pipelines caused what could be the largest anthropogenic methane leak ever recorded. We report on Landsat 8 (L8) and Sentinel-2B (S-2B) observations of the sea-foam patch produced by the Nord Stream 2 (NS2) leak located close to Bornholm island, acquired on 29 and 30 September, respectively. Usually, reflected sunlight over sea is insufficient for these Earth imagers to observe any methane signal in nadir-viewing geometry. However, the NS2 foam patch observed here is bright enough to possibly allow the detection of methane above it. We apply the multi-band single-pass (MBSP) method to infer methane enhancement above the NS2 foam patch and then use the integrated mass enhancement (IME) method in a Monte Carlo ensemble approach to estimate methane leak rates and their uncertainties. This very specific NS2 observation case challenges some of MBSP and IME implicit assumptions and thus calls for customized calibrations: (1) for MBSP, we perform an empirical calibration of sea-foam albedo spectral dependence by using sea-foam observations in ship trails, and (2) for IME, we yield a tailored effective wind speed calibration that accounts for a partial plume observation, as methane enhancement may only be seen above the NS2 sea-foam patch. Our comprehensive uncertainty analysis yields large methane leak rate uncertainty ranges that include zero for single overpasses and, assuming they are independent, a best estimate of 502 ± 464 t h−1 for the combined averaged L8 and S-2B emission rate. Within all our Monte Carlo ensembles, positive methane leak rates have higher probabilities (80 %–88 %) than negative ones (12 %–20 %), thus indicating that L8 and S-2B likely captured a methane-related signal. Overall, we see our work both as a nuanced analysis of L8 and S-2B contributions to quantifying the NS2 leak emissions and as a methodological cautionary tale that builds insight into MBSP and IME sensitivities.

Application Study on the Reinforcement Learning Strategies in the Network Awareness Risk Perception and Prevention

The intricacy of wireless network ecosystems and Internet of Things (IoT) connected devices have increased rapidly as technology advances and cyber threats increase. The existing methods cannot make sequential decisions in complex network environments, particularly in scenarios with partial observability and non-stationarity. Network awareness monitors and comprehends the network's assets, vulnerabilities, and ongoing activities in real-time. Advanced analytics, machine learning algorithms, and artificial intelligence are used to improve risk perception by analyzing massive amounts of information, identifying trends, and anticipating future security breaches. Hence, this study suggests the Deep Reinforcement Learning-assisted Network Awareness Risk Perception and Prevention Model (DRL-NARPP) for detecting malicious activity in cybersecurity. The proposed system begins with the concept of network awareness, which uses DRL algorithms to constantly monitor and evaluate the condition of the network in terms of factors like asset configurations, traffic patterns, and vulnerabilities. DRL provides autonomous learning and adaptation to changing network settings, revealing the ever-changing nature of network awareness risks in real time. Incorporating DRL into risk perception increases the system's capacity to recognize advanced attack methods while simultaneously decreasing the number of false positives and enhancing the reliability of risk assessments. DRL algorithms drive dynamic and context-aware response mechanisms, making up the adaptive network prevention component of the development. Predicting new threats and proactively deploying preventive measures, such as changing firewall rules, isolating compromised devices, or dynamically reallocating resources to reduce developing risks, is made possible by the system's ability to learn from historical data and prevailing network activity. The suggested DRL-NARPP model increases the anomaly detection rate by 98.3%, the attack prediction accuracy rate by 97.4%, and the network risk assessment ratio by 96.4%, reducing the false positive ratio by 11.2% compared to other popular methodologies.

The quantum cartpole: A benchmark environment for non-linear reinforcement learning

Feedback-based control is the de-facto standard when it comes to controlling classical stochastic systems and processes. However, standard feedback-based control methods are challenged by quantum systems due to measurement induced backaction and partial observability. Here we remedy this by using weak quantum measurements and model-free reinforcement learning agents to perform quantum control. By comparing control algorithms with and without state estimators to stabilize a quantum particle in an unstable state near a local potential energy maximum, we show how a trade-off between state estimation and controllability arises. For the scenario where the classical analogue is highly nonlinear, the reinforcement learned controller has an advantage over the standard controller. Additionally, we demonstrate the feasibility of using transfer learning to develop a quantum control agent trained via reinforcement learning on a classical surrogate of the quantum control problem. Finally, we present results showing how the reinforcement learning control strategy differs from the classical controller in the non-linear scenarios.

Characterization of multi-channel denial-of-service

The embedded closed-loop in integrated systems can be compromised when attackers execute a successful malicious attack. Hence, over the past decade, interest in enhancing the safety of Cyber-Physical Systems (CPSs) has risen. This article examines the resilient control problem for CPSs with numerous transmission channels under Denial-of-Service (DoS). To begin with, a partial observer technique is developed in response to the Multi-Channel DoS (MCDoS) condition. Secondly, the changing frequency of MCDoS is characterized while maintaining the Input-to-State Stability (ISS) of the closed-loop system. Next, an event-based control strategy is developed to address Full-Scale DoS (FSDoS). Finally, the optimal control parameters are designed to make the system resilient against maximum MCDoS changing frequency.

GMIM: Self-supervised pre-training for 3D medical image segmentation with adaptive and hierarchical masked image modeling

Self-supervised pre-training and fully supervised fine-tuning paradigms have received much attention to solve the data annotation problem in deep learning fields. Compared with traditional pre-training on large natural image datasets, medical self-supervised learning methods learn rich representations derived from unlabeled data itself thus avoiding the distribution shift between different image domains. However, nowadays state-of-the-art medical pre-training methods were specifically designed for downstream tasks making them less flexible and difficult to apply to new tasks. In this paper, we propose grid mask image modeling, a flexible and general self-supervised method to pre-train medical vision transformers for 3D medical image segmentation. Our goal is to guide networks to learn the correlations between organs and tissues by reconstructing original images based on partial observations. The relationships are consistent within the human body and invariant to disease type or imaging modality. To achieve this, we design a Siamese framework consisting of an online branch and a target branch. An adaptive and hierarchical masking strategy is employed in the online branch to (1) learn the boundaries or small contextual mutation regions within images; (2) to learn high-level semantic representations from deeper layers of the multiscale encoder. In addition, the target branch provides representations for contrastive learning to further reduce representation redundancy. We evaluate our method through segmentation performance on two public datasets. The experimental results demonstrate our method outperforms other self-supervised methods. Codes are available at https://github.com/mobiletomb/Gmim.

Identifying the body force from partial observations of a two-dimensional incompressible velocity field

Identifying the body force from partial observations of a two-dimensional incompressible velocity field

The role of video games in enhancing managers' strategic thinking and cognitive abilities: An experiential survey

Using gamification and video games is one of the modern approaches to cognitive enhancement and improving the abilities and competencies of managers, including strategic thinking skills. Many organizations use video games in the fields of education, marketing, business, and entrepreneurship. This research aimed to investigate the role of video games in enhancing managers' strategic thinking and their potential contribution to developing cognitive capabilities. The sample included 30 students actively involved in the innovation and entrepreneurship ecosystem. To measure the strategic thinking of the participants, Pisapia’s strategic thinking questionnaire was used, and the CANTAB test was employed to measure their cognitive capabilities. To identify the individuals’ game-playing styles, indicators of micro-management, planning, plan recognition, predictions, gathering resources, partial observability, and damage avoidance were designed. The findings indicated that 53.3 percent of the participants had reflective thinking, 30 percent had systems thinking, and the rest had a reframing thinking style as their strategic thinking dominant dimension. On the other hand, given the identified correlation between the damage avoidance criteria with the thinking and reflective style, as well as the inverse and significant correlation of the gathering resources criteria with the results of the PRM test, it seems that strategic games have the potential to change and even develop some cognitive functions such as attention, reaction, and memory, and can be considered as tools to improve cognitive ability. These games can be used to design tasks to enhance managers' cognitive abilities and subsequently promote their strategic thinking and decision-making skills.

Learning-Based Autonomous Channel Access in the Presence of Hidden Terminals

We consider the problem of autonomous channel access (AutoCA), where a group of terminals tries to discover a communication strategy with an access point (AP) via a common wireless channel in a distributed fashion. Due to the irregular topology and the limited communication range of terminals, a practical challenge for AutoCA is the hidden terminal problem, which is notorious in wireless networks for deteriorating throughput and delay performances. To meet the challenge, this paper presents a new multi-agent deep reinforcement learning paradigm, dubbed MADRL-HT, tailored for AutoCA in the presence of hidden terminals. MADRL-HT exploits topological insights and transforms the observation space of each terminal into a scalable form independent of the number of terminals. To compensate for the partial observability, we put forth a look-back mechanism such that the terminals can infer behaviors of their hidden terminals from the carrier-sensed channel states as well as feedback from the AP. A window-based global reward function is proposed, whereby the terminals are instructed to maximize the system throughput while balancing the terminals' transmission opportunities over the course of learning. Considering short-packet machine-type communications, extensive numerical experiments verified the superior performance of our solution benchmarked against the legacy carrier-sense multiple access with collision avoidance (CSMA/CA) protocol.

Model predictive manipulation of compliant objects with multi-objective optimizer and adversarial network for occlusion compensation

Background:The manipulation of compliant objects by robotic systems remains a challenging task, largely due to their variable shapes and the complex, high-dimensional nature of their interaction dynamics. Traditional robotic manipulation strategies struggle with the accurate modeling and control necessary to handle such materials, especially in the presence of visual occlusions that frequently occur in dynamic environments. Meanwhile, for most unstructured environments, robots are required to have autonomous interactions with their surroundings. Methods:To solve the shape manipulation of compliant objects in an unstructured environment, we begin by exploring the regression-based algorithm of representing the high-dimensional configuration space of deformable objects in a compressed form that enables efficient and effective manipulation. Simultaneously, we address the issue of visual occlusions by proposing the integration of an adversarial network, enabling guiding the shaping task even with partial observations of the object. Afterwards, we propose a receding-time estimator to coordinate the robot action with the computed shape features while satisfying various performance criteria. Finally, model predictive controller is utilized to compute the robot’s shaping motions subject to safety constraints. Detailed experiments are presented to evaluate the proposed manipulation framework. Significant findings:Our MPC framework utilizes the compressed representation and occlusion-compensated information to predict the object’s behavior, while the multi-objective optimizer ensures that the resulting control actions meet multiple performance criteria. Through rigorous experimental validation, our approach demonstrates superior manipulation capabilities in scenarios with visual obstructions, outperforming existing methods in terms of precision and operational reliability. The findings highlight the potential of our integrated approach to significantly enhance the manipulation of compliant objects in real-world robotic applications.

Fully Data-Driven Robust Output Formation Tracking Control for Heterogeneous Multiagent System With Multiple Leaders and Actuator Faults.

This article investigates a fully data-driven method to solve the robust output formation tracking control problem for the multiagent system (MAS) under actuator faults. The outputs of the followers are controlled to track those of multiple leaders with respect to a convex point while achieving an expected time-varying formation. To obviate the requirement of various system prior knowledge in typical MAS control, a hierarchical frame is developed with three learning and control stages using the online measured data. First, a distributed adaptive observer is designed to coordinate the state convex of multiple leaders while estimating unknown dynamics. The adaptive mechanism relaxes the demand for global topology. Second, by collecting and reusing the online system data, an off-policy reinforcement learning (RL) method is proposed in a continuous form to acquire nominal feedback gains from partial observations of the followers. Essential system models are learned along with the RL process, while solutions to the output regulation equations are implicitly obtained. Third, a comprehensive robust controller is further presented based on the previous learning results. To address the actuator faults with efficiency loss and bias, the adaptive neural networks and robust compensations are utilized in a model-free manner. The output formation tracking is achieved under a derived feasibility condition while stabilities of the learning and control methods are analyzed. Finally, simulation results demonstrate the validity of this fully data-driven control frame.

Advancements in Reinforcement Learning

Reinforcement Learning (RL) stands at the forefront of machine learning, offering a paradigm where agents learn to navigate complex environments through trial and error interactions. This paper provides a comprehensive overview of RL concepts, algorithms, challenges, and future directions. Key concepts such as agent-environment interaction, Markov Decision Processes (MDPs), value-based and policy-based methods are elucidated. Challenges including the exploration-exploitation trade-off, sample efficiency, and safety concerns are discussed, alongside potential breakthroughs in deep RL, handling continuous action spaces, and dealing with partial observability. The integration of RL with emerging technologies like IoT and its implications for real-world applications such as robotics, autonomous vehicles, and healthcare are explored. Finally, future trends and directions in RL research, including AI engineering and automated feature engineering, are outlined, highlighting the potential for transformative advancements and their impact on various domains. This paper serves as a comprehensive resource for researchers, practitioners, and enthusiasts interested in the evolving landscape of Reinforcement Learning

Partially observable deep reinforcement learning for multi-agent strategy optimization of human-robot collaborative disassembly: A case of retired electric vehicle battery

The burgeoning electric vehicle (EV) industry has precipitated a commensurate surge in the consumption of EV batteries, which are currently labor-intensive and inefficient for the recycling and disassembly of EV batteries. However, it is a potential trend to enhance the efficacy and safety of the disassembly of EV batteries based on human-robot collaboration (HRC) method. Because of the uncertainty of retired EV battery disassembly and the inefficiency of the existing disassembling sequence, it is difficult to be fully accomplish through HRC disassembly. The collaborative disassembly of EV batteries by humans and robots can be conceptualized as agents engaging with and learning from the environment, and modeled as a multi-agent Markov game process. This paper aims to address the challenge of HRC in the disassembly of EV batteries by recognizing the dual attributes of partial observability and non-smoothness in the suitable disassembly scenario. A partially observable multi-agent reinforcement learning environment is constructed, incorporating the structural aspects of the EV battery and the disassembly task. The framework is extended to the QMIX-HRC algorithm on the QMIX architecture (as a value-based multi-agent deep reinforcement learning algorithm), specifically designed to tackle the sequence problem in human-robot collaborative disassembly of EV batteries. The optimization results would yield a task sequence to offer maximal global co-benefit during the exploration iteration, facilitating a reduction in labor costs and an enhancement of co-efficiency. The viability of the QMIX-HRC disassembly strategy would be verified through the eventual disassembly sequence of a simulated battery pack through a real human-robot collaborative disassembly station.

Modeling bycatch abundance in tropical tuna purse seine fisheries on floating objects using the Δ method

Abstract Bycatch rates are essential to estimating fishery impacts and making management decisions, but data on bycatch are often limited. Tropical tuna purse seine (PS) fisheries catch numerous bycatch species, including vulnerable silky sharks. Even if bycatch proportion is relatively low, impacts on pelagic ecosystems may be important due to the large size of these fisheries. Partial observer coverage of bycatch is a major impediment to assessing impacts. Here we develop a generic Δ modeling approach for predicting catch of four major bycatch species, including silky sharks, in floating object-associated fishing sets of the French Indian Ocean PS fleet from 2011 to 2018 based on logbook and observer data. Cross-validation and variable selection are used to identify optimal models consisting of a random forest model for presence–absence and a negative binomial general-additive model for abundance when present. Though models explain small to moderate amounts of variance (5–15%), they outperform a simpler approach commonly used for reporting, and they allow us to estimate total annual bycatch for the four species with robust estimates of uncertainty. Interestingly, uncertainty relative to mean catch is lower for top predators than forage species, consistent with these species having similar behavior and ecological niches to tunas.