Partial Observations Research Articles

Discovering dynamics and parameters of nonlinear oscillatory and chaotic systems from partial observations

Despite rapid progress in data acquisition techniques, many complex physical, chemical, and biological systems remain only partially observable, thus posing the challenge to identify valid theoretical models and estimate their parameters from an incomplete set of experimentally accessible time series. Here, we combine sensitivity methods and ranked-choice model selection to construct an automated hidden dynamics inference framework that can discover predictive nonlinear dynamical models for both observable and latent variables from noise-corrupted incomplete data in oscillatory and chaotic systems. After validating the framework for prototypical FitzHugh-Nagumo oscillations, we demonstrate its applicability to experimental data from squid neuron activity measurements and Belousov-Zhabotinsky reactions, as well as to the Lorenz system in the chaotic regime. Published by the American Physical Society 2024

Formal contracts mitigate social dilemmas in multi-agent reinforcement learning

Multi-agent Reinforcement Learning (MARL) is a powerful tool for training autonomous agents acting independently in a common environment. However, it can lead to sub-optimal behavior when individual incentives and group incentives diverge. Humans are remarkably capable at solving these social dilemmas. It is an open problem in MARL to replicate such cooperative behaviors in selfish agents. In this work, we draw upon the idea of formal contracting from economics to overcome diverging incentives between agents in MARL. We propose an augmentation to a Markov game where agents voluntarily agree to binding transfers of reward, under pre-specified conditions. Our contributions are theoretical and empirical. First, we show that this augmentation makes all subgame-perfect equilibria of all Fully Observable Markov Games exhibit socially optimal behavior, given a sufficiently rich space of contracts. Next, we show that for general contract spaces, and even under partial observability, richer contract spaces lead to higher welfare. Hence, contract space design solves an exploration-exploitation tradeoff, sidestepping incentive issues. We complement our theoretical analysis with experiments. Issues of exploration in the contracting augmentation are mitigated using a training methodology inspired by multi-objective reinforcement learning: Multi-Objective Contract Augmentation Learning. We test our methodology in static, single-move games, as well as dynamic domains that simulate traffic, pollution management, and common pool resource management.

Iterative algorithms for solving one-sided partially observable stochastic shortest path games

Real-world scenarios often involve dynamic interactions among competing agents, where decisions are made considering actions taken by others. These situations can be modeled as partially observable stochastic games (POSGs), with zero-sum variants capturing strictly competitive interactions (e.g., security scenarios). While such models address a broad range of problems, they commonly focus on infinite-horizon scenarios with discounted-sum objectives. Using the discounted-sum objective, however, can lead to suboptimal solutions in cases where the length of the interaction does not directly affect the gained rewards of the players.We thus focus on games with undiscounted objective and an indefinite horizon where every realization of the game is guaranteed to terminate after some unspecified number of turns. To manage the computational complexity of solving POSGs in general, we restrict to games with one-sided partial observability where only one player has imperfect information while their opponent is provided with full information about the current situation. We introduce two novel algorithms based on the heuristic search value iteration (HSVI) algorithm that iteratively solve sequences of easier-to-solve approximations of the game using fundamentally different approaches for constructing the sequences: (1) in GoalHorizon, the game approximations are based on a limited number of turns in which players can change their actions, (2) in GoalDiscount, the game approximations are constructed using an increasing discount factor. We provide theoretical qualitative guarantees for algorithms, and we also experimentally demonstrate that these algorithms are able to find near-optimal solutions on pursuit-evasion games and a game modeling privilege escalation problem from computer security.

Collaborative Learning-Based Spectrum Sensing Under Partial Observations

Collaborative Learning-Based Spectrum Sensing Under Partial Observations

Neural network-based reconstruction of steady-state temperature systems with unknown material composition

Effective thermal management is crucial for the performance and longevity of devices such as computer chips and batteries. A fundamental challenge in this field is accurately determining temperature distributions, which are often limited by incomplete observations. We introduce a neural-network-based approach to overcome these challenges. Specifically, a physics-informed fully convolutional auto-encoder coupled with a preceding spatial propagator network module is used to reconstruct complete temperature fields from partial observations, without requiring additional information about the spatial distribution of materials with varying thermal properties. This paper details the development and validation of our model, demonstrating its effectiveness in virtual sensing setups for both simulated and real-world thermal systems. The reconstruction capabilities for different sampling strategies are investigated. The most challenging of which is the “grid-edge” sampling where only discrete temperature sensors on the periphery of the systems are taken as input, amounting to a sampling ratio as low as 3.12%. The presented method is able to reconstruct the full temperature field with a relative average error of 1.1%. The NN-based approach outperforms the standard Kriging method in predicting the maximum temperature in a system, and proves more robust when the available measurements contain little information.

Мультиагентное тестирование на проникновение на основе AIRL

This paper explores the application of a multi-agent approach based on the Adversarial Inverse Reinforcement Learning (AIRL) method for penetration testing in information systems. Theoretical aspects of multi-agent AIRL are discussed, including the ability to model complex, multi-stage attacks, coordinate agent actions, and learn with partial observability, which accounts for limitations in information access. The practical application of this approach will demonstrate its effectiveness in identifying vulnerabilities, providing a deeper and more accurate security analysis.

Improving community detection in blockmodel by distance-based observation selection

Community detection is an important research topic in complex systems and has plenty of applications in real-world networks. Probabilistic methods, such as the Expectation–Maximization (EM), are developed to classify nodes that have similar connection patterns in a network based on blockmodels. However, the detection procedures in these models are typically started from randomly generated initial community distributions without prior knowledge. In biological and social networks, there are practical measures to obtain prior knowledge for a subset of nodes, such as local observations. These facts lead us to question how we can select a subset of nodes with known community labels to enhance the accuracy of the EM method. The current selection methods lack the relationship between detection accuracy and structural characteristics and most approaches consider the nodes as the center of communities, which is not suitable for block models. In this paper, we first study the relationships between the structural distance and detection accuracy without prior knowledge. Then we propose a distance-based indicator to describe the performance of the observation node set in the EM method. Finally, we introduce a scoring method based on the indicator to select a partial observation set, improving the accuracy of community detection using the EM method. Empirical results from synthetic and real-world networks corroborate that the proposed indicator could contribute to a better performance in kinds of scenarios.

A model template for reachability-based containment checking of imprecise observations in timed automata

AbstractVerifying safety requirements by model checking becomes increasingly important for safety-critical applications. For the validity of such proof in practice, the model needs to capture the actual behavior of the real system, which could be tested by containment checks of real observation traces. Basic equivalence checks, however, are not applicable if the system is only partially or imprecisely observable, if the model abstracts from explicit states with symbolic semantics, or if the checks are not expressible in the logics supported by a model checker. In this article, we solve the problem of observation containment checking in timed automata via reachability checking on tester systems. We introduce the logic SRL (sequence reachability logic) to express observations as sequences of delayed reachability properties. Through SBLL (introduced by Aceto et al.) as intermediate logic, we synthesize a set of matcher model templates for partial and imprecise observations and further extend these templates for the case of limited state accessibility in a model. For the obtained matching traces, we define the back-transformation into the original model domain and formally prove the correctness of the transformation. We implemented the observation matching approach, and apply it to a set of 7 demo and 3 case study models with different levels of observability. The results show that all positive and negative observations are correctly classified, and that the most advanced matcher model instance still offers average run times between 0.1 and 1 s in all but 3 scenarios.

Active Inference and Reinforcement Learning: A Unified Inference on Continuous State and Action Spaces Under Partial Observability.

Reinforcement learning (RL) has garnered significant attention for developing decision-making agents that aim to maximize rewards, specified by an external supervisor, within fully observable environments. However, many real-world problems involve partial or noisy observations, where agents cannot access complete and accurate information about the environment. These problems are commonly formulated as partially observable Markov decision processes (POMDPs). Previous studies have tackled RL in POMDPs by either incorporating the memory of past actions and observations or by inferring the true state of the environment from observed data. Nevertheless, aggregating observations and actions over time becomes impractical in problems with large decision-making time horizons and high-dimensional spaces. Furthermore, inference-based RL approaches often require many environmental samples to perform well, as they focus solely on reward maximization and neglect uncertainty in the inferred state. Active inference (AIF) is a framework naturally formulated in POMDPs and directs agents to select actions by minimizing a function called expected free energy (EFE). This supplies reward-maximizing (or exploitative) behavior, as in RL, with information-seeking (or exploratory) behavior. Despite this exploratory behavior of AIF, its use is limited to problems with small time horizons and discrete spaces due to the computational challenges associated with EFE. In this article, we propose a unified principle that establishes a theoretical connection between AIF and RL, enabling seamless integration of these two approaches and overcoming their limitations in continuous space POMDP settings. We substantiate our findings with rigorous theoretical analysis, providing novel perspectives for using AIF in designing and implementing artificial agents. Experimental results demonstrate the superior learning capabilities of our method compared to other alternative RL approaches in solving partially observable tasks with continuous spaces. Notably, our approach harnesses information-seeking exploration, enabling it to effectively solve reward-free problems and rendering explicit task reward design by an external supervisor optional.

Deep Reinforcement Learning for Fluid Mechanics: Control, Optimization, and Automation

A comprehensive review of recent advancements in applying deep reinforcement learning (DRL) to fluid dynamics problems is presented. Applications in flow control and shape optimization, the primary fields where DRL is currently utilized, are thoroughly examined. Moreover, the review introduces emerging research trends in automation within computational fluid dynamics, a promising field for enhancing the efficiency and reliability of numerical analysis. Emphasis is placed on strategies developed to overcome challenges in applying DRL to complex, real-world engineering problems, such as data efficiency, turbulence, and partial observability. Specifically, the implementations of transfer learning, multi-agent reinforcement learning, and the partially observable Markov decision process are discussed, illustrating how these techniques can provide solutions to such issues. Finally, future research directions that could further advance the integration of DRL in fluid dynamics research are highlighted.

Deep reinforcement learning of airfoil pitch control in a highly disturbed environment using partial observations

Deep reinforcement learning of airfoil pitch control in a highly disturbed environment using partial observations

Path detectability verification for time-dependent systems with application to flexible manufacturing systems

This paper addresses the path detectability verification problem for time-dependent systems modeled by time labeled Petri nets (TLPNs). To capture the information precisely, it may not be sufficient to estimate the current state by resorting to the partial system observation, and it is usually crucial to decide the path of a system to reach the current state. Path detectability characterizes a time-dependent system whose current state and the corresponding path can be uniquely determined after a real-time observation (RTO). Revised state class graphs (RSCGs) are proposed to capture the time information for the evolution of the RTO in a TLPN system. We demonstrate the time information overlap problem in the RSCG, i.e., several paths are associated with the same observable events and the same time instants, which leads to such paths that cannot be distinguished. The nodes required to be computed in the proposed RSCGs are always less or equal to those of the modified state class graphs reported in the literature, since the enumeration of all the states is avoided. Based on the RSCG, an RSCG observer is formulated to address the time information overlap problem and capture the number of such paths in the TLPN system. The efficiency analysis of this verification method is provided. In this paper, the results are applied to a real production system, exposing the practical value of the reported method.

Inference on the Macroscopic Dynamics of Spiking Neurons.

The process of inference on networks of spiking neurons is essential to decipher the underlying mechanisms of brain computation and function. In this study, we conduct inference on parameters and dynamics of a mean-field approximation, simplifying the interactions of neurons. Estimating parameters of this class of generative model allows one to predict the system's dynamics and responses under changing inputs and, indeed, changing parameters. We first assume a set of known state-space equationsand address the problem of inferring the lumped parameters from observed time series. Crucially, we consider this problem in the setting of bistability, random fluctuations in system dynamics, and partial observations, in which some states are hidden. To identify the most efficient estimation or inversion scheme in this particular system identification, we benchmark against state-of-the-art optimization and Bayesian estimation algorithms, highlighting their strengths and weaknesses. Additionally, we explore how well the statistical relationships between parameters are maintained across different scales. We found that deep neural density estimators outperform other algorithms in the inversion scheme, despite potentially resulting in overestimated uncertainty and correlation between parameters. Nevertheless, this issue can be improved by incorporating time-delay embedding. We then eschew the mean-field approximation and employ deep neural ODEs on spiking neurons, illustrating prediction of system dynamics and vector fields from microscopic states. Overall, this study affords an opportunity to predict brain dynamics and responses to various perturbations or pharmacological interventions using deep neural networks.

Stochastic EM algorithm for partially observed stochastic epidemics with individual heterogeneity.

We develop a stochastic epidemic model progressing over dynamic networks, where infection rates are heterogeneous and may vary with individual-level covariates. The joint dynamics are modeled as a continuous-time Markov chain such that disease transmission is constrained by the contact network structure, and network evolution is in turn influenced by individual disease statuses. To accommodate partial epidemic observations commonly seen in real-world data, we propose a stochastic EM algorithm for inference, introducing key innovations that include efficient conditional samplers for imputing missing infection and recovery times which respect the dynamic contact network. Experiments on both synthetic and real datasets demonstrate that our inference method can accurately and efficiently recover model parameters and provide valuable insight at the presence of unobserved disease episodes in epidemic data.

Joint computation offloading and resource allocation for end-edge collaboration in internet of vehicles via multi-agent reinforcement learning

Vehicular edge computing (VEC), a promising paradigm for the development of emerging intelligent transportation systems, can provide lower service latency for vehicular applications. However, it is still a challenge to fulfill the requirements of such applications with stringent latency requirements in the VEC system with limited resources. In addition, existing methods focus on handling the offloading task in a certain time slot with statically allocated resources, but ignore the heterogeneous tasks’ different resource requirements, resulting in resource wastage. To solve the real-time task offloading and heterogeneous resource allocation problem in VEC system, we propose a decentralized solution based on the attention mechanism and recurrent neural networks (RNN) with a multi-agent distributed deep deterministic policy gradient (AR-MAD4PG). First, to address the partial observability of agents, we construct a shared agent graph and propose a periodic communication mechanism that enables edge nodes to aggregate information from other edge nodes. Second, to help agents better understand the current system state, we design an RNN-based feature extraction network to capture the historical state and resource allocation information of the VEC system. Thirdly, to tackle the challenges of excessive joint observation-action space and ineffective information interference, we adopt the multi-head attention mechanism to compress the dimension of the observation-action space of agents. Finally, we build a simulation model based on the actual vehicle trajectories, and the experimental results show that our proposed method outperforms the existing approaches.

ANISE: Assembly-Based Neural Implicit Surface Reconstruction.

We present ANISE, a method that reconstructs a 3D shape from partial observations (images or sparse point clouds) using a part-aware neural implicit shape representation. The shape is formulated as an assembly of neural implicit functions, each representing a different part instance. In contrast to previous approaches, the prediction of this representation proceeds in a coarse-to-fine manner. Our model first reconstructs a structural arrangement of the shape in the form of geometric transformations of its part instances. Conditioned on them, the model predicts part latent codes encoding their surface geometry. Reconstructions can be obtained in two ways: (i) by directly decoding the part latent codes to part implicit functions, then combining them into the final shape; or (ii) by using part latents to retrieve similar part instances in a part database and assembling them in a single shape. We demonstrate that, when performing reconstruction by decoding part representations into implicit functions, our method achieves state-of-the-art part-aware reconstruction results from both images and sparse point clouds. When reconstructing shapes by assembling parts retrieved from a dataset, our approach significantly outperforms traditional shape retrieval methods even when significantly restricting the database size. We present our results in well-known sparse point cloud reconstruction and single-view reconstruction benchmarks.

Graph learning from incomplete graph signals: From batch to online methods

Inferring graph topologies from data is crucial in many graph-related applications. Existing works typically assume that signals are observed at all nodes, which may not hold due to application-specific constraints. The problem becomes more challenging when data are sequentially available and no delay is tolerated. To address these issues, we propose an approach for learning graphs from incomplete data. First, the problem of learning graphs with missing data is formulated as maximizing the posterior distribution with hidden variables from a Bayesian perspective. Then, we propose an expectation maximization (EM) algorithm to solve the induced problem, in which graph learning and graph signal recovery are jointly performed. Furthermore, we extend the proposed EM algorithm to an online version to accommodate the delay-sensitive situations of sequential data. Theoretically, we analyze the dynamic regret of the proposed online algorithm, illustrating the effectiveness of our algorithm in tracking graphs from partial observations in an online manner. Finally, extensive experiments on synthetic and real data are conducted, and the results corroborate that our approach can learn graphs effectively from incomplete data in both batch and online situations.

Codiscovering graphical structure and functional relationships within data: A Gaussian Process framework for connecting the dots

Most problems within and beyond the scientific domain can be framed into one of the following three levels of complexity of function approximation. Type 1: Approximate an unknown function given input/output data. Type 2: Consider a collection of variables and functions, some of which are unknown, indexed by the nodes and hyperedges of a hypergraph (a generalized graph where edges can connect more than two vertices). Given partial observations of the variables of the hypergraph (satisfying the functional dependencies imposed by its structure), approximate all the unobserved variables and unknown functions. Type 3: Expanding on Type 2, if the hypergraph structure itself is unknown, use partial observations of the variables of the hypergraph to discover its structure and approximate its unknown functions. These hypergraphs offer a natural platform for organizing, communicating, and processing computational knowledge. While most scientific problems can be framed as the data-driven discovery of unknown functions in a computational hypergraph whose structure is known (Type 2), many require the data-driven discovery of the structure (connectivity) of the hypergraph itself (Type 3). We introduce an interpretable Gaussian Process (GP) framework for such (Type 3) problems that does not require randomization of the data, access to or control over its sampling, or sparsity of the unknown functions in a known or learned basis. Its polynomial complexity, which contrasts sharply with the super-exponential complexity of causal inference methods, is enabled by the nonlinear ANOVA capabilities of GPs used as a sensing mechanism.

Data Assimilation in Chaotic Systems Using Deep Reinforcement Learning

AbstractData assimilation (DA) plays a pivotal role in diverse applications, ranging from weather forecasting to trajectory planning for autonomous vehicles. A prime example is the widely used ensemble Kalman filter (EnKF), which relies on the Kalman filter's linear update equation to correct each of the ensemble forecast member's state with incoming observations. Recent advancements have witnessed the emergence of deep learning approaches in this domain, primarily within a supervised learning framework. However, the adaptability of such models to untrained scenarios remains a challenge. In this study, we introduce a new DA strategy that utilizes reinforcement learning (RL) to apply state corrections using full or partial observations of the state variables. Our investigation focuses on demonstrating this approach to the chaotic Lorenz 63 and 96 systems, where the agent's objective is to maximize the geometric series with terms that are proportional to the negative root‐mean‐squared error (RMSE) between the observations and corresponding forecast states. Consequently, the agent develops a correction strategy, enhancing model forecasts based on available observations. Our strategy employs a stochastic action policy, enabling a Monte Carlo‐based DA framework that relies on randomly sampling the policy to generate an ensemble of assimilated realizations. Numerical results demonstrate that the developed RL algorithm performs favorably when compared to the EnKF. Additionally, we illustrate the agent's capability to assimilate non‐Gaussian observations, addressing one of the limitations of the EnKF.

Path-Wise Continuous-Time Transmission with Applications in Source Identification from Partial Observations

Path-Wise Continuous-Time Transmission with Applications in Source Identification from Partial Observations