Observable Systems Research Articles

Cooperative multi-target hunting by unmanned surface vehicles based on multi-agent reinforcement learning

To solve the problem of multi-target hunting by an unmanned surface vehicle (USV) fleet, a hunting algorithm based on multi-agent reinforcement learning is proposed. Firstly, the hunting environment and kinematic model without boundary constraints are built, and the criteria for successful target capture are given. Then, the cooperative hunting problem of a USV fleet is modeled as a decentralized partially observable Markov decision process (Dec-POMDP), and a distributed partially observable multi-target hunting Proximal Policy Optimization (DPOMH-PPO) algorithm applicable to USVs is proposed. In addition, an observation model, a reward function and the action space applicable to multi-target hunting tasks are designed. To deal with the dynamic change of observational feature dimension input by partially observable systems, a feature embedding block is proposed. By combining the two feature compression methods of column-wise max pooling (CMP) and column-wise average-pooling (CAP), observational feature encoding is established. Finally, the centralized training and decentralized execution framework is adopted to complete the training of hunting strategy. Each USV in the fleet shares the same policy and perform actions independently. Simulation experiments have verified the effectiveness of the DPOMH-PPO algorithm in the test scenarios with different numbers of USVs. Moreover, the advantages of the proposed model are comprehensively analyzed from the aspects of algorithm performance, migration effect in task scenarios and self-organization capability after being damaged, the potential deployment and application of DPOMH-PPO in the real environment is verified.

Patterns in the Public Square: Reference Models for Regulatory Science.

Science and engineering involve discovery, representation, explanation, and exploitation of recurrent patterns, observed as phenomena. Model-based representations describe not only natural phenomena and engineered products, but also the socio-technical systems of systems that carry out scientific study, product engineering, medical practice, public health, commerce, and regulation. The term “Regulatory Science” invites us to represent and understand innovation, regulation and their intended and actual consequences as observable system phenomena in their own right, using scientific and engineering principles, tools, and insights. This article summarizes three classes of model-based reference patterns central to representing, understanding, communicating, and enhancing systems of innovation, regulation, and improvement over life cycles. In order of increasing scale, these pattern classes are (1) the domain-independent pattern of model-based representation of system phenomena (the S*Metamodel) in the sciences and engineering disciplines, underlying all modeling and simulation; (2) domain-specific patterns representing families of natural systems and engineered products in their life cycle contexts; and (3) the large-scale Innovation Ecosystem Pattern, in which science, engineering, commerce, medicine, and regulation are performed, planned, and advanced—including sharing of managed models and data across ecosystems. All three are applied by the Model-Based Patterns Working Group of the International Council on Systems Engineering (INCOSE).

Optimal joint inspection and mission abort policy for a partially observable system

his paper presents a joint optimization of inspection and condition-based mission abort policy for a partially observable safety-critical system. The system deterioration is modeled by a hidden continuous-time Markov process. Condition monitoring can only reveal partial information about the hidden state of the system. A joint inspection and mission abort policy is developed by using multi-variate Bayesian control approach. The posterior probability of the system being in warning state is updated at each inspection by Bayes’ rule. The mission is aborted and the rescue procedure is initiated once the posterior probability exceeds the control limit. The problem is formulated and solved within the framework of Markov decision process which aims to minimize the expected total cost including inspection, mission failure and system failure cost. Some structural properties of the control limit are presented. Finally, numerical examples are provided to demonstrate the model.

A condition-based prognostic approach for age- and state-dependent partially observable nonlinear degrading system

To deal with age- and state-dependent degradation, unit-to-unit variability and partially observable or hidden degradation in remaining useful life prediction jointly, this paper proposes a condition-based prognostic approach for age- and state-dependent partially observable nonlinear degrading system. The dynamics and nonlinearity of the system degradation process are described by the age- and state-dependent nonlinear diffusion process. The nonlinear relationship between observations and the hidden degradation state is characterized by a state space model. To derive the distribution of the remaining useful life, we apply the extended Kalman filtering and expectation-maximization algorithm to adaptively estimate the degradation states and the unknown model parameters. Based on the estimated degradation states and model parameters, we derive the approximately analytical distribution of remaining useful life in the concept of the first hitting time. Furthermore, the distribution of remaining useful life can be updated according to the newly available data, thereby realizing real-time remaining useful life estimation. An illustrative example is given to explain the application of the proposed approach in the specific age- and state-dependent nonlinear degradation model. Finally, a numerical example and a case study for bearing degradation data are presented to verify the accuracy and effectiveness of the proposed model.

71 Agent-Based Modeling: A Historical Perspective and Comparison to Other Modeling Techniques

Abstract The earliest known history of agent-based modeling (ABM) is traced back to Von Neumann’s self-reproducing cellular automata, designed in the 1940s. The usage of ABM in different scientific fields accelerated in the 1990s, supported by the phenomenal increase in computing capabilities. ABM is gaining acceptance in animal systems, as daily decision-making is becoming complex due to multiple competing outcomes of interest to food systems stakeholders. ABM typically replicates complex real-world systems, for example, a herd of animals that dynamically interact based on simple rules. ABM simulate the heterogenous, stochastic characteristics of agents observed in the real-world. The dynamic interaction of agents replicates the observable real-world complex system patterns. Application of ABM in animal systems ranges from modeling cell behavior, precision nutrition and herd management to predicting the spread of epidemics, and food animal supply chain optimization. Animal science's most widely used alternative modeling techniques include system dynamic models, differential equations-based models, and statistical modeling. The list of unrealistic assumptions that limit the utility of these modeling techniques includes the assumptions of linearity, homogeneity, normality, and stationarity. The advantageous characteristics of an agent in ABM that set it apart from other techniques include being identifiable, capable of existing in an environment where it interacts with other agents while being autonomous and self-directed, goal-oriented behavior, flexible learning, and capability of adaptations in its behavior over time, based on experience. Identifying the purpose of the model, the questions the model will answer, and the potential users are the key decision variables modelers should ponder upon before embarking on building ABM. The commonly used ABM software includes Repast, Swarm, Netlogo, and MASON. Developing an ABM has several highly interleaved stages: concept development, requirements definition, design, implementation, and operationalization, each of which will be illustrated during the hands-on training session on ABM.

Sequential Measurements, Topological Quantum Field Theories, and Topological Quantum Neural Networks

AbstractWe introduce novel methods for implementing generic quantum information within a scale‐free architecture. For a given observable system, we show how observational outcomes are taken to be finite bit strings induced by measurement operators derived from a holographic screen bounding the system. In this framework, measurements of identified systems with respect to defined reference frames are represented by semantically‐regulated information flows through distributed systems of finite sets of binary‐valued Barwise‐Seligman classifiers. Specifically, we construct a functor from the category of cone‐cocone diagrams (CCCDs) over finite sets of classifiers, to the category of finite cobordisms of Hilbert spaces. We show that finite CCCDs provide a generic representation of finite quantum reference frames (QRFs). Hence the constructed functor shows how sequential finite measurements can induce TQFTs. The only requirement is that each measurement in a sequence, by itself, satisfies Bayesian coherence, hence that the probabilities it assigns satisfy the Kolmogorov axioms. We extend the analysis too develop topological quantum neural networks (TQNNs), which enable machine learning with functorial evolution of quantum neural 2‐complexes (TQN2Cs) governed by TQFTs amplitudes, and resort to the Atiyah‐Singer theorems in order to classify topological data processed by TQN2Cs. We then comment about the quiver representation of CCCDs and generalized spin‐networks, a basis of the Hilbert spaces of both TQNNs and TQFTs. We finally review potential implementations of this framework in solid state physics and suggest applications to quantum simulation and biological information processing.

Observer Synthesis for Uncertain Nonlinear Systems With Nonuniformly Sampled and Delayed Output

This article presents a novel observer synthesis for nonuniform observable uncertain nonlinear systems with sampled (constant and non-uniform intervals) and disturbed delayed output. An observer with a cascade structure is proposed in order to obtain (in a continuous way) the estimated free-delay state vector. The base of the cascade structure is a continuous-discrete observer that achieve the continuous estimation of the delayed state, which reduces the convergence time and allows to reduce the remaining subsystems in the cascade. The rest of the cascade structure is composed by predictors that compensate the delay induced by the original signal. The performance of the observer is evaluated by estimating the state of an uncertain chaotic system with delayed measurements by considering different scenarios.

Deep reinforcement learning for gearshift controllers in automatic transmissions

Control design for gearshifts in modern automotive automatic transmissions constitutes a challenging, time consuming task performed by highly trained experts. This is due to the fact that a variety of non-linear and partially observable systems need to be actuated, such that a comfortable shifting behavior is achieved within an sufficiently low shifting time. The presented approach leverages deep reinforcement learning (DRL) to control gear shifts, outperforming current state of the art controller performance. This requires formulating the shifting task as a Markov decision process by designing suitable action and observation spaces as well as a meaningful reward function. Due to the sample complexity of DRL methods, the control agents are trained in simulation and are subsequently transferred to a real transmission on a test bench. To successfully transfer DRL agents from simulation to reality, methods such as domain randomization and domain adaption leveraging evolutionary optimization are applied. To the best of the authors’ knowledge, this work is the first to successfully apply DRL for the closed loop control of a real world automotive automatic transmission of realistic complexity.

Analytical and Microsimulation Model Calibration and Validation: Application to Roundabouts under Sight-Restricted Conditions

Analytical models and traffic microsimulation are two widely used platforms for evaluating roundabout operations. The application of the correct inputs and proper specification of calibration parameters should precede the actual simulation, to replicate field traffic conditions. In this sense, simultaneous data collection and estimation of the input, calibration, and validation variables, along with knowledge of their definitions, are crucial. Although simultaneity of data gathering is virtually guaranteed with the use of wide-frame videos captured with an unmanned aerial system (UAS), there are cases where sight distance restrictions may obscure observations of the back of queue and arrival patterns. This paper explores the calibration and validation efforts associated with an analytical platform, SIDRA 9, and a microsimulation model, TransModeler 5, conducted under sight-restricted conditions. Video captured from a drone, followed by trajectory extraction using video processing software, was used to analyze operations on two approaches at a single-lane roundabout. In the process, the team employed a specialized demand estimation method, and developed a novel data collection scheme for estimating the critical headway distribution in TransModeler 5. Because of sight distance constraints, the model validation was limited to the use of the observable system travel time and associated travel speed within the field of view. The comparison results, for both platforms, have confirmed the value of model calibration in more accurately describing field performance. The calibrated models performed differently between the two approaches, with the approach having a larger presence of buses and heavy vehicles yielding slightly poorer results.

Adaptive improved super-twisting integral sliding mode guidance law against maneuvering target with terminal angle constraint

This paper investigates the maneuvering target interception problem considering terminal angle constraint, the input saturation and the second-order autopilot lag with the unavailable missile's acceleration. The considering problem is formulated as an output feedback control problem of a four-order nonlinear uncertain weakly observable system, which is challenging to address by general composite methods directly. Hence, the liner normal equivalent control model of the original guidance system is firstly derived to facilitate the extended state observer (ESO) based method design. Then, a nonlinear ESO is introduced to estimate the unmeasured system states and modeled disturbances as well as compensate for control saturation. Thereafter, the ESO-based guidance scheme is developed by integrating a novel four-order integral sliding mode manifold and an adaptive improved super-twisting algorithm. The main novelty is that a novel integral sliding mode manifold is proposed to replace the linear sliding mode manifold of super-twisting sliding mode to further reduce the convergent time of closed-loop system states. Besides, the conventional super-twisting algorithm is improved by modifying the reaching law and designing the adaptive gains to guarantee the continuity of guidance command and suppress the large constant gains and parameter uncertainties. Notably, only the output relative information is utilized in its design process. Numerical simulations validate the previous analyses and show the superiorities in reducing the energy consumption in high maneuvering target interception and achieving robustness to a certain margin under various uncertainties.

Traffic signal optimization for partially observable traffic system and low penetration rate of connected vehicles

AbstractObservability and controllability are two critical requirements for a partially observable transportation system. This paper proposes a stepwise signal optimization framework with connected vehicle (CV) data as input to solve both challenges. First, a Bayesian deduction method based on low‐penetration CV data is established to estimate the traffic volume. Thereafter, an offline signal optimization model is constructed to simultaneously optimize the flexible lane settings and signal timings, which are set as the prior information for the third step. In the third step, an online deep recurrent Q‐learning (DRQN) signal optimization model dynamically adjusts signal settings based on real‐time traffic information. Numerical experiments demonstrate that the model outperforms the actuated control, the online DQRN model without offline filter, and the back‐pressure model by 9%–66% and 7%–29% in two networks. This study innovatively combines traffic state estimation and traffic signal control as an integrated process. It contributes to an improved understanding of traffic control in a CV environment and lays a solid foundation for future traffic control strategies.

Markov Decision Processes with Incomplete Information and Semiuniform Feller Transition Probabilities

This paper deals with control of partially observable discrete-time stochastic systems. It introduces and studies the class of Markov Decision Processes with Incomplete information and with semi-uniform Feller transition probabilities. The important feature of this class of models is that the classic reduction of such a model with incomplete observation to the completely observable Markov Decision Process with belief states preserves semi-uniform Feller continuity of transition probabilities. Under mild assumptions on cost functions, optimal policies exist, optimality equations hold, and value iterations converge to optimal values for this class of models. In particular, for Partially Observable Markov Decision Processes the results of this paper imply new and generalize several known sufficient conditions on transition and observation probabilities for the existence of optimal policies, validity of optimality equations, and convergence of value iterations.

Development of the Flourishing Classroom System Observation Framework and Rubric: a Delphi Study

This paper describes the development of the Flourishing Classroom System Observation Framework and Rubric, which provides a framework and practical approach to defining and describing multiple interconnected observable characteristics of a classroom system that individually and together can be targeted to cultivate collective flourishing within schools. Beginning with a working theoretical model based on existing literature, a three-round Delphi study was used to develop the framework and related rubric. In round 1, 35 experts answered open-ended questions regarding observable behaviours of collective wellbeing in the classroom. Analysis of responses resulted in a framework with five dimensions, each with three sub-dimensions. In round 2, 23 experts sorted and categorised statements that potentially described each of the 15 sub-dimensions. Analyses created definitions of each sub-dimension and statements describing how they manifest for teachers, students, and the class, along with a description of the learning environment for each dimension. In round 3, 18 experts reviewed the definitions and descriptions, resulting in a final 15-dimension framework with a related rubric of 45 descriptive statements. The resulting framework and rubric provide an organising structure to identify observable system elements that shape a flourishing classroom culture.

Optimal extended state observer based control for vibration reduction on a flexible cantilever beam with using air thrust actuator

The idea of using air thrusts for vibration control of structures has gained attention since the 1990s. However, only discontinuous controllers and switching functions have been studied with these actuators. Therefore, the main motivation of this study is to investigate air thrusters with continuous control methods for vibration reduction on a flexible cantilever beam. First, the finite element model of the flexible cantilever beam is defined. The order of the model is reduced according to the effective mass participation factor using the mode displacement method to obtain a controllable and observable system. The fundamental dynamics of the air thrust actuator, such as time constant, delay time and bandwidth, have been determined by system identification and considered in the control system. In this study, optimal extended state observer-based control (optimal ESOBC) tuned by optimal control law and particle swarm optimization (PSO) is proposed and compared to linear quadratic Gaussian (LQG) and positive position feedback (PPF) controllers. The performance of the controllers has been evaluated under several experiments such as initial displacement, impulsive disturbance, structure uncertainty and actuator uncertainty. The effectiveness of the controllers is determined by the RMS value of the beam displacement and the total air consumption. The results show that the ESOBC controlled air thrust actuator has suppressed beam vibrations more successfully than LQG and PPF controllers. In addition, the proposed method is found to be effective for low-frequency vibration suppression on flexible structures.

Stochastic optimal well control in subsurface reservoirs using reinforcement learning

We present a case study of model-free reinforcement learning (RL) framework to solve stochastic optimal control for a predefined parameter uncertainty distribution and partially observable system. We focus on robust optimal well control problem which is a subject of intensive research activities in the field of subsurface reservoir management. For this problem, the system is partially observed since the data is only available at well locations. Furthermore, the model parameters are highly uncertain due to sparsity of available field data. In principle, RL algorithms are capable of learning optimal action policies – a map from states to actions – to maximize a numerical reward signal. In deep RL, this mapping from state to action is parameterized using a deep neural network. In the RL formulation of the robust optimal well control problem, the states are represented by saturation and pressure values at well locations while the actions represent the valve openings controlling the flow through wells. The numerical reward refers to the total sweep efficiency and the uncertain model parameter is the subsurface permeability field. The model parameter uncertainties are handled by introducing a domain randomization scheme that exploits cluster analysis on its uncertainty distribution. We present numerical results using two state-of-the-art RL algorithms, proximal policy optimization (PPO) and advantage actor–critic (A2C), on two subsurface flow test cases representing two distinct uncertainty distributions of permeability field. The results were benchmarked against optimization results obtained using differential evolution algorithm. Furthermore, we demonstrate the robustness of the proposed use of RL by evaluating the learned control policy on unseen samples drawn from the parameter uncertainty distribution that were not used during the training process.

Information exchange, meaning and redundancy generation in anticipatory systems: self-organization of expectations – the case of Covid-19

The paper focuses on the link between the dynamics of information and system evolution. Information, exchanged between different system’s parts, before being processed is first provided with meaning by the system. Meanings are generated from the perspective of hindsight, i.e. against the arrow of time. The same information can be differently interpreted by different system’s parts (i.e. provided with different meanings) so that the number of options for possible system development is proliferated. Some options eventually turn into observable system states. So that system evolutionary dynamics can be considered as due to information processing within the system. The model under study is Triple Helix (TH) model, conceptualized in the framework of three compartment model used to describe infectious diseases. The paper demonstrates how the dynamics of information and meaning can be incorporated in the description of Covid-19 infection propagation. The results show correspondence of model predictions with observable infection dynamics.

A predictive Markov decision process for optimizing inspection and maintenance strategies of partially observable multi-state systems

Optimizing both inspection and maintenance strategies for multi-state systems is challenging, especially when the inspected conditions contain uncertainties. One classic approach for addressing this issue is the Partially Observable Markov Decision Process (POMDP). However, the POMDP often considers the system is periodically inspected, resulting in a waste of inspection resources (cost and manpower) in the early stage of the system. To predictively optimize the inspection strategies, we formulate a new model-Predictive Markov Decision Process (PMDP). It extends the POMDP by embedding the Forward algorithm for inspection timing prediction and the Baum–Welch algorithm for model parameters estimation. Therefore, it could harvest the inspection information for predicting the successive inspection timing in an online updating scheme based on the new observation. In this manner, maintenance actions can take place at the predicted inspection timing that reduces unnecessary inspections. The PMDP manifests the power of predictive maintenance. As illustrated by the case study, the PMDP outperforms the POMDP under routine inspection by saving 26.3% of the cost on average.

Interdependence Analysis in collaborative robot applications from a joint cognitive functional perspective

This paper sets up a framework to assess co-agency in human-robot interactions, and applies it specifically to the socio-technical safety analysis of collaborative robots. We also examine to what extent the concept of Situation Awareness can be applied to assess collaborative robots as efficient team members in socio-technical systems. We explain some theoretical concerns with traditional concepts of Situation Awareness and defend why the concept of Joint Cognitive Systems, which maps the conceptualization of the cognitive system onto the work system as a whole, is best suited for issues of distributed cognition and controllability in human-robot interaction. Thereafter we present a five-step methodology specifically conceived for cobot applications serving the aim of goal coordination between multiple agents by functional interactions. The proposed framework merges two existing safety and resilience analysis methods, being the Functional Resonance Analysis Method and Interdependence Analysis. These methods are used in combination to assess shared control in safe and efficient human-robot interaction from a systems-thinking perspective. This allows to describe the systemic conditions for Distributed Situation Awareness in terms of observable system interactions and as an emergent object of distributed cognition. Instead of looking at undesirable safety outcomes, we have imposed the focus of co-agency as the unit of analysis in line with the Joint Cognitive Systems perspective. The theoretical insights from this paper are additionally applied to a hypothetical but credible demonstration case study with collaborative warehouse robots.

Technical Note—Average Cost Optimality in Partially Observable Lost-Sales Inventory Systems

In many real-life situations, the inventory record may not match the actual stock perfectly. This can happen due to distortion of inventory data, such as transaction errors, misplaced inventories, and spoilage. In these cases, because the decision maker only has incomplete information about the inventory levels, many well-known inventory policies are not even admissible, and our understanding of the optimal policies, even their existence, is very limited. In “Average Cost Optimality in Partially Observable Lost-Sales Inventory Systems,” Bai et al. consider the classical lost-sales inventory model, in which the inventory level is only observed when it becomes zero. They formulate the cost-minimization problem as a partially observable Markov decision process. By exploiting the vanishing discount factor approach, they provide a way to verify the existence of optimal policies under the average cost criterion. The key step in their analysis is the construction of a valid policy, which, in a certain sense, copies the actions of another policy for the process starting from another initial state.

Active opacity of discrete-event systems

As an essential attribute of information flow, opacity is used to characterise whether or not the intruders can infer the system secrets, which has caused considerable concern. For a complex system with multiple secret states, it is difficult to ensure the system opacity, which makes it necessary to actively control the system to enforce their opacity. In this regard, based on the framework of partially observable and controllable discrete event system, the active current-state opacity is studied. The system being actively opaque means that we can find a control for each secret state such that the behaviour of system reaching the secret state is opaque. Under such control, the intruder is unable to confirm the current state as secret state through limited observation. We construct an active-opacity verifier and derive a necessary and sufficient condition to check the active opacity of the system.