Dynamic Team Problems Research Articles

Optimal Policies for Convex Symmetric Stochastic Dynamic Teams and their Mean-Field Limit

This paper studies convex stochastic dynamic team problems with finite and infinite time horizons under decentralized information structures. First, we introduce two notions called exchangeable tea...

Dynamic Teams and Decentralized Control Problems With Substitutable Actions

This technical note considers two problems—a dynamic team problem and a decentralized control problem. The problems we consider do not belong to the known classes of “simpler” dynamic team/decentralized control problems such as partially nested or quadratically invariant problems. However, we show that our problems admit simple solutions under an assumption referred to as the substitutability assumption. Intuitively, substitutability in a team (resp. decentralized control) problem means that the effects of one team member's (resp. controller's) action on the cost function and the information (resp. state dynamics) can be achieved by an action of another member (resp. controller). For the non-partially-nested LQG dynamic team problem, it is shown that under certain conditions linear strategies are optimal. For the non-partially-nested decentralized LQG control problem, the state structure can be exploited to obtain optimal control strategies with recursively update-able sufficient statistics. These results suggest that substitutability can work as a counterpart of the information structure requirements that enable simplification of dynamic teams and decentralized control problems.

Finite Model Approximations and Asymptotic Optimality of Quantized Policies in Decentralized Stochastic Control

We consider finite model approximations of a large class of static and dynamic team problems where these models are constructed through uniform quantization of the observation and action spaces of the agents. The strategies obtained from these finite models are shown to approximate the optimal cost with arbitrary precision under mild technical assumptions. In particular, quantized team policies are asymptotically optimal. This result is then applied to Witsenhausen's celebrated counterexample and the Gaussian relay channel problem. For Witsenhausen's counterexample, our approximation approach provides, to our knowledge, the first rigorously established result that one can construct an $\varepsilon$ -optimal strategy for any $\varepsilon > 0$ through a solution of a simpler problem.

Convex Analysis in Decentralized Stochastic Control, Strategic Measures, and Optimal Solutions

This paper is concerned with the properties of the sets of strategic measures induced by admissible team policies in decentralized stochastic control and the convexity properties in dynamic team problems. To facilitate a convex analytical approach, strategic measures for team problems are introduced. Properties such as convexity, compactness, and Borel measurability under weak convergence topology are studied, and sufficient conditions for each of these properties are presented. These lead to existence of and structural results for optimal policies. It will be shown that the set of strategic measures for teams which are not classical is in general nonconvex, but the extreme points of a relaxed set consist of deterministic team policies, which lead to their optimality for a given team problem under an expected cost criterion. Externally provided independent common randomness for static teams or private randomness for dynamic teams do not improve the team performance. The problem of when a sequential team p...

Optimal Design of Sequential Real-Time Communication Systems

Optimal design of sequential real-time communication of a Markov source over a noisy channel is investigated. In such a system, the delay between the source output and its reconstruction at the receiver should equal a fixed prespecified amount. An optimal communication strategy must minimize the total expected symbol-by-symbol distortion between the source output and its reconstruction. Design techniques or performance bounds for such real-time communication systems are unknown. In this paper a systematic methodology, based on the concepts of information structures and information states, to search for an optimal real-time communication strategy is presented. This methodology trades off complexity in communication length (linear in contrast to doubly exponential) with complexity in alphabet sizes (doubly exponential in contrast to exponential). As the communication length is usually order of magnitudes bigger than the alphabet sizes, the proposed methodology simplifies the search for an optimal communication strategy. In spite of this simplification, the resultant optimality equations cannot be solved efficiently using existing algorithmic techniques. The main idea is to formulate a zero-delay communication problem as a dynamic team with nonclassical information structure. Then, an appropriate choice of information states converts the dynamic team problem into a centralized stochastic control problem in function space. Thereafter, Markov decision theory is used to derive nested optimality equations for choosing an optimal design. For infinite horizon problems, these optimality equations give rise to a fixed point functional equation. Communication systems with fixed finite delay constraint, a higher-order Markov source, and channels with memory are treated in the same manner after an appropriate expansion of the state space. Thus, this paper presents a comprehensive methodology to study different variations of real-time communication.

Multiuser rate-based flow control

Flow and congestion control allow the users of a telecommunication network to regulate the traffic that they send into the network in accordance with the quality of service that they require. Flow control may be performed by the network, as is the case in asynchronous transfer mode (ATM) networks (the available bit rate (ABR) transfer capacity), or by the users themselves, as is the case in the Internet [transmission control protocol/Internet protocol (TCP/IP)]. We study both situations using optimal control and dynamic game techniques. The first situation leads to the formulation of a dynamic team problem, while the second one leads to a dynamic noncooperative game, for which we establish the existence and uniqueness of a linear Nash equilibrium and obtain a characterization of the corresponding equilibrium policies along with the performance costs. We further show that when the users update their policies in a greedy manner, not knowing a priori the utilities of the other players, the sequence of policies thus generated converges to the Nash equilibrium. Finally, we study an extension of the model that accommodates multiple traffic types for each user, with the switching from one type of traffic to another being governed by a Markov jump process. Presentation of some numerical results complements this study.

Solutions to a class of linear-quadratic-Gaussian (LQG) stochastic team problems with nonclassical information

We consider a stochastic dynamic team problem with two controllers and nonclassical information, which can be viewed as the transmission of a garbled version of a Gaussian message over a number of noisy channels under a fidelity criterion. We show that the optimum solution (under a quadratic loss functional) consists of linearly transforming the garbled message to a certain (optimum) power level P ∗ and then optimally decoding it by using a linear transformation at the receiving end. The power level P ∗ is determined by the solution of a fifth order algebraic equation. The paper also discusses an extension of this result to the case when the channel noise is correlated with the input random variable, and shows that for the single channel case the optimum solution is again linear.

A decentralized closed-loop solution to the routing problem in networks

Several approaches have been attempted in the literature dealing with the problem of designing a routing protocol in a packet-switched computer communication network. However, most of the literature considers the problem only from a static optimization viewpoint, that is, it aims at determining routing policies where decisions are preordered and are not functions of the current state of the network. On the other hand, a control theoretic viewpoint would imply the determination of routing decisions at each control instant on the basis of a state feedback law (dynamic routing), and would be more responsive to variations in the network state. The first significant results in this direction are given by Moss and Segall, who introduced a state space model and developed a closed- loop optimal policy. The main characteristics of this approach are the following: i) a continuous-time linear time-invariant model of the network; ii) the practical lack of consideration of stochastic events; iii) the centralized nature of the solution. The third point represents a serious drawback for the practical implementation in a data network, requiring control information about the network state to traverse the network at a velocity much higher than that of proper message information. A decentralized control strategy, where the routing decisions are determined, at each instant, on the basis of local information at each node (with a moderate information exchange among nodes), seems to be more attractive in this respect. The approach of this paper defines a decentralized stochastic team control problem. On the basis of suitable assumptions on the information exchange mechanism among the various decision makers located at the nodes the information structure of the team is found to be partially nested; this fact allows the reduction of the dynamic team problem to a static one. The optimal control strategies are determined in a tabular form via the solution of a linear integer programming problem, whose complexity increases with the control horizon. Some possibilities are explored in order to reduce the computational complexity of the whole procedure: i) application of a receding horizon control scheme; ii) "partial" solution of the integer programming problem; iii) analysis of the conditions under which the optimal solution of the linear relaxed problem can be shown to be itself integer.

A multimodel approach to stochastic team problems

This paper develops decentralized team strategies for decision makers using different models of the same large-scale system. Multiparameter singular perturbations are employed to capture the multimodel nature of the fast dynamic subsystems interconnected through slow dynamic variables. The small parameters are appropriately scaled so that the variables in both time scales are well defined. The system considered is linear and the cost criterion is quadratic. First, a multimodel solution is obtained when the decision makers make different noisy linear observations of the random initial state only. Then the solution to this static team problem is utilized to obtain a multimodel solution to the dynamic team problem with sampled observations under the one-step-delay observation sharing pattern. In both cases, the well-posedness of the multimodel solution is demonstrated.

The dynamic linear exponential Gaussian team problem

The dynamic team problem for a linear system with Gaussian noise, exponential of a quadratic performance index, and one-step delayed sharing information pattern is considered. It is shown, via dynamic programming, that the multistage problem can be decomposed into a series of static team problems. Moreover, the optimal policy of the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</tex> th team member at time <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</tex> is an affine function of both the one-step predicted Kalman filter estimate and the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</tex> th team member's observation at time <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</tex> . Efficient algorithms are available for determining the gains of this affine controller. This model and solution are applied to an approximate resource allocation problem associated with a defense network, and a numerical example is discussed.

Decomposition of dynamic team decision problems

This paper considers the question when can a dynamic team problem be decomposed into several smaller dynamic team problems? A team is partitioned into several groups of members. Precedence relation a nd nestedness relation of information among these groups are defined. Then the concepts of independent partition and sequential partition of a team are introduced, which are partitions satisfying some conditions on their precedence relation, nestedness relation, and cost function. It is shown that if a team has an independent (sequential) partition, the problem can be decomposed into several independent (related) subproblems. It is also shown that the finest independent (sequential) partition is unique. In addition, decomposition of decentralized control problems is discussed.

Two-criteria LQG decision problems with one-step delay observation sharing pattern

This paper is concerned with the development of a general theory for the class of multistage linear\3-quadratic\3-Gaussian (LQG) decision problems characterized by (i) two decision makers (DM) each with a different objective functional to optimize, (ii) one-step delay observation sharing information pattern which provides each DM with the observation (but not the action) of the other DM with a one-step delay, (iii) a noncooperative equilibrium solution concept. In particular, it is proven that, under certain conditions, this class of optimization problems admit unique equilibrium strategies for each DM, which are linear in the information available. Moreover, exact expressions for those unique strategies are given in the paper. When specialized to the case of a single objective functional, the theory developed generalizes and unifies some of the results found in the literature on dynamic LQG team and zero-sum game problems.

動的チーム決定問題の分解

This paper considers the question when can a dynamic team problem be decomposed into several smaller dynamic team problems? A team is partitioned into several groups of members. Precedence relation a nd nestedness relation of information among these groups are defined. Then the concepts of independent partition and sequential partition of a team are introduced, which are partitions satisfying some conditions on their precedence relation, nestedness relation, and cost function. It is shown that if a team has an independent (sequential) partition, the problem can be decomposed into several independent (related) subproblems. It is also shown that the finest independent (sequential) partition is unique. In addition, decomposition of decentralized control problems is discussed.

Multi person decision analysis in large-scale hierarchical systems—team decision theory†

This paper represents a portion of a study of decision analysis in large-scale systems in which an effort is made to attack problem complexity by taking advantage of any inherent structure in the problem. By arranging the elements in the problem in an appropriate graphical form, usually a tree or hierarchy, it may often be possible to decompose the problem into a number of loosely coupled subproblems which can be solved independently and then coordinated in such a way as to provide a solution to the overall problem. We first survey team decision theory in which a multiplicity of decision makers, each of whom has access to different information, is attempting to extremize the same cost function. Under the conditions of a linear system with quadratic costs and Gaussian noises, the solution of a static team problem, in which no member's decision affects any other member's information, reduces to the solution of a set of linear algebraic equations. A large class of dynamic team problems which possess certain information structures is found to reduce to a static problem and hence can also be solved as a set of linear algebraic equations. Team problems can often be represented as cooperative stochastic differential games. Finally a technique is proposed for reducing the computational burden by decomposing the problem into a set of subproblems which can be solved relatively independently employing the techniques of hierarchical systems theory.

Dynamic team problems with subteam structure

Dynamic team problems with subteam structure