Value Function Research Articles

Intelligent Interference Waveform Design for Radar Detection Based on the Cross-Correlation Value Function

Intelligent Interference Waveform Design for Radar Detection Based on the Cross-Correlation Value Function

Controlled Reflected McKean–Vlasov SDEs and Neumann Problem for Backward SPDEs

This paper is concerned with the stochastic optimal control problem of a 1-dimensional McKean–Vlasov stochastic differential equation (SDE) with reflection, of which the drift coefficient and diffusion coefficient can be both dependent on the state of the solution process along with its law and control. One backward stochastic partial differential equation (BSPDE) with the Neumann boundary condition can represent the value function of this control problem. Existence and uniqueness of the solution to the above equation are obtained. Finally, the optimal feedback control can be constructed by the BSPDE.

Research on Behavioral Decision at an Unsignalized Roundabout for Automatic Driving Based on Proximal Policy Optimization Algorithm

Unsignalized roundabouts have a significant impact on traffic flow and vehicle safety. To address the challenge of autonomous vehicles passing through roundabouts with low penetration, improve their efficiency, and ensure safety and stability, we propose the proximal policy optimization (PPO) algorithm to enhance decision-making behavior. We develop an optimization-based behavioral choice model for autonomous vehicles that incorporates gap acceptance theory and deep reinforcement learning using the PPO algorithm. Additionally, we employ the CoordConv network to establish an aerial view for spatial perception information gathering. Furthermore, a dynamic multi-objective reward mechanism is introduced to maximize the PPO algorithm’s reward pool function while quantifying environmental factors. Through simulation experiments, we demonstrate that our optimized PPO algorithm significantly improves training efficiency by enhancing the reward value function by 2.85%, 7.17%, and 19.58% in scenarios with 20, 100, and 200 social vehicles, respectively, compared to the PPO+CCMR algorithm. The effectiveness of simulation training also increases by 11.1%, 13.8%, and 7.4%. Moreover, there is a reduction in crossing time by 2.37%, 2.62%, and 13.96%. Our optimized PPO algorithm enhances path selection during autonomous vehicle simulation training as they tend to drive in the inner ring over time; however, the influence of social vehicles on path selection diminishes as their quantity increases. The safety of autonomous vehicles remains largely unaffected by our optimized PPO algorithm.

Heuristic Expanding Disconnected Graph: A Rapid Path Planning Method for Mobile Robots

Existing mobile robots mostly use graph search algorithms for path planning, which suffer from relatively low planning efficiency owing to high redundancy and large computational complexity. Due to the limitations of the neighborhood search strategy, the robots could hardly obtain the most optimal global path. A global path planning algorithm, denoted as EDG*, is proposed by expanding nodes using a well-designed expanding disconnected graph operator (EDG) in this paper. Firstly, all obstacles are marked and their corners are located through the map pre-processing. Then, the EDG operator is designed to find points in non-obstruction areas to complete the rapid expansion of disconnected nodes. Finally, the EDG* heuristic iterative algorithm is proposed. It selects the candidate node through a specific valuation function and realizes the node expansion while avoiding collision with a minimum offset. Path planning experiments were conducted in a typical indoor environment and on the public dataset CSM. The result shows that the proposed EDG* reduced the planning time by more than 90% and total length of paths reduced by more than 4.6%. Compared to A*, Dijkstra and JPS, EDG* does not show an exponential explosion effect in map size. The EDG* showed better performance in terms of path smoothness, and collision avoidance. This shows that the EDG* algorithm proposed in this paper can improve the efficiency of path planning and enhance path quality.

Constrained optimal stopping under a regime-switching model

Abstract We investigate an optimal stopping problem for the expected value of a discounted payoff on a regime-switching geometric Brownian motion under two constraints on the possible stopping times: only at exogenous random times, and only during a specific regime. The main objectives are to show that an optimal stopping time exists as a threshold type and to derive expressions for the value functions and the optimal threshold. To this end, we solve the corresponding variational inequality and show that its solution coincides with the value functions. Some numerical results are also introduced. Furthermore, we investigate some asymptotic behaviors.

Probabilistic linguistic prospect outranking risk decision making method based on stochastic dominance and application in emergency plan evaluation

In the decision-making process, probabilistic linguistic term sets (PLTSs) could represent decision-makers’ uncertain evaluation information. The combination of probabilistic and linguistic information assists decision-makers to depict their evaluation information more flexibly. In this paper, a prospect outranking decision-making method based on stochastic dominance is proposed for solving the multi-attribute risk decision-making problem under the probabilistic linguistic information environment. Firstly, to fully preserve the original probabilistic linguistic information, we give the definitions of the generalized probability distribution and the corresponding generalized cumulative distribution function of a PLTS for defining the distance measure between PLTSs. Secondly, based on the positive and negative ideal cumulative distribution functions, we propose the probabilistic linguistic prospect value functions. After that, considering the uncertainty of the relationship between alternatives, several kinds of novel prospect stochastic dominance and almost stochastic dominance relations are defined, which consider the decision makers’ distinction between gains and losses to measure the subtle relationship between alternatives. The stochastic dominance approach and the almost stochastic dominance approach under the probabilistic linguistic decision-making environment are put forward. Besides, we use the superiority and inferiority ranking method to solve the dominance ranking problem of the uncertainty relationship between any two alternatives. Finally, the proposed method is applied in the evaluation of emergency plans to show the accuracy and reliability, and comparative analyses are made to illustrate its availability.

Prognostic value and potential function of a novel heme-related LncRNAs signature in gastric cancer

Heme is a coordination complex formed by the binding of iron ions and porphyrin rings. Its metabolic processes are associated with various cancers, including gastric cancer (GC). In recent years, long non-coding RNAs (LncRNAs) have been identified as key regulatory factors in GC. However, the role of LncRNAs associated with heme metabolism in GC and their relationship with prognosis have not been reported. In this study, we constructed a novel LncRNAs signature related to heme metabolism (HMlncSig) and validated its prognostic value for predicting the survival of GC patients through training, test, and entire cohorts. Kaplan-Meier analysis demonstrated that patients in the high-risk group had shorter survival times. Univariate and multivariate Cox regression analysis showed that HMlncSig was an independent prognostic indicator for GC patients, regardless of other clinical pathological features. Gene ontology and Kyoto Encyclopedia of Genes and Genomes enrichment analysis and gene set variation analysis pathways showed that the activation of these markers may be involved in tumor progression, influencing the survival of GC patients. The nomogram, based on HMlncSig score and clinical features, demonstrated the strong predictive ability of this signature. Additionally, significant differences were observed between the high-risk and low-risk groups in terms of immune cell subtypes, expression of immune checkpoint genes, and response to chemotherapy and immunotherapy. Through clinical validation, we found that the risk score and heme levels of GC patients were both significantly elevated and correlated with the degree of malignancy. Furthermore, we found that AP000692.1, a key gene in this signature, promoted the proliferation, migration, and invasion of GC cells. In conclusion, our HMlncSig model has significant predictive value for the prognosis of GC patients and can provide clinical guidance for personalized immunotherapy.

The motivation and engagement of student volunteers in volunteerism at the University of Gondar

This study examines student volunteering at the University of Gondar in Ethiopia, aiming to understand motivation and areas of engagement. Using a mixed research method, data from surveys, in-depth interviews, and key informant interviews were analyzed. The sample included 422 participants selected through probability and non-probability sampling techniques. Quantitative findings reveal that understanding the functions of volunteering motivates students, with a desire for learning experiences and applying their skills being primary motivators. The value function and career enhancement function also contribute to motivation. Survey results indicate that students engage in volunteer activities to enhance their resumes and have a strong belief in the importance of helping others. Notably, 47% of volunteer students focus on social and disability affairs. This study emphasizes the need for careful planning and implementation to encourage student volunteerism and maximize its benefits for all involved parties.

Accelerated Value Iteration for Nonlinear Zero-Sum Games with Convergence Guarantee

In this paper, an accelerated value iteration (VI) algorithm is established to solve the zero-sum game problem with convergence guarantee. First, inspired by the successive over relaxation theory, the convergence rate of the iterative value function sequence is accelerated significantly with the relaxation factor. Second, the convergence and monotonicity of the value function sequence are analyzed under different ranges of the relaxation factor. Third, two practical approaches, namely the integrated scheme and the relaxation function, are introduced into the accelerated VI algorithm to guarantee the convergence of the iterative value function sequence for zero-sum games. The integrated scheme consists of the accelerated stage and the convergence stage, and the relaxation function can adjust the value of the relaxation factor. Finally, including the autopilot controller, the fantastic performance of the accelerated VI algorithm is verified through two examples with practical physical backgrounds.

From Past to Future: Rethinking Eligibility Traces

In this paper, we introduce a fresh perspective on the challenges of credit assignment and policy evaluation. First, we delve into the nuances of eligibility traces and explore instances where their updates may result in unexpected credit assignment to preceding states. From this investigation emerges the concept of a novel value function, which we refer to as the ????????????? ????? ????????. Unlike traditional state value functions, bidirectional value functions account for both future expected returns (rewards anticipated from the current state onward) and past expected returns (cumulative rewards from the episode's start to the present). We derive principled update equations to learn this value function and, through experimentation, demonstrate its efficacy in enhancing the process of policy evaluation. In particular, our results indicate that the proposed learning approach can, in certain challenging contexts, perform policy evaluation more rapidly than TD(λ)–a method that learns forward value functions, v^π, ????????. Overall, our findings present a new perspective on eligibility traces and potential advantages associated with the novel value function it inspires, especially for policy evaluation.

Symmetric Q-learning: Reducing Skewness of Bellman Error in Online Reinforcement Learning

In deep reinforcement learning, estimating the value function to evaluate the quality of states and actions is essential. The value function is often trained using the least squares method, which implicitly assumes a Gaussian error distribution. However, a recent study suggested that the error distribution for training the value function is often skewed because of the properties of the Bellman operator, and violates the implicit assumption of normal error distribution in the least squares method. To address this, we proposed a method called Symmetric Q-learning, in which the synthetic noise generated from a zero-mean distribution is added to the target values to generate a Gaussian error distribution. We evaluated the proposed method on continuous control benchmark tasks in MuJoCo. It improved the sample efficiency of a state-of-the-art reinforcement learning method by reducing the skewness of the error distribution.

Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge

The problem of sample complexity of online reinforcement learning is often studied in the literature without taking into account any partial knowledge about the system dynamics that could potentially accelerate the learning process. In this paper, we study the sample complexity of online Q-learning methods when some prior knowledge about the dynamics is available or can be learned efficiently. We focus on systems that evolve according to an additive disturbance model of the form S_{h+1} = ƒ(S_h, A_h) + W_h, where ƒ represents the underlying system dynamics, and W_h are unknown disturbances independent of states and actions. In the setting of finite episodic Markov decision processes with S states, A actions, and episode length H, we present an optimistic Q-learning algorithm that achieves Õ(Poly(H)√T) regret under perfect knowledge of ƒ, where T is the total number of interactions with the system. This is in contrast to the typical Õ(Poly(H)√SAT) regret for existing Q-learning methods. Further, if only a noisy estimate ƒ_hat of ƒ is available, our method can learn an approximately optimal policy in a number of samples that is independent of the cardinalities of state and action spaces. The sub-optimality gap depends on the approximation error ƒ_hat − ƒ, as well as the Lipschitz constant of the corresponding optimal value function. Our approach does not require modeling of the transition probabilities and enjoys the same memory complexity as model-free methods.

Generalizable Policy Improvement via Reinforcement Sampling (Student Abstract)

Current policy gradient techniques excel in refining policies over sampled states but falter when generalizing to unseen states. To address this, we introduce Reinforcement Sampling (RS), a novel method leveraging a generalizable action value function to sample improved decisions. RS is able to improve the decision quality whenever the action value estimation is accurate. It works by improving the agent's decision on the fly on the states the agent is visiting. Compared with the historically experienced states in which conventional policy gradient methods improve the policy, the currently visited states are more relevant to the agent. Our method sufficiently exploits the generalizability of the value function on unseen states and sheds new light on the future development of generalizable reinforcement learning.

Discerning Temporal Difference Learning

Temporal difference learning (TD) is a foundational concept in reinforcement learning (RL), aimed at efficiently assessing a policy's value function. TD(λ), a potent variant, incorporates a memory trace to distribute the prediction error into the historical context. However, this approach often neglects the significance of historical states and the relative importance of propagating the TD error, influenced by challenges such as visitation imbalance or outcome noise. To address this, we propose a novel TD algorithm named discerning TD learning (DTD), which allows flexible emphasis functions—predetermined or adapted during training—to allocate efforts effectively across states. We establish the convergence properties of our method within a specific class of emphasis functions and showcase its promising potential for adaptation to deep RL contexts. Empirical results underscore that employing a judicious emphasis function not only improves value estimation but also expedites learning across diverse scenarios.

Reinforcement Learning and Data-Generation for Syntax-Guided Synthesis

Program synthesis is the task of automatically generating code based on a specification. In Syntax-Guided Synthesis (SyGuS) this specification is a combination of a syntactic template and a logical formula, and the result is guaranteed to satisfy both. We present a reinforcement-learning guided algorithm for SyGuS which uses Monte-Carlo Tree Search (MCTS) to search the space of candidate solutions. Our algorithm learns policy and value functions which, combined with the upper confidence bound for trees, allow it to balance exploration and exploitation. A common challenge in applying machine learning approaches to syntax-guided synthesis is the scarcity of training data. To address this, we present a method for automatically generating training data for SyGuS based on anti-unification of existing first-order satisfiability problems, which we use to train our MCTS policy. We implement and evaluate this setup and demonstrate that learned policy and value improve the synthesis performance over a baseline by over 26 percentage points in the training and testing sets. Our tool outperforms state-of-the-art tool cvc5 on the training set and performs comparably in terms of the total number of problems solved on the testing set (solving 23% of the benchmarks on which cvc5 fails). We make our data set publicly available, to enable further application of machine learning methods to the SyGuS problem.

Critic-Guided Decision Transformer for Offline Reinforcement Learning

Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Return-Conditioned Supervised Learning (RCSL), a paradigm that learns the action distribution based on target returns for each state in a supervised manner. However, prevailing RCSL methods largely focus on deterministic trajectory modeling, disregarding stochastic state transitions and the diversity of future trajectory distributions. A fundamental challenge arises from the inconsistency between the sampled returns within individual trajectories and the expected returns across multiple trajectories. Fortunately, value-based methods offer a solution by leveraging a value function to approximate the expected returns, thereby addressing the inconsistency effectively. Building upon these insights, we propose a novel approach, termed the Critic-Guided Decision Transformer (CGDT), which combines the predictability of long-term returns from value-based methods with the trajectory modeling capability of the Decision Transformer. By incorporating a learned value function, known as the critic, CGDT ensures a direct alignment between the specified target returns and the expected returns of actions. This integration bridges the gap between the deterministic nature of RCSL and the probabilistic characteristics of value-based methods. Empirical evaluations on stochastic environments and D4RL benchmark datasets demonstrate the superiority of CGDT over traditional RCSL methods. These results highlight the potential of CGDT to advance the state of the art in offline RL and extend the applicability of RCSL to a wide range of RL tasks.

Parameterized Projected Bellman Operator

Approximate value iteration (AVI) is a family of algorithms for reinforcement learning (RL) that aims to obtain an approximation of the optimal value function. Generally, AVI algorithms implement an iterated procedure where each step consists of (i) an application of the Bellman operator and (ii) a projection step into a considered function space. Notoriously, the Bellman operator leverages transition samples, which strongly determine its behavior, as uninformative samples can result in negligible updates or long detours, whose detrimental effects are further exacerbated by the computationally intensive projection step. To address these issues, we propose a novel alternative approach based on learning an approximate version of the Bellman operator rather than estimating it through samples as in AVI approaches. This way, we are able to (i) generalize across transition samples and (ii) avoid the computationally intensive projection step. For this reason, we call our novel operator projected Bellman operator (PBO). We formulate an optimization problem to learn PBO for generic sequential decision-making problems, and we theoretically analyze its properties in two representative classes of RL problems. Furthermore, we theoretically study our approach under the lens of AVI and devise algorithmic implementations to learn PBO in offline and online settings by leveraging neural network parameterizations. Finally, we empirically showcase the benefits of PBO w.r.t. the regular Bellman operator on several RL problems.

Physics-Informed Representation and Learning: Control and Risk Quantification

Optimal and safety-critical control are fundamental problems for stochastic systems, and are widely considered in real-world scenarios such as robotic manipulation and autonomous driving. In this paper, we consider the problem of efficiently finding optimal and safe control for high-dimensional systems. Specifically, we propose to use dimensionality reduction techniques from a comparison theorem for stochastic differential equations together with a generalizable physics-informed neural network to estimate the optimal value function and the safety probability of the system. The proposed framework results in substantial sample efficiency improvement compared to existing methods. We further develop an autoencoder-like neural network to automatically identify the low-dimensional features in the system to enhance the ease of design for system integration. We also provide experiments and quantitative analysis to validate the efficacy of the proposed method. Source code is available at https://github.com/jacobwang925/path-integral-PINN.

1/2-Approximate MMS Allocation for Separable Piecewise Linear Concave Valuations

We study fair distribution of a collection of m indivisible goods among a group of n agents, using the widely recognized fairness principles of Maximin Share (MMS) and Any Price Share (APS). These principles have undergone thorough investigation within the context of additive valuations. We explore these notions for valuations that extend beyond additivity. First, we study approximate MMS under the separable (piecewise-linear) concave (SPLC) valuations, an important class generalizing additive, where the best known factor was 1/3-MMS. We show that 1/2-MMS allocation exists and can be computed in polynomial time, significantly improving the state-of-the-art. We note that SPLC valuations introduce an elevated level of intricacy in contrast to additive. For instance, the MMS value of an agent can be as high as her value for the entire set of items. We use a relax-and-round paradigm that goes through competitive equilibrium and LP relaxation. Our result extends to give (symmetric) 1/2-APS, a stronger guarantee than MMS. APS is a stronger notion that generalizes MMS by allowing agents with arbitrary entitlements. We study the approximation of APS under submodular valuation functions. We design and analyze a simple greedy algorithm using concave extensions of submodular functions. We prove that the algorithm gives a 1/3-APS allocation which matches the best-known factor. Concave extensions are hard to compute in polynomial time and are, therefore, generally not used in approximation algorithms. Our approach shows a way to utilize it within analysis (while bypassing its computation), and hence might be of independent interest.

Maximizing Nash Social Welfare under Two-Sided Preferences

The maximum Nash social welfare (NSW)---which maximizes the geometric mean of agents' utilities---is a fundamental solution concept with remarkable fairness and efficiency guarantees. The computational aspects of NSW have been extensively studied for *one-sided* preferences where a set of agents have preferences over a set of resources. Our work deviates from this trend and studies NSW maximization for *two-sided* preferences, wherein a set of workers and firms, each having a cardinal valuation function, are matched with each other. We provide a systematic study of the computational complexity of maximizing NSW for many-to-one matchings under two-sided preferences. Our main negative result is that maximizing NSW is NP-hard even in a highly restricted setting where each firm has capacity 2, all valuations are in the range {0,1,2}, and each agent positively values at most three other agents. In search of positive results, we develop approximation algorithms as well as parameterized algorithms in terms of natural parameters such as the number of workers, the number of firms, and the firms' capacities. We also provide algorithms for restricted domains such as symmetric binary valuations and bounded degree instances.