Value Function Research Articles

On Piecewise Cubic Estimates of the Value Function in a Target Control Problem for a Nonlinear System

On Piecewise Cubic Estimates of the Value Function in a Target Control Problem for a Nonlinear System

Improving the Performance of Learned Controllers in Behavior Trees Using Value Function Estimates at Switching Boundaries

Improving the Performance of Learned Controllers in Behavior Trees Using Value Function Estimates at Switching Boundaries

Event-Triggered Constrained Optimal Control for Organic Rankine Cycle Systems via Safe Reinforcement Learning.

The organic Rankine cycle (ORC) is an effective application for converting low-grade heat sources into power and is crucial for environmentally friendly production and energy recovery. However, the inherent complexity of the mechanism, its strong and unidentified nonlinearity, and the presence of control constraints severely impair the design of its optimal controller. To solve these issues, this study provides a novel event-triggered (ET) constrained optimal control approach for the ORC systems based on a safe reinforcement learning technique to find the optimal control law. Instead of employing the usual non-quadratic integral form to solve the control-limited optimal control problems, a constraint handling strategy based on a relaxed weighted barrier function (BF) technique is proposed. By adding the BF terms to the original value function, a modified value iteration algorithm is developed to make the control input solutions that tend to violate the constraints be pushed back and maintained in their safe sets. In addition, the ET mechanism proposed in this article is critically required for the ORC systems, and it can significantly reduce the computational load. The combination of these two techniques allows the ORC systems to achieve set-point tracking control and satisfy the control restrictions. The proposed approach is conducted based on a heuristic dynamic programming framework with three neural networks (NNs) involved. The safety and convergence of the proposed approach and the stability of the closed-loop system are analyzed. Simulation results and comparisons are presented to demonstrate its effectiveness.

Safe tracking in games: Achieving optimal control with unknown dynamics and constraints

AbstractThis paper introduces mix‐zero‐sum differential (MZSD) game theory to address multi‐player tracking systems, offering a better understanding of the coexistence of cooperation and competition among players. Within this framework, we present an optimal safety tracking control (OSTC) method, which incorporates a control barrier function (CBF) into the value function to ensure that the tracking error remains within a specified range, thus guaranteeing safety while achieving optimization. Simultaneously, to eliminate the need for system dynamics, we propose a novel approach leveraging off‐policy integral reinforcement learning (IRL) technology to obtain the Nash equilibrium solution of the MZSD games. We establish a unique critics–actors neural network (NN) structure that updates concurrently. Furthermore, we analyze stability and convergence using the Lyapunov method. We conduct two simulations to demonstrate the effectiveness of the proposed algorithm.

Dynamic Analysis, Image Encryption Application and FPGA Implementation of a Discrete Memristor-Coupled Neural Network

This paper presents a novel discrete memristor model that incorporates exponential and absolute value functions. A discrete coupled memristor neural network model is constructed based on this memristor design. The periodic and chaotic regions of the discrete neural network model are determined using bifurcation and Lyapunov exponent spectrum. Furthermore, by varying the initial values of the discrete memristive neural network, we observe the coexistence of chaos and periodic attractors, as well as periodic attractors. Additionally, an application to color image encryption based on the discrete system model is given. Security analysis is conducted in the aspects of key space, histogram analysis, correlation analysis, sensitivity analysis, Peak Signal-to-Noise Ratio (PSNR), and information entropy analysis. The analysis results show that the algorithm has a key space size of [Formula: see text], and the information entropy of baboon graph is 7.9993, which is very close to the ideal value of 8. It shows that the image encryption algorithm is feasible and effective. Finally, the implementation of the discrete memristive neural network model is realized using Field Programmable Gate Array (FPGA). The experimental implementation is conducted using the Verilog language on the Vivado 2018.3 platform, and the obtained results align with the numerical simulation results obtained through MATLAB.

A Reinforcement Learning-Based Pantograph Control Strategy for Improving Current Collection Quality in High-Speed Railways.

In high-speed railways, the pantograph-catenary system (PCS) is a critical subsystem of the train power supply system. In particular, when the double-PCS (DPCS) is in operation, the passing of the leading pantograph (LP) causes the contact force of the trailing pantograph (TP) to fluctuate violently, affecting the power collection quality of the electric multiple units (EMUs). The actively controlled pantograph is the most promising technique for reducing the pantograph-catenary contact force (PCCF) fluctuation and improving the current collection quality. Based on the Nash equilibrium framework, this study proposes a multiagent reinforcement learning (MARL) algorithm for active pantograph control called cooperative proximity policy optimization (Coo-PPO). In the algorithm implementation, the heterogeneous agents play a unique role in a cooperative environment guided by the global value function. Then, a novel reward propagation channel is proposed to reveal implicit associations between agents. Furthermore, a curriculum learning approach is adopted to strike a balance between reward maximization and rational movement patterns. An existing MARL algorithm and a traditional control strategy are compared in the same scenario to validate the proposed control strategy's performance. The experimental results show that the Coo-PPO algorithm obtains more rewards, significantly suppresses the fluctuation in PCCF (up to 41.55%), and dramatically decreases the TP's offline rate (up to 10.77%). This study adopts MARL technology for the first time to address the coordinated control of double pantographs in DPCS.

Implicit Posteriori Parameter Distribution Optimization in Reinforcement Learning.

Efficient and intelligent exploration remains a major challenge in the field of deep reinforcement learning (DRL). Bayesian inference with a distributional representation is usually an effective way to improve the exploration ability of the RL agent. However, when optimizing Bayesian neural networks (BNNs), most algorithms need to specify an explicit parameter distribution such as a multivariate Gaussian distribution. This may reduce the flexibility of model representation and affect the algorithm performance. Therefore, to improve sample efficiency and exploration based on Bayesian methods, we propose a novel implicit posteriori parameter distribution optimization (IPPDO) algorithm. First, we adopt a distributional perspective on the parameter and model it with an implicit distribution, which is approximated by generative models. Each model corresponds to a learned latent space, providing structured stochasticity for each layer in the network. Next, to make it possible to optimize an implicit posteriori parameter distribution, we build an energy-based model (EBM) with value function to represent the implicit distribution which is not constrained by any analytic density function. Then, we design a training algorithm based on amortized Stein variational gradient descent (SVGD) to improve the model learning efficiency. We compare IPPDO with other prevailing DRL algorithms on the OpenAI Gym, MuJoCo, and Box2D platforms. Experiments on various tasks demonstrate that the proposed algorithm can represent the parameter uncertainty implicitly for a learned policy and can consistently outperform competing approaches.

The Enlightenment Significance and Value Function of Aesthetic Psychological Structure Theory to Contemporary Aesthetic Education

In today's quality education, aesthetic education has become an important subject of modern education. Combined with the practice of aesthetic education, this paper discusses the nature, function, process and method of the construction of aesthetic psychology comprehensively. Using the "intermediary" of aesthetic psychological structure, cultivate and improve students 'aesthetic imagination, as the key point and breakthrough of aesthetic education teaching, and improve students' aesthetic psychological structure, so as to create a new situation of "hard teaching" into "happy teaching" and "hard learning" into "happy learning". The paper discusses the internal relationship between aesthetic education and the construction of aesthetic psychology. Only when the aesthetic psychological structure of the learning subject is constantly improved, the power of aesthetics will gradually emerge. In the construction of aesthetic psychology, the learning subject can be self-shaping, self-perfection until self-realization. Through aesthetic education to change people's way of thinking, create the comprehensive and harmonious development of human beings, teach people how to have the ability to appreciate and create beauty, and at the same time is closely related to everyone's aesthetic psychological structure. Therefore, aesthetic education is actually the educational process of how to construct people's aesthetic psychological structure at a higher level. Therefore, the aesthetic psychological structure is a very important concept, which not only affects the improvement of individual aesthetic ability, but also has enlightening significance on how to conduct aesthetic education. It is the most fundamental task and purpose to promote students to construct their own aesthetic psychological structure and improve the national quality.

Deep Q-Network Algorithm-Based Cyclic Air Braking Strategy for Heavy-Haul Trains

Cyclic air braking is a key element for ensuring safe train operation when running on a long and steep downhill railway section. In reality, the cyclic braking performance of a train is affected by its operating environment, speed and air-refilling time. Existing optimization algorithms have the problem of low learning efficiency. To solve this problem, an intelligent control method based on the deep Q-network (DQN) algorithm for heavy-haul trains running on long and steep downhill railway sections is proposed. Firstly, the environment of heavy-haul train operation is designed by considering the line characteristics, speed limits and constraints of the train pipe’s air-refilling time. Secondly, the control process of heavy-haul trains running on long and steep downhill sections is described as the reinforcement learning (RL) of a Markov decision process. By designing the critical elements of RL, a cyclic braking strategy for heavy-haul trains is established based on the reinforcement learning algorithm. Thirdly, the deep neural network and Q-learning are combined to design a neural network for approximating the action value function so that the algorithm can achieve the optimal action value function faster. Finally, simulation experiments are conducted on the actual track data pertaining to the Shuozhou–Huanghua line in China to compare the performance of the Q-learning algorithm against the DQN algorithm. Our findings revealed that the DQN-based intelligent control strategy decreased the air braking distance by 2.1% and enhanced the overall average speed by more than 7%. These experiments unequivocally demonstrate the efficacy and superiority of the DQN-based intelligent control strategy.

A Method of Reliability Assessment Based on Trend Analysis for Open Source Software

Software reliability growth model (SRGM) is used as one of the reliability assessment methods to assess the software reliability. In SRGM, the degree of reliability growth may fluctuate greatly according to change in the internal state of the software. It is called the change point (CP). Several researchers proposed the SRGM considering CP. In the open source software (OSS), there are many projects that continue development even after the software is released. Therefore, major updates with breaking changes may occur in it. The major updates can be a factor that causes a CP because it greatly changes the internal state of the OSS. This paper focuses on the relationship between CP and software updates. We collect OSS fault data from a bug tracking system. Moreover, we examine the behavior of SRGM before and after software updates. Furthermore, we discuss the applicability of SRGM for CP in OSS. Also, we compare the proposed model based on CP with the model without CP. As a result, we have confirmed that the SRGM can evaluate the reliability in the environment with major updates. Moreover, the proposed method performs better than without considering CP model. Especially, the exponential model’s mean value function is the suitable method to assess the OSS reliability for the proposed method.

Reference path for ships in ports and waterways based on optimal control

In the context of maritime transportation, waterborne shipping has developed quickly and attracted significant attention. This paper aims to propose a ship path planning model based on optimal control to provide reference paths for ships, prioritizing navigational efficiency and safety considerations. Based on the optimal control, the ship path planning problem is transformed into a multi-objective optimization problem, taking into account the speed constraints of ships during navigation. Then, the cost function based on the multi-objective optimization is constructed. Dynamic programming and numerical simulation are utilized to solve the value function, which is inducted to generate reference paths for ships. In addition, a case study including 12 Origin-Destination (OD) pairs in the Port of Rotterdam is conducted to generate reference paths. The simulation results showed that the model is proven to have high applicability in generating reference paths of ships navigating within the confines of port waters. By enhancing navigational safety and efficiency, the model could be used to provide invaluable guidance to both the bridge team and pilotage work.

Multi-agent reinforcement learning method for cutting parameters optimization based on simulation and experiment dual drive environment

Improving production efficiency while ensuring product surface quality is a constant focus of manufacturers. Cutting parameter optimization is an important technique for ensuring high-efficiency and high-quality production. In this paper, a novel method for cutting parameter optimization that integrates multi-agent reinforcement learning with a dual-drive virtual machining environment is proposed. First, a feature extraction, fusion and generation model for cutting simulation and experimental data is proposed to solve the problem of incomplete data acquisition in the production process. Second, a Markov decision model for optimizing cutting parameters is defined, and a virtual machining environment driven by both simulation and experimental data is constructed. Third, a novel multi-agent reinforcement learning method called Q-MIX-MATD3, in which a twin delay deep deterministic policy gradient, value function decomposition and a teacher model are combined, is proposed to explore the cutting parameter optimization policy by interacting with the virtual machining environment. Finally, the proposed method is verified on a commutator production line. Moreover, the results show that the accuracy of the virtual machining environment driven by both simulation and experiment increases by more than 5 %, response efficiency increases by 31 %, and Q-MIX-MATD3-based cutting parameter optimization method reduces time cost by 98 % and achieves the optimization effect of the classical optimization method.

Optimal stopping of conditional McKean–Vlasov jump diffusions

The purpose of this paper is to study the optimal stopping problem of conditional McKean–Vlasov (mean-field) stochastic differential equations with jumps (conditional McKean–Vlasov jump diffusions, for short). We obtain sufficient variational inequalities for a function to be the value function of such a problem and for a stopping time to be optimal.The key is that we combine the conditional McKean–Vlasov equation with the associated stochastic Fokker–Planck partial integro-differential equation for the conditional law of the state. This leads to a Markovian system which can be handled by using a version of a Dynkin formula.Our verification result is illustrated by finding the optimal time to sell in a market with common noise and jumps.

PREDICTING THE LEARNING PATH TO LEARNER’S OPTIMUM COMPREHENSION

The essence of learning is for the learner to attain a significant level of comprehension after the learning process is completed. The quest to achieve this singular purpose has led to the introduction of several learning techniques in the conventional learning environment, such as asking questions and conducting test after class, just to mention a few. Additionally, technology has been introduced in learning. Even with technological advancements, the learning experience still faces the challenge of learners not attaining the optimum comprehension state after the learning process. This is due to the present systems' inability to model the learner to determine the best methods for achieving maximum comprehension. Hence, this research paper focuses on deriving an improved mathematical model for predicting the learning path to a learner’s optimum comprehension. The paper presented three learning instructional media (learning paths); textual, audio and a hybrid of audio and video, which this research uses in modelling the learner. This is to enable the improved system predict the best learning path to optimum comprehension for learners. This research paper adopted Reinforcement Learning and the Markov decision process, specifically the Markov Chain approach, in developing an improved model for prediction. The evaluation of this research involved brainstorming on the Bellman’s equation with the aid of the Markov Chain transition state framework, resulting in an improved mean value function of 71.7. This indicates an enhanced comprehension state for the learning students compared to the existing mean value function of 46.0. The results obtained from this research clearly demonstrate that the improved model was able to predict and assign the best learning path to achieve optimum comprehension state for learners.

Linear quadratic optimal control turnpike in finite and infinite dimension: Two-term expansion of the value function

In this paper, we consider a linear quadratic (LQ) optimal control problem in both finite and infinite dimensions. We derive an asymptotic expansion of the value function as the fixed time horizon T tends to infinity. The leading term in this expansion, proportional to T, corresponds to the optimal value attained through the classical turnpike theory in the associated static problem. The remaining terms are associated with optimal stabilization problems towards the turnpike.

Hermite kernel surrogates for the value function of high-dimensional nonlinear optimal control problems

Numerical methods for the optimal feedback control of high-dimensional dynamical systems typically suffer from the curse of dimensionality. In the current presentation, we devise a mesh-free data-based approximation method for the value function of optimal control problems, which partially mitigates the dimensionality problem. The method is based on a greedy Hermite kernel interpolation scheme and incorporates context knowledge by its structure. Especially, the value function surrogate is elegantly enforced to be 0 in the target state, non-negative and constructed as a correction of a linearized model. The algorithm allows formulation in a matrix-free way which ensures efficient offline and online evaluation of the surrogate, circumventing the large-matrix problem for multivariate Hermite interpolation. Additionally, an incremental Cholesky factorization is utilized in the offline generation of the surrogate. For finite time horizons, both convergence of the surrogate to the value function and for the surrogate vs. the optimal controlled dynamical system are proven. Experiments support the effectiveness of the scheme, using among others a new academic model with an explicitly given value function. It may also be useful for the community to validate other optimal control approaches.

Multi-omics pan-cancer study of SPTBN2 and its value as a potential therapeutic target in pancreatic cancer

SPTBN2 is a protein-coding gene that is closely related to the development of malignant tumors. However, its prognostic value and biological function in pan-cancer, especially pancreatic cancer (PAAD), have not been reported. In the present study, a novel exploration of the value and potential mechanism of SPTBN2 in PAAD was conducted using multi-omics in the background of pan-cancer. Via various database analysis, up-regulated expression of SPTBN2 was detected in most of the tumor tissues examined. Overexpression of SPTBN2 in PAAD and kidney renal clear cell cancer patients potentially affected overall survival, disease-specific survival, and progression-free interval. In PAAD, SPTBN2 can be used as an independent factor affecting prognosis. Mutations and amplification of SPTBN2 were detected, with abnormal methylation of SPTBN2 affecting its expression and the survival outcome of PAAD patients. Immunoassay results demonstrate that SPTBN2 was a potential biomarker for predicting therapeutic response in PAAD, and may influence the immunotherapy efficacy of PAAD by regulating levels of CD8 + T cells and neutrophil infiltration. Results from an enrichment analysis indicated that SPTBN2 may regulate the development of PAAD via immune pathways. Thus, SPTBN2 is a potential prognostic biomarker and immunotherapy target based on its crucial role in the development of PAAD.

A Hybrid Multi-Agent Reinforcement Learning Approach for Spectrum Sharing in Vehicular Networks

Efficient spectrum sharing is essential for maximizing data communication performance in Vehicular Networks (VNs). In this article, we propose a novel hybrid framework that leverages Multi-Agent Reinforcement Learning (MARL), thereby combining both centralized and decentralized learning approaches. This framework addresses scenarios where multiple vehicle-to-vehicle (V2V) links reuse the frequency spectrum preoccupied by vehicle-to-infrastructure (V2I) links. We introduce the QMIX technique with the Deep Q Networks (DQNs) algorithm to facilitate collaborative learning and efficient spectrum management. The DQN technique uses a neural network to approximate the Q value function in high-dimensional state spaces, thus mapping input states to (action, Q value) tables that facilitate self-learning across diverse scenarios. Similarly, the QMIX is a value-based technique for multi-agent environments. In the proposed model, each V2V agent having its own DQN observes the environment, receives observation, and obtains a common reward. The QMIX network receives Q values from all agents considering individual benefits and collective objectives. This mechanism leads to collective learning while V2V agents dynamically adapt to real-time conditions, thus improving VNs performance. Our research finding highlights the potential of hybrid MARL models for dynamic spectrum sharing in VNs and paves the way for advanced cooperative learning strategies in vehicular communication environments. Furthermore, we conducted an in-depth exploration of the simulation environment and performance evaluation criteria, thus concluding in a comprehensive comparative analysis with cutting-edge solutions in the field. Simulation results show that the proposed framework efficiently performs against the benchmark architecture in terms of V2V transmission probability and V2I peak data transfer.

Prognostic value and immunological function of cuproptosis-related genes in lung adenocarcinoma

BackgroundLung cancer is one of the malignant tumors with the highest morbidity and fatality rates worldwide. The overall survival (OS) of lung adenocarcinoma (LUAD) is poor. Cuproptosis is a copper-triggered modality of mitochondrial cell death, however, its contribution to the emergence of lung cancer is unknown. The clinical implication and immunological function of cuproptosis-related genes (CRGs) in LUAD has yet to be established. MethodsTCGA, HPA, GEPIA, Kaplan-Meier, TIMER and CancerSEA database were used to explore the prognostic value and biological function of CRGs in LUAD. ResultsCRGs are primarily involved in copper ion transport, the citrate cycle (TCA cycle) and central carbon metabolism in LUAD. The mRNA expression of COA6, UBE2D1, DLAT, SLC25A3, and DBH was significantly increased. The expression of COA6, DLAT, SLC25A3, DBH, and LOXL2 were all strongly associated with the clinicopathological stages in LUAD. Moreover, high expression of COA6, UBE2D1, DLAT, SLC25A3 and LOXL2 was related to poor OS. The expression of SLC25A3 and LOXL2 showed different association with immune cell infiltration. The single cell sequencing demonstrated that CRGs play important roles in the regulation of DNA damage response, inflammation and metastasis in LUAD. ConclusionsIn summary, this study comprehensively uncovered that CRGs could be identified as potential prognostic and immunological biomarkers in LUAD. Our current research could provide a solid theoretical basis for LUAD survival research and clinical decision-making.

USV Trajectory Tracking Control Based on Receding Horizon Reinforcement Learning.

We present a novel approach for achieving high-precision trajectory tracking control in an unmanned surface vehicle (USV) through utilization of receding horizon reinforcement learning (RHRL). The control architecture for the USV involves a composite of feedforward and feedback components. The feedforward control component is derived directly from the curvature of the reference path and the dynamic model. Feedback control is acquired through application of the RHRL algorithm, effectively addressing the problem of achieving optimal tracking control. The methodology introduced in this paper synergizes with the rolling time domain optimization mechanism, converting the perpetual time domain optimal control predicament into a succession of finite time domain control problems amenable to resolution. In contrast to Lyapunov model predictive control (LMPC) and sliding mode control (SMC), our proposed method employs the RHRL controller, which yields an explicit state feedback control law. This characteristic endows the controller with the dual capabilities of direct offline and online learning deployment. Within each prediction time domain, we employ a time-independent executive-evaluator network structure to glean insights into the optimal value function and control strategy. Furthermore, we substantiate the convergence of the RHRL algorithm in each prediction time domain through rigorous theoretical proof, with concurrent analysis to verify the stability of the closed-loop system. To conclude, USV trajectory control tests are carried out within a simulated environment.