Many optimal and robust control problems are nonconvex and potentially nonsmooth in their policy optimization forms. In Part II of this paper, we introduce a new and unified Extended Convex Lifting (ECL) framework to reveal <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">hidden convexity</i> in classical optimal and robust control problems from a modern optimization perspective. Our optimization perspective offers a bridge between nonconvex policy optimization and convex reformulations, enabling convex analysis for nonconvex problems. Despite non-convexity and non-smoothness, the existence of an ECL not only reveals that minimizing the original function is equivalent to a convex problem but also certifies a class of first-order <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">non-degenerate</i> stationary points to be globally optimal. This ECL framework can cover many benchmark control problems, including LQR, LQG, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {H}_\infty$</tex-math></inline-formula> robust control. We also believe that the new ECL framework will be of independent interest for analyzing nonconvex problems beyond control.

This paper introduces novel Bellman mappings (BMaps) for value iteration (VI) in distributed reinforcement learning (DRL), where agents are deployed over an undirected, connected graph/network with arbitrary topology—but without a centralized node, that is, a node capable of aggregating all data and performing computations. Each agent constructs a nonparametric B-Map from its private data, operating on Qfunctions represented in a reproducing kernel Hilbert space, with flexibility in choosing the basis for their representation. Agents exchange their Q-function estimates only with direct neighbors, and unlike existing DRL approaches that restrict communication to Q-functions, the proposed framework also enables the transmission of basis information in the form of covariance matrices, thereby conveying additional structural details. Linear convergence rates are established for both Q-function and covariance-matrix estimates toward their consensus values, regardless of the network topology, with optimal learning rates determined by the ratio of the smallest positive eigenvalue (the graph’s Fiedler value) to the largest eigenvalue of the graph Laplacian matrix. A detailed performance analysis further shows that the proposed DRL framework effectively approximates the performance of a centralized node, had such a node existed. Numerical tests on distributed extensions of three benchmark control problems confirm the effectiveness of the proposed nonparametric B-Maps relative to state-of-the-art methods. Notably, the tests reveal a counter-intuitive outcome: although the framework involves richer information exchange—specifically, through the transmission of covariance matrices as information on the basis of a subspace—it achieves the desired performance at a lower cumulative communication cost than several state-of-the-art DRL schemes, underscoring the critical role of sharing basis information in accelerating the learning process.

This paper presents a framework for numerical continuation that can transform a previously known, potentially suboptimal, control history into a minimum effort control history without needing to find the appropriate initial costate values for the known solution. This formulation is motivated by the fact that analytical and/or approximate solutions to aerospace control problems generally are significantly easier to compute compared to corresponding optimal trajectories for the same boundary conditions. Moreover, numerical methods for computing indirect optimal control solutions often greatly benefit from having an initial guess that is “close” to the optimal solution. For this reason, it is often desirable to produce a quickly computable approximate solution that can bootstrap an optimal control solution process. Salient to note here is that using an approximate solution as an initial guess for an indirect optimal control solver requires the user to find the appropriate initial costate values corresponding to the previously known control history, which is generally a nontrivial problem itself. The proposed algorithm provides a systematic framework for addressing this initial costate generation hurdle together with strong convergence properties. The methodology is applied and illustrated for a wide array of benchmark control problems.

The maRTHS benchmark problem (Condori Uribe et al., 2023) offers a framework to assess and improve control techniques, enabling the research community to handle the intricate requirements of multi-dimensional testing better. This initiative fosters a deeper understanding of structural responses by encouraging the shift from single-actuator to multi-actuator experiments, paving the way for safer, more resilient designs. In particular, this benchmark is the secondgeneration problem in virtual RTHS. The scientific community received the first-generation RTHS benchmark (Silva et al., 2020), which provided a virtual testbed for researchers to validate complex control schemes and allowed for training and development purposes to build the necessary tools to conduct single-actuator RTHS tests. Following this successful approach, the second-generation maRTHS benchmark dramatically increases the virtual test&#39;s sophistication by incorporating two actuators to command two degrees of freedom (displacement and rotation) at the interface between numerical and experimental substructures. The problem was designed and implemented in the laboratory at Purdue University, where the properties from the specimen and loading equipment were obtained for the relevant models included in the problem. A virtual representation of the maRTHS experiment was encapsulated and stored in a shared repository, providing researchers with the necessary tools to develop control algorithms and envision more sophistication in these experimental techniques to advance the discipline further. Meanwhile, other authors have proposed innovative developments to improve the accuracy and capabilities of maRTHS. Saeger et al. (2024) considered a computer vision-based displacement tracking algorithm (Lucas-Kanade optical flow method) in a virtual reality environment to allow contactless feedback control in virtual maRTHS. Sudvarg et al. (2024) studied real-time scheduling for virtual maRTHS computations and proposed adaptive event handling for intensive simulations over larger computational models and complex control architectures. Tian et al. (2024) proposed another maRTHS problem inspired by this research topic&#39;s benchmark, where a boundary-coordinating device (BDC) comprising double shaking tables in parallel configuration can enforce a shear-moment boundary condition over a physical substructure.Finally, the collective contributions in this special issue mark a significant step forward in the design and control techniques essential for multi-actuator systems. Nonetheless, several challenges remain that future research must address to advance this experimental approach. First, understanding how the proposed compensation algorithms scale and generalize to higher degrees of multi-actuator loading, especially regarding computational complexity and real-time performance, is critical. Second, better insights into multi-actuator coupling in Cartesian coordinates are necessary to enable direct control of specimen boundary conditions, rather than indirect control through actuator coordinates. Third, integrating mixed-mode displacement-force testing is key to simulating more realistic and complex loading scenarios. By confronting these challenges, the continued progress in dynamic testing methodologies is expected to open avenues for more accurate simulation of real-world conditions and consider multiple natural hazards.

Biological systems often choose actions without an explicit reward signal, a phenomenon known as intrinsic motivation. The computational principles underlying this behavior remain poorly understood. In this study, we investigate an information-theoretic approach to intrinsic motivation, based on maximizing an agent's empowerment (the mutual information between its past actions and future states). We show that this approach generalizes previous attempts to formalize intrinsic motivation, and we provide a computationally efficient algorithm for computing the necessary quantities. We test our approach on several benchmark control problems, and we explain its success in guiding intrinsically motivated behaviors by relating our information-theoretic control function to fundamental properties of the dynamical system representing the combined agent-environment system. This opens the door for designing practical artificial, intrinsically motivated controllers and for linking animal behaviors to their dynamical properties. Published by the American Physical Society 2024

Real-time hybrid simulation (RTHS) is a widely applied test method in structural engineering, which is developed from pseudo-dynamic test. Much of the past work has been centered on one-dimensional RTHS using a single hydraulic actuator. When the complexity of the problem demands to increase the number of degrees of freedom to be enforced on the boundary conditions, more than one hydraulic actuator must be used. Multiple-actuator or multi-axial RTHS (maRTHS) requires that more than one hydraulic actuator exerts the required motion on experimental substructures demanding the implementation of multiple-input multiple-output (MIMO) control strategies. A new maRTHS benchmark control problem has been developed, focusing on a frame subjected to seismic load at the base, substantially transforming and intensifying the complexity of the problem. The time delay generated by the dynamic characteristics of the loading system and the transmission process as well as the high coupling between the hydraulic actuators and the nonlinear kinematics escalates the complexity of the actuator control tracking. A sliding mode adaptive delay compensation method suitable for maRTHS is proposed, which utilizes a MIMO sliding mode method to reduce the coupling effects of actuators and the adaptive compensation method to compensate the residual delay. The effectiveness of the method is verified by numerical simulating different working conditions in the Benchmark Problem Platform.

This paper presents an end-to-end learning approach to developing a Nonlinear Model Predictive Control (NMPC) policy, which does not require an explicit first-principles model and assumes that the system dynamics are either unknown or partially known. The paper proposes the use of available measurements to identify a nominal Recurrent Neural Network (RNN) model to capture the nonlinear dynamics, which includes constraints on the state variables and inputs. To address the issue of suboptimal control policies resulting from simply fitting the model to the data, this paper uses Reinforcement learning (RL) to tune the NMPC scheme and generate an optimal policy for the real system. The approach’s novelty lies in the use of RL to overcome the limitations of the nominal RNN model and generate a more accurate control policy. The paper discusses the implementation aspects of initial state estimation for RNN models and integration of neural models in MPC. The presented method is demonstrated on a classic benchmark control problem: cascaded two tank system (CTS).

The structural performance of critical infrastructure during extreme events requires testing to understand the complex dynamics. Shake table testing of buildings to evaluate structural integrity is expensive and requires special facilities that can allow for the construction of large-scale test specimens. An attractive alternative is a cyber-physical testing technique known as Real-Time Hybrid Simulation (RTHS), where a large-scale structure is decomposed into physical and numerical substructures. A transfer system creates the interface between physical and numerical substructures. The challenge occurs when using multiple actuators connected with a coupler (i.e., transfer system) to create translation and rotation at the interface. Tracking control strategies aim to reduce time delay errors to create the desired displacements that account for the complex dynamics. This paper proposes two adaptive control methodologies for multi-axial real-time hybrid simulations that improve capabilities for a higher degree of coupling, boundary, complexity, and noise reduction. One control method integrates the feedback proportional derivative integrator (PID) control with a conditional adaptive time series (CATS) compensation and inverse decoupler. The second proposed control method is based on a coupled Model Predictive Control (MPC) with the CATS compensation. The performance of the proposed methods is evaluated using the virtual multi-axial benchmark control problem consisting of a steel frame as the experimental substructure. The transfer system consists of a coupler that connects two hydraulic actuators generating the translation and rotation acting at the joint. Through sensitivity analysis, parameters were tuned for the decoupler components, CATS compensation, and the control design for PID, LQG, and MPC. Comparative results among different control methods are evaluated based on performance criteria, including critical factors such as reduction in the time delay of bothactuators. The research findings in this paper improve the tracking control systems for the multi-axial RTHS of building structures subjected to earthquake loading. It provides insight into the robustness of the proposed tracking control methods in addressing uncertainty and improves the understanding of multiple output controllers that could be used in future cyber-physical testing of civil infrastructure subjected to natural hazards.

Abstract A classic way to design a nonlinear model predictive control (NMPC) scheme with guaranteed stability is to incorporate a terminal cost and a terminal constraint into the problem formulation. While a long prediction horizon is often desirable to obtain a large domain of attraction and good closed‐loop performance, the related computational burden can hinder its real‐time deployment. In this article, we propose an NMPC scheme with prediction horizon and no terminal constraint to drastically decrease the numerical complexity without significantly impacting closed‐loop stability and performance. This is attained by constructing a suitable terminal cost from data that estimates the cost‐to‐go of a given NMPC scheme with long prediction horizon. We demonstrate the advantages of the proposed control scheme in two benchmark control problems.

Although fuel gas systems represent a large part of industrial chemical processes, there has been limited literature on their modelling and control. The available literature typically neglects the effects of fuel gas consumer dynamics, leaving much of the system's important dynamic behaviour omitted. This paper aims to contribute to the existing literature on fuel gas control and improve an existing fuel gas control benchmark problem by including the effects of fuel gas consumer dynamics on the system. Two model predictive controllers (MPC) were designed, where the first MPC uses a model that neglects the consumer dynamics and the second MPC uses a model that includes the consumer dynamics. It was found that the MPC neglecting consumer dynamics has a pressure variability 5.8 times higher than the MPC that includes the dynamics. It also has a relative sensitivity index (RSI) of 7.2, indicating the presence of model-plant mismatches (MPM) affecting controller performance.

Topology and weight evolving artificial neural network (TWEANN) algorithms optimize the structure and weights of artificial neural networks (ANNs) simultaneously. The resulting networks are typically used as policy models for solving control and reinforcement learning (RL) type problems. This paper presents a neuroevolution algorithm that aims to address the typical stagnation and sluggish convergence issues present in other neuroevolution algorithms. These issues are often caused by inadequacies in population diversity preservation, exploration/exploitation balance, and search flexibility. This new algorithm, called the Adaptive Genomic Evolution of Neural-Network Topologies (AGENT), builds on the neuroevolution of augmenting topologies (NEAT) concept. Novel mechanisms for adapting the selection and mutation operations are proposed to favorably control population diversity and exploration/exploitation balance. The former is founded on a fundamentally new way of quantifying diversity by taking a graph-theoretic perspective of the population of genomes and inter-genomic differences. Further advancements to the NEAT paradigm occur through the incorporation of variable neuronal properties and new mutation operations that uniquely allow both the growth and pruning of ANN topologies during evolution. Numerical experiments with benchmark control problems adopted from the OpenAI Gym illustrate the competitive performance of AGENT against standard RL methods and adaptive HyperNEAT, and superiority over the original NEAT algorithm. Further parametric analysis provides key insights into the impact of the new features in AGENT. This is followed by evaluation on an unmanned aerial vehicle collision avoidance problem where maneuver planning models are learnt by AGENT with 33% reward improvement over 15 generations.

Advancing RTHS methods to readily handle multi-dimensional problems has great potential for enabling more advanced testing and synergistically using existing laboratory facilities that have the capacity for such experimentation. However, the high internal coupling between hydraulics actuators and the nonlinear kinematics escalates the complexity of actuator control and boundary condition tracking. To enable researchers in the RTHS community to develop and compare advanced control algorithms, this paper proposes a benchmark control problem for a multi-axial real-time hybrid simulation (maRTHS) and presents its definition and implementation on a steel frame excited by seismic loads at the base. The benchmark problem enables the development and validation of control techniques for tracking both translation and rotation degrees of freedom of a plant that consists of a steel frame, two hydraulic actuators, and a steel coupler with high stiffness that couples the axial displacements of the hydraulic actuators resulting in the required motion of the frame node. In this investigation, the different components of this benchmark were developed, tested, and a set of maRTHS were conducted to demonstrate its feasibility in order to provide a realistic virtual platform. To offer flexibility in the control design process, experimental data for identification purposes, finite element models for the reference structure, numerical, and physical substructure, and plant models with model uncertainties are provided. Also, a sample example of an RTHS design based on a linear quadratic Gaussian controller is included as part of a computational code package, which facilitates the exploration of the tradeoff between robustness and performance of tracking control designs. The goals of this benchmark are to: extend existing control or develop new control techniques; provide a computational tool for investigation of the challenging aspects of maRTHS; encourage a transition to multiple actuator RTHS scenarios; and make available a challenging problem for new researchers to investigate maRTHS approaches. We believe that this benchmark problem will encourage the advancing of the next-generation of controllers for more realistic RTHS methods.

Abstract In this paper, a new mechanism for detecting population stagnation based on the analysis of the local improvement of the evaluation function and the infinite impulse response filter is proposed. The purpose of this mechanism is to improve the population stagnation detection capability for various optimization scenarios, and thus to improve multi-population-based algorithms (MPBAs) performance. In addition, various other approaches have been proposed to eliminate stagnation, including approaches aimed at both improving performance and reducing the complexity of the algorithms. The developed methods were tested, among the others, for various migration topologies and various MPBAs, including the MNIA algorithm, which allows the use of many different base algorithms and thus eliminates the need to select the population-based algorithm for a given simulation problem. The simulations were performed for typical benchmark functions and control problems. The obtained results confirm the validity of the developed method.

Abstract The quadruple tank (QT) system consists of four interacting tanks and can switch between the minimum and non‐minimum phase behavior with changes in the positions of pump valves and is considered a benchmark control problem. In the present study, long‐short term memory (LSTM), a type of recurrent neural networks (RNN) is designed for the benchmark QT system based on the model‐based control framework. Random input–output sequences are generated from the white box model of the QT system to train an LSTM network model. The LSTM network is tuned by adjusting its hyperparameters such as the number of hidden layers, hidden units, and epochs to minimize the prediction error on the test data. The trained model is cross validated both during and after training to avoid overfitting. Once a reasonably reliable model is obtained, another LSTM network is trained for use as a controller. The network architecture is constantly modified till the controller is able to track the test setpoints with minimum error. This procedure is repeated with a gated recurrent unit (GRU) network and the servo and regulatory response of both the network models and controller are evaluated in terms of standard performance measure namely root mean square error (RMSE), integral square error (ISE), and control effort (CE). It is observed that the controller designed based on RNN performs better than a conventional centralized controller.

PANOC is an algorithm for nonconvex optimization that has recently gained popularity in real-time control applications due to its fast, global convergence. The present work proposes a variant of PANOC that makes use of Gauss–Newton directions to accelerate the method. Furthermore, we show that when applied to optimal control problems, the computation of this Gauss–Newton step can be cast as a linear quadratic regulator (LQR) problem, allowing for an efficient solution through the Riccati recursion. Finally, we demonstrate that the proposed algorithm is more than twice as fast as the traditional L–BFGS variant of PANOC when applied to an optimal control benchmark problem, and that the performance scales favorably with increasing horizon length.

The utilization of Le`vy flight to create new candidate solutions is one of the most powerful elements of CS. Candidate solutions are modified using this method by making a lot of minor modifications and a few big jumps. As a result, CS will be able to significantly increase the link between exploration and exploitation while also improving its search capabilities. The cuckoo search optimization (CSO) algorithm is applied to interval type-2 fuzzy logic controller (IT2FLC) in this research to determine the optimal parameters of membership functions (MFs) of interval type-2 fuzzy logic systems (IT2FLSs). The study takes into account two forms of MFs: triangular and trapezoidal. When perturbations are applied during the execution of each control issue, the CSO algorithm's performance and efficiency improve significantly. The proposed approach is tested using two benchmark control problems: water tank controller and inverted pendulum controller.

The current journal paper proposes an end-to-end analysis for the numerical implementation of a two-degrees-of-freedom (2DOF) control structure, starting from the sampling rate selection mechanism via a quasi-optimal manner, along with the estimation of the worst-case execution time (WCET) for the specified controller. For the sampling rate selection, the classical Shannon–Nyquist sampling theorem is replaced by an optimization problem that encompasses the trade-off between the fidelity of the controllers’ representation, along with the fidelity of the resulting closed-loop systems, and the implementation difficulty of the controllers. Additionally, the WCET analysis can be seen as a verification step before automatic code generation, a computational model being provided. The proposed computational model encompasses infinite-impulse response (IIR) and finite-impulse response (FIR) filter models for the controller implementation, along with additional relevant phenomena being discussed, such as saturation, signal scaling and anti-windup techniques. All proposed results will be illustrated on a DC motor benchmark control problem.

In this article, we discuss continuous-time H2 control for the unknown nonlinear system. We use differential neural networks to model the system, then apply the H2 tracking control based on the neural model. Since the neural H2 control is very sensitive to the neural modeling error, we use reinforcement learning to improve the control performance. The stabilities of the neural modeling and the H2 tracking control are proven. The convergence of the approach is also given. The proposed method is validated with two benchmark control problems.

The general motivation of our work is to meet the main time constraint when implementing a control loop: the Controller’s execution time is less than the sampling period. This paper proposes a practical method to diminish the computational complexity of the controllers using predictions based on the Evolutionary Algorithm (EA). It is the case of Model Predictive Control or, more generally, Receding Horizon Control structures. The main drawback of the metaheuristic algorithms (including EAs) working in control structures is their great complexity. Usually, the control variables take values between minimum and maximum technological limits. This work’s main idea is to consider the control variables’ domain inside a predefined control profile’s neighbourhood. The Controller takes into account a smaller domain of the control variables without tracking the predefined control profile or a reference trajectory. The convergence of the EA under consideration is not affected; hence, the same best predictions are found. The predefined control profile is already known or can be determined by solving the optimal control problem without time constraints in open-loop and offline. This work also presents a simulation study applying the proposed technique that involves two benchmark control problems. The results prove that the computational complexity decreases significantly.

Interval type-2 generalized fuzzy hyperbolic modelling and control of nonlinear systems