Model-free Features Research Articles

Optimal energy system scheduling using a constraint-aware reinforcement learning algorithm

The massive integration of renewable-based distributed energy resources (DERs) inherently increases the energy system’s complexity, especially when it comes to defining its operational schedule. Deep reinforcement learning (DRL) algorithms arise as a promising solution due to their data-driven and model-free features. However, current DRL algorithms fail to enforce rigorous operational constraints (e.g., power balance, ramping up or down constraints) limiting their implementation in real systems. To overcome this, in this paper, a DRL algorithm (namely MIP-DQN) is proposed, capable of strictly enforcing all operational constraints in the action space, ensuring the feasibility of the defined schedule in real-time operation. This is done by leveraging recent optimization advances for deep neural networks (DNNs) that allow their representation as a MIP formulation, enabling further consideration of any action space constraints. Comprehensive numerical simulations show that the proposed algorithm outperforms existing state-of-the-art DRL algorithms, obtaining a lower error when compared with the optimal global solution (upper boundary) obtained after solving a mathematical programming formulation with perfect forecast information; while strictly enforcing all operational constraints (even in unseen test days).

Multi-objective optimal control for proton exchange membrane fuel cell via large-scale deep reinforcement learning

To achieve a high operation efficiency and control performance of proton exchange membrane fuel cell (PEMFC), a multi-objective optimal fractional-order proportion integration differentiation (FOPID) controller is proposed to maintain the oxygen excess ratio (OER) and output voltage at the reference value simultaneously. In addition, a novel large-scale deep reinforcement learning called demonstration curriculum strategy large-scale multi-delay deep deterministic policy gradient (DCSL-MD3PG) algorithm is designed to operate as the tuner of this controller. This algorithm applies the teacher–student large-scale deep reinforcement learning strategy as well as the demonstration curriculum learning strategy and a number of techniques in the offline training to improve robustness and optimize control ability of the controller. Simulation results show that, taking the benefits of the excellent adaptability and model-free features of deep reinforcement learning, the adaptive optimal FOPID algorithm can ensures the optimal and comprehensive control performance of OER and output voltage while satisfying the security constraints of PEMFC.

Optimal adaptive control for solid oxide fuel cell with operating constraints via large-scale deep reinforcement learning

Since a solid oxide fuel cell (SOFC) is a complicated nonlinear, time-varying and constrained system, it is difficult to control the fuel flow to stabilize the output voltage while considering fuel utilization operating constraints. To overcome this problem, an adaptive fractional-order proportional integral derivative (FOPID) controller, taking advantage of the adaptability and model-free features of large-scale deep reinforcement learning, is proposed in this paper. Furthermore, a fittest survival strategy large-scale twin delayed deep deterministic policy gradient (FSSL-TD3) algorithm is designed as the tuner of this controller. In this algorithm, the exploration efficacy is improved by way of the fittest survival strategy and imitation learning. Other techniques are also applied to this algorithm in order to improve the robustness of FOPID controller. In addition, by formulating the reward function of the FSSL-TD3 algorithm, the fuel utilization of the SOFC can always be kept in a safe range, which is not possible for conventional control algorithms. The simulation results in this paper show that the output voltage of SOFCs can be controlled effectively by this controller while fuel utilization is retained within a reasonable range.

A novel data-driven controller for solid oxide fuel cell via deep reinforcement learning

Solid oxide fuel cells (SOFC) are complex nonlinear and time-varying systems with operational constraints. How to effectively control and stabilize the output voltage while preventing constraint violations becomes a main challenge to its wide application. To improve the efficiency of its operation and power tracking control and prevent constraint violations, this paper designs a data-driven adaptive proportional integral derivative (PID) controller, which maintains the output voltage at the reference value via the optimal control of the hydrogen flow. Moreover, a novel large-scale deep reinforcement learning (DRL) algorithm, called the two-stage training strategy large-scale twin delayed deep determination policy gradient (TGSL-TD3PG), is adopted to adaptively adjust the baseline coefficients of the designed controller scaffolded by the high adaptability and model-free features of reinforcement learning. In the training of TGSL-TD3PG, multiple agents are simultaneously employed to acquire the best policy, whereby the training of optimal agents in practice is structured upon the principles of imitation learning and curriculum learning. This method solves the common problem of low robustness in conventional deep reinforcement learning and can be applied to the field of control. Moreover, the TGSL-TD3PG algorithm incorporates the baseline PID coefficients into the design objective and offers the controller with the online coefficient-adjusting ability through learning. Therefore, such limitations as no adaptability, low robustness and neglect of constraints associated with conventional PID control are significantly addressed. Simulation results show that the proposed controller can promote the SOFC to track the load power demand effectively while maintaining the fuel utilization ratio constant. The proposed algorithm reduces 45.2% of the output voltage setting time and 30% of the voltage overshoot to the maximum extent, while generating the constraint violation time of fuel utilization at 0.

A New Adaptive Type-II Fuzzy-Based Deep Reinforcement Learning Control: Fuel Cell Air-Feed Sensors Control

This paper proposes a new adaptive controller for air-feed on proton exchange membrane fuel cell (PEMFC) plants. The oxygen excess ratio is first regulated by a single input interval type-2 fuzzy PI (SIT2-FPI) controller during the current variation of the PEMFC system. Then, a deep deterministic policy gradient (DDPG) algorithm is adopted to adaptively adjust the baseline PI coefficients of the SIT2 controller by taking the benefits of the online learning and model-free features of reinforcement learning. In the DDPG structure, an actor-network determines the policy signals, while a critic-network evaluates the quality of the policy provided by the actor. The suggested DDPG algorithm naturally takes into account the baseline PI coefficients into the design objective and offers the SIT2-FPI structure with online coefficient adjusting ability through learning. Based on reward feedback of the oxygen excess ratio error, the weights of the actor and critic networks are updated by the gradient descent technique. Detailed real-time model-in-the loop (MIL) simulation outcomes and comparative analysis are presented to confirm the adaptation capability of the DDPG-based online SIT2-FPI coefficient tuning strategy.

Robust gait recognition using hybrid descriptors based on Skeleton Gait Energy Image

Gait features have been widely applied in human identification. The commonly-used representations for gait recognition can be roughly classified into two categories: model-free features and model-based features. However, due to the view variances and clothes changes, model-free features are sensitive to the appearance changes. For model-based features, there is great difficulty in extracting the underlying models from gait sequences. Based on the confidence maps and the part affinity fields produced by a two-branch multi-stage CNN network, a new model-based representation, Skeleton Gait Energy Image (SGEI), has been proposed in this paper. Another contribution is that a hybrid representation has been produced, which uses SGEI to remedy the deficiency of model-free features, Gait Energy Image (GEI) for instance. The experimental performances indicate that our proposed methods are more robust to the cloth changes, and contribute to increasing the robustness of gait recognition in the unconstrained environments with view variances and clothes changes.

Methyl dynamics of a Ca2+-calmodulin-peptide complex from NMR/SRLS.

We developed the slowly relaxing local structure (SRLS) approach for analyzing NMR spin relaxation in proteins. SRLS accounts for dynamical coupling between the tumbling of the protein and the local motion of the probe and for general tensorial properties. It is the generalization of the traditional model-free (MF) method, which does not account for mode-coupling and treats only simple tensorial properties. SRLS is applied herein to ²H relaxation of ¹³CDH₂ groups in the complex of Ca(2+)-calmodulin with the peptide smMLCKp. Literature data comprising ²H T₁ and T₂ acquired at 14.1 and 17.6 T, and 288, 295, 308, and 320 K, are used. We find that mode-coupling is a small effect for methyl dynamics. On the other hand, general tensorial properties are important. In particular, it is important to allow for the asymmetry of the local spatial restrictions, which can be represented in SRLS by a rhombic local ordering tensor with components S(0)(2) and S(2)(2). The principal axes frame of this tensor is obviously different from the axial frames of the magnetic tensors. Here, we find that -0.2 ≤ S(0)(2) ≤ 0.5 and -0.4 ≤ S(2)(2) ≤ 0. MF features a single "generalized" order parameter, S, confined to the 0-0.316 range; the local geometry is inherently simple. The parameter S is inaccurate, having absorbed unaccounted for effects, notably S(2)(2) ≠ 0. We find that the methionine methyls (the other methyl types) reorient with rates of 8.6 × 10⁹ to 21.4 × 10⁹ (0.67 × 10⁹ to 6.5 × 10⁹) 1/s. The corresponding activation energies are 10 (10-27) kJ/mol. By contrast, MF yields inaccurate effective local motional correlation times, τ(e), with nonphysical temperature dependence. Thus, the problematic S- and τ(e)-based MF picture of methyl dynamics has been replaced with an insightful physical picture based on a local ordering tensor related to structural features, and a local diffusion tensor that yields accurate activation energies.

Tracking rigid objects using integration of model-based and model-free cues

Model-based 3-D object tracking has earned significant importance in areas such as augmented reality, surveillance, visual servoing, robotic object manipulation and grasping. Key problems to robust and precise object tracking are the outliers caused by occlusion, self-occlusion, cluttered background, reflections and complex appearance properties of the object. Two of the most common solutions to the above problems have been the use of robust estimators and the integration of visual cues. The tracking system presented in this paper achieves robustness by integrating model-based and model-free cues together with robust estimators. As a model-based cue, a wireframe edge model is used. As model-free cues, automatically generated surface texture features are used. The particular contribution of this work is the integration framework where not only polyhedral objects are considered. In particular, we deal also with spherical, cylindrical and conical objects for which the complete pose cannot be estimated using only wireframe models. Using the integration with the model-free features, we show how a full pose estimate can be obtained. Experimental evaluation demonstrates robust system performance in realistic settings with highly textured objects and natural backgrounds.

GREY PREDICTION BASED PARTICLE FILTER FOR MANEUVERING TARGET TRACKING

For maneuvering target tracking, we propose a novel grey prediction based particle fllter (GP-PF), which incorporates the grey prediction algorithm into the standard particle fllter (SPF). The basic idea of the GP-PF is that new particles are sampled by both the state transition prior and the grey prediction algorithm. Since the grey prediction algorithm is a kind of model-free method and is able to predict the system state based on historical measurements other than establishing a priori dynamic model, the GP-PF can signiflcantly alleviate the sample degeneracy problem which is common in SPF, especially when it is used for maneuvering target tracking. Simulations are conducted in the context of two typical maneuvering motion scenarios and the results indicate that the overall performance of the proposed GP-PF is better than the SPF and the multiple model particle fllter (MMPF) when the tracking accuracy, computational complexity and tracking lost probability are considered. The performance improvements can be attributed to that the GP-PF has both model-based and model-free features.

Combining hard- and soft-modelling to solve kinetic problems

A novel approach mixing the qualities of hard-modelling and soft-modelling methods is proposed to analyse kinetic data monitored spectrometrically. Taking as a basis the Multivariate Curve Resolution–Alternating Least Squares method (MCR–ALS), which obtains the pure concentration profiles and spectra of all absorbing species present in the raw measurements by using typical soft-modelling constraints, a new hard constraint is introduced to force some or all the concentration profiles to fulfill a kinetic model, which is refined at each iterative cycle of the optimisation process. This modification of MCR–ALS drastically decreases the rotational ambiguity associated with the kinetic profiles obtained using exclusively soft-modelling constraints. The optional inclusion of some or all the absorbing species into the kinetic model allows the successful treatment of data matrices whose instrumental response is not exclusively due to the chemical components involved in the kinetic process, an impossible scenario for classical hard-modelling approaches. Moreover, the possible distinct constraint of each of the matrices in a three-way data set allows for the simultaneous analysis of kinetic runs with diverse kinetic models and rate constants. Thus, the introduction of model-based and model-free features in the treatment of kinetic data sets yields more satisfactory results than the application of pure hard- or pure soft-modelling approaches. Simulated and real examples are used to confirm this statement.

A robotic motion controller using a self-organizing fuzzy logic algorithm with grey prediction

It is well known that robotic manipulators are highly non-linear coupled dynamic systems. It is difficult to establish an appropriate mathematical model for the design of a model-based controller. Although fuzzy logic control has model-free features, it still needs time-consuming work for rule bank and fuzzy parameter adjustment. Hence the self-organizing fuzzy controller is proposed to manipulate the motion trajectory of robots with multiple degrees of freedom. This approach has learning ability for responding to the time-varying characteristic of a robot. Its control rule bank can be established and modified continuously by on-line learning with zero initial fuzzy rules. However, this control strategy has larger oscillatory behaviour during initial learning. Here a self-organizing fuzzy controller with grey prediction is proposed to improve this behaviour. The experimental results show that this intelligent controller can reduce significantly the oscillatory amplitude of the output trajectory and tracking error.