Model-free Way Research Articles

Advanced isoconversional kinetic analysis: Insight in mechanisms and simulations. Successes and future

Various examples of application of advanced isoconversional kinetic analysis are discussed. The paper focuses on polymerization and crystallization mechanisms, protein denaturation and simulations in a model-free way under different temperature programs. It is shown how isoconversional kinetic analysis was developed in the recent years to gain meaningful kinetic information on changes in rate-limiting steps and was applied to multi-step processes. The complementary of the information obtained by estimating pre-exponential factors, rate constants and mathematical function that describes the reaction mechanism, in addition to the usual effective activation energy dependency, is highlighted. In addition, combination of model-free and model-based methods can be used for the elucidation of multi-step kinetics.

Reinforcement learning to achieve real-time control of triple inverted pendulum

This work uses reinforcement learning (RL) to achieve the first-ever data-driven real-time control of an actual, not simulated, triple inverted pendulum (TIP) in a model-free way. A swing-up control task for the TIP is formulated as a Markov decision process with a dense reward function, then conducted in real time by using a model-free RL approach. To increase the sample efficiency of learning, a structure-aware virtual experience replay (VER) method is proposed; it works together with an off-policy actor-critic algorithm. The VER exploits the geometrically-symmetric property of TIPs to create virtual sample trajectories from measured ones, then uses the resulting multifold augmented dataset to effectively train actor and critic networks during the learning process. These structure-infused training data serve to obtain additional information and hence increase the convergence speed of network learning. We combine the proposed VER with a state-of-the-art actor-critic algorithm, and then validate its effectiveness through numerical simulations. Notably, the inclusion of VER amplifies computational efficiency, slashing the requisite trials, training steps, and overall duration by approximately 66.67%. Finally experiments demonstrate the real-time control capability of the proposed approach on an actual TIP system.

Self-Attention Generative Adversarial Network Enhanced Learning Method for Resilient Defense of Networked Microgrids Against Sequential Events

The resilient responses of networked microgrids (MGs) can greatly improve the survival of critical loads during extreme events. In order to efficiently handle the scarce data issue as well as improve the adaptability of deep reinforcement learning (DRL) methods for complex sequential extreme events (SEEs) such as hurricanes and tornadoes, a new learning-based method is proposed for the survival of critical loads in MGs during SEEs. A generative adversarial network (GAN) is applied to generate a sufficient extreme event-related database in a model-free way. Specifically, a self-attention GAN (SA-GAN) is developed to capture sequential features of the SEE process. Then, the SA-GAN is integrated into a DRL framework, and the corresponding Markov decision process (MDP) and the environment are designed to realize adaptive networked MG reconfiguration for the survival of critical loads. Faced with uncertain distributed generator (DG) output and sequential line damage, the SA-GAN-DRL method provides an adaptive model-free solution to continuously supply critical loads during SEEs. The effectiveness of the proposed method is validated using a 7-bus test system and the IEEE 123-bus system, and the results demonstrate both a strong learning ability with limited practice data, and robustness and adaptability for highly changeable SEE processes.

Neuron-Glia-Ratio-Like Approach Evidenced for Limited Variability and In-Aggregate Circadian Shifts in Cortical Cell-Specific Transcriptomes.

Regardless of shifts in levels of individual transcripts, it remains elusive whether natural variability in cell-specific transcriptomes within the cerebral cortex is limited in aggregate. It is also unclear whether cortical cell-specific transcriptomes might change dynamically in absence of cell number changes. Total variation in neuron- and glia-specific in-aggregate transcriptomes could be identified in a model-free way via glia-neuron ratio approach, by univariate median-to-median ratios comparing integral levels of cell-specific transcripts within a tissue sample. When deleterious, regenerative or developmental events affecting cortical cell numbers were subtle, median-to-median ratios demonstrated within-group variability not exceeding <20-25% in most cases. These levels of total variability might be explained in part by limited (~5-10%) circadian and stress-induced shifts in cell-specific cortical transcriptomes. Relevant in-aggregate transcriptomic alterations were identified after shifts in cell numbers induced by well-validated deleterious events including ischemia, traumatic injury, microglia's activation/depletion or specific mutations. Cortical median-to-median ratios also follow naturally occurring changes in the numbers of excitatory, inhibitory neurons and glial cells during perinatal brain development. These findings characterize cortical cell-specific transcriptomes as subjects to circadian shifts and lifetime events, urging the importance of reporting full details on an origin of any transcriptomic sample collected in vivo.

Sampled-data control through model-free reinforcement learning with effective experience replay

Reinforcement Learning (RL) based control algorithms can learn the control strategies for nonlinear and uncertain environment during interacting with it. Guided by the rewards generated by environment, a RL agent can learn the control strategy directly in a model-free way instead of investigating the dynamic model of the environment. In the paper, we propose the sampled-data RL control strategy to reduce the computational demand. In the sampled-data control strategy, the whole control system is of a hybrid structure, in which the plant is of continuous structure while the controller (RL agent) adopts a discrete structure. Given that the continuous states of the plant will be the input of the agent, the state–action value function is approximated by the fully connected feed-forward neural networks (FCFFNN). Instead of learning the controller at every step during the interaction with the environment, the learning and acting stages are decoupled to learn the control strategy more effectively through experience replay. In the acting stage, the most effective experience obtained during the interaction with the environment will be stored and during the learning stage, the stored experience will be replayed to customized times, which helps enhance the experience replay process.The effectiveness of proposed approach will be verified by simulation examples.

Nonequilibrium dynamics of adaptation in sensory systems.

Adaptation is used by biological sensory systems to respond to a wide range of environmental signals, by adapting their response properties to the statistics of the stimulus in order to maximize information transmission. We derive rules of optimal adaptation to changes in the mean and variance of a continuous stimulus in terms of Bayesian filters and map them onto stochastic equationsthat couple the state of the environment to an internal variable controlling the response function. We calculate numerical and exact results for the speed and accuracy of adaptation and its impact on information transmission. We find that, in the regime of efficient adaptation, the speed of adaptation scales sublinearly with the rate of change of the environment. Finally, we exploit the mathematical equivalence between adaptation and stochastic thermodynamics to quantitatively relate adaptation to the irreversibility of the adaptation time course, defined by the rate of entropy production. Our results suggest a means to empirically quantify adaptation in a model-free and nonparametric way.

Online knowledge distillation network for single image dehazing

Single image dehazing, as a key prerequisite of high-level computer vision tasks, catches more and more attentions. Traditional model-based methods recover haze-free images via atmospheric scattering model, which achieve favorable dehazing effect but endure artifacts, halos, and color distortion. By contrast, recent learning-based methods dehaze images by a model-free way, which achieve better color fidelity but tend to acquire under-dehazed results due to lacking of knowledge guiding. To combine these merits, we propose a novel online knowledge distillation network for single image dehazing named OKDNet. Specifically, the proposed OKDNet firstly preprocesses hazy images and acquires abundant shared features by a multiscale network constructed with attention guided residual dense blocks. After that, these features are sent to different branches to generate two preliminary dehazed images via supervision training: one branch acquires dehazed images via the atmospheric scattering model; another branch directly establishes the mapping relationship between hazy images and clear images, which dehazes images by a model-free way. To effectively fuse useful information from these two branches and acquire a better dehazed results, we propose an efficient feature aggregation block consisted of multiple parallel convolutions with different receptive. Moreover, we adopt a one-stage knowledge distillation strategy named online knowledge distillation to joint optimization of our OKDNet. The proposed OKDNet achieves superior performance compared with state-of-the-art methods on both synthetic and real-world images with fewer model parameters. Project website: https://github.com/lanyunwei/OKDNet.

Deep Learning Based Model-Free Robust Load Restoration to Enhance Bulk System Resilience With Wind Power Penetration

This paper proposes a new deep learning (DL) based model-free robust method for bulk system on-line load restoration with high penetration of wind power. Inspired by the iterative calculation of the two-stage robust load restoration model, the deep neural network (DNN) and deep convolutional neural network (CNN) are respectively designed to find the worst-case system condition of a load pickup decision and evaluate the corresponding security. In order to find the optimal result within a limited number of checks, a load pickup checklist generation (LPCG) algorithm is developed to ensure the optimality. Then, the fast robust load restoration strategy acquisition is achieved based on the designed one-line strategy generation (OSG) algorithm. The proposed method finds the optimal result in a model-free way, holds the robustness to handle uncertainties, and provides real-time computation. It can completely replace conventional robust optimization and supports on-line robust load restoration which better satisfies the changeable restoration process. The effectiveness of the proposed method is validated using the IEEE 30-bus system and the IEEE 118-bus system, showing high computational efficiency and considerable accuracy.

The dynamical Matryoshka model: 3. Diffusive nature of the atomic motions contained in a new dynamical model for deciphering local lipid dynamics

In accompanying papers [Bicout et al., BioRxiv https://doi.org/10.1101/2021.09.21.461198 (2021); Cissé et al., BioRxiv https://doi.org/10.1101/2022.03.30.486370 (2022)], a new model called Matryoshka model has been proposed to describe the geometry of atomic motions in phospholipid molecules in bilayers and multilamellar vesicles based on their quasielastic neutron scattering (QENS) spectra. Here, in order to characterize the relaxational aspects of this model, the energy widths of the QENS spectra of the samples were analyzed first in a model-free way. The spectra were decomposed into three Lorentzian functions, which are classified as slow, intermediate, and fast motions depending on their widths. The analysis provides the diffusion coefficients, residence times, and geometrical parameters for the three classes of motions. The results corroborate the parameter values such as the amplitudes and the mobile fractions of atomic motions obtained by the application of the Matryoshka model to the same samples. Since the current analysis was carried out independently of the development of the Matryoshka model, the present results enhance the validity of the model. The model will serve as a powerful tool to decipher the dynamics of lipid molecules not only in model systems, but also in more complex systems such as mixtures of different kinds of lipids or natural cell membranes.

Model-Free Quantum Control with Reinforcement Learning

Model bias is an inherent limitation of the current dominant approach to optimal quantum control, which relies on a system simulation for optimization of control policies. To overcome this limitation, we propose a circuit-based approach for training a reinforcement learning agent on quantum control tasks in a model-free way. Given a continuously parameterized control circuit, the agent learns its parameters through trial-and-error interaction with the quantum system, using measurement outcomes as the only source of information about the quantum state. Focusing on control of a harmonic oscillator coupled to an ancilla qubit, we show how to reward the learning agent using measurements of experimentally available observables. We train the agent to prepare various non-classical states using both unitary control and control with adaptive measurement-based quantum feedback, and to execute logical gates on encoded qubits. This approach significantly outperforms widely used model-free methods in terms of sample efficiency. Our numerical work is of immediate relevance to superconducting circuits and trapped ions platforms where such training can be implemented in experiment, allowing complete elimination of model bias and the adaptation of quantum control policies to the specific system in which they are deployed.

Regulation With Guaranteed Convergence Rate for Continuous-Time Systems With Completely Unknown Dynamics in the Presence of Disturbance

This paper presents the design of a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -based control framework for state regulation of continuous-time linear systems with completely unknown dynamics. The proposed method solves the regulation problem with the desired convergence rate and simultaneously seeks to attenuate the adverse effect of disturbance on the system. The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> regulation problem assumes a cost function that considers regulation with a guaranteed rate of convergence as well as disturbance attenuation. The problem is then turned into a two-player zero-sum game optimization problem that can be solved by solving the associated algebraic Riccati equation (ARE), which provides a model-based solution. To solve this problem in a model-free way, a novel integral reinforcement learning (IRL) algorithm is designed to learn the solution online without requiring any prior knowledge of the system dynamics. It is shown that the model-free method (i.e., IRL-based method) provides the same solution as the model-based method (i.e., ARE). The effectiveness of the proposed method is ascertained through simulation examples; it is shown that the proposed method effectively addresses the problem for both stable and unstable systems.

PI Controller Tuning via Data-Driven Algorithms for Shape Memory Alloy Systems

The current paper proposes to compare the performances of a proportional-integral (PI) controller whose parameters are determined in a model-based way using a metaheuristic search algorithm, with the performances of a PI determined in a model-free way by using virtual reference feedback tuning (VRFT) algorithm and using iterative feedback tuning (IFT) algorithm. The PI controller is included in a control system structure to control the position of a shape memory alloy (SMA) laboratory equipment.

Measuring preferences over the temporal resolution of consumption uncertainty

Timing premia measure how much consumption people are willing to forgo to resolve all consumption uncertainty immediately. We develop a novel experiment to elicit these attitudes directly in a model-free way. On average, subjects forgo around 5% of their total consumption to resolve all uncertainty immediately. Recursive utility models postulate a structural link between timing premia and deep preference parameters. We elicit these preference parameters separately and estimate corresponding predicted timing premia. Comparing directly elicited and predicted timing premia allows us to test this structural link. Surprisingly, we find a negative correlation between predicted and elicited timing premia.

On the use of the model-free way method for kinetic analysis of thermoanalytical data – advantages and limitations

This paper analyzes the possibility of using the model-free way method (MFW method) to evaluate the kinetic triplet (activation energy (E), pre-exponential factor (A) and conversion function) of a single-step process by processing non-isothermal data recorded at several constant heating rates. For this purpose, simulated non-isothermal data and experimental thermogravimetric data obtained to investigate the thermo-oxidative degradation of a sort of polyvinyl chloride (PVC) have been kinetically analyzed. Two MFW procedures, one differential and one iterative, are presented and used to determine E and A, as well as the reconstruction of the conversion function corresponding to the investigated processes. It is highlighted the dependence of these parameters on the set of kinetic models considered for performing the application stage of the MFW procedure, which consists in evaluating the parameters of the compensation effect apparently due to the variation of reaction model. It has been shown that, for the whole range of the degree of conversion in which the value of the activation energy is constant, the values of numerically reconstructed conversion function are proportional to the real values of the conversion function. Therefore, the results obtained by the MFW procedures give indications regarding the real expression of the conversion function, but they cannot be used for the unambiguous determination of the real conversion function. To solve the problem of determining the real kinetic parameters, a modified MFW method in which the linear relation of the compensation effect is replaced by a polynomial regression of lnA vs. E has been suggested and verified. This method can lead to the real values of the conversion function even if its algebraic expression is not part of their known theoretical set.

Bootstrap Prediction Bands for Functional Time Series

A bootstrap procedure for constructing prediction bands for a stationary functional time series is proposed. The procedure exploits a general vector autoregressive representation of the time-reversed series of Fourier coefficients appearing in the Karhunen–Loève representation of the functional process. It generates backward-in-time functional replicates that adequately mimic the dependence structure of the underlying process in a model-free way and have the same conditionally fixed curves at the end of each functional pseudo-time series. The bootstrap prediction error distribution is then calculated as the difference between the model-free, bootstrap-generated future functional observations and the functional forecasts obtained from the model used for prediction. This allows the estimated prediction error distribution to account for the innovation and estimation errors associated with prediction and the possible errors due to model misspecification. We establish the asymptotic validity of the bootstrap procedure in estimating the conditional prediction error distribution of interest, and we also show that the procedure enables the construction of prediction bands that achieve (asymptotically) the desired coverage. Prediction bands based on a consistent estimation of the conditional distribution of the studentized prediction error process also are introduced. Such bands allow for taking more appropriately into account the local uncertainty of the prediction. Through a simulation study and the analysis of two datasets, we demonstrate the capabilities and the good finite-sample performance of the proposed method.

Model-Free Optimal Control of Linear Multiagent Systems via Decomposition and Hierarchical Approximation

Designing the optimal linear quadratic regulator (LQR) for a large-scale multiagent system is time consuming since it involves solving a large-size matrix Riccati equation. The situation is further exasperated when the design needs to be done in a model-free way using schemes such as reinforcement learning (RL). To reduce this computational complexity, we decompose the large-scale LQR design problem into multiple small-size LQR design problems. We consider the objective function to be specified over an undirected graph, and cast the decomposition as a graph clustering problem. The graph is decomposed into two parts, one consisting of independent clusters of connected components, and the other containing edges that connect different clusters. Accordingly, the resulting controller has a hierarchical structure, consisting of two components. The first component optimizes the performance of each independent cluster by solving the small-size LQR design problem in a model-free way using an RL algorithm. The second component accounts for the objective coupling different clusters, which is achieved by solving a least-squares problem in one shot. Although suboptimal, the hierarchical controller adheres to a particular structure as specified by interagent couplings in the objective function and by the decomposition strategy. Mathematical formulations are established to find a decomposition that minimizes the number of required communication links or reduces the optimality gap. Numerical simulations are provided to highlight the pros and cons of the proposed designs.

Machine Learning in Model-free Mechanical Property Imaging: Novel Integration of Physics With the Constrained Optimization Process

The Autoprogressive Method (AutoP) is a fundamentally different approach to solving the inverse problem in quasi-static ultrasonic elastography (QUSE). By exploiting the nonlinear adaptability of artificial neural networks and physical constraints imposed through finite element analysis, AutoP is able to build patient specific soft-computational material models from a relatively sparse set of force-displacement measurement data. Physics-guided, data-driven models offer a new path to the discovery of mechanical properties most effective for diagnostic imaging. AutoP was originally applied to modeling mechanical properties of materials in geotechnical and civil engineering applications. The method was later adapted to reconstructing maps of linear-elastic material properties for cancer imaging applications. Previous articles describing AutoP focused on high-level concepts to explain the mechanisms driving the training process. In this review, we focus on AutoP as applied to QUSE to present a more thorough explanation of the ways in which the method fundamentally differs from classic model-based and other machine learning approaches. We build intuition for the method through analogy to conventional optimization methods and explore how maps of stresses and strains are extracted from force-displacement measurements in a model-free way. In addition, we discuss a physics-based regularization term unique to AutoP that illuminates the comparison to typical optimization procedures. The insights gained from our hybrid inverse method will hopefully inspire others to explore combinations of rigorous mathematical techniques and conservation principles with the power of machine learning to solve difficult inverse problems.

Addressing indirect frequency coupling via partial generalized coherence

Distinguishing between direct and indirect frequency coupling is an important aspect of functional connectivity analyses because this distinction can determine if two brain regions are directly connected. Although partial coherence quantifies partial frequency coupling in the linear Gaussian case, we introduce a general framework that can address even the nonlinear and non-Gaussian case. Our technique, partial generalized coherence (PGC), expands prior work by allowing pairwise frequency coupling analyses to be conditioned on other processes, enabling model-free partial frequency coupling results. By taking advantage of recent advances in conditional mutual information estimation, we are able to implement our technique in a way that scales well with dimensionality, making it possible to condition on many processes and produce a partial frequency coupling graph. We analyzed both linear Gaussian and nonlinear simulated networks. We then performed PGC analysis of calcium recordings from mouse olfactory bulb glomeruli under anesthesia and quantified the dominant influence of breathing-related activity on the pairwise relationships between glomeruli for breathing-related frequencies. Overall, we introduce a technique capable of eliminating indirect frequency coupling in a model-free way, empowering future research to correct for potentially misleading frequency interactions in functional connectivity analyses.

Isoconversional computations for nonisothermal kinetic predictions

Isoconversional analysis is a model-free and powerful approach that readily provide quantitative information of the kinetics of a thermally activated transformation. Activation energy and preexponential factor are the main output of isoconversional analysis. Another practical application is the kinetic predictions that are computed in a model-free way. In this communication, a computational tool of predicting nonisothermal kinetics is presented. This approach is directly derived from an advanced isoconversional analysis. It assumes that a nonisothermal program can be decomposed in a series of infinitesimal isothermal steps. For the development of computations, four kinetic models are simulated and are used as input data. Predicted nonisothermal kinetic profiles are compared to simulated curves and show excellent agreements. Accuracy, effect of sampling and noise are assessed. This approach gives excellent predictions for the four models (among them one comprises variable activation energy). The predictions are also accurate when dealing with a multistep kinetic program (alternating isothermal and nonisothermal stages). Furthermore, the authors briefly discuss the dependance of the heating rates on the shift of nonisothermal kinetics profiles.

Model-free isothermal and nonisothermal predictions using advanced isoconversional methods

Four complex mechanisms of two parallel independent reactions were simulated under various isothermal and nonisothermal conditions. Equations to perform isothermal predictions and nonisothermal simulations at a different temperature program are presented, based on the isoconversional methods of Vyazovkin and Friedman. Isoconversional kinetic parameters were used to simulate the reaction progress under different isothermal and nonisothermal conditions. It was shown that accurate simulations can be performed in a model-free way, i.e. without knowledge of the reaction mechanism. A higher accuracy was obtained for mechanism involving more separated individual steps. The methods proposed are applicable to both DSC and TG data.