Linear Function Approximation Research Articles

Structural and Statistical Analysis of Multidimensional Linear Approximations of Random Functions and Permutations

Structural and Statistical Analysis of Multidimensional Linear Approximations of Random Functions and Permutations

Finite-Sample Analysis of Off-Policy Natural Actor–Critic With Linear Function Approximation

In this paper, we develop a novel variant of natural actor-critic algorithm using off-policy sampling and linear function approximation, and establish a sample complexity of O(ε-3), outperforming all the previously known convergence bounds of such algorithms. In order to overcome the divergence due to deadly triad in off-policy policy evaluation under function approximation, we develop a critic that employs n-step TD-learning algorithm with a properly chosen n. We present finite-sample convergence bounds on this critic, which are of independent interest. Furthermore, we develop a variant of natural policy gradient under function approximation, with an improved convergence rate of O(1/T) after T iterations. Combining the finite sample bounds of the actor and the critic, we obtain an overall O(ε-3) sample complexity. Our results were derived solely based on the assumption that the behavior policy sufficiently explores the state-action space, which is a much lighter assumption compared to the related literature.

Momentary finite element for elasticity 3D problems

A description of a new 8-node finite element in the form of a hexahedron is given for solving elasticity 3D problems. This finite element has the following features. This is a linear approximation of functions in the element, one point of integration and taking into account the moments of forces in the element. The finite element is based on “rare mesh” FEM schemes—finite element schemes in the form of n-dimensional cubes (square, cube, etc.) with templates in the form of inscribed simplexes (triangle, tetrahedron, etc.). Among the rare mesh schemes, schemes in 3-dimensional and 7-dimensional spaces are successful, in which the simplex can be arranged symmetrically with respect to the center of the n-dimensional cube. The rare mesh FEM schemes have not the hourglass instability due to the fact that the template of the finite element operator has the form of a simplex. Compared to traditional linear finite elements in the form of a simplex, rare mesh schemes are more economical and converge better, since they do not have the effect of overestimated shear stiffness. Moment FEM schemes are constructed by rare mesh schemes higher dimensional projection, respectively, on a two-dimensional or three-dimensional finite element mesh. The resulting finite elements are close to the known polylinear elements and surpass them in efficiency. The schemes contain parameters that allow you to control the convergence of numerical solutions. The possibility of applying this approach to the construction of numerical schemes for solving other problems of mathematical physics is discussed.

Finite-Time Error Bounds of Biased Stochastic Approximation With Application to TD-Learning

Motivated by the recent success of reinforcement learning algorithms, this paper studies a class of biased stochastic approximation (SA) procedures under a mild “ergodicity-like” assumption on the random noise sequence. Building on a multistep Lyapunov function that looks ahead to several future updates to accommodate the stochastic perturbations (thus gaining control over the bias), we prove a general result on the convergence of the SA iterates, and use it to derive non-asymptotic bounds on the mean-square error in the case of constant stepsizes. This novel viewpoint renders finite-time analysis of biased SA algorithms under a family of stochastic perturbations possible. For direct comparison with prior work, we demonstrate these bounds by applying them to TD-learning with linear function approximation, under the Markov chain observation model. The resultant finite-time error bound for TD-learning is the first of its kind, in the sense that it holds i) for the unmodified versions (i.e., without any modification to the updates) using even nonlinear approximators; as well as for Markov chains ii) under sublinear mixing conditions and iii) starting from any initial distribution, at least one of which has to be violated for existing results to be applicable.

Network Synchronization Revisited: Time Delays in Mutually Coupled Oscillators

Coordinated and efficient operation in large, complex systems requires the synchronization of the rhythms of spatially distributed components. Such systems are the basis for critical infrastructure such as satellite navigation, mobile communications, and services like the precision time protocol and Universal Coordinated Time. Different concepts for the synchronization of oscillator networks have been proposed, in particular mutual synchronization without and hierarchical synchronization from a reference clock. Established network synchronization models in electrical engineering address the role of inevitable cross-coupling time delays for network synchronization. Mutual synchronization has been studied using linear approximations of the coupling functions of these models. We review previous work and present a general model in which we study synchronization taking into account nonlinearities and finite time delays. As a result, dynamical phenomena in networks of coupled electronic oscillators induced by time delays, such as the multistability and stabilization of synchronized states can be predicted and observed. We study the linear stability of nonlinear states and predict for which system parameters synchronized states can be stable. We use these results to discuss the implementation of mutual synchronization for complex system architectures. A key finding is that mutual synchronization can result in stable in- and anti-phase synchronized states in the presence of large time delays. We provide the condition for which such synchronized states are guaranteed to be stable.

A Simultaneous Training and Input Selection Algorithm for Classification Problems Using Piecewise Approximations

Heart disease diagnosis using few measurements is a challenging an important task considering the increasing population. Artificial Neural Networks (ANNs) are promising mathematical architectures once the training is performed in an elegant manner to avoid theoretical challenges related to high nonlinearity, nonconvexity using few input variables to ensure generalization capability. This study shows the impact of the piecewise linear approximation of nonlinear functions in ANN architecture and training problem to benefit from the mixed integer linear problem formulation for the simultaneous input selection and training to obtain mixed integer programming based ANN (MIP-ANN). Proposed formulation is further tailored through linking constraints to remove the connections from the eliminated inputs to favor parameter identifiability. A publicly available dataset is considered as a case study of whose results are also compared to traditional ANN with all inputs (FC-ANN) and a relatively more straightforward but common input selection method (SKB-ANN). The results provide a comparable performance despite significant reduction in the input space in addition to significant computational and theoretical advantages thanks to advanced formulation.

Objective-sensitive principal component analysis for high-dimensional inverse problems

We introduce a novel approach of data-driven dimensionality reduction for solving high-dimensional optimization problems, including history matching. Objective-Sensitive parameterization of the argument accounts for the corresponding change of objective function value. The result is achieved via an extension of the conventional loss function, which only quantifies approximation error over realizations. This paper contains three instances of such an approach based on Principal Component Analysis (PCA). Gradient-Sensitive PCA (GS-PCA) exploits a linear approximation of the objective function. Two other approaches solve the problem approximately within the framework of stationary perturbation theory (SPT). All the algorithms are verified and tested with a synthetic reservoir model. The results demonstrate improvements in parameterization quality regarding the reveal of the unconstrained objective function minimum. Also, we provide possible extensions and analyze the overall applicability of the Objective-Sensitive approach, which can be combined with modern parameterization techniques beyond PCA.

Prediction of nuclide accumulation spread based on the volume of enhancing magnetic resonance imaging lesion in glioblastoma patients.

11C-methionine-PET (MET) and Thallium-201 chloride-SPECT (TL) are useful for predictive proliferation ability and tumor invasion range identification in glioma patients, however they are not always possible in any hospital or country. Our study aimed to assess whether the range of MET and Tl accumulation could be predicted from the contrast-enhanced lesions in Gadolinium (Gd)-T1 weighted magnetic resonance image in glioblastoma multiforme (GBM) patients. In 25 cases, the MET-area, TL-area, O-area where MET and TL overlap, and all accumulation area (AA-area) were measured in the same axial cross section as the Gd enhanced maximum area (Gd-area). This tracing operation was repeated with all axial fusion slices, and each volume was also measured (Gd-V, MET-V, TL-V, O-V, AA-V). The maximum accumulation distance of MET and TL beyond the Gd-area was limited to within 30 mm, 35 mm, respectively. Significant positive correlations were showed in all combinations with Gd-area: MET-area (r=0.851, P<0.0001), TL-area (r=0.955, P<0.0001), O-area (r=0.935, P<0.0001) and AA-area (r=0.893, P<0.0001), respectively. All combinations with Gd-V showed significant positive correlation: MET-V (r=0.867, P<0.0001), TL-V (r=0.952, P<0.0001), O-V (r=0.935, P<0.0001) and AA-V (r=0.897, P<0.0001), respectively. Approximate tumor volume Gd-V can be calculated using the formula A * B * C / 2, where A, B, and C represent the dimensions of Gd-enhanced lesion in 3 axes perpendicular to each other. The nuclide accumulation predictive table created using the obtained linear approximation functions can be used to predict the average tumor invasion range from the Gd-V without preoperative nuclear examinations.

The Self Organization of Context for Learning in Multiagent Games

Reinforcement learning is an effective machine learning paradigm in domains represented by compact and discrete state-action spaces. In high-dimensional and continuous domains, tile coding with linear function approximation has been widely used to circumvent the curse of dimensionality, but it suffers from the drawback that human-guided identification of features is required to create effective tilings. The challenge is to find tilings that preserve the context necessary to evaluate the value of a state-action pair while limit- ing memory requirements. The technique presented in this paper addresses the difficulty of identifying context in high-dimensional domains. We have chosen RoboCup simulated soccer as a domain because its high-dimensional continuous state space makes it a formidable challenge for reinforcement learning algorithms. Using self-organizing maps and reinforcement learning in a two-pass process, our technique scales to large state spaces without requiring a large amount of domain knowledge to automatically form abstractions over the state space. Results show that our algorithm learns to play the game of soccer better than a contemporary hand-coded opponent.

Adaptive Step-Size for Online Temporal Difference Learning

The step-size, often denoted as α, is a key parameter for most incremental learning algorithms. Its importance is especially pronounced when performing online temporal difference (TD) learning with function approximation. Several methods have been developed to adapt the step-size online. These range from straightforward back-off strategies to adaptive algorithms based on gradient descent. We derive an adaptive upper bound on the step-size parameter to guarantee that online TD learning with linear function approximation will not diverge. We then empirically evaluate algorithms using this upper bound as a heuristic for adapting the step-size parameter online. We compare performance with related work including HL(λ) and Autostep. Our results show that this adaptive upper bound heuristic out-performs all existing methods without requiring any meta-parameters. This effectively eliminates the need to tune the learning rate of temporal difference learning with linear function approximation.

Distributed Value Function Approximation for Collaborative Multiagent Reinforcement Learning

In this article, we propose several novel distributed gradient-based temporal-difference algorithms for multiagent off-policy learning of linear approximation of the value function in Markov decision processes with strict information structure constraints, limiting interagent communications to small neighborhoods. The algorithms are composed of the following: first, local parameter updates based on the single-agent off-policy gradient temporal-difference learning algorithms, including the eligibility traces with state-dependent parameters and, second, linear stochastic time-varying consensus schemes, represented by directed graphs. The proposed algorithms differ in their form, definition of eligibility traces, selection of time scales, and the way of incorporating consensus iterations. The main contribution of this article is a convergence analysis based on the general properties of the underlying Feller-Markov processes and the stochastic time-varying consensus model. We prove under general assumptions that the parameter estimates generated by all the proposed algorithms weakly converge to the corresponding ordinary differential equations with precisely defined invariant sets. It is demonstrated how the adopted methodology can be applied to temporal-difference algorithms under weaker information structure constraints. The variance reduction effect of the proposed algorithms is demonstrated by formulating and analyzing an asymptotic stochastic differential equation. Specific guidelines for the communication network design are provided. The algorithms' superior properties are illustrated by characteristic simulation results.

Gradient temporal-difference learning for off-policy evaluation using emphatic weightings

The problem of off-policy evaluation (OPE) has long been advocated as one of the foremost challenges in reinforcement learning. Gradient-based and emphasis-based temporal-difference (TD) learning algorithms comprise the major part of off-policy TD learning methods. In this work, we investigate the derivation of efficient OPE algorithms from a novel perspective based on the advantages of these two categories. The gradient-based framework is adopted, and the emphatic approach is used to improve convergence performance. We begin by proposing a new analogue of the on-policy objective, called the distribution-correction-based mean square projected Bellman error (DC-MSPBE). The key to the construction of DC-MSPBE is the use of emphatic weightings on the representable subspace of the original MSPBE. Based on this objective function, the emphatic TD with lower-variance gradient correction (ETD-LVC) algorithm is proposed. Under standard off-policy and stochastic approximation conditions, we provide the convergence analysis of the ETD-LVC algorithm in the case of linear function approximation. Further, we generalize the algorithm to nonlinear smooth function approximation. Finally, we empirically demonstrate the improved performance of our ETD-LVC algorithm on off-policy benchmarks. Taken together, we hope that our work can guide the future discovery of a better alternative in the off-policy TD learning algorithm family.

Displacement Measurement in the Vertical Axis of the Measuring Microscope using Laser Triangulation Sensor

The article deals with assessment of the suitability of using a laser triangulation sensor to measure the displacement in vertical axis of a measuring microscope. Data obtained from sensor calibration are used for determination of measuring range in which the smallest measurement error occurs. Linear approximation of the inverse calibration function is applied to correct systematic errors in reduced measuring range of the sensor. Residual measurement errors of corrected sensor output can be applied in uncertainty calculation for sensor future measurements.

Automatic Generation Control for Distributed Multi-Region Interconnected Power System with Function Approximation

Solving the energy crisis and environmental pollution requires large-scale access to distributed energy and the popularization of electric vehicles. However, distributed energy sources and loads are characterized by randomness, intermittence and difficulty in accurate prediction, which bring great challenges to the security, stability and economic operation of power system. Therefore, this paper explores an integrated energy system model that contains a large amount of new energy and combined cooling heating and power (CCHP) from the perspective of automatic generation control (AGC). Then, a gradient Q(σ,λ) [GQ (σ,λ)] algorithm for distributed multi-region interconnected power system is proposed to solve it. The proposed algorithm integrates unified mixed sampling parameter and linear function approximation on the basis of the Q(λ) algorithm with characteristics of interactive collaboration and self-learning. The GQ (σ,λ) algorithm avoids the disadvantages of large action spaces required by traditional reinforcement learning, so as to obtain multi-region optimal cooperative control. Under such control, the energy autonomy of each region can be achieved, and the strong stochastic disturbance caused by the large-scale access of distributed energy to grid can be resolved. In this paper, the improved IEEE two-area load frequency control (LFC) model and the integrated energy system model incorporating a large amount of new energy and CCHP are used for simulation analysis. Results show that compared with other algorithms, the proposed algorithm has optimal cooperative control performance, fast convergence speed and good robustness, which can solve the strong stochastic disturbance caused by the large-scale grid connection of distributed energy.

Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling

Despite the wide applications of Adam in reinforcement learning (RL), the theoretical convergence of Adam-type RL algorithms has not been established. This paper provides the first such convergence analysis for two fundamental RL algorithms of policy gradient (PG) and temporal difference (TD) learning that incorporate AMSGrad updates (a standard alternative of Adam in theoretical analysis), referred to as PG-AMSGrad and TD-AMSGrad, respectively. Moreover, our analysis focuses on Markovian sampling for both algorithms. We show that under general nonlinear function approximation, PG-AMSGrad with a constant stepsize converges to a neighborhood of a stationary point at the rate of O(1/T) (where T denotes the number of iterations), and with a diminishing stepsize converges exactly to a stationary point at the rate of O(log^2 T/√T). Furthermore, under linear function approximation, TD-AMSGrad with a constant stepsize converges to a neighborhood of the global optimum at the rate of O(1/T), and with a diminishing stepsize converges exactly to the global optimum at the rate of O(log T/√T). Our study develops new techniques for analyzing the Adam-type RL algorithms under Markovian sampling.

ABO4 type scheelite phases in (Ca/Sr)MoO4 - BiVO4 - Bi2Mo3O12 systems: synthesis, structure and optical properties

The cation deficient complex oxides of (Ca/Sr)MoO4 - BiVO4 - Bi2Mo3O12 triple system are promising photocatalysts and pigments. Compounds with general formula of Ca1−1.5x-yBix+yФxMo1-yVyO4 and Sr1−1.5x-yBix+yФxMo1-yVyO4 were synthesized byconvention solid state technique in the range of 550–720 °C. Two wide regions of the solid solutions (ordinary and superstructured scheelite-type phases respectively) were found for each system. The diffuse scatering of homogeneous samples was investigated in the range of 190–1100 nm. Energy gaps calculated with linear approximation of Kubelka-Munk function decreases with bismuth and vanadium content.

Gaussian Based Non-linear Function Approximation for Reinforcement Learning

Reinforcement learning (RL) problems with continuous states and discrete actions (CSDA) can be found in classic examples such as Cart Pole and Puck World, as well as real world applications such as Market Making. Solutions to CSDA problems typically involve a function approximation (FA) of the mapping from states to actions and can be linear or nonlinear. Linear FAs such as tile-coding (Sutton and Barto in Reinforcement learning, 2nd ed, 2009) suffer from state information loss due to state discretization, whilst non-linear FAs such as DQN (Mnih et al. in Playing atari with deep reinforcement learning, https://arxiv.org/abs/1312.5602, 2013) are practically infeasible in infinitely large state spaces due to their cubic time complexity (O(n^3)). In this paper, we propose a novel, general solution to CSDA problems, called Gaussian distribution based non-linear function approximation (GBNLFA). Experimentation on three CSDA RL problems (Cart Pole, Puck World, Market Marking) demonstrates the superiority of GBNLFA over state-of-the-art FAs, namely tile-coding and DQN. In particular, GBNLFA resolves the state information loss problem with linear FAs and provides an asymptotically faster algorithm (O(n)) than linear FAs (O(n^2)) and neural network based nonlinear FAs (O(n^3)).

Sampling based approximation of linear functionals in reproducing kernel Hilbert spaces

In this paper we analyze a greedy procedure to approximate a linear functional defined in a Reproducing Kernel Hilbert Space by nodal values. This procedure computes a quadrature rule which can be applied to general functionals, including integration functionals. For a large class of functionals, we prove convergence results for the approximation by means of uniform and greedy points which generalize in various ways several known results. A perturbation analysis of the weights and node computation is also discussed. Beyond the theoretical investigations, we demonstrate numerically that our algorithm is effective in treating various integration densities, and that it is even very competitive when compared to existing methods for Uncertainty Quantification.

A Case study of Exergy Losses of a Ground Heat Pump and Photovoltaic Cells System and Their Optimization

The aim of this scientific research is to experimentally determine the exergy losses of a ground heat pump and further optimization for more efficient use of operating modes and improvement of individual structural elements. In addition, it is proposed to use photovoltaic panels as a backup power source for the experimental installation under study. The exergetic losses are calculated, not only for the ground heat pump itself, with R407C refrigerant. The research methodology consists in a comprehensive assessment of exergetic flows, their optimization using new methods of approximation of piecewise linear functions, and the development of prerequisites for the use of anergy as one of the components of a new type of analysis of the efficiency of low-potential energy sources. As a result of processing the experimental data, the values of Coefficient of performance (COP) 4.136, exergetic temperature for the lower heat source 0.0253 and for the upper heat source 0.155, exergetic efficiency of the installation 0.62, and total loss of specific exergy of the heat pump 24.029 kJ/kg were obtained. Controllers with the Modbus protocol were used for data collection. Matlab Simulink was used to process the experimental data. When carrying out the procedure for optimizing the operating modes and selecting several modes with minimal exergetic losses, an important role is given to mathematical methods of processing statistical data. The method of increasing the efficiency of the heat pump is shown, first of all, based on the use of photovoltaic panels as a backup power source and optimization of exergetic losses due to exergo-anergetic evaluation of operating modes. The authors present the measurement errors of the heat pump plant parameters in the form of a 3D Gaussian curve, which becomes possible only when applying new approximation methods in the processing of measurements.

ИССЛЕДОВАНИЕ ВОЗМОЖНОСТИ ПРИМЕНЕНИЯ ЛИНЕЙНОГО АНАЛИЗА К ARX АЛГОРИТМАМ СТОХАСТИЧЕСКОГО ПРЕОБРАЗОВАНИЯ ДАННЫХ В ЗАВИСИМОСТИ ОТ ФУНКЦИИ СМЕШЕНИЯ С РАУНДОВЫМ КЛЮЧОМ

ARX stochastic algorithms are a promising solution in the development of unpredictable pseudo- random number generators for low-resource systems. The ease of implementation of their round operations, as well as their high energy efficiency, make the choice of such algorithms attractive for ensuring the confidentiality of information. Existing studies on the possibility of applying linear analysis to ARX stochastic algorithms use imprecise linear approximations of round functions. The key idea of linear analysis is the use of linear statistical analogs of non-linear functions. Linear approximations are used to express the inputs of the stochastic transformation algorithm by its outputs as a linear function. The resulting linear function is true with a certain probability, which depends on the probability of fulfilling the used linear approximations. The only non-linear operation in ARX algorithms is addition modulo 2n. In this paper we study the limitations of the applicability of the linear analysis to ARX stochastic transformation algorithms. The study is carried out for various cases of the implementation of round key addition: the use of the addition modulo 2n, the addition modulo 2, and the cyclic shift. For ARX stochastic transformation algorithms, using addition modulo 2n or addition modulo 2 as a round key mix operation, estimates are obtained for the number of addition operations modulo 2n needed to ensure their stability to linear analysis.