Dynamics Of On-line Learning Research Articles

Dynamics of Online Learning during COVID-19 in Indonesia: Bibliometric Analysis and Future Research Agenda

Dynamics of Online Learning during COVID-19 in Indonesia: Bibliometric Analysis and Future Research Agenda

An empirical study into finding optima in stochastic optimization of neural networks

Mini-batch sub-sampling (MBSS) in neural network training is unavoidable due to growing data demands, memory-limited computational resources such as graphical processing units, and the dynamics of on-line learning. This study distinguishes between static MBSS and dynamic MBSS. In static MBSS, mini-batches are intermittently fixed during training, resulting in smooth but biased loss functions. During dynamic MBSS, mini-batches are resampled at every loss evaluation, resulting in sampling induced discontinuities by trading sampling bias for sampling variance. This renders classical minimization strategies ineffective in dynamic MBSS losses, as these may locate spurious sampling induced minima, while critical points may not exist. This paper re-evaluates the information used to define optima in stochastic loss functions of neural networks by defining the solution to a stochastic optimization problem as the stochastic non-negative associated gradient projection point (SNN-GPP). We demonstrate that SNN-GPPs offer a more robust description of full-batch optima than minimizers and critical points. An empirical investigation compares local minima to SNN-GPPs for the potential training of a simple neural network training problem with different activation functions. Since SNN-GPPs better approximate the location of true optima, we conclude that line searches locating SNN-GPPs can contribute significantly to automating neural network training.

The dynamics of online learning at the workplace: Peer‐facilitated social learning and the application in practice

AbstractResearch stresses the importance of social components in learning. The social contact with peers and tutors stimulates reflection and supports higher processes of learning necessary for the internalisation and application of new knowledge. However, merely proposing opportunities for interaction does not necessarily lead to fruitful discussion and collaboration. Social presence and facilitation are key concepts for successful mutual learning. Both are represented in Murphy’s collaboration model; social presence forms the basis of collaboration on which discussions and co‐construction of knowledge evolve. Facilitation supports the entire collaboration process. In this paper, an adjusted version of Murphy’s model was applied to analyse 1085 comments shared in an online course between career practitioners of a public employment service. The results show that without a dedicated tutor, learners can still be involved in collaborative learning and co‐construction of new knowledge provided that the topic under discussion is highly relevant and controversial. Learners themselves take over social presence and facilitation activities, but less frequently than when a professional tutor facilitates discussions. Ex post summative evaluation revealed that only a few learners applied the gained knowledge in the long‐term. As comparisons with related research suggest, higher facilitation support leading to a higher cognitive interaction with the learning could have better supported the transfer to practice.

Theoretical and numerical analysis of learning dynamics near singularity in multilayer perceptrons

The multilayer perceptron is one of the most widely used neural networks in applications, however, its learning behavior often becomes very slow, which is due to the singularities in the parameter space. In this paper, we analyze the learning dynamics near singularities in multilayer perceptrons by using traditional methods. We obtain the explicit expressions of the averaged learning equations which play a significant role in theoretical and numerical analysis. After obtaining the best approximation on overlap singularity, the stability of overlap singularity is analyzed. Then we take the numerical analysis on singular regions. Real averaged dynamics near the singularities are obtained in comparison with the theoretical learning trajectories near singularity. In the simulation we analyze the averaged learning dynamics, batch mode learning dynamics and on-line learning dynamics, respectively.

Theory of neural information processing systems

I INTRODUCTION TO NEURAL NETWORKS 1. General introduction 2. Layered networks 3. Recurrent networks with binary neurons II ADVANCED NEURAL NETWORKS 4. Competitive unsupervised learning processes 5. Bayesian techniques in supervised learning 6. Gaussian processes 7. Support vector machines for binary classification III INFORMATION THEORY AND NEURAL NETWORKS 8. Measuring information 9. Identification of entropy as an information measure 10. Building blocks of Shannon's information theory 11. Information theory and statistical inference 12. Applications to neural networks IV MACROSCOPIC ANALYSIS OF DYNAMICS 13. Network operation: macroscopic dynamics 14. Dynamics of online learning in binary perceptrons 15. Dynamics of online gradient descent learning V EQUILIBRIUM STATISTICAL MECHANICS OF NEURAL NETWORKS 16. Basics of equilibrium statistical mechanics 17. Network operation: equilibrium analysis 18. Gardner theory of task realizability APPENDICES A. Historical and bibliographical notes B. Probability theory in a nutshell C. Conditions for central limit theorem to apply D. Some simple summation identities E. Gaussian integrals and probability distributions F. Matrix identities G. The delta-distribution H. Inequalities based on convexity I. Metrics for parametrized probability distributions J. Saddle-point integration REFERENCES

Slow Dynamics Due to Singularities of Hierarchical Learning Machines

Recently, slow dynamics in learning of neural networks has been known to be closely related to singularities, which exist in parameter spaces of hierarchical learning models. To show the influence of singular structure on learning dynamics, we take statistical mechanical approaches and investigate online-learning dynamics under various learning scenario with different relationship between optimum and singularities. From the investigation, we found a quasi-plateau phenomenon which differs from the well known plateau. The quasi-plateau and plateau become extremely serious when an optimal point is in a neighborhood of a singularity. The quasi-plateau and plateau disappear in the natural gradient learning, which takes singular structures into account and uses Riemannian measure for the parameter space.

On-line learning of unrealizable tasks.

The dynamics of on-line learning is investigated for structurally unrealizable tasks in the context of two-layer neural networks with an arbitrary number of hidden neurons. Within a statistical mechanics framework, a closed set of differential equations describing the learning dynamics can be derived, for the general case of unrealizable isotropic tasks. In the asymptotic regime one can solve the dynamics analytically in the limit of a large number of hidden neurons, providing an analytical expression for the residual generalization error, the optimal and critical asymptotic training parameters, and the corresponding prefactor of the generalization error decay.

Dynamics of on-line learning in radial basis function networks

On-line learning is examined for the radial basis function network, an important and practical type of neural network. The evolution of generalization error is calculated within a framework which allows the phenomena of the learning process, such as the specialization of the hidden units, to be analyzed. The distinct stages of training are elucidated, and the role of the learning rate described. The three most important stages of training, the symmetric phase, the symmetry-breaking phase, and the convergence phase, are analyzed in detail; the convergence phase analysis allows derivation of maximal and optimal learning rates. As well as finding the evolution of the mean system parameters, the variances of these parameters are derived and shown to be typically small. Finally, the analytic results are strongly confirmed by simulations.

On-line learning in parity machines

The optimal algorithm for on-line learning in the tree K-parity machine is studied. We introduce a set of recursion relations for the relevant probability distributions, which permit study of the general K case. The generalization error curve is determined and shown to decay to zero for large as even in the presence of noise. There is no critical noise level. The dynamics of on-line learning is studied analytically near the origin. In the absence of previous knowledge, the learning dynamics has a fixed point at . Previous knowledge is needed in at least K - 1 branches for the learning to take place.