A pricing model is tied to its ability of capturing the dynamics of the spot price process. Its misspecification will lead to pricing and hedging errors. Parametric pricing formula depends on the particular form of the dynamics of the underlying asset. For tractability reasons, some assumptions are made which are not consistent with the multifractal properties of market returns. On the other hand, non-parametric models such as neural networks use market data to estimate the implicit stochastic process driving the spot price and its relationship with contingent claims. We propose to use neural networks for learning the dynamics of the implied volatility surface. Rather than learning the dynamics of the whole volatility surface, we perform a dimensionality reduction and choose to learn that of the main volatility risk factors from the data. These risk factors form a vector basis for the implied volatility surface. They can be the first three eigenmodes of a PCA decomposition, the parameters of a polynomial model, or any other model parameters from a stochastic volatility model. We assume that these factors are driven by explanatory variables, such as spot price, volume, estimated volatility, and financial indicators like the VIX and options from the main indices. We use neural networks to capture the relation between the volatility risk factors and their explanatory variables. We model the dynamics of the IV surfaces by letting the parameters of a polynomial model, or a stochastic volatility model with an explicit expression for the smile such as the SVI and the SABR model, be statistically evolved, by considering one neural network per parameter. The smile is reconstructed by using the polynomial model, or the parametric smile representation, where each parameter is replaced by a trained network. That is, not only do we learn the model parameters at a fixed time, but we use the fact that the networks are universal interpolators to learn about the dynamics of their term-structures. We can then reconstruct the volatility surface and use it for pricing and hedging options as well as for risk analysis, forecasting, and volatility trading. We also describe how to use the model for pricing forward starting contracts such as cliquet options.