We propose a semiparametric factor model, which approximates the implied volatility surface (IVS) in a finite dimensional function space. Unlike standard principal component approaches typically used to reduce complexity, our approach is tailored to the degenerated design of IVS data. In particular, we only fit in the local neighborhood of the design points by exploiting the expiry effect present in option data. Using DAX index option data, we estimate the nonparametric components and a low-dimensional time series of latent factors. The modeling approach is completed by studying vector autoregressive models fitted to the latent factors.