Sinusoidal and monotonic transfer functions: Implications for VC dimension

Abstract

It is sometimes stated that neural networks that employ units with nonmonotonic transfer functions are more difficult to train than networks that use monotonic transfer functions, because the former can be expected to have more local minima. That this is often true arises from the fact that networks using monotonic transfer functions tend to have a smaller VC (Vapnik-Chervonenkis) dimension than networks using nonmonotonic transfer junctions. But the VC dimension of a network is not solely influenced by the nature of the transfer function. We give an example of a network with an infinite VC dimension and demonstrate that it is equivalent to a network which contains only monotonic transfer functions. Thus we show that monotonicity alone is not a sufficient criterion to avoid large VC dimension.