In this paper we study intermittency for the parabolic Anderson equation $\partial u/\partial t=\kappa\Delta u+\gamma\xi u$ with $u:\mathbb{Z}^d\times[0,\infty)\to\mathbb{R}$, where $\kappa\in[0,\infty)$ is the diffusion constant, $\Delta$ is the discrete Laplacian, $\gamma\in(0,\infty)$ is the coupling constant, and $\xi:\mathbb{Z}^d\times[0,\infty)\to\mathbb{R}$ is a space--time random medium. The solution of this equation describes the evolution of a ``reactant'' $u$ under the influence of a ``catalyst'' $\xi$. We focus on the case where $\xi$ is the voter model with opinions 0 and 1 that are updated according to a random walk transition kernel, starting from either the Bernoulli measure $\nu_{\rho}$ or the equilibrium measure $\mu_{\rho}$, where $\rho\in(0,1)$ is the density of 1's. We consider the annealed Lyapunov exponents, that is, the exponential growth rates of the successive moments of $u$. We show that if the random walk transition kernel has zero mean and finite variance, then these exponents are trivial for $1\leq d\leq4$, but display an interesting dependence on the diffusion constant $\kappa$ for $d\geq 5$, with qualitatively different behavior in different dimensions. In earlier work we considered the case where $\xi$ is a field of independent simple random walks in a Poisson equilibrium, respectively, a symmetric exclusion process in a Bernoulli equilibrium, which are both reversible dynamics. In the present work a main obstacle is the nonreversibility of the voter model dynamics, since this precludes the application of spectral techniques. The duality with coalescing random walks is key to our analysis, and leads to a representation formula for the Lyapunov exponents that allows for the application of large deviation estimates.