We investigate the acceleration of particles via the second-order Fermi process in the lobes of giant radio galaxies. Such sites are candidates for the accelerators of ultra-high energy cosmic rays. We focus on the nearby FR I radio galaxy Centaurus A. This is motivated by the coincidence of its position with the arrival direction of several of the highest energy Auger events. The conditions necessary for consistency with the acceleration timescales predicted by quasi-linear theory are reviewed. Test particle calculations are performed in fields which guarantee electric fields with no component parallel to the local magnetic field. The results of quasilinear theory are, to order of magnitude, found to be accurate at low turbulence levels for non-relativistic Alfven waves and at both low and high turbulence levels in the mildly relativistic case. We conclude that for pure stochastic acceleration to be plausible as the generator of ultra-high energy cosmic rays in Centaurus A, the baryon number density would need to be several orders of magnitude below currently held upper-limits.