Spherical means are a widespread model in modern imaging modalities like photoacoustic tomography. Besides direct inversion methods for specific geometries, iterative methods are often used as reconstruction scheme such that each iteration asks for the efficient and accurate computation of spherical means. We consider a spectral discretization via trigonometric polynomials such that the computation can be done via nonequispaced fast Fourier transforms. Moreover, a recently developed sparse fast Fourier transform is used in the three dimensional case and gives optimal arithmetic complexity. All theoretical results are illustrated by numerical experiments.