In this paper we propose and analyze a new, fast method for the numerical solution of time domain boundary integral formulations of the wave equation. We employ Lubich's convolution quadrature method for the time discretization and a Galerkin boundary element method for the spatial discretization. The coefficient matrix of the arising system of linear equations is a triangular block Toeplitz matrix. Possible choices for solving the linear system arising from the above discretization include the use of fast Fourier transform (FFT) techniques and the use of data-sparse approximations. By using FFT techniques, the computational complexity can be reduced substantially while the storage cost remains unchanged and is, typically, high. Using data-sparse approximations, the gain is reversed; i.e., the computational cost is (approximately) unchanged while the storage cost is substantially reduced. The method proposed in this paper combines the advantages of these two approaches. First, the discrete convolution (related to the block Toeplitz system) is transformed into the (discrete) Fourier image, thereby arriving at a decoupled system of discretized Helmholtz equations with complex wave numbers. A fast data-sparse (e.g., fast multipole or panel-clustering) method can then be applied to the transformed system. Additionally, significant savings can be achieved if the boundary data are smooth and time-limited. In this case the right-hand sides of many of the Helmholtz problems are almost zero, and hence can be disregarded. Finally, the proposed method is inherently parallel. We analyze the stability and convergence of these methods, thereby deriving the choice of parameters that preserves the convergence rates of the unperturbed convolution quadrature. We also present numerical results which illustrate the predicted convergence behavior.