Time delays of electrons emitted from an isotropic initial state with the absorption of a single photon and leaving behind an isotropic ion are angle independent. Using an interferometric method involving XUV attosecond pulse trains and an IR-probe field in combination with a detection scheme, which allows for full three-dimensional momentum resolution, we show that measured time delays between electrons liberated from the $1{s}^{2}$ spherically symmetric ground state of helium depend on the emission direction of the electrons relative to the common linear polarization axis of the ionizing XUV light and the IR-probing field. Such time delay anisotropy, for which we measure values as large as 60 as, is caused by the interplay between final quantum states with different symmetry and arises naturally whenever the photoionization process involves the exchange of more than one photon. With the support of accurate theoretical models, the angular dependence of the time delay is attributed to small phase differences that are induced in the laser-driven continuum transitions to the final states. Since most measurement techniques tracing attosecond electron dynamics involve the exchange of at least two photons, this is a general and significant effect that must be taken into account in all measurements of time delays involving photoionization processes.