Compton scattering plays an important role in various astrophysical objects such as accreting black holes and neutron stars, pulsars, and relativistic jets, clusters of galaxies as well as the early Universe. In most of the calculations it is assumed that the electrons have isotropic angular distribution in some frame. However, there are situations where the anisotropy may be significant due to the bulk motions, or anisotropic cooling by synchrotron radiation, or anisotropic source of seed soft photons. We develop here an analytical theory of Compton scattering by anisotropic distribution of electrons that can simplify significantly the calculations. Assuming that the electron angular distribution can be represented by a second order polynomial over cosine of some angle (dipole and quadrupole anisotropy), we integrate the exact Klein-Nishina cross-section over the angles. Exact analytical and approximate formulae valid for any photon and electron energies are derived for the redistribution functions describing Compton scattering of photons with arbitrary angular distribution by anisotropic electrons. The analytical expressions for the corresponding photon scattering cross-section on such electrons as well as the mean energy of scattered photons, its dispersion and radiation pressure force are also derived. We applied the developed formalism to the accurate calculations of the thermal and kinematic Sunyaev-Zeldovich effects for arbitrary electron distributions.