We consider billiard ball motion in a convex domain of the Euclidean plane bounded by a piece-wise smooth curve under the action of a constant magnetic field. We show that if there exists a first integral polynomial in the velocities of the magnetic billiard flow, then every smooth piece γ of the boundary must be algebraic, and either is a circle or satisfies very strong restrictions. In particular, it follows that any non-circular magnetic Birkhoff billiard is not algebraically integrable for all but finitely many values of the magnitude of the magnetic field. Moreover, a magnetic billiard in ellipse is not algebraically integrable for all values of the magnitude of the magnetic field. We conjecture that the circle is the only integrable magnetic billiard, not only in the algebraic sense, but also for a broader meaning of integrability. We also introduce what we call outer magnetic billiards. As an application of our method, we prove analogous results on algebraically integrable outer magnetic billiards.