If the graviton possesses a non-zero charge qg , gravitational waves (GW) originating from astrophysical sources would experience an additional time delay due to intergalactic magnetic fields. This would result in a modification of the phase evolution of the observed GW signal similar to the effect induced by a massive graviton. As a result, we can reinterpret the most recent upper limits on the graviton's mass as constraints on the joint mass-charge parameter space, finding |qg |/e < 3 × 10-34 where e represents the charge of an electron. Additionally, we illustrate that a charged graviton would introduce a constant phase difference in the gravitational waves detected by two spatially separated GW detectors due to the Aharonov-Bohm effect. Using the non-observation of such a phase difference for the GW event GW190814, we establish a mass-independent constraint |qg |/e < 2 × 10-26. To the best of our knowledge, our results constitute the first-ever bounds on the charge of the graviton. We also discuss various caveats involved in our measurements and prospects for strengthening these bounds with future GW observations.