We investigated temporal changes in seismic velocity associated with active seismic swarms and crustal deformations that occurred in 2006 and 2009 in the eastern Izu peninsula of Japan. To detect changes in seismic velocity during and after these swarm episodes, we used a passive image interferometry technique, by estimating the phase shift of auto‐correlation functions of continuous seismic waveforms in a frequency band of 1–3 Hz. During the 2006 and 2009 swarm episodes, a significant velocity decrease (>0.3%) was detected at the station nearest the swarm area. This velocity decrease occurred when the volumetric strain change at the surface due to magma intrusion was larger than 10−6. We could not, however, find a clear correlation between the velocity decrease and the peak ground acceleration caused by moderate earthquakes (M ∼ 5) that occurred during the swarms. We also found that during the interseismic period the velocity decrease gradually recovered, and appears to be proportional to displacement at the surface measured by GPS stations. These results suggest that the change in the subsurface velocity structure in the eastern Izu peninsula strongly reflects strain changes caused by magma intrusion into the crust.