Brownian-type motion of helium bubbles in aluminum and its dynamical response to irradiation with 100-keV Al+ ions at high temperatures has been studied using in situ irradiation and transmission electron microscopy. It is found that, for most bubbles, the Brownian-type motion is retarded under irradiation, while the mobility returns when the irradiation is stopped. In contrast, under irradiation, a small number of bubbles display exceptionally rapid motion associated with the change in bubble size. These effects are discussed in terms of the dynamical interaction of helium bubbles with cascade damage formed by the high-energy self-ion irradiation.