Flowering plants rely on the polarized growth of pollen tubes to deliver sperm cells (SCs) to the embryo sac for double fertilization. In pollen, the vegetative nucleus (VN) and two SCs form the male germ unit (MGU). However, the mechanism underlying directional transportation of MGU is not well understood. In this study, we provide the first full picture of the dynamic interplay among microtubules, actin filaments, and MGU during pollen germination and tube growth. Depolymerization of microtubules and inhibition of kinesin activity result in an increased velocity and magnified amplitude of VN’s forward and backward movement. Pharmacological washout experiments further suggest that microtubules participate in coordinating the directional movement of MGU. In contrast, suppression of the actomyosin system leads to a reduced velocity of VN mobility but without a moving pattern change. Moreover, detailed observation shows that the direction and velocity of VN’s movement are in close correlations with those of the actomyosin-driven cytoplasmic streaming surrounding VN. Therefore, we propose that while actomyosin-based cytoplasmic streaming influences on the oscillational movement of MGU, microtubules and kinesins avoid MGU drifting with the cytoplasmic streaming and act as the major regulator for fine-tuning the proper positioning and directional migration of MGU in pollen.