Semiconductor heterojunctions have gained significant attention for efficient optoelectronic devices owing to their unique interfaces and synergistic effects. Interaction between charge carriers with the heterojunction plays a crucial role in determining device performance, while its spatial-temporal mapping remains lacking. In this study, we employ scanning ultrafast electron microscopy (SUEM), an emerging technique that combines high spatial-temporal resolution and surface sensitivity, to investigate photocarrier dynamics across a Si/Ge heterojunction. Charge dynamics are selectively examined across the junction and compared to far bulk areas, through which the impact of the built-in potential, band offsets, and surface effects is directly visualized. In particular, we find that the heterojunction drastically modifies the hot photocarrier diffusivities in both Si and Ge regions due to charge trapping. These findings are further elucidated with insights from the band structure and surface potential measured by complementary techniques. This work demonstrates the tremendous effect of heterointerfaces on hot photocarrier dynamics and showcases the potential of SUEM in characterizing realistic optoelectronic devices.