Kinesin and dynein-dynactin motors move endosomes and other vesicles bidirectionally along microtubules, a process mainly studied under invitro conditions. Here, we provide a physiological bidirectional transport model following color-coded, endogenously tagged transport-related proteins as they move through a crowded cellular environment. Late endosomes (LEs) surf bidirectionally on Protrudin-enriched endoplasmic reticulum (ER) membrane contact sites, while hopping and gliding along microtubules and bypassing cellular obstacles, such as mitochondria. During bidirectional transport, late endosomes do not switch between opposing Rab7 GTPase effectors, RILP and FYCO1, or their associated dynein and KIF5B motor proteins, respectively. In the endogenous setting, far fewer motors associate with endosomal membranes relative to effectors, implying coordination of transport with other aspects of endosome physiology through GTPase-regulated mechanisms. We find that directionality of transport is provided in part by various microtubule-associated proteins (MAPs), including MID1, EB1, and CEP169, which recruit Lis1-activated dynein motors to microtubule plus ends for transport of early and late endosomal populations. At these microtubule plus ends, activated dynein motors encounter the dynactin subunit p150glued and become competent for endosomal capture and minus-end movement in collaboration with membrane-associated Rab7-RILP. We show that endosomes surf over the ER through the crowded cell and move bidirectionally under the control of MAPs for motor activation and through motor replacement and capture by endosomal anchors.