The olivo-cerebellar system plays an important role in vertebrate sensorimotor control. Here, we investigate sensory representations in the inferior olive (IO) of larval zebrafish and their spatial organization. Using single-cell labeling of genetically identified IO neurons, we find that they can be divided into at least two distinct groups based on their spatial location, dendritic morphology, and axonal projection patterns. In the same genetically targeted population, we recorded calcium activity in response to a set of visual stimuli using two-photon imaging. We found that most IO neurons showed direction-selective and binocular responses to visual stimuli and that the functional properties were spatially organized within the IO. Light-sheet functional imaging that allowed for simultaneous activity recordings at the soma and axonal level revealed tight coupling between functional properties, soma location, and axonal projection patterns of IO neurons. Taken together, our results suggest that anatomically defined classes of IO neurons correspond to distinct functional types, and that topographic connections between IO and cerebellum contribute to organization of the cerebellum into distinct functional zones.