We report on the direct observation of spin-exchanging interactions in a two-orbital SU(N)-symmetric quantum gas of ytterbium in an optical lattice. The two orbital states are represented by two different (meta-)stable electronic configurations of fermionic Yb-173. A strong spin-exchange between particles in the two separate orbitals is mediated by the contact interaction between atoms, which we characterize by clock shift spectroscopy in a 3D optical lattice. We find the system to be SU(N)-symmetric within our measurement precision and characterize all relevant scattering channels for atom pairs in combinations of the ground and the excited state. Elastic scattering between the orbitals is dominated by the antisymmetric channel, which leads to the strong spin-exchange coupling. The exchange process is directly observed, by characterizing the dynamic equilibration of spin imbalances between two large ensembles in the two orbital states, as well as indirectly in atom pairs via interaction shift spectroscopy in a 3D lattice. The realization of a stable SU(N)-symmetric two-orbital Hubbard Hamiltonian opens the route towards experimental quantum simulation of condensed-matter models based on orbital interactions, such as the Kondo lattice model.