A high-speed, low-loss modular cross-bar switching element suitable for single photons is demonstrated. Entanglement preserving voltage-driven switching of single photons is achieved with a unique combination of ultra-fast and low-loss characteristics (less than 50 ps 10%–90% transition time and greater than 90% transmission), and we present a roadmap towards even lower losses. The switch properties are well matched to the wavelength, bandwidth and lifetime of photons generated by spontaneous four-wave mixing sources also realized in silicon photonics. Such a switch can enable reconfigurability of each single photon and addressing of each time bin, and benefit future quantum photonic circuits.