The heterostructure consisting of the Mott insulator LaVO$_3$ and the band insulator SrTiO$_3$ is considered a promising candidate for future photovoltaic applications. Not only does the (direct) excitation gap of LaVO$_3$ match well the solar spectrum, but its correlated nature and predicted built-in potential, owing to the non-polar/polar interface when integrated with SrTiO$_3$, also offer remarkable advantages over conventional solar cells. However, experimental data beyond the observation of a thickness-dependent metal-insulator transition is scarce and a profound, microscopic understanding of the electronic properties is still lacking. By means of soft and hard X-ray photoemission spectroscopy as well as resistivity and Hall effect measurements we study the electrical properties, band bending, and band alignment of LaVO$_3$/SrTiO$_3$ heterostructures. We find a critical LaVO$_3$ thickness of five unit cells, confinement of the conducting electrons to exclusively Ti 3$d$ states at the interface, and a potential gradient in the film. From these findings we conclude on electronic reconstruction as the driving mechanism for the formation of the metallic interface in LaVO$_3$/SrTiO$_3$.