Power grids are evolving toward a highly distributed architecture with power-electronics circuits interfacing a majority of generation, storage, and loads. To ensure stability as synchronous machines are retired, grid-forming (GFM) inverters will be needed across the board. In this article, we investigate whether systems built with interconnected single-phase droop-controlled GFM inverters are capable of self organizing into balanced three-phase systems. We model, analyze, and build a system comprised of three delta-connected droop-controlled single-phase inverters connected across loads. After deriving a model of the angle dynamics for this system, we show that its stable equilibria coincide with balanced conditions, where each phase is offset by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1/3$</tex-math></inline-formula> of an ac cycle. Furthermore, we observe that the desirable equilibrium with balanced phase offsets is robust against voltage and load imbalances. This demonstrates the feasibility of assembling three-phase systems using fleets of decentralized single-phase GFM inverters. The analytical developments are substantiated experimentally on a system of three delta-connected single-phase inverters. Spontaneous emergence of phase balancing from startup with nonidentical initial conditions are validated in the experimental setup. Empirical observations further illustrate robust operation during unbalanced conditions.