Soft gravitons produced by the expansion of de Sitter can be viewed as the Nambu-Goldstone bosons of spontaneously broken asymptotic symmetries of the de Sitter spacetime. We explicitly construct the associated charges, and show that acting with the charges on the vacuum creates a new state equivalent to a change in the local coordinates induced by the soft graviton. While the effect remains unobservable within the domain of a single observer where the symmetry is unbroken, this change is physical when comparing different asymptotic observers, or between a transformed and un-transformed initial state, consistent with the scale-dependent statistical anisotropies previously derived using semiclassical relations. We then compute the overlap, ⟨0| 0′⟩, between the unperturbed de Sitter vacuum |0⟩, and the state | 0′⟩ obtained by acting \U0001d4a9 times with the charge. We show that when \U0001d4a9→ Mp2/H2 this overlap receives order one corrections and 0⟨0| 0′⟩→ , which corresponds to an infrared perturbative breakdown after a time tdS ∼ Mp2/H3 has elapsed, consistent with earlier arguments in the literature arguing for a perturbative breakdown on this timescale. We also discuss the generalization to inflation, and rederive the 3-point and one-loop consistency relations.