Factorization violating effects in hadron scattering are due mainly to spectator-spectator interactions. While it is known that these interactions cancel in inclusive cross sections, like for the Drell-Yan process, not much is known about for what classes of observables factorization is violated. We show that for pure Glauber ladder graphs, all amplitude-level factorization violating effects completely cancel at cross section level for any single-scale observable (such as hadronic transverse energy or beam thrust). This result disproves previous claims that these pure Glauber graphs are factorization-violating. Our proof exploits scale invariance of two-to-two scattering amplitudes in an essential way. The leading factorization-violating effects therefore come from graphs with at least one soft gluon, involving the Lipatov vertex off of the Glauber ladders. This implies that real soft radiation must be involved in factorization-violation, shedding light on the connection between factorization-violation and the underlying event.