This paper describes the relationship between the first non-vanishing Milnor invariants of a classical link and the intersection invariant of a twisted Whitney tower. This is a certain 2-complex in the 4-ball, built from immersed disks bounded by the given link in the 3-sphere together with finitely many `layers' of Whitney disks. The intersection invariant is a higher-order generalization of the intersection number between two immersed disks in the 4-ball, well known to give the linking number of the link on the boundary, which measures intersections among the Whitney disks and the disks bounding the given link, together with information that measures the twists (framing obstructions) of the Whitney disks. This interpretation of Milnor invariants as higher-order intersection invariants plays a key role in the classifications of both the framed and twisted Whitney tower filtrations on link concordance (as sketched in this paper). Here we show how to realize the higher-order Arf invariants, which also play a role in the classifications, and derive new geometric characterizations of links with vanishing Milnor invariants of length less than or equal to 2k.