Abstract Interaction with the interstellar medium (ISM) cannot be ignored in understanding planetary nebula (PN) evolution and shaping. In an effort to understand the range of shapes observed in the outer envelopes of PNe, we have run a comprehensive set of three-dimensional hydrodynamic simulations, from the beginning of the asymptotic giant branch (AGB) superwind phase until the end of the post-AGB/PN phase. A ‘triple-wind’ model is used, including a slow AGB wind, fast post-AGB wind and third wind reflecting the linear movement through the ISM. A wide range of stellar velocities, mass-loss rates and ISM densities have been considered. We find that ISM interaction strongly affects outer PN structures, with the dominant shaping occurring during the AGB phase. The simulations predict four stages of PN–ISM interaction whereby (i) the PN is initially unaffected, (ii) then limb-brightened in the direction of motion, (iii) then distorted with the star moving away from the geometric centre, and (iv) finally so distorted that the object is no longer recognizable as a PN and may not be classed as such. Parsec-size shells around PNe are predicted to be common. The structure and brightness of ancient PNe are largely determined by the ISM interaction, caused by rebrightening during the second stage; this effect may address the current discrepancies in Galactic PN abundance. The majority of PNe will have tail structures. Evidence for strong interaction is found for all known PNe in globular clusters.