Adaptation aftereffects offer a critical window onto sensory processing in the brain. However, such sensory processing is hierarchical, progressing from the extraction of simple features to the representation of complex patterns. The way that adaptation depends on coordinated changes across different levels of the hierarchy has been studied. However, when a given adapting stimulus produces both a low- and a high-level aftereffect, it remains unclear whether the high-level aftereffect is a passive reflection of low-level adaptation, or whether it is generated, at least partially, de novo in high-level areas. We assembled the two key ingredients needed for investigating this question psychophysically. One ingredient involves perceptual tasks that depend rather exclusively on low or high levels of processing, and yet involve partially identical stimuli that inspire cross-level adaptation. For this, we considered the discrimination of curvature or facial expression using curves or cartoon faces. The other ingredient is spatial or temporal stimulus manipulations that limit adaptation to either low or high levels. For this, we used crowding and brief presentations. We found that crowding an adapting curve with flanking curves reduces the curvature aftereffect much more than the facial-expression aftereffect, and vice versa for crowding the adapting face with flanking faces. Additionally, reducing adaptation time to a cartoon face diminishes the curvature aftereffect more drastically than the facial-expression aftereffect. These results suggest that high-level aftereffects, even when generated by a low-level adaptor, are not completely inherited from lower levels, and offer a window into the determining factors.