Childhood adversity, a prevalent experience, is related to a higher risk for externalizing and internalizing psychopathology. Alterations in the development of cognitive processes, for example in the attention-interference domain may link childhood adversity and psychopathology. Interfering stimuli can vary in their salience, i.e. ability to capture attentional focus, and valence. However, it is not known if interference by salience or valence is associated with self-reported adversity. In two independent study samples of healthy men (Study 1: n = 44; mean age [standard deviation (SD)] = 25.9 [3.4] years; Study 2: n = 37; 43.5 [9.7] years) we used the attention modulation task (AMT) that probed interference by two attention-modulating conditions, salience and valence separately across repeated target stimuli. The AMT measures the effects of visual distractors (pictures) on the performance of auditory discrimination tasks (target stimuli). We hypothesized that participants reporting higher levels of childhood adversity, measured with the childhood trauma questionnaire, would show sustained interference in trials with lower salience. Due to conflicting reports on the valence-modulation, we tested the valence condition in an exploratory manner. Linear mixed models revealed an interaction between reported childhood adversity and the salience condition across tone presentations in both study samples (Sample 1: p = .03; Sample 2: p = .04), while there were no effects for the valence condition across both studies. Our study suggests that higher self-reported childhood adversity is related to faster processing of target cues during high salience, but slower during low salience conditions. These results hint to the mechanisms linking childhood adversity and psychopathological symptoms in the attentional domain.