Neutral memories can be modulated via intentional memory control paradigms such as directed forgetting. In addition, previous studies have shown that neutral visual memories can be modulated indirectly, via remember and forget instructions towards competing verbal memories. Here we show that direct modulation of neutral verbal memory strength is impaired by negative visual context, and that negative visual context is resistant to indirect memory modulation. Participants were directly instructed to intentionally remember or forget newly encoded neutral verbal information. Importantly, this verbal information was interleaved with embedded negative visual context. Results showed that negative visual context eliminated the well-documented effect of direct instructions to intentionally remember verbal content. Furthermore, negative visual memory was highly persistent, overcoming its sensitivity to indirect modulation shown in previous studies. Finally, these memory effects persisted to the following day. These results demonstrate the dominance of negative visual context over neutral content, highlighting the challenges associated with memory modulation in psychopathologies involving maladaptive processing of negative visual memories.