We analyze the origin of the structure observed in the reaction $J/\psi \to \gamma \eta'\pi^+\pi^-$ for $\eta'\pi^+\pi^-$ invariant masses close to the antiproton-proton ($\bar pp$) threshold, commonly associated with the $X(1835)$ resonance. Specifically, we explore the effect of a possible contribution from the two-step process $J/\psi \to \gamma \bar NN \to \gamma \eta'\pi^+\pi^-$. The calculation is performed in distorted-wave Born approximation which allows an appropriate inclusion of the $\bar NN$ interaction in the transition amplitude. The $\bar NN$ amplitude itself is generated from a corresponding potential recently derived within chiral effective field theory. We are able to reproduce the measured spectra for the reactions $J/\psi \to \gamma \bar pp$ and $J/\psi \to \gamma \eta'\pi^+\pi^-$ for invariant masses around the $\bar pp$ threshold. The structure seen in the $\eta'\pi^+\pi^-$ spectrum emerges as a threshold effect due to the opening of the $\bar pp$ channel.