When applied to the nonlinear matter distribution of the universe, neural networks have been shown to be very statistically sensitive probes of cosmological parameters, such as the linear perturbation amplitude ${\ensuremath{\sigma}}_{8}$. However, when used as a ``black box,'' neural networks are not robust to baryonic uncertainty. We propose a robust architecture for constraining primordial non-Gaussianity ${f}_{NL}$, by training a neural network to locally estimate ${\ensuremath{\sigma}}_{8}$, and correlating these local estimates with the large-scale density field. We apply our method to $N$-body simulations, and show that $\ensuremath{\sigma}({f}_{NL})$ is 3.5 times better than the constraint obtained from a standard halo-based approach. We show that our method has the same robustness property as large-scale halo bias: baryonic physics can change the normalization of the estimated ${f}_{NL}$, but cannot change whether ${f}_{NL}$ is detected.