Numerical studies of the N\'eel to valence-bond solid phase transition in 2D quantum antiferromagnets give strong evidence for the remarkable scenario of deconfined criticality, but display strong violations of finite-size scaling that are not yet understood. We show how to realise the universal physics of the Neel-VBS transition in a 3D classical loop model (this includes the interference effect that suppresses N\'eel hedgehogs). We use this model to simulate unprecedentedly large systems (of size $L\leq 512$). Our results are compatible with a direct continuous transition at which both order parameters are critical, and we do not see conventional signs of first-order behaviour. However, we find that the scaling violations are stronger than previously realised and are incompatible with conventional finite-size scaling over the size range studied, even if allowance is made for a weakly/marginally irrelevant scaling variable. In particular, different determinations of the anomalous dimensions $\eta_\text{VBS}$ and $\eta_\text{N\'eel}$ yield very different results. The assumption of conventional finite-size scaling gives estimates which drift to negative values at large $L$, in violation of unitarity bounds. In contrast, the behaviour of correlators on scales much smaller than $L$ is consistent with large positive anomalous dimensions. Barring an unexpected reversal in behaviour at still larger sizes, this implies that the transition, if continuous, must show unconventional finite-size scaling, e.g. from a dangerously irrelevant scaling variable. Another possibility is an anomalously weak first-order transition. By analysing the renormalisation group flows for the non-compact $CP^{n-1}$ model ($n$-component Abelian Higgs model) between two and four dimensions, we give the simplest scenario by which an anomalously weak first-order transition can arise without fine-tuning of the Hamiltonian.