In this work, we consider the recently proposed well-defined theory that permits a healthy $D\to 4$ limit of the Einstein-Gauss-Bonnet combination, which requires the addition of a scalar degree of freedom. We continue the construction of exact, hairy black hole solutions in this theory in the presence of matter sources, by considering a nonlinear electrodynamics source, constructed through the Pleba\'nski tensor and a precise structural function $\mathcal{H}(P)$. Computing the thermodynamic quantities with the Wald formalism, we identify a region in parameter space where the hairy black holes posses well-defined, non-vanishing, finite thermodynamic quantities, in spite of the relaxed asymptotic approach to planar AdS. We test its local stability under thermal and electrical fluctuations and we also show that a Smarr relation is satisfied for these black hole configurations.