When the length scale of external stimulations is comparable to most chain lengths within a polymeric solid, the nonlocal and microstructure-dependent strain-gradient effects become significant. The present work proposes a physically-based nonlocal strain gradient theory of polymer networks, where the kernel functions and intrinsic length scales have unambiguous physical meanings. The main contribution lies in the establishment of the general framework that takes in an arbitrary complete set of microscopic descriptions (chain energetics, chain-length distribution, structure of interpenetrating network) and outputs a corresponding nonlocal strain gradient constitutive relation. Based on the hypotheses of interpolatory correlation, homogenization, and superposition, the physically-based construction of constitutive relation is specified to a degree to uniquely determine two linear functionals, which are used to evaluate the nonlocal stress and nonlocal hyper-stress. Application to eight-chain interpenetrating networks yields a specific theory of nonlocal strain gradient elasticity in which the explicit functional form of the kernel is analytically derived from the chain length distribution function. It is shown that the physically derived kernel for the strain gradient field results from the superposition of a spectrum of length scales. The proposed physically-based nonlocal strain gradient theory can be beneficial to clarify the relationship between the microstructure-dependent intrinsic lengths and properties (or responses) of crosslinked polymer network structures.