We reexamine the range of validity of finite-size scaling in the $\phi^4$ lattice model and the $\phi^4$ field theory below four dimensions. We show that general renormalization-group arguments based on the renormalizability of the $\phi^4$ theory do not rule out the possibility of a violation of finite-size scaling due to a finite lattice constant and a finite cutoff. For a confined geometry of linear size $L$ with periodic boundary conditions we analyze the approach towards bulk critical behavior as $L \to \infty$ at fixed $\xi$ for $T > T_c$ where $\xi$ is the bulk correlation length. We show that for this analysis ordinary renormalized perturbation theory is sufficient. On the basis of one-loop results and of exact results in the spherical limit we find that finite-size scaling is violated for both the $\phi^4$ lattice model and the $\phi^4$ field theory in the region $L \gg \xi$. The non-scaling effects in the field theory and in the lattice model differ significantly from each other.