A sheet of glassy polymers placed in a solvent shows swelling behaviors quite different from that of soft polymers (rubbers and gels). (1) Non-Fickian diffusion (called case II diffusion): As solvent permeates into the sample, a sharp front is created between the swollen part and the glassy part, and it moves toward the center at constant speed. (2) Nonmonotonous swelling: The thickness of the sample first increases and then decreases toward the equilibrium value. Here we propose a theory to explain such anomalous behavior by extending the previous theory for swelling of soft gels. We regard the material as a continuum mixture of a glassy polymer network and solvent. We assume that the polymer network is a viscoelastic gel of glassy polymers, and its relaxation time depends strongly on solvent concentration. We show that this theory explains the above two characteristics of glassy polymers in a simple and unified framework. The theory predicts how the permeation speed of the solvent and the characteristic times of the swelling process depend on material parameters and experimental conditions, which can be checked experimentally.