Shape memory polymers (SMPs) can fix a temporary shape and recover their permanent shape upon activation by an external stimulus. Most SMPs require programming at above their transition temperatures, normally well above the room temperature. In addition, most SMPs are programmed into shapes that are affine to the high temperature deformation. Recently, a cold-programmed anisotropic shape memory elastomeric composite was developed and showed interesting low temperature stretching induced shape memory behavior. There, simple, uniaxial stretching at low temperature transformed the composites into curled temporary shapes upon unloading. The exact geometry of the curled state depended on the microstructure of the composite, and the curled shape showed no affinity to the deformed shape. Heating the sample recovered the sample back to its original shape. This new composite consisted of an elastomeric matrix reinforced by aligned amorphous polymer fibers. By utilizing the plastic-like behavior of the amorphous polymer phase at low temperatures, a temporary shape could be fixed upon unloading since the induced plastic-like strain resists the recovery of the elastomer matrix. After heating to a high temperature, the permanent shape was recovered when the amorphous polymer softened and the elastomer matrix contracted. To set a theoretical foundation for capturing the cold-programmed shape memory effects and the dramatic non-affine shape change of this composite, a 3D anisotropic thermoviscoelastic constitutive model is developed in this paper. In this model, the matrix is modeled as a hyperelastic solid, and the amorphous phase of the fibrous mat is considered as a nonlinear thermoviscoplastic solid, whose viscous flow resistance is sensitive to both temperature and stress. The plastic-deformation like behavior demonstrated in the fiber is treated as nonlinear viscoplasticity with extremely high viscosity or long relaxation time at zero-stress state at low temperature. The anisotropic viscoplastic property of the fibrous mat is captured by an isotropic fibrous network superimposed with an oriented fibrous network. The material parameters in the model are identified from the experiments on the fibrous mat and on the composites, respectively. The cold-programmed shape memory behaviors of the composite are predicted by simulations and compared with experiments without further adjusting the material parameters. Good agreement is observed, indicating the ability of the present model to capture the anisotropic viscoplastic and shape memory behaviors. By using the developed constitutive model, effects of loading rate and fiber volume fraction on cold programmed shape memory behavior are discussed. Furthermore, the constitutive relation is applied to a mechanical model to study the cold-programmed curling of the laminates.