A multi-branch thermoviscoelastic-themoviscoplastic finite deformation constitutive model incorporated with structural and stress relaxation is developed for a thermally activated shape memory polymer (SMP) based syntactic foam. In this paper, the total mechanical deformation of the foam is divided into the components of the SMP and the elastic glass microballoons by using the mixture rule. The nonlinear Adam-Gibbs model is used to describe the structural relaxation of the SMP as the temperature crosses the glass transition temperature (Tg). Further, a multi-branch model combined with the modified Eying model of viscous flow is used to capture the multitude of relaxation processes of the SMP. The deformation of the glass microballoons could be split into elastic and inelastic components. In addition, the phenomenological evolution rule is implemented in order to further characterize the macroscopic post-yield strain softening behaviors of the syntactic foam. A comparison between the numerical simulation and the thermomechanical experiment shows an acceptable agreement. Moreover, a parametric study is conducted to examine the predictability of the model and to provide guidance for reasonable design of the syntactic foam.