Shape memory polymer composites (SMPCs) are emerging as an attractive material for space deployable structures due to their variable stiffness, programmable deformation, high packing ratio, and controllable deployment. However, the research on the deformation behaviors and damage mechanisms of SMPC was scarce. The buckling behaviors and damage mechanisms of out-of-plane buckling and in-plane buckling of fiber-reinforced SMPC were investigated in this study. The expression and evolution law of strain energy and key parameters of fiber buckling were studied. The effects of material parameters and size parameters on damage mode and critical damage curvature were analyzed quantitatively. It was found that in-plane buckling was more likely to occur without external constraints. The damage modes of in-plane buckling are classified as delamination, matrix cracking and fiber tensile fracture. Out-of-plane buckling is more susceptible to damage and has one more fiber buckling fracture damage mode than in-plane buckling. When the fiber volume content is below a certain value, only fiber tensile fracture damage mode will appear, regardless of the thickness. Delamination is more likely to occur in cases of out-of-plane buckling, and matrix cracking is more frequent in cases of in-plane buckling. Subsequently, the correctness of the theoretical analysis was verified by the bending test. The theoretical analysis in this study provides a theoretical foundation for SMPCs-based structural design.