Azobenzene-containing crosslinked liquid crystalline polymer (CLCP) exhibits macroscopic deformation through irradiation with light, and recently, carbon nanotubes (CNTs) were inserted to improve the efficiency of the photo-mechanical energy conversion. We conducted molecular dynamics simulations to study the effects of the arrangement of the single-walled CNT (SWCNT) and photo-isomerization of the photochromic moieties on the opto-mechanical properties of the composite. We characterized the composites in terms of the interfacial shear strength (IFSS), photostrain, and elastic modulus according to the isomerization ratio. The SWCNT-CLCP composite enables 19–29% larger photostrain and 20–41% higher modulus compared to those of neat azo-PRP. As the angle between the CNT axis and the nematic director increases, mechanical reinforcing effect decreases. Nevertheless, the applicable photo-shrinkage increases because more space for polymer diffusion is made by the nematic defects in the interphase. The isomerization of azobenzene weakens the interfacial adhesion due to the decrease in the π-π interaction. However, changes in the elastic stiffness of the composite can be modulated by adjusting the arrangement between the SWCNT and matrix. These results provide insight into the mechanism of changes in the interfacial and opto-mechanical properties of the CNT-PRP composite caused by photo-isomerization and the mechanical design of a photo-responsive actuator.