The molecular dynamics and optical switching behavior (photo alignment) of novel azobenzene containing side-group (co)polymers were studied by time-resolved optical spectroscopy and broadband dielectric spectroscopy (BDS). To elucidate the effect of the molecular structure on the photochromic properties, two series of poly(methacrylates) with different aliphatic spacer units and four different concentrations of the chromophoric monomer were investigated. For both series, an inverse relation between the switching (retardation and relaxation) times and the chromophore content was found as well as a strong correlation between switching times and alignment efficiency. Dielectric spectroscopy on all materials revealed up to three relaxation processes (α, α*, δ) above the glass transition temperature that were assigned to the dynamic glass transition of the polymer backbone and the fast and slow fluctuation of the chromophores around their long and short molecular axes. In the glassy state, occasionally two Arrhenius-type relaxations were observed that were identified as local motions of the butyl side group and the chromophore in its anisotropic environment. Both materials series showed a monotonic increase in Tg and the dynamic fragility with increasing chromophore concentration, which was explained by an increasing effect of physical crosslinking provided by the increasingly dense packed side groups.