A novel azobenzene polymer (PBA) containing push-pull electronic structure chromophore and chirality skeleton was synthesized based on the azo chromophore molecule 4-(4′-nitro-phenyl-diazenyl)-benzylamine (NPDB), chiral reagent L(-)-tartaric acid, acryloyl chloride, and methacrylate. The PBA was characterized by FT-IR, UV-vis spectroscopy and 1H NMR, differential scanning calorimeter (DSC), and thermogravimetric analysis (TGA). The PBA had high thermal stability up to its glass-transition temperature (Tg) of 108°C and 5% heat weight loss temperature of 188°C and optical rotation of +4.30°. The UV-induced trans/cis photoisomerization and reflex-isomerization behaviors were investigated for the PBA solution. The results indicated that the PBA solution could undergo photochromism after being irradiated by 365 nm UV light. In different polar solvents, PBA exhibits obvious solvatochromism. With the polarity of solvent enhancing, the maximum absorption peak shows red-shifting. The fluorescence emission spectra of PBA in DMF were explored and the fluorescence intensity decreased continuously as the polymer concentration increased. The optical parameters refractive index (n) and thermo-optic coefficient (dn/dT), the dielectric constant (ϵ), and its variation with temperature (dϵ/dT) of PBA were obtained. The thermo-optic coefficients were in the range of −3.2400 to −3.7514 × 10−4°C−1. The thermo-optic coefficients are one order of magnitude larger than those of the inorganic materials, such as SiO2 (1.1 × 10−5 °C−1 and LiNbO3 (4 × 10−5 °C−1) and were larger than those of organic material such as polystyrene (−1.23 × 10−4 °C−1) and PMMA (−1.20 × 10−4 °C−1). The results indicated that PBA could be used to develop a new digital optical switch with low driving power and optical communication.