Recently, much research has focused on photo-responsive materials since the responsiveness of materials is convenient to control. The application of photo-responsive materials in drug delivery is widely reported. The o-nitrobenzyl group and coumarin group are two common functional groups that can respond to lights. The o-nitrobenzyl group is photo-cleavable and responds to 365 nm UV lights, while the coumarin group is both photo-cleavable and photo-dimerizable and responds to lights with various wavelengths. The dimerization process of the coumarin group is reversible. Materials based on these two groups are also easy to be prepared. In this review, photo-responsive materials for drug delivery systems are introduced. The whole delivery strategy for each work will be described in detail. All delivery systems show good drug-releasing ability in light stimuli in vitro. Due to limited experiment data, the effectiveness of the delivery system in vivo cannot be determined. The near-infrared ray responsive coumarin group-based materials are more recommended due to the better penetrability of near-infrared rays.